From robert.virding@REDACTED Sun Jul 1 04:12:01 2012 From: robert.virding@REDACTED (Robert Virding) Date: Sun, 01 Jul 2012 03:12:01 +0100 (BST) Subject: [erlang-questions] Leex scanners and default token matching In-Reply-To: <0B878C56-EC7A-4DC2-B02A-A4D5866BCF63@gmail.com> Message-ID: The reason for the infinite loop is the macro definition: AOP = (\\+|-|\\*|/) The double \\ means you are quoting the '\' not the '+' and '-'. So that regex means: match one-or-more '+' or match '-' or match zero-or-more '*' <<<=== or match '/' This will match zero of any non-matching character so you get a match but no character will be consumed and the scanner will loop over the same character again. For matching characters this is not a problem as you always get the longest match which is always longer than the empty match. When you add your illegal regex this is what happens. Having regex which contain just '*' qualified regex is very dangerous as they can match the empty string and so create a loop. I know of no good way to handle this as it is not a bug, it is doing what you told it to do. The only way would be to disallow empty matches. Your string regex '([^''])*' looks a little strange. Robert ----- Original Message ----- > Well the 'solution' is probably as simple as filtering them out - not > quite as onerous as I'd imagined once I realised that any character > that isn't single quoted falls into a pretty small range. An extra > macro: > > ILLEGAL = ([^\s\w=><*/]) > > and then an extra Rule: > > {ILLEGAL}+ : {error, "Illegal Character(s): " ++ > TokenChars}. > > Still, it should be documented that Leex *can* produce a scanner that > is non-terminating. Even though there's nothing to say that it > shouldn't, whilst the Yecc documentation explains something about > the implementation that helps you understand how it works (LALR-1) > there is nothing in the leex docs to suggest that I should *beware* > of this. Luckily I decided to randomly generate input strings using > PropEr and quickly came across the issue in my implementation. > > What do other think about this? Should there be a caution in the > documentation explaining this potential situation? > > On 30 Jun 2012, at 12:31, Tim Watson wrote: > > > Hi all, > > > > I've got a simple Leex scanner, which appears to go into a > > non-terminating state for certain inputs, consuming 100% CPU and > > quickly eating up all available memory. I found this *very* > > surprising - should the generated scanner really be able to get > > itself into this state? Is there some way for me to provide a > > default rule that will execute if no other regex matches, so I can > > return {error, Reason} for this??? > > > > Below is a copy of the xrl file - running scanner:string("aa = !") > > will cause it to hang. Am I missing some obvious way of preventing > > this? > > > > Definitions. > > COMMA = [,] > > PARENS = [\(\)] > > L = [A-Za-z_\$] > > D = [0-9-] > > F = (\+|-)?[0-9]+\.[0-9]+((E|e)(\+|-)?[0-9]+)? > > HEX = 0x[0-9]+ > > WS = ([\000-\s]|%.*) > > S = ({COMMA}|{PARENS}) > > CMP = (=|>|>=|<|<=|<>) > > AOP = (\\+|-|\\*|/) > > > > Rules. > > > > LIKE : {token, {op_like, TokenLine, like}}. > > IN : {token, {op_in, TokenLine, in}}. > > AND : {token, {op_and, TokenLine, conjunction}}. > > OR : {token, {op_or, TokenLine, disjunction}}. > > NOT : {token, {op_not, TokenLine, negation}}. > > IS{WS}NULL : {token, {op_null, TokenLine, is_null}}. > > IS{WS}NOT{WS}NULL : {token, {op_null, TokenLine, not_null}}. > > BETWEEN : {token, {op_between, TokenLine, range}}. > > ESCAPE : {token, {escape, TokenLine, escape}}. > > {CMP} : {token, {op_cmp, TokenLine, > > atomize(TokenChars)}}. > > {AOP} : {token, {op_arith, TokenLine, > > atomize(TokenChars)}}. > > {L}({L}|{D})* : {token, {ident, TokenLine, TokenChars}}. > > '([^''])*' : {token, {lit_string, TokenLine, > > strip(TokenChars,TokenLen)}}. > > {S} : {token, > > {list_to_atom(TokenChars),TokenLine}}. > > {D}+ : {token, {lit_int, TokenLine, > > list_to_integer(TokenChars)}}. > > {F} : {token, {lit_flt, TokenLine, > > list_to_float(TokenChars)}}. > > {HEX} : {token, {lit_hex, TokenLine, > > hex_to_int(TokenChars)}}. > > {WS}+ : skip_token. > > > > Erlang code. > > > > strip(TokenChars,TokenLen) -> > > lists:sublist(TokenChars, 2, TokenLen - 2). > > > > hex_to_int([_,_|R]) -> > > {ok,[Int],[]} = io_lib:fread("~16u", R), > > Int. > > > > atomize(TokenChars) -> > > list_to_atom(TokenChars). > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From msegalis@REDACTED Sun Jul 1 13:36:11 2012 From: msegalis@REDACTED (Morgan Segalis) Date: Sun, 1 Jul 2012 13:36:11 +0200 Subject: [erlang-questions] gproc : 2 nodes out of sync behaviour. Message-ID: <0FCCE367-276F-409B-B22D-E905C413E73A@gmail.com> Hello everyone, I have 2 nodes which use gproc. Both are well connected to each other? But sometimes (doesn't happen really often, but it does) both server gets disconnected to each other, once their are connected again, gproc is out of sync. Here's what happen : 1- A is connected to B. 2- a new value X set by A is saw by B 3- a new value Y set by B is saw by A -------- they get disconnect for a second or two -------- 4- Clusters lost connection -------- they reconnect ---------- 5- Clusters regain connection 6- the old value X set by A is not saw anymore by B 7- the old value Y set by B is not saw anymore by B 8- a new value Z set by A is saw by B 9- a new value V set by B is not saw by A how come in "8" the new value Z set by A is saw by B and in "9" a new value V set by B is not saw by A ? I know that there is a leader, which is probably B, but I can't explain why new value are not seen symmetrically. what should I do for reconnecting correctly both cluster, so old value and new value are saw in both cluster again ? From ulf@REDACTED Sun Jul 1 13:49:02 2012 From: ulf@REDACTED (Ulf Wiger) Date: Sun, 1 Jul 2012 13:49:02 +0200 Subject: [erlang-questions] gproc : 2 nodes out of sync behaviour. In-Reply-To: <0FCCE367-276F-409B-B22D-E905C413E73A@gmail.com> References: <0FCCE367-276F-409B-B22D-E905C413E73A@gmail.com> Message-ID: <512AD519-5761-4EBB-B399-79A7455C77C5@feuerlabs.com> It's a feature of gproc, carefully crafted to make life more interesting. ;-) There is no resynch after netsplit in gproc, since gen_leader didn't use to have a way to handle netsplits. Still, there is no hook to inform the callback (gproc_dist) about what's happened. One way to deal with this, is to set -kernel dist_auto_connect false, and add a "backdoor ping" (e.g. over UDP). If you get a ping from a known node that's not in the nodes() list, you have a netsplit situation. You can then select which node(s) to restart. After restart, normal synch will ensue, and since the nodes never auto-connected, you will have concistency (but quite possibly data loss, of course). BR, Ulf W Ulf Wiger, Feuerlabs, Inc. http://www.feuerlabs.com 1 jul 2012 kl. 13:36 skrev Morgan Segalis : > Hello everyone, > > I have 2 nodes which use gproc. > Both are well connected to each other? > But sometimes (doesn't happen really often, but it does) both server gets disconnected to each other, once their are connected again, gproc is out of sync. > > Here's what happen : > 1- A is connected to B. > 2- a new value X set by A is saw by B > 3- a new value Y set by B is saw by A > -------- they get disconnect for a second or two -------- > 4- Clusters lost connection > -------- they reconnect ---------- > 5- Clusters regain connection > 6- the old value X set by A is not saw anymore by B > 7- the old value Y set by B is not saw anymore by B > 8- a new value Z set by A is saw by B > 9- a new value V set by B is not saw by A > > how come in "8" the new value Z set by A is saw by B and in "9" a new value V set by B is not saw by A ? > I know that there is a leader, which is probably B, but I can't explain why new value are not seen symmetrically. > what should I do for reconnecting correctly both cluster, so old value and new value are saw in both cluster again ? > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From watson.timothy@REDACTED Sun Jul 1 14:20:40 2012 From: watson.timothy@REDACTED (Tim Watson) Date: Sun, 1 Jul 2012 13:20:40 +0100 Subject: [erlang-questions] Leex scanners and default token matching In-Reply-To: References: Message-ID: Hi Robert, Thanks for the explanation, that makes sense now. I wasn't sure about the escape sequence, but clearly I had it wrong, as the match wants to be something more like ([\+\*-/]{1}), although clearly my ILLEGAL regex isn't account for the '+' which is wrong. On 1 Jul 2012, at 03:12, Robert Virding wrote: > The reason for the infinite loop is the macro definition: > > AOP = (\\+|-|\\*|/) > > The double \\ means you are quoting the '\' not the '+' and '-'. So that regex means: > > match one-or-more '+' Because what I'm quoting is the '\', presumably you mean 'match one or more (+) of '\' yes? > or > match '-' > or > match zero-or-more '*' <<<=== Yep, I see why that's screwed up - thanks! > or > match '/' > > This will match zero of any non-matching character so you get a match but no character will be consumed and the scanner will loop over the same character again. For matching characters this is not a problem as you always get the longest match which is always longer than the empty match. When you add your illegal regex this is what happens. > > Having regex which contain just '*' qualified regex is very dangerous as they can match the empty string and so create a loop. I know of no good way to handle this as it is not a bug, it is doing what you told it to do. The only way would be to disallow empty matches. > > Your string regex '([^''])*' looks a little strange. > Well it's meant to say something more like '([^']+)' [immediately following a single quote, match one or more of any character apart from a single quote, followed immediately after by a single quote] but I it was late at night! :) > Robert From essen@REDACTED Sun Jul 1 15:31:51 2012 From: essen@REDACTED (=?windows-1252?Q?Lo=EFc_Hoguin?=) Date: Sun, 01 Jul 2012 15:31:51 +0200 Subject: [erlang-questions] conditional pub-sub in gproc In-Reply-To: <0C235F84-F92B-414D-A6B3-AB8B23A9CF2D@feuerlabs.com> References: <4FDBB494.3000106@gmail.com> <4FDBC0CC.6080600@ninenines.eu> <4FDBC46D.3070305@ninenines.eu> <0C235F84-F92B-414D-A6B3-AB8B23A9CF2D@feuerlabs.com> Message-ID: <4FF05147.1030905@ninenines.eu> I've been using wide_await in production for a few days now. It works great. Thanks again! On 06/16/2012 02:58 AM, Ulf Wiger wrote: > > Good points. > > Please try out the latest version, which includes a few new functions: > > gproc:await(Node, Key, Timeout) > https://github.com/esl/gproc/blob/master/doc/gproc.md#await-3 > > gproc:wide_await(Nodes, Key, Timeout) > https://github.com/esl/gproc/blob/master/doc/gproc.md#wide_await3 > > gproc:nb_wait(Node, Key) > https://github.com/esl/gproc/blob/master/doc/gproc.md#nb_wait-2 > > gproc:cancel_wait(Node, Key, Ref) > https://github.com/esl/gproc/blob/master/doc/gproc.md#cancel_wait-3 > > > Note that await/3 also returns the Value, so can be throught of as a > distributed get_value() function. > > BR, > Ulf W > > > On 15 Jun 2012, at 16:25, Lo?c Hoguin wrote: > >> Sorry I should have been clearer. >> >> I'm using global gproc to locate processes that may be on any node of the cluster. But sometimes I run into gen_leader's issues (for example if a node crashes) so I was wondering if I could manage still using gproc for my purposes without gen_leader. >> >> Currently it's easy to broadcast information to other local gprocs but not so much to retrieve remote data, similar to what get_value/2 or where/1 would do in a distributed context. >> >> It would be nice to be able to have a cluster of local gprocs and easily access them from any remote node, similar to how bcast works for broadcasting. >> >> On 06/16/2012 01:18 AM, Ulf Wiger wrote: >>> >>> On 15 Jun 2012, at 16:10, Lo?c Hoguin wrote: >>> >>>> That removes some functionality though, doesn't it. Like :get_value/1 and where/1. How would you suggest using these without global gproc? >>> >>> Uhm, those functions work perfectly fine in a local context? >>> >>> Can you expand on that? >>> >>> Global gproc (gproc_dist) is disabled by default. >>> My assumption is that most people leave it that way. >>> >>> BR, >>> Ulf W >>> >>> Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. >>> http://feuerlabs.com >>> >>> >>> >> >> >> -- >> Lo?c Hoguin >> Erlang Cowboy >> Nine Nines >> >> > > Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. > http://feuerlabs.com > > > -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From msegalis@REDACTED Sun Jul 1 17:01:46 2012 From: msegalis@REDACTED (Morgan Segalis) Date: Sun, 1 Jul 2012 17:01:46 +0200 Subject: [erlang-questions] gproc : 2 nodes out of sync behaviour. In-Reply-To: <512AD519-5761-4EBB-B399-79A7455C77C5@feuerlabs.com> References: <0FCCE367-276F-409B-B22D-E905C413E73A@gmail.com> <512AD519-5761-4EBB-B399-79A7455C77C5@feuerlabs.com> Message-ID: <7E913273-5E1F-476A-A061-486E8EA11908@gmail.com> Gproc may make life more interesting, but right now, I certainly know that gproc made my life easier, thanks to you :-) (Sorry for my late answer, but I wanted to think about the solution before posting it) When you say : " since gen_leader didn't use to have a way to handle netsplits." 1- this means that gen_leader handles netsplits now ? 2- If so, gproc_dist would only need a way to know when a netsplits happened, right ? What about this solution ? (if previous 1 & 2 are true) =============================== nodemonitor.erl =================================== -module(nodemonitor). -behaviour(gen_server). -record(nodemonitor, {nodes=none}). -export([start_link/0, init/1, handle_call/3, handle_cast/2, terminate/2, code_change/3, handle_info/2]). start_link() -> gen_server:start_link(?MODULE, [], []). init([]) -> net_kernel:monitor_nodes(true, [{node_type, visible}]), {ok, #nodemonitor{nodes=dict:new()}}. handle_call(_, _, NM) -> {noreply, NM}. handle_cast(_, NM) -> {noreply, NM}. handle_info({nodeup, Node, Params}, NM) -> case dict:find(Node, NM#nodemonitor.nodes) of {ok, disconnected} -> %% Let know gproc_dist about this netsplit io:fwrite("netsplit occured: ~p~n", [Node]); {ok, connected} -> io:fwrite("Error occured: ~p~n", [Node]); error -> io:fwrite("New Node detected: ~p~n", [Node]) end, Dict = dict:store(Node, connected, NM#nodemonitor.nodes), {noreply, NM#nodemonitor{nodes=Dict}}; handle_info({nodedown, Node, Params}, NM) -> Dict = dict:store(Node, disconnected, NM#nodemonitor.nodes), {noreply, NM#nodemonitor{nodes=Dict}}. code_change(_, NM, _) -> {noreply, NM}. terminate(normal, _) -> ok. ----------------------------------------------------------------------------------------------------------------------------------------- I'm surely not pretending that this solution would not have been thought by you, so there is something I don't get. Do you think it would be possible to do something about it ? 3 - if 1 & 2 are not true then would it be possible, in your opinion, to stop & start gproc and re-register every value so every cluster are in sync again ? Le 1 juil. 2012 ? 13:49, Ulf Wiger a ?crit : > It's a feature of gproc, carefully crafted to make life more interesting. ;-) > > There is no resynch after netsplit in gproc, since gen_leader didn't use to have a way to handle netsplits. Still, there is no hook to inform the callback (gproc_dist) about what's happened. > > One way to deal with this, is to set -kernel dist_auto_connect false, and add a "backdoor ping" (e.g. over UDP). If you get a ping from a known node that's not in the nodes() list, you have a netsplit situation. You can then select which node(s) to restart. After restart, normal synch will ensue, and since the nodes never auto-connected, you will have concistency (but quite possibly data loss, of course). > > BR, > Ulf W > > Ulf Wiger, Feuerlabs, Inc. > http://www.feuerlabs.com > > 1 jul 2012 kl. 13:36 skrev Morgan Segalis : > >> Hello everyone, >> >> I have 2 nodes which use gproc. >> Both are well connected to each other? >> But sometimes (doesn't happen really often, but it does) both server gets disconnected to each other, once their are connected again, gproc is out of sync. >> >> Here's what happen : >> 1- A is connected to B. >> 2- a new value X set by A is saw by B >> 3- a new value Y set by B is saw by A >> -------- they get disconnect for a second or two -------- >> 4- Clusters lost connection >> -------- they reconnect ---------- >> 5- Clusters regain connection >> 6- the old value X set by A is not saw anymore by B >> 7- the old value Y set by B is not saw anymore by B >> 8- a new value Z set by A is saw by B >> 9- a new value V set by B is not saw by A >> >> how come in "8" the new value Z set by A is saw by B and in "9" a new value V set by B is not saw by A ? >> I know that there is a leader, which is probably B, but I can't explain why new value are not seen symmetrically. >> what should I do for reconnecting correctly both cluster, so old value and new value are saw in both cluster again ? >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions From erlang@REDACTED Sun Jul 1 18:36:11 2012 From: erlang@REDACTED (Stefan Marr) Date: Sun, 1 Jul 2012 18:36:11 +0200 Subject: [erlang-questions] RACES'12: Relaxing Synchronization for Multicore and Manycore Scalability, SPLASH'12 Workshop Message-ID: ============================================================================== Call for Participation R A C E S 2 0 1 2 Relaxing Synchronization for Multicore and Manycore Scalability Workshop Co-located with SPLASH in Tucson, Arizona Sunday, October 21 Submission deadline: Monday, August 6 http://soft.vub.ac.be/races/ ============================================================================== Massively-parallel systems are coming: core counts keep rising - whether conventional cores as in multicore and manycore systems, or specialized cores as in GPUs. Conventional wisdom has been to utilize this parallelism by reducing synchronization to the minimum required to preserve determinism - in particular, by eliminating data races. However, Amdahl's law implies that on highly-parallel systems even a small amount of synchronization that introduces serialization will limit scaling. Thus, we are forced to confront the trade-off between synchronization and the ability of an implementation to scale performance with the number of processors: synchronization inherently limits parallelism. This workshop focuses on harnessing parallelism by limiting synchronization, even to the point where programs will compute inconsistent or approximate rather than exact answers. Organizers: Andrew P. Black, Portland State University Theo D'Hondt, Vrije Universiteit Brussel Doug Kimelman, IBM Thomas J. Watson Research Center Martin Rinard, MIT CSAIL David Ungar, IBM Thomas J. Watson Research Center Theme and Topics ---------------- A new school of thought is arising: one that accepts and even embraces nondeterminism (including data races), and in return is able to dramatically reduce synchronization, or even eliminate it completely. However, this approach requires that we leave the realm of the certain and enter the realm of the merely probable. How can we cast aside the security of correctness, the logic of a proof, and adopt a new way of thinking, where answers are good enough but not certain, and where many processors work together in parallel without quite knowing the states that the others are in? We may need some amount of synchronization, but how much? Or better yet, how little? What mental tools and linguistic devices can we give programmers to help them adapt to this challenge? This workshop focuses on these questions and related ones: harnessing parallelism by limiting synchronization, even to the point where programs will compute inconsistent or approximate rather than exact answers. This workshop aims to bring together researchers who, in the quest for scalability, have been exploring the limits of how much synchronization can be avoided. We invite submissions on any topic related to the theme of the workshop, pro or con. We want to hear from those who have experimented with formalisms, algorithms, data structures, programming languages, and mental models that push the limits. In addition, we hope to hear from a few voices with wilder ideas: those who may not have reduced their notions to practice yet, but who have thoughts that can inspire us as we head towards this yet-uncertain future. For example, biology may yield fruitful insights. The ideal presentation for this workshop will focus on a grand idea, but will be backed by some experimental result. Submission ---------- Authors are invited to submit short position papers, technical papers, or experience reports. Submissions may range from a single paragraph to as long as desired, but the committee can only commit to reading one full page. Nonetheless, we expect that in many cases reviewers will read farther than that. Submissions should be formatted according to the ACM SIG Proceedings style at http://www.acm.org/sigs/publications/proceedings-templates and should be submitted via EasyChair at http://www.easychair.org/conferences/?conf=races2012 in PDF format. PLEASE NOTE: All submissions (except for those retracted by their authors) will be posted on the workshop website, along with reviews, which will be signed by the reviewers, and a rating assigned by the program committee. Further, the submissions to be presented at the workshop will be selected by a vote of all registered attendees. As well, submissions to be published in an official proceedings will be selected by the program committee. Please see the sections below concerning the rationale and details for this process. Program Committee ----------------- Andrew P. Black, Portland State University Yvonne Coady, University of Victoria Tom Van Cutsem, Vrije Universiteit Brussel Theo D'Hondt, Vrije Universiteit Brussel Phil Howard, Portland State University Doug Kimelman, IBM Thomas J. Watson Research Center Eddie Kohler, Harvard SEAS Jim Larus, Microsoft Research Stefan Marr, Vrije Universiteit Brussel Tim Mattson, Intel Paul McKenney, IBM Hannes Payer, University of Salzburg Dan Prenner, IBM Lakshmi Renganarayana, IBM David Ungar, IBM Thomas J. Watson Research Center - TBC - Important Dates --------------- August 6 Submission deadline. August 29 Reviews sent to authors. September 3 Last date for retraction by authors. September 4 Papers, reviews, ratings posted on web site. Voting opens. September 11 Voting closes. September 14 Notification of papers accepted for presentation and/or publication. August 21 SPLASH early registration deadline. October 21 Workshop. mid-November Camera-ready copy due for papers selected for proceedings. Goals and Outcomes ------------------ We will consider the workshop a success if attendees come away with new insights into fundamental principles, and new ideas for algorithms, data structures, programming languages, and mental models, leading to improving scaling by limiting synchronization, even to the point where programs will compute inconsistent or approximate rather than exact answers. The goal of this workshop is both to influence current programming practice and to initiate the coalescence of a new research community giving rise to a new subfield within the general area of concurrent and parallel programming. Results generated by the workshop will be made persistent via the workshop website and possibly via the ACM Digital Library. The RACES 2012 Review Process and Workshop Presentation Selection Process ========================================================================= David Ungar IBM Thomas J. Watson Research Center PC Chair for Workshop Presentations Technology has changed the economic tradeoffs that once shaped the reviewing process. It has become cheap and easy to share submissions, reviews and the preferences of the attendees. What remains scarce is the number of hours in a day, and as a consequence the time we have in our workshop in which to learn and share with each other. I believe that this change in the balance of factors affords us the opportunity to significantly improve the review and selection processes. Sadly, all too often, those who spend their precious time attending a workshop are not served as well they could be with respect to enlightenment, thought provoking discussions, and being challenged by new ideas. The fault lies not in the people who generously donate their time to serve on program committees and do external reviews. Rather, the fault lies in the process itself. The very notion of acceptance by committee forces us to boil a rich stew of reactions, insights, and opinions, down to a single carrot. As a result, it is common for PC members to come away from a meeting feeling that either some fraud will be perpetrated on the audience by a fundamentally flawed paper, or, more often, feeling that a sin of omission will be committed on the audience by the suppression of a significant but controversial new idea. Sometimes instead of a carrot we get a lump of gristle. There are other, lesser, flaws in this process. Although reviewer anonymity protects negative reviewers from resentment and reprisal, all too often it prevents an open debate that would promote mutual understanding. Further, in some cases anonymity allows a reviewer to cast aspersions on authors without being accountable. Finally, we fail to take maximal advantage of the time and effort spent in creating insightful reviews when we withhold them from the audience. Attendees and readers could benefit from expert reactions as they try to glean the wisdom embedded in the authors' papers. In this workshop, we have an opportunity to try a different process, one that we hope will serve all parties better: All reviews will be signed, all submissions and reviews will be posted on the web (unless an author chooses to retract a submission), and the attendees will be the ones selecting which papers will be presented. Here are the details: --------------------- At least three committee members will review each submission, and each review will be signed. Once all the reviews for a submission are in, they will be sent to the author, who can decide to retract the paper if so desired. Then, all submissions (except any that are retracted) will be posted on the workshop website, along with all reviews and a net score determined for each submission by the program committee. At this point, prior to the workshop, all registered attendees will be invited to read the submissions and the reviews, and vote on which of the papers they want to see presented. Of course, an attendee who so wishes will be free to merely vote according to the recommendation of the PC, or to not vote and to accept the wisdom of the rest of the attendees. But the important point remains: it will be those who will be spending the time in the room who get to decide how that time is spent. Please note that a submission being posted on the workshop website and/or presented at the workshop are not intended to constitute prior publication for purposes of publishing in other workshops, major conferences, or journals. This process is a grand experiment, designed to exploit the technologies we Computer Scientists have created, in order to better serve the advancement of Computer Science. We hope that its potential excites you as much as it excites us! The RACES 2012 Published Proceedings Paper Selection Process ============================================================ Theo D'Hondt Vrije Universiteit Brussel PC Chair for Proceedings Papers We understand that many submitters may want to publish their paper in an official proceedings in addition to having it posted on the workshop website. In order to satisfy that desire, we will publish a proceedings via the ACM Digital Library. To satisfy ACM DL selectivity requirements, a separate and more conventional process will be employed for selecting papers to be included in the published proceedings: Even though all submissions will be posted on the workshop website (unless retracted by the author), the program committee will select a smaller number of papers to be included in the published proceedings based on the signed and posted reviews. Authors of the selected papers will be asked to submit revised and extended papers mid-November, taking into account the reviews and the publisher's guidelines. Page limits for the revised and extended papers to be included in the published proceedings are anticipated to be 10 pages for research papers, and 5 pages for position papers. Please note that inclusion in the ACM Digital Library published proceedings may well be considered to be a prior publication for purposes of publication in other workshops, major conferences, or journals. For that reason, authors may choose to decline to have their submission included in the published proceedings, even if it was presented at the workshop. For questions please contact: races@REDACTED For updates, follow us on Twitter: @races_workshop ============================================================================== From ulf@REDACTED Sun Jul 1 19:15:32 2012 From: ulf@REDACTED (Ulf Wiger) Date: Sun, 1 Jul 2012 19:15:32 +0200 Subject: [erlang-questions] gproc : 2 nodes out of sync behaviour. In-Reply-To: <7E913273-5E1F-476A-A061-486E8EA11908@gmail.com> References: <0FCCE367-276F-409B-B22D-E905C413E73A@gmail.com> <512AD519-5761-4EBB-B399-79A7455C77C5@feuerlabs.com> <7E913273-5E1F-476A-A061-486E8EA11908@gmail.com> Message-ID: <48FA97FF-3D3C-4099-9E02-1DB292EE1B41@feuerlabs.com> On 1 Jul 2012, at 17:01, Morgan Segalis wrote: > Gproc may make life more interesting, but right now, I certainly know that gproc made my life easier, thanks to you :-) Why, thanks. :) > When you say : " since gen_leader didn't use to have a way to handle netsplits." > > 1- this means that gen_leader handles netsplits now ? Yes, later versions of gen_leader, e.g. Garrett Smith's fork, [1], have some point in the code where they detect a netsplit and call a new election. [1] https://github.com/garret-smith/gen_leader_revival/blob/master/src/gen_leader.erl#L874 I picked Garrett's purely on the basis of the Github network graph, seeing as it is farthest to the right, and has recent netsplit-related commits. It's not a value judgement based on having tried them or deeply studied the code. > 2- If so, gproc_dist would only need a way to know when a netsplits happened, right ? Well, a few things may need to be addressed: 1. Currently, gen_leader simply calls a new election and decides who's the new leader. This *may* be sufficient - probably is in many cases - but in some cases, what's happened is that one node has become isolated, and while gen_leader may think it has the best claim to the throne, the people may disagree. The choice that might cause the least disruption may be if the application reads the situation and picks the node that is actually busy servicing customers. A possibility could then be that gen_leader first asks the applications: "these leaders are in contention; do you have a favorite?" Answering this question with anything other than 'no' can risk destabilizing the system unless the two candidates disagree. Gen_leader *might* be able to detect t his, and call an election instead. 2. Next problem is the reconciliation. There are essentially two ways: a) Once a new leader is elected, you call elected() with some info signaling that there has been a netsplit, and rely on it to take care of things. It might not be possible to reuse elected() (not in a position to check right now). b) Enter a protected state machine, similar to the election state machine, and allow the leaders to signal each other asynchronously until they agree. The two may be able to combine, by allowing the elected() callback order a reconciliation procedure. I'm sure the hackers at e.g. Basho could offer good insights into this. BR, Ulf W > 3 - if 1 & 2 are not true then would it be possible, in your opinion, to stop & start gproc and re-register every value so every cluster are in sync again ? > > Le 1 juil. 2012 ? 13:49, Ulf Wiger a ?crit : > >> It's a feature of gproc, carefully crafted to make life more interesting. ;-) >> >> There is no resynch after netsplit in gproc, since gen_leader didn't use to have a way to handle netsplits. Still, there is no hook to inform the callback (gproc_dist) about what's happened. >> >> One way to deal with this, is to set -kernel dist_auto_connect false, and add a "backdoor ping" (e.g. over UDP). If you get a ping from a known node that's not in the nodes() list, you have a netsplit situation. You can then select which node(s) to restart. After restart, normal synch will ensue, and since the nodes never auto-connected, you will have concistency (but quite possibly data loss, of course). >> >> BR, >> Ulf W >> >> Ulf Wiger, Feuerlabs, Inc. >> http://www.feuerlabs.com >> >> 1 jul 2012 kl. 13:36 skrev Morgan Segalis : >> >>> Hello everyone, >>> >>> I have 2 nodes which use gproc. >>> Both are well connected to each other? >>> But sometimes (doesn't happen really often, but it does) both server gets disconnected to each other, once their are connected again, gproc is out of sync. >>> >>> Here's what happen : >>> 1- A is connected to B. >>> 2- a new value X set by A is saw by B >>> 3- a new value Y set by B is saw by A >>> -------- they get disconnect for a second or two -------- >>> 4- Clusters lost connection >>> -------- they reconnect ---------- >>> 5- Clusters regain connection >>> 6- the old value X set by A is not saw anymore by B >>> 7- the old value Y set by B is not saw anymore by B >>> 8- a new value Z set by A is saw by B >>> 9- a new value V set by B is not saw by A >>> >>> how come in "8" the new value Z set by A is saw by B and in "9" a new value V set by B is not saw by A ? >>> I know that there is a leader, which is probably B, but I can't explain why new value are not seen symmetrically. >>> what should I do for reconnecting correctly both cluster, so old value and new value are saw in both cluster again ? >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions > Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. http://feuerlabs.com From ulf@REDACTED Sun Jul 1 19:24:13 2012 From: ulf@REDACTED (Ulf Wiger) Date: Sun, 1 Jul 2012 19:24:13 +0200 Subject: [erlang-questions] gproc : 2 nodes out of sync behaviour. In-Reply-To: <48FA97FF-3D3C-4099-9E02-1DB292EE1B41@feuerlabs.com> References: <0FCCE367-276F-409B-B22D-E905C413E73A@gmail.com> <512AD519-5761-4EBB-B399-79A7455C77C5@feuerlabs.com> <7E913273-5E1F-476A-A061-486E8EA11908@gmail.com> <48FA97FF-3D3C-4099-9E02-1DB292EE1B41@feuerlabs.com> Message-ID: <27271B5D-4C73-4954-B527-1B50BE2125A4@feuerlabs.com> On 1 Jul 2012, at 19:15, Ulf Wiger wrote: > Answering this question with anything other than 'no' can risk destabilizing the system unless the two candidates disagree. s/disagree/agree of course. :) BR, Ulf Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. http://feuerlabs.com From robert.virding@REDACTED Mon Jul 2 02:48:52 2012 From: robert.virding@REDACTED (Robert Virding) Date: Mon, 02 Jul 2012 01:48:52 +0100 (BST) Subject: [erlang-questions] Leex scanners and default token matching In-Reply-To: Message-ID: It would probably be a good idea to catch that and flag it as an error, if I can work out a good way to do it as for some cases it might work. Robert ----- Original Message ----- > Hi Robert, > > Thanks for the explanation, that makes sense now. I wasn't sure about > the escape sequence, but clearly I had it wrong, as the match wants > to be something more like ([\+\*-/]{1}), although clearly my ILLEGAL > regex isn't account for the '+' which is wrong. > > On 1 Jul 2012, at 03:12, Robert Virding wrote: > > > The reason for the infinite loop is the macro definition: > > > > AOP = (\\+|-|\\*|/) > > > > The double \\ means you are quoting the '\' not the '+' and '-'. So > > that regex means: > > > > match one-or-more '+' > > Because what I'm quoting is the '\', presumably you mean 'match one > or more (+) of '\' yes? > > > or > > match '-' > > or > > match zero-or-more '*' <<<=== > > Yep, I see why that's screwed up - thanks! > > > or > > match '/' > > > > This will match zero of any non-matching character so you get a > > match but no character will be consumed and the scanner will loop > > over the same character again. For matching characters this is not > > a problem as you always get the longest match which is always > > longer than the empty match. When you add your illegal regex this > > is what happens. > > > > Having regex which contain just '*' qualified regex is very > > dangerous as they can match the empty string and so create a loop. > > I know of no good way to handle this as it is not a bug, it is > > doing what you told it to do. The only way would be to disallow > > empty matches. > > > > Your string regex '([^''])*' looks a little strange. > > > > Well it's meant to say something more like '([^']+)' [immediately > following a single quote, match one or more of any character apart > from a single quote, followed immediately after by a single quote] > but I it was late at night! :) > > > Robert > > From ok@REDACTED Mon Jul 2 03:19:49 2012 From: ok@REDACTED (Richard O'Keefe) Date: Mon, 2 Jul 2012 13:19:49 +1200 Subject: [erlang-questions] Why are records not 'export'-able? In-Reply-To: <8367670F-7092-4618-ADF7-EF0843BF65E8@dieswaytoofast.com> References: <8367670F-7092-4618-ADF7-EF0843BF65E8@dieswaytoofast.com> Message-ID: On 30/06/2012, at 2:06 AM, Mahesh Paolini-Subramanya wrote: > If I have a record I use in just one module, I declare it in that module. > If I have a record I use across a number of modules, I declare it in an included file. > If I have a record that I use across a number of applications, I end up declaring it once in each application. Er, what stops you using the same -include file in more than one application? If I found myself wanting to use the same -record in more than one application, I'd lie down until the feeling wore off. More precisely, I'd definitely want to hide that inside some sort of access module. > > The simple-minded me says "Hmf. If I could just have a '-export_record' declaration, then I could do some fun things like '-record(Name, Application:Record)' But there is no _thing_ for it to bind to. Records are just a kind of macro that the compiler understands. With frames, there'd be no -record declaration in the first place, you'd just use the data like you can just use a list or tuple. With abstract patterns, you'd import them from the defining module the way you'd import ordinary functions from their defining module. Records are *neither* a data type *nor* something function-like. Consider the following module: -module(foo). -export([bar/2]). -record(ugh, {ugh}). bar(i, #ugh{ugh=X}) -> #ugh{ugh=X+1}; bar(d, {ugh,Y}) -> {ugh,Y-1}. To me, it looks like a very bad idea to use both record syntax and plain tuple syntax for the same kind of data. I think that if there is a -record declaration for a record name, and there is also any tuple with the same name as its first element, the compiler should warn about that. Currently it doesn't. From sam@REDACTED Mon Jul 2 04:06:58 2012 From: sam@REDACTED (Sam Bobroff) Date: Mon, 2 Jul 2012 12:06:58 +1000 Subject: [erlang-questions] Bug introduced in R15B01 gen_server.erl? Message-ID: Hi All, After upgrading to R15B01, one of my gen_servers began crashing during start up with {error, timeout_value}. The server is started via gen_server:enter_loop and is globally registerd using a name of this form {global, server_name}. The reason seems to be these lines in gen_server.erl, which look like a cut-and-paste error: enter_loop(Mod, Options, State, ServerName = {Scope, _}) when Scope == local; Scope == local -> enter_loop(Mod, Options, State, ServerName, infinity); Should one of the guards be testing for global? (I assume enter_loop isn't heavily used!) Cheers, Sam. -------------- next part -------------- An HTML attachment was scrubbed... URL: From gustav@REDACTED Mon Jul 2 09:24:39 2012 From: gustav@REDACTED (Gustav Simonsson) Date: Mon, 2 Jul 2012 09:24:39 +0200 Subject: [erlang-questions] Bug introduced in R15B01 gen_server.erl? In-Reply-To: References: Message-ID: <4FF14CB7.9040403@erlang.org> Hi Sam, This is indeed a bug. We will create an internal patch for this. Thank you for reporting it! Regards, Gustav Simonsson Erlang/OTP team On 2012-07-02 04:06, Sam Bobroff wrote: > Hi All, > > After upgrading to R15B01, one of my gen_servers began crashing during > start up with {error, timeout_value}. The server is started via > gen_server:enter_loop and is globally registerd using a name of this > form {global, server_name}. > > The reason seems to be these lines in gen_server.erl, which look like > a cut-and-paste error: > > enter_loop(Mod, Options, State, ServerName = {Scope, _}) > when Scope == local; Scope == local -> > enter_loop(Mod, Options, State, ServerName, infinity); > > Should one of the guards be testing for global? > > (I assume enter_loop isn't heavily used!) > > Cheers, > Sam. > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From bourinov@REDACTED Mon Jul 2 12:29:32 2012 From: bourinov@REDACTED (Max Bourinov) Date: Mon, 2 Jul 2012 14:29:32 +0400 Subject: [erlang-questions] How would you do it? Message-ID: Hi Erlangers, I need to build a ranking system for online game. I think I will have a ranking gen_server that will accept cast messages like this: {score, Player :: profile(), Score :: integer()}. So, the question is what would be most appropriate data structure if: 1. I have to rank about 50 000 - 100 000 different players. 2. On each score message I have to re-sort whole ranking table. 3. It must be very cheap to get: 1. top 100 players 2. player's rating +/- 10 players about current player 4. I expect about 20-50 score messages per seconds 5. Size of score message is about 4KB (profile takes most of the space). Any ideas or suggestions are welcome! Best regards, Max -------------- next part -------------- An HTML attachment was scrubbed... URL: From f@REDACTED Mon Jul 2 13:06:05 2012 From: f@REDACTED (Francesco Mazzoli) Date: Mon, 02 Jul 2012 12:06:05 +0100 Subject: [erlang-questions] How would you do it? In-Reply-To: References: Message-ID: <87fw9as9r6.wl%f@mazzo.li> At Mon, 2 Jul 2012 14:29:32 +0400, Max Bourinov wrote: > 1. I have to rank about 50 000 - 100 000 different players. > 2. On each score message I have to re-sort whole ranking table. > 3. It must be very cheap to get: > 1. top 100 players > 2. player's rating +/- 10 players about current player > 4. I expect about 20-50 score messages per seconds > 5. Size of score message is about 4KB (profile takes most of the space). > > Any ideas or suggestions are welcome! A priority search queue will fit your bill. You can to 2 in and 3.1 in logarithmic time, and 3.1 in constant time. I'm not aware of an Erlang implementation, here is an Haskell module with link to a paper with implementation details: http://hackage.haskell.org/packages/archive/PSQueue/1.1/doc/html/Data-PSQueue.html . If you know some Haskell, this is an - untested - stub that will do what you want: -------------8<----------------------------------- import Data.PSQueue as PSQ type Ranking = PSQ Player Score newRanking :: Ranking newRanking = PSQ.empty updateScore :: Player -> Score -> Ranking -> Ranking updateScore player score rank = case PSQ.lookup player rank of Nothing -> PSQ.insert player score rank Just score' -> let score'' = magic to get new score in PSQ.insert player score'' rank topNPlayers :: Int -> Ranking -> [Player] topNPlayers n = take n . PSQ.toDescList worsePlayers :: Player -> Int -> Ranking -> Maybe [Binding Player Score] worsePlayers player n rank = case PSQ.lookup player rank of Nothing -> Nothing Just score -> Just (take n (PSQ.atMost score rank)) ------------->8----------------------------------- For some reason you don't have the dual of `atMost' which returns the items with a priority *greater* then the one given. I'd be surprised if you can't implement that. -- Francesco * Often in error, never in doubt From f@REDACTED Mon Jul 2 13:44:37 2012 From: f@REDACTED (Francesco Mazzoli) Date: Mon, 02 Jul 2012 12:44:37 +0100 Subject: [erlang-questions] How would you do it? In-Reply-To: <87fw9as9r6.wl%f@mazzo.li> References: <87fw9as9r6.wl%f@mazzo.li> Message-ID: <87ehous7yy.wl%f@mazzo.li> At Mon, 02 Jul 2012 12:06:05 +0100, Francesco Mazzoli wrote: > I'm not aware of an Erlang implementation, here is an Haskell module > with link to a paper with implementation details: > http://hackage.haskell.org/packages/archive/PSQueue/1.1/doc/html/Data-PSQueue.html . Mpf, I noticed that the paper linked there requires logging in CiteSeer, thankfully a quick Googling lead to a direct source: www.cs.ox.ac.uk/ralf.hinze/publications/ICFP01.pdf -- Francesco * Often in error, never in doubt From icfp.publicity@REDACTED Mon Jul 2 13:46:09 2012 From: icfp.publicity@REDACTED (Wouter Swierstra) Date: Mon, 2 Jul 2012 13:46:09 +0200 Subject: [erlang-questions] ICFP 2012: Call for participation Message-ID: ===================================================================== Call for Participation The 17th ACM SIGPLAN International Conference on Functional Programming (ICFP 2012) and affiliated events http://www.icfpconference.org/icfp2012/ Copenhagen, Denmark, Sep 9-15, 2012 ===================================================================== ICFP provides a forum for researchers and developers to hear about the latest work on the design, implementations, principles, and uses of functional programming. The conference covers the entire spectrum of work, from practice to theory, including its peripheries. A full week dedicated to functional programming: 1 conference, 1 symposium, 9 workshops, 8 tutorials, programming contest results, student research contest * Accepted Papers: http://www.icfpconference.org/icfp2012/accepted.html * Local arrangements (including travel and accommodation): http://icfpconference.org/icfp2012/local.html Conference hotel reservation cutoff: July 9, 2012 * Registration is available via: http://icfpconference.org/icfp2012/ Electronic registration will open shortly. * Follow @icfp_conference on twitter for the latest news: http://twitter.com/#!/icfp_conference There are several events affiliated with ICFP: September 9 Workshop on Cross-paradigm Language Design and Implementation Workshop on Generic Programming Workshop on Higher-Order Programming with Effects Workshop on Logical Frameworks and Meta-languages: Theory and Practice September 10-12 ICFP - main conference September 13 Commercial Users of Functional Programming ? Day 1 (CUFP Tutorials) Haskell Symposium Workshop on ML September 14 Commercial Users of Functional Programming ? Day 2 (CUFP Tutorials) Erlang Workshop Haskell Implementors' Workshop OCaml Users and Developers Workshop September 15 Commercial Users of Functional Programming ? Day 3 (CUFP Talks) Workshop on Functional High-Performance Computing Tutorial on Compiler Construction in Haskell Tutorial on the Grammatical Framework Conference organizers: * General Chair: Peter Thiemann, University of Freiburg * Program Chair: Robby Findler, Northwestern University * Local Arrangements Chair: Fritz Henglein, University of Copenhagen * Industrial Relations Chair: Andy Adams-Moran, Galois * Workshop Co-Chairs: Patrik Jansson, Chalmers University of Technology Gabriele Keller, University of New South Wales * Programming Contest Chair: Edwin Brady, University of St. Andrews Kevin Hammond, University of St. Andrews * Publicity Chair: Wouter Swierstra, Utrecht University * Video Chair: Malcolm Wallace, Standard Chartered Bank * Student Research Competition Chair: Doaitse Swierstra, Utrecht University ===================================================================== From fritchie@REDACTED Mon Jul 2 14:39:43 2012 From: fritchie@REDACTED (Scott Lystig Fritchie) Date: Mon, 02 Jul 2012 07:39:43 -0500 Subject: [erlang-questions] SMP for IO-bound applications In-Reply-To: Message of "Thu, 21 Jun 2012 20:37:44 +0200." Message-ID: <201207021239.q62CdhGl029106@snookles.snookles.com> >>> Erisa Dervishi wrote: ed> As part of my studies, I have recently been doing some performance ed> evaluations on Erlang SMP improvements for IO-bound applications. ed> [...] ed> The tests I run have these parameters: Hi, Erisa. Sorry to jump into this email thread so late. I didn't see any mention of a couple of parameters in your summary, so I'm wondering if you've used or studied the effect of these "erl" command line flags: +A n (where N is an integer > 0) +K true The first flag tells the VM to use the asynchronous I/O thread pool for local disk I/O. The second uses a different socket ready/activity mechanism rather than the old-and-frequently-slower select(2) system call. (See http://www.erlang.org/doc/man/erl.html for summary.) -Scott From dmkolesnikov@REDACTED Mon Jul 2 14:43:49 2012 From: dmkolesnikov@REDACTED (dmitry kolesnikov) Date: Mon, 2 Jul 2012 15:43:49 +0300 Subject: [erlang-questions] How would you do it? In-Reply-To: References: Message-ID: <2304181223424698845@unknownmsgid> Hello, You have to split your data structure. The player profile has to go into hash table (e.g. Ets) and ranking into separate index. It should give you better performance since your player profile is big. The first candidate for index is gb_trees. It should be good enough to cover 3.1 but 3.2 requires a range search, which is available in my Branch of gbt ( http://github.com/fogfish/feta) If you are not fine with gbt performance then you have to shard your scores into multiple trees. Best Regards, Dmitry >-|-|-*> On 2.7.2012, at 13.30, Max Bourinov wrote: Hi Erlangers, I need to build a ranking system for online game. I think I will have a ranking gen_server that will accept cast messages like this: {score, Player :: profile(), Score :: integer()}. So, the question is what would be most appropriate data structure if: 1. I have to rank about 50 000 - 100 000 different players. 2. On each score message I have to re-sort whole ranking table. 3. It must be very cheap to get: 1. top 100 players 2. player's rating +/- 10 players about current player 4. I expect about 20-50 score messages per seconds 5. Size of score message is about 4KB (profile takes most of the space). Any ideas or suggestions are welcome! Best regards, Max _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From garret.smith@REDACTED Mon Jul 2 16:31:46 2012 From: garret.smith@REDACTED (Garret Smith) Date: Mon, 2 Jul 2012 07:31:46 -0700 Subject: [erlang-questions] gproc : 2 nodes out of sync behaviour. In-Reply-To: <27271B5D-4C73-4954-B527-1B50BE2125A4@feuerlabs.com> References: <0FCCE367-276F-409B-B22D-E905C413E73A@gmail.com> <512AD519-5761-4EBB-B399-79A7455C77C5@feuerlabs.com> <7E913273-5E1F-476A-A061-486E8EA11908@gmail.com> <48FA97FF-3D3C-4099-9E02-1DB292EE1B41@feuerlabs.com> <27271B5D-4C73-4954-B527-1B50BE2125A4@feuerlabs.com> Message-ID: Regarding the gen_leader fork above, it does have basic support for netsplit situations merged from Andrew Thompson's (of Basho) fork. There has not been any rigorous model checking that I'm aware of. My own testing has shown that it basically works, with 2 caveats (so far): 1) the process is periodically unavailable for a few seconds at a time doing election related things (even on stable networks), so frequent requests like "who is the leader" can sometimes take a while, 2) if 2 gen_leader instances are started during a netsplit, they will eventually deadlock. I'm hoping to take a look at these issues over the next few weeks. -Garret Smith On Sun, Jul 1, 2012 at 10:24 AM, Ulf Wiger wrote: > > On 1 Jul 2012, at 19:15, Ulf Wiger wrote: > > > Answering this question with anything other than 'no' can risk > destabilizing the system unless the two candidates disagree. > > > s/disagree/agree > > of course. :) > > BR, > Ulf > > Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. > http://feuerlabs.com > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mahesh@REDACTED Mon Jul 2 17:23:14 2012 From: mahesh@REDACTED (Mahesh Paolini-Subramanya) Date: Mon, 2 Jul 2012 11:23:14 -0400 Subject: [erlang-questions] Why are records not 'export'-able? In-Reply-To: References: <8367670F-7092-4618-ADF7-EF0843BF65E8@dieswaytoofast.com> Message-ID: <7E05367B-00F5-43CC-A4F8-8DDAC3889AFA@dieswaytoofast.com> > > On 30/06/2012, at 2:06 AM, Mahesh Paolini-Subramanya wrote: > >> If I have a record I use in just one module, I declare it in that module. >> If I have a record I use across a number of modules, I declare it in an included file. >> If I have a record that I use across a number of applications, I end up declaring it once in each application. > > Er, what stops you using the same -include file in more than one application? > > If I found myself wanting to use the same -record in more than one > application, I'd lie down until the feeling wore off. More precisely, > I'd definitely want to hide that inside some sort of access module. Oh ouch :-) Mahesh Paolini-Subramanya That Tall Bald Indian Guy... Blog | Twitter | Google+ -------------- next part -------------- An HTML attachment was scrubbed... URL: From rostislav.romanin@REDACTED Mon Jul 2 17:39:37 2012 From: rostislav.romanin@REDACTED (Rostislav Romanin) Date: Mon, 2 Jul 2012 18:39:37 +0300 Subject: [erlang-questions] ssl:listen and certs from binary Message-ID: Hi, I have key,cert (pem format)as a binary. According to the documentation ssl:listen() accepts them as der_encoded(). What is the path that should be traveled in order to get there ? (guess it starts with pem_decode and ends with der_encode but can't figure the exact process). Thanks, Rostislav -------------- next part -------------- An HTML attachment was scrubbed... URL: From dan@REDACTED Mon Jul 2 17:44:54 2012 From: dan@REDACTED (Daniel Dormont) Date: Mon, 2 Jul 2012 11:44:54 -0400 Subject: [erlang-questions] Mnesia: simplest way to export/import tables across installations Message-ID: Hi all, Since I have been unable to restore order and goodness to my Mnesia cluster as discussed in my recent thread, I've decided to start over with a fresh node. But I need to move certain data across. My question now is much simpler: what is the best way to export a table (preferably _not_ including the table definition, since it will already exist, just the set of records) in one Mnesia installation and import it into another? I'm aware that there's a risk of things being slightly out of sync and I'm willing to completely ignore transactions for the purpose of this exercise anyway. dan -------------- next part -------------- An HTML attachment was scrubbed... URL: From ulf@REDACTED Mon Jul 2 18:05:23 2012 From: ulf@REDACTED (Ulf Wiger) Date: Mon, 2 Jul 2012 18:05:23 +0200 Subject: [erlang-questions] Mnesia: simplest way to export/import tables across installations In-Reply-To: References: Message-ID: Depends a bit on how dirty you want to be, but assuming that the tables are disc_copies, you can actually simply swipe the .DCD and .DCL files for each table you want to move over - then drop them in the same place in the new mnesia directory (assuming schema already created), then start the new instance. The .DCD file is a snapshot of the ets table, and the .DCL file is a set of operations on the table, updated each log dump. Thus, the pair forms a transaction-consistent view of the table (and *across tables*, if you do this on several tables at once), though not guaranteed to be the very latest. When the .DCL file grows too large, a new snapshot may be created (depending on mnesia settings). You need to make sure that no log dump is ongoing while you do this. The only officially sanctioned way to get such a guarantee is to stop mnesia on the node in question. Before doing so, you can always call mnesia:dump_log() to ensure that you don't have a bunch of stuff still lingering in the transaction log. BR, Ulf W On 2 Jul 2012, at 17:44, Daniel Dormont wrote: > Hi all, > > Since I have been unable to restore order and goodness to my Mnesia cluster as discussed in my recent thread, I've decided to start over with a fresh node. But I need to move certain data across. My question now is much simpler: what is the best way to export a table (preferably _not_ including the table definition, since it will already exist, just the set of records) in one Mnesia installation and import it into another? I'm aware that there's a risk of things being slightly out of sync and I'm willing to completely ignore transactions for the purpose of this exercise anyway. > > dan > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. http://feuerlabs.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From rostislav.romanin@REDACTED Mon Jul 2 21:17:18 2012 From: rostislav.romanin@REDACTED (Rostislav Romanin) Date: Mon, 2 Jul 2012 22:17:18 +0300 Subject: [erlang-questions] ssl:listen and certs from binary In-Reply-To: References: Message-ID: In case somebody will look for same thing some day: get_der(Pem)-> {_,Der,_} = hd(public_key:pem_decode(Pem)), Der. ssl:listen(Port [{key, {'RSAPrivateKey', get_der(KeyPem)}}, {cert, get_der(CertPem)}, {cacerts, [get_der(CaPem)]}]) On Mon, Jul 2, 2012 at 6:39 PM, Rostislav Romanin < rostislav.romanin@REDACTED> wrote: > Hi, > > I have key,cert (pem format)as a binary. According to the documentation > ssl:listen() > accepts them as der_encoded(). > > What is the path that should be traveled in order to get there ? > (guess it starts with pem_decode and ends with der_encode but can't figure > the > exact process). > > Thanks, > Rostislav > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wojtek@REDACTED Mon Jul 2 21:41:28 2012 From: wojtek@REDACTED (=?UTF-8?B?V29qdGVrIE5hcmN6ecWEc2tp?=) Date: Mon, 02 Jul 2012 21:41:28 +0200 Subject: [erlang-questions] How would you do it? In-Reply-To: References: Message-ID: <4FF1F968.5040405@power.com.pl> On 2012-07-02 12:29, Max Bourinov wrote: > > I need to build a ranking system for online game. > If this is a 1 vs 1 game, you might be interested in Elo Rating system. If it is team vs. team, it gets more complex. I am not sure how gamer 4KB Profile fits into ranking. Well, okay, maybe it is a good idea to keep gamer profiles and ranks in the same data structure, but they don't have to be updated with the same message. I would expect an interfrace along these lines: {set_status, GamerID, offline / ready / playing} - info for match making {get_opponents, GamerID} -> [{OpponentID, Rank}] - select some potential opponents, ready to play {update_rank, MatchID, WinnerID, LooserID} -> {NewWinnerRank, NewLooserRank} - ranks after a match, Elo here Pleasantries (you may not dispense with): {register, GamerID, Profile} {unregister, GamerID} {set_profile, GamerID, Profile} {get_profile, GamerID} -> Profile {get_rank, GamerID} -> Rank {get_top, N} -> [{GamerID, Rank}] Sorting and searching is going to be the easy part, the performance can easily be assessed by an automated test. I have never done anything even remotely gaming related, so don't listen to me. --Regards, Wojtek Narczynski From eriksoe@REDACTED Mon Jul 2 23:52:39 2012 From: eriksoe@REDACTED (=?ISO-8859-1?Q?Erik_S=F8e_S=F8rensen?=) Date: Mon, 2 Jul 2012 23:52:39 +0200 Subject: [erlang-questions] How would you do it? In-Reply-To: References: Message-ID: (forgot to Cc the list.) ---------- Videresendt besked ---------- Fra: "Erik S?e S?rensen" Dato: 02/07/2012 23.50 Emne: Re: [erlang-questions] How would you do it? Til: "Max Bourinov" I think I'd do this...: 0) Not include the profile info if the score messages unless it really does change that often. 1) Keep two ets tables: one with player-id as key and current score as a column (and possibly other columns, for profile info etc., but for this I'd only fetch the score column). 2) And the other ets table would have {Score,PlayerID} as key. No other columns needed for the tasks you mention. 3) The first table would be a normal set, the second a sorted set. This means that player rows can be found and updated in constant time, and the score table row updated in logarithmic time (using the old score from the first table). Top 100, and neighbouring scores, can be found using ets iterators, in log time. The performance of this should be quite adequate. There is at least one advantage of using ets tables over in-heap data structures for this, namely that it keeps the heap size down, so there's less work for the GC. Happy hacking, whichever approach you settle on. /Erik (Reminds me, I ought to add mention of two-table approaches to AGttES soonish. It can be quite handy to have your data indexed in several ways.) /Erik Den 02/07/2012 12.30 skrev "Max Bourinov" : > Hi Erlangers, > > I need to build a ranking system for online game. > > I think I will have a ranking gen_server that will accept cast messages > like this: {score, Player :: profile(), Score :: integer()}. > > So, the question is what would be most appropriate data structure if: > > 1. I have to rank about 50 000 - 100 000 different players. > 2. On each score message I have to re-sort whole ranking table. > 3. It must be very cheap to get: > 1. top 100 players > 2. player's rating +/- 10 players about current player > 4. I expect about 20-50 score messages per seconds > 5. Size of score message is about 4KB (profile takes most of the > space). > > > Any ideas or suggestions are welcome! > > Best regards, > Max > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From desired.mta@REDACTED Tue Jul 3 09:40:41 2012 From: desired.mta@REDACTED (=?UTF-8?Q?Motiejus_Jak=C5=A1tys?=) Date: Tue, 3 Jul 2012 10:40:41 +0300 Subject: [erlang-questions] including observer.ebin to release Message-ID: Hi, I am trying to get observer included to my release. I specified it in my reltool.config[1] like Garret Smith suggested[2], but I do not get an observer.beam file in my release[3]. What am I doing wrong? Thanks. [1]: http://paste.debian.net/177500/ [2]: http://erlang.org/pipermail/erlang-questions/2012-May/066348.html [3]: http://paste.debian.net/177502/ Motiejus Jak?tys From YurinVV@REDACTED Tue Jul 3 09:52:03 2012 From: YurinVV@REDACTED (Slava Yurin) Date: Tue, 03 Jul 2012 14:52:03 +0700 Subject: [erlang-questions] including observer.ebin to release Message-ID: <477511341301923@web3f.yandex.ru> An HTML attachment was scrubbed... URL: From desired.mta@REDACTED Tue Jul 3 09:57:23 2012 From: desired.mta@REDACTED (=?UTF-8?Q?Motiejus_Jak=C5=A1tys?=) Date: Tue, 3 Jul 2012 10:57:23 +0300 Subject: [erlang-questions] including observer.ebin to release In-Reply-To: <477511341301923@web3f.yandex.ru> References: <477511341301923@web3f.yandex.ru> Message-ID: On Tue, Jul 3, 2012 at 10:52 AM, Slava Yurin wrote: > Hi Motiejus. > > I think it is because you specified {mod_cond, derived} in reltool.config. > And no module from "observer" application was included as they didn't use. Hi, thanks, that's correct, I missed that option somehow. > Try {app, observer, [{incl_cond, include}, {mod_cond, include}]} instead. You most likely meant {app, observer, [{incl_cond, include}, {mod_cond, all}]} But I really ment {mod_cond, all}. Thanks for your help! -- Motiejus Jak?tys From erlangsiri@REDACTED Tue Jul 3 10:01:50 2012 From: erlangsiri@REDACTED (Siri Hansen) Date: Tue, 3 Jul 2012 10:01:50 +0200 Subject: [erlang-questions] More on this: simple_one_by_one trouble and rare start_link/2 In-Reply-To: <201206221104.38808.clist@uah.es> References: <201206210030.08517.clist@uah.es> <20120621042238.GA15583@aluminum.wavenet.lk> <201206210930.15698.clist@uah.es> <201206221104.38808.clist@uah.es> Message-ID: Hi Angel! Thanks for pointing out this problem! While one might agree that the design decision was not very good here, it is, as Robert points out, a totally backwards incompatible change you wish to do. And even though simple_one_for_one is not the most used supervisor type, it is for sure used and it is not an option for us to do such a change. However, I do agree that the documentation could be better here, and I will write a ticket to make sure the issue is not forgotten. As always, a user contribution might very well make this change happen faster than our own priorities else would allow :) Best regards /siri@REDACTED 2012/6/22 Angel J. Alvarez Miguel > > i did'nt find any example of mixing args from the supervirsor childspec > and start_child(...) so im going to make my own digging in this issue: > > Let see How the simple_one_by_one works... > > handle_call({start_child, EArgs}, _From, State) when ?is_simple(State) -> > Child = hd(State#state.children), > #child{mfargs = {M, F, A}} = Child, > Args = A ++ EArgs, > case do_start_child_i(M, F, Args) of > <------>{ok, undefined} when Child#child.restart_type =:= temporary -> > <------> {reply, {ok, undefined}, State}; > <------>{ok, Pid} -> > <------> NState = save_dynamic_child(Child#child.restart_type, Pid, > Args, State), > <------> {reply, {ok, Pid}, NState}; > <------>{ok, Pid, Extra} -> > <------> NState = save_dynamic_child(Child#child.restart_type, Pid, > Args, State), > <------> {reply, {ok, Pid, Extra}, NState}; > <------>What -> > <------> {reply, What, State} > end; > > > Shouldnt be do_start_child_i(M,F,EArgs) of the form : > > do_start_child(M,F,[EArgs]) ?? > > or the inner call apply(M,F,[A])? > > Provided the docs state that your unique exported funcion must be of > arity one you should > expect that code will take measures in order to enforce arity one, > enclosing whatever you pass it in a List on the apply phase. > > The semanctics of apply use a list to KNOW how many args the target call > has, while the > semantics of sup childspec + sup start_child use a list ++ op to know how > many args will be passed > to your callback of arity ONE. > > So for the apply part all your functions are arity ONE > > Without the ability to join Args from the ChildSpec and from the start > child call i would concur that you should > enclose in a list whatever you want to be as a sole argument and this code > be ok, but as soon as you can put > two or more args one on the ChildSpec and one on the call Sup code must > enforce that only one arg APPLY > call is made so you end calling start_link/1 whatever you pass on the > args... > > Even the DOCS state "[ term() ]" as the MFA on the ChildSpec > > > My stdlib-1.18.1.pdf says (Pag 369) > > "start_child(SupRef, ChildSpec) -> startchild_ret() > Types: > SupRef = sup_ref() > ChildSpec = child_spec() | (List :: [term()]) > ... > ... > > ...If the case of a simple_one_for_one supervisor, the child specification > defined in Module:init/1 will be > used and ChildSpec should instead be an arbitrary list of terms List. The > child process will then be started by > appending List to the existing start function arguments, i.e. by calling > apply(M, F, A++List) where {M,F,A} > is the start function defined in the child specification." > > Here IMHO lies the error, as A is a list we have [any()...] ++ [any()...] > is a list [any()...,any()...] and thats get us an apply > call of arity > 1 ALWAYS even when you chilkdspec has a {M,F,[]} and you > try provide a [..] in your start_child it will fail > unless both argslists are void. > > So a combination of: > > init(Opts) -> > io:format("[GROUP SUP] Ready to manage MuC with opts: ~p\'s\n",[Opts]), > > {ok, { > {simple_one_for_one, 1, 60}, > [ > {mucfsm, {sim_group_fsm, start_link, [arg1,arg2]}, > transient, 60, worker, [sim_group]} > ] > } > }. > > > and a client that makes a call like: > ... > {ok,ChildPid} = supervisor:start_child(SupPid,[arg3,arg4]), > ... > > Gets you a final apply call {M,F,A} where A is in the form > [arg1,arg2,arg3,arg4] > not [[arg1,arg2,argf3,arg4]] at is should be when you intent to call a > start_link/1.... > > > > Erlang R15B01 (erts-5.9.1) [source] [64-bit] [smp:4:4] [async-threads:0] > [hipe] [kernel-poll:false] > > [QuickCheck] [PropEr] [Erl0MQ] > Starting simple_one_by_one group supervisor... > [GROUP SUP] staring with argument: [{option1,40}] > Eshell V5.9.1 (abort with ^G) > 1> [GROUP SUP] Ready to manage MuC with opts: [{option1,40}]'s > Adding a child dynamically... > {"init terminating in > do_boot",{{badmatch,{error,{'EXIT',{undef,[{sim_group_fsm,start_link,[arg1,arg2,arg3,arg4],[]},{supervisor,do_start_child_i,3,[{file,"supervisor.erl"},{line,319}]},{supervisor,handle_call,3,[{file,"supervisor.erl"},{line,344}]}, > > {gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,588}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}}}},[{test,main,0,[{file,"test.erl"},{line,11}]},{init,start_it,1,[]},{init,start_em,1,[]}]}} > > > /Angel > > On Jueves, 21 de Junio de 2012 09:30:15 Angel J. Alvarez Miguel escribi?: > > That's right... > > > > But start_link it is expected to be or arity 3 and Opts is a property > list. > > > > but then on test:main/0 when i do supervisor:start_child(SupPid,Opts2) > > > > being Opts2 another property list, then suppervisor should call: > > > > apply(sim_group_fsm,start_link,Opts ++ Opts2) on its line 319 which in > turn > > calls gen_fsm:start_link, thus providing a Pid in the end... > > > > but what it really does is apply(sim_group_fsm,start_link,Opts ++ Opts2,[ > > ]) and then crashes unless i declare a start_link/2 on my gen_fsm... > > > > { > > {badmatch, > > {error, > > {'EXIT', > > {undef, > > [ > > > {sim_group_fsm,start_link,[[{option1,40}],[{option29,100}]],[]}, > > {supervisor,do_start_child_i,3,[ > > > {file,"supervisor.erl"}, > > > {line,319} > > > ]}...... > > > > i dont see why i finish having a apply/4 call in supervisor: 319 instead > of > > an apply/3... > > > > I want the sup to propagate one property list to children specs while i > > will provide another during children spawns providing that sup combines > > two in any way children will manage to demunge ...both property lists > > > > So cant just see what im doing wrong.... > > > > /angel > > > > > > PS: i changed a bit the test code to force sup crash > > > > On Jueves, 21 de Junio de 2012 06:22:54 Vance Shipley escribi?: > > > On Thu, Jun 21, 2012 at 12:30:08AM +0200, Angel J. Alvarez Miguel > wrote: > > > } While calling a simple_one_by_one supervisor to start a gen fsm I > > > found } it causes calling a start_link/2 function on the callback > > > module of the } gen_fsm. > > > > > > You are specifying that function to be called to start the worker > > > yourself. In your module group_supervisor.erl you provide a > child_spec() > > > in the > > > > > > return value of init/1: > > > {mucfsm, {sim_group_fsm, start_link, [Opts]}, transient, 60, worker, > > > > > > [sim_group]} > > > > > > The form of which is: > > > {Id, StartFunc, Restart, Shutdown, Type, Modules} > > > > > > The second argument, StartFunc, has the form: > > > {Module, Function, Arguments} > > > > > > You provided: > > > {sim_group_fsm, start_link, [Opts]} > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Tue Jul 3 10:28:49 2012 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Tue, 03 Jul 2012 10:28:49 +0200 Subject: [erlang-questions] How would you do it? In-Reply-To: References: Message-ID: <4FF2AD41.9090407@erlang-solutions.com> On 7/2/12 11:52 PM, Erik S?e S?rensen wrote: > > I think I'd do this...: > 0) Not include the profile info if the score messages unless it really > does change that often. > 1) Keep two ets tables: one with player-id as key and current score as > a column (and possibly other columns, for profile info etc., but for > this I'd only fetch the s... > This is my solution as well. The PSQueue data structure can be quite heavy in memory operations, so while it has nice properties on paper, it does have some hefty constant factors. I used a PSQueue in Combinatorrent, but eTorrent uses the above solution for its piece histogram. -- Jesper Louis Andersen Erlang Solutions Ltd., Copenhagen, DK From jesper.louis.andersen@REDACTED Tue Jul 3 10:34:24 2012 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Tue, 03 Jul 2012 10:34:24 +0200 Subject: [erlang-questions] How would you do it? In-Reply-To: <4FF1F968.5040405@power.com.pl> References: <4FF1F968.5040405@power.com.pl> Message-ID: <4FF2AE90.3020006@erlang-solutions.com> On 7/2/12 9:41 PM, Wojtek Narczy?ski wrote: > If this is a 1 vs 1 game, you might be interested in Elo Rating > system. If it is team vs. team, it gets more complex. In a modern world of 1v1 games, it is better to use a more complex ranking system with better results. ELO is designed for hand-calculation. My Glicko2 code, which is a bayesian ranking algorithm, is on-line: https://github.com/jlouis/erl-glicko2/blob/master/src/glicko2.erl Do note that the Volatility root finding in step 5 is not numerically stable in that code. If you need a stable variant, prod me and I'll push a later version, which uses another variant of root-finding which has proven to be stable enough in practice. -- Jesper Louis Andersen Erlang Solutions Ltd., Copenhagen, DK From erlangsiri@REDACTED Tue Jul 3 11:27:08 2012 From: erlangsiri@REDACTED (Siri Hansen) Date: Tue, 3 Jul 2012 11:27:08 +0200 Subject: [erlang-questions] common_test vs supervised process In-Reply-To: References: Message-ID: It's a while since you wrote this, so you might have come to some conclusion by now, but anyway - here's what happens: Since your top supervisor is started with start_link, it will be linked to the test case process. The test case process does not trap exit. When stopping your service, the supervisor is terminated with reason shutdown. Due to the link and not trapping exits, the test case process will also terminate with reason shutdown and thus common_test believes that the test has failed with this reason. After all, a terminating test case process is the definition of a failed test case in common_test world. Regards /siri 2012/6/21 Attila Rajmund Nohl > Hello! > > I try to test our code with common_test (from the OTP master branch). > This code starts a supervisor, which starts a process. For some reason > the test fails with: > > *** CT Error Notification 2012-06-21 19:38:59.236 *** > Error detected: shutdown > > even though the started process doesn't crash (or at least I don't see > it). I'm probably making something really stupid, do you have any idea > what? > > Here's the important part of the code. The testcase (the init_* > functions are empty): > > start_stop_test(_Config) -> > {ok, Pid} = inets:start(ftpd, [{port, 2021}]), > inets:stop(ftpd, Pid). > > The ftpd module: > > start_service(Config) -> > ftpd_sup:start_link(Config). > > stop_service(Pid) when is_pid(Pid) -> > case ftpd_sup:stop(Pid) of > true -> ok; > R -> { error, R } > end. > > The ftpd_sup module: > > start_link(Args) -> > supervisor:start_link(ftpd_sup, Args). > > stop(Pid) -> > exit(Pid,shutdown), > true. > > init(Args) -> > NewArgs = [ {sup_pid, self()} | Args], > {ok, {{one_for_one, 10, 60}, > [{ftpd_listener, > {ftpd_listener, start_link, [NewArgs]}, > permanent, > brutal_kill, > worker, > [ftpd_listener]}]}}. > > And the ftpd_listener: > > start_link(Args) -> > proc_lib:start_link(?MODULE, listener_init, [self(), Args]). > > listener_init(Parent, Args) -> > proc_lib:init_ack(Parent, {ok, self()}), > Port = proplists:get_value(port,Args), > SupPid = proplists:get_value(sup_pid,Args), > {ok, LSock} = gen_tcp:listen(Port, [binary, {packet, 0}, {active, > false}]), > loop(LSock, SupPid). > > loop(LSock, SupPid) -> > case gen_tcp:accept(LSock) of > {ok, Sock} -> > loop(LSock, SupPid); > {_, Res} -> ok > end. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From f@REDACTED Tue Jul 3 13:18:47 2012 From: f@REDACTED (Francesco Mazzoli) Date: Tue, 03 Jul 2012 12:18:47 +0100 Subject: [erlang-questions] Removing mnesia cluster node while mnesia is offline In-Reply-To: References: <878vf6chx0.wl%f@mazzo.li> Message-ID: <87fw995bzc.wl%f@mazzo.li> Hi Rudolph, thanks for the answer. At Tue, 03 Jul 2012 12:01:34 +0100, Rudolph van Graan wrote: > I would suggest that you think differently about the > problem. Instead of trying to 'remove' the node from the schema > (which requires you to modify it and all the nodes must be online), > think that the offline node that is destroyed. So in this case, > delete the mnesia data on the failed node (all the files in the > mnesia data directory) and bring it back online. Or alternative, > start a clean instance with the same node name as the failed node. Well, that will surely work, but it's not an attractive option, since it will lead to loss of data (the local tables) on that node. I was looking for a way to retain the local data on the offline node. -- Francesco * Often in error, never in doubt From Sergey_Zhemzhitsky@REDACTED Tue Jul 3 13:20:23 2012 From: Sergey_Zhemzhitsky@REDACTED (Zhemzhitsky Sergey) Date: Tue, 3 Jul 2012 11:20:23 +0000 Subject: [erlang-questions] Does port driver block scheduler threads? Message-ID: <06139A918ACCA041BF46A0F36940C7FA39CD79D6@exch-mbx1.msk.trd.ru> Hi erlangers, Recently we have developed port (linked-in) driver and faced with some performance issues. The port uses ERL_DRV_FLAG_USE_PORT_LOCKING locking mode, so as we understand calls to different port instances should have not blocked each other. SMP count is greater than 1. So the first question is: Is it possible to have multiple port instances executing time-consuming operation in parallel, i.e. at the same time? We expected port instances do not influence on each other. However, all the calls are blocked but the first one. So the scheduler thread seems to be blocked during the port instance invocation. Is driver_async function required for the issues like above? Is it possible to unblock the scheduler thread before erlang:port_call completes? Best Regards, Sergey _______________________________________________________ The information contained in this message may be privileged and conf idential and protected from disclosure. If you are not the original intended recipient, you are hereby notified that any review, retransmission, dissemination, or other use of, or taking of any action in reliance upon, this information is prohibited. If you have received this communication in error, please notify the sender immediately by replying to this message and delete it from your computer. Thank you for your cooperation. Troika Dialog, Russia. If you need assistance please contact our Contact Center (+7495) 258 0500 or go to www.troika.ru/eng/Contacts/system.wbp -------------- next part -------------- An HTML attachment was scrubbed... URL: From rvg.mailing@REDACTED Tue Jul 3 13:31:11 2012 From: rvg.mailing@REDACTED (Rudolph van Graan) Date: Tue, 03 Jul 2012 12:31:11 +0100 Subject: [erlang-questions] Removing mnesia cluster node while mnesia is offline In-Reply-To: <87fw995bzc.wl%f@mazzo.li> References: <878vf6chx0.wl%f@mazzo.li> <87fw995bzc.wl%f@mazzo.li> Message-ID: <698B603A-C83F-4726-B4EA-FE215D997987@me.com> Hi Francesco, > Well, that will surely work, but it's not an attractive option, since > it will lead to loss of data (the local tables) on that node. I was > looking for a way to retain the local data on the offline node. You have to set things up so that you have other replicas of those tables on other nodes. > The problem is that the offline node still thinks it is clustered with > the offline (or worse, not clustered anymore nodes). Starting mnesia > on that node will either "pollute" the other nodes by propagating the > schema with the not-in-the-cluster-anymore nodes, or worse hangs if the > nodes that the node thinks are still clustered are online. This is not how mnesia works. The list of nodes in the distributed database is static, i.e. all the nodes have the same list. This is why all of them needs to be online when you add or remove nodes. The only problem that you need to solve is dealing with partitioning (i.e. splits) and you have two sets of data on two different segments. It is not possible for a node to "pollute" other nodes. So "... offline node still thinks it is clustered..." - it doesn't just think so, it has been configured so when you added it into the cluster. It will stay part of the cluster until you remove it. On 3 Jul 2012, at 12:18, Francesco Mazzoli wrote: > Hi Rudolph, thanks for the answer. > > At Tue, 03 Jul 2012 12:01:34 +0100, > Rudolph van Graan wrote: >> I would suggest that you think differently about the >> problem. Instead of trying to 'remove' the node from the schema >> (which requires you to modify it and all the nodes must be online), >> think that the offline node that is destroyed. So in this case, >> delete the mnesia data on the failed node (all the files in the >> mnesia data directory) and bring it back online. Or alternative, >> start a clean instance with the same node name as the failed node. > > Well, that will surely work, but it's not an attractive option, since > it will lead to loss of data (the local tables) on that node. I was > looking for a way to retain the local data on the offline node. > > -- > Francesco * Often in error, never in doubt > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From serge@REDACTED Tue Jul 3 13:37:04 2012 From: serge@REDACTED (Serge Aleynikov) Date: Tue, 03 Jul 2012 07:37:04 -0400 Subject: [erlang-questions] Does port driver block scheduler threads? In-Reply-To: <06139A918ACCA041BF46A0F36940C7FA39CD79D6@exch-mbx1.msk.trd.ru> References: <06139A918ACCA041BF46A0F36940C7FA39CD79D6@exch-mbx1.msk.trd.ru> Message-ID: <4FF2D960.7030505@aleynikov.org> Driver callbacks are executed by a scheduler thread in synchronous manner, so what you are describing is a normal behavior of a linked-in driver. Use driver_async to off-load a long computation to another thread from a pool different from the scheduling threads. On 7/3/2012 7:20 AM, Zhemzhitsky Sergey wrote: > Hi erlangers, > > > > Recently we have developed port (linked-in) driver and faced with some > performance issues. > > > > The port uses ERL_DRV_FLAG_USE_PORT_LOCKING locking mode, so as we > understand calls to different port instances should have not blocked > each other. > > SMP count is greater than 1. > > > > So the first question is: > > Is it possible to have multiple port instances executing time-consuming > operation in parallel, i.e. at the same time? > > > > We expected port instances do not influence on each other. However, all > the calls are blocked but the first one. So the scheduler thread seems > to be blocked during the port instance invocation. > > Is driver_async function required for the issues like above? > > Is it possible to unblock the scheduler thread before erlang:port_call > completes? > > > > Best Regards, > > Sergey > > > > _______________________________________________________ > > > > The information contained in this message may be privileged and conf > idential and protected from disclosure. If you are not the original > intended recipient, you are hereby notified that any review, > retransmission, dissemination, or other use of, or taking of any action > in reliance upon, this information is prohibited. If you have received > this communication in error, please notify the sender immediately by > replying to this message and delete it from your computer. Thank you for > your cooperation. Troika Dialog, Russia. > > If you need assistance please contact our Contact Center (+7495) 258 > 0500 or go to www.troika.ru/eng/Contacts/system.wbp > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From f@REDACTED Tue Jul 3 14:12:49 2012 From: f@REDACTED (Francesco Mazzoli) Date: Tue, 03 Jul 2012 13:12:49 +0100 Subject: [erlang-questions] Removing mnesia cluster node while mnesia is offline In-Reply-To: <698B603A-C83F-4726-B4EA-FE215D997987@me.com> References: <878vf6chx0.wl%f@mazzo.li> <87fw995bzc.wl%f@mazzo.li> <698B603A-C83F-4726-B4EA-FE215D997987@me.com> Message-ID: <87d34d59h9.wl%f@mazzo.li> Hi Rudolph, At Tue, 03 Jul 2012 12:31:11 +0100, Rudolph van Graan wrote: > You have to set things up so that you have other replicas of those > tables on other nodes. Well, some data is inherently "local" and not relevant to the other nodes, that have similar local tables. > This is not how mnesia works. The list of nodes in the distributed > database is static, i.e. all the nodes have the same list. This is > why all of them needs to be online when you add or remove nodes. I understand this, but I still want to know if it's possible to recover from that situation more or less gracefully. > The only problem that you need to solve is dealing with partitioning > (i.e. splits) and you have two sets of data on two different > segments. It is not possible for a node to "pollute" other nodes. OK, let's say that I have node A, B, and C, in a cluster. A goes down, and C gets removed from the cluster. So B now has [A, B] in the schema as cluster nodes. A's data still has [A, B, C] as cluster nodes, and if we start A again, that will be propagated to B. That's what I meant with "pollute". > So "... offline node still thinks it is clustered..." - it doesn't > just think so, it has been configured so when you added it into the > cluster. It will stay part of the cluster until you remove it. ...but to remove it I have to bring it online, and if the other online nodes have a different configuration, things get ugly. -- Francesco * Often in error, never in doubt From erlangsiri@REDACTED Tue Jul 3 15:25:26 2012 From: erlangsiri@REDACTED (Siri Hansen) Date: Tue, 3 Jul 2012 15:25:26 +0200 Subject: [erlang-questions] Reltool error: "Module potentially included by two different applications" In-Reply-To: References: Message-ID: Hi Eduard - sorry for the long delay! Here are some thoughts about this issue. The principle here is that reltool shall never guess. I.e. as long as it can not be 100% sure which of the two modules to use, then it will not choose but expects you to explicitely make the decision. Since you haven't included your reltool config file, I'm just guessing that it looks (maybe simplified) something like this: {sys,[{lib_dirs,[""]}, {incl_cond,derived}, {app,,[{incl_cond,include}]} ]}. and that exists in . If this is close to correct, then what you ask reltool to do is "Take my erlang installation and also look in and figure out which applications and modules I need to include in my release so that all dependencies of are taken care of." And to be very strict - it could(!!) be that you want to use the from and not from itself... but I realize that it is not very likely. At least not if is the only relation between and ... So the question is if this is ALWAYS the case?? A simple workaround for the above config file would be to skip the system level lib_dirs parameter, and specify directory per application instead: {sys,[{incl_cond,derived}, {app,,[{incl_cond,include},{lib_dir,"/"]} ]}. This will of course only work as long as there are no other dependecies from towards other applications in ... Regards /siri 2012/6/15 Eduard Sergeev > Hi, > > I was wondering what was the reason behind the following behaviour of > reltool: > > If my reltool.config uses default 'mod_cond' and 'incl_cond' options > and if one of my included applications has a module which also happens > to be a part of some application installed on my machine but NOT > included in my release `relltool:get_target_spec/1` returns: > > {error, "Module potentially included by two different > applications: and ."} > > Which is annoying since is NOT a part of my release > (neither directly nor indirectly). Can't reltool actually figure out > that will not be included in my release? Id that why it > is "potentially included"? > > Anyway, in order to generate my release I have to either explicitly > exclude via `{app, [{incl_cond, exclude}]}` > which is ugly since this is just happens to be installed > in 'lib' directory of the machine where I do the build (it may not be > installed on other build machines) and has nothing to do with my > release. The actual example: 'tsung-1.4.3' includes 'mochijson2' > module so I have a problem building my release which should include > 'mochiweb' app on the machine that has 'tsung' installed (but not on > other machines). > Another option would be to change top-level 'incl_cond' from > {incl_cond, derived} to {incl_cond, exclude} and then manually include > all the application which I want to be a part my release which is > better (will work on any build machine) but still not great since it > has to be done manually (I want to rely on relltool to figure out the > dependencies). > > So the question is why do we have such a situation? Why just a mere > presence of some application on the build machine lead to the above > reltool error? > > Regards, > Eduard > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Sergey_Zhemzhitsky@REDACTED Tue Jul 3 16:04:16 2012 From: Sergey_Zhemzhitsky@REDACTED (Zhemzhitsky Sergey) Date: Tue, 3 Jul 2012 14:04:16 +0000 Subject: [erlang-questions] Does port driver block scheduler threads? In-Reply-To: <4FF2D960.7030505@aleynikov.org> References: <06139A918ACCA041BF46A0F36940C7FA39CD79D6@exch-mbx1.msk.trd.ru> <4FF2D960.7030505@aleynikov.org> Message-ID: <06139A918ACCA041BF46A0F36940C7FA39CD8B3C@exch-mbx1.msk.trd.ru> Hi Serge, Thanks a lot. Whether it is correct that the same behavior is applicable to the ordinary ports? Best Regards, Sergey -----Original Message----- From: erlang-questions-bounces@REDACTED [mailto:erlang-questions-bounces@REDACTED] On Behalf Of Serge Aleynikov Sent: Tuesday, July 03, 2012 3:37 PM To: erlang-questions@REDACTED Subject: Re: [erlang-questions] Does port driver block scheduler threads? Driver callbacks are executed by a scheduler thread in synchronous manner, so what you are describing is a normal behavior of a linked-in driver. Use driver_async to off-load a long computation to another thread from a pool different from the scheduling threads. On 7/3/2012 7:20 AM, Zhemzhitsky Sergey wrote: > Hi erlangers, > > > > Recently we have developed port (linked-in) driver and faced with some > performance issues. > > > > The port uses ERL_DRV_FLAG_USE_PORT_LOCKING locking mode, so as we > understand calls to different port instances should have not blocked > each other. > > SMP count is greater than 1. > > > > So the first question is: > > Is it possible to have multiple port instances executing > time-consuming operation in parallel, i.e. at the same time? > > > > We expected port instances do not influence on each other. However, > all the calls are blocked but the first one. So the scheduler thread > seems to be blocked during the port instance invocation. > > Is driver_async function required for the issues like above? > > Is it possible to unblock the scheduler thread before erlang:port_call > completes? > > > > Best Regards, > > Sergey > > > > _______________________________________________________ > > > > The information contained in this message may be privileged and conf > idential and protected from disclosure. If you are not the original > intended recipient, you are hereby notified that any review, > retransmission, dissemination, or other use of, or taking of any > action in reliance upon, this information is prohibited. If you have > received this communication in error, please notify the sender > immediately by replying to this message and delete it from your > computer. Thank you for your cooperation. Troika Dialog, Russia. > > If you need assistance please contact our Contact Center (+7495) 258 > 0500 or go to www.troika.ru/eng/Contacts/system.wbp > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions From f@REDACTED Tue Jul 3 16:08:03 2012 From: f@REDACTED (Francesco Mazzoli) Date: Tue, 03 Jul 2012 15:08:03 +0100 Subject: [erlang-questions] Removing mnesia cluster node while mnesia is offline In-Reply-To: <878vf6chx0.wl%f@mazzo.li> References: <878vf6chx0.wl%f@mazzo.li> Message-ID: <87wr2lx7i4.wl%f@mazzo.li> It seems that you can delete the node schema while mnesia is offline (via `mnesia:delete_schema/2'). This makes it possible to uncluster the node while offline, and then re-cluster with the correct nodes afterwards. -- Francesco * Often in error, never in doubt From zabrane3@REDACTED Tue Jul 3 16:25:10 2012 From: zabrane3@REDACTED (Zabrane Mickael) Date: Tue, 3 Jul 2012 16:25:10 +0200 Subject: [erlang-questions] Does port driver block scheduler threads? In-Reply-To: <4FF2D960.7030505@aleynikov.org> References: <06139A918ACCA041BF46A0F36940C7FA39CD79D6@exch-mbx1.msk.trd.ru> <4FF2D960.7030505@aleynikov.org> Message-ID: Hi Serge, Where one can find a simple driver_async example to learn from? thx Zab On Jul 3, 2012, at 1:37 PM, Serge Aleynikov wrote: > Driver callbacks are executed by a scheduler thread in synchronous > manner, so what you are describing is a normal behavior of a linked-in > driver. Use driver_async to off-load a long computation to another > thread from a pool different from the scheduling threads. > > On 7/3/2012 7:20 AM, Zhemzhitsky Sergey wrote: >> Hi erlangers, >> >> >> >> Recently we have developed port (linked-in) driver and faced with some >> performance issues. >> >> >> >> The port uses ERL_DRV_FLAG_USE_PORT_LOCKING locking mode, so as we >> understand calls to different port instances should have not blocked >> each other. >> >> SMP count is greater than 1. >> >> >> >> So the first question is: >> >> Is it possible to have multiple port instances executing time-consuming >> operation in parallel, i.e. at the same time? >> >> >> >> We expected port instances do not influence on each other. However, all >> the calls are blocked but the first one. So the scheduler thread seems >> to be blocked during the port instance invocation. >> >> Is driver_async function required for the issues like above? >> >> Is it possible to unblock the scheduler thread before erlang:port_call >> completes? >> >> >> >> Best Regards, >> >> Sergey >> >> >> >> _______________________________________________________ >> >> >> >> The information contained in this message may be privileged and conf >> idential and protected from disclosure. If you are not the original >> intended recipient, you are hereby notified that any review, >> retransmission, dissemination, or other use of, or taking of any action >> in reliance upon, this information is prohibited. If you have received >> this communication in error, please notify the sender immediately by >> replying to this message and delete it from your computer. Thank you for >> your cooperation. Troika Dialog, Russia. >> >> If you need assistance please contact our Contact Center (+7495) 258 >> 0500 or go to www.troika.ru/eng/Contacts/system.wbp >> >> >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions Regards, Zabrane From barys_ilyushonak@REDACTED Tue Jul 3 16:43:09 2012 From: barys_ilyushonak@REDACTED (Ilyushonak Barys) Date: Tue, 3 Jul 2012 14:43:09 +0000 Subject: [erlang-questions] Does port driver block scheduler threads? In-Reply-To: References: <06139A918ACCA041BF46A0F36940C7FA39CD79D6@exch-mbx1.msk.trd.ru> <4FF2D960.7030505@aleynikov.org> Message-ID: Hi, http://www.erlang.org/doc/apps/erts/driver.html#id82769 and source code from OTP erts/example/next_perm.cc Please, note, there is no implementation of do_free method in this file: Line 106, driver_async(port, NULL, do_perm, async_data, NULL); Hope this helps. To be honest it would be great to find more complex example. It would great contribution If someone can share it. Regards, Boris -----Original Message----- From: erlang-questions-bounces@REDACTED [mailto:erlang-questions-bounces@REDACTED] On Behalf Of Zabrane Mickael Sent: Tuesday, July 03, 2012 6:26 PM To: Serge Aleynikov Cc: erlang-questions@REDACTED Subject: Re: [erlang-questions] Does port driver block scheduler threads? Hi Serge, Where one can find a simple driver_async example to learn from? thx Zab On Jul 3, 2012, at 1:37 PM, Serge Aleynikov wrote: > Driver callbacks are executed by a scheduler thread in synchronous > manner, so what you are describing is a normal behavior of a linked-in > driver. Use driver_async to off-load a long computation to another > thread from a pool different from the scheduling threads. > > On 7/3/2012 7:20 AM, Zhemzhitsky Sergey wrote: >> Hi erlangers, >> >> >> >> Recently we have developed port (linked-in) driver and faced with >> some performance issues. >> >> >> >> The port uses ERL_DRV_FLAG_USE_PORT_LOCKING locking mode, so as we >> understand calls to different port instances should have not blocked >> each other. >> >> SMP count is greater than 1. >> >> >> >> So the first question is: >> >> Is it possible to have multiple port instances executing >> time-consuming operation in parallel, i.e. at the same time? >> >> >> >> We expected port instances do not influence on each other. However, >> all the calls are blocked but the first one. So the scheduler thread >> seems to be blocked during the port instance invocation. >> >> Is driver_async function required for the issues like above? >> >> Is it possible to unblock the scheduler thread before >> erlang:port_call completes? >> >> >> >> Best Regards, >> >> Sergey >> >> >> >> _______________________________________________________ >> >> >> >> The information contained in this message may be privileged and conf >> idential and protected from disclosure. If you are not the original >> intended recipient, you are hereby notified that any review, >> retransmission, dissemination, or other use of, or taking of any >> action in reliance upon, this information is prohibited. If you have >> received this communication in error, please notify the sender >> immediately by replying to this message and delete it from your >> computer. Thank you for your cooperation. Troika Dialog, Russia. >> >> If you need assistance please contact our Contact Center (+7495) 258 >> 0500 or go to www.troika.ru/eng/Contacts/system.wbp >> >> >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions Regards, Zabrane _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions From zabrane3@REDACTED Tue Jul 3 16:44:06 2012 From: zabrane3@REDACTED (Zabrane Mickael) Date: Tue, 3 Jul 2012 16:44:06 +0200 Subject: [erlang-questions] Does port driver block scheduler threads? In-Reply-To: References: <06139A918ACCA041BF46A0F36940C7FA39CD79D6@exch-mbx1.msk.trd.ru> <4FF2D960.7030505@aleynikov.org> Message-ID: Thanks for sharing Joseph. Zabrane On Jul 3, 2012, at 4:35 PM, Joseph Wayne Norton wrote: > > Zabrane - > > Here is a nif and driver implementation for interfacing with leveldb. The contrast/comparison between the two implementations might be helpful. > > https://github.com/norton/lets/tree/master/c_src > > and a pointer to one of the commands with optional async support: > > https://github.com/norton/lets/blob/master/c_src/lets_drv.cc#L577 > > Not sure if it is simple or not ... but it might help you with your development. > > thanks, > > > > On Jul 3, 2012, at 11:25 PM, Zabrane Mickael wrote: > >> Hi Serge, >> >> Where one can find a simple driver_async example to learn from? >> >> thx >> Zab >> >> On Jul 3, 2012, at 1:37 PM, Serge Aleynikov wrote: >> >>> Driver callbacks are executed by a scheduler thread in synchronous >>> manner, so what you are describing is a normal behavior of a linked-in >>> driver. Use driver_async to off-load a long computation to another >>> thread from a pool different from the scheduling threads. >>> >>> On 7/3/2012 7:20 AM, Zhemzhitsky Sergey wrote: >>>> Hi erlangers, >>>> >>>> >>>> >>>> Recently we have developed port (linked-in) driver and faced with some >>>> performance issues. >>>> >>>> >>>> >>>> The port uses ERL_DRV_FLAG_USE_PORT_LOCKING locking mode, so as we >>>> understand calls to different port instances should have not blocked >>>> each other. >>>> >>>> SMP count is greater than 1. >>>> >>>> >>>> >>>> So the first question is: >>>> >>>> Is it possible to have multiple port instances executing time-consuming >>>> operation in parallel, i.e. at the same time? >>>> >>>> >>>> >>>> We expected port instances do not influence on each other. However, all >>>> the calls are blocked but the first one. So the scheduler thread seems >>>> to be blocked during the port instance invocation. >>>> >>>> Is driver_async function required for the issues like above? >>>> >>>> Is it possible to unblock the scheduler thread before erlang:port_call >>>> completes? >>>> >>>> >>>> >>>> Best Regards, >>>> >>>> Sergey >>>> >>>> >>>> >>>> _______________________________________________________ >>>> >>>> >>>> >>>> The information contained in this message may be privileged and conf >>>> idential and protected from disclosure. If you are not the original >>>> intended recipient, you are hereby notified that any review, >>>> retransmission, dissemination, or other use of, or taking of any action >>>> in reliance upon, this information is prohibited. If you have received >>>> this communication in error, please notify the sender immediately by >>>> replying to this message and delete it from your computer. Thank you for >>>> your cooperation. Troika Dialog, Russia. >>>> >>>> If you need assistance please contact our Contact Center (+7495) 258 >>>> 0500 or go to www.troika.ru/eng/Contacts/system.wbp >>>> >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >> >> Regards, >> Zabrane >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > From lukas@REDACTED Tue Jul 3 16:56:41 2012 From: lukas@REDACTED (Lukas Larsson) Date: Tue, 3 Jul 2012 16:56:41 +0200 Subject: [erlang-questions] Does port driver block scheduler threads? In-Reply-To: References: <06139A918ACCA041BF46A0F36940C7FA39CD79D6@exch-mbx1.msk.trd.ru> <4FF2D960.7030505@aleynikov.org> Message-ID: If you want a much more complex example, go looking through https://github.com/erlang/otp/blob/maint/erts/emulator/drivers/common/efile_drv.c :D 4000 lines of asynch driver goodness :) Lukas On Tue, Jul 3, 2012 at 4:44 PM, Zabrane Mickael wrote: > Thanks for sharing Joseph. > > Zabrane > > On Jul 3, 2012, at 4:35 PM, Joseph Wayne Norton wrote: > >> >> Zabrane - >> >> Here is a nif and driver implementation for interfacing with leveldb. ?The contrast/comparison between the two implementations might be helpful. >> >> https://github.com/norton/lets/tree/master/c_src >> >> and a pointer to one of the commands with optional async support: >> >> https://github.com/norton/lets/blob/master/c_src/lets_drv.cc#L577 >> >> Not sure if it is simple or not ... but it might help you with your development. >> >> thanks, >> >> >> >> On Jul 3, 2012, at 11:25 PM, Zabrane Mickael wrote: >> >>> Hi Serge, >>> >>> Where one can find a simple driver_async example to learn from? >>> >>> thx >>> Zab >>> >>> On Jul 3, 2012, at 1:37 PM, Serge Aleynikov wrote: >>> >>>> Driver callbacks are executed by a scheduler thread in synchronous >>>> manner, so what you are describing is a normal behavior of a linked-in >>>> driver. ?Use driver_async to off-load a long computation to another >>>> thread from a pool different from the scheduling threads. >>>> >>>> On 7/3/2012 7:20 AM, Zhemzhitsky Sergey wrote: >>>>> Hi erlangers, >>>>> >>>>> >>>>> >>>>> Recently we have developed port (linked-in) driver and faced with some >>>>> performance issues. >>>>> >>>>> >>>>> >>>>> The port uses ERL_DRV_FLAG_USE_PORT_LOCKING locking mode, so as we >>>>> understand calls to different port instances should have not blocked >>>>> each other. >>>>> >>>>> SMP count is greater than 1. >>>>> >>>>> >>>>> >>>>> So the first question is: >>>>> >>>>> Is it possible to have multiple port instances executing time-consuming >>>>> operation in parallel, i.e. at the same time? >>>>> >>>>> >>>>> >>>>> We expected port instances do not influence on each other. However, all >>>>> the calls are blocked but the first one. So the scheduler thread seems >>>>> to be blocked during the port instance invocation. >>>>> >>>>> Is driver_async function required for the issues like above? >>>>> >>>>> Is it possible to unblock the scheduler thread before erlang:port_call >>>>> completes? >>>>> >>>>> >>>>> >>>>> Best Regards, >>>>> >>>>> Sergey >>>>> >>>>> >>>>> >>>>> _______________________________________________________ >>>>> >>>>> >>>>> >>>>> The information contained in this message may be privileged and conf >>>>> idential and protected from disclosure. If you are not the original >>>>> intended recipient, you are hereby notified that any review, >>>>> retransmission, dissemination, or other use of, or taking of any action >>>>> in reliance upon, this information is prohibited. If you have received >>>>> this communication in error, please notify the sender immediately by >>>>> replying to this message and delete it from your computer. Thank you for >>>>> your cooperation. Troika Dialog, Russia. >>>>> >>>>> If you need assistance please contact our Contact Center (+7495) 258 >>>>> 0500 or go to www.troika.ru/eng/Contacts/system.wbp >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> erlang-questions mailing list >>>>> erlang-questions@REDACTED >>>>> http://erlang.org/mailman/listinfo/erlang-questions >>>>> >>>> >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> Regards, >>> Zabrane >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From f@REDACTED Tue Jul 3 17:12:21 2012 From: f@REDACTED (Francesco Mazzoli) Date: Tue, 03 Jul 2012 16:12:21 +0100 Subject: [erlang-questions] Removing mnesia cluster node while mnesia is offline In-Reply-To: <87wr2lx7i4.wl%f@mazzo.li> References: <878vf6chx0.wl%f@mazzo.li> <87wr2lx7i4.wl%f@mazzo.li> Message-ID: <87vci4yj3e.wl%f@mazzo.li> At Tue, 03 Jul 2012 15:08:03 +0100, Francesco Mazzoli wrote: > It seems that you can delete the node schema while mnesia is offline > (via `mnesia:delete_schema/2'). > > This makes it possible to uncluster the node while offline, and then > re-cluster with the correct nodes afterwards. Just to clarify, this in the end is very similar to what Rudolph suggested, since deleting the schema will wipe everything out of mnesia - it's just a nicer way of doing so. -- Francesco * Often in error, never in doubt From mjtruog@REDACTED Tue Jul 3 18:02:03 2012 From: mjtruog@REDACTED (Michael Truog) Date: Tue, 03 Jul 2012 09:02:03 -0700 Subject: [erlang-questions] Does port driver block scheduler threads? In-Reply-To: References: <06139A918ACCA041BF46A0F36940C7FA39CD79D6@exch-mbx1.msk.trd.ru> <4FF2D960.7030505@aleynikov.org> Message-ID: <4FF3177B.2080502@gmail.com> Hi Zabrane, If you want to make it a bit easier, you can use GEPD and just set a flag (1 instead of 0) to make the function async (so the code generation uses driver_async). If you wanted to, you could look at the generated code, but it may be simpler to look at the GEPD code... it depends if the preprocessor scares you. The link is https://github.com/okeuday/generic-erlang-port--driver- Best Regards, Michael On 07/03/2012 07:25 AM, Zabrane Mickael wrote: > Hi Serge, > > Where one can find a simple driver_async example to learn from? > > thx > Zab > > On Jul 3, 2012, at 1:37 PM, Serge Aleynikov wrote: > >> Driver callbacks are executed by a scheduler thread in synchronous >> manner, so what you are describing is a normal behavior of a linked-in >> driver. Use driver_async to off-load a long computation to another >> thread from a pool different from the scheduling threads. >> >> On 7/3/2012 7:20 AM, Zhemzhitsky Sergey wrote: >>> Hi erlangers, >>> >>> >>> >>> Recently we have developed port (linked-in) driver and faced with some >>> performance issues. >>> >>> >>> >>> The port uses ERL_DRV_FLAG_USE_PORT_LOCKING locking mode, so as we >>> understand calls to different port instances should have not blocked >>> each other. >>> >>> SMP count is greater than 1. >>> >>> >>> >>> So the first question is: >>> >>> Is it possible to have multiple port instances executing time-consuming >>> operation in parallel, i.e. at the same time? >>> >>> >>> >>> We expected port instances do not influence on each other. However, all >>> the calls are blocked but the first one. So the scheduler thread seems >>> to be blocked during the port instance invocation. >>> >>> Is driver_async function required for the issues like above? >>> >>> Is it possible to unblock the scheduler thread before erlang:port_call >>> completes? >>> >>> >>> >>> Best Regards, >>> >>> Sergey >>> >>> >>> >>> _______________________________________________________ >>> >>> >>> >>> The information contained in this message may be privileged and conf >>> idential and protected from disclosure. If you are not the original >>> intended recipient, you are hereby notified that any review, >>> retransmission, dissemination, or other use of, or taking of any action >>> in reliance upon, this information is prohibited. If you have received >>> this communication in error, please notify the sender immediately by >>> replying to this message and delete it from your computer. Thank you for >>> your cooperation. Troika Dialog, Russia. >>> >>> If you need assistance please contact our Contact Center (+7495) 258 >>> 0500 or go to www.troika.ru/eng/Contacts/system.wbp >>> >>> >>> >>> >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > Regards, > Zabrane > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From spawn.think@REDACTED Wed Jul 4 09:32:29 2012 From: spawn.think@REDACTED (Ahmed Omar) Date: Wed, 4 Jul 2012 09:32:29 +0200 Subject: [erlang-questions] More Erlang Users' groups Message-ID: Hi everyone, In order to help local erlang users' communicate and organize, more location-oriented subgroups were created under Erlang/OTP professionals group (http://www.linkedin.com/groups?gid=90878). Recently added: EUG-SWISS: http://www.linkedin.com/groups?gid=4478020 EUG-India: http://www.linkedin.com/groups?gid=4506394 EUG-Finland: http://www.linkedin.com/groups?gid=4514705 Best Regards, Ahmed Omar -------------- next part -------------- An HTML attachment was scrubbed... URL: From henrik@REDACTED Wed Jul 4 14:10:28 2012 From: henrik@REDACTED (Henrik Nord) Date: Wed, 4 Jul 2012 14:10:28 +0200 Subject: [erlang-questions] [erlang-bugs] www.erlang.org not responding In-Reply-To: <1CD1916B-14C7-403D-ABD4-EAAF11A04573@feuerlabs.com> References: <1CD1916B-14C7-403D-ABD4-EAAF11A04573@feuerlabs.com> Message-ID: <4FF432B4.3090807@erlang.org> Hello The site was down due to a backup-script that went in to a tailspin. The issue is now resolved and everything is back up! -- /Henrik Nord Erlang/OTP From vasily@REDACTED Wed Jul 4 14:12:57 2012 From: vasily@REDACTED (Vasily Sulatskov) Date: Wed, 4 Jul 2012 14:12:57 +0200 Subject: [erlang-questions] Querying release version Message-ID: Hi All, I am using a release system for deployment. That means I have a properly generated .rel file with a meaningful release version string. Something like this: {release, {"myproject", "1.0.18-5256c9d80442"}, ... ... Is there a way to read that release version string during system's runtime? -- Best regards, Vasily Sulatskov From gustav.simonsson@REDACTED Wed Jul 4 14:24:08 2012 From: gustav.simonsson@REDACTED (Gustav Simonsson) Date: Wed, 04 Jul 2012 13:24:08 +0100 (BST) Subject: [erlang-questions] Querying release version In-Reply-To: Message-ID: <399011f0-e0c1-4274-9dd2-4e965b64b1d3@knuth> http://www.erlang.org/doc/man/reltool.html#get_rel-2 http://www.erlang.org/doc/apps/reltool/reltool_examples.html#id61354 // Gustav Sent from my PC ----- Original Message ----- > From: "Vasily Sulatskov" > To: erlang-questions@REDACTED > Sent: Wednesday, 4 July, 2012 2:12:57 PM > Subject: [erlang-questions] Querying release version > > Hi All, > > I am using a release system for deployment. That means I have a > properly generated .rel file with a meaningful release version > string. > > Something like this: > > {release, > {"myproject", "1.0.18-5256c9d80442"}, > ... > ... > > > Is there a way to read that release version string during system's > runtime? > > -- > Best regards, > Vasily Sulatskov > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From andre@REDACTED Wed Jul 4 15:04:00 2012 From: andre@REDACTED (=?ISO-8859-1?Q?Andr=E9_Graf?=) Date: Wed, 4 Jul 2012 15:04:00 +0200 Subject: [erlang-questions] Querying release version In-Reply-To: <399011f0-e0c1-4274-9dd2-4e965b64b1d3@knuth> References: <399011f0-e0c1-4274-9dd2-4e965b64b1d3@knuth> Message-ID: release_handler:which_releases() will do it. On 4 July 2012 14:24, Gustav Simonsson wrote: > > http://www.erlang.org/doc/man/reltool.html#get_rel-2 > http://www.erlang.org/doc/apps/reltool/reltool_examples.html#id61354 > > // Gustav > > Sent from my PC > > ----- Original Message ----- >> From: "Vasily Sulatskov" >> To: erlang-questions@REDACTED >> Sent: Wednesday, 4 July, 2012 2:12:57 PM >> Subject: [erlang-questions] Querying release version >> >> Hi All, >> >> I am using a release system for deployment. That means I have a >> properly generated .rel file with a meaningful release version >> string. >> >> Something like this: >> >> {release, >> {"myproject", "1.0.18-5256c9d80442"}, >> ... >> ... >> >> >> Is there a way to read that release version string during system's >> runtime? >> >> -- >> Best regards, >> Vasily Sulatskov >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From vasily@REDACTED Wed Jul 4 15:21:53 2012 From: vasily@REDACTED (Vasily Sulatskov) Date: Wed, 4 Jul 2012 15:21:53 +0200 Subject: [erlang-questions] Querying release version In-Reply-To: <399011f0-e0c1-4274-9dd2-4e965b64b1d3@knuth> References: <399011f0-e0c1-4274-9dd2-4e965b64b1d3@knuth> Message-ID: That would probably work, however I am getting a following error when I am trying to start a reltool server: exception error: no match of right hand side value {error,beam_lib, {missing_chunk,'/usr/opt/foo/lib/bar/ebin/bar_app.beam', "Attr"}} On Wed, Jul 4, 2012 at 2:24 PM, Gustav Simonsson wrote: > > http://www.erlang.org/doc/man/reltool.html#get_rel-2 > http://www.erlang.org/doc/apps/reltool/reltool_examples.html#id61354 > > // Gustav > > Sent from my PC > > ----- Original Message ----- >> From: "Vasily Sulatskov" >> To: erlang-questions@REDACTED >> Sent: Wednesday, 4 July, 2012 2:12:57 PM >> Subject: [erlang-questions] Querying release version >> >> Hi All, >> >> I am using a release system for deployment. That means I have a >> properly generated .rel file with a meaningful release version >> string. >> >> Something like this: >> >> {release, >> {"myproject", "1.0.18-5256c9d80442"}, >> ... >> ... >> >> >> Is there a way to read that release version string during system's >> runtime? >> >> -- >> Best regards, >> Vasily Sulatskov >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- Best regards, Vasily Sulatskov From ivan@REDACTED Wed Jul 4 15:40:34 2012 From: ivan@REDACTED (Ivan Uemlianin) Date: Wed, 04 Jul 2012 14:40:34 +0100 Subject: [erlang-questions] starting a release with nohup Message-ID: <4FF447D2.5020408@llaisdy.com> Dear All We have a web application which is a hybrid of python and erlang. As part of the deployment infrastructure we can restart the application over ssh, using a command like nohup webapp start "webapp start" just calls "pythonapp start" and "erlangapp start". Although nohup isn't necessary for the erlang app, it seems to be necessary for the python app. Recently we packaged the erlang app as a release, with start and stop functions. So, "erlangapp start" calls "/path/to/erlangapp/rel/eappnode/bin/eappnode start". The release set up is much nicer than before, *but* it doesn't seem to want to play with nohup. Running the above returns an erlang ebadf error message (see below). Is there a way to start a release using nohup? With thanks and best wishes Ivan ** error messages {error_logger,{{2012,7,4},{9,14,46}},supervisor_report, [{supervisor,{<0.21.0>,user_sup}},{errorContext,child_terminated},{reason,ebadf},{offender,[{pid,<0.22.0>},{mod,user_sup}]}]} {error_logger,{{2012,7,4},{9,14,46}}, "** Generic server ~p terminating \n ** Last message in was ~p~n ** When Server state == ~p~n ** Reason for termination == ~n ** ~p~n", [<0.21.0>, {'EXIT',<0.22.0>,ebadf}, {state,user_sup,undefined,<0.22.0>,{<0.21.0>,user_sup}}, ebadf]} {error_logger,{{2012,7,4},{9,14,46}},crash_report, [[{initial_call,{supervisor_bridge,user_sup,['Argument__1']}},{pid,<0.21.0>},{registered_name,[]},{error_info,{exit,ebadf,[{gen_server,terminate,6},{proc_lib,init_p_do_apply,3}]}},{ancestors,[kernel_sup,<0.9.0>]},{messages,[]},{links,[<0.10.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,987},{stack_size,24},{reductions,206}],[]]} {error_logger,{{2012,7,4},{9,14,46}},supervisor_report, [{supervisor,{local,kernel_sup}},{errorContext,child_terminated},{reason,ebadf},{offender,[{pid,<0.21.0>},{name,user},{mfargs,{user_sup,start,[]}},{restart_type,temporary},{shutdown,2000},{child_type,supervisor}]}]} -- ============================================================ Ivan A. Uemlianin PhD Llaisdy Speech Technology Research and Development ivan@REDACTED www.llaisdy.com llaisdy.wordpress.com github.com/llaisdy www.linkedin.com/in/ivanuemlianin "hilaritas excessum habere nequit" (Spinoza, Ethica, IV, XLII) ============================================================ From vasily@REDACTED Wed Jul 4 17:36:39 2012 From: vasily@REDACTED (Vasily Sulatskov) Date: Wed, 4 Jul 2012 17:36:39 +0200 Subject: [erlang-questions] Querying release version In-Reply-To: References: <399011f0-e0c1-4274-9dd2-4e965b64b1d3@knuth> Message-ID: I suppose I am not following all instructions for deploying a release correctly because release_handler:which_releases() lists only standard releases like OTP My release is not included in that list, but this function seems to be doing exactly what I need. On Wed, Jul 4, 2012 at 3:04 PM, Andr? Graf wrote: > release_handler:which_releases() > will do it. > > On 4 July 2012 14:24, Gustav Simonsson > wrote: >> >> http://www.erlang.org/doc/man/reltool.html#get_rel-2 >> http://www.erlang.org/doc/apps/reltool/reltool_examples.html#id61354 >> >> // Gustav >> >> Sent from my PC >> >> ----- Original Message ----- >>> From: "Vasily Sulatskov" >>> To: erlang-questions@REDACTED >>> Sent: Wednesday, 4 July, 2012 2:12:57 PM >>> Subject: [erlang-questions] Querying release version >>> >>> Hi All, >>> >>> I am using a release system for deployment. That means I have a >>> properly generated .rel file with a meaningful release version >>> string. >>> >>> Something like this: >>> >>> {release, >>> {"myproject", "1.0.18-5256c9d80442"}, >>> ... >>> ... >>> >>> >>> Is there a way to read that release version string during system's >>> runtime? >>> >>> -- >>> Best regards, >>> Vasily Sulatskov >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- Best regards, Vasily Sulatskov From anders.nygren@REDACTED Wed Jul 4 18:37:42 2012 From: anders.nygren@REDACTED (Anders Nygren) Date: Wed, 4 Jul 2012 11:37:42 -0500 Subject: [erlang-questions] Querying release version In-Reply-To: References: <399011f0-e0c1-4274-9dd2-4e965b64b1d3@knuth> Message-ID: Also useful is init:script_id() /Anders On Wed, Jul 4, 2012 at 10:36 AM, Vasily Sulatskov wrote: > I suppose I am not following all instructions for deploying a release > correctly because > > release_handler:which_releases() lists only standard releases like OTP > > My release is not included in that list, but this function seems to be > doing exactly what I need. > > On Wed, Jul 4, 2012 at 3:04 PM, Andr? Graf wrote: >> release_handler:which_releases() >> will do it. >> >> On 4 July 2012 14:24, Gustav Simonsson >> wrote: >>> >>> http://www.erlang.org/doc/man/reltool.html#get_rel-2 >>> http://www.erlang.org/doc/apps/reltool/reltool_examples.html#id61354 >>> >>> // Gustav >>> >>> Sent from my PC >>> >>> ----- Original Message ----- >>>> From: "Vasily Sulatskov" >>>> To: erlang-questions@REDACTED >>>> Sent: Wednesday, 4 July, 2012 2:12:57 PM >>>> Subject: [erlang-questions] Querying release version >>>> >>>> Hi All, >>>> >>>> I am using a release system for deployment. That means I have a >>>> properly generated .rel file with a meaningful release version >>>> string. >>>> >>>> Something like this: >>>> >>>> {release, >>>> {"myproject", "1.0.18-5256c9d80442"}, >>>> ... >>>> ... >>>> >>>> >>>> Is there a way to read that release version string during system's >>>> runtime? >>>> >>>> -- >>>> Best regards, >>>> Vasily Sulatskov >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > > > -- > Best regards, > Vasily Sulatskov > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From bchesneau@REDACTED Wed Jul 4 21:57:10 2012 From: bchesneau@REDACTED (Benoit Chesneau) Date: Wed, 4 Jul 2012 21:57:10 +0200 Subject: [erlang-questions] releases & plugins Message-ID: Hi all, Why a release can be extended with custom apps, I am looking for a way to install a plugin directly in a release to extend it on demand and distribute custom add-ons more easily. I'm not sure how to do that in Erlang actually. Does some project alreaydy do that? How? I could eventually distribute binaries but then it imply I do it for each platforms supported. Is there any smarter trick? - beno?t From zabrane3@REDACTED Thu Jul 5 01:44:49 2012 From: zabrane3@REDACTED (Zabrane Mickael) Date: Thu, 5 Jul 2012 01:44:49 +0200 Subject: [erlang-questions] [ANN] etest and etest_http 1.0.0.beta In-Reply-To: <4FD85D89.8040809@berlin.ccc.de> References: <4FD85D89.8040809@berlin.ccc.de> Message-ID: <5FB1B946-BBE3-4991-8547-F7BD7555D58A@gmail.com> Hi John, On Jun 13, 2012, at 11:29 AM, John-Paul Bader wrote: > A video demonstrating etest_http will follow shortly. Any video on etest_http? Any progress on etest in general? Regards, Zabrane -------------- next part -------------- An HTML attachment was scrubbed... URL: From hukl@REDACTED Thu Jul 5 10:22:40 2012 From: hukl@REDACTED (John-Paul Bader) Date: Thu, 05 Jul 2012 10:22:40 +0200 Subject: [erlang-questions] [ANN] etest and etest_http 1.0.0.beta In-Reply-To: <5FB1B946-BBE3-4991-8547-F7BD7555D58A@gmail.com> References: <4FD85D89.8040809@berlin.ccc.de> <5FB1B946-BBE3-4991-8547-F7BD7555D58A@gmail.com> Message-ID: <4FF54ED0.5070709@berlin.ccc.de> Hey, i've just uploaded a video showing etest_http in action. It does not have an audio track but I'm working on a proper setup for that. https://vimeo.com/45212367 Right now in my team at work we are using etest heavily and while we have no general problems with it we have a couple of nice to have features on the list which will probably make it into etest in the following weeks. * random test execution order (each run the order of tests is different) * focus tests (prefix a test with focus_ and only that test will be executed * run tests in a separated process ~ John Zabrane Mickael wrote: > Hi John, > > On Jun 13, 2012, at 11:29 AM, John-Paul Bader wrote: > >> A video demonstrating etest_http will follow shortly. > > Any video on etest_http? > Any progress on etest in general? > > Regards, > Zabrane > From hans.bolinder@REDACTED Thu Jul 5 10:36:46 2012 From: hans.bolinder@REDACTED (Hans Bolinder) Date: Thu, 5 Jul 2012 10:36:46 +0200 Subject: [erlang-questions] Trouble with 'make docs' In-Reply-To: References: <242A68FD-469F-4B31-959C-D8F0566F0415@cs.otago.ac.nz> Message-ID: <20469.21022.400935.313184@ornendil.otp.ericsson.se> [Richard O'Keefe:] > I looked at one of the generated PDFs, > erl-interface-3.7.7.pdf > ... > The PDF contains stuff that looks like > > intei_encode_ulong(char *buf, int *index, unsigned long p) > intei_x_encode_ulong(ei_x_buff* x, unsigned long p) > > In no case do I see a space generated between and . > ... > I'm seeing exactly the same issue in the man pages. Thank you for pointing out this bug. Best regards, Hans Bolinder, Erlang/OTP team, Ericsson From hirotnkg@REDACTED Thu Jul 5 10:53:58 2012 From: hirotnkg@REDACTED (Yoshihiro Tanaka) Date: Thu, 5 Jul 2012 01:53:58 -0700 Subject: [erlang-questions] port/socket leak ? Message-ID: Hi, I'm using lhttpc and seeing some ports/sockets are leaking on R14B-04. Using inet:i(), it looks like below: 1437362 inet_tcp 0 0 <0.18131.596> *:* CLOSED Using erlang:port_info/1, it looks like below: [[{name,"tcp_inet"}, {links,[]}, {id,1437362}, {connected,<4283.18131.596>}, {input,0}, {output,0}]] My question is, when a port does not have linked process, or connected process is already gone, shouldn't it recycled by Erlang VM ? If not, why ? (These ports can be recycled using erlang:port_close/1) Thank you Yoshihiro From watson.timothy@REDACTED Thu Jul 5 11:10:03 2012 From: watson.timothy@REDACTED (Tim Watson) Date: Thu, 5 Jul 2012 10:10:03 +0100 Subject: [erlang-questions] Reading comments/annotations inside parse_transforms Message-ID: There doesn't appear to be any way of doing this currently. There is support in erl_syntax for working with these and there is also an erl_comment_scan module in the syntax-tools application which parses and returns comments from a file (or whatever). What there doesn't appear to be, is a means to get erl_parse to return comments as attributes (?) in the AST that the compiler passes to a parse_transform. My question then, is this: what is the best (only!?) way to process comments at the same time as source code in a parse_transform? So far, given that this seems unlikely to work OOTB, I'm assuming I'll have to do something like: 1. once you hit the 'file' attribute, parse the comments and stash these away in a useful place 2. process the forms as usual, comparing each line number for each form, against the line numbers stored for the comments 3. when you get an interleaving of code and comments, you know where the comments reside in the source file in relation to the form you're currently working with I *can* do this of course, but it seems like an awful lot of hard work. Why can't comments be preserved in the AST passed to a parse transform, and then just dropped later on!? Is there an easier way of doing this that I'm currently missing? Cheers, Tim From zerthurd@REDACTED Thu Jul 5 11:12:01 2012 From: zerthurd@REDACTED (Maxim Treskin) Date: Thu, 5 Jul 2012 16:12:01 +0700 Subject: [erlang-questions] Dialyzer cannot catch gen_server callback return type error Message-ID: Hello I got error when gen_server callback returns not a proper process state, but another erlang term(), and dialyzer not warns me about this. After specification of Module:handle_cast/2 with -spec handle_cast(Msg :: term(), State :: #state{}) -> {noreply, NewState :: #state{}}. where #state{} is record of process state, it not warns too (it also not warns about {stop, normal, State} return value). Code of callback: handle_cast({message}, _State) -> NewState = {bugotak}, {noreply, NewState}; Is it possible to make that dialyzer can see this mistake? Does it check types of gen_server callbacks? -- Max Treskin -------------- next part -------------- An HTML attachment was scrubbed... URL: From zabrane3@REDACTED Thu Jul 5 11:25:32 2012 From: zabrane3@REDACTED (Zabrane Mickael) Date: Thu, 5 Jul 2012 11:25:32 +0200 Subject: [erlang-questions] [ANN] etest and etest_http 1.0.0.beta In-Reply-To: <4FF54ED0.5070709@berlin.ccc.de> References: <4FD85D89.8040809@berlin.ccc.de> <5FB1B946-BBE3-4991-8547-F7BD7555D58A@gmail.com> <4FF54ED0.5070709@berlin.ccc.de> Message-ID: <80596240-1047-4238-9F57-D2FC7ADC815A@gmail.com> Hi John, Very nice video! Last time, I suggested you if possible to "handle sessions" inside etest_http (i.e be able to test a complete HTTP session and not only separated requests). Are you considering this in your TODO? Regards, Zabrane On Jul 5, 2012, at 10:22 AM, John-Paul Bader wrote: > Hey, > > > i've just uploaded a video showing etest_http in action. > It does not have an audio track but I'm working on a proper setup for that. > > https://vimeo.com/45212367 > > Right now in my team at work we are using etest heavily and while we have no general problems with it we have a couple of nice to have features on the list which will probably make it into etest in the following weeks. > > * random test execution order (each run the order of tests is different) > * focus tests (prefix a test with focus_ and only that test will be executed > * run tests in a separated process > > ~ John > > Zabrane Mickael wrote: >> Hi John, >> >> On Jun 13, 2012, at 11:29 AM, John-Paul Bader wrote: >> >>> A video demonstrating etest_http will follow shortly. >> >> Any video on etest_http? >> Any progress on etest in general? >> >> Regards, >> Zabrane >> > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From desired.mta@REDACTED Thu Jul 5 13:17:48 2012 From: desired.mta@REDACTED (Motiejus =?utf-8?Q?Jak=C5=A1tys?=) Date: Thu, 5 Jul 2012 14:17:48 +0300 Subject: [erlang-questions] Dialyzer cannot catch gen_server callback return type error In-Reply-To: References: Message-ID: <20120705111748.GB3724@localhost> On Thu, Jul 05, 2012 at 04:12:01PM +0700, Maxim Treskin wrote: > Hello > > I got error when gen_server callback returns not a proper process state, > but another erlang term(), and dialyzer not warns me about this. > After specification of Module:handle_cast/2 with > > -spec handle_cast(Msg :: term(), State :: #state{}) -> {noreply, NewState > :: #state{}}. > > where #state{} is record of process state, I don't know about this one, > it not warns too (it also not warns about {stop, normal, State} return > value). but this is fine. Your function returns a *subtype* of what Mod:handle_cast/2 is supposed to return. If you added something that is not in return values of handle_cast, like this: -spec handle_cast(term(), #state{}) -> {noreply, NewState} | wrong. Dialyzer will warn you that gen_server is not expecting 'wrong'. > Code of callback: > > handle_cast({message}, _State) -> > NewState = {bugotak}, > {noreply, NewState}; > > Is it possible to make that dialyzer can see this mistake? Does it check > types of gen_server callbacks? But this one would be interesting indeed. Motiejus From zerthurd@REDACTED Thu Jul 5 13:44:12 2012 From: zerthurd@REDACTED (Maxim Treskin) Date: Thu, 5 Jul 2012 18:44:12 +0700 Subject: [erlang-questions] Dialyzer cannot catch gen_server callback return type error In-Reply-To: <20120705111748.GB3724@localhost> References: <20120705111748.GB3724@localhost> Message-ID: On 5 July 2012 18:17, Motiejus Jak?tys wrote: > On Thu, Jul 05, 2012 at 04:12:01PM +0700, Maxim Treskin wrote: > > Hello > > > > I got error when gen_server callback returns not a proper process state, > > but another erlang term(), and dialyzer not warns me about this. > > After specification of Module:handle_cast/2 with > > > > -spec handle_cast(Msg :: term(), State :: #state{}) -> {noreply, NewState > > :: #state{}}. > > > > where #state{} is record of process state, > > I don't know about this one, > > > it not warns too (it also not warns about {stop, normal, State} return > > value). > > but this is fine. Your function returns a *subtype* of what > Mod:handle_cast/2 is supposed to return. > > No, handle_cast really returns {stop, normal, State}, but this return type not described in -spec of handle_cast, function returns not a subtype of declared type. Dialyzer is silent. > If you added something that is not in return values of handle_cast, like > this: > > -spec handle_cast(term(), #state{}) -> {noreply, NewState} | wrong. > > Dialyzer will warn you that gen_server is not expecting 'wrong'. > Yes, I see such warning: my_module.erl:215: The return type 'wrong' in the specification of handle_cast/2 is not a subtype of {'noreply',_} | {'noreply',_,'hibernate' | 'infinity' | non_neg_integer()} | {'stop',_,_}, which is the expected return type for the callback of gen_server behaviour my_module.erl:215: The specification for my_module:handle_cast/2 states that the function might also return 'wrong' but the inferred return is {'noreply',_} | {'stop',{'error',{_,_}},{'state',_}} It seems dialyzer uses less strict declaration of callback from behaviour -callback specification. I think it should be more appreciated to overlap -callback specification from behavior with -spec in callback module. When my_module.erl not contains -spec handle_cast(...) -> ..., dialyzer should use type description from gen_server.erl -callback declaration, but if we want to restrict this type in my_module.erl with -spec declaration this value should be used. > > > Code of callback: > > > > handle_cast({message}, _State) -> > > NewState = {bugotak}, > > {noreply, NewState}; > > > > Is it possible to make that dialyzer can see this mistake? Does it check > > types of gen_server callbacks? > > But this one would be interesting indeed. > > Motiejus > -- Max Treskin -------------- next part -------------- An HTML attachment was scrubbed... URL: From Sergey_Zhemzhitsky@REDACTED Thu Jul 5 14:07:09 2012 From: Sergey_Zhemzhitsky@REDACTED (Zhemzhitsky Sergey) Date: Thu, 5 Jul 2012 12:07:09 +0000 Subject: [erlang-questions] What is the best location to place common type specs? Message-ID: <06139A918ACCA041BF46A0F36940C7FA39CE0E22@exch-mbx1.msk.trd.ru> Hi guys, I have multiple .hrl files with multiple records. I?d like to define separate types for these records and their fields. These records will be used across multiple modules, and I?m wondering whether it is legal to define type specs in the hrl files? Or would it be better to have a separate module with exported common type specs? Best Regards, Sergey _______________________________________________________ The information contained in this message may be privileged and conf idential and protected from disclosure. If you are not the original intended recipient, you are hereby notified that any review, retransmission, dissemination, or other use of, or taking of any action in reliance upon, this information is prohibited. If you have received this communication in error, please notify the sender immediately by replying to this message and delete it from your computer. Thank you for your cooperation. Troika Dialog, Russia. If you need assistance please contact our Contact Center (+7495) 258 0500 or go to www.troika.ru/eng/Contacts/system.wbp -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Thu Jul 5 14:11:21 2012 From: essen@REDACTED (=?windows-1252?Q?Lo=EFc_Hoguin?=) Date: Thu, 05 Jul 2012 14:11:21 +0200 Subject: [erlang-questions] What is the best location to place common type specs? In-Reply-To: <06139A918ACCA041BF46A0F36940C7FA39CE0E22@exch-mbx1.msk.trd.ru> References: <06139A918ACCA041BF46A0F36940C7FA39CE0E22@exch-mbx1.msk.trd.ru> Message-ID: <4FF58469.8030805@ninenines.eu> Hello, Your specs should belong to a single module. module1:myrec() isn't the same type as module2:myrec(). A bit off-topic but I wouldn't use a record over multiple modules (even though I've done it in the past, this is getting fixed). It's better to have your record belong to one module, make it opaque and to add functions to manipulate it without having to know what's in it. This way you can modify your record only in one place and not have to worry about breaking code everywhere. On 07/05/2012 02:07 PM, Zhemzhitsky Sergey wrote: > Hi guys, > > I have multiple .hrl files with multiple records. I?d like to define > separate types for these records and their fields. > > These records will be used across multiple modules, and I?m wondering > whether it is legal to define type specs in the hrl files? > > Or would it be better to have a separate module with exported common > type specs? > > Best Regards, > > Sergey > > _______________________________________________________ > > The information contained in this message may be privileged and conf > idential and protected from disclosure. If you are not the original > intended recipient, you are hereby notified that any review, > retransmission, dissemination, or other use of, or taking of any action > in reliance upon, this information is prohibited. If you have received > this communication in error, please notify the sender immediately by > replying to this message and delete it from your computer. Thank you for > your cooperation. Troika Dialog, Russia. > > If you need assistance please contact our Contact Center (+7495) 258 > 0500 or go to www.troika.ru/eng/Contacts/system.wbp > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From Sergey_Zhemzhitsky@REDACTED Thu Jul 5 14:25:10 2012 From: Sergey_Zhemzhitsky@REDACTED (Zhemzhitsky Sergey) Date: Thu, 5 Jul 2012 12:25:10 +0000 Subject: [erlang-questions] What is the best location to place common type specs? In-Reply-To: <4FF58469.8030805@ninenines.eu> References: <06139A918ACCA041BF46A0F36940C7FA39CE0E22@exch-mbx1.msk.trd.ru> <4FF58469.8030805@ninenines.eu> Message-ID: <06139A918ACCA041BF46A0F36940C7FA39CE0E77@exch-mbx1.msk.trd.ru> Hello Lo?c, Thanks a lot for suggestions. Best Regards, Sergey -----Original Message----- From: Lo?c Hoguin [mailto:essen@REDACTED] Sent: Thursday, July 05, 2012 4:11 PM To: Zhemzhitsky Sergey Cc: erlang-questions@REDACTED Subject: Re: [erlang-questions] What is the best location to place common type specs? Hello, Your specs should belong to a single module. module1:myrec() isn't the same type as module2:myrec(). A bit off-topic but I wouldn't use a record over multiple modules (even though I've done it in the past, this is getting fixed). It's better to have your record belong to one module, make it opaque and to add functions to manipulate it without having to know what's in it. This way you can modify your record only in one place and not have to worry about breaking code everywhere. On 07/05/2012 02:07 PM, Zhemzhitsky Sergey wrote: > Hi guys, > > I have multiple .hrl files with multiple records. I?d like to define > separate types for these records and their fields. > > These records will be used across multiple modules, and I?m wondering > whether it is legal to define type specs in the hrl files? > > Or would it be better to have a separate module with exported common > type specs? > > Best Regards, > > Sergey > > _______________________________________________________ > > The information contained in this message may be privileged and conf > idential and protected from disclosure. If you are not the original > intended recipient, you are hereby notified that any review, > retransmission, dissemination, or other use of, or taking of any > action in reliance upon, this information is prohibited. If you have > received this communication in error, please notify the sender > immediately by replying to this message and delete it from your > computer. Thank you for your cooperation. Troika Dialog, Russia. > > If you need assistance please contact our Contact Center (+7495) 258 > 0500 or go to www.troika.ru/eng/Contacts/system.wbp > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From aronisstav@REDACTED Thu Jul 5 15:22:06 2012 From: aronisstav@REDACTED (Stavros Aronis) Date: Thu, 5 Jul 2012 06:22:06 -0700 (PDT) Subject: [erlang-questions] Dialyzer cannot catch gen_server callback return type error In-Reply-To: References: <20120705111748.GB3724@localhost> Message-ID: <21efd459-0e1a-41d4-a043-e472c064c85d@googlegroups.com> Hi! Your description is not very clear, so it might be better to send me some code snippet off-list, but let me put some facts here to make sure that the expectations from Dialyzer are 'reasonable'! - Dialyzer first uses the -callback attibutes of the behaviour and will always warn if the arguments or return value of a callback function are not subtypes of those attributes. If e.g. the return value of a callback is expected to be {noreply, term()} and the callback always returns {reply, term(), term()}, Dialyzer will warn about this. - As long as your implementation can possibly return an acceptable value, Dialyzer is happy. You might have a callback that either returns a correct value or something else that cannot be handled by the behaviour, but Dialyzer will assume that the correct term will always be returned. This happens because Dialyzer should never emit a false warning and this additional bad value might have been inferred because Dialyzer's analysis was not strong enough to exclude it. - Spec attributes are also taken into account when anayzing callbacks in the following manner: -- First the specs themselves are checked for compatibility with the -callback attributes of the behaviour. Here it is the user (and not Dialyzer) that writes these specs, so if these specs allow more values you get a warning. The rationale is that if you are providing a spec it should be at least as restrictive as the callback attribute. -- Second, the spec is used to check the arguments and return value of the function, as is the case with any spec. For gen_server, the -callback attributes can be seen here. There, the State and NewState can be any terms. If you also have a spec saying that this callback function should return {noreply, State :: #state{}} and you return {noreply, {bugotak}} then you are violating your own spec (but not the callback attribute of gen_server) and you should get a warning. Is this not the case when you run Dialyzer? Stavros -------------- next part -------------- An HTML attachment was scrubbed... URL: From zerthurd@REDACTED Thu Jul 5 15:31:42 2012 From: zerthurd@REDACTED (Maxim Treskin) Date: Thu, 5 Jul 2012 20:31:42 +0700 Subject: [erlang-questions] Dialyzer cannot catch gen_server callback return type error In-Reply-To: <21efd459-0e1a-41d4-a043-e472c064c85d@googlegroups.com> References: <20120705111748.GB3724@localhost> <21efd459-0e1a-41d4-a043-e472c064c85d@googlegroups.com> Message-ID: Hello! File attached. Just compile it with erlc +debug_info my_module.erl then dialyzer . --plt ~/.r15b01_dialyzer.plt --no_native -Werror_handling -Wrace_conditions -Wunderspecs and I see: Checking whether the PLT /home/zert/.r15b01_dialyzer.plt is up-to-date... yes Proceeding with analysis... done in 0m0.39s done (passed successfully) On 5 July 2012 20:22, Stavros Aronis wrote: > Hi! > > Your description is not very clear, so it might be better to send me some > code snippet off-list, but let me put some facts here to make sure that the > expectations from Dialyzer are 'reasonable'! > > - Dialyzer first uses the -callback attibutes of the behaviour and will > always warn if the arguments or return value of a callback function are not > subtypes of those attributes. If e.g. the return value of a callback is > expected to be {noreply, term()} and the callback always returns {reply, > term(), term()}, Dialyzer will warn about this. > - As long as your implementation can possibly return an acceptable value, > Dialyzer is happy. You might have a callback that either returns a correct > value or something else that cannot be handled by the behaviour, but > Dialyzer will assume that the correct term will always be returned. This > happens because Dialyzer should never emit a false warning and this > additional bad value might have been inferred because Dialyzer's analysis > was not strong enough to exclude it. > - Spec attributes are also taken into account when anayzing callbacks in > the following manner: > -- First the specs themselves are checked for compatibility with the > -callback attributes of the behaviour. Here it is the user (and not > Dialyzer) that writes these specs, so if these specs allow more values you > get a warning. The rationale is that if you are providing a spec it should > be at least as restrictive as the callback attribute. > -- Second, the spec is used to check the arguments and return value of the > function, as is the case with any spec. > > For gen_server, the -callback attributes can be seen here. > There, the State and NewState can be any terms. If you also have a spec > saying that this callback function should return {noreply, State :: > #state{}} and you return {noreply, {bugotak}} then you are violating your > own spec (but not the callback attribute of gen_server) and you should get > a warning. Is this not the case when you run Dialyzer? > > Stavros > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -- Max Treskin -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: my_module.erl Type: application/octet-stream Size: 957 bytes Desc: not available URL: From mattevans123@REDACTED Thu Jul 5 16:16:39 2012 From: mattevans123@REDACTED (Matthew Evans) Date: Thu, 5 Jul 2012 10:16:39 -0400 Subject: [erlang-questions] starting a release with nohup In-Reply-To: <4FF447D2.5020408@llaisdy.com> References: <4FF447D2.5020408@llaisdy.com> Message-ID: I'm using it all the time. Not sure if you are doing it, but I set the "-detached" and "-noshell" options to the Erlang "erl" command. i.e: nohup erl -detached -noshell Matt ---------------------------------------- > Date: Wed, 4 Jul 2012 14:40:34 +0100 > From: ivan@REDACTED > To: erlang-questions@REDACTED > Subject: [erlang-questions] starting a release with nohup > > Dear All > > We have a web application which is a hybrid of python and erlang. As > part of the deployment infrastructure we can restart the application > over ssh, using a command like > > nohup webapp start > > "webapp start" just calls "pythonapp start" and "erlangapp start". > Although nohup isn't necessary for the erlang app, it seems to be > necessary for the python app. > > Recently we packaged the erlang app as a release, with start and stop > functions. So, "erlangapp start" calls > "/path/to/erlangapp/rel/eappnode/bin/eappnode start". > > The release set up is much nicer than before, *but* it doesn't seem to > want to play with nohup. Running the above returns an erlang ebadf > error message (see below). > > Is there a way to start a release using nohup? > > With thanks and best wishes > > Ivan > > > ** error messages > > {error_logger,{{2012,7,4},{9,14,46}},supervisor_report, > > [{supervisor,{<0.21.0>,user_sup}},{errorContext,child_terminated},{reason,ebadf},{offender,[{pid,<0.22.0>},{mod,user_sup}]}]} > > {error_logger,{{2012,7,4},{9,14,46}}, > "** Generic server ~p terminating \n > ** Last message in was ~p~n > ** When Server state == ~p~n > ** Reason for termination == ~n > ** ~p~n", > [<0.21.0>, > {'EXIT',<0.22.0>,ebadf}, > {state,user_sup,undefined,<0.22.0>,{<0.21.0>,user_sup}}, > ebadf]} > > {error_logger,{{2012,7,4},{9,14,46}},crash_report, > > [[{initial_call,{supervisor_bridge,user_sup,['Argument__1']}},{pid,<0.21.0>},{registered_name,[]},{error_info,{exit,ebadf,[{gen_server,terminate,6},{proc_lib,init_p_do_apply,3}]}},{ancestors,[kernel_sup,<0.9.0>]},{messages,[]},{links,[<0.10.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,987},{stack_size,24},{reductions,206}],[]]} > > {error_logger,{{2012,7,4},{9,14,46}},supervisor_report, > > [{supervisor,{local,kernel_sup}},{errorContext,child_terminated},{reason,ebadf},{offender,[{pid,<0.21.0>},{name,user},{mfargs,{user_sup,start,[]}},{restart_type,temporary},{shutdown,2000},{child_type,supervisor}]}]} > > > -- > ============================================================ > Ivan A. Uemlianin PhD > Llaisdy > Speech Technology Research and Development > > ivan@REDACTED > www.llaisdy.com > llaisdy.wordpress.com > github.com/llaisdy > www.linkedin.com/in/ivanuemlianin > > "hilaritas excessum habere nequit" > (Spinoza, Ethica, IV, XLII) > ============================================================ > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From aronisstav@REDACTED Thu Jul 5 16:36:52 2012 From: aronisstav@REDACTED (Stavros Aronis) Date: Thu, 5 Jul 2012 17:36:52 +0300 Subject: [erlang-questions] Dialyzer cannot catch gen_server callback return type error In-Reply-To: References: <20120705111748.GB3724@localhost> <21efd459-0e1a-41d4-a043-e472c064c85d@googlegroups.com> Message-ID: Hi again, your example shows another case of the "might also be correct" rationale of Dialyzer. Your full code is: -spec handle_cast(term(), #state{}) -> {noreply, NewState :: #state{}}. handle_cast({asd}, _State) -> NewState = {bugotak}, {noreply, NewState}; handle_cast(_Msg, State) -> {noreply, State}. It is true that the return value of the first clause violates the spec, but this information is lost within Dialyzer as the "general" type signature that Dialyzer will find and use is my_module:handle_cast(term(),term()) -> {'noreply',term()} due to the second clause. This more general type signature "absorbs" and hides the incorrect result. The function "can work" if you only call with a message different than {asd} and provide a correct #state{} as input, so Dialyzer remains silent. If, on the other hand, you had just the first clause, or both clauses returned a value that couldn't be #state{}, you would get a warning. Remember that Dialyzer does not promise to find all the bugs in your code. It might be interesting and possible to check the return value of each separate clause of a function, if there exists a spec, however... Thanks for the example and idea! Stavros On Thu, Jul 5, 2012 at 4:31 PM, Maxim Treskin wrote: > Hello! > > File attached. > Just compile it with > > erlc +debug_info my_module.erl > > then > > dialyzer . --plt ~/.r15b01_dialyzer.plt --no_native -Werror_handling > -Wrace_conditions -Wunderspecs > > and I see: > > Checking whether the PLT /home/zert/.r15b01_dialyzer.plt is > up-to-date... yes > Proceeding with analysis... done in 0m0.39s > done (passed successfully) > > > On 5 July 2012 20:22, Stavros Aronis wrote: > >> Hi! >> >> Your description is not very clear, so it might be better to send me some >> code snippet off-list, but let me put some facts here to make sure that the >> expectations from Dialyzer are 'reasonable'! >> >> - Dialyzer first uses the -callback attibutes of the behaviour and will >> always warn if the arguments or return value of a callback function are not >> subtypes of those attributes. If e.g. the return value of a callback is >> expected to be {noreply, term()} and the callback always returns {reply, >> term(), term()}, Dialyzer will warn about this. >> - As long as your implementation can possibly return an acceptable value, >> Dialyzer is happy. You might have a callback that either returns a correct >> value or something else that cannot be handled by the behaviour, but >> Dialyzer will assume that the correct term will always be returned. This >> happens because Dialyzer should never emit a false warning and this >> additional bad value might have been inferred because Dialyzer's analysis >> was not strong enough to exclude it. >> - Spec attributes are also taken into account when anayzing callbacks in >> the following manner: >> -- First the specs themselves are checked for compatibility with the >> -callback attributes of the behaviour. Here it is the user (and not >> Dialyzer) that writes these specs, so if these specs allow more values you >> get a warning. The rationale is that if you are providing a spec it should >> be at least as restrictive as the callback attribute. >> -- Second, the spec is used to check the arguments and return value of >> the function, as is the case with any spec. >> >> For gen_server, the -callback attributes can be seen here. >> There, the State and NewState can be any terms. If you also have a spec >> saying that this callback function should return {noreply, State :: >> #state{}} and you return {noreply, {bugotak}} then you are violating your >> own spec (but not the callback attribute of gen_server) and you should get >> a warning. Is this not the case when you run Dialyzer? >> >> Stavros >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > > > -- > Max Treskin > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hirotnkg@REDACTED Thu Jul 5 17:09:26 2012 From: hirotnkg@REDACTED (Yoshihiro Tanaka) Date: Thu, 5 Jul 2012 08:09:26 -0700 Subject: [erlang-questions] port/socket leak ? Message-ID: Hi, I'm using lhttpc and seeing some ports/sockets are leaking on R14B-04. Using inet:i(), it looks like below: 1437362 inet_tcp 0 0 <0.18131.596> *:* CLOSED Using erlang:port_info/1, it looks like below: [[{name,"tcp_inet"}, {links,[]}, {id,1437362}, {connected,<4283.18131.596>}, {input,0}, {output,0}]] My question is, when a port does not have linked process, or connected process is already gone, shouldn't it recycled by Erlang VM ? If not, why ? (These ports can be recycled using erlang:port_close/1) Thank you Yoshihiro From zerthurd@REDACTED Thu Jul 5 19:12:21 2012 From: zerthurd@REDACTED (Maxim Treskin) Date: Fri, 6 Jul 2012 00:12:21 +0700 Subject: [erlang-questions] Dialyzer cannot catch gen_server callback return type error In-Reply-To: References: <20120705111748.GB3724@localhost> <21efd459-0e1a-41d4-a043-e472c064c85d@googlegroups.com> Message-ID: Thank you, Stavros! On 5 July 2012 21:36, Stavros Aronis wrote: > Hi again, > > your example shows another case of the "might also be correct" rationale > of Dialyzer. Your full code is: > > -spec handle_cast(term(), #state{}) -> {noreply, NewState :: #state{}}. > > handle_cast({asd}, _State) -> > NewState = {bugotak}, > {noreply, NewState}; > handle_cast(_Msg, State) -> > {noreply, State}. > > It is true that the return value of the first clause violates the spec, > but this information is lost within Dialyzer as the "general" type > signature that Dialyzer will find and use > is my_module:handle_cast(term(),term()) -> {'noreply',term()} due to the > second clause. > Why signature `my_module:handle_cast(term(),term()) -> {'noreply',term()}` used? Is it taken from -callback signature, all right? > This more general type signature "absorbs" and hides the incorrect result. > The function "can work" if you only call with a message different than > {asd} and provide a correct #state{} as input, so Dialyzer remains silent. > If, on the other hand, you had just the first clause, or both clauses > returned a value that couldn't be #state{}, you would get a warning. > > Remember that Dialyzer does not promise to find all the bugs in your code. > It might be interesting and possible to check the return value of each > separate clause of a function, if there exists a spec, however... Thanks > for the example and idea! > > Stavros > > > On Thu, Jul 5, 2012 at 4:31 PM, Maxim Treskin wrote: > >> Hello! >> >> File attached. >> Just compile it with >> >> erlc +debug_info my_module.erl >> >> then >> >> dialyzer . --plt ~/.r15b01_dialyzer.plt --no_native -Werror_handling >> -Wrace_conditions -Wunderspecs >> >> and I see: >> >> Checking whether the PLT /home/zert/.r15b01_dialyzer.plt is >> up-to-date... yes >> Proceeding with analysis... done in 0m0.39s >> done (passed successfully) >> >> >> On 5 July 2012 20:22, Stavros Aronis wrote: >> >>> Hi! >>> >>> Your description is not very clear, so it might be better to send me >>> some code snippet off-list, but let me put some facts here to make sure >>> that the expectations from Dialyzer are 'reasonable'! >>> >>> - Dialyzer first uses the -callback attibutes of the behaviour and will >>> always warn if the arguments or return value of a callback function are not >>> subtypes of those attributes. If e.g. the return value of a callback is >>> expected to be {noreply, term()} and the callback always returns {reply, >>> term(), term()}, Dialyzer will warn about this. >>> - As long as your implementation can possibly return an acceptable >>> value, Dialyzer is happy. You might have a callback that either returns a >>> correct value or something else that cannot be handled by the behaviour, >>> but Dialyzer will assume that the correct term will always be returned. >>> This happens because Dialyzer should never emit a false warning and this >>> additional bad value might have been inferred because Dialyzer's analysis >>> was not strong enough to exclude it. >>> - Spec attributes are also taken into account when anayzing callbacks in >>> the following manner: >>> -- First the specs themselves are checked for compatibility with the >>> -callback attributes of the behaviour. Here it is the user (and not >>> Dialyzer) that writes these specs, so if these specs allow more values you >>> get a warning. The rationale is that if you are providing a spec it should >>> be at least as restrictive as the callback attribute. >>> -- Second, the spec is used to check the arguments and return value of >>> the function, as is the case with any spec. >>> >>> For gen_server, the -callback attributes can be seen here. >>> There, the State and NewState can be any terms. If you also have a spec >>> saying that this callback function should return {noreply, State :: >>> #state{}} and you return {noreply, {bugotak}} then you are violating your >>> own spec (but not the callback attribute of gen_server) and you should get >>> a warning. Is this not the case when you run Dialyzer? >>> >>> Stavros >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >> >> >> -- >> Max Treskin >> > > -- Max Treskin -------------- next part -------------- An HTML attachment was scrubbed... URL: From vychodil.hynek@REDACTED Thu Jul 5 21:23:14 2012 From: vychodil.hynek@REDACTED (Hynek Vychodil) Date: Thu, 5 Jul 2012 21:23:14 +0200 Subject: [erlang-questions] Keeping ErlNifBinary between NIF calls Message-ID: Hello, there is mentioned at several places in documentation that ErlNifBinary can be kept mutable between NIF calls but there is not more details how it should be done. I guess it can be stored in static variable, module private data or inside resource object as any other structure. Am I right? Hynek Vychodil -------------- next part -------------- An HTML attachment was scrubbed... URL: From andre.graf@REDACTED Fri Jul 6 00:27:56 2012 From: andre.graf@REDACTED (=?ISO-8859-1?Q?Andr=E9_Graf?=) Date: Fri, 6 Jul 2012 00:27:56 +0200 Subject: [erlang-questions] Problem attaching a shell to an Erlang node started through SSH Message-ID: Hello, I am working on a small library that uses Python Fabric together with Rebar deploying of Erlang applications. Fabric provides a scriptable (it's Python) abstraction over SSH. While I am able to deploy and upgrade new releases through such a fabric script I struggle with a simple Erlang node startup. A fabric call e.g. run("/path/to/my/release/bin/myproject start") invokes following SSH command ssh user@REDACTED "/bin/bash -l -c '/path/to/my/release/bin/myproject start'" This starts the node and I can use the remote shell to operate on the node. However, attaching a shell to the node triggers a strange behavior, the shell is available but unusable. It seems the startup through SSH has somehow confused the involved pipes. Do you have any hints on that? I am using rebar 2.0.0 R15B 20120625_185716 git 2.0.0-45-gc41fda6. Cheers, Andre From ivan@REDACTED Fri Jul 6 07:27:25 2012 From: ivan@REDACTED (Ivan Uemlianin) Date: Fri, 6 Jul 2012 06:27:25 +0100 Subject: [erlang-questions] Problem attaching a shell to an Erlang node started through SSH In-Reply-To: References: Message-ID: <43D0FCE5-625C-4903-8EF0-0133F44122C3@llaisdy.com> Dear Andr? This is what we're doing (see my recent email) and the erlang is working fine. I'll be at the office this afternoon to check properly but here are a few differences: - our fab script uses sudo() instead of run(); - this function calls an unit script instead of calling the release start directly. > However, attaching a shell to the > node triggers a strange > behavior, the shell is available but > unusable. What are the symptoms? Best wishes Ivan Sent from my iPhone On 5 Jul 2012, at 23:27, Andr? Graf wrote: > Hello, > > I am working on a small library that uses Python Fabric together with > Rebar deploying of Erlang applications. Fabric provides a scriptable > (it's Python) abstraction over SSH. > While I am able to deploy and upgrade new releases through such a > fabric script I struggle with a simple Erlang node startup. > > A fabric call e.g. run("/path/to/my/release/bin/myproject start") > invokes following SSH command > ssh user@REDACTED "/bin/bash -l -c '/path/to/my/release/bin/myproject start'" > > This starts the node and I can use the remote shell to operate on the > node. However, attaching a shell to the node triggers a strange > behavior, the shell is available but unusable. It seems the startup > through SSH has somehow confused the involved pipes. > > Do you have any hints on that? I am using rebar 2.0.0 R15B > 20120625_185716 git 2.0.0-45-gc41fda6. > > Cheers, > Andre > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From andre.graf@REDACTED Fri Jul 6 10:17:47 2012 From: andre.graf@REDACTED (=?ISO-8859-1?Q?Andr=E9_Graf?=) Date: Fri, 6 Jul 2012 10:17:47 +0200 Subject: [erlang-questions] Problem attaching a shell to an Erlang node started through SSH In-Reply-To: <43D0FCE5-625C-4903-8EF0-0133F44122C3@llaisdy.com> References: <43D0FCE5-625C-4903-8EF0-0133F44122C3@llaisdy.com> Message-ID: Hi Ivan, Running EUnit works. I am pretty sure the problem is somehow related with the runner script generated by Rebar (which I haven't changed). I tried both sudo and run. I forgot to mention that you must disable fabric pty when starting the node through the runner script otherwise the start call succeeds but no node (and also no logs) are generated. Symptoms are difficult to describe. The resulting shell looks like that: user@REDACTED:~/test_project/rel/mynode/bin$ sudo ./mynode attach Attaching to /tmp//home/user/test_project/rel/mynode/erlang.pipe.3 (^D to exit) ^R So I won't get the normal Erlang shell prompt, I can run some Erlang commands though, but I don't have any tab-completion. Ctrl-D will properly exit, Ctrl-G is not giving me the User switch command, Ctrl-C gives the proper Break menu. Funny here, If I don't use sudo for the attach I get a "Node is not running!" although the node was started without using sudo, but this problem also appears if I start the node directly on the server without going through SSH. Cheers, Andr? On 6 July 2012 07:27, Ivan Uemlianin wrote: > Dear Andr? > > This is what we're doing (see my recent email) and the erlang is working > fine. I'll be at the office this afternoon to check properly but here are a > few differences: > > - our fab script uses sudo() instead of run(); > - this function calls an unit script instead of calling the release start > directly. > >> However, attaching a shell to the >> node triggers a strange >> behavior, the shell is available but >> unusable. > > What are the symptoms? > > Best wishes > > Ivan > > > Sent from my iPhone > > On 5 Jul 2012, at 23:27, Andr? Graf wrote: > > Hello, > > I am working on a small library that uses Python Fabric together with > Rebar deploying of Erlang applications. Fabric provides a scriptable > (it's Python) abstraction over SSH. > While I am able to deploy and upgrade new releases through such a > fabric script I struggle with a simple Erlang node startup. > > A fabric call e.g. run("/path/to/my/release/bin/myproject start") > invokes following SSH command > ssh user@REDACTED "/bin/bash -l -c '/path/to/my/release/bin/myproject start'" > > This starts the node and I can use the remote shell to operate on the > node. However, attaching a shell to the node triggers a strange > behavior, the shell is available but unusable. It seems the startup > through SSH has somehow confused the involved pipes. > > Do you have any hints on that? I am using rebar 2.0.0 R15B > 20120625_185716 git 2.0.0-45-gc41fda6. > > Cheers, > Andre > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From zerthurd@REDACTED Fri Jul 6 10:39:58 2012 From: zerthurd@REDACTED (Maxim Treskin) Date: Fri, 6 Jul 2012 15:39:58 +0700 Subject: [erlang-questions] Dialyzer cannot catch gen_server callback return type error In-Reply-To: References: <20120705111748.GB3724@localhost> <21efd459-0e1a-41d4-a043-e472c064c85d@googlegroups.com> Message-ID: I just really not understand. Function handle_cast returns, and returns incorrect type according to -spec signature, right? Why Dialyzer lose this information? It is sort of missbehaviour i think. On 6 July 2012 00:12, Maxim Treskin wrote: > Thank you, Stavros! > > On 5 July 2012 21:36, Stavros Aronis wrote: > >> Hi again, >> >> your example shows another case of the "might also be correct" rationale >> of Dialyzer. Your full code is: >> >> -spec handle_cast(term(), #state{}) -> {noreply, NewState :: #state{}}. >> >> handle_cast({asd}, _State) -> >> NewState = {bugotak}, >> {noreply, NewState}; >> handle_cast(_Msg, State) -> >> {noreply, State}. >> >> It is true that the return value of the first clause violates the spec, >> but this information is lost within Dialyzer as the "general" type >> signature that Dialyzer will find and use >> is my_module:handle_cast(term(),term()) -> {'noreply',term()} due to the >> second clause. >> > > Why signature `my_module:handle_cast(term(),term()) -> > {'noreply',term()}` used? Is it taken from -callback signature, all right? > > >> This more general type signature "absorbs" and hides the incorrect >> result. The function "can work" if you only call with a message different >> than {asd} and provide a correct #state{} as input, so Dialyzer remains >> silent. If, on the other hand, you had just the first clause, or both >> clauses returned a value that couldn't be #state{}, you would get a warning. >> >> Remember that Dialyzer does not promise to find all the bugs in your >> code. It might be interesting and possible to check the return value of >> each separate clause of a function, if there exists a spec, however... >> Thanks for the example and idea! >> >> Stavros >> >> >> On Thu, Jul 5, 2012 at 4:31 PM, Maxim Treskin wrote: >> >>> Hello! >>> >>> File attached. >>> Just compile it with >>> >>> erlc +debug_info my_module.erl >>> >>> then >>> >>> dialyzer . --plt ~/.r15b01_dialyzer.plt --no_native -Werror_handling >>> -Wrace_conditions -Wunderspecs >>> >>> and I see: >>> >>> Checking whether the PLT /home/zert/.r15b01_dialyzer.plt is >>> up-to-date... yes >>> Proceeding with analysis... done in 0m0.39s >>> done (passed successfully) >>> >>> >>> On 5 July 2012 20:22, Stavros Aronis wrote: >>> >>>> Hi! >>>> >>>> Your description is not very clear, so it might be better to send me >>>> some code snippet off-list, but let me put some facts here to make sure >>>> that the expectations from Dialyzer are 'reasonable'! >>>> >>>> - Dialyzer first uses the -callback attibutes of the behaviour and will >>>> always warn if the arguments or return value of a callback function are not >>>> subtypes of those attributes. If e.g. the return value of a callback is >>>> expected to be {noreply, term()} and the callback always returns {reply, >>>> term(), term()}, Dialyzer will warn about this. >>>> - As long as your implementation can possibly return an acceptable >>>> value, Dialyzer is happy. You might have a callback that either returns a >>>> correct value or something else that cannot be handled by the behaviour, >>>> but Dialyzer will assume that the correct term will always be returned. >>>> This happens because Dialyzer should never emit a false warning and this >>>> additional bad value might have been inferred because Dialyzer's analysis >>>> was not strong enough to exclude it. >>>> - Spec attributes are also taken into account when anayzing callbacks >>>> in the following manner: >>>> -- First the specs themselves are checked for compatibility with the >>>> -callback attributes of the behaviour. Here it is the user (and not >>>> Dialyzer) that writes these specs, so if these specs allow more values you >>>> get a warning. The rationale is that if you are providing a spec it should >>>> be at least as restrictive as the callback attribute. >>>> -- Second, the spec is used to check the arguments and return value of >>>> the function, as is the case with any spec. >>>> >>>> For gen_server, the -callback attributes can be seen here. >>>> There, the State and NewState can be any terms. If you also have a spec >>>> saying that this callback function should return {noreply, State :: >>>> #state{}} and you return {noreply, {bugotak}} then you are violating your >>>> own spec (but not the callback attribute of gen_server) and you should get >>>> a warning. Is this not the case when you run Dialyzer? >>>> >>>> Stavros >>>> >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> >>>> >>> >>> >>> -- >>> Max Treskin >>> >> >> > > > -- > Max Treskin > -- Max Treskin -------------- next part -------------- An HTML attachment was scrubbed... URL: From rvg.mailing@REDACTED Fri Jul 6 10:49:29 2012 From: rvg.mailing@REDACTED (Rudolph van Graan) Date: Fri, 06 Jul 2012 09:49:29 +0100 Subject: [erlang-questions] Removing mnesia cluster node while mnesia is offline In-Reply-To: <87d34d59h9.wl%f@mazzo.li> References: <878vf6chx0.wl%f@mazzo.li> <87fw995bzc.wl%f@mazzo.li> <698B603A-C83F-4726-B4EA-FE215D997987@me.com> <87d34d59h9.wl%f@mazzo.li> Message-ID: Hi Francesco, > OK, let's say that I have node A, B, and C, in a cluster. A goes down, > and C gets removed from the cluster. So B now has [A, B] in the schema > as cluster nodes. A's data still has [A, B, C] as cluster nodes, and > if we start A again, that will be propagated to B. That's what I meant > with "pollute". This scenario is impossible. I will rewrite it as follows, depending on your definition of "remove". One correct interpretation would be: 1. A,B & C are in the cluster 2. A goes down 3. C is turned off and the schema is destroyed on C (instead of your term "removed") 4. So B has [A,B,C] in the schema (and so does A) 5. A is started again. 6. A copies the data from B because it knows it was down and B has the latest data Another correct sequence would be: 1. A,B & C are in the cluster 2. A goes down 3. Someone tries to remove C from the cluster via API call. Mnesia will refuse to do this, because A is down. > A's data still has [A, B, C] as cluster nodes, and > if we start A again, that will be propagated to B. This is the bit that happen unless you had partitioning/netsplit. A knows that B was online when it went down. B knows that A went down. When A starts up again, they communicate and figure this out. Then A replaces all its copies of non-local/shared data from the current data at B. Rudolph van Graan Please have a look at my blogs: - Random Views of London -- A photographer's interpretation of London http://randomlondonview.blogspot.co.uk/ On 3 Jul 2012, at 13:12, Francesco Mazzoli wrote: > Hi Rudolph, > At Tue, 03 Jul 2012 12:31:11 +0100, > Rudolph van Graan wrote: >> You have to set things up so that you have other replicas of those >> tables on other nodes. > > Well, some data is inherently "local" and not relevant to the other > nodes, that have similar local tables. > >> This is not how mnesia works. The list of nodes in the distributed >> database is static, i.e. all the nodes have the same list. This is >> why all of them needs to be online when you add or remove nodes. > > I understand this, but I still want to know if it's possible to > recover from that situation more or less gracefully. > >> The only problem that you need to solve is dealing with partitioning >> (i.e. splits) and you have two sets of data on two different >> segments. It is not possible for a node to "pollute" other nodes. > > OK, let's say that I have node A, B, and C, in a cluster. A goes down, > and C gets removed from the cluster. So B now has [A, B] in the schema > as cluster nodes. A's data still has [A, B, C] as cluster nodes, and > if we start A again, that will be propagated to B. That's what I meant > with "pollute". > >> So "... offline node still thinks it is clustered..." - it doesn't >> just think so, it has been configured so when you added it into the >> cluster. It will stay part of the cluster until you remove it. > > ...but to remove it I have to bring it online, and if the other online > nodes have a different configuration, things get ugly. > > -- > Francesco * Often in error, never in doubt -------------- next part -------------- An HTML attachment was scrubbed... URL: From aronisstav@REDACTED Fri Jul 6 12:35:28 2012 From: aronisstav@REDACTED (Stavros Aronis) Date: Fri, 6 Jul 2012 13:35:28 +0300 Subject: [erlang-questions] Dialyzer cannot catch gen_server callback return type error In-Reply-To: References: <20120705111748.GB3724@localhost> <21efd459-0e1a-41d4-a043-e472c064c85d@googlegroups.com> Message-ID: Hi again, The signature (or "success typing" in Dialyzer's terminology) is `my_module:handle_cast(term(),term()) -> {'noreply',term()}` because the second clause: handle_cast(_Msg, State) -> {noreply, State}. will accept any terms as _Msg and State and return {noreply, State} in which State again is any term. In Dialyzer's type system, this {noreply, term()} return value will 'absorb' the definitely incorrect {noreply, {bugotak}} as {bugotak} is also a term(). From that point on, Dialyzer can assume that term() might also be #state{} and emit no warnings. This happens because Dialyzer checks the whole range of return values at once and not each particular clause. If all the return values were definitely different than #state{}, then and only then would Dialyzer warn you. Again, trying to check each particular return value against a given spec might be an interesting extension and we will consider it in the future. Stavros On Fri, Jul 6, 2012 at 11:39 AM, Maxim Treskin wrote: > I just really not understand. Function handle_cast returns, and returns > incorrect type according to -spec signature, right? > Why Dialyzer lose this information? It is sort of missbehaviour i think. > > -- > Max Treskin > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sam@REDACTED Fri Jul 6 13:10:26 2012 From: sam@REDACTED (Samuel Elliott) Date: Fri, 6 Jul 2012 12:10:26 +0100 Subject: [erlang-questions] et_viewer instability Message-ID: Hi I'm having trouble with et_viewer at the moment. - If i resize the window at all, it locks up (i get a spinning Beachball, and the option to Force Quit it instead of just plain Quit) - If i try to scroll, sometimes it also locks up. Any ideas what could be causing this, and how I could fix it? I'd love to actually be able to use the viewer. My erlang install details - homebrew (mac os x 10.7.4) - Erlang R15B01 (erts-5.9.1) [source] [64-bit] [smp:2:2] [async-threads:0] [hipe] [kernel-poll:false] - wxWidgets version: No idea, so probably the default one. Any Idea how I find out? To start the viewer, I'm using: et_viewer:start([ {detail_level, max}, {trace_pattern, {et, max}}, {trace_global, true}, {scale, 2}, {max_actors, infinity}, {width, 1000}, {height, 800} ]). Thanks! Sam -- Samuel Elliott sam@REDACTED http://lenary.co.uk/ +44 (0)7891 993 664 -------------- next part -------------- An HTML attachment was scrubbed... URL: From cgsmcmlxxv@REDACTED Fri Jul 6 13:21:26 2012 From: cgsmcmlxxv@REDACTED (CGS) Date: Fri, 6 Jul 2012 13:21:26 +0200 Subject: [erlang-questions] simple question about list memory consumption Message-ID: Hi everyone, I am puzzled by a behavior I didn't expect, so, if anyone can help to understand it, I would appreciate. Let's consider a string of latin1 characters. When dumped to a file on a harddisk it occupies, as expected, 1 B per character. When loaded in a variable in Erlang shell, I get about 21 B per character (at least free is reporting that). Of course, it is hard to test on one character due to changing RAM consumption of other processes. So, my test case consisted in (I voided writing down and reading from a file in the test case): 1. stabilizing the environment (no other processes were started, but the OS ones and watching for the RAM consumption stabilization at the level of MB - give or take 1MB); 2. opening an Erlang shell (few MB consumption - stable); 3. executing: L = lists:map(fun(_) -> 107 end,lists:seq(1,10000000)), ok. where (I know it's obvious, but just for completeness): - 107 is representing "k" character; - the 'ok' atom at the end is for not writing the content of L in the shell; - the list [107,107,...] is equivalent to "kk..." string (which written in a plain text file it gives 1 B per character as it is expected). The occupied RAM jumped by 214-217 MB for 10^7 characters (at 10^6 characters, I got 21 MB, so, it is related to the string length only), which gives about 21 B per character (one can compute within the errors the exact the size, but I took that value as the minimum possible). Erlang shell reports "Erlang R15B01 (erts-5.9.1) [source] [smp:4:4] [async-threads:0] [hipe] [kernel-poll:false] [dtrace]". I know that this isn't a big problem, but I am missing something here for sure. Can anyone point what I am missing? Thank you in advance. Regards, CGS -------------- next part -------------- An HTML attachment was scrubbed... URL: From zerthurd@REDACTED Fri Jul 6 13:36:55 2012 From: zerthurd@REDACTED (Maxim Treskin) Date: Fri, 6 Jul 2012 18:36:55 +0700 Subject: [erlang-questions] Dialyzer cannot catch gen_server callback return type error In-Reply-To: References: <20120705111748.GB3724@localhost> <21efd459-0e1a-41d4-a043-e472c064c85d@googlegroups.com> Message-ID: On 6 July 2012 17:35, Stavros Aronis wrote: > Hi again, > > The signature (or "success typing" in Dialyzer's terminology) is > `my_module:handle_cast(term(),term()) -> {'noreply',term()}` because the > second clause: > > handle_cast(_Msg, State) -> > {noreply, State}. > > will accept any terms as _Msg and State and return {noreply, State} in > which State again is any term. In Dialyzer's type system, this {noreply, > term()} return value will 'absorb' the definitely incorrect {noreply, > {bugotak}} as {bugotak} is also a term(). From that point on, Dialyzer can > assume that term() might also be #state{} and emit no warnings. This > happens because Dialyzer checks the whole range of return values at once > and not each particular clause. If all the return values were definitely > different than #state{}, then and only then would Dialyzer warn you. > > Thanks for the clarification, Stavros! Now I understand how Dialyzer checks types of function clauses. > Again, trying to check each particular return value against a given spec > might be an interesting extension and we will consider it in the future. > > I hope so. It would be a significant level up! > Stavros > > On Fri, Jul 6, 2012 at 11:39 AM, Maxim Treskin wrote: > >> I just really not understand. Function handle_cast returns, and returns >> incorrect type according to -spec signature, right? >> Why Dialyzer lose this information? It is sort of missbehaviour i think. >> >> -- >> Max Treskin >> > > -- Max Treskin -------------- next part -------------- An HTML attachment was scrubbed... URL: From magnus.henoch@REDACTED Fri Jul 6 14:29:12 2012 From: magnus.henoch@REDACTED (Magnus Henoch) Date: Fri, 06 Jul 2012 13:29:12 +0100 (BST) Subject: [erlang-questions] simple question about list memory consumption In-Reply-To: Message-ID: An Erlang list uses at least 2 words per element - one word for the element itself, and one word to point to the next list element. (Of course, if the list element doesn't fit in a word, it would need space for that too, but ASCII characters do fit.) So on a 32-bit machine, you'd expect an ASCII string to use 8 (2 * 4) bytes per character, and on a 64-bit machine it should be 16 (2 * 8) bytes per character (unless you're using the halfword emulator). I'm not sure why you see 21 bytes per character, though... If you need more memory-efficient storage, consider using binaries instead. They use one byte per, well, byte; however, some completely different considerations apply: http://www.erlang.org/doc/efficiency_guide/binaryhandling.html Hope this helps, Magnus ----- Original Message ----- > > Hi everyone, > > > I am puzzled by a behavior I didn't expect, so, if anyone can help to > understand it, I would appreciate. > > > Let's consider a string of latin1 characters. When dumped to a file > on a harddisk it occupies, as expected, 1 B per character. When > loaded in a variable in Erlang shell, I get about 21 B per character > (at least free is reporting that). Of course, it is hard to test on > one character due to changing RAM consumption of other processes. > So, my test case consisted in (I voided writing down and reading > from a file in the test case): > 1. stabilizing the environment (no other processes were started, but > the OS ones and watching for the RAM consumption stabilization at > the level of MB - give or take 1MB); > 2. opening an Erlang shell (few MB consumption - stable); > 3. executing: > > > L = lists:map(fun(_) -> 107 end,lists:seq(1,10000000)), ok. > > > where (I know it's obvious, but just for completeness): > - 107 is representing "k" character; > - the 'ok' atom at the end is for not writing the content of L in the > shell; > - the list [107,107,...] is equivalent to "kk..." string (which > written in a plain text file it gives 1 B per character as it is > expected). > > > The occupied RAM jumped by 214-217 MB for 10^7 characters (at 10^6 > characters, I got 21 MB, so, it is related to the string length > only), which gives about 21 B per character (one can compute within > the errors the exact the size, but I took that value as the minimum > possible). > > > Erlang shell reports "Erlang R15B01 (erts-5.9.1) [source] [smp:4:4] > [async-threads:0] [hipe] [kernel-poll:false] [dtrace]". > > > I know that this isn't a big problem, but I am missing something here > for sure. Can anyone point what I am missing? Thank you in advance. > > > Regards, > CGS > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Magnus Henoch Erlang Solutions Ltd http://www.erlang-solutions.com/ From bob@REDACTED Fri Jul 6 14:32:49 2012 From: bob@REDACTED (Bob Ippolito) Date: Fri, 6 Jul 2012 05:32:49 -0700 Subject: [erlang-questions] simple question about list memory consumption In-Reply-To: References: Message-ID: http://www.erlang.org/doc/efficiency_guide/advanced.html The data structure you want for space efficiency is binary. On Friday, July 6, 2012, CGS wrote: > Hi everyone, > > I am puzzled by a behavior I didn't expect, so, if anyone can help to > understand it, I would appreciate. > > Let's consider a string of latin1 characters. When dumped to a file on a > harddisk it occupies, as expected, 1 B per character. When loaded in a > variable in Erlang shell, I get about 21 B per character (at least free is > reporting that). Of course, it is hard to test on one character due to > changing RAM consumption of other processes. So, my test case consisted in > (I voided writing down and reading from a file in the test case): > 1. stabilizing the environment (no other processes were started, but the > OS ones and watching for the RAM consumption stabilization at the level of > MB - give or take 1MB); > 2. opening an Erlang shell (few MB consumption - stable); > 3. executing: > > L = lists:map(fun(_) -> 107 end,lists:seq(1,10000000)), ok. > > where (I know it's obvious, but just for completeness): > - 107 is representing "k" character; > - the 'ok' atom at the end is for not writing the content of L in the > shell; > - the list [107,107,...] is equivalent to "kk..." string (which written in > a plain text file it gives 1 B per character as it is expected). > > The occupied RAM jumped by 214-217 MB for 10^7 characters (at 10^6 > characters, I got 21 MB, so, it is related to the string length only), > which gives about 21 B per character (one can compute within the errors the > exact the size, but I took that value as the minimum possible). > > Erlang shell reports "Erlang R15B01 (erts-5.9.1) [source] [smp:4:4] > [async-threads:0] [hipe] [kernel-poll:false] [dtrace]". > > I know that this isn't a big problem, but I am missing something here for > sure. Can anyone point what I am missing? Thank you in advance. > > Regards, > CGS > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From magnus.henoch@REDACTED Fri Jul 6 14:38:36 2012 From: magnus.henoch@REDACTED (Magnus Henoch) Date: Fri, 06 Jul 2012 13:38:36 +0100 (BST) Subject: [erlang-questions] simple question about list memory consumption In-Reply-To: Message-ID: <48564154-48ae-4d26-9a06-aa1091360cc0@knuth> Hm, ignore my previous answer, then; I was under the impression that small enough integers would be stored as immediates in the list cell... ----- Original Message ----- > > http://www.erlang.org/doc/efficiency_guide/advanced.html > > > The data structure you want for space efficiency is binary. > > On Friday, July 6, 2012, CGS wrote: > > > Hi everyone, > > > I am puzzled by a behavior I didn't expect, so, if anyone can help to > understand it, I would appreciate. > > > Let's consider a string of latin1 characters. When dumped to a file > on a harddisk it occupies, as expected, 1 B per character. When > loaded in a variable in Erlang shell, I get about 21 B per character > (at least free is reporting that). Of course, it is hard to test on > one character due to changing RAM consumption of other processes. > So, my test case consisted in (I voided writing down and reading > from a file in the test case): > 1. stabilizing the environment (no other processes were started, but > the OS ones and watching for the RAM consumption stabilization at > the level of MB - give or take 1MB); > 2. opening an Erlang shell (few MB consumption - stable); > 3. executing: > > > L = lists:map(fun(_) -> 107 end,lists:seq(1,10000000)), ok. > > > where (I know it's obvious, but just for completeness): > - 107 is representing "k" character; > - the 'ok' atom at the end is for not writing the content of L in the > shell; > - the list [107,107,...] is equivalent to "kk..." string (which > written in a plain text file it gives 1 B per character as it is > expected). > > > The occupied RAM jumped by 214-217 MB for 10^7 characters (at 10^6 > characters, I got 21 MB, so, it is related to the string length > only), which gives about 21 B per character (one can compute within > the errors the exact the size, but I took that value as the minimum > possible). > > > Erlang shell reports "Erlang R15B01 (erts-5.9.1) [source] [smp:4:4] > [async-threads:0] [hipe] [kernel-poll:false] [dtrace]". > > > I know that this isn't a big problem, but I am missing something here > for sure. Can anyone point what I am missing? Thank you in advance. > > > Regards, > CGS > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Magnus Henoch Erlang Solutions Ltd http://www.erlang-solutions.com/ From rvg.mailing@REDACTED Fri Jul 6 14:46:09 2012 From: rvg.mailing@REDACTED (Rudolph van Graan) Date: Fri, 06 Jul 2012 13:46:09 +0100 Subject: [erlang-questions] How would you do it? References: Message-ID: <924C63BF-9335-4E10-B34F-3D6D1B16B9FF@me.com> > I have to rank about 50 000 - 100 000 different players. > On each score message I have to re-sort whole ranking table. > It must be very cheap to get: > top 100 players > player's rating +/- 10 players about current player > I expect about 20-50 score messages per seconds > Size of score message is about 4KB (profile takes most of the space). Trying to be pragmatic, I would do this: 1. Every player has a unique key (say ID) and this is the key to a Mnesia table that stores the player records. 2. I would have a second table with the rankings, keyed on score. 3. Each slot contains the ID and score of the player at that position: - {score,30,{player,200},undefined,20} - {score,20,{player,5},30,19} - {score,19,{player,18},20,undefined} ... So the top player is the one with rank 30, i.e. player 200, the 2nd one is player 5 and the third is player 18. You can get to the first one by reading the first entry or last entry in the table. Each score record contains a "pointer" to the one before and the one after. The record will look like this: {score, Score, PlayerID, Next, Previous} 4. The player table is keyed on the player id and contains the player's current score. So in the example above Player 200 will be rank 1 (score 30) and Player 5 will be rank 2 (score 20) etc. So you can easily get the top N, simply read the records starting at the end (using dirty last) and following the next pointer. That will give you two database reads per entry. If you want to cache this, make a simple gen_server that keeps this list in memory and refreshes every 10 seconds. 4. Process your score messages concurrently, i.e. each operation does the folling a) Read the player record according to ID and get the score from the player ID. b) Read the score record using the current score and retrieve the values of "previous" and "next" c) Delete the score record and modify "previous" to point to "next" and "next" to point to previous d) Insert a new #score{score=NewScore,previous=undefined, next=undefined} e) Call mnesia:next(Tab,{score,NewScore}) to get the logical Next and mnesia:prev/2 to get the logical previous f) Update the three records with the appropriate next/prev values (all of this protected by a transaction of course) This approach is heavy compared to the other approaches suggested, but it gives you distribution for free and no single process bottlenecks. It should be very fast if you use only memory tables and fast enough if you add persistent tables. Obviously I haven't worked out all the details above - I am sure you can figure that out. Hope it helps. Rudolph van Graan Please have a look at my blogs: - Random Views of London -- A photographer's interpretation of London http://randomlondonview.blogspot.co.uk/ On 2 Jul 2012, at 11:29, Max Bourinov wrote: > Hi Erlangers, > > I need to build a ranking system for online game. > > I think I will have a ranking gen_server that will accept cast messages like this: {score, Player :: profile(), Score :: integer()}. > > So, the question is what would be most appropriate data structure if: > I have to rank about 50 000 - 100 000 different players. > On each score message I have to re-sort whole ranking table. > It must be very cheap to get: > top 100 players > player's rating +/- 10 players about current player > I expect about 20-50 score messages per seconds > Size of score message is about 4KB (profile takes most of the space). > > Any ideas or suggestions are welcome! > > Best regards, > Max > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From sverker.eriksson@REDACTED Fri Jul 6 14:50:03 2012 From: sverker.eriksson@REDACTED (Sverker Eriksson) Date: Fri, 6 Jul 2012 14:50:03 +0200 Subject: [erlang-questions] Keeping ErlNifBinary between NIF calls In-Reply-To: References: Message-ID: <4FF6DEFB.40108@erix.ericsson.se> Hynek Vychodil wrote: > Hello, > > there is mentioned at several places in documentation that ErlNifBinary can be > kept mutable between NIF calls but there is not more details how it should be > done. I guess it can be stored in static variable, module private data or inside > resource object as any other structure. Am I right? > Yes. /Sverker, Erlang/OTP From cgsmcmlxxv@REDACTED Fri Jul 6 14:54:24 2012 From: cgsmcmlxxv@REDACTED (CGS) Date: Fri, 6 Jul 2012 14:54:24 +0200 Subject: [erlang-questions] simple question about list memory consumption In-Reply-To: <48564154-48ae-4d26-9a06-aa1091360cc0@knuth> References: <48564154-48ae-4d26-9a06-aa1091360cc0@knuth> Message-ID: Thank you all for your time in answering. It seems my memory entered holiday already. :) Regards, CGS On Fri, Jul 6, 2012 at 2:38 PM, Magnus Henoch < magnus.henoch@REDACTED> wrote: > Hm, ignore my previous answer, then; I was under the impression that > small enough integers would be stored as immediates in the > list cell... > > ----- Original Message ----- > > > > http://www.erlang.org/doc/efficiency_guide/advanced.html > > > > > > The data structure you want for space efficiency is binary. > > > > On Friday, July 6, 2012, CGS wrote: > > > > > > Hi everyone, > > > > > > I am puzzled by a behavior I didn't expect, so, if anyone can help to > > understand it, I would appreciate. > > > > > > Let's consider a string of latin1 characters. When dumped to a file > > on a harddisk it occupies, as expected, 1 B per character. When > > loaded in a variable in Erlang shell, I get about 21 B per character > > (at least free is reporting that). Of course, it is hard to test on > > one character due to changing RAM consumption of other processes. > > So, my test case consisted in (I voided writing down and reading > > from a file in the test case): > > 1. stabilizing the environment (no other processes were started, but > > the OS ones and watching for the RAM consumption stabilization at > > the level of MB - give or take 1MB); > > 2. opening an Erlang shell (few MB consumption - stable); > > 3. executing: > > > > > > L = lists:map(fun(_) -> 107 end,lists:seq(1,10000000)), ok. > > > > > > where (I know it's obvious, but just for completeness): > > - 107 is representing "k" character; > > - the 'ok' atom at the end is for not writing the content of L in the > > shell; > > - the list [107,107,...] is equivalent to "kk..." string (which > > written in a plain text file it gives 1 B per character as it is > > expected). > > > > > > The occupied RAM jumped by 214-217 MB for 10^7 characters (at 10^6 > > characters, I got 21 MB, so, it is related to the string length > > only), which gives about 21 B per character (one can compute within > > the errors the exact the size, but I took that value as the minimum > > possible). > > > > > > Erlang shell reports "Erlang R15B01 (erts-5.9.1) [source] [smp:4:4] > > [async-threads:0] [hipe] [kernel-poll:false] [dtrace]". > > > > > > I know that this isn't a big problem, but I am missing something here > > for sure. Can anyone point what I am missing? Thank you in advance. > > > > > > Regards, > > CGS > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > > -- > Magnus Henoch > Erlang Solutions Ltd > http://www.erlang-solutions.com/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sverker.eriksson@REDACTED Fri Jul 6 15:08:30 2012 From: sverker.eriksson@REDACTED (Sverker Eriksson) Date: Fri, 6 Jul 2012 15:08:30 +0200 Subject: [erlang-questions] simple question about list memory consumption In-Reply-To: <48564154-48ae-4d26-9a06-aa1091360cc0@knuth> References: <48564154-48ae-4d26-9a06-aa1091360cc0@knuth> Message-ID: <4FF6E34E.1000102@erix.ericsson.se> Magnus Henoch wrote: > Hm, ignore my previous answer, then; I was under the impression that > small enough integers would be stored as immediates in the > list cell... > > Small enough integers (28/60 bit signed integers) *are* stored as immediates in the list cells. /Sverker, Erlang/OTP From magnus.henoch@REDACTED Fri Jul 6 15:15:09 2012 From: magnus.henoch@REDACTED (Magnus Henoch) Date: Fri, 06 Jul 2012 14:15:09 +0100 (BST) Subject: [erlang-questions] simple question about list memory consumption In-Reply-To: <4FF6E34E.1000102@erix.ericsson.se> Message-ID: > Small enough integers (28/60 bit signed integers) *are* stored as > immediates in the list cells. > > /Sverker, Erlang/OTP Oh, now I see - I was misreading the page Bob linked. It says "1 word + 2 words per character" which I read as "(1 word + 2 words) per character"... Makes more sense now :) -- Magnus Henoch Erlang Solutions Ltd http://www.erlang-solutions.com/ From f@REDACTED Fri Jul 6 15:27:30 2012 From: f@REDACTED (Francesco Mazzoli) Date: Fri, 06 Jul 2012 14:27:30 +0100 Subject: [erlang-questions] simple question about list memory consumption In-Reply-To: <4FF6E34E.1000102@erix.ericsson.se> References: <48564154-48ae-4d26-9a06-aa1091360cc0@knuth> <4FF6E34E.1000102@erix.ericsson.se> Message-ID: <87sjd5ow8t.wl%f@mazzo.li> At Fri, 6 Jul 2012 15:08:30 +0200, Sverker Eriksson wrote: > Small enough integers (28/60 bit signed integers) *are* stored as immediates > in the list cells. How can this be, without some kind of tagging, since Erlang list are heterogeneous - which would increase memory overhead anyway? In other words, if I have [foo, 1, bar], if the 1 is unboxed, how does Erlang know it is an integer? -- Francesco * Often in error, never in doubt From sverker.eriksson@REDACTED Fri Jul 6 16:00:22 2012 From: sverker.eriksson@REDACTED (Sverker Eriksson) Date: Fri, 6 Jul 2012 16:00:22 +0200 Subject: [erlang-questions] simple question about list memory consumption In-Reply-To: <87sjd5ow8t.wl%f@mazzo.li> References: <48564154-48ae-4d26-9a06-aa1091360cc0@knuth> <4FF6E34E.1000102@erix.ericsson.se> <87sjd5ow8t.wl%f@mazzo.li> Message-ID: <4FF6EF76.1050008@erix.ericsson.se> Francesco Mazzoli wrote: > At Fri, 6 Jul 2012 15:08:30 +0200, > Sverker Eriksson wrote: > >> Small enough integers (28/60 bit signed integers) *are* stored as immediates >> in the list cells. >> > > How can this be, without some kind of tagging, since Erlang list are > heterogeneous - which would increase memory overhead anyway? > > In other words, if I have [foo, 1, bar], if the 1 is unboxed, how does Erlang > know it is an integer? > It is done with tagging. The two least significant bits are used for tagging as they are always 0 for a pointer to word aligned data. /Sverker From carlsson.richard@REDACTED Fri Jul 6 16:02:07 2012 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Fri, 06 Jul 2012 16:02:07 +0200 Subject: [erlang-questions] simple question about list memory consumption In-Reply-To: <87sjd5ow8t.wl%f@mazzo.li> References: <48564154-48ae-4d26-9a06-aa1091360cc0@knuth> <4FF6E34E.1000102@erix.ericsson.se> <87sjd5ow8t.wl%f@mazzo.li> Message-ID: <4FF6EFDF.4030905@gmail.com> On 07/06/2012 03:27 PM, Francesco Mazzoli wrote: > At Fri, 6 Jul 2012 15:08:30 +0200, > Sverker Eriksson wrote: >> Small enough integers (28/60 bit signed integers) *are* stored as immediates >> in the list cells. > > How can this be, without some kind of tagging, since Erlang list are > heterogeneous - which would increase memory overhead anyway? Because all data is tagged. > In other words, if I have [foo, 1, bar], if the 1 is unboxed, how does Erlang > know it is an integer? Simple - there's a difference between "tagged" and "boxed". Boxed means the first word is a pointer (also tagged) to a location on the heap where the real data is stored. An "immediate" like an integer is just that first word, with a couple of bits used as tags. /Richard From f@REDACTED Fri Jul 6 16:09:12 2012 From: f@REDACTED (Francesco Mazzoli) Date: Fri, 06 Jul 2012 15:09:12 +0100 Subject: [erlang-questions] simple question about list memory consumption In-Reply-To: <4FF6EFDF.4030905@gmail.com> References: <48564154-48ae-4d26-9a06-aa1091360cc0@knuth> <4FF6E34E.1000102@erix.ericsson.se> <87sjd5ow8t.wl%f@mazzo.li> <4FF6EFDF.4030905@gmail.com> Message-ID: <87pq89oubb.wl%f@mazzo.li> At Fri, 06 Jul 2012 16:02:07 +0200, Richard Carlsson wrote: > Because all data is tagged. OK. > Simple - there's a difference between "tagged" and "boxed". Boxed means > the first word is a pointer (also tagged) to a location on the heap > where the real data is stored. An "immediate" like an integer is just > that first word, with a couple of bits used as tags. I know that, the thing I was pointing out is that we need some additional space to accommodate the tag as well, while people in this thread were accounting for pointers only. What I did not realise is that > The two least significant bits are used for tagging as they are always 0 for a > pointer to word aligned data. Which makes it possible to include the tag in the word. -- Francesco * Often in error, never in doubt From carlsson.richard@REDACTED Fri Jul 6 16:19:12 2012 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Fri, 06 Jul 2012 16:19:12 +0200 Subject: [erlang-questions] simple question about list memory consumption In-Reply-To: <87pq89oubb.wl%f@mazzo.li> References: <48564154-48ae-4d26-9a06-aa1091360cc0@knuth> <4FF6E34E.1000102@erix.ericsson.se> <87sjd5ow8t.wl%f@mazzo.li> <4FF6EFDF.4030905@gmail.com> <87pq89oubb.wl%f@mazzo.li> Message-ID: <4FF6F3E0.7040407@gmail.com> On 07/06/2012 04:09 PM, Francesco Mazzoli wrote: > What I did not realise is that > >> The two least significant bits are used for tagging as they are always 0 for a >> pointer to word aligned data. > > Which makes it possible to include the tag in the word. That's what a tag is, in the usual terminology. Otherwise it would have been called a header word. /Richard From f@REDACTED Fri Jul 6 16:29:34 2012 From: f@REDACTED (Francesco Mazzoli) Date: Fri, 06 Jul 2012 15:29:34 +0100 Subject: [erlang-questions] simple question about list memory consumption In-Reply-To: <4FF6F3E0.7040407@gmail.com> References: <48564154-48ae-4d26-9a06-aa1091360cc0@knuth> <4FF6E34E.1000102@erix.ericsson.se> <87sjd5ow8t.wl%f@mazzo.li> <4FF6EFDF.4030905@gmail.com> <87pq89oubb.wl%f@mazzo.li> <4FF6F3E0.7040407@gmail.com> Message-ID: <87obntotdd.wl%f@mazzo.li> At Fri, 06 Jul 2012 16:19:12 +0200, Richard Carlsson wrote: > That's what a tag is, in the usual terminology. Otherwise it would have > been called a header word. Well, in Haskell the word (it cant be 2 bits - that would restrict datatypes to have 4 constructors) that identifies the constructor is called "tag". I guess in erlang the terminology is different. -- Francesco * Often in error, never in doubt From daniel.goertzen@REDACTED Fri Jul 6 18:28:33 2012 From: daniel.goertzen@REDACTED (Daniel Goertzen) Date: Fri, 6 Jul 2012 11:28:33 -0500 Subject: [erlang-questions] specs for fixed length lists Message-ID: While working with lists of fixed length, I noted that type specification for such lists are conspicuously missing. For example, the following does not work... -spec get_buttons() -> {ok, [boolean(), boolean(), boolean(), boolean()]}. Is there some sneaky way to make this go? Now I know you would normally use a tuple in this situation, but I will be using the lists module to process this data so I want to stick with lists. I would love to see something like this for list specification... List :: list(Type) %% Proper list ([]-terminated) + | list(Type, Length) %% Proper list of fixed size + | tlist(TList) %% Proper list specified like a tuple | improper_list(Type1, Type2) %% Type1=contents, Type2=termination | maybe_improper_list(Type1, Type2) %% Type1 and Type2 as above Tuple :: tuple() %% stands for a tuple of any size | {} | {TList} TList :: Type | Type, TList Thanks, Dan. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bourinov@REDACTED Fri Jul 6 23:08:34 2012 From: bourinov@REDACTED (Max Bourinov) Date: Sat, 7 Jul 2012 01:08:34 +0400 Subject: [erlang-questions] What is the best location to place common type specs? In-Reply-To: <06139A918ACCA041BF46A0F36940C7FA39CE0E22@exch-mbx1.msk.trd.ru> References: <06139A918ACCA041BF46A0F36940C7FA39CE0E22@exch-mbx1.msk.trd.ru> Message-ID: <4715714536353355546@unknownmsgid> Hi Sergey, Dedicated module that holds types is your friend. Sent from my iPhone On 05.07.2012, at 16:07, Zhemzhitsky Sergey wrote: Hi guys, I have multiple .hrl files with multiple records. I'd like to define separate types for these records and their fields. These records will be used across multiple modules, and I'm wondering whether it is legal to define type specs in the hrl files? Or would it be better to have a separate module with exported common type specs? Best Regards, Sergey _______________________________________________________ The information contained in this message may be privileged and conf idential and protected from disclosure. If you are not the original intended recipient, you are hereby notified that any review, retransmission, dissemination, or other use of, or taking of any action in reliance upon, this information is prohibited. If you have received this communication in error, please notify the sender immediately by replying to this message and delete it from your computer. Thank you for your cooperation. Troika Dialog, Russia. If you need assistance please contact our Contact Center (+7495) 258 0500 or go to www.troika.ru/eng/Contacts/system.wbp _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang@REDACTED Sat Jul 7 12:15:18 2012 From: erlang@REDACTED (Joe Armstrong) Date: Sat, 7 Jul 2012 12:15:18 +0200 Subject: [erlang-questions] specs for fixed length lists In-Reply-To: References: Message-ID: I don't understand. Lists are intended as containers for a variable number of terms. Tuples are intended as a container for a fixed number of terms. Not using lists and tuples as they are intended will just lead to confusion. /Joe On Fri, Jul 6, 2012 at 6:28 PM, Daniel Goertzen wrote: > While working with lists of fixed length, I noted that type specification > for such lists are conspicuously missing. For example, the following does > not work... > > -spec get_buttons() -> {ok, [boolean(), boolean(), boolean(), boolean()]}. > > Is there some sneaky way to make this go? Now I know you would normally use > a tuple in this situation, but I will be using the lists module to process > this data so I want to stick with lists. > > I would love to see something like this for list specification... > > > List :: list(Type) %% Proper list ([]-terminated) > + | list(Type, Length) %% Proper list of fixed size > + | tlist(TList) %% Proper list specified like a > tuple > | improper_list(Type1, Type2) %% Type1=contents, > Type2=termination > | maybe_improper_list(Type1, Type2) %% Type1 and Type2 as above > > Tuple :: tuple() %% stands for a tuple of any > size > | {} > | {TList} > > TList :: Type > | Type, TList > > > Thanks, > Dan. > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From ivan@REDACTED Sat Jul 7 12:27:20 2012 From: ivan@REDACTED (Ivan Uemlianin) Date: Sat, 07 Jul 2012 11:27:20 +0100 Subject: [erlang-questions] starting a release with nohup In-Reply-To: References: <4FF447D2.5020408@llaisdy.com> Message-ID: <4FF80F08.4080408@llaisdy.com> Dear Matthew Thanks for your comment. I didn't think those arguments would work the the release script (which is a shell script, and which I'd rather leave alone I think). Best wishes Ivan On 05/07/2012 15:16, Matthew Evans wrote: > > I'm using it all the time. > Not sure if you are doing it, but I set the "-detached" and "-noshell" options to the Erlang "erl" command. > i.e: > nohup erl -detached -noshell > > Matt > > ---------------------------------------- >> Date: Wed, 4 Jul 2012 14:40:34 +0100 >> From: ivan@REDACTED >> To: erlang-questions@REDACTED >> Subject: [erlang-questions] starting a release with nohup >> >> Dear All >> >> We have a web application which is a hybrid of python and erlang. As >> part of the deployment infrastructure we can restart the application >> over ssh, using a command like >> >> nohup webapp start >> >> "webapp start" just calls "pythonapp start" and "erlangapp start". >> Although nohup isn't necessary for the erlang app, it seems to be >> necessary for the python app. >> >> Recently we packaged the erlang app as a release, with start and stop >> functions. So, "erlangapp start" calls >> "/path/to/erlangapp/rel/eappnode/bin/eappnode start". >> >> The release set up is much nicer than before, *but* it doesn't seem to >> want to play with nohup. Running the above returns an erlang ebadf >> error message (see below). >> >> Is there a way to start a release using nohup? >> >> With thanks and best wishes >> >> Ivan >> >> >> ** error messages >> >> {error_logger,{{2012,7,4},{9,14,46}},supervisor_report, >> >> [{supervisor,{<0.21.0>,user_sup}},{errorContext,child_terminated},{reason,ebadf},{offender,[{pid,<0.22.0>},{mod,user_sup}]}]} >> >> {error_logger,{{2012,7,4},{9,14,46}}, >> "** Generic server ~p terminating \n >> ** Last message in was ~p~n >> ** When Server state == ~p~n >> ** Reason for termination == ~n >> ** ~p~n", >> [<0.21.0>, >> {'EXIT',<0.22.0>,ebadf}, >> {state,user_sup,undefined,<0.22.0>,{<0.21.0>,user_sup}}, >> ebadf]} >> >> {error_logger,{{2012,7,4},{9,14,46}},crash_report, >> >> [[{initial_call,{supervisor_bridge,user_sup,['Argument__1']}},{pid,<0.21.0>},{registered_name,[]},{error_info,{exit,ebadf,[{gen_server,terminate,6},{proc_lib,init_p_do_apply,3}]}},{ancestors,[kernel_sup,<0.9.0>]},{messages,[]},{links,[<0.10.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,987},{stack_size,24},{reductions,206}],[]]} >> >> {error_logger,{{2012,7,4},{9,14,46}},supervisor_report, >> >> [{supervisor,{local,kernel_sup}},{errorContext,child_terminated},{reason,ebadf},{offender,[{pid,<0.21.0>},{name,user},{mfargs,{user_sup,start,[]}},{restart_type,temporary},{shutdown,2000},{child_type,supervisor}]}]} >> >> >> -- >> ============================================================ >> Ivan A. Uemlianin PhD >> Llaisdy >> Speech Technology Research and Development >> >> ivan@REDACTED >> www.llaisdy.com >> llaisdy.wordpress.com >> github.com/llaisdy >> www.linkedin.com/in/ivanuemlianin >> >> "hilaritas excessum habere nequit" >> (Spinoza, Ethica, IV, XLII) >> ============================================================ >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > -- ============================================================ Ivan A. Uemlianin PhD Llaisdy Speech Technology Research and Development ivan@REDACTED www.llaisdy.com llaisdy.wordpress.com github.com/llaisdy www.linkedin.com/in/ivanuemlianin "hilaritas excessum habere nequit" (Spinoza, Ethica, IV, XLII) ============================================================ From ivan@REDACTED Sat Jul 7 12:31:43 2012 From: ivan@REDACTED (Ivan Uemlianin) Date: Sat, 07 Jul 2012 11:31:43 +0100 Subject: [erlang-questions] Problem attaching a shell to an Erlang node started through SSH In-Reply-To: References: <43D0FCE5-625C-4903-8EF0-0133F44122C3@llaisdy.com> Message-ID: <4FF8100F.6000609@llaisdy.com> Dear Andr? Sorry I made a typo in my email (never try to be helpful before breakfast;). I should have written: >> - this function calls an *init* script instead of calling the >> release start directly. e.g.: sudo("webapp start") Then the shell script "webapp" calls start functions for the python app and the erlang app (i.e., "/path/to/erlangapp/rel/eappnode/bin/eappnode start"). > user@REDACTED:~/test_project/rel/mynode/bin$ sudo ./mynode attach > ... I didn't know about attach! I've just verified I can ssh onto the remote machine and attach to the node. Why is sshing onto the remote machine and attaching better than running an erlang shell on your local machine and connecting to the remote machine from there? Thanks Ivan On 06/07/2012 09:17, Andr? Graf wrote: > Hi Ivan, > > Running EUnit works. I am pretty sure the problem is somehow related > with the runner script generated by Rebar (which I haven't changed). > I tried both sudo and run. I forgot to mention that you must disable > fabric pty when starting the node through the runner script otherwise > the start call succeeds but no node (and also no logs) are generated. > > Symptoms are difficult to describe. The resulting shell looks like that: > user@REDACTED:~/test_project/rel/mynode/bin$ sudo ./mynode attach > Attaching to /tmp//home/user/test_project/rel/mynode/erlang.pipe.3 (^D to exit) > > ^R > > So I won't get the normal Erlang shell prompt, I can run some Erlang > commands though, but I don't have any tab-completion. > Ctrl-D will properly exit, Ctrl-G is not giving me the User switch > command, Ctrl-C gives the proper Break menu. > > Funny here, If I don't use sudo for the attach I get a "Node is not > running!" although the node was started without using sudo, but this > problem also appears if I start the node directly on the server > without going through SSH. > > Cheers, > Andr? > > > > > > > On 6 July 2012 07:27, Ivan Uemlianin wrote: >> Dear Andr? >> >> This is what we're doing (see my recent email) and the erlang is working >> fine. I'll be at the office this afternoon to check properly but here are a >> few differences: >> >> - our fab script uses sudo() instead of run(); >> - this function calls an unit script instead of calling the release start >> directly. >> >>> However, attaching a shell to the >>> node triggers a strange >>> behavior, the shell is available but >>> unusable. >> >> What are the symptoms? >> >> Best wishes >> >> Ivan >> >> >> Sent from my iPhone >> >> On 5 Jul 2012, at 23:27, Andr? Graf wrote: >> >> Hello, >> >> I am working on a small library that uses Python Fabric together with >> Rebar deploying of Erlang applications. Fabric provides a scriptable >> (it's Python) abstraction over SSH. >> While I am able to deploy and upgrade new releases through such a >> fabric script I struggle with a simple Erlang node startup. >> >> A fabric call e.g. run("/path/to/my/release/bin/myproject start") >> invokes following SSH command >> ssh user@REDACTED "/bin/bash -l -c '/path/to/my/release/bin/myproject start'" >> >> This starts the node and I can use the remote shell to operate on the >> node. However, attaching a shell to the node triggers a strange >> behavior, the shell is available but unusable. It seems the startup >> through SSH has somehow confused the involved pipes. >> >> Do you have any hints on that? I am using rebar 2.0.0 R15B >> 20120625_185716 git 2.0.0-45-gc41fda6. >> >> Cheers, >> Andre >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> -- ============================================================ Ivan A. Uemlianin PhD Llaisdy Speech Technology Research and Development ivan@REDACTED www.llaisdy.com llaisdy.wordpress.com github.com/llaisdy www.linkedin.com/in/ivanuemlianin "hilaritas excessum habere nequit" (Spinoza, Ethica, IV, XLII) ============================================================ From andre.graf@REDACTED Sat Jul 7 13:38:16 2012 From: andre.graf@REDACTED (=?ISO-8859-1?Q?Andr=E9_Graf?=) Date: Sat, 7 Jul 2012 13:38:16 +0200 Subject: [erlang-questions] Problem attaching a shell to an Erlang node started through SSH In-Reply-To: <4FF8100F.6000609@llaisdy.com> References: <43D0FCE5-625C-4903-8EF0-0133F44122C3@llaisdy.com> <4FF8100F.6000609@llaisdy.com> Message-ID: Hi Ivan Thank you for your reply! Good question, maybe somebody more experienced can shed some light on that one. Actually the main reason why I stumbled upon these problems is that I wanted to start the node using the privbind tool for letting a non-privileged run-user bind to privileged ports such as 80 or 443. If I used fabric e.g. sudo('privbind -u myuser /path/to/my/release/bin/myproject start', pty=False) the fabric call froze my terminal since the privbind process will terminate after the Erlang node has terminated (hopefully never). At this point I have two problems: first, why behaves an attached shell so weird if you start the node through SSH. Second how to use privbind within a fabric command. For the first one I can well live with the workaround through the remote shell. For the second problem I am on the wrong mailing list. Cheers, Andr? On 7 July 2012 12:31, Ivan Uemlianin wrote: > Dear Andr? > > Sorry I made a typo in my email (never try to be helpful before breakfast;). > I should have written: > >>> - this function calls an *init* script instead of calling the >>> release start directly. > > e.g.: > > sudo("webapp start") > > Then the shell script "webapp" calls start functions for the python app and > the erlang app (i.e., "/path/to/erlangapp/rel/eappnode/bin/eappnode start"). > > >> user@REDACTED:~/test_project/rel/mynode/bin$ sudo ./mynode attach >> ... > > I didn't know about attach! I've just verified I can ssh onto the remote > machine and attach to the node. > > Why is sshing onto the remote machine and attaching better than running an > erlang shell on your local machine and connecting to the remote machine from > there? > > Thanks > > Ivan > > > > On 06/07/2012 09:17, Andr? Graf wrote: >> >> Hi Ivan, >> >> Running EUnit works. I am pretty sure the problem is somehow related >> with the runner script generated by Rebar (which I haven't changed). >> I tried both sudo and run. I forgot to mention that you must disable >> fabric pty when starting the node through the runner script otherwise >> the start call succeeds but no node (and also no logs) are generated. >> >> Symptoms are difficult to describe. The resulting shell looks like that: >> user@REDACTED:~/test_project/rel/mynode/bin$ sudo ./mynode attach >> Attaching to /tmp//home/user/test_project/rel/mynode/erlang.pipe.3 (^D to >> exit) >> >> ^R >> >> So I won't get the normal Erlang shell prompt, I can run some Erlang >> commands though, but I don't have any tab-completion. >> Ctrl-D will properly exit, Ctrl-G is not giving me the User switch >> command, Ctrl-C gives the proper Break menu. >> >> Funny here, If I don't use sudo for the attach I get a "Node is not >> running!" although the node was started without using sudo, but this >> problem also appears if I start the node directly on the server >> without going through SSH. >> >> Cheers, >> Andr? >> >> >> >> >> >> >> On 6 July 2012 07:27, Ivan Uemlianin wrote: >>> >>> Dear Andr? >>> >>> This is what we're doing (see my recent email) and the erlang is working >>> fine. I'll be at the office this afternoon to check properly but here are >>> a >>> few differences: >>> >>> - our fab script uses sudo() instead of run(); >>> - this function calls an unit script instead of calling the release start >>> directly. >>> >>>> However, attaching a shell to the >>>> node triggers a strange >>>> behavior, the shell is available but >>>> unusable. >>> >>> >>> What are the symptoms? >>> >>> Best wishes >>> >>> Ivan >>> >>> >>> Sent from my iPhone >>> >>> On 5 Jul 2012, at 23:27, Andr? Graf wrote: >>> >>> Hello, >>> >>> I am working on a small library that uses Python Fabric together with >>> Rebar deploying of Erlang applications. Fabric provides a scriptable >>> (it's Python) abstraction over SSH. >>> While I am able to deploy and upgrade new releases through such a >>> fabric script I struggle with a simple Erlang node startup. >>> >>> A fabric call e.g. run("/path/to/my/release/bin/myproject start") >>> invokes following SSH command >>> ssh user@REDACTED "/bin/bash -l -c '/path/to/my/release/bin/myproject >>> start'" >>> >>> This starts the node and I can use the remote shell to operate on the >>> node. However, attaching a shell to the node triggers a strange >>> behavior, the shell is available but unusable. It seems the startup >>> through SSH has somehow confused the involved pipes. >>> >>> Do you have any hints on that? I am using rebar 2.0.0 R15B >>> 20120625_185716 git 2.0.0-45-gc41fda6. >>> >>> Cheers, >>> Andre >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> > > > -- > ============================================================ > Ivan A. Uemlianin PhD > Llaisdy > Speech Technology Research and Development > > ivan@REDACTED > www.llaisdy.com > llaisdy.wordpress.com > github.com/llaisdy > www.linkedin.com/in/ivanuemlianin > > "hilaritas excessum habere nequit" > (Spinoza, Ethica, IV, XLII) > ============================================================ > > From tony@REDACTED Sat Jul 7 13:57:45 2012 From: tony@REDACTED (Tony Rogvall) Date: Sat, 7 Jul 2012 13:57:45 +0200 Subject: [erlang-questions] specs for fixed length lists In-Reply-To: References: Message-ID: <16D9EF7B-2735-4D8A-8BCE-301E6C60FE1D@rogvall.se> Hi! On 6 jul 2012, at 18:28, Daniel Goertzen wrote: > While working with lists of fixed length, I noted that type specification for such lists are conspicuously missing. For example, the following does not work... > > -spec get_buttons() -> {ok, [boolean(), boolean(), boolean(), boolean()]}. > Try this: -type buttons() :: [boolean() | [boolean() | [boolean() | [boolean() | []]]]]. -spec get_buttons() -> {ok, buttons()}. get_buttons() -> [true | [false | [true | [false | [] ] ] ] ]. Of course my formatting of the get_buttons return value is intensional. You could write it as normal [true,false,true,false]. You can of course write it as: -spec get_buttons() -> {ok,[boolean() | [boolean() | [boolean() | [boolean() | []]]]] }. Regards /Tony > Is there some sneaky way to make this go? Now I know you would normally use a tuple in this situation, but I will be using the lists module to process this data so I want to stick with lists. > > I would love to see something like this for list specification... > > > List :: list(Type) %% Proper list ([]-terminated) > + | list(Type, Length) %% Proper list of fixed size > + | tlist(TList) %% Proper list specified like a tuple > | improper_list(Type1, Type2) %% Type1=contents, Type2=termination > | maybe_improper_list(Type1, Type2) %% Type1 and Type2 as above > > Tuple :: tuple() %% stands for a tuple of any size > | {} > | {TList} > > TList :: Type > | Type, TList > > > Thanks, > Dan. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions "Installing applications can lead to corruption over time. Applications gradually write over each other's libraries, partial upgrades occur, user and system errors happen, and minute changes may be unnoticeable and difficult to fix" -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.goertzen@REDACTED Sat Jul 7 15:03:19 2012 From: daniel.goertzen@REDACTED (Daniel Goertzen) Date: Sat, 7 Jul 2012 08:03:19 -0500 Subject: [erlang-questions] specs for fixed length lists In-Reply-To: References: Message-ID: But tuples lack things like lists:map() and lists:foreach(). If there were such functions for tuples, I would agree with you and just use tuples. Dan. On Sat, Jul 7, 2012 at 5:15 AM, Joe Armstrong wrote: > I don't understand. > > Lists are intended as containers for a variable number of terms. Tuples are > intended as a container for a fixed number of terms. Not using lists and > tuples > as they are intended will just lead to confusion. > > /Joe > > On Fri, Jul 6, 2012 at 6:28 PM, Daniel Goertzen > wrote: > > While working with lists of fixed length, I noted that type specification > > for such lists are conspicuously missing. For example, the following > does > > not work... > > > > -spec get_buttons() -> {ok, [boolean(), boolean(), boolean(), > boolean()]}. > > > > Is there some sneaky way to make this go? Now I know you would normally > use > > a tuple in this situation, but I will be using the lists module to > process > > this data so I want to stick with lists. > > > > I would love to see something like this for list specification... > > > > > > List :: list(Type) %% Proper list > ([]-terminated) > > + | list(Type, Length) %% Proper list of fixed size > > + | tlist(TList) %% Proper list specified > like a > > tuple > > | improper_list(Type1, Type2) %% Type1=contents, > > Type2=termination > > | maybe_improper_list(Type1, Type2) %% Type1 and Type2 as above > > > > Tuple :: tuple() %% stands for a tuple of any > > size > > | {} > > | {TList} > > > > TList :: Type > > | Type, TList > > > > > > Thanks, > > Dan. > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gianfranco.alongi@REDACTED Sat Jul 7 15:08:43 2012 From: gianfranco.alongi@REDACTED (Gianfranco Alongi) Date: Sat, 7 Jul 2012 15:08:43 +0200 Subject: [erlang-questions] specs for fixed length lists In-Reply-To: References: Message-ID: You can (of course with an overhead penalty), do tuple_to_list and use map / foreach. But yes, it would be nice if you could somehow define your tuples as an instance of a Haskell functor. /G On Sat, Jul 7, 2012 at 3:03 PM, Daniel Goertzen wrote: > But tuples lack things like lists:map() and lists:foreach(). If there were > such functions for tuples, I would agree with you and just use tuples. > > Dan. > > On Sat, Jul 7, 2012 at 5:15 AM, Joe Armstrong wrote: >> >> I don't understand. >> >> Lists are intended as containers for a variable number of terms. Tuples >> are >> intended as a container for a fixed number of terms. Not using lists and >> tuples >> as they are intended will just lead to confusion. >> >> /Joe >> >> On Fri, Jul 6, 2012 at 6:28 PM, Daniel Goertzen >> wrote: >> > While working with lists of fixed length, I noted that type >> > specification >> > for such lists are conspicuously missing. For example, the following >> > does >> > not work... >> > >> > -spec get_buttons() -> {ok, [boolean(), boolean(), boolean(), >> > boolean()]}. >> > >> > Is there some sneaky way to make this go? Now I know you would normally >> > use >> > a tuple in this situation, but I will be using the lists module to >> > process >> > this data so I want to stick with lists. >> > >> > I would love to see something like this for list specification... >> > >> > >> > List :: list(Type) %% Proper list >> > ([]-terminated) >> > + | list(Type, Length) %% Proper list of fixed size >> > + | tlist(TList) %% Proper list specified >> > like a >> > tuple >> > | improper_list(Type1, Type2) %% Type1=contents, >> > Type2=termination >> > | maybe_improper_list(Type1, Type2) %% Type1 and Type2 as above >> > >> > Tuple :: tuple() %% stands for a tuple of any >> > size >> > | {} >> > | {TList} >> > >> > TList :: Type >> > | Type, TList >> > >> > >> > Thanks, >> > Dan. >> > >> > >> > _______________________________________________ >> > erlang-questions mailing list >> > erlang-questions@REDACTED >> > http://erlang.org/mailman/listinfo/erlang-questions >> > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From carlsson.richard@REDACTED Sat Jul 7 21:53:25 2012 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Sat, 07 Jul 2012 21:53:25 +0200 Subject: [erlang-questions] specs for fixed length lists In-Reply-To: <16D9EF7B-2735-4D8A-8BCE-301E6C60FE1D@rogvall.se> References: <16D9EF7B-2735-4D8A-8BCE-301E6C60FE1D@rogvall.se> Message-ID: <4FF893B5.1090302@gmail.com> On 2012-07-07 13:57, Tony Rogvall wrote: > Try this: > > -type buttons() :: [boolean() | [boolean() | [boolean() | > [boolean() | []]]]]. You certainly can _write_ it like that, just like you can write "1+1" instead of "2". To a tool like Dialyzer, it will still just be interpreted as a (nonempty) list of boolean(). The exact number of elements is not tracked. And if you for example try to say "[atom() | [integer() | []]]", it will just be the same as saying "nonempty list of atom()|integer()" - the order of occurrence of the element types is also not part of the list type. /Richard From ok@REDACTED Sun Jul 8 07:22:36 2012 From: ok@REDACTED (ok@REDACTED) Date: Sun, 8 Jul 2012 17:22:36 +1200 Subject: [erlang-questions] simple question about list memory consumption In-Reply-To: <48564154-48ae-4d26-9a06-aa1091360cc0@knuth> References: <48564154-48ae-4d26-9a06-aa1091360cc0@knuth> Message-ID: > Hm, ignore my previous answer, then; I was under the impression that > small enough integers would be stored as immediates in the > list cell... They are. See the Efficiency guide. http://www.erlang.org/doc/efficiency_guide/advanced.html#id68680 "String ... 2 words per character" Each list cell holds one integer, which *is* an immediate in the list cell and points nowhere else, and one pointer to the next element in the list. >> 2. opening an Erlang shell (few MB consumption - stable); >> 3. executing: >> >> >> L = lists:map(fun(_) -> 107 end,lists:seq(1,10000000)), ok. Two things strike me about this. (A) you construct TWO lists here, so at 8 bytes per list cell on a 32-bit machine, there's 16 bytes each already. (B) You are doing this in the shell, where expressions are interpreted. I have no idea what space overheads that adds. Is there a way of measuring the amount of heap space currently _used_ in a process? process_info>heap_size and process_info>total_heap_size seem to be about the amount of space allocated for a process, including its stack and any quantity of free space. So the following only gave me an approximate idea: -module(ss). -export([ss/1]). heap_size() -> erlang:garbage_collect(), element(2, process_info(self(), heap_size)). ss(N) -> Before = heap_size(), Result = loop(N), After = heap_size(), (After-Before)*1.0 / length(Result). loop(0) -> []; loop(N) -> [$k | loop(N-1)]. However, the idea it _did_ give me was "between 2 and 3 words per character", so it looks like "2 words per character" as stated in the Efficiency Guide is, after all, true. It's certainly nowhere near 21 bytes. From ok@REDACTED Sun Jul 8 08:22:37 2012 From: ok@REDACTED (ok@REDACTED) Date: Sun, 8 Jul 2012 18:22:37 +1200 Subject: [erlang-questions] specs for fixed length lists In-Reply-To: References: Message-ID: I'm with /Joe on this one. There aren't all _that_ many list operations that make sense on tuples, and map4(F, {A,B,C,D}) -> {F(A), F(B), F(C), F(D)}. is easy enough to define. Perhaps the code needs turning inside out. I think we need a bit more detail about what actually needs to be done. From bchesneau@REDACTED Sun Jul 8 08:59:39 2012 From: bchesneau@REDACTED (Benoit Chesneau) Date: Sun, 8 Jul 2012 08:59:39 +0200 Subject: [erlang-questions] heartbeat timeout + computer sleep Message-ID: Each time the computer wake up from sleep the heartbeat timeout on a detached release and make it crash. What would be the other way to handle a detached process in this case. I looked at erld but it is only working on unix platforms. - benoit From tony@REDACTED Sun Jul 8 11:48:28 2012 From: tony@REDACTED (Tony Rogvall) Date: Sun, 8 Jul 2012 11:48:28 +0200 Subject: [erlang-questions] specs for fixed length lists In-Reply-To: <4FF893B5.1090302@gmail.com> References: <16D9EF7B-2735-4D8A-8BCE-301E6C60FE1D@rogvall.se> <4FF893B5.1090302@gmail.com> Message-ID: On 7 jul 2012, at 21:53, Richard Carlsson wrote: > On 2012-07-07 13:57, Tony Rogvall wrote: >> Try this: >> >> -type buttons() :: [boolean() | [boolean() | [boolean() | >> [boolean() | []]]]]. > > You certainly can _write_ it like that, just like you can write "1+1" instead of "2". To a tool like Dialyzer, it will still just be interpreted as a (nonempty) list of boolean(). The exact number of elements is not tracked. And if you for example try to say "[atom() | [integer() | []]]", it will just be the same as saying "nonempty list of atom()|integer()" - the order of occurrence of the element types is also not part of the list type. > But you can not _write_ (why do you use underscore before and after write? ) things like -spec get_buttons1() -> {ok, [boolean(),boolean(),boolean(),boolean()]}. Because you will get a syntax error!. Why is the "dotted pair" notation supported if it does not work ? It can not be harder to support then supporting pairs ? (For the record I do think it could be a bit harder :) BTW The 1+1 = 2 example was totally silly in my opinion. /Tony > /Richard > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions "Installing applications can lead to corruption over time. Applications gradually write over each other's libraries, partial upgrades occur, user and system errors happen, and minute changes may be unnoticeable and difficult to fix" -------------- next part -------------- An HTML attachment was scrubbed... URL: From zerthurd@REDACTED Sun Jul 8 14:24:03 2012 From: zerthurd@REDACTED (Maxim Treskin) Date: Sun, 8 Jul 2012 19:24:03 +0700 Subject: [erlang-questions] Dialyzer cannot catch gen_server callback return type error In-Reply-To: References: <20120705111748.GB3724@localhost> <21efd459-0e1a-41d4-a043-e472c064c85d@googlegroups.com> Message-ID: On 8 July 2012 07:13, Sven Heyll wrote: > Acutally in gen_server I find: > > -callback handle_cast(Request :: term(), State :: term()) -> > {noreply, NewState :: term()} | > {noreply, NewState :: term(), timeout() | hibernate} | > {stop, Reason :: term(), NewState :: term()}. > > Why not define it this way: > > -callback handle_cast(Request :: term(), State) -> > {noreply, State} | > {noreply, State, timeout() | hibernate} | > {stop, Reason :: term(), State}. > > Is this possible?? I think this would help well written apps that are > careful with type consistency, and hurt only bad code - but one who > writes bad code wouldn't invoke dialyzer. > > No, it is impossible. What type of State would be here? > wbr > Sven > > > 2012/7/6 Maxim Treskin : > > > > > > On 6 July 2012 17:35, Stavros Aronis wrote: > >> > >> Hi again, > >> > >> The signature (or "success typing" in Dialyzer's terminology) is > >> `my_module:handle_cast(term(),term()) -> {'noreply',term()}` because the > >> second clause: > >> > >> handle_cast(_Msg, State) -> > >> {noreply, State}. > >> > >> will accept any terms as _Msg and State and return {noreply, State} in > >> which State again is any term. In Dialyzer's type system, this {noreply, > >> term()} return value will 'absorb' the definitely incorrect {noreply, > >> {bugotak}} as {bugotak} is also a term(). From that point on, Dialyzer > can > >> assume that term() might also be #state{} and emit no warnings. This > happens > >> because Dialyzer checks the whole range of return values at once and not > >> each particular clause. If all the return values were definitely > different > >> than #state{}, then and only then would Dialyzer warn you. > >> > > > > Thanks for the clarification, Stavros! Now I understand how Dialyzer > checks > > types of function clauses. > > > >> > >> Again, trying to check each particular return value against a given spec > >> might be an interesting extension and we will consider it in the future. > >> > > > > I hope so. It would be a significant level up! > > > >> > >> Stavros > >> > >> On Fri, Jul 6, 2012 at 11:39 AM, Maxim Treskin > wrote: > >>> > >>> I just really not understand. Function handle_cast returns, and returns > >>> incorrect type according to -spec signature, right? > >>> Why Dialyzer lose this information? It is sort of missbehaviour i > think. > >>> > >>> -- > >>> Max Treskin > >> > >> > > > > > > > > -- > > Max Treskin > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > -- Max Treskin -------------- next part -------------- An HTML attachment was scrubbed... URL: From tony@REDACTED Sun Jul 8 19:47:16 2012 From: tony@REDACTED (Tony Rogvall) Date: Sun, 8 Jul 2012 19:47:16 +0200 Subject: [erlang-questions] specs for fixed length lists In-Reply-To: <4FF893B5.1090302@gmail.com> References: <16D9EF7B-2735-4D8A-8BCE-301E6C60FE1D@rogvall.se> <4FF893B5.1090302@gmail.com> Message-ID: <72E7AD09-75F1-4E54-8094-CED0D42F4BF8@rogvall.se> While thinking about this problem I wrote the following spec: -spec get_buttons() -> {ok,{boolean(),{boolean(),{boolean(),{boolean(),[]}}}}}. get_buttons() -> {ok,{true,{true,{false,12}}}}. This passed dialyzer! The reason (I suppose) is that the analysis is limited to depth = 2 (or 3). Running typer while commenting out the spec gives: -spec get_buttons() -> {'ok',{'true',{'true',{_,_}}}}. I guess this is the real reason why dialyzer can not handle fixed size lists (at least not a list with a length greater than 3) /Tony On 7 jul 2012, at 21:53, Richard Carlsson wrote: > On 2012-07-07 13:57, Tony Rogvall wrote: >> Try this: >> >> -type buttons() :: [boolean() | [boolean() | [boolean() | >> [boolean() | []]]]]. > > You certainly can _write_ it like that, just like you can write "1+1" instead of "2". To a tool like Dialyzer, it will still just be interpreted as a (nonempty) list of boolean(). The exact number of elements is not tracked. And if you for example try to say "[atom() | [integer() | []]]", it will just be the same as saying "nonempty list of atom()|integer()" - the order of occurrence of the element types is also not part of the list type. > > /Richard > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions "Installing applications can lead to corruption over time. Applications gradually write over each other's libraries, partial upgrades occur, user and system errors happen, and minute changes may be unnoticeable and difficult to fix" -------------- next part -------------- An HTML attachment was scrubbed... URL: From n.oxyde@REDACTED Sun Jul 8 20:14:34 2012 From: n.oxyde@REDACTED (Anthony Ramine) Date: Sun, 8 Jul 2012 20:14:34 +0200 Subject: [erlang-questions] specs for fixed length lists In-Reply-To: <4FF893B5.1090302@gmail.com> References: <16D9EF7B-2735-4D8A-8BCE-301E6C60FE1D@rogvall.se> <4FF893B5.1090302@gmail.com> Message-ID: Doesn't this spec mean "a list of atom() or list of integer() and empty list"? I don't see any nonempty constraint in this type. Regards, -- Anthony Ramine Le 7 juil. 2012 ? 21:53, Richard Carlsson a ?crit : > On 2012-07-07 13:57, Tony Rogvall wrote: >> Try this: >> >> -type buttons() :: [boolean() | [boolean() | [boolean() | >> [boolean() | []]]]]. > > You certainly can _write_ it like that, just like you can write "1+1" instead of "2". To a tool like Dialyzer, it will still just be interpreted as a (nonempty) list of boolean(). The exact number of elements is not tracked. And if you for example try to say "[atom() | [integer() | []]]", it will just be the same as saying "nonempty list of atom()|integer()" - the order of occurrence of the element types is also not part of the list type. > > /Richard > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From tony@REDACTED Sun Jul 8 20:42:14 2012 From: tony@REDACTED (Tony Rogvall) Date: Sun, 8 Jul 2012 20:42:14 +0200 Subject: [erlang-questions] specs for fixed length lists In-Reply-To: <4FF893B5.1090302@gmail.com> References: <16D9EF7B-2735-4D8A-8BCE-301E6C60FE1D@rogvall.se> <4FF893B5.1090302@gmail.com> Message-ID: <757E7DFC-BF3A-4115-A9CC-B3771A49E001@rogvall.se> Ah, you are right (of course) Stupid mistake trying to use | for dotted pair, sorry about that. I was suppose to "write" something like: -type buttons() :: maybe_improper_list(boolean(), maybe_improper_list(boolean(), maybe_improper_list(boolean(), maybe_improper_list(boolean(),[])))). Several attempts writing this kind of specs, crashed dialyzer: -type buttons() :: maybe_improper_list(boolean(), integer()). -spec get_buttons() -> {ok,buttons()}. get_buttons() -> {ok,[true|12]}. --- dialyzer list_type.erl Checking whether the PLT /Users/tony/.dialyzer_plt is up-to-date... yes Proceeding with analysis... =ERROR REPORT==== 8-Jul-2012::20:41:14 === Error in process <0.29.0> with exit value: {{badmatch,false},[{erl_types,t_maybe_improper_list,2,[{file,"erl_types.erl"},{line,1361}]},{erl_types,t_from_form,5,[{file,"erl_types.erl"},{line,3665}]},{erl_types,t_from_form,5,[{file,"erl_types.erl"},{line,3724}]},{erl_types,list_from_form... dialyzer: Analysis failed with error: {{badmatch,false}, [{erl_types,t_maybe_improper_list,2,[{file,"erl_types.erl"},{line,1361}]}, {erl_types,t_from_form,5,[{file,[...]},{line,...}]}, {erl_types,t_from_form,5,[{file,...},{...}]}, {erl_types,list_from_form,5,[{...}|...]}, {erl_types,list_from_form,5,[...]}, {erl_types,t_from_form,5,...}, {erl_types,t_from_form,...}, {erl_types,...}]} Last messages in the log cache: Reading files and computing callgraph... done in 0.13 secs Removing edges... done in 0.00 secs /Tony On 7 jul 2012, at 21:53, Richard Carlsson wrote: > On 2012-07-07 13:57, Tony Rogvall wrote: >> Try this: >> >> -type buttons() :: [boolean() | [boolean() | [boolean() | >> [boolean() | []]]]]. > > You certainly can _write_ it like that, just like you can write "1+1" instead of "2". To a tool like Dialyzer, it will still just be interpreted as a (nonempty) list of boolean(). The exact number of elements is not tracked. And if you for example try to say "[atom() | [integer() | []]]", it will just be the same as saying "nonempty list of atom()|integer()" - the order of occurrence of the element types is also not part of the list type. > > /Richard > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions "Installing applications can lead to corruption over time. Applications gradually write over each other's libraries, partial upgrades occur, user and system errors happen, and minute changes may be unnoticeable and difficult to fix" -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlsson.richard@REDACTED Sun Jul 8 21:40:35 2012 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Sun, 08 Jul 2012 21:40:35 +0200 Subject: [erlang-questions] specs for fixed length lists In-Reply-To: References: <16D9EF7B-2735-4D8A-8BCE-301E6C60FE1D@rogvall.se> <4FF893B5.1090302@gmail.com> Message-ID: <4FF9E233.20200@gmail.com> On 07/08/2012 08:14 PM, Anthony Ramine wrote: > Doesn't this spec mean "a list of atom() or list of integer() and > empty list"? I don't see any nonempty constraint in this type. > > Regards, > > -- Anthony Ramine Yes, you're right. I abused the notation in the same way that Tony's original question was stated. In the type notation, "|" just means type union, so "[T1 | T2]" means "list of T1|T2", and not "list starting with T1 followed by a tail of T2". Basically, there are no individual cons cells in the type notation. /Richard > Le 7 juil. 2012 ? 21:53, Richard Carlsson > a ?crit : > >> On 2012-07-07 13:57, Tony Rogvall wrote: >>> Try this: >>> >>> -type buttons() :: [boolean() | [boolean() | [boolean() | >>> [boolean() | []]]]]. >> >> You certainly can _write_ it like that, just like you can write >> "1+1" instead of "2". To a tool like Dialyzer, it will still just >> be interpreted as a (nonempty) list of boolean(). The exact number >> of elements is not tracked. And if you for example try to say >> "[atom() | [integer() | []]]", it will just be the same as saying >> "nonempty list of atom()|integer()" - the order of occurrence of >> the element types is also not part of the list type. >> >> /Richard _______________________________________________ >> erlang-questions mailing list erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions From cgsmcmlxxv@REDACTED Sun Jul 8 21:41:14 2012 From: cgsmcmlxxv@REDACTED (CGS) Date: Sun, 8 Jul 2012 21:41:14 +0200 Subject: [erlang-questions] simple question about list memory consumption In-Reply-To: References: <48564154-48ae-4d26-9a06-aa1091360cc0@knuth> Message-ID: Hi, About the two lists you were speaking about (I suppose you were referring to the output of lists:map/2 and lists:seq/2), the second is destroyed as soon as lists:map/2 exits. So, there shouldn't be any overhead here. And, yes, the memory consumption peaked at higher value, but stabilized itself at the value I mentioned before. On a 64-bits machine, 2 words = 16 B and 3 words = 24 B, so, my value of 21 B is exactly in between 2 and 3 words. I suppose you worked on 32-bits machine or with half-world emulator and you got half of that value (or a bit less if we are to consider that the shell is adding bigger overhead). CGS On Sun, Jul 8, 2012 at 7:22 AM, wrote: > > Hm, ignore my previous answer, then; I was under the impression that > > small enough integers would be stored as immediates in the > > list cell... > > They are. > See the Efficiency guide. > http://www.erlang.org/doc/efficiency_guide/advanced.html#id68680 > "String ... 2 words per character" > > Each list cell holds one integer, which *is* an immediate in the > list cell and points nowhere else, and one pointer to the next > element in the list. > > > > >> 2. opening an Erlang shell (few MB consumption - stable); > >> 3. executing: > >> > >> > >> L = lists:map(fun(_) -> 107 end,lists:seq(1,10000000)), ok. > > Two things strike me about this. > (A) you construct TWO lists here, so at 8 bytes per list cell > on a 32-bit machine, there's 16 bytes each already. > (B) You are doing this in the shell, where expressions are > interpreted. I have no idea what space overheads that adds. > > Is there a way of measuring the amount of heap space currently > _used_ in a process? process_info>heap_size and > process_info>total_heap_size seem to be about the amount of > space allocated for a process, including its stack and any quantity > of free space. So the following only gave me an approximate idea: > > -module(ss). > -export([ss/1]). > > heap_size() -> > erlang:garbage_collect(), > element(2, process_info(self(), heap_size)). > > ss(N) -> > Before = heap_size(), > Result = loop(N), > After = heap_size(), > (After-Before)*1.0 / length(Result). > > loop(0) -> > []; > loop(N) -> > [$k | loop(N-1)]. > > > However, the idea it _did_ give me was "between 2 and 3 words > per character", so it looks like "2 words per character" as > stated in the Efficiency Guide is, after all, true. > > It's certainly nowhere near 21 bytes. > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlsson.richard@REDACTED Sun Jul 8 21:50:54 2012 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Sun, 08 Jul 2012 21:50:54 +0200 Subject: [erlang-questions] specs for fixed length lists In-Reply-To: References: <16D9EF7B-2735-4D8A-8BCE-301E6C60FE1D@rogvall.se> <4FF893B5.1090302@gmail.com> Message-ID: <4FF9E49E.8000002@gmail.com> On 07/08/2012 11:48 AM, Tony Rogvall wrote: > BTW > The 1+1 = 2 example was totally silly in my opinion. I was just trying to make the point that in the type notation, the elements of a list are the union of all possible types of elements that can occur anywhere in the list. This discards all information about how many elements there may be in a list, and in which order. In that way, it's exactly like how e.g. "3" may represent any and all of "1+2", "2+1", "1+1+1", and "3". /Richard From tony@REDACTED Sun Jul 8 21:54:04 2012 From: tony@REDACTED (Tony Rogvall) Date: Sun, 8 Jul 2012 21:54:04 +0200 Subject: [erlang-questions] specs for fixed length lists In-Reply-To: <4FF9E49E.8000002@gmail.com> References: <16D9EF7B-2735-4D8A-8BCE-301E6C60FE1D@rogvall.se> <4FF893B5.1090302@gmail.com> <4FF9E49E.8000002@gmail.com> Message-ID: <96CD1CFC-0BFD-4F14-95EC-A4F06BD6879A@rogvall.se> On 8 jul 2012, at 21:50, Richard Carlsson wrote: > On 07/08/2012 11:48 AM, Tony Rogvall wrote: >> BTW >> The 1+1 = 2 example was totally silly in my opinion. > > I was just trying to make the point that in the type notation, the elements of a list are the union of all possible types of elements that can occur anywhere in the list. This discards all information about how many elements there may be in a list, and in which order. In that way, it's exactly like how e.g. "3" may represent any and all of "1+2", "2+1", "1+1+1", and "3". > Point taken! But I was totally out of context :-) Sorry about that. /Tony > /Richard "Installing applications can lead to corruption over time. Applications gradually write over each other's libraries, partial upgrades occur, user and system errors happen, and minute changes may be unnoticeable and difficult to fix" -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlsson.richard@REDACTED Sun Jul 8 21:58:58 2012 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Sun, 08 Jul 2012 21:58:58 +0200 Subject: [erlang-questions] specs for fixed length lists In-Reply-To: <72E7AD09-75F1-4E54-8094-CED0D42F4BF8@rogvall.se> References: <16D9EF7B-2735-4D8A-8BCE-301E6C60FE1D@rogvall.se> <4FF893B5.1090302@gmail.com> <72E7AD09-75F1-4E54-8094-CED0D42F4BF8@rogvall.se> Message-ID: <4FF9E682.8030005@gmail.com> On 07/08/2012 07:47 PM, Tony Rogvall wrote: > While thinking about this problem I wrote the following spec: > > -spec get_buttons() -> > {ok,{boolean(),{boolean(),{boolean(),{boolean(),[]}}}}}. > > get_buttons() -> > {ok,{true,{true,{false,12}}}}. > > This passed dialyzer! > > The reason (I suppose) is that the analysis is limited to depth = 2 (or 3). > Running typer while commenting out the spec gives: > > -spec get_buttons() -> {'ok',{'true',{'true',{_,_}}}}. > > I guess this is the real reason why dialyzer can not handle fixed size > lists (at least not a list with a length greater than 3) Dialyzer is still pretty poor at handling recursive tuple structures, as you noted, and uses depth-k limitation (I don't remember the details, and they've probably changed since I last looked at that code). But lists are treated differently from tuples. It is assumed that most sane uses of lists will not depend on the order of elements from a type perspective. This lets Dialyzer use a less exact approximation, which still captures most of the useful information for most programs but without slowing down the type analysis too much. /Richard From paul.joseph.davis@REDACTED Mon Jul 9 00:53:52 2012 From: paul.joseph.davis@REDACTED (Paul Davis) Date: Sun, 8 Jul 2012 17:53:52 -0500 Subject: [erlang-questions] Selective receive optimization for NIFs Message-ID: I've got a bit of a curious question in relation to OTP-8623 (which is the selective receive for a newly created ref skips messages in the inbox). Apparently this doesn't appear to apply to messages generated from NIFs. Or at least, not in the code I've got. I did spend some time trawling around erts internals but the few references I found weren't enough for the whole thing to click. I'll paste the sample test code I've been using in case I'm doing something supremely silly. Thanks, Paul Davis https://gist.github.com/3073295 From ok@REDACTED Mon Jul 9 01:39:44 2012 From: ok@REDACTED (Richard O'Keefe) Date: Mon, 9 Jul 2012 11:39:44 +1200 Subject: [erlang-questions] simple question about list memory consumption In-Reply-To: References: <48564154-48ae-4d26-9a06-aa1091360cc0@knuth> Message-ID: On 9/07/2012, at 7:41 AM, CGS wrote: > Hi, > > About the two lists you were speaking about (I suppose you were referring to the output of lists:map/2 and lists:seq/2), the second is destroyed as soon as lists:map/2 exits. No it isn't. It is only reclaimed when the garbage collector runs. > On a 64-bits machine, 2 words = 16 B and 3 words = 24 B, so, my value of 21 B is exactly in between 2 and 3 words. I suppose you worked on 32-bits machine or with half-world emulator and you got half of that value (or a bit less if we are to consider that the shell is adding bigger overhead). No, I used process_info, which reports the size in words, not bytes. From zhengsyao@REDACTED Mon Jul 9 06:00:49 2012 From: zhengsyao@REDACTED (Siyao Zheng) Date: Sun, 8 Jul 2012 21:00:49 -0700 (PDT) Subject: [erlang-questions] About Erlang SMP scheduler In-Reply-To: <4FC877D9.7020004@erlang.org> References: <0F4D9489-7755-4FD2-8831-E4BEE2A8C8F1@gmail.com> <2FF9F129-C96C-48C2-BFE9-223F591D624B@gmail.com> <0773DEE9-E2B7-440A-814E-33C55A472A7C@gmail.com> <4FC877D9.7020004@erlang.org> Message-ID: <2aa71fec-06e3-4a5a-b63e-daa5a22f8934@googlegroups.com> So, as much as I understand, the "balance" in Erlang and in other computation oriented languages such as Cilk are kind of different. The computation processes in Erlang might be serving other processes, so they can't be stacked in one scheduler getting lower effective priority. But the "tasks" in Cilk are working together to complete a whole computation, so the scheduler just needs to keep every core busy to make the computation finishes fast. Am I right? Cheers Siyao On Friday, June 1, 2012 4:05:45 PM UTC+8, Rickard Green wrote: > > On 05/31/2012 01:29 PM, Emilio De Camargo Francesquini wrote: > > Hello, > > > >> check_balance() sets up > >> migration paths and migration limits that are used in order to balance > >> the load between schedulers. Without balancing, processes with the > >> same assigned priority can get very different effective priorities. > > > > I have some questions relating to that. > > > > First, after the decision to rebalance has been made, what are the > > criteria to select which processes from each queue will be migrated? > > The last ones in the queue, the first, at random, ...? > > > > Each priority is balanced separately (while considering the effects > different priorities have on each other). Run-queue lengths on a > specific priority is the most important factor. Within each priority > all processes are considered equal. > > We migrate away processes from an overloaded run queue at (sort of) both > ends. New processes will be redirected instead of enqueued and > underloaded schedulers will pick from the front of the queue. > > > Second, how are these migration paths and limits defined? Does the > > definition of the migration paths take into consideration the > > architecture (SMP, multisocket SMP, NUMA, ...) of the machine? If we > > consider that the lifespan of the majority of the process is quite > > short, and if we are running on a NUMA machine for example, migrating > > processes at random to another numa-node might hurt the performance > > more than letting the execution queues unbalanced. > > > > The primary purpose of the load balancing is to preserve the soft real > time properties. That is, a load balancing decision that hurt > performance isn't necessarily unwanted. We do, of course, not want to > unnecessarily hurt performance. A performance improvement is, of course, > preferred. > > The CPU topology is currently taken into account in a situation where > schedulers run out of work. In this case we try to move work towards > schedulers with low ids. Scheduler bind type will determine the effect > of this compaction of work (for more info see cpu topology > and scheduler bind type > in the documentation erl). > > We have on the todo-list to take the CPU topology into account also in > other load situations. Most likely some kind of hierarchical balancing > strategy based on the CPU topology. However, at the time being there are > other things ahead of this on the todo-list, but at some point this will > be introduced. > > Regards, > Rickard > > > Thanks! > > > > Best Regards > > > > Emilio Francesquini > > > > 2012/5/8 Rickard Green: > >> The work stealing is there to quickly distribute work between > >> schedulers, but it doesn't balance the load. check_balance() sets up > >> migration paths and migration limits that are used in order to balance > >> the load between schedulers. Without balancing, processes with the > >> same assigned priority can get very different effective priorities. > >> > >> Example: With 4 schedulers and 400 cpu bound processes executing the > >> same code, each scheduler will eventually end up with 100 processes to > >> manage. That is, all processes will have the same access to the cpu. > >> If you disable check_balance(), you may end up with three schedulers > >> only having one process each, and one scheduler with 397 processes. > >> These 397 processes will effectively have a much lower priority than > >> the other three processes. > >> > >> Regards, > >> Rickard Green, Erlang/OTP, Ericsson AB > >> > >> 2012/5/2 Siyao Zheng(???): > >>> Hi, > >>> > >>> Yes, that's true. But the "try_steal_task()" afterwards is also > controlled > >>> by ERTS_SMP macro. It doesn't make any sense to talk about workload > balance > >>> outside multi-processor environment. I'm just wondering why > "check_balance" > >>> is needed while there exists work stealing(try_steal_task). > >>> > >>> Cheers > >>> > >>> On May 2, 2012, at 12:04 PM, xu yu wrote: > >>> > >>> Hi, > >>> > >>> "check_balance" controlled by "ifdef ERTS_SMP", so... > >>> > >>> > >>> 2012/4/27 "Siyao Zheng(???)" > >>>> > >>>> Hi, > >>>> > >>>> schedule(), in which check_balance() is only called. > >>>> > >>>> In schedule(), when check_balance_reds reaches zero, scheduler > decides to > >>>> "check_balance". But later in schedule(), when scheduler finds run > queue > >>>> being empty, it will try to steal task from other schedulers (by > calling > >>>> try_steal_task()), which is also a load balance mechanism. I'm just > >>>> wondering what the purpose of check_balance(). > >>>> > >>>> Cheers > >>>> Siyao > >>>> > >>>> On Apr 27, 2012, at 9:40 PM, Zabrane Mickael wrote: > >>>> > >>>>> Hi Siyao, > >>>>> > >>>>> Which "check_balance" you've commented out? > >>>>> > >>>>> Here is what I found so far: > >>>>> > >>>>> $ find /otp_src_R15B01 -type f | xargs grep check_balance | grep -v > >>>>> matches > >>>>> /opt/otp_src_R15B01/erts/emulator/beam/erl_process.c: int > >>>>> forced_check_balance; > >>>>> > >>>>> > /opt/otp_src_R15B01/erts/emulator/beam/erl_process.c:check_balance(ErtsRunQueue > >>>>> *c_rq) > >>>>> /opt/otp_src_R15B01/erts/emulator/beam/erl_process.c:# error > >>>>> check_balance() assumes ERTS_MAX_PROCESS< (1<< 27) > >>>>> /opt/otp_src_R15B01/erts/emulator/beam/erl_process.c: > >>>>> c_rq->check_balance_reds = INT_MAX; > >>>>> /opt/otp_src_R15B01/erts/emulator/beam/erl_process.c: > >>>>> c_rq->check_balance_reds = INT_MAX; > >>>>> /opt/otp_src_R15B01/erts/emulator/beam/erl_process.c: > >>>>> rq->check_balance_reds = ERTS_RUNQ_CALL_CHECK_BALANCE_REDS; > >>>>> /opt/otp_src_R15B01/erts/emulator/beam/erl_process.c: * > >>>>> check_balance() is never called in more threads > >>>>> /opt/otp_src_R15B01/erts/emulator/beam/erl_process.c: forced = > >>>>> balance_info.forced_check_balance; > >>>>> /opt/otp_src_R15B01/erts/emulator/beam/erl_process.c: > >>>>> balance_info.forced_check_balance = 0; > >>>>> /opt/otp_src_R15B01/erts/emulator/beam/erl_process.c: > >>>>> c_rq->check_balance_reds = INT_MAX; > >>>>> /opt/otp_src_R15B01/erts/emulator/beam/erl_process.c: > >>>>> rq->check_balance_reds = INT_MAX; > >>>>> /opt/otp_src_R15B01/erts/emulator/beam/erl_process.c: > >>>>> rq->check_balance_reds = ERTS_RUNQ_CALL_CHECK_BALANCE_REDS; > >>>>> /opt/otp_src_R15B01/erts/emulator/beam/erl_process.c: > >>>>> rq->check_balance_reds = ERTS_RUNQ_CALL_CHECK_BALANCE_REDS; > >>>>> /opt/otp_src_R15B01/erts/emulator/beam/erl_process.c: > >>>>> balance_info.forced_check_balance = 0; > >>>>> /opt/otp_src_R15B01/erts/emulator/beam/erl_process.c: > >>>>> (RQ)->check_balance_reds = ERTS_RUNQ_CALL_CHECK_BALANCE_REDS; > \ > >>>>> /opt/otp_src_R15B01/erts/emulator/beam/erl_process.c: > >>>>> balance_info.forced_check_balance = 1; > >>>>> /opt/otp_src_R15B01/erts/emulator/beam/erl_process.c: > >>>>> ERTS_RUNQ_IX(0)->check_balance_reds = 0; > >>>>> /opt/otp_src_R15B01/erts/emulator/beam/erl_process.c: if > >>>>> (rq->check_balance_reds<= 0) > >>>>> /opt/otp_src_R15B01/erts/emulator/beam/erl_process.c: > >>>>> check_balance(rq); > >>>>> /opt/otp_src_R15B01/erts/emulator/beam/erl_process.h: int > >>>>> check_balance_reds; > >>>>> /opt/otp_src_R15B01/erts/emulator/beam/erl_process.h: > >>>>> (RQ)->check_balance_reds -= (REDS); \ > >>>>> /opt/otp_src_R15B01/erts/emulator/beam/erl_process.h: > >>>>> (RQ)->check_balance_reds -= (REDS); > >>>>> [...] > >>>>> > >>>>> For OTP Team: is this dangerous ? > >>>>> > >>>>> Regards, > >>>>> Zabrane > >>>>> > >>>>> > >>>>> On Apr 27, 2012, at 1:23 PM, Siyao Zheng(???) wrote: > >>>>> > >>>>>> Hi, > >>>>>> > >>>>>> In SMP version of Erlang, every scheduler periodically call > >>>>>> "check_balance()" to check load balance among all schedulers, then > it might > >>>>>> migrate some processes to balance load, or shut down some > schedulers with > >>>>>> low load. Does anyone know why scheduler should do this? > check_balance() is > >>>>>> quite a big one, and it has to lock every run queue when it checks > the run > >>>>>> queue. I think it's quite a big cost here. The work stealing at each > >>>>>> schedule step afterwards actually does work load balance very well. > Actually > >>>>>> after I comment out the check_balance() step and run BigBang and > Hackbench > >>>>>> benchmarks, they are really faster! > >>>>>> > >>>>>> So, I wonder what is the purpose of check_balance() step here, if > it is > >>>>>> related to scalability on many core processors or something, and if > there is > >>>>>> any other benchmark I can run to prove its usefulness. > >>>>>> > >>>>>> Cheers! > >>>>>> > >>>>>> Siyao > >>>>>> _______________________________________________ > >>>>>> erlang-questions mailing list > >>>>>> erlang-questions@REDACTED > >>>>>> http://erlang.org/mailman/listinfo/erlang-questions > >>>>> > >>>>> > >>>> > >>>> _______________________________________________ > >>>> erlang-questions mailing list > >>>> erlang-questions@REDACTED > >>>> http://erlang.org/mailman/listinfo/erlang-questions > >>> > >>> > >>> > >>> > >>> _______________________________________________ > >>> erlang-questions mailing list > >>> erlang-questions@REDACTED > >>> http://erlang.org/mailman/listinfo/erlang-questions > >>> > >> _______________________________________________ > >> erlang-questions mailing list > >> erlang-questions@REDACTED > >> http://erlang.org/mailman/listinfo/erlang-questions > > > -- > Rickard Green, Erlang/OTP, Ericsson AB. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cyg.cao@REDACTED Mon Jul 9 11:01:12 2012 From: cyg.cao@REDACTED (Wei Cao) Date: Mon, 9 Jul 2012 17:01:12 +0800 Subject: [erlang-questions] Improve performance of IO bounded server written in Erlang via having pollset for each scheduler and bind port to scheduler together with process Message-ID: Hi, all We wrote a proxy server in Erlang, the proxy's logic is rather simple, it listens on some TCP port, establishes new connection from user client, forward packets back and forth between the client and backend server after authentication until connection is closed. It's very easy to write such a proxy in Erlang, fork a process for each new user connection and connect to the backend server in the same process, the process works like a pipe, sockets from both side is set to the active once mode, whenever a tcp packet is received from one socket, the packet will be sent to other socket. (A simplified version of proxy code is attached at the end of the mail) However, the performance is not quite satisfying, the proxy can handle maximum only 40k requests on our 16 core machine(Intel Xeon L5630, 2.13GHz) with heavy stress(100 concurrent clients). We then analyzed the behavior of beam.smp use tools like tcprstat, mpstat, perf, and SystemTap. tcprstat shows QPS is about 40k, have average 1.7 millisecond latency for each request. timestamp count max min avg med stddev 95_max 95_avg 95_std 99_max 99_avg 99_std 1341813326 39416 17873 953 1718 1519 724 2919 1609 340 3813 1674 462 1341813327 40528 9275 884 1645 1493 491 2777 1559 290 3508 1619 409 1341813328 40009 18105 925 1694 1507 714 2868 1586 328 3753 1650 450 mpstat shows 30% CPU is idle, 03:30:19 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle 03:30:20 PM all 38.69 0.00 21.92 0.00 0.06 7.52 0.00 0.00 31.80 03:30:21 PM all 37.56 0.00 21.99 0.00 0.00 7.50 0.00 0.00 32.95 and perf top shows, much time is wasted in scheduler_wait, in spin wait I guess. 9320.00 19.8% scheduler_wait /home/mingsong.cw/erlangr16b/lib/erlang/erts-5.10/bin/beam.smp 1813.00 3.9% process_main /home/mingsong.cw/erlangr16b/lib/erlang/erts-5.10/bin/beam.smp 1379.00 2.9% _spin_lock /usr/lib/debug/lib/modules/2.6.32-131.21.1.tb477.el6.x86_64/vmlinux 1201.00 2.6% schedule /home/mingsong.cw/erlangr16b/lib/erlang/erts-5.10/bin/beam.smp It seems the performance may be associated with scheduler_wait() and erts_check_io(), with a SystemTap script(attached at the end of this mail), we can find out how many times the system call epoll_wait is invoked by beam.smp and each time, how many revents it gets. cpu process times revents min max avg timeouts all 1754 128042 - - 73 3 [14] beam.smp 151 14127 82 97 93 0 [ 5] beam.smp 142 13291 83 97 93 0 [13] beam.smp 127 11948 86 96 94 0 [ 6] beam.smp 127 11836 81 96 93 0 [ 4] beam.smp 121 11323 81 96 93 0 [15] beam.smp 117 10935 83 96 93 0 [12] beam.smp 486 10128 0 96 20 2 [ 1] beam.smp 71 6549 71 100 92 0 [ 2] beam.smp 62 5695 82 96 91 0 [ 7] beam.smp 55 5102 81 95 92 0 [11] beam.smp 52 4822 85 95 92 0 [ 9] beam.smp 52 4799 85 95 92 0 [ 8] beam.smp 51 4680 78 95 91 0 [10] beam.smp 49 4508 85 97 92 0 [ 3] beam.smp 46 4211 81 95 91 0 [ 0] beam.smp 44 4088 83 95 92 0 The resuls shows, epoll_wait is invoked 1754 times each second, and get 73 io events in average. This is unacceptable for writing high performance server. Because if epoll_wait is invoked no more than 2k times per second, then read/write a packet would cost more than 500ms, which causes long delay and affects the throughput finally. The problem relies on there is only one global pollset in system wide, so at a time there is no more than one scheduler can call erts_check_io() to obtain pending io tasks from underlying pollset, and no scheduler can call erts_check_io() before all pending io tasks're processed, so for IO bounded application, it's very likely that a scheduler finish its own job, but must wait idly for other schedulers to complete theirs. Hence, we develops a patch to slove this problem, by having a pollset for each scheduler, so that each scheduler can invoke erts_check_io() on its own pollset concurrently. After a scheduler complete its tasks, it can invoke erts_check_io() immediately no matter what state other schedulers're in. This patch also handles port migration situation, all used file descriptors in each port're recorded, when a port is migrated, these fd 're removed from original scheduler's pollset, and added to new scheduler's. Bind port to scheduler together with process is also helpful to performance, it reduces the cost of thread switches and synchronization, and bound port won't be migrated between schedulers. After apply the two patches, with the same pressure(100 concurrent clients),epoll_wait is invoked 49332 times per second, and get 3 revents each time in average, that is to say, our server responds quicker and become more realtime. cpu process times revents min max avg timeouts all 49332 217186 - - 4 3 [ 2] beam.smp 3219 16050 2 7 4 0 [11] beam.smp 4275 16033 1 6 3 0 [ 8] beam.smp 4240 15992 1 6 3 0 [ 9] beam.smp 4316 15964 0 6 3 2 [10] beam.smp 4139 15851 1 6 3 0 [ 3] beam.smp 4256 15816 1 6 3 0 [ 1] beam.smp 3107 15800 2 7 5 0 [ 0] beam.smp 3727 15259 1 6 4 0 [ 7] beam.smp 2810 14722 3 7 5 0 [13] beam.smp 1981 11499 4 7 5 0 [15] beam.smp 2285 10942 3 6 4 0 [14] beam.smp 2258 10866 3 6 4 0 [ 4] beam.smp 2246 10849 3 6 4 0 [ 6] beam.smp 2206 10730 3 6 4 0 [12] beam.smp 2173 10573 3 6 4 0 [ 5] beam.smp 2093 10240 3 6 4 0 scheduler_wait no longer take so much time now, 169.00 6.2% process_main beam.smp 55.00 2.0% _spin_lock [kernel] 45.00 1.7% driver_deliver_term beam.smp so is idle CPU time 04:30:44 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle 04:30:45 PM all 60.34 0.00 21.44 0.00 0.06 16.45 0.00 0.00 1.71 04:30:46 PM all 60.99 0.00 21.22 0.00 0.00 16.26 0.00 0.00 1.52 and tcprstat shows, QPS is getting 100K, latency is less than 1 millisecond timestamp count max min avg med stddev 95_max 95_avg 95_std 99_max 99_avg 99_std 1341822689 96078 11592 314 910 817 311 1447 869 228 1777 897 263 1341822690 100651 24245 209 914 819 381 1458 870 229 1800 899 263 I also write a extreamly simple keep-alive http server(attached at the end of mail), to compare performance before and after applying the patches, mearused with apache ab tool(ab -n 1000000 -c 100 -k ), a 30% performance gain can be get. before Requests per second: 103671.70 [#/sec] (mean) Time per request: 0.965 [ms] (mean) after Requests per second: 133701.24 [#/sec] (mean) Time per request: 0.748 [ms] (mean) Patches can be found at github, compile with ./configure CFLAGS=-DERTS_POLLSET_PER_SCHEDULER Pollset per scheduler: git fetch git://github.com/weicao/otp.git pollset_per_scheduler https://github.com/weicao/otp/compare/weicao:master...weicao:pollset_per_scheduler https://github.com/weicao/otp/compare/weicao:master...weicao:pollset_per_scheduler.patch Bind port to scheduler: git fetch git://github.com/weicao/otp.git bind_port_to_scheduler https://github.com/weicao/otp/compare/weicao:pollset_per_scheduler...weicao:bind_port_to_scheduler https://github.com/weicao/otp/compare/weicao:pollset_per_scheduler...weicao:bind_port_to_scheduler.patch Appendix: ----------------------------------- proxy.erl ------------------------------------ -module(proxy). -compile(export_all). -define(RECBUF_SIZE, 8192). -define(ACCEPT_TIMEOUT, 2000). start([MyPortAtom, DestIpAtom, DestPortAtom]) -> {MyPort, []} = string:to_integer(atom_to_list(MyPortAtom)), DestIp = atom_to_list(DestIpAtom), {DestPort, []} = string:to_integer(atom_to_list(DestPortAtom)), listen(MyPort, DestIp, DestPort), receive Any -> io:format("recv ~p~n", [Any]) end. listen(MyPort, DestIp, DestPort) -> io:format("start proxy on [local] 0.0.0.0:~p -> [remote] ~p:~p~n", [MyPort, DestIp, DestPort]), case gen_tcp:listen(MyPort, [inet, {ip, {0,0,0,0}}, binary, {reuseaddr, true}, {recbuf, ?RECBUF_SIZE}, {active, false}, {nodelay, true} ]) of {ok, Listen} -> N = erlang:system_info(schedulers), lists:foreach(fun(I) -> accept(Listen, DestIp, DestPort, I) end, lists:seq(1,N)); {error, Reason} -> io:format("error listen ~p~n", [Reason]) end. accept(Listen, DestIp, DestPort, I) -> spawn_opt(?MODULE, loop, [Listen, DestIp, DestPort, I], [{scheduler, I}]). loop(Listen, DestIp, DestPort, I) -> case gen_tcp:accept(Listen, ?ACCEPT_TIMEOUT) of {ok, S1} -> accept(Listen, DestIp, DestPort, I), case catch gen_tcp:connect(DestIp, DestPort, [inet, binary, {active, false}, {reuseaddr, true}, {nodelay, true}]) of {ok, S2} -> io:format("new connection~n"), loop1(S1, S2); {error, Reason} -> io:format("error connect ~p~n", [Reason]) end; {error, timeout} -> loop(Listen, DestIp, DestPort, I); Error -> io:format("error accept ~p~n", [Error]), accept(Listen, DestIp, DestPort, I) end. loop1(S1, S2) -> active(S1, S2), receive {tcp, S1, Data} -> gen_tcp:send(S2, Data), loop1(S1, S2); {tcp, S2, Data} -> gen_tcp:send(S1, Data), loop1(S1, S2); {tcp_closed, S1} -> io:format("S1 close~n"), gen_tcp:close(S1), gen_tcp:close(S2); {tcp_closed, S2} -> io:format("S2 close~n"), gen_tcp:close(S1), gen_tcp:close(S2) end. active(S1,S2)-> inet:setopts(S1, [{active, once}]), inet:setopts(S2, [{active, once}]). ----------------------------------- epollwait.stp ------------------------------------ #! /usr/bin/env stap # # global epoll_timeout_flag, epoll_count, epoll_min, epoll_max, epoll_times, epoll_timeouts probe syscall.epoll_wait { if(timeout > 0) { epoll_timeout_flag[pid()] = 1 } } probe syscall.epoll_wait.return { c = cpu() p = execname() epoll_times[c,p] ++ epoll_count[c,p] += $return if($return == 0 && pid() in epoll_timeout_flag) { epoll_timeouts[c,p] ++ delete epoll_timeout_flag[pid()] } if(!([c, p] in epoll_min)) { epoll_min[c,p] = $return } else if($return < epoll_min[c,p]) { epoll_min[c,p] = $return } if($return > epoll_max[c,p]) { epoll_max[c,p] = $return } } probe timer.s($1) { printf ("%4s %45s %10s %10s %10s %10s %10s %10s\n", "cpu", "process", "times", "revents", "min", "max", "avg", "timeouts" ) foreach ([cpu, process] in epoll_count- limit 20) { all_epoll_times += epoll_times[cpu,process] all_epoll_count += epoll_count[cpu,process] all_epoll_timeouts += epoll_timeouts[cpu,process] } printf ("%4s %45s %10d %10d %10s %10s %10d %10d\n", "all", "", all_epoll_times, all_epoll_count, "-", "-", all_epoll_count == 0? 0:all_epoll_count/all_epoll_times, all_epoll_timeouts) foreach ([cpu, process] in epoll_count- limit 20) { printf ("[%2d] %45s %10d %10d %10d %10d %10d %10d\n", cpu, process, epoll_times[cpu, process], epoll_count[cpu, process], epoll_min[cpu, process], epoll_max[cpu, process], epoll_count[cpu,process]/epoll_times[cpu,process], epoll_timeouts[cpu,process]) } delete epoll_count delete epoll_min delete epoll_max delete epoll_times delete epoll_timeouts printf ("--------------------------------------------------------------------------\n\n" ) } ------------------------------------------------ ehttpd.erl ------------------------------------------------- -module(ehttpd). -compile(export_all). start() -> start(8888). start(Port) -> N = erlang:system_info(schedulers), listen(Port, N), io:format("ehttpd ready with ~b schedulers on port ~b~n", [N, Port]), register(?MODULE, self()), receive Any -> io:format("~p~n", [Any]) end. %% to stop: ehttpd!stop. listen(Port, N) -> Opts = [inet, binary, {active, false}, {recbuf, 8192}, {nodelay,true}, {reuseaddr, true}], {ok, S} = gen_tcp:listen(Port, Opts), lists:foreach(fun(I)-> spawn_opt(?MODULE, accept, [S, I], [{scheduler, I}]) end, lists:seq(1, N)). accept(S, I) -> case gen_tcp:accept(S) of {ok, Socket} -> spawn_opt(?MODULE, accept, [S, I],[{scheduler,I}] ), io:format("new connection @~p~n", [I]), loop(Socket,<<>>); Error -> erlang:error(Error) end. loop(S,B) -> inet:setopts(S, [{active, once}]), receive {tcp, S, Data} -> B1 = <>, case binary:part(B1,{byte_size(B1), -4}) of <<"\r\n\r\n">> -> Response = <<"HTTP/1.1 200 OK\r\nContent-Length: 12\r\nConnection: Keep-Alive\r\n\r\nhello world!">>, gen_tcp:send(S, Response), loop(S, <<>>); _ -> loop(S, B1) end; {tcp_closed, S} -> io:format("connection closed forced~n"), gen_tcp:close(S); Error -> io:format("unexpected message~p~n", [Error]), Error end. -- Best, Wei Cao From mrtndimitrov@REDACTED Mon Jul 9 11:13:07 2012 From: mrtndimitrov@REDACTED (Martin Dimitrov) Date: Mon, 09 Jul 2012 12:13:07 +0300 Subject: [erlang-questions] Erlang based graph database Message-ID: <4FFAA0A3.7000200@gmail.com> Hello, Can you share your experience with Phoebus (https://github.com/xslogic/phoebus/tree/master/src)** or any other graph database written in Erlang? Thanks, Martin From sverker.eriksson@REDACTED Mon Jul 9 11:30:47 2012 From: sverker.eriksson@REDACTED (Sverker Eriksson) Date: Mon, 9 Jul 2012 11:30:47 +0200 Subject: [erlang-questions] Selective receive optimization for NIFs In-Reply-To: References: Message-ID: <4FFAA4C7.4040800@erix.ericsson.se> Hi Paul The OTP-8623 selective receive optimization was achieved partly with static code analysis done by the Erlang compiler. In your case the NIF is creating the ref. To do the same optimization for that, would need some sort of dynamic analysis done in runtime. /Sverker, Erlang/OTP Paul Davis wrote: > I've got a bit of a curious question in relation to OTP-8623 (which is > the selective receive for a newly created ref skips messages in the > inbox). Apparently this doesn't appear to apply to messages generated > from NIFs. Or at least, not in the code I've got. > > I did spend some time trawling around erts internals but the few > references I found weren't enough for the whole thing to click. I'll > paste the sample test code I've been using in case I'm doing something > supremely silly. > > Thanks, > Paul Davis > > https://gist.github.com/3073295 > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > From jesper.louis.andersen@REDACTED Mon Jul 9 11:34:14 2012 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Mon, 9 Jul 2012 11:34:14 +0200 Subject: [erlang-questions] About Erlang SMP scheduler In-Reply-To: <2aa71fec-06e3-4a5a-b63e-daa5a22f8934@googlegroups.com> References: <0F4D9489-7755-4FD2-8831-E4BEE2A8C8F1@gmail.com> <2FF9F129-C96C-48C2-BFE9-223F591D624B@gmail.com> <0773DEE9-E2B7-440A-814E-33C55A472A7C@gmail.com> <4FC877D9.7020004@erlang.org> <2aa71fec-06e3-4a5a-b63e-daa5a22f8934@googlegroups.com> Message-ID: <68C533CB-F5A7-4327-96C2-DEA26C2E1ED4@erlang-solutions.com> On Jul 9, 2012, at 6:00 AM, Siyao Zheng wrote: > So, as much as I understand, the "balance" in Erlang and in other computation oriented languages such as Cilk are kind of different. The computation processes in Erlang might be serving other processes, so they can't be stacked in one scheduler getting lower effective priority. But the "tasks" in Cilk are working together to complete a whole computation, so the scheduler just needs to keep every core busy to make the computation finishes fast. Spot on. The two systems have different goals. So naturally, their implementations are different. From essen@REDACTED Mon Jul 9 11:56:58 2012 From: essen@REDACTED (=?ISO-8859-1?Q?Lo=EFc_Hoguin?=) Date: Mon, 09 Jul 2012 11:56:58 +0200 Subject: [erlang-questions] Looking for slides of a lightning talk Message-ID: <4FFAAAEA.9060605@ninenines.eu> Someone did a lightning talk in Stockholm that was quite funny but also quite true, saying that Erlang isn't ready for the web and doing comparisons with other languages/platforms. Any chance I could get the slides? Thanks. -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From cgsmcmlxxv@REDACTED Mon Jul 9 12:34:17 2012 From: cgsmcmlxxv@REDACTED (CGS) Date: Mon, 9 Jul 2012 12:34:17 +0200 Subject: [erlang-questions] simple question about list memory consumption In-Reply-To: References: <48564154-48ae-4d26-9a06-aa1091360cc0@knuth> Message-ID: On Mon, Jul 9, 2012 at 1:39 AM, Richard O'Keefe wrote: > > On 9/07/2012, at 7:41 AM, CGS wrote: > > > Hi, > > > > About the two lists you were speaking about (I suppose you were > referring to the output of lists:map/2 and lists:seq/2), the second is > destroyed as soon as lists:map/2 exits. > > No it isn't. It is only reclaimed when the garbage collector runs. > That is a bit puzzling. Looking at the code for lists:map/2: map(F, [H|T]) -> [F(H)|map(F, T)]; map(F, []) when is_function(F, 1) -> []. I can see that when lists:map/2 exits, the only remaining list is an empty list. I might be wrong, but I see only one word remaining for the garbage collector to take care of. That means, at the end, there is only 1 full list remaining (1 word really can be neglected compared with 214 MB which I got for 10^7 characters long list). > > > On a 64-bits machine, 2 words = 16 B and 3 words = 24 B, so, my value of > 21 B is exactly in between 2 and 3 words. I suppose you worked on 32-bits > machine or with half-world emulator and you got half of that value (or a > bit less if we are to consider that the shell is adding bigger overhead). > > No, I used process_info, which reports the size in words, not bytes. > You got the result from process_info, I got it from `free' under Linux. In the end, I don't suppose it matters too much how you or I got the info, but it matters that the results are compatible and, moreover, they are compatible with the official documentation, don't you think? The results are consistent with 2 words per character in the list. The remaining is an artifact from shell and/or memory management (which can come from lower levels as well). My main question here was because I forgot about the list memory management in Erlang (as described in the efficiency guide) and I was wondering about 1 B vs. 21 B per character. As I said, I knew I was missing something and I was looking for someone to help me remember. :) Thank you all for investing time in answering my question. CGS -------------- next part -------------- An HTML attachment was scrubbed... URL: From ulf@REDACTED Mon Jul 9 14:13:46 2012 From: ulf@REDACTED (Ulf Wiger) Date: Mon, 9 Jul 2012 14:13:46 +0200 Subject: [erlang-questions] simple question about list memory consumption In-Reply-To: References: <48564154-48ae-4d26-9a06-aa1091360cc0@knuth> Message-ID: <27DC8581-2D1F-4ABB-B14F-7A309843A6E0@feuerlabs.com> On 9 Jul 2012, at 12:34, CGS wrote: > > About the two lists you were speaking about (I suppose you were referring to the output of lists:map/2 and lists:seq/2), the second is destroyed as soon as lists:map/2 exits. > > No it isn't. It is only reclaimed when the garbage collector runs. > > That is a bit puzzling. Looking at the code for lists:map/2: > > map(F, [H|T]) -> > [F(H)|map(F, T)]; > map(F, []) when is_function(F, 1) -> []. > > I can see that when lists:map/2 exits, the only remaining list is an empty list. I might be wrong, but I see only one word remaining for the garbage collector to take care of. That means, at the end, there is only 1 full list remaining (1 word really can be neglected compared with 214 MB which I got for 10^7 characters long list). I believe Richard was referring to: a) the list created using lists:seq(1,10000000) b) the list that results from the call to lists:map/2 Actually, when you do this in the shell, the second list will be remembered in the command history (as well as in the variable bindings). The first list will, as Richard said, disappear when the garbage collector runs. The quickest and most precise way of finding out how much RAM is needed for a particular data structure is: Eshell V5.9 (abort with ^G) 1> L = lists:map(fun(_) -> 107 end,lists:seq(1,10000000)), ok. ok 2> erts_debug:flat_size(L). 20000000 In this case, 20M words, which is very much in line with the documentation. If you want to figure out how much memory is actually used by the VM, this is more complicated. This will depend on how the data structure was built - i.e. how it exercised the garbage collector - and how much temporary fragmentation that caused. The Erlang VM is not designed to keep the memory footprint as small as possible, but, in a sense, to maintain a reasonably low level of fragmentation in a non-stop system without ruining the soft-real-time characteristics. If you perform an operation such as creating a list with 10M elements, and from that creating another list of similar size, this can be pretty hard on Erlang's garbage collector, since it can't predict when you are going to quit accumulating data. Let's try a very unscientific experiment: 3> exit(foo). ** exception exit: foo 4> timer:tc(fun() -> lists:map(fun(_) -> 107 end,lists:seq(1,10000000)), ok end). {10025590,ok} %% create a fun that evaluates F() in a fresh process with a really big heap 5> RPC = fun(F) -> {_,R} = spawn_opt(fun() -> exit({ok,F()}) end,[{min_heap_size,50000000},monitor]), receive {'DOWN',R,_,_,Res} -> {ok,Return} = Res, Return end end. #Fun 6> exit(foo). ** exception exit: foo 7> timer:tc(fun() -> RPC(fun() -> lists:map(fun(_) -> 107 end,lists:seq(1,10000000)), ok end) end). {5378929,ok} (I did, in fact, run the timing operations several times and kept the best result.) Of course, by creating a slave process, we add some processing overhead, compared to calling the function directly, but in this case it's insignificant, due to the long execution time. We almost reduce the time by half by eliminating a lot of expensive GC. (If we actually return the big list from the slave process, this increases the cost a bit, but not by nearly as much, since the VM knows how much it needs to grow the heap in order to make room for the result.) BR, Ulf W Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. http://feuerlabs.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.joseph.davis@REDACTED Mon Jul 9 15:52:07 2012 From: paul.joseph.davis@REDACTED (Paul Davis) Date: Mon, 9 Jul 2012 08:52:07 -0500 Subject: [erlang-questions] Selective receive optimization for NIFs In-Reply-To: <4FFAA4C7.4040800@erix.ericsson.se> References: <4FFAA4C7.4040800@erix.ericsson.se> Message-ID: Sverker, Ah! That would explain a lot of my confusion. I spent a lot of time reading through code thinking the magic was that refs were tagged with when they were created or had some special flag when they got copied out of a process. If I have time later this week I'll take some time to start reading through decompiled beam to try and get a better understanding and see what I can come up with. Thanks, Paul On Mon, Jul 9, 2012 at 4:30 AM, Sverker Eriksson wrote: > Hi Paul > > The OTP-8623 selective receive optimization was achieved partly with static > code analysis done by the Erlang compiler. > > In your case the NIF is creating the ref. To do the same optimization for > that, would need some sort of dynamic analysis done in runtime. > > > /Sverker, Erlang/OTP > > > > Paul Davis wrote: >> >> I've got a bit of a curious question in relation to OTP-8623 (which is >> the selective receive for a newly created ref skips messages in the >> inbox). Apparently this doesn't appear to apply to messages generated >> from NIFs. Or at least, not in the code I've got. >> >> I did spend some time trawling around erts internals but the few >> references I found weren't enough for the whole thing to click. I'll >> paste the sample test code I've been using in case I'm doing something >> supremely silly. >> >> Thanks, >> Paul Davis >> >> https://gist.github.com/3073295 >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > > From cgsmcmlxxv@REDACTED Mon Jul 9 16:12:21 2012 From: cgsmcmlxxv@REDACTED (CGS) Date: Mon, 9 Jul 2012 16:12:21 +0200 Subject: [erlang-questions] simple question about list memory consumption In-Reply-To: <27DC8581-2D1F-4ABB-B14F-7A309843A6E0@feuerlabs.com> References: <48564154-48ae-4d26-9a06-aa1091360cc0@knuth> <27DC8581-2D1F-4ABB-B14F-7A309843A6E0@feuerlabs.com> Message-ID: On Mon, Jul 9, 2012 at 2:13 PM, Ulf Wiger wrote: > > On 9 Jul 2012, at 12:34, CGS wrote: > > > About the two lists you were speaking about (I suppose you were >> referring to the output of lists:map/2 and lists:seq/2), the second is >> destroyed as soon as lists:map/2 exits. >> >> No it isn't. It is only reclaimed when the garbage collector runs. >> > > That is a bit puzzling. Looking at the code for lists:map/2: > > map(F, [H|T]) -> > [F(H)|map(F, T)]; > map(F, []) when is_function(F, 1) -> []. > > I can see that when lists:map/2 exits, the only remaining list is an empty > list. I might be wrong, but I see only one word remaining for the garbage > collector to take care of. That means, at the end, there is only 1 full > list remaining (1 word really can be neglected compared with 214 MB which I > got for 10^7 characters long list). > > > I believe Richard was referring to: > > a) the list created using lists:seq(1,10000000) > b) the list that results from the call to lists:map/2 > > Actually, when you do this in the shell, the second list will be > remembered in the command history (as well as in the variable bindings). > The first list will, as Richard said, disappear when the garbage collector > runs. > Agree. Only that I have the impression that the first list is an empty list when lists:map/2 exits. As I have no way to stabilize my OS RAM consumption at the level of bytes, I don't really care about few words error and therefore I went for 10^6 - 10^7 characters list. The fact that I got in between 2 and 3 words per character supports my first impression. I might be wrong though. > > The quickest and most precise way of finding out how much RAM is needed > for a particular data structure is: > ...theoretically... > > Eshell V5.9 (abort with ^G) > 1> L = lists:map(fun(_) -> 107 end,lists:seq(1,10000000)), ok. > ok > 2> erts_debug:flat_size(L). > 20000000 > I am a bit skeptical to use erts_debug:flat_size/1 as it is using the internal representation to compute the size and it doesn't report the actual physical memory consumption (see lib/hipe/cerl/erl_bif_types.erl and the list type definition in erl_types.erl from the same path), while `free' is providing the RAM usage as it is. Using erts_debug:flat_size(List) is quite the same with length(List)*(2 words) because for each list element ([Type,Term]) it adds 2 words (2*t_integer()) and not checking the actual memory consumption. > In this case, 20M words, which is very much in line with the documentation. > Obviously. :) > > If you want to figure out how much memory is actually used by the VM, this > is more complicated. This will depend on how the data structure was built - > i.e. how it exercised the garbage collector - and how much temporary > fragmentation that caused. The Erlang VM is not designed to keep the memory > footprint as small as possible, but, in a sense, to maintain a reasonably > low level of fragmentation in a non-stop system without ruining the > soft-real-time characteristics. If you perform an operation such as > creating a list with 10M elements, and from that creating another list of > similar size, this can be pretty hard on Erlang's garbage collector, since > it can't predict when you are going to quit accumulating data. > That is what I was referring to lower level memory management. > > Let's try a very unscientific experiment: > > 3> exit(foo). > ** exception exit: foo > 4> timer:tc(fun() -> lists:map(fun(_) -> 107 end,lists:seq(1,10000000)), > ok end). > {10025590,ok} > > %% create a fun that evaluates F() in a fresh process with a really big > heap > 5> RPC = fun(F) -> {_,R} = spawn_opt(fun() -> exit({ok,F()}) > end,[{min_heap_size,50000000},monitor]), receive {'DOWN',R,_,_,Res} -> > {ok,Return} = Res, Return end end. > #Fun > 6> exit(foo). > ** exception exit: foo > 7> timer:tc(fun() -> RPC(fun() -> lists:map(fun(_) -> 107 > end,lists:seq(1,10000000)), ok end) end). > {5378929,ok} > > (I did, in fact, run the timing operations several times and kept the best > result.) > Yeah, it's not easy to get the accurate time of execution. I usually perform the execution several times and make a weighted average value (by eliminating the extrema exceeding a certain error) to get an idea about the execution time. > > Of course, by creating a slave process, we add some processing overhead, > compared to calling the function directly, but in this case it's > insignificant, due to the long execution time. We almost reduce the time by > half by eliminating a lot of expensive GC. > > (If we actually return the big list from the slave process, this increases > the cost a bit, but not by nearly as much, since the VM knows how much it > needs to grow the heap in order to make room for the result.) > Agree. CGS -------------- next part -------------- An HTML attachment was scrubbed... URL: From f@REDACTED Mon Jul 9 16:24:01 2012 From: f@REDACTED (Francesco Mazzoli) Date: Mon, 09 Jul 2012 15:24:01 +0100 Subject: [erlang-questions] Removing mnesia cluster node while mnesia is offline In-Reply-To: References: <878vf6chx0.wl%f@mazzo.li> <87fw995bzc.wl%f@mazzo.li> <698B603A-C83F-4726-B4EA-FE215D997987@me.com> <87d34d59h9.wl%f@mazzo.li> Message-ID: <87k3yd2eta.wl%f@mazzo.li> At Fri, 06 Jul 2012 09:49:29 +0100, Rudolph van Graan wrote: > This scenario is impossible. I will rewrite it as follows, depending on your > definition of "remove". One correct interpretation would be: If I say "C gets removed" I mean that it goes offline and some other nodes invokes `del_table_copy(schema, C)'. Sorry about the lack of clarity. > 1. A,B & C are in the cluster > 2. A goes down > 3. C is turned off and the schema is destroyed on C (instead of your term "removed") > 4. So B has [A,B,C] in the schema (and so does A) With my definition of remove, B knows has [A,B], while A has [A,B,C]. > 5. A is started again. > 6. A copies the data from B because it knows it was down and B has the latest > data Now I don't want to test it (I'm kind of sick of mnesia testing :P) but iirc when A has [A,B,C] and B has [A,B], when A gets started B will get C has well, which is what I wanted to avoid. -- Francesco * Often in error, never in doubt From paul.joseph.davis@REDACTED Mon Jul 9 21:07:24 2012 From: paul.joseph.davis@REDACTED (Paul Davis) Date: Mon, 9 Jul 2012 14:07:24 -0500 Subject: [erlang-questions] Selective receive optimization for NIFs In-Reply-To: References: <4FFAA4C7.4040800@erix.ericsson.se> Message-ID: On Mon, Jul 9, 2012 at 9:05 AM, Dave Smith wrote: > In the past, I think I solved this problem on a NIF by just passing > the ref in. It's a bit more of a pain, but should do the trick. > > D. > That was the method I tried first and it didn't have any effect. I'll spend some time reading the generated beam code for both versions to see if I can't figure out a way around it. > On Mon, Jul 9, 2012 at 7:52 AM, Paul Davis wrote: >> Sverker, >> >> Ah! That would explain a lot of my confusion. I spent a lot of time >> reading through code thinking the magic was that refs were tagged with >> when they were created or had some special flag when they got copied >> out of a process. If I have time later this week I'll take some time >> to start reading through decompiled beam to try and get a better >> understanding and see what I can come up with. >> >> Thanks, >> Paul >> >> On Mon, Jul 9, 2012 at 4:30 AM, Sverker Eriksson >> wrote: >>> Hi Paul >>> >>> The OTP-8623 selective receive optimization was achieved partly with static >>> code analysis done by the Erlang compiler. >>> >>> In your case the NIF is creating the ref. To do the same optimization for >>> that, would need some sort of dynamic analysis done in runtime. >>> >>> >>> /Sverker, Erlang/OTP >>> >>> >>> >>> Paul Davis wrote: >>>> >>>> I've got a bit of a curious question in relation to OTP-8623 (which is >>>> the selective receive for a newly created ref skips messages in the >>>> inbox). Apparently this doesn't appear to apply to messages generated >>>> from NIFs. Or at least, not in the code I've got. >>>> >>>> I did spend some time trawling around erts internals but the few >>>> references I found weren't enough for the whole thing to click. I'll >>>> paste the sample test code I've been using in case I'm doing something >>>> supremely silly. >>>> >>>> Thanks, >>>> Paul Davis >>>> >>>> https://gist.github.com/3073295 >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> >>>> >>> >>> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions From daniel.goertzen@REDACTED Mon Jul 9 21:59:40 2012 From: daniel.goertzen@REDACTED (Daniel Goertzen) Date: Mon, 9 Jul 2012 14:59:40 -0500 Subject: [erlang-questions] specs for fixed length lists In-Reply-To: References: Message-ID: On Sun, Jul 8, 2012 at 1:22 AM, wrote: > I'm with /Joe on this one. There aren't all _that_ many > list operations that make sense on tuples, and > map4(F, {A,B,C,D}) -> {F(A), F(B), F(C), F(D)}. > is easy enough to define. Perhaps the code needs turning > inside out. > > Thanks for the idea; my functional instincts are still developing. I will go with this approach since my sequence size really is only 4. Dan. -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas@REDACTED Mon Jul 9 22:54:07 2012 From: lukas@REDACTED (Lukas Larsson) Date: Mon, 9 Jul 2012 22:54:07 +0200 Subject: [erlang-questions] specs for fixed length lists In-Reply-To: References: Message-ID: Writing a generic tuple:map/2 would not be that hard either: map(F, Tuple) -> map(F, Tuple, size(Tuple)). map(F,Tuple,1) -> setelement(1,Tuple,F(element(1,Tuple))); map(F,Tuple,N) -> map(F,setelement(N,Tuple,F(element(N,Tuple))),N-1). It's not very pretty, or efficient, but it gets the job done. Lukas On Mon, Jul 9, 2012 at 9:59 PM, Daniel Goertzen wrote: > > On Sun, Jul 8, 2012 at 1:22 AM, wrote: >> >> I'm with /Joe on this one. There aren't all _that_ many >> list operations that make sense on tuples, and >> map4(F, {A,B,C,D}) -> {F(A), F(B), F(C), F(D)}. >> is easy enough to define. Perhaps the code needs turning >> inside out. >> > > Thanks for the idea; my functional instincts are still developing. I will > go with this approach since my sequence size really is only 4. > > Dan. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From erlang@REDACTED Mon Jul 9 23:54:11 2012 From: erlang@REDACTED (Joe Armstrong) Date: Mon, 9 Jul 2012 23:54:11 +0200 Subject: [erlang-questions] specs for fixed length lists In-Reply-To: References: Message-ID: I find this very strange it violates the principle of least astonishment. I expect the elements of a list to have the same type, but I expect the elements of a tuple to have different types, which is why map is defined over lists and not tuples. And yes you can do it, but that it not to say that you should do it. /Joe On Mon, Jul 9, 2012 at 10:54 PM, Lukas Larsson wrote: > Writing a generic tuple:map/2 would not be that hard either: > > map(F, Tuple) -> > map(F, Tuple, size(Tuple)). > > map(F,Tuple,1) -> > setelement(1,Tuple,F(element(1,Tuple))); > map(F,Tuple,N) -> > map(F,setelement(N,Tuple,F(element(N,Tuple))),N-1). > > It's not very pretty, or efficient, but it gets the job done. > > Lukas > > On Mon, Jul 9, 2012 at 9:59 PM, Daniel Goertzen > wrote: >> >> On Sun, Jul 8, 2012 at 1:22 AM, wrote: >>> >>> I'm with /Joe on this one. There aren't all _that_ many >>> list operations that make sense on tuples, and >>> map4(F, {A,B,C,D}) -> {F(A), F(B), F(C), F(D)}. >>> is easy enough to define. Perhaps the code needs turning >>> inside out. >>> >> >> Thanks for the idea; my functional instincts are still developing. I will >> go with this approach since my sequence size really is only 4. >> >> Dan. >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From zabrane3@REDACTED Tue Jul 10 01:14:33 2012 From: zabrane3@REDACTED (Zabrane Mickael) Date: Tue, 10 Jul 2012 01:14:33 +0200 Subject: [erlang-questions] Improve performance of IO bounded server written in Erlang via having pollset for each scheduler and bind port to scheduler together with process In-Reply-To: References: Message-ID: Hi, Performance of our HTTP Web Server drops down after applying your patches. Box: Linux F17, 4GB of RAM: $ lscpu Architecture: i686 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 4 On-line CPU(s) list: 0-3 Thread(s) per core: 1 Core(s) per socket: 4 Socket(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 23 Stepping: 7 CPU MHz: 2499.772 BogoMIPS: 4999.54 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 3072K Bench: before: 77787 rps after: 53056 rps Any hint to explain this result Wei ? Regards, Zabrane On Jul 9, 2012, at 11:01 AM, Wei Cao wrote: > Hi, all > > We wrote a proxy server in Erlang, the proxy's logic is rather simple, > it listens on some TCP port, establishes new connection from user > client, forward packets back and forth between the client and backend > server after authentication until connection is closed. > > It's very easy to write such a proxy in Erlang, fork a process for > each new user connection and connect to the backend server in the same > process, the process works like a pipe, sockets from both side is set > to the active once mode, whenever a tcp packet is received from one > socket, the packet will be sent to other socket. (A simplified version > of proxy code is attached at the end of the mail) > > However, the performance is not quite satisfying, the proxy can handle > maximum only 40k requests on our 16 core machine(Intel Xeon L5630, > 2.13GHz) with heavy stress(100 concurrent clients). We then analyzed > the behavior of beam.smp use tools like tcprstat, mpstat, perf, and > SystemTap. > > tcprstat shows QPS is about 40k, have average 1.7 millisecond latency > for each request. > > timestamp count max min avg med stddev 95_max 95_avg 95_std 99_max 99_avg 99_std > 1341813326 39416 17873 953 1718 1519 724 2919 1609 340 3813 1674 462 > 1341813327 40528 9275 884 1645 1493 491 2777 1559 290 3508 1619 409 > 1341813328 40009 18105 925 1694 1507 714 2868 1586 328 3753 1650 450 > > > mpstat shows 30% CPU is idle, > > 03:30:19 PM CPU %usr %nice %sys %iowait %irq %soft > %steal %guest %idle > 03:30:20 PM all 38.69 0.00 21.92 0.00 0.06 7.52 > 0.00 0.00 31.80 > 03:30:21 PM all 37.56 0.00 21.99 0.00 0.00 7.50 > 0.00 0.00 32.95 > > and perf top shows, much time is wasted in scheduler_wait, in spin wait I guess. > > 9320.00 19.8% scheduler_wait > /home/mingsong.cw/erlangr16b/lib/erlang/erts-5.10/bin/beam.smp > 1813.00 3.9% process_main > /home/mingsong.cw/erlangr16b/lib/erlang/erts-5.10/bin/beam.smp > 1379.00 2.9% _spin_lock > /usr/lib/debug/lib/modules/2.6.32-131.21.1.tb477.el6.x86_64/vmlinux > 1201.00 2.6% schedule > /home/mingsong.cw/erlangr16b/lib/erlang/erts-5.10/bin/beam.smp > > It seems the performance may be associated with scheduler_wait() and > erts_check_io(), with a SystemTap script(attached at the end of this > mail), we can find out how many times the system call epoll_wait is > invoked by beam.smp and each time, how many revents it gets. > > cpu process times > revents min max avg timeouts > all > 1754 128042 - - 73 3 > [14] beam.smp 151 > 14127 82 97 93 0 > [ 5] beam.smp 142 > 13291 83 97 93 0 > [13] beam.smp 127 > 11948 86 96 94 0 > [ 6] beam.smp 127 > 11836 81 96 93 0 > [ 4] beam.smp 121 > 11323 81 96 93 0 > [15] beam.smp 117 > 10935 83 96 93 0 > [12] beam.smp 486 > 10128 0 96 20 2 > [ 1] beam.smp 71 > 6549 71 100 92 0 > [ 2] beam.smp 62 > 5695 82 96 91 0 > [ 7] beam.smp 55 > 5102 81 95 92 0 > [11] beam.smp 52 > 4822 85 95 92 0 > [ 9] beam.smp 52 > 4799 85 95 92 0 > [ 8] beam.smp 51 > 4680 78 95 91 0 > [10] beam.smp 49 > 4508 85 97 92 0 > [ 3] beam.smp 46 > 4211 81 95 91 0 > [ 0] beam.smp 44 > 4088 83 95 92 0 > > The resuls shows, epoll_wait is invoked 1754 times each second, and > get 73 io events in average. This is unacceptable for writing high > performance server. Because if epoll_wait is invoked no more than 2k > times per second, then read/write a packet would cost more than 500ms, > which causes long delay and affects the throughput finally. > > The problem relies on there is only one global pollset in system wide, > so at a time there is no more than one scheduler can call > erts_check_io() to obtain pending io tasks from underlying pollset, > and no scheduler can call erts_check_io() before all pending io > tasks're processed, so for IO bounded application, it's very likely > that a scheduler finish its own job, but must wait idly for other > schedulers to complete theirs. > > Hence, we develops a patch to slove this problem, by having a pollset > for each scheduler, so that each scheduler can invoke erts_check_io() > on its own pollset concurrently. After a scheduler complete its tasks, > it can invoke erts_check_io() immediately no matter what state other > schedulers're in. This patch also handles port migration situation, > all used file descriptors in each port're recorded, when a port is > migrated, these > fd 're removed from original scheduler's pollset, and added to new scheduler's. > > Bind port to scheduler together with process is also helpful to > performance, it reduces the cost of thread switches and > synchronization, and bound port won't be migrated between schedulers. > > After apply the two patches, with the same pressure(100 concurrent > clients),epoll_wait is invoked 49332 times per second, and get 3 > revents each time in average, that is to say, our server responds > quicker and become more realtime. > > cpu process times > revents min max avg timeouts > all > 49332 217186 - - 4 3 > [ 2] beam.smp 3219 > 16050 2 7 4 0 > [11] beam.smp 4275 > 16033 1 6 3 0 > [ 8] beam.smp 4240 > 15992 1 6 3 0 > [ 9] beam.smp 4316 > 15964 0 6 3 2 > [10] beam.smp 4139 > 15851 1 6 3 0 > [ 3] beam.smp 4256 > 15816 1 6 3 0 > [ 1] beam.smp 3107 > 15800 2 7 5 0 > [ 0] beam.smp 3727 > 15259 1 6 4 0 > [ 7] beam.smp 2810 > 14722 3 7 5 0 > [13] beam.smp 1981 > 11499 4 7 5 0 > [15] beam.smp 2285 > 10942 3 6 4 0 > [14] beam.smp 2258 > 10866 3 6 4 0 > [ 4] beam.smp 2246 > 10849 3 6 4 0 > [ 6] beam.smp 2206 > 10730 3 6 4 0 > [12] beam.smp 2173 > 10573 3 6 4 0 > [ 5] beam.smp 2093 > 10240 3 6 4 0 > > scheduler_wait no longer take so much time now, > > 169.00 6.2% process_main beam.smp > 55.00 2.0% _spin_lock [kernel] > 45.00 1.7% driver_deliver_term beam.smp > > so is idle CPU time > 04:30:44 PM CPU %usr %nice %sys %iowait %irq %soft > %steal %guest %idle > 04:30:45 PM all 60.34 0.00 21.44 0.00 0.06 16.45 > 0.00 0.00 1.71 > 04:30:46 PM all 60.99 0.00 21.22 0.00 0.00 16.26 > 0.00 0.00 1.52 > > and tcprstat shows, QPS is getting 100K, latency is less than 1 millisecond > > timestamp count max min avg med stddev 95_max 95_avg 95_std 99_max 99_avg 99_std > 1341822689 96078 11592 314 910 817 311 1447 869 228 1777 897 263 > 1341822690 100651 24245 209 914 819 381 1458 870 229 1800 899 263 > > I also write a extreamly simple keep-alive http server(attached at the > end of mail), to compare performance before and after applying the > patches, mearused with apache ab tool(ab -n 1000000 -c 100 -k ), a 30% > performance gain can be get. > > before > Requests per second: 103671.70 [#/sec] (mean) > Time per request: 0.965 [ms] (mean) > > after > Requests per second: 133701.24 [#/sec] (mean) > Time per request: 0.748 [ms] (mean) > > Patches can be found at github, compile with > ./configure CFLAGS=-DERTS_POLLSET_PER_SCHEDULER > > Pollset per scheduler: > > git fetch git://github.com/weicao/otp.git pollset_per_scheduler > > https://github.com/weicao/otp/compare/weicao:master...weicao:pollset_per_scheduler > https://github.com/weicao/otp/compare/weicao:master...weicao:pollset_per_scheduler.patch > > > Bind port to scheduler: > > git fetch git://github.com/weicao/otp.git bind_port_to_scheduler > > https://github.com/weicao/otp/compare/weicao:pollset_per_scheduler...weicao:bind_port_to_scheduler > https://github.com/weicao/otp/compare/weicao:pollset_per_scheduler...weicao:bind_port_to_scheduler.patch > > > Appendix: > > ----------------------------------- > proxy.erl > ------------------------------------ > > -module(proxy). > -compile(export_all). > > -define(RECBUF_SIZE, 8192). > -define(ACCEPT_TIMEOUT, 2000). > > start([MyPortAtom, DestIpAtom, DestPortAtom]) -> > {MyPort, []} = string:to_integer(atom_to_list(MyPortAtom)), > DestIp = atom_to_list(DestIpAtom), > {DestPort, []} = string:to_integer(atom_to_list(DestPortAtom)), > listen(MyPort, DestIp, DestPort), > receive Any -> > io:format("recv ~p~n", [Any]) > end. > > listen(MyPort, DestIp, DestPort) -> > io:format("start proxy on [local] 0.0.0.0:~p -> [remote] ~p:~p~n", > [MyPort, DestIp, DestPort]), > case gen_tcp:listen(MyPort, > [inet, > {ip, {0,0,0,0}}, > binary, > {reuseaddr, true}, > {recbuf, ?RECBUF_SIZE}, > {active, false}, > {nodelay, true} > ]) of > {ok, Listen} -> > N = erlang:system_info(schedulers), > lists:foreach(fun(I) -> accept(Listen, DestIp, DestPort, > I) end, lists:seq(1,N)); > {error, Reason} -> > io:format("error listen ~p~n", [Reason]) > end. > > accept(Listen, DestIp, DestPort, I) -> > spawn_opt(?MODULE, loop, [Listen, DestIp, DestPort, I], [{scheduler, I}]). > > loop(Listen, DestIp, DestPort, I) -> > case gen_tcp:accept(Listen, ?ACCEPT_TIMEOUT) of > {ok, S1} -> > accept(Listen, DestIp, DestPort, I), > case catch gen_tcp:connect(DestIp, DestPort, > [inet, binary, {active, false}, > {reuseaddr, true}, {nodelay, true}]) of > {ok, S2} -> > io:format("new connection~n"), > loop1(S1, S2); > {error, Reason} -> > io:format("error connect ~p~n", [Reason]) > end; > {error, timeout} -> > loop(Listen, DestIp, DestPort, I); > Error -> > io:format("error accept ~p~n", [Error]), > accept(Listen, DestIp, DestPort, I) > end. > > loop1(S1, S2) -> > active(S1, S2), > receive > {tcp, S1, Data} -> > gen_tcp:send(S2, Data), > loop1(S1, S2); > {tcp, S2, Data} -> > gen_tcp:send(S1, Data), > loop1(S1, S2); > {tcp_closed, S1} -> > io:format("S1 close~n"), > gen_tcp:close(S1), > gen_tcp:close(S2); > {tcp_closed, S2} -> > io:format("S2 close~n"), > gen_tcp:close(S1), > gen_tcp:close(S2) > end. > > active(S1,S2)-> > inet:setopts(S1, [{active, once}]), > inet:setopts(S2, [{active, once}]). > > ----------------------------------- > epollwait.stp > ------------------------------------ > #! /usr/bin/env stap > # > # > > global epoll_timeout_flag, epoll_count, epoll_min, epoll_max, > epoll_times, epoll_timeouts > > probe syscall.epoll_wait { > if(timeout > 0) { > epoll_timeout_flag[pid()] = 1 > } > } > > probe syscall.epoll_wait.return { > c = cpu() > p = execname() > > epoll_times[c,p] ++ > epoll_count[c,p] += $return > > if($return == 0 && pid() in epoll_timeout_flag) { > epoll_timeouts[c,p] ++ > delete epoll_timeout_flag[pid()] > } > > if(!([c, p] in epoll_min)) { > epoll_min[c,p] = $return > } else if($return < epoll_min[c,p]) { > epoll_min[c,p] = $return > } > > > if($return > epoll_max[c,p]) { > epoll_max[c,p] = $return > } > } > > probe timer.s($1) { > printf ("%4s %45s %10s %10s %10s %10s %10s %10s\n", "cpu", > "process", "times", "revents", "min", "max", "avg", "timeouts" ) > foreach ([cpu, process] in epoll_count- limit 20) { > all_epoll_times += epoll_times[cpu,process] > all_epoll_count += epoll_count[cpu,process] > all_epoll_timeouts += epoll_timeouts[cpu,process] > } > printf ("%4s %45s %10d %10d %10s %10s %10d %10d\n", > "all", "", all_epoll_times, all_epoll_count, "-", "-", > all_epoll_count == 0? 0:all_epoll_count/all_epoll_times, all_epoll_timeouts) > > foreach ([cpu, process] in epoll_count- limit 20) { > printf ("[%2d] %45s %10d %10d %10d %10d %10d %10d\n", > cpu, process, epoll_times[cpu, process], epoll_count[cpu, process], > epoll_min[cpu, process], epoll_max[cpu, process], > epoll_count[cpu,process]/epoll_times[cpu,process], > epoll_timeouts[cpu,process]) > } > delete epoll_count > delete epoll_min > delete epoll_max > delete epoll_times > delete epoll_timeouts > printf ("--------------------------------------------------------------------------\n\n" > ) > } > > ------------------------------------------------ > ehttpd.erl > ------------------------------------------------- > -module(ehttpd). > -compile(export_all). > > start() -> > start(8888). > start(Port) -> > N = erlang:system_info(schedulers), > listen(Port, N), > io:format("ehttpd ready with ~b schedulers on port ~b~n", [N, Port]), > > register(?MODULE, self()), > receive Any -> io:format("~p~n", [Any]) end. %% to stop: ehttpd!stop. > > listen(Port, N) -> > Opts = [inet, > binary, > {active, false}, > {recbuf, 8192}, > {nodelay,true}, > {reuseaddr, true}], > > {ok, S} = gen_tcp:listen(Port, Opts), > lists:foreach(fun(I)-> spawn_opt(?MODULE, accept, [S, I], > [{scheduler, I}]) end, lists:seq(1, N)). > > accept(S, I) -> > case gen_tcp:accept(S) of > {ok, Socket} -> > spawn_opt(?MODULE, accept, [S, I],[{scheduler,I}] ), > io:format("new connection @~p~n", [I]), > loop(Socket,<<>>); > Error -> erlang:error(Error) > end. > > loop(S,B) -> > inet:setopts(S, [{active, once}]), > receive > {tcp, S, Data} -> > B1 = <>, > case binary:part(B1,{byte_size(B1), -4}) of > <<"\r\n\r\n">> -> > Response = <<"HTTP/1.1 200 OK\r\nContent-Length: > 12\r\nConnection: Keep-Alive\r\n\r\nhello world!">>, > gen_tcp:send(S, Response), > loop(S, <<>>); > _ -> > loop(S, B1) > end; > {tcp_closed, S} -> > io:format("connection closed forced~n"), > gen_tcp:close(S); > Error -> > io:format("unexpected message~p~n", [Error]), > Error > end. > > > -- > > Best, > > Wei Cao > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From eriksoe@REDACTED Tue Jul 10 01:50:03 2012 From: eriksoe@REDACTED (=?ISO-8859-1?Q?Erik_S=F8e_S=F8rensen?=) Date: Tue, 10 Jul 2012 01:50:03 +0200 Subject: [erlang-questions] specs for fixed length lists In-Reply-To: References: Message-ID: Using setelement is a bad idea, at least unless the tuples are very small: the cost in time and allocation is n^2. A better scaling approach is to use tuple_to_list + map + list_to_tuple; that should be linear, and my guess is that it'd be faster even for small tuple sizes. (For the actual n=4 case, I don't know which is better; must measure...) /Erik Den 09/07/2012 22.54 skrev "Lukas Larsson" : > Writing a generic tuple:map/2 would not be that hard either: > > map(F, Tuple) -> > map(F, Tuple, size(Tuple)). > > map(F,Tuple,1) -> > setelement(1,Tuple,F(element(1,Tuple))); > map(F,Tuple,N) -> > map(F,setelement(N,Tuple,F(element(N,Tuple))),N-1). > > It's not very pretty, or efficient, but it gets the job done. > > Lukas > > On Mon, Jul 9, 2012 at 9:59 PM, Daniel Goertzen > wrote: > > > > On Sun, Jul 8, 2012 at 1:22 AM, wrote: > >> > >> I'm with /Joe on this one. There aren't all _that_ many > >> list operations that make sense on tuples, and > >> map4(F, {A,B,C,D}) -> {F(A), F(B), F(C), F(D)}. > >> is easy enough to define. Perhaps the code needs turning > >> inside out. > >> > > > > Thanks for the idea; my functional instincts are still developing. I > will > > go with this approach since my sequence size really is only 4. > > > > Dan. > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Tue Jul 10 03:21:42 2012 From: ok@REDACTED (Richard O'Keefe) Date: Tue, 10 Jul 2012 13:21:42 +1200 Subject: [erlang-questions] simple question about list memory consumption In-Reply-To: References: <48564154-48ae-4d26-9a06-aa1091360cc0@knuth> Message-ID: <8604E6F2-85ED-4CB4-8257-C216BCD6C713@cs.otago.ac.nz> On 9/07/2012, at 10:34 PM, CGS wrote: > > About the two lists you were speaking about (I suppose you were referring to the output of lists:map/2 and lists:seq/2), the second is destroyed as soon as lists:map/2 exits. > > No it isn't. It is only reclaimed when the garbage collector runs. > > That is a bit puzzling. It seems that you are muddling up "is LIVE" with "EXISTS". When you compute map(F, L) and there are no other references to L anywhere, L becomes *DEAD* as soon as the call to map/2 completes. (In fact, each cell of L becomes dead in turn during the call.) But DEAD and DESTROYED are very different properties. L is not "destroyed" until the garbage collector runs; until the garbage collector runs the cells of L still occupy space on the heap. When, if ever, that space is reclaimed depends on how much other need there is for it. > Looking at the code for lists:map/2: > > map(F, [H|T]) -> > [F(H)|map(F, T)]; > map(F, []) when is_function(F, 1) -> []. > > I can see that when lists:map/2 exits, the only remaining list is an empty list. No you can't see that, because there is nothing in the code for map/2 to release any storage whatever. > I might be wrong, but I see only one word remaining for the garbage collector to take care of. What do you think makes the other words go away? > > > You got the result from process_info, I got it from `free' under Linux. Not the best way to do it. That tells you how much memory the operating has handed out but not to which process; even ps -o size tells you only how much memory Erlang has claimed from the operating system, and nothing about what Erlang is doing with that memory. From ok@REDACTED Tue Jul 10 03:51:09 2012 From: ok@REDACTED (Richard O'Keefe) Date: Tue, 10 Jul 2012 13:51:09 +1200 Subject: [erlang-questions] specs for fixed length lists In-Reply-To: References: Message-ID: <527994A9-EB45-48C8-84B7-EB1672335EBA@cs.otago.ac.nz> On 10/07/2012, at 8:54 AM, Lukas Larsson wrote: > Writing a generic tuple:map/2 would not be that hard either: > > map(F, Tuple) -> > map(F, Tuple, size(Tuple)). > > map(F,Tuple,1) -> > setelement(1,Tuple,F(element(1,Tuple))); > map(F,Tuple,N) -> > map(F,setelement(N,Tuple,F(element(N,Tuple))),N-1). Code golf! tuple_map(F, Tuple) -> list_to_tuple([F(X) || X <- tuple_to_list(Tuple)]). Exercise for the reader: why is my version O(|Tuple|) but Lukas Larsson's O(|Tuple|**2)? From cyg.cao@REDACTED Tue Jul 10 04:30:01 2012 From: cyg.cao@REDACTED (Wei Cao) Date: Tue, 10 Jul 2012 10:30:01 +0800 Subject: [erlang-questions] Improve performance of IO bounded server written in Erlang via having pollset for each scheduler and bind port to scheduler together with process In-Reply-To: References: Message-ID: My tests were all keepalived/persistent connections, and there was significant performance gain in these situations. but I wrote a http server with short connection just now, the performance did drop down, I'll find it out later :). 2012/7/10 Zabrane Mickael : > Hi, > > Performance of our HTTP Web Server drops down after applying your patches. > > Box: Linux F17, 4GB of RAM: > $ lscpu > Architecture: i686 > CPU op-mode(s): 32-bit, 64-bit > Byte Order: Little Endian > CPU(s): 4 > On-line CPU(s) list: 0-3 > Thread(s) per core: 1 > Core(s) per socket: 4 > Socket(s): 1 > Vendor ID: GenuineIntel > CPU family: 6 > Model: 23 > Stepping: 7 > CPU MHz: 2499.772 > BogoMIPS: 4999.54 > Virtualization: VT-x > L1d cache: 32K > L1i cache: 32K > L2 cache: 3072K > > Bench: > before: 77787 rps > after: 53056 rps > > Any hint to explain this result Wei ? > > Regards, > Zabrane > > > On Jul 9, 2012, at 11:01 AM, Wei Cao wrote: > > Hi, all > > We wrote a proxy server in Erlang, the proxy's logic is rather simple, > it listens on some TCP port, establishes new connection from user > client, forward packets back and forth between the client and backend > server after authentication until connection is closed. > > It's very easy to write such a proxy in Erlang, fork a process for > each new user connection and connect to the backend server in the same > process, the process works like a pipe, sockets from both side is set > to the active once mode, whenever a tcp packet is received from one > socket, the packet will be sent to other socket. (A simplified version > of proxy code is attached at the end of the mail) > > However, the performance is not quite satisfying, the proxy can handle > maximum only 40k requests on our 16 core machine(Intel Xeon L5630, > 2.13GHz) with heavy stress(100 concurrent clients). We then analyzed > the behavior of beam.smp use tools like tcprstat, mpstat, perf, and > SystemTap. > > tcprstat shows QPS is about 40k, have average 1.7 millisecond latency > for each request. > > timestamp count max min avg med stddev 95_max 95_avg 95_std 99_max 99_avg > 99_std > 1341813326 39416 17873 953 1718 1519 724 2919 1609 340 3813 1674 462 > 1341813327 40528 9275 884 1645 1493 491 2777 1559 290 3508 1619 409 > 1341813328 40009 18105 925 1694 1507 714 2868 1586 328 3753 1650 450 > > > mpstat shows 30% CPU is idle, > > 03:30:19 PM CPU %usr %nice %sys %iowait %irq %soft > %steal %guest %idle > 03:30:20 PM all 38.69 0.00 21.92 0.00 0.06 7.52 > 0.00 0.00 31.80 > 03:30:21 PM all 37.56 0.00 21.99 0.00 0.00 7.50 > 0.00 0.00 32.95 > > and perf top shows, much time is wasted in scheduler_wait, in spin wait I > guess. > > 9320.00 19.8% scheduler_wait > /home/mingsong.cw/erlangr16b/lib/erlang/erts-5.10/bin/beam.smp > 1813.00 3.9% process_main > /home/mingsong.cw/erlangr16b/lib/erlang/erts-5.10/bin/beam.smp > 1379.00 2.9% _spin_lock > /usr/lib/debug/lib/modules/2.6.32-131.21.1.tb477.el6.x86_64/vmlinux > 1201.00 2.6% schedule > /home/mingsong.cw/erlangr16b/lib/erlang/erts-5.10/bin/beam.smp > > It seems the performance may be associated with scheduler_wait() and > erts_check_io(), with a SystemTap script(attached at the end of this > mail), we can find out how many times the system call epoll_wait is > invoked by beam.smp and each time, how many revents it gets. > > cpu process times > revents min max avg timeouts > all > 1754 128042 - - 73 3 > [14] beam.smp 151 > 14127 82 97 93 0 > [ 5] beam.smp 142 > 13291 83 97 93 0 > [13] beam.smp 127 > 11948 86 96 94 0 > [ 6] beam.smp 127 > 11836 81 96 93 0 > [ 4] beam.smp 121 > 11323 81 96 93 0 > [15] beam.smp 117 > 10935 83 96 93 0 > [12] beam.smp 486 > 10128 0 96 20 2 > [ 1] beam.smp 71 > 6549 71 100 92 0 > [ 2] beam.smp 62 > 5695 82 96 91 0 > [ 7] beam.smp 55 > 5102 81 95 92 0 > [11] beam.smp 52 > 4822 85 95 92 0 > [ 9] beam.smp 52 > 4799 85 95 92 0 > [ 8] beam.smp 51 > 4680 78 95 91 0 > [10] beam.smp 49 > 4508 85 97 92 0 > [ 3] beam.smp 46 > 4211 81 95 91 0 > [ 0] beam.smp 44 > 4088 83 95 92 0 > > The resuls shows, epoll_wait is invoked 1754 times each second, and > get 73 io events in average. This is unacceptable for writing high > performance server. Because if epoll_wait is invoked no more than 2k > times per second, then read/write a packet would cost more than 500ms, > which causes long delay and affects the throughput finally. > > The problem relies on there is only one global pollset in system wide, > so at a time there is no more than one scheduler can call > erts_check_io() to obtain pending io tasks from underlying pollset, > and no scheduler can call erts_check_io() before all pending io > tasks're processed, so for IO bounded application, it's very likely > that a scheduler finish its own job, but must wait idly for other > schedulers to complete theirs. > > Hence, we develops a patch to slove this problem, by having a pollset > for each scheduler, so that each scheduler can invoke erts_check_io() > on its own pollset concurrently. After a scheduler complete its tasks, > it can invoke erts_check_io() immediately no matter what state other > schedulers're in. This patch also handles port migration situation, > all used file descriptors in each port're recorded, when a port is > migrated, these > fd 're removed from original scheduler's pollset, and added to new > scheduler's. > > Bind port to scheduler together with process is also helpful to > performance, it reduces the cost of thread switches and > synchronization, and bound port won't be migrated between schedulers. > > After apply the two patches, with the same pressure(100 concurrent > clients),epoll_wait is invoked 49332 times per second, and get 3 > revents each time in average, that is to say, our server responds > quicker and become more realtime. > > cpu process times > revents min max avg timeouts > all > 49332 217186 - - 4 3 > [ 2] beam.smp 3219 > 16050 2 7 4 0 > [11] beam.smp 4275 > 16033 1 6 3 0 > [ 8] beam.smp 4240 > 15992 1 6 3 0 > [ 9] beam.smp 4316 > 15964 0 6 3 2 > [10] beam.smp 4139 > 15851 1 6 3 0 > [ 3] beam.smp 4256 > 15816 1 6 3 0 > [ 1] beam.smp 3107 > 15800 2 7 5 0 > [ 0] beam.smp 3727 > 15259 1 6 4 0 > [ 7] beam.smp 2810 > 14722 3 7 5 0 > [13] beam.smp 1981 > 11499 4 7 5 0 > [15] beam.smp 2285 > 10942 3 6 4 0 > [14] beam.smp 2258 > 10866 3 6 4 0 > [ 4] beam.smp 2246 > 10849 3 6 4 0 > [ 6] beam.smp 2206 > 10730 3 6 4 0 > [12] beam.smp 2173 > 10573 3 6 4 0 > [ 5] beam.smp 2093 > 10240 3 6 4 0 > > scheduler_wait no longer take so much time now, > > 169.00 6.2% process_main beam.smp > 55.00 2.0% _spin_lock [kernel] > 45.00 1.7% driver_deliver_term beam.smp > > so is idle CPU time > 04:30:44 PM CPU %usr %nice %sys %iowait %irq %soft > %steal %guest %idle > 04:30:45 PM all 60.34 0.00 21.44 0.00 0.06 16.45 > 0.00 0.00 1.71 > 04:30:46 PM all 60.99 0.00 21.22 0.00 0.00 16.26 > 0.00 0.00 1.52 > > and tcprstat shows, QPS is getting 100K, latency is less than 1 millisecond > > timestamp count max min avg med stddev 95_max 95_avg 95_std 99_max 99_avg > 99_std > 1341822689 96078 11592 314 910 817 311 1447 869 228 1777 897 263 > 1341822690 100651 24245 209 914 819 381 1458 870 229 1800 899 263 > > I also write a extreamly simple keep-alive http server(attached at the > end of mail), to compare performance before and after applying the > patches, mearused with apache ab tool(ab -n 1000000 -c 100 -k ), a 30% > performance gain can be get. > > before > Requests per second: 103671.70 [#/sec] (mean) > Time per request: 0.965 [ms] (mean) > > after > Requests per second: 133701.24 [#/sec] (mean) > Time per request: 0.748 [ms] (mean) > > Patches can be found at github, compile with > ./configure CFLAGS=-DERTS_POLLSET_PER_SCHEDULER > > Pollset per scheduler: > > git fetch git://github.com/weicao/otp.git pollset_per_scheduler > > https://github.com/weicao/otp/compare/weicao:master...weicao:pollset_per_scheduler > https://github.com/weicao/otp/compare/weicao:master...weicao:pollset_per_scheduler.patch > > > Bind port to scheduler: > > git fetch git://github.com/weicao/otp.git bind_port_to_scheduler > > https://github.com/weicao/otp/compare/weicao:pollset_per_scheduler...weicao:bind_port_to_scheduler > https://github.com/weicao/otp/compare/weicao:pollset_per_scheduler...weicao:bind_port_to_scheduler.patch > > > Appendix: > > ----------------------------------- > proxy.erl > ------------------------------------ > > -module(proxy). > -compile(export_all). > > -define(RECBUF_SIZE, 8192). > -define(ACCEPT_TIMEOUT, 2000). > > start([MyPortAtom, DestIpAtom, DestPortAtom]) -> > {MyPort, []} = string:to_integer(atom_to_list(MyPortAtom)), > DestIp = atom_to_list(DestIpAtom), > {DestPort, []} = string:to_integer(atom_to_list(DestPortAtom)), > listen(MyPort, DestIp, DestPort), > receive Any -> > io:format("recv ~p~n", [Any]) > end. > > listen(MyPort, DestIp, DestPort) -> > io:format("start proxy on [local] 0.0.0.0:~p -> [remote] ~p:~p~n", > [MyPort, DestIp, DestPort]), > case gen_tcp:listen(MyPort, > [inet, > {ip, {0,0,0,0}}, > binary, > {reuseaddr, true}, > {recbuf, ?RECBUF_SIZE}, > {active, false}, > {nodelay, true} > ]) of > {ok, Listen} -> > N = erlang:system_info(schedulers), > lists:foreach(fun(I) -> accept(Listen, DestIp, DestPort, > I) end, lists:seq(1,N)); > {error, Reason} -> > io:format("error listen ~p~n", [Reason]) > end. > > accept(Listen, DestIp, DestPort, I) -> > spawn_opt(?MODULE, loop, [Listen, DestIp, DestPort, I], [{scheduler, > I}]). > > loop(Listen, DestIp, DestPort, I) -> > case gen_tcp:accept(Listen, ?ACCEPT_TIMEOUT) of > {ok, S1} -> > accept(Listen, DestIp, DestPort, I), > case catch gen_tcp:connect(DestIp, DestPort, > [inet, binary, {active, false}, > {reuseaddr, true}, {nodelay, true}]) of > {ok, S2} -> > io:format("new connection~n"), > loop1(S1, S2); > {error, Reason} -> > io:format("error connect ~p~n", [Reason]) > end; > {error, timeout} -> > loop(Listen, DestIp, DestPort, I); > Error -> > io:format("error accept ~p~n", [Error]), > accept(Listen, DestIp, DestPort, I) > end. > > loop1(S1, S2) -> > active(S1, S2), > receive > {tcp, S1, Data} -> > gen_tcp:send(S2, Data), > loop1(S1, S2); > {tcp, S2, Data} -> > gen_tcp:send(S1, Data), > loop1(S1, S2); > {tcp_closed, S1} -> > io:format("S1 close~n"), > gen_tcp:close(S1), > gen_tcp:close(S2); > {tcp_closed, S2} -> > io:format("S2 close~n"), > gen_tcp:close(S1), > gen_tcp:close(S2) > end. > > active(S1,S2)-> > inet:setopts(S1, [{active, once}]), > inet:setopts(S2, [{active, once}]). > > ----------------------------------- > epollwait.stp > ------------------------------------ > #! /usr/bin/env stap > # > # > > global epoll_timeout_flag, epoll_count, epoll_min, epoll_max, > epoll_times, epoll_timeouts > > probe syscall.epoll_wait { > if(timeout > 0) { > epoll_timeout_flag[pid()] = 1 > } > } > > probe syscall.epoll_wait.return { > c = cpu() > p = execname() > > epoll_times[c,p] ++ > epoll_count[c,p] += $return > > if($return == 0 && pid() in epoll_timeout_flag) { > epoll_timeouts[c,p] ++ > delete epoll_timeout_flag[pid()] > } > > if(!([c, p] in epoll_min)) { > epoll_min[c,p] = $return > } else if($return < epoll_min[c,p]) { > epoll_min[c,p] = $return > } > > > if($return > epoll_max[c,p]) { > epoll_max[c,p] = $return > } > } > > probe timer.s($1) { > printf ("%4s %45s %10s %10s %10s %10s %10s %10s\n", "cpu", > "process", "times", "revents", "min", "max", "avg", "timeouts" ) > foreach ([cpu, process] in epoll_count- limit 20) { > all_epoll_times += epoll_times[cpu,process] > all_epoll_count += epoll_count[cpu,process] > all_epoll_timeouts += epoll_timeouts[cpu,process] > } > printf ("%4s %45s %10d %10d %10s %10s %10d %10d\n", > "all", "", all_epoll_times, all_epoll_count, "-", "-", > all_epoll_count == 0? 0:all_epoll_count/all_epoll_times, > all_epoll_timeouts) > > foreach ([cpu, process] in epoll_count- limit 20) { > printf ("[%2d] %45s %10d %10d %10d %10d %10d %10d\n", > cpu, process, epoll_times[cpu, process], epoll_count[cpu, process], > epoll_min[cpu, process], epoll_max[cpu, process], > epoll_count[cpu,process]/epoll_times[cpu,process], > epoll_timeouts[cpu,process]) > } > delete epoll_count > delete epoll_min > delete epoll_max > delete epoll_times > delete epoll_timeouts > printf > ("--------------------------------------------------------------------------\n\n" > ) > } > > ------------------------------------------------ > ehttpd.erl > ------------------------------------------------- > -module(ehttpd). > -compile(export_all). > > start() -> > start(8888). > start(Port) -> > N = erlang:system_info(schedulers), > listen(Port, N), > io:format("ehttpd ready with ~b schedulers on port ~b~n", [N, Port]), > > register(?MODULE, self()), > receive Any -> io:format("~p~n", [Any]) end. %% to stop: ehttpd!stop. > > listen(Port, N) -> > Opts = [inet, > binary, > {active, false}, > {recbuf, 8192}, > {nodelay,true}, > {reuseaddr, true}], > > {ok, S} = gen_tcp:listen(Port, Opts), > lists:foreach(fun(I)-> spawn_opt(?MODULE, accept, [S, I], > [{scheduler, I}]) end, lists:seq(1, N)). > > accept(S, I) -> > case gen_tcp:accept(S) of > {ok, Socket} -> > spawn_opt(?MODULE, accept, [S, I],[{scheduler,I}] ), > io:format("new connection @~p~n", [I]), > loop(Socket,<<>>); > Error -> erlang:error(Error) > end. > > loop(S,B) -> > inet:setopts(S, [{active, once}]), > receive > {tcp, S, Data} -> > B1 = <>, > case binary:part(B1,{byte_size(B1), -4}) of > <<"\r\n\r\n">> -> > Response = <<"HTTP/1.1 200 OK\r\nContent-Length: > 12\r\nConnection: Keep-Alive\r\n\r\nhello world!">>, > gen_tcp:send(S, Response), > loop(S, <<>>); > _ -> > loop(S, B1) > end; > {tcp_closed, S} -> > io:format("connection closed forced~n"), > gen_tcp:close(S); > Error -> > io:format("unexpected message~p~n", [Error]), > Error > end. > > > -- > > Best, > > Wei Cao > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > -- Best, Wei Cao From n.oxyde@REDACTED Tue Jul 10 10:09:19 2012 From: n.oxyde@REDACTED (Anthony Ramine) Date: Tue, 10 Jul 2012 10:09:19 +0200 Subject: [erlang-questions] specs for fixed length lists In-Reply-To: References: Message-ID: <671132D1-1699-4246-BC2A-9D49944F3267@gmail.com> I wouldn't be so sure about that, you need to check whether the destructive updates pass (dsetel) does its magic here. -- Anthony Ramine Le 10 juil. 2012 ? 01:50, Erik S?e S?rensen a ?crit : > Using setelement is a bad idea, at least unless the tuples are very small: the cost in time and allocation is n^2. > A better scaling approach is to use tuple_to_list + map + list_to_tuple; that should be linear, and my guess is that it'd be faster even for small tuple sizes. > (For the actual n=4 case, I don't know which is better; must measure...) > /Erik From ward@REDACTED Tue Jul 10 10:33:11 2012 From: ward@REDACTED (Ward Bekker) Date: Tue, 10 Jul 2012 10:33:11 +0200 Subject: [erlang-questions] Memory size of Erlang term Message-ID: <20120710083311.dd65bf1c@groupware.tty.nl> Hi, Is there an Erlang function that returns the actual word size / byte size of a term? (I know of http://www.erlang.org/doc/efficiency_guide/advanced.html#id68680 , but wondering if there is a function for it ) Regards, Ward -------------- next part -------------- An HTML attachment was scrubbed... URL: From lambdadmitry@REDACTED Tue Jul 10 10:43:49 2012 From: lambdadmitry@REDACTED (Dmitry Groshev) Date: Tue, 10 Jul 2012 12:43:49 +0400 Subject: [erlang-questions] Local namespaces in case/try Message-ID: Is there any reason why we don't have them? It's quite annoying to have potential error in places like this: case do_something() of {ok, Result} -> Result; {error, Error} -> Error end, case do_another() of {ok, Result} -> Result; {error, Error} -> Error end, Result and Error are bound in first case and we will probably have a match failure in second one. Compiler warns about this, but it's still very unwieldy to fix it with names like Error1, Error2, etc. I also earnestly ask to avoid answers like "you should name that variables in a meaningful way so there is no such problem". Speaking about backward compatibility: first, I think it's a very bad style to use variables that was bound this way and second, there is nothing wrong in introducing one more compile-time option (like inline one). From gustav.simonsson@REDACTED Tue Jul 10 10:49:36 2012 From: gustav.simonsson@REDACTED (Gustav Simonsson) Date: Tue, 10 Jul 2012 09:49:36 +0100 (BST) Subject: [erlang-questions] Memory size of Erlang term In-Reply-To: <20120710083311.dd65bf1c@groupware.tty.nl> Message-ID: <0684411b-2a03-45cb-9401-b62edbc56cea@knuth> Hi Ward, Examples of erts_debug:size/1 and flat_size/1 are in: http://www.erlang.org/doc/efficiency_guide/processes.html // Gustav Sent from my PC ----- Original Message ----- > From: "Ward Bekker" > To: erlang-questions@REDACTED > Sent: Tuesday, 10 July, 2012 10:33:11 AM > Subject: [erlang-questions] Memory size of Erlang term > > > Hi, > > > Is there an Erlang function that returns the actual word size / byte > size of a term? > > > > (I know of > http://www.erlang.org/doc/efficiency_guide/advanced.html#id68680 , > but wondering if there is a function for it ) > > > Regards, > > > Ward > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From dmitriid@REDACTED Tue Jul 10 10:52:24 2012 From: dmitriid@REDACTED (Dmitrii 'Mamut' Dimandt) Date: Tue, 10 Jul 2012 01:52:24 -0700 (PDT) Subject: [erlang-questions] Looking for slides of a lightning talk In-Reply-To: <4FFAAAEA.9060605@ninenines.eu> References: <4FFAAAEA.9060605@ninenines.eu> Message-ID: <0d6c6c5d-2b0f-48da-bae8-f440d232df92@googlegroups.com> Heh. That was me :) I have the slides at home, and I *WILL* forget to send them to you :) So feel free to bug me directly until I do > Someone did a lightning talk in Stockholm that was quite funny but also > quite true, saying that Erlang isn't ready for the web and doing > comparisons with other languages/platforms. Any chance I could get the > slides? > > Thanks. > > -- > Lo?c Hoguin > Erlang Cowboy > Nine Nines > http://ninenines.eu > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eriksoe@REDACTED Tue Jul 10 11:00:29 2012 From: eriksoe@REDACTED (=?ISO-8859-1?Q?Erik_S=F8e_S=F8rensen?=) Date: Tue, 10 Jul 2012 11:00:29 +0200 Subject: [erlang-questions] specs for fixed length lists In-Reply-To: <671132D1-1699-4246-BC2A-9D49944F3267@gmail.com> References: <671132D1-1699-4246-BC2A-9D49944F3267@gmail.com> Message-ID: It can optimize setelement used in a loop? I'd be surprised, although pleasantly so. (I didn't even know it could handle non-fixed indexes, but then I am not generally very familiar with it.) One more thing to research... Den 10/07/2012 10.10 skrev "Anthony Ramine" : > I wouldn't be so sure about that, you need to check whether the > destructive updates pass (dsetel) does its magic here. > > -- > Anthony Ramine > > Le 10 juil. 2012 ? 01:50, Erik S?e S?rensen a ?crit : > > > Using setelement is a bad idea, at least unless the tuples are very > small: the cost in time and allocation is n^2. > > A better scaling approach is to use tuple_to_list + map + list_to_tuple; > that should be linear, and my guess is that it'd be faster even for small > tuple sizes. > > (For the actual n=4 case, I don't know which is better; must measure...) > > /Erik > -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlsson.richard@REDACTED Tue Jul 10 11:02:29 2012 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Tue, 10 Jul 2012 11:02:29 +0200 Subject: [erlang-questions] Local namespaces in case/try In-Reply-To: References: Message-ID: <4FFBEFA5.6030305@gmail.com> On 07/10/2012 10:43 AM, Dmitry Groshev wrote: > case do_something() of > {ok, Result} -> Result; > {error, Error} -> Error > end, > case do_another() of > {ok, Result} -> Result; > {error, Error} -> Error > end, > > Result and Error are bound in first case and we will probably have a > match failure in second one. Compiler warns about this, but it's still > very unwieldy to fix it with names like Error1, Error2, etc. Take it as a sign that you should break out those case expressions into separate functions, or restructure the code in some other way. /Richard From zabrane3@REDACTED Tue Jul 10 11:08:13 2012 From: zabrane3@REDACTED (Zabrane Mickael) Date: Tue, 10 Jul 2012 11:08:13 +0200 Subject: [erlang-questions] Improve performance of IO bounded server written in Erlang via having pollset for each scheduler and bind port to scheduler together with process In-Reply-To: References: Message-ID: <0593CC0D-1B2B-4974-934E-C460C952CD44@gmail.com> On Jul 10, 2012, at 4:30 AM, Wei Cao wrote: > My tests were all keepalived/persistent connections, and there was > significant performance gain in these situations. Of course I enabled keepalive on my test. > but I wrote a http server with short connection just now, the performance did drop down, > I'll find it out later :). Great. Let me know if you make any progress. Regards, Zabrane > > 2012/7/10 Zabrane Mickael : >> Hi, >> >> Performance of our HTTP Web Server drops down after applying your patches. >> >> Box: Linux F17, 4GB of RAM: >> $ lscpu >> Architecture: i686 >> CPU op-mode(s): 32-bit, 64-bit >> Byte Order: Little Endian >> CPU(s): 4 >> On-line CPU(s) list: 0-3 >> Thread(s) per core: 1 >> Core(s) per socket: 4 >> Socket(s): 1 >> Vendor ID: GenuineIntel >> CPU family: 6 >> Model: 23 >> Stepping: 7 >> CPU MHz: 2499.772 >> BogoMIPS: 4999.54 >> Virtualization: VT-x >> L1d cache: 32K >> L1i cache: 32K >> L2 cache: 3072K >> >> Bench: >> before: 77787 rps >> after: 53056 rps >> >> Any hint to explain this result Wei ? >> >> Regards, >> Zabrane >> >> >> On Jul 9, 2012, at 11:01 AM, Wei Cao wrote: >> >> Hi, all >> >> We wrote a proxy server in Erlang, the proxy's logic is rather simple, >> it listens on some TCP port, establishes new connection from user >> client, forward packets back and forth between the client and backend >> server after authentication until connection is closed. >> >> It's very easy to write such a proxy in Erlang, fork a process for >> each new user connection and connect to the backend server in the same >> process, the process works like a pipe, sockets from both side is set >> to the active once mode, whenever a tcp packet is received from one >> socket, the packet will be sent to other socket. (A simplified version >> of proxy code is attached at the end of the mail) >> >> However, the performance is not quite satisfying, the proxy can handle >> maximum only 40k requests on our 16 core machine(Intel Xeon L5630, >> 2.13GHz) with heavy stress(100 concurrent clients). We then analyzed >> the behavior of beam.smp use tools like tcprstat, mpstat, perf, and >> SystemTap. >> >> tcprstat shows QPS is about 40k, have average 1.7 millisecond latency >> for each request. >> >> timestamp count max min avg med stddev 95_max 95_avg 95_std 99_max 99_avg >> 99_std >> 1341813326 39416 17873 953 1718 1519 724 2919 1609 340 3813 1674 462 >> 1341813327 40528 9275 884 1645 1493 491 2777 1559 290 3508 1619 409 >> 1341813328 40009 18105 925 1694 1507 714 2868 1586 328 3753 1650 450 >> >> >> mpstat shows 30% CPU is idle, >> >> 03:30:19 PM CPU %usr %nice %sys %iowait %irq %soft >> %steal %guest %idle >> 03:30:20 PM all 38.69 0.00 21.92 0.00 0.06 7.52 >> 0.00 0.00 31.80 >> 03:30:21 PM all 37.56 0.00 21.99 0.00 0.00 7.50 >> 0.00 0.00 32.95 >> >> and perf top shows, much time is wasted in scheduler_wait, in spin wait I >> guess. >> >> 9320.00 19.8% scheduler_wait >> /home/mingsong.cw/erlangr16b/lib/erlang/erts-5.10/bin/beam.smp >> 1813.00 3.9% process_main >> /home/mingsong.cw/erlangr16b/lib/erlang/erts-5.10/bin/beam.smp >> 1379.00 2.9% _spin_lock >> /usr/lib/debug/lib/modules/2.6.32-131.21.1.tb477.el6.x86_64/vmlinux >> 1201.00 2.6% schedule >> /home/mingsong.cw/erlangr16b/lib/erlang/erts-5.10/bin/beam.smp >> >> It seems the performance may be associated with scheduler_wait() and >> erts_check_io(), with a SystemTap script(attached at the end of this >> mail), we can find out how many times the system call epoll_wait is >> invoked by beam.smp and each time, how many revents it gets. >> >> cpu process times >> revents min max avg timeouts >> all >> 1754 128042 - - 73 3 >> [14] beam.smp 151 >> 14127 82 97 93 0 >> [ 5] beam.smp 142 >> 13291 83 97 93 0 >> [13] beam.smp 127 >> 11948 86 96 94 0 >> [ 6] beam.smp 127 >> 11836 81 96 93 0 >> [ 4] beam.smp 121 >> 11323 81 96 93 0 >> [15] beam.smp 117 >> 10935 83 96 93 0 >> [12] beam.smp 486 >> 10128 0 96 20 2 >> [ 1] beam.smp 71 >> 6549 71 100 92 0 >> [ 2] beam.smp 62 >> 5695 82 96 91 0 >> [ 7] beam.smp 55 >> 5102 81 95 92 0 >> [11] beam.smp 52 >> 4822 85 95 92 0 >> [ 9] beam.smp 52 >> 4799 85 95 92 0 >> [ 8] beam.smp 51 >> 4680 78 95 91 0 >> [10] beam.smp 49 >> 4508 85 97 92 0 >> [ 3] beam.smp 46 >> 4211 81 95 91 0 >> [ 0] beam.smp 44 >> 4088 83 95 92 0 >> >> The resuls shows, epoll_wait is invoked 1754 times each second, and >> get 73 io events in average. This is unacceptable for writing high >> performance server. Because if epoll_wait is invoked no more than 2k >> times per second, then read/write a packet would cost more than 500ms, >> which causes long delay and affects the throughput finally. >> >> The problem relies on there is only one global pollset in system wide, >> so at a time there is no more than one scheduler can call >> erts_check_io() to obtain pending io tasks from underlying pollset, >> and no scheduler can call erts_check_io() before all pending io >> tasks're processed, so for IO bounded application, it's very likely >> that a scheduler finish its own job, but must wait idly for other >> schedulers to complete theirs. >> >> Hence, we develops a patch to slove this problem, by having a pollset >> for each scheduler, so that each scheduler can invoke erts_check_io() >> on its own pollset concurrently. After a scheduler complete its tasks, >> it can invoke erts_check_io() immediately no matter what state other >> schedulers're in. This patch also handles port migration situation, >> all used file descriptors in each port're recorded, when a port is >> migrated, these >> fd 're removed from original scheduler's pollset, and added to new >> scheduler's. >> >> Bind port to scheduler together with process is also helpful to >> performance, it reduces the cost of thread switches and >> synchronization, and bound port won't be migrated between schedulers. >> >> After apply the two patches, with the same pressure(100 concurrent >> clients),epoll_wait is invoked 49332 times per second, and get 3 >> revents each time in average, that is to say, our server responds >> quicker and become more realtime. >> >> cpu process times >> revents min max avg timeouts >> all >> 49332 217186 - - 4 3 >> [ 2] beam.smp 3219 >> 16050 2 7 4 0 >> [11] beam.smp 4275 >> 16033 1 6 3 0 >> [ 8] beam.smp 4240 >> 15992 1 6 3 0 >> [ 9] beam.smp 4316 >> 15964 0 6 3 2 >> [10] beam.smp 4139 >> 15851 1 6 3 0 >> [ 3] beam.smp 4256 >> 15816 1 6 3 0 >> [ 1] beam.smp 3107 >> 15800 2 7 5 0 >> [ 0] beam.smp 3727 >> 15259 1 6 4 0 >> [ 7] beam.smp 2810 >> 14722 3 7 5 0 >> [13] beam.smp 1981 >> 11499 4 7 5 0 >> [15] beam.smp 2285 >> 10942 3 6 4 0 >> [14] beam.smp 2258 >> 10866 3 6 4 0 >> [ 4] beam.smp 2246 >> 10849 3 6 4 0 >> [ 6] beam.smp 2206 >> 10730 3 6 4 0 >> [12] beam.smp 2173 >> 10573 3 6 4 0 >> [ 5] beam.smp 2093 >> 10240 3 6 4 0 >> >> scheduler_wait no longer take so much time now, >> >> 169.00 6.2% process_main beam.smp >> 55.00 2.0% _spin_lock [kernel] >> 45.00 1.7% driver_deliver_term beam.smp >> >> so is idle CPU time >> 04:30:44 PM CPU %usr %nice %sys %iowait %irq %soft >> %steal %guest %idle >> 04:30:45 PM all 60.34 0.00 21.44 0.00 0.06 16.45 >> 0.00 0.00 1.71 >> 04:30:46 PM all 60.99 0.00 21.22 0.00 0.00 16.26 >> 0.00 0.00 1.52 >> >> and tcprstat shows, QPS is getting 100K, latency is less than 1 millisecond >> >> timestamp count max min avg med stddev 95_max 95_avg 95_std 99_max 99_avg >> 99_std >> 1341822689 96078 11592 314 910 817 311 1447 869 228 1777 897 263 >> 1341822690 100651 24245 209 914 819 381 1458 870 229 1800 899 263 >> >> I also write a extreamly simple keep-alive http server(attached at the >> end of mail), to compare performance before and after applying the >> patches, mearused with apache ab tool(ab -n 1000000 -c 100 -k ), a 30% >> performance gain can be get. >> >> before >> Requests per second: 103671.70 [#/sec] (mean) >> Time per request: 0.965 [ms] (mean) >> >> after >> Requests per second: 133701.24 [#/sec] (mean) >> Time per request: 0.748 [ms] (mean) >> >> Patches can be found at github, compile with >> ./configure CFLAGS=-DERTS_POLLSET_PER_SCHEDULER >> >> Pollset per scheduler: >> >> git fetch git://github.com/weicao/otp.git pollset_per_scheduler >> >> https://github.com/weicao/otp/compare/weicao:master...weicao:pollset_per_scheduler >> https://github.com/weicao/otp/compare/weicao:master...weicao:pollset_per_scheduler.patch >> >> >> Bind port to scheduler: >> >> git fetch git://github.com/weicao/otp.git bind_port_to_scheduler >> >> https://github.com/weicao/otp/compare/weicao:pollset_per_scheduler...weicao:bind_port_to_scheduler >> https://github.com/weicao/otp/compare/weicao:pollset_per_scheduler...weicao:bind_port_to_scheduler.patch >> >> >> Appendix: >> >> ----------------------------------- >> proxy.erl >> ------------------------------------ >> >> -module(proxy). >> -compile(export_all). >> >> -define(RECBUF_SIZE, 8192). >> -define(ACCEPT_TIMEOUT, 2000). >> >> start([MyPortAtom, DestIpAtom, DestPortAtom]) -> >> {MyPort, []} = string:to_integer(atom_to_list(MyPortAtom)), >> DestIp = atom_to_list(DestIpAtom), >> {DestPort, []} = string:to_integer(atom_to_list(DestPortAtom)), >> listen(MyPort, DestIp, DestPort), >> receive Any -> >> io:format("recv ~p~n", [Any]) >> end. >> >> listen(MyPort, DestIp, DestPort) -> >> io:format("start proxy on [local] 0.0.0.0:~p -> [remote] ~p:~p~n", >> [MyPort, DestIp, DestPort]), >> case gen_tcp:listen(MyPort, >> [inet, >> {ip, {0,0,0,0}}, >> binary, >> {reuseaddr, true}, >> {recbuf, ?RECBUF_SIZE}, >> {active, false}, >> {nodelay, true} >> ]) of >> {ok, Listen} -> >> N = erlang:system_info(schedulers), >> lists:foreach(fun(I) -> accept(Listen, DestIp, DestPort, >> I) end, lists:seq(1,N)); >> {error, Reason} -> >> io:format("error listen ~p~n", [Reason]) >> end. >> >> accept(Listen, DestIp, DestPort, I) -> >> spawn_opt(?MODULE, loop, [Listen, DestIp, DestPort, I], [{scheduler, >> I}]). >> >> loop(Listen, DestIp, DestPort, I) -> >> case gen_tcp:accept(Listen, ?ACCEPT_TIMEOUT) of >> {ok, S1} -> >> accept(Listen, DestIp, DestPort, I), >> case catch gen_tcp:connect(DestIp, DestPort, >> [inet, binary, {active, false}, >> {reuseaddr, true}, {nodelay, true}]) of >> {ok, S2} -> >> io:format("new connection~n"), >> loop1(S1, S2); >> {error, Reason} -> >> io:format("error connect ~p~n", [Reason]) >> end; >> {error, timeout} -> >> loop(Listen, DestIp, DestPort, I); >> Error -> >> io:format("error accept ~p~n", [Error]), >> accept(Listen, DestIp, DestPort, I) >> end. >> >> loop1(S1, S2) -> >> active(S1, S2), >> receive >> {tcp, S1, Data} -> >> gen_tcp:send(S2, Data), >> loop1(S1, S2); >> {tcp, S2, Data} -> >> gen_tcp:send(S1, Data), >> loop1(S1, S2); >> {tcp_closed, S1} -> >> io:format("S1 close~n"), >> gen_tcp:close(S1), >> gen_tcp:close(S2); >> {tcp_closed, S2} -> >> io:format("S2 close~n"), >> gen_tcp:close(S1), >> gen_tcp:close(S2) >> end. >> >> active(S1,S2)-> >> inet:setopts(S1, [{active, once}]), >> inet:setopts(S2, [{active, once}]). >> >> ----------------------------------- >> epollwait.stp >> ------------------------------------ >> #! /usr/bin/env stap >> # >> # >> >> global epoll_timeout_flag, epoll_count, epoll_min, epoll_max, >> epoll_times, epoll_timeouts >> >> probe syscall.epoll_wait { >> if(timeout > 0) { >> epoll_timeout_flag[pid()] = 1 >> } >> } >> >> probe syscall.epoll_wait.return { >> c = cpu() >> p = execname() >> >> epoll_times[c,p] ++ >> epoll_count[c,p] += $return >> >> if($return == 0 && pid() in epoll_timeout_flag) { >> epoll_timeouts[c,p] ++ >> delete epoll_timeout_flag[pid()] >> } >> >> if(!([c, p] in epoll_min)) { >> epoll_min[c,p] = $return >> } else if($return < epoll_min[c,p]) { >> epoll_min[c,p] = $return >> } >> >> >> if($return > epoll_max[c,p]) { >> epoll_max[c,p] = $return >> } >> } >> >> probe timer.s($1) { >> printf ("%4s %45s %10s %10s %10s %10s %10s %10s\n", "cpu", >> "process", "times", "revents", "min", "max", "avg", "timeouts" ) >> foreach ([cpu, process] in epoll_count- limit 20) { >> all_epoll_times += epoll_times[cpu,process] >> all_epoll_count += epoll_count[cpu,process] >> all_epoll_timeouts += epoll_timeouts[cpu,process] >> } >> printf ("%4s %45s %10d %10d %10s %10s %10d %10d\n", >> "all", "", all_epoll_times, all_epoll_count, "-", "-", >> all_epoll_count == 0? 0:all_epoll_count/all_epoll_times, >> all_epoll_timeouts) >> >> foreach ([cpu, process] in epoll_count- limit 20) { >> printf ("[%2d] %45s %10d %10d %10d %10d %10d %10d\n", >> cpu, process, epoll_times[cpu, process], epoll_count[cpu, process], >> epoll_min[cpu, process], epoll_max[cpu, process], >> epoll_count[cpu,process]/epoll_times[cpu,process], >> epoll_timeouts[cpu,process]) >> } >> delete epoll_count >> delete epoll_min >> delete epoll_max >> delete epoll_times >> delete epoll_timeouts >> printf >> ("--------------------------------------------------------------------------\n\n" >> ) >> } >> >> ------------------------------------------------ >> ehttpd.erl >> ------------------------------------------------- >> -module(ehttpd). >> -compile(export_all). >> >> start() -> >> start(8888). >> start(Port) -> >> N = erlang:system_info(schedulers), >> listen(Port, N), >> io:format("ehttpd ready with ~b schedulers on port ~b~n", [N, Port]), >> >> register(?MODULE, self()), >> receive Any -> io:format("~p~n", [Any]) end. %% to stop: ehttpd!stop. >> >> listen(Port, N) -> >> Opts = [inet, >> binary, >> {active, false}, >> {recbuf, 8192}, >> {nodelay,true}, >> {reuseaddr, true}], >> >> {ok, S} = gen_tcp:listen(Port, Opts), >> lists:foreach(fun(I)-> spawn_opt(?MODULE, accept, [S, I], >> [{scheduler, I}]) end, lists:seq(1, N)). >> >> accept(S, I) -> >> case gen_tcp:accept(S) of >> {ok, Socket} -> >> spawn_opt(?MODULE, accept, [S, I],[{scheduler,I}] ), >> io:format("new connection @~p~n", [I]), >> loop(Socket,<<>>); >> Error -> erlang:error(Error) >> end. >> >> loop(S,B) -> >> inet:setopts(S, [{active, once}]), >> receive >> {tcp, S, Data} -> >> B1 = <>, >> case binary:part(B1,{byte_size(B1), -4}) of >> <<"\r\n\r\n">> -> >> Response = <<"HTTP/1.1 200 OK\r\nContent-Length: >> 12\r\nConnection: Keep-Alive\r\n\r\nhello world!">>, >> gen_tcp:send(S, Response), >> loop(S, <<>>); >> _ -> >> loop(S, B1) >> end; >> {tcp_closed, S} -> >> io:format("connection closed forced~n"), >> gen_tcp:close(S); >> Error -> >> io:format("unexpected message~p~n", [Error]), >> Error >> end. >> >> >> -- >> >> Best, >> >> Wei Cao >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> > > > > -- > > Best, > > Wei Cao -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmitrii@REDACTED Tue Jul 10 11:14:37 2012 From: dmitrii@REDACTED (Dmitrii Dimandt) Date: Tue, 10 Jul 2012 11:14:37 +0200 Subject: [erlang-questions] Looking for slides of a lightning talk In-Reply-To: <0d6c6c5d-2b0f-48da-bae8-f440d232df92@googlegroups.com> References: <4FFAAAEA.9060605@ninenines.eu> <0d6c6c5d-2b0f-48da-bae8-f440d232df92@googlegroups.com> Message-ID: I'll send the slides to everyone interested, and to the list as well :) > Heh. That was me :) > > I have the slides at home, and I *WILL* forget to send them to you :) So feel free to bug me directly until I do > > Someone did a lightning talk in Stockholm that was quite funny but also > quite true, saying that Erlang isn't ready for the web and doing > comparisons with other languages/platforms. Any chance I could get the > slides? > > Thanks. > > -- > Lo?c Hoguin > Erlang Cowboy > Nine Nines > http://ninenines.eu > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From watson.timothy@REDACTED Tue Jul 10 11:15:56 2012 From: watson.timothy@REDACTED (Tim Watson) Date: Tue, 10 Jul 2012 10:15:56 +0100 Subject: [erlang-questions] Looking for slides of a lightning talk In-Reply-To: References: <4FFAAAEA.9060605@ninenines.eu> <0d6c6c5d-2b0f-48da-bae8-f440d232df92@googlegroups.com> Message-ID: Yes please. :) On 10 Jul 2012, at 10:14, Dmitrii Dimandt wrote: > I'll send the slides to everyone interested, and to the list as well :) > >> Heh. That was me :) >> >> I have the slides at home, and I *WILL* forget to send them to you :) So feel free to bug me directly until I do >> >> Someone did a lightning talk in Stockholm that was quite funny but also >> quite true, saying that Erlang isn't ready for the web and doing >> comparisons with other languages/platforms. Any chance I could get the >> slides? >> >> Thanks. >> >> -- >> Lo?c Hoguin >> Erlang Cowboy >> Nine Nines >> http://ninenines.eu >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From eriksoe@REDACTED Tue Jul 10 11:19:56 2012 From: eriksoe@REDACTED (=?ISO-8859-1?Q?Erik_S=F8e_S=F8rensen?=) Date: Tue, 10 Jul 2012 11:19:56 +0200 Subject: [erlang-questions] specs for fixed length lists In-Reply-To: References: <671132D1-1699-4246-BC2A-9D49944F3267@gmail.com> Message-ID: Just read up on it; the dsetel pass doesn't have that kind of magic. Specifically, - it can only handle fixed indexes; - the resulting destructive update doesn't check the tuple size, so the elements must be updated in a descending-index order; - there must be no chance of GC kicking in between creation of the tuple and the destructive update (or else there might be introduced an old-generation-to-new-generation pointer). So not only doesn't dsetel at present optimize setelement calls across a loop, it is not likely to get that ability in the future either. Apparently dsetel is rather narrowly aimed at optimizing record updates; it doesn't try to be much cleverer than needed for that job. 2012/7/10 Erik S?e S?rensen > It can optimize setelement used in a loop? I'd be surprised, although > pleasantly so. > (I didn't even know it could handle non-fixed indexes, but then I am not > generally very familiar with it.) > One more thing to research... > Den 10/07/2012 10.10 skrev "Anthony Ramine" : > > I wouldn't be so sure about that, you need to check whether the >> destructive updates pass (dsetel) does its magic here. >> >> -- >> Anthony Ramine >> >> Le 10 juil. 2012 ? 01:50, Erik S?e S?rensen a ?crit : >> >> > Using setelement is a bad idea, at least unless the tuples are very >> small: the cost in time and allocation is n^2. >> > A better scaling approach is to use tuple_to_list + map + >> list_to_tuple; that should be linear, and my guess is that it'd be faster >> even for small tuple sizes. >> > (For the actual n=4 case, I don't know which is better; must >> measure...) >> > /Erik >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lambdadmitry@REDACTED Tue Jul 10 11:32:28 2012 From: lambdadmitry@REDACTED (Dmitry Groshev) Date: Tue, 10 Jul 2012 02:32:28 -0700 (PDT) Subject: [erlang-questions] Local namespaces in case/try In-Reply-To: <4FFBEFA5.6030305@gmail.com> References: <4FFBEFA5.6030305@gmail.com> Message-ID: Excuse me, Richard, but it's not an answer. On Tuesday, July 10, 2012 1:02:29 PM UTC+4, Richard Carlsson wrote: > > On 07/10/2012 10:43 AM, Dmitry Groshev wrote: > > case do_something() of > > {ok, Result} -> Result; > > {error, Error} -> Error > > end, > > case do_another() of > > {ok, Result} -> Result; > > {error, Error} -> Error > > end, > > > > Result and Error are bound in first case and we will probably have a > > match failure in second one. Compiler warns about this, but it's still > > very unwieldy to fix it with names like Error1, Error2, etc. > > Take it as a sign that you should break out those case expressions into > separate functions, or restructure the code in some other way. > > /Richard > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From superbobry@REDACTED Tue Jul 10 11:40:17 2012 From: superbobry@REDACTED (Sergei Lebedev) Date: Tue, 10 Jul 2012 13:40:17 +0400 Subject: [erlang-questions] Local namespaces in case/try In-Reply-To: <4FFBEFA5.6030305@gmail.com> References: <4FFBEFA5.6030305@gmail.com> Message-ID: Richard, While this indeed *may* be a sign of a problem in your code, for the original example this is hardly the case. Splitting the function apart *only* because of compiler limitations sounds like an ad-hoc hack, not a real solution. -- Sergei On Jul 10, 2012, at 1:02 PM, Richard Carlsson wrote: > On 07/10/2012 10:43 AM, Dmitry Groshev wrote: >> case do_something() of >> {ok, Result} -> Result; >> {error, Error} -> Error >> end, >> case do_another() of >> {ok, Result} -> Result; >> {error, Error} -> Error >> end, >> >> Result and Error are bound in first case and we will probably have a >> match failure in second one. Compiler warns about this, but it's still >> very unwieldy to fix it with names like Error1, Error2, etc. > > Take it as a sign that you should break out those case expressions into separate functions, or restructure the code in some other way. > > /Richard > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From ivan@REDACTED Tue Jul 10 11:54:11 2012 From: ivan@REDACTED (Ivan Uemlianin) Date: Tue, 10 Jul 2012 10:54:11 +0100 Subject: [erlang-questions] Local namespaces in case/try In-Reply-To: References: <4FFBEFA5.6030305@gmail.com> Message-ID: <4FFBFBC3.30207@llaisdy.com> > On 07/10/2012 10:43 AM, Dmitry Groshev wrote: > case do_something() of > {ok, Result} -> Result; > {error, Error} -> Error > end, > case do_another() of > {ok, Result} -> Result; > {error, Error} -> Error > end, > > Result and Error are bound in first case and we will probably have a > match failure in second one. Compiler warns about this, but it's still > very unwieldy to fix it with names like Error1, Error2, etc. Are you expecting that the "Result" variable returned from do_another() should have the same value as the "Result" returned from do_something()? (and likewise for Error.) If so, I don't see why you would get a match failure: e.g., you surely wouldn't get a match failure if do_something() and do_another() both returned {ok, 5}? If not, yes you will get match failures. To this naive observer, the code looks like the "Result"s (and the "Errors"s) are expected to be identical. Best wishes Ivan -- ============================================================ Ivan A. Uemlianin PhD Llaisdy Speech Technology Research and Development ivan@REDACTED www.llaisdy.com llaisdy.wordpress.com github.com/llaisdy www.linkedin.com/in/ivanuemlianin "hilaritas excessum habere nequit" (Spinoza, Ethica, IV, XLII) ============================================================ From lambdadmitry@REDACTED Tue Jul 10 12:00:01 2012 From: lambdadmitry@REDACTED (Dmitry Groshev) Date: Tue, 10 Jul 2012 03:00:01 -0700 (PDT) Subject: [erlang-questions] Local namespaces in case/try In-Reply-To: References: <4FFBEFA5.6030305@gmail.com> Message-ID: Exactly what I was talking about. I can't see any sense in bloating the code with nonsensical functions just to get something let-like -- AFAIK, every functional language out there has a local namespace for such constructs. Moreover, there is a huge inconsistency right now with list comprehensions that have their own namespace because of their transformation to functions: test1() -> [begin _A = X, ok end || X <- [1, 2, 3]], _A = 1. test2() -> case 2 of _A -> _A end, _A = 1. I think it's a sign of a bigger problem with Erlang as a language: we have an amazing BEAM, some brilliant libraries, a lot of godlike developers, but Erlang as a language evolves extremely slow. One reason that I can think about is a lack of ultimate authority that can decide what is right and what is wrong ? theoretically there is a EEP process (and some proposals are out of EEPs scope, like an e2 thing), but in practice some EEPs are here for years (like Frames proposal or JSON BIFs) and nothing really moves on. On Tuesday, July 10, 2012 1:40:17 PM UTC+4, Sergei Lebedev wrote: > > Richard, > > While this indeed *may* be a sign of a problem in your code, for the > original example this is hardly the case. Splitting the function apart > *only* because of compiler limitations sounds like an ad-hoc hack, not a > real solution. > > -- Sergei > > On Jul 10, 2012, at 1:02 PM, Richard Carlsson wrote: > > > On 07/10/2012 10:43 AM, Dmitry Groshev wrote: > >> case do_something() of > >> {ok, Result} -> Result; > >> {error, Error} -> Error > >> end, > >> case do_another() of > >> {ok, Result} -> Result; > >> {error, Error} -> Error > >> end, > >> > >> Result and Error are bound in first case and we will probably have a > >> match failure in second one. Compiler warns about this, but it's still > >> very unwieldy to fix it with names like Error1, Error2, etc. > > > > Take it as a sign that you should break out those case expressions into > separate functions, or restructure the code in some other way. > > > > /Richard > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > On Tuesday, July 10, 2012 1:40:17 PM UTC+4, Sergei Lebedev wrote: > > Richard, > > While this indeed *may* be a sign of a problem in your code, for the > original example this is hardly the case. Splitting the function apart > *only* because of compiler limitations sounds like an ad-hoc hack, not a > real solution. > > -- Sergei > > On Jul 10, 2012, at 1:02 PM, Richard Carlsson wrote: > > > On 07/10/2012 10:43 AM, Dmitry Groshev wrote: > >> case do_something() of > >> {ok, Result} -> Result; > >> {error, Error} -> Error > >> end, > >> case do_another() of > >> {ok, Result} -> Result; > >> {error, Error} -> Error > >> end, > >> > >> Result and Error are bound in first case and we will probably have a > >> match failure in second one. Compiler warns about this, but it's still > >> very unwieldy to fix it with names like Error1, Error2, etc. > > > > Take it as a sign that you should break out those case expressions into > separate functions, or restructure the code in some other way. > > > > /Richard > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lambdadmitry@REDACTED Tue Jul 10 12:10:45 2012 From: lambdadmitry@REDACTED (Dmitry Groshev) Date: Tue, 10 Jul 2012 03:10:45 -0700 (PDT) Subject: [erlang-questions] Local namespaces in case/try In-Reply-To: <4FFBFBC3.30207@llaisdy.com> References: <4FFBEFA5.6030305@gmail.com> <4FFBFBC3.30207@llaisdy.com> Message-ID: Ivan, There is a concept called "lexical scoping" ? it means that some lexical constructs have a local namespace. In that example I would expect that all variables that were introduced inside case statement (Error/Result) won't be bound after the statement. On top of that, Erlang compiler already has some similar logic ? it will warn you about "exported" variables and will not compile code with "unsafe" variables, so it's only a matter of a minor (and maybe optional) language change. I'm sorry but I can't see why do you think that that values should be identical. On Tuesday, July 10, 2012 1:54:11 PM UTC+4, Ivan Uemlianin wrote: > > > On 07/10/2012 10:43 AM, Dmitry Groshev wrote: > > case do_something() of > > {ok, Result} -> Result; > > {error, Error} -> Error > > end, > > case do_another() of > > {ok, Result} -> Result; > > {error, Error} -> Error > > end, > > > > Result and Error are bound in first case and we will probably have a > > match failure in second one. Compiler warns about this, but it's still > > very unwieldy to fix it with names like Error1, Error2, etc. > > Are you expecting that the "Result" variable returned from do_another() > should have the same value as the "Result" returned from do_something()? > (and likewise for Error.) > > If so, I don't see why you would get a match failure: e.g., you surely > wouldn't get a match failure if do_something() and do_another() both > returned {ok, 5}? > > If not, yes you will get match failures. > > To this naive observer, the code looks like the "Result"s (and the > "Errors"s) are expected to be identical. > > Best wishes > > Ivan > > > -- > ============================================================ > Ivan A. Uemlianin PhD > Llaisdy > Speech Technology Research and Development > > ivan@REDACTED > www.llaisdy.com > llaisdy.wordpress.com > github.com/llaisdy > www.linkedin.com/in/ivanuemlianin > > "hilaritas excessum habere nequit" > (Spinoza, Ethica, IV, XLII) > ============================================================ > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmitrii@REDACTED Tue Jul 10 12:19:01 2012 From: dmitrii@REDACTED (Dmitrii Dimandt) Date: Tue, 10 Jul 2012 12:19:01 +0200 Subject: [erlang-questions] Local namespaces in case/try In-Reply-To: <4FFBFBC3.30207@llaisdy.com> References: <4FFBEFA5.6030305@gmail.com> <4FFBFBC3.30207@llaisdy.com> Message-ID: <48509C31-8054-4F23-B24D-E716AF256A52@dmitriid.com> On Jul 10, 2012, at 11:54 , Ivan Uemlianin wrote: >> On 07/10/2012 10:43 AM, Dmitry Groshev wrote: >> case do_something() of >> {ok, Result} -> Result; >> {error, Error} -> Error >> end, >> case do_another() of >> {ok, Result} -> Result; >> {error, Error} -> Error >> end, >> >> Result and Error are bound in first case and we will probably have a >> match failure in second one. Compiler warns about this, but it's still >> very unwieldy to fix it with names like Error1, Error2, etc. > > Are you expecting that the "Result" variable returned from do_another() should have the same value as the "Result" returned from do_something()? (and likewise for Error.) > > If so, I don't see why you would get a match failure: e.g., you surely wouldn't get a match failure if do_something() and do_another() both returned {ok, 5}? > > If not, yes you will get match failures. > > To this naive observer, the code looks like the "Result"s (and the "Errors"s) are expected to be identical. It's easy to extend the example as Val1 = case do_something() of {ok, Result} -> Result; {error, Error} -> Error end, Val2 = case do_another() of {ok, Result} -> Result; {error, Error} -> Error end, Why would Result from the first case need to match Result in the second case is beyond my comprehension. Of course, we can do to the extreme and refactor each and every case into its own function. Should we? From ivan@REDACTED Tue Jul 10 12:20:44 2012 From: ivan@REDACTED (Ivan Uemlianin) Date: Tue, 10 Jul 2012 11:20:44 +0100 Subject: [erlang-questions] Local namespaces in case/try In-Reply-To: References: <4FFBEFA5.6030305@gmail.com> <4FFBFBC3.30207@llaisdy.com> Message-ID: <4FFC01FC.50801@llaisdy.com> On 10/07/2012 11:10, Dmitry Groshev wrote: > Ivan, > > There is a concept called "lexical scoping" ? it means that some lexical > constructs have a local namespace. In that example I would expect that > all variables that were introduced inside case statement (Error/Result) > won't be bound after the statement. In erlang it seems that they are bound: http://www.erlang.org/course/advanced.html#scope http://www.erlang.org/doc/reference_manual/expressions.html#id75881 > ... I'm sorry > but I can't see why do you think that that values should be identical. I am only a beginner. Ivan > > On Tuesday, July 10, 2012 1:54:11 PM UTC+4, Ivan Uemlianin wrote: > > > On 07/10/2012 10:43 AM, Dmitry Groshev wrote: > > case do_something() of > > {ok, Result} -> Result; > > {error, Error} -> Error > > end, > > case do_another() of > > {ok, Result} -> Result; > > {error, Error} -> Error > > end, > > > > Result and Error are bound in first case and we will probably have a > > match failure in second one. Compiler warns about this, but it's > still > > very unwieldy to fix it with names like Error1, Error2, etc. > > Are you expecting that the "Result" variable returned from do_another() > should have the same value as the "Result" returned from > do_something()? > (and likewise for Error.) > > If so, I don't see why you would get a match failure: e.g., you surely > wouldn't get a match failure if do_something() and do_another() both > returned {ok, 5}? > > If not, yes you will get match failures. > > To this naive observer, the code looks like the "Result"s (and the > "Errors"s) are expected to be identical. > > Best wishes > > Ivan > > > -- > ============================================================ > Ivan A. Uemlianin PhD > Llaisdy > Speech Technology Research and Development > > ivan@REDACTED > www.llaisdy.com > llaisdy.wordpress.com > github.com/llaisdy > www.linkedin.com/in/ivanuemlianin > > > "hilaritas excessum habere nequit" > (Spinoza, Ethica, IV, XLII) > ============================================================ > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- ============================================================ Ivan A. Uemlianin PhD Llaisdy Speech Technology Research and Development ivan@REDACTED www.llaisdy.com llaisdy.wordpress.com github.com/llaisdy www.linkedin.com/in/ivanuemlianin "hilaritas excessum habere nequit" (Spinoza, Ethica, IV, XLII) ============================================================ From carlsson.richard@REDACTED Tue Jul 10 12:30:15 2012 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Tue, 10 Jul 2012 12:30:15 +0200 Subject: [erlang-questions] Local namespaces in case/try In-Reply-To: References: <4FFBEFA5.6030305@gmail.com> Message-ID: <4FFC0437.3060105@gmail.com> On 07/10/2012 11:40 AM, Sergei Lebedev wrote: > > While this indeed *may* be a sign of a problem in your code, for the > original example this is hardly the case. Splitting the function > apart *only* because of compiler limitations sounds like an ad-hoc > hack, not a real solution. What you have to understand first is that it's not a compiler limitation - it's how the language was *designed*. The question about local scopes is not new; it has been discussed several times over the years, and it has been decided that mixing local scopes with Erlang's existing scoping rules would create a schizofrenic language. Variable scoping in Erlang is different from most other languages, but it's not random or poorly designed. You just have to learn how to work with it, and the result is usually (although not always) more readable code, when you do it right. /Richard Here follows a quote from a mail that I had saved from 1996 by H?kan Millroth when the addition of let-expressions was discussed: "One of the really attractive characteristics of Erlang as we know it today, is that all constructs in the language has simple semantics that make sense also to imperative programmers. I understand that several quite radical extensions to the language are now being considered by the Erlang language designers. Several of these extensions are, in my opinion, bad ideas. Below I explain why. [I am not 100% sure that I describe the proposed extensions correctly below. The reason for this is that there is, as far as I know, no "official" description of the extensions.] Extension 1: let Pat_1 = Expr_1, ... Pat_n = Expr_n in BodyExpr end This introduces a new scope for the variables in Pat_1,...,Pat_n. In most other functional langauges, this is how you introduce new variables. It works just fine in these languages. Clean and nice. BUT: Introduce this construct in Erlang and you get a bad design. The principle in Erlang today is that a variable exists from its first occurrence to its last occurrence in a clause. Very simple (maybe not as elegant as the first method, but simple). It is also consistent that case/if/receive follow this principle; this is not at all inelegant if we consider case/if/receive as control structures in the language. When we mix these two principles in one language, we get an inconsistent and incoherent language design. My advice to Erlang language (re)designers is: stick to one of the principles. If you prefer the first one, then you should design a NEW language which is consistent with that principle. ...[discussion of other language extensions removed]" (I could note that another of H?kan's comments was: "'++' is OK. However, there are several reasonable interpretations of what '--' should do, none of them more natural than the others. Please, keep your feet on the ground and forget '--'." Sadly, the advice was not taken.) From superbobry@REDACTED Tue Jul 10 12:52:30 2012 From: superbobry@REDACTED (Sergei Lebedev) Date: Tue, 10 Jul 2012 14:52:30 +0400 Subject: [erlang-questions] Local namespaces in case/try In-Reply-To: <4FFC0437.3060105@gmail.com> References: <4FFBEFA5.6030305@gmail.com> <4FFC0437.3060105@gmail.com> Message-ID: <283C9941-B1E9-473B-A557-D18D51E8B56A@gmail.com> Your argument makes sense, but I wonder why list comprehensions don't follow this design decision? Of course, from the implementation point of view the answer is obvious -- list comprehensions are compiled to functions, which indeed have a separate namespace. But for me, as a user, it's one more rule to learn; I think scoping rules should be consistent, especially if the goal is to keep Erlang semantics as simple as possible. -- Sergei On Jul 10, 2012, at 2:30 PM, Richard Carlsson wrote: > On 07/10/2012 11:40 AM, Sergei Lebedev wrote: >> >> While this indeed *may* be a sign of a problem in your code, for the >> original example this is hardly the case. Splitting the function >> apart *only* because of compiler limitations sounds like an ad-hoc >> hack, not a real solution. > > What you have to understand first is that it's not a compiler limitation - it's how the language was *designed*. The question about local scopes is not new; it has been discussed several times over the years, and it has been decided that mixing local scopes with Erlang's existing scoping rules would create a schizofrenic language. Variable scoping in Erlang is different from most other languages, but it's not random or poorly designed. You just have to learn how to work with it, and the result is usually (although not always) more readable code, when you do it right. > > /Richard > > Here follows a quote from a mail that I had saved from 1996 by H?kan Millroth when the addition of let-expressions was discussed: > > "One of the really attractive characteristics of Erlang as we know it today, is that all constructs in the language has simple semantics > that make sense also to imperative programmers. > > I understand that several quite radical extensions to the language are now being considered by the Erlang language designers. Several of these extensions are, in my opinion, bad ideas. Below I explain why. > > [I am not 100% sure that I describe the proposed extensions correctly below. The reason for this is that there is, as far as I know, no "official" description of the extensions.] > > Extension 1: > > let > Pat_1 = Expr_1, > ... > Pat_n = Expr_n > in > BodyExpr > end > > This introduces a new scope for the variables in Pat_1,...,Pat_n. > > In most other functional langauges, this is how you introduce new variables. It works just fine in these languages. Clean and nice. BUT: Introduce this construct in Erlang and you get a bad design. > > The principle in Erlang today is that a variable exists from its first occurrence to its last occurrence in a clause. Very simple (maybe not as elegant as the first method, but simple). It is also consistent that case/if/receive follow this principle; this is not at all inelegant if we consider case/if/receive as control structures in the language. > > When we mix these two principles in one language, we get an inconsistent and incoherent language design. > > My advice to Erlang language (re)designers is: stick to one of the principles. If you prefer the first one, then you should design a NEW language which is consistent with that principle. > > ...[discussion of other language extensions removed]" > > (I could note that another of H?kan's comments was: "'++' is OK. However, there are several reasonable interpretations of what '--' should do, none of them more natural than the others. Please, keep your feet on the ground and forget '--'." Sadly, the advice was not taken.) > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From cgsmcmlxxv@REDACTED Tue Jul 10 12:58:11 2012 From: cgsmcmlxxv@REDACTED (CGS) Date: Tue, 10 Jul 2012 12:58:11 +0200 Subject: [erlang-questions] simple question about list memory consumption In-Reply-To: <8604E6F2-85ED-4CB4-8257-C216BCD6C713@cs.otago.ac.nz> References: <48564154-48ae-4d26-9a06-aa1091360cc0@knuth> <8604E6F2-85ED-4CB4-8257-C216BCD6C713@cs.otago.ac.nz> Message-ID: Well, I think I have to apologize. I had the impression that the machine I was working on was 64-bits. It is 32-bits, so, yes, 5 words per character means the list from the lists:seq/2 resides fully in memory. My bad, sorry. I did the test again using the loop from Richard's test and, indeed, I got about 80 MB memory consumption for 10^7 characters which is consistent with 2 words per character in the list. Now, to answer how free can be used in such a test: 1. execute command: watch -n 1 'free -m' (every 1 second calls free -m (output in MB) and reports its output); 2. stabilize your OS by killing all the processes which have a non-constant memory usage at the level of MB (that is, for several seconds the output of the previous command should remain constant); 3. start an Erlang shell (that adds usually something like 4 MB to the used memory); 4. load the module (usually not visible in the output of (1), but it may add 1 MB, depending on the system state by that time); 5. execute `:loop(10000000), ok.'; 6. the difference in between the output from (1) after step (5) and the output from (1) after step (4) is giving the memory used by Erlang for the list (+ some overhead). This is much of how Richard did it in Erlang, but using external tools to monitor the RAM consumption. I repeated several times this and I got almost the same result (1 MB error, but that may be an artifact of the shell or system). Again, I am sorry for my mistake thinking that the sequence list was empty when lists:map/2 exited, but I was really under the impression that the machine I was working on is 64-bits (1 word = 8 B). And thank you for spotting my mistakes in judgement. CGS On Tue, Jul 10, 2012 at 3:21 AM, Richard O'Keefe wrote: > > On 9/07/2012, at 10:34 PM, CGS wrote: > > > About the two lists you were speaking about (I suppose you were > referring to the output of lists:map/2 and lists:seq/2), the second is > destroyed as soon as lists:map/2 exits. > > > > No it isn't. It is only reclaimed when the garbage collector runs. > > > > That is a bit puzzling. > > It seems that you are muddling up "is LIVE" with "EXISTS". > When you compute > map(F, L) > and there are no other references to L anywhere, > L becomes *DEAD* as soon as the call to map/2 completes. > (In fact, each cell of L becomes dead in turn during the > call.) But DEAD and DESTROYED are very different properties. > L is not "destroyed" until the garbage collector runs; > until the garbage collector runs the cells of L still occupy > space on the heap. When, if ever, that space is reclaimed > depends on how much other need there is for it. > > > Looking at the code for lists:map/2: > > > > map(F, [H|T]) -> > > [F(H)|map(F, T)]; > > map(F, []) when is_function(F, 1) -> []. > > > > I can see that when lists:map/2 exits, the only remaining list is an > empty list. > > No you can't see that, because there is nothing in the code > for map/2 to release any storage whatever. > > > I might be wrong, but I see only one word remaining for the garbage > collector to take care of. > > What do you think makes the other words go away? > > > > > > You got the result from process_info, I got it from `free' under Linux. > > Not the best way to do it. That tells you how much memory > the operating has handed out but not to which process; even > ps -o size tells you only how much memory Erlang has claimed from > the operating system, and nothing about what Erlang is doing with > that memory. > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlsson.richard@REDACTED Tue Jul 10 13:03:36 2012 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Tue, 10 Jul 2012 13:03:36 +0200 Subject: [erlang-questions] Local namespaces in case/try In-Reply-To: <283C9941-B1E9-473B-A557-D18D51E8B56A@gmail.com> References: <4FFBEFA5.6030305@gmail.com> <4FFC0437.3060105@gmail.com> <283C9941-B1E9-473B-A557-D18D51E8B56A@gmail.com> Message-ID: <4FFC0C08.6010301@gmail.com> On 07/10/2012 12:52 PM, Sergei Lebedev wrote: > Your argument makes sense, but I wonder why list comprehensions don't > follow this design decision? Of course, from the implementation point > of view the answer is obvious -- list comprehensions are compiled to > functions, which indeed have a separate namespace. But for me, as a > user, it's one more rule to learn; I think scoping rules should be > consistent, especially if the goal is to keep Erlang semantics as > simple as possible. Funs and list comprehensions are indeed exceptions, and they do somewhat complicate the language. Not everyone was happy when funs were added, for example. I think funs tend to work because they are not so heavily used, and you do get warnings about shadowed variables - something that I personally thought was stupid to begin with, but eventually I realized that taken with Erlang's normal scoping rules, it's best to avoid variable shadowing to keep the code readable and maintainable. When it comes to list comprehensions, I think that they were added a bit too quickly without much peer review of the finer details. It is nice to be able to have only local scope for variables bound in generators, but on the other hand it trips people up sometimes. But list comprehensions are well delimited visually and don't usually cause confusion. /Richard From send2philip@REDACTED Tue Jul 10 13:18:06 2012 From: send2philip@REDACTED (Philip Clarke) Date: Tue, 10 Jul 2012 12:18:06 +0100 Subject: [erlang-questions] What is the best location to place common type specs? In-Reply-To: <4715714536353355546@unknownmsgid> References: <06139A918ACCA041BF46A0F36940C7FA39CE0E22@exch-mbx1.msk.trd.ru> <4715714536353355546@unknownmsgid> Message-ID: Hi, Some useful suggestions are given in response to this question, but I'd also like to know if it is ok to define a spec in a .hrl file and use the record in multiple modules. This is how I have been doing it: In the .hrl file: ---------------------------- -record(my_record, { a = 1 :: integer() }). -type(my_record() :: #my_record{}). In the .erl file: ----------------------------- -module(spec_test). -include("spec_test.hrl"). -export([start/0]). -spec(start() -> {ok, MyRecord :: my_record()}). start() -> MyRecord = #my_record{}, {ok, MyRecord}. This certainly compiles, but I am not up to speed with tools like dialyzer to verify that this works. But this approach is correct, or ? Philip On Fri, Jul 6, 2012 at 10:08 PM, Max Bourinov wrote: > Hi Sergey, > > Dedicated module that holds types is your friend. > > Sent from my iPhone > > On 05.07.2012, at 16:07, Zhemzhitsky Sergey > wrote: > > Hi guys, > > > > I have multiple .hrl files with multiple records. I?d like to define > separate types for these records and their fields. > > These records will be used across multiple modules, and I?m wondering > whether it is legal to define type specs in the hrl files? > > Or would it be better to have a separate module with exported common type > specs? > > > > > > Best Regards, > > Sergey > > > > _______________________________________________________ > > > > The information contained in this message may be privileged and conf > idential and protected from disclosure. If you are not the original > intended recipient, you are hereby notified that any review, > retransmission, dissemination, or other use of, or taking of any action in > reliance upon, this information is prohibited. If you have received this > communication in error, please notify the sender immediately by replying to > this message and delete it from your computer. Thank you for your > cooperation. Troika Dialog, Russia. > > If you need assistance please contact our Contact Center (+7495) 258 0500or go to > www.troika.ru/eng/Contacts/system.wbp > > > > _______________________________________________ > > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.eugene.turner@REDACTED Tue Jul 10 13:43:30 2012 From: michael.eugene.turner@REDACTED (Michael Turner) Date: Tue, 10 Jul 2012 20:43:30 +0900 Subject: [erlang-questions] Local namespaces in case/try In-Reply-To: <4FFC0C08.6010301@gmail.com> References: <4FFBEFA5.6030305@gmail.com> <4FFC0437.3060105@gmail.com> <283C9941-B1E9-473B-A557-D18D51E8B56A@gmail.com> <4FFC0C08.6010301@gmail.com> Message-ID: I agree with Richard's reasoning, though I'm not sure I agree 100% with his prescription, in part because the following expression compiles just fine, even if it means something slightly different. If breaking the passage into two functions seems artificial, so what? You don't need to do that. ------- case do_something() of {error, Error} -> Error; {ok, Result} -> case do_another() of {ok, Result} -> Result; {error, Error} -> Error end end ------- And it's a difference that makes a (good) difference: why are you throwing out the first Error value anyway? If the first case/end clause throws away its values, why not just have "case do_something() of {error, _} -> ok; {ok, _} -> ok end", with a reasonable exception being thrown only when you don't get either of those two to match? And if you REALLY don't care what do_something/0 returns here, just call it w/o checking return value. -michael turner On Tue, Jul 10, 2012 at 8:03 PM, Richard Carlsson wrote: > On 07/10/2012 12:52 PM, Sergei Lebedev wrote: >> >> Your argument makes sense, but I wonder why list comprehensions don't >> follow this design decision? Of course, from the implementation point >> of view the answer is obvious -- list comprehensions are compiled to >> functions, which indeed have a separate namespace. But for me, as a >> user, it's one more rule to learn; I think scoping rules should be >> consistent, especially if the goal is to keep Erlang semantics as >> simple as possible. > > > Funs and list comprehensions are indeed exceptions, and they do somewhat > complicate the language. Not everyone was happy when funs were added, for > example. I think funs tend to work because they are not so heavily used, and > you do get warnings about shadowed variables - something that I personally > thought was stupid to begin with, but eventually I realized that taken with > Erlang's normal scoping rules, it's best to avoid variable shadowing to keep > the code readable and maintainable. > > When it comes to list comprehensions, I think that they were added a bit too > quickly without much peer review of the finer details. It is nice to be able > to have only local scope for variables bound in generators, but on the other > hand it trips people up sometimes. But list comprehensions are well > delimited visually and don't usually cause confusion. > > /Richard > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From michael.eugene.turner@REDACTED Tue Jul 10 13:50:38 2012 From: michael.eugene.turner@REDACTED (Michael Turner) Date: Tue, 10 Jul 2012 20:50:38 +0900 Subject: [erlang-questions] Local namespaces in case/try In-Reply-To: References: <4FFBEFA5.6030305@gmail.com> <4FFC0437.3060105@gmail.com> <283C9941-B1E9-473B-A557-D18D51E8B56A@gmail.com> <4FFC0C08.6010301@gmail.com> Message-ID: Oops. On second thought, the first {ok, Result} in my suggested code should really be "{ok, _}". Unless there's really some need to know both the Result from do_something/0 and the Error from do_another/0, in which case I suppose the original complaint still applies. -michael turner On Tue, Jul 10, 2012 at 8:43 PM, Michael Turner wrote: > I agree with Richard's reasoning, though I'm not sure I agree 100% > with his prescription, in part because the following expression > compiles just fine, even if it means something slightly different. If > breaking the passage into two functions seems artificial, so what? You > don't need to do that. > ------- > ------- > > And it's a difference that makes a (good) difference: why are you > throwing out the first Error value anyway? If the first case/end > clause throws away its values, why not just have "case do_something() > of {error, _} -> ok; {ok, _} -> ok end", with a reasonable exception > being thrown only when you don't get either of those two to match? And > if you REALLY don't care what do_something/0 returns here, just call > it w/o checking return value. > > -michael turner > > On Tue, Jul 10, 2012 at 8:03 PM, Richard Carlsson > wrote: >> On 07/10/2012 12:52 PM, Sergei Lebedev wrote: >>> >>> Your argument makes sense, but I wonder why list comprehensions don't >>> follow this design decision? Of course, from the implementation point >>> of view the answer is obvious -- list comprehensions are compiled to >>> functions, which indeed have a separate namespace. But for me, as a >>> user, it's one more rule to learn; I think scoping rules should be >>> consistent, especially if the goal is to keep Erlang semantics as >>> simple as possible. >> >> >> Funs and list comprehensions are indeed exceptions, and they do somewhat >> complicate the language. Not everyone was happy when funs were added, for >> example. I think funs tend to work because they are not so heavily used, and >> you do get warnings about shadowed variables - something that I personally >> thought was stupid to begin with, but eventually I realized that taken with >> Erlang's normal scoping rules, it's best to avoid variable shadowing to keep >> the code readable and maintainable. >> >> When it comes to list comprehensions, I think that they were added a bit too >> quickly without much peer review of the finer details. It is nice to be able >> to have only local scope for variables bound in generators, but on the other >> hand it trips people up sometimes. But list comprehensions are well >> delimited visually and don't usually cause confusion. >> >> /Richard >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions From erlang@REDACTED Tue Jul 10 14:41:54 2012 From: erlang@REDACTED (Joe Armstrong) Date: Tue, 10 Jul 2012 14:41:54 +0200 Subject: [erlang-questions] Local namespaces in case/try In-Reply-To: References: <4FFBEFA5.6030305@gmail.com> Message-ID: On Tue, Jul 10, 2012 at 12:00 PM, Dmitry Groshev wrote: > Exactly what I was talking about. I can't see any sense in bloating the code > with nonsensical functions just to get something let-like -- AFAIK, every > functional language out there has a local namespace for such constructs. > Moreover, there is a huge inconsistency right now with list comprehensions > that have their own namespace because of their transformation to functions: > > test1() -> > [begin > _A = X, > ok > end || X <- [1, 2, 3]], > _A = 1. > > test2() -> > case 2 of > _A -> _A > end, > _A = 1. > > I think it's a sign of a bigger problem with Erlang as a language: we have > an amazing BEAM, some brilliant libraries, a lot of godlike developers, but > Erlang as a language evolves extremely slow. One reason that I can think > about is a lack of ultimate authority that can decide what is right and what > is wrong ? theoretically there is a EEP process (and some proposals are out > of EEPs scope, like an e2 thing), but in practice some EEPs are here for > years (like Frames proposal or JSON BIFs) and nothing really moves on. No that's not the reason. There is no ultimate authority. Change is driven by the interplay of several factors: - consensus (ie general agreement that something is good) - payment (the people who finance Erlang pay to have something done) - voluntary effort (individuals make changes, which are then adopted) As regards the changes you suggest: 1) Millions of lines of code have been written with the assumption that if you see a variable twice in the same lexical scope it has the same value. Changing this would mean that all old code must be retested and modified. 2) When funs and list comprehensions where introduce the lexical scoping rules where changed, but there were no requirements for backwards compatibility 3) A change like this invalidate all existing Erlang books and all books in the pipeline. Lack of good documentation is probably our single greatest problem, so we don't like changes that invalidate all old documents. 4) Something like frames and JSON bifs is being implemented and will be released as soon as it is stable. And yes there is slow progress on things that are not financed. The priorities for development are set by the people who pay for the development. This means in practice that stuff that is need for Ericsson products gets prioritized. 5) Slow development of a language is a sign of maturity - languages that are over twenty years old don't change quickly. Most effort has not gone into language development but into the run-time system. As each new generation of multicores has come along the system has evolved to accommodate them We want to keep the language constant, while adapting the run-time system to modern architectures. This takes priority over language changes. 6) If you made a change like this and distributed the code somebody who read the code later would not know what the code meant. You can't have the same syntax for something in a program that means two different things at two different times. 7) I'm not against change as such - but we do have to take backwards comparability seriously for many reasons. We want to improve the system, subject to the condition that we don't break the existing stuff. Even small change require a lot of work, it's not just a question of changing the parser and compiler. All third-part tools have to be changed the dialyzer, quickcheck, eclipse IDE, refactoring tools etc. 8) Erlang could do with some major changes - I proposed erl2 a while back but these changes are too dramatic to be accommodated in a incremental and backwards compatible manner. I perceive erl2 as a code generator that produces regular erlang programs. /Joe > > On Tuesday, July 10, 2012 1:40:17 PM UTC+4, Sergei Lebedev wrote: >> >> Richard, >> >> While this indeed *may* be a sign of a problem in your code, for the >> original example this is hardly the case. Splitting the function apart >> *only* because of compiler limitations sounds like an ad-hoc hack, not a >> real solution. >> >> -- Sergei >> >> On Jul 10, 2012, at 1:02 PM, Richard Carlsson wrote: >> >> > On 07/10/2012 10:43 AM, Dmitry Groshev wrote: >> >> case do_something() of >> >> {ok, Result} -> Result; >> >> {error, Error} -> Error >> >> end, >> >> case do_another() of >> >> {ok, Result} -> Result; >> >> {error, Error} -> Error >> >> end, >> >> >> >> Result and Error are bound in first case and we will probably have a >> >> match failure in second one. Compiler warns about this, but it's still >> >> very unwieldy to fix it with names like Error1, Error2, etc. >> > >> > Take it as a sign that you should break out those case expressions into >> > separate functions, or restructure the code in some other way. >> > >> > /Richard >> > _______________________________________________ >> > erlang-questions mailing list >> > erlang-questions@REDACTED >> > http://erlang.org/mailman/listinfo/erlang-questions >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > > On Tuesday, July 10, 2012 1:40:17 PM UTC+4, Sergei Lebedev wrote: >> >> Richard, >> >> While this indeed *may* be a sign of a problem in your code, for the >> original example this is hardly the case. Splitting the function apart >> *only* because of compiler limitations sounds like an ad-hoc hack, not a >> real solution. >> >> -- Sergei >> >> On Jul 10, 2012, at 1:02 PM, Richard Carlsson wrote: >> >> > On 07/10/2012 10:43 AM, Dmitry Groshev wrote: >> >> case do_something() of >> >> {ok, Result} -> Result; >> >> {error, Error} -> Error >> >> end, >> >> case do_another() of >> >> {ok, Result} -> Result; >> >> {error, Error} -> Error >> >> end, >> >> >> >> Result and Error are bound in first case and we will probably have a >> >> match failure in second one. Compiler warns about this, but it's still >> >> very unwieldy to fix it with names like Error1, Error2, etc. >> > >> > Take it as a sign that you should break out those case expressions into >> > separate functions, or restructure the code in some other way. >> > >> > /Richard >> > _______________________________________________ >> > erlang-questions mailing list >> > erlang-questions@REDACTED >> > http://erlang.org/mailman/listinfo/erlang-questions >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From thomasl_erlang@REDACTED Tue Jul 10 17:23:04 2012 From: thomasl_erlang@REDACTED (Thomas Lindgren) Date: Tue, 10 Jul 2012 08:23:04 -0700 (PDT) Subject: [erlang-questions] Local namespaces in case/try In-Reply-To: References: <4FFBEFA5.6030305@gmail.com> Message-ID: <1341933784.8337.YahooMailNeo@web111409.mail.gq1.yahoo.com> ----- Original Message ----- > From: Joe Armstrong > ... > There is no ultimate authority. Change is driven by the interplay of > several factors: > > ? ? ? - consensus (ie general agreement that something is good) > ? ? ? - payment (the people who finance Erlang pay to have something done) > ? ? ? - voluntary effort (individuals make changes, which are then adopted) True. Still, I might agree with the OP in that Erlang probably would gain by having a "chief architect" or someone along those lines to make the final decision. (Also regarding BIFs, APIs, ...)? > As regards the changes you suggest: > > 1) Millions of lines of code have been written with the assumption > that if you see a variable twice > in the same lexical scope it has the same value. Changing this would > mean that all old code > must be retested and modified. On the other hand, there _are_ ways to deprecate things these days. Erlang has replaced the original type tests with long-winded ones, for instance. Aren't the old tests even illegal? nowadays? Another more interesting example is that of how users were discouraged from exporting variables from cases.? I think this policy was?changed (for whatever reason) by nagging, ie, making the compiler warn when it occurred. The same could be done by emitting a warning when an already bound variable is matched. I personally wouldn't mind, actually -- this is IMO a marginal feature and appears to occasionally confuse even skilled practitioners,?for instance on a?silent pattern match failure. Best, Thomas From jbarrett@REDACTED Tue Jul 10 17:55:34 2012 From: jbarrett@REDACTED (Jeremey Barrett) Date: Tue, 10 Jul 2012 10:55:34 -0500 Subject: [erlang-questions] Local namespaces in case/try In-Reply-To: <48509C31-8054-4F23-B24D-E716AF256A52@dmitriid.com> References: <4FFBEFA5.6030305@gmail.com> <4FFBFBC3.30207@llaisdy.com> <48509C31-8054-4F23-B24D-E716AF256A52@dmitriid.com> Message-ID: <5E5C827E-80AC-470E-A37C-0021447119BE@alertlogic.com> Hi all, +1 for real let. However, you can do something like this: -define(lets(C), begin fun() -> C end() end). Val1 = ?lets(case do_something() of {ok, Result} -> Result; {error, Error} -> Error end). Val2 = ?lets(case do_another() of {ok, Result} -> Result; {error, Error} -> Error end). Only Val1 and Val2 are bound outside the macro calls. Regards, Jeremey. On Jul 10, 2012, at 5:19 AM, Dmitrii Dimandt wrote: > Val1 = case do_something() of > {ok, Result} -> Result; > {error, Error} -> Error > end, > Val2 = case do_another() of > {ok, Result} -> Result; > {error, Error} -> Error > end, From jbarrett@REDACTED Tue Jul 10 18:35:46 2012 From: jbarrett@REDACTED (Jeremey Barrett) Date: Tue, 10 Jul 2012 11:35:46 -0500 Subject: [erlang-questions] Local namespaces in case/try In-Reply-To: <5E5C827E-80AC-470E-A37C-0021447119BE@alertlogic.com> References: <4FFBEFA5.6030305@gmail.com> <4FFBFBC3.30207@llaisdy.com> <48509C31-8054-4F23-B24D-E716AF256A52@dmitriid.com> <5E5C827E-80AC-470E-A37C-0021447119BE@alertlogic.com> Message-ID: <5373C811-9CFC-4C88-AD98-4B422C04CD09@alertlogic.com> ...and the begin/end are superfluous (leftovers from another macro), so: -define(lets(C), fun() -> C end()). Obviously new bindings performed inside the ?lets() call do not leak out, but anything already bound is already bound (you're getting a closure). Jeremey. On Jul 10, 2012, at 10:55 AM, Jeremey Barrett wrote: > -define(lets(C), begin fun() -> C end() end). From watson.timothy@REDACTED Tue Jul 10 19:20:39 2012 From: watson.timothy@REDACTED (Tim Watson) Date: Tue, 10 Jul 2012 18:20:39 +0100 Subject: [erlang-questions] Reading comments/annotations inside parse_transforms In-Reply-To: References: Message-ID: <043C9387-7B91-4FE1-9D00-CED8C3E0EC4D@gmail.com> Does anyone have any suggestions about this, or do I just resort to brute force to get it done? On 5 Jul 2012, at 10:10, Tim Watson wrote: > There doesn't appear to be any way of doing this currently. There is > support in erl_syntax for working with these and there is also an > erl_comment_scan module in the syntax-tools application which parses > and returns comments from a file (or whatever). What there doesn't > appear to be, is a means to get erl_parse to return comments as > attributes (?) in the AST that the compiler passes to a > parse_transform. > > My question then, is this: what is the best (only!?) way to process > comments at the same time as source code in a parse_transform? So far, > given that this seems unlikely to work OOTB, I'm assuming I'll have to > do something like: > > 1. once you hit the 'file' attribute, parse the comments and stash > these away in a useful place > 2. process the forms as usual, comparing each line number for each > form, against the line numbers stored for the comments > 3. when you get an interleaving of code and comments, you know where > the comments reside in the source file in relation to the form you're > currently working with > > I *can* do this of course, but it seems like an awful lot of hard > work. Why can't comments be preserved in the AST passed to a parse > transform, and then just dropped later on!? Is there an easier way of > doing this that I'm currently missing? > > Cheers, > Tim From dmitrii@REDACTED Tue Jul 10 20:39:24 2012 From: dmitrii@REDACTED (Dmitrii Dimandt) Date: Tue, 10 Jul 2012 20:39:24 +0200 Subject: [erlang-questions] Looking for slides of a lightning talk In-Reply-To: <489B3810-49B2-4085-B468-DA037F7DE646@dmitriid.com> References: <4FFAAAEA.9060605@ninenines.eu> <0d6c6c5d-2b0f-48da-bae8-f440d232df92@googlegroups.com> <489B3810-49B2-4085-B468-DA037F7DE646@dmitriid.com> Message-ID: <1709242D-0AD5-4070-A6EC-AB039E62ADFD@dmitriid.com> Ok, just the slides are here: http://www.slideshare.net/dmitriid/erlang-sucks-euc-2012 (you can download them there as well) A PDF with some notes is here: http://www.scribd.com/doc/99721085/Erlang-Sucks-EUC-2012 Notes isn't as good as talking in from of the audience, but oh well... > >> Heh. That was me :) >> >> I have the slides at home, and I *WILL* forget to send them to you :) So feel free to bug me directly until I do >> >> Someone did a lightning talk in Stockholm that was quite funny but also >> quite true, saying that Erlang isn't ready for the web and doing >> comparisons with other languages/platforms. Any chance I could get the >> slides? >> >> Thanks. >> >> -- >> Lo?c Hoguin >> Erlang Cowboy >> Nine Nines >> http://ninenines.eu >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlsson.richard@REDACTED Tue Jul 10 21:20:13 2012 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Tue, 10 Jul 2012 21:20:13 +0200 Subject: [erlang-questions] Reading comments/annotations inside parse_transforms In-Reply-To: References: Message-ID: <4FFC806D.7090805@gmail.com> On 07/05/2012 11:10 AM, Tim Watson wrote: > There doesn't appear to be any way of doing this currently. There is > support in erl_syntax for working with these and there is also an > erl_comment_scan module in the syntax-tools application which parses > and returns comments from a file (or whatever). What there doesn't > appear to be, is a means to get erl_parse to return comments as > attributes (?) in the AST that the compiler passes to a > parse_transform. > > My question then, is this: what is the best (only!?) way to process > comments at the same time as source code in a parse_transform? So far, > given that this seems unlikely to work OOTB, I'm assuming I'll have to > do something like: > > 1. once you hit the 'file' attribute, parse the comments and stash > these away in a useful place > 2. process the forms as usual, comparing each line number for each > form, against the line numbers stored for the comments > 3. when you get an interleaving of code and comments, you know where > the comments reside in the source file in relation to the form you're > currently working with > > I *can* do this of course, but it seems like an awful lot of hard > work. Why can't comments be preserved in the AST passed to a parse > transform, and then just dropped later on!? Is there an easier way of > doing this that I'm currently missing? The compiler toolchain just isn't targeted at preserving comments; they are treated as whitespace and are discarded already at the tokenization stage (erl_scan), even before the preprocessing stage (epp). After that, there's the parsing stage, and then the parse transforms are called. So, as you said, you'll need to use the -file("...") hints to locate the source files and read them again to extract the comments. It would certainly be possible to make a compiler that preserves comments for later passes, but the way it's currently done is the "traditional" way of writing a compiler, and changing it afterwards can be difficult, since all the code that expects the current behaviour has to be updated. The good news is that the work of digging out comments and connecting them to the abstract syntax trees has already been done, in the syntax_tools library - EDoc uses exactly this (although not as a parse transform). You can call erl_comment_scan:file/1 to get the comment lines, and then use erl_recomment:recomment_forms/2 to attach the comments to the abstract syntax trees that the parse transform got. The result is a extended abstract syntax tree as defined by the erl_syntax module. You can use the erl_syntax functions to manipulate the tree, and when you're done, you need to call erl_syntax:revert/1 to revert the representation to the form that the compiler understands (this loses the comments again). For a detailed usage example, see edoc_extract:source/3 and friends. /Richard From fritchie@REDACTED Tue Jul 10 21:57:09 2012 From: fritchie@REDACTED (Scott Lystig Fritchie) Date: Tue, 10 Jul 2012 14:57:09 -0500 Subject: [erlang-questions] Erlang and Solaris SMF's In-Reply-To: Message of "Thu, 28 Jun 2012 22:24:43 +0200." Message-ID: <87222.1341950229@snookles.snookles.com> Heinz, I don't know if you got any answers from the OpenIndiana mailing list ... I haven't seen any here on the erlang-questions list. I don't have a good answer for you, other than to give an example SMF config file for Riak. The full text (below) comes from https://raw.github.com/gist/6d76de3af5eba57134d2/cdc659038ac75bc646bdae0ad839271ad6477ff1/manifest.xml -- many thanks to Jared Morrow! Note that we are big fans of the features of libumem. Also, explicit management of "epmd" appears to be missing, which suggests that it isn't necessary? {shrug} This is the manifest that is being used, as far as I know, right now by Riak deployments using SmartOS at Joyent's hosting service. -Scott --- snip --- snip --- snip --- snip --- snip --- snip --- ?xml version="1.0"?> From nem@REDACTED Tue Jul 10 22:57:35 2012 From: nem@REDACTED (Geoff Cant) Date: Tue, 10 Jul 2012 13:57:35 -0700 Subject: [erlang-questions] Escript archives and the load path Message-ID: Hi all, after looking for a while, I couldn't figure out how to tell ERTS or the code server that /some/path/rebar was a valid place to load modules from. Is there a magic function / command-line flag I can use to get the contents of this escript archive into the code path? Cheers, -- Geoff Cant From lferlang@REDACTED Tue Jul 10 23:14:54 2012 From: lferlang@REDACTED (erlang) Date: Tue, 10 Jul 2012 23:14:54 +0200 Subject: [erlang-questions] Improve performance of IO bounded server written in Erlang via having pollset for each scheduler and bind port to scheduler together with process In-Reply-To: References: Message-ID: <4FFC9B4E.6040603@gmail.com> Hi, all (first time) on start - i'm sorry for my englisch language. i think that the problem is on all lines where you use "io:format" In my system when i start wrote in erlang I used many times io:format for debug. It wos big bottleneck. JanM W dniu 2012-07-09 11:01, Wei Cao pisze: > Hi, all > > We wrote a proxy server in Erlang, the proxy's logic is rather simple, > it listens on some TCP port, establishes new connection from user > client, forward packets back and forth between the client and backend > server after authentication until connection is closed. > > It's very easy to write such a proxy in Erlang, fork a process for > each new user connection and connect to the backend server in the same > process, the process works like a pipe, sockets from both side is set > to the active once mode, whenever a tcp packet is received from one > socket, the packet will be sent to other socket. (A simplified version > of proxy code is attached at the end of the mail) > > However, the performance is not quite satisfying, the proxy can handle > maximum only 40k requests on our 16 core machine(Intel Xeon L5630, > 2.13GHz) with heavy stress(100 concurrent clients). We then analyzed > the behavior of beam.smp use tools like tcprstat, mpstat, perf, and > SystemTap. > > tcprstat shows QPS is about 40k, have average 1.7 millisecond latency > for each request. > > timestamp count max min avg med stddev 95_max 95_avg 95_std 99_max 99_avg 99_std > 1341813326 39416 17873 953 1718 1519 724 2919 1609 340 3813 1674 462 > 1341813327 40528 9275 884 1645 1493 491 2777 1559 290 3508 1619 409 > 1341813328 40009 18105 925 1694 1507 714 2868 1586 328 3753 1650 450 > > > mpstat shows 30% CPU is idle, > > 03:30:19 PM CPU %usr %nice %sys %iowait %irq %soft > %steal %guest %idle > 03:30:20 PM all 38.69 0.00 21.92 0.00 0.06 7.52 > 0.00 0.00 31.80 > 03:30:21 PM all 37.56 0.00 21.99 0.00 0.00 7.50 > 0.00 0.00 32.95 > > and perf top shows, much time is wasted in scheduler_wait, in spin wait I guess. > > 9320.00 19.8% scheduler_wait > /home/mingsong.cw/erlangr16b/lib/erlang/erts-5.10/bin/beam.smp > 1813.00 3.9% process_main > /home/mingsong.cw/erlangr16b/lib/erlang/erts-5.10/bin/beam.smp > 1379.00 2.9% _spin_lock > /usr/lib/debug/lib/modules/2.6.32-131.21.1.tb477.el6.x86_64/vmlinux > 1201.00 2.6% schedule > /home/mingsong.cw/erlangr16b/lib/erlang/erts-5.10/bin/beam.smp > > It seems the performance may be associated with scheduler_wait() and > erts_check_io(), with a SystemTap script(attached at the end of this > mail), we can find out how many times the system call epoll_wait is > invoked by beam.smp and each time, how many revents it gets. > > cpu process times > revents min max avg timeouts > all > 1754 128042 - - 73 3 > [14] beam.smp 151 > 14127 82 97 93 0 > [ 5] beam.smp 142 > 13291 83 97 93 0 > [13] beam.smp 127 > 11948 86 96 94 0 > [ 6] beam.smp 127 > 11836 81 96 93 0 > [ 4] beam.smp 121 > 11323 81 96 93 0 > [15] beam.smp 117 > 10935 83 96 93 0 > [12] beam.smp 486 > 10128 0 96 20 2 > [ 1] beam.smp 71 > 6549 71 100 92 0 > [ 2] beam.smp 62 > 5695 82 96 91 0 > [ 7] beam.smp 55 > 5102 81 95 92 0 > [11] beam.smp 52 > 4822 85 95 92 0 > [ 9] beam.smp 52 > 4799 85 95 92 0 > [ 8] beam.smp 51 > 4680 78 95 91 0 > [10] beam.smp 49 > 4508 85 97 92 0 > [ 3] beam.smp 46 > 4211 81 95 91 0 > [ 0] beam.smp 44 > 4088 83 95 92 0 > > The resuls shows, epoll_wait is invoked 1754 times each second, and > get 73 io events in average. This is unacceptable for writing high > performance server. Because if epoll_wait is invoked no more than 2k > times per second, then read/write a packet would cost more than 500ms, > which causes long delay and affects the throughput finally. > > The problem relies on there is only one global pollset in system wide, > so at a time there is no more than one scheduler can call > erts_check_io() to obtain pending io tasks from underlying pollset, > and no scheduler can call erts_check_io() before all pending io > tasks're processed, so for IO bounded application, it's very likely > that a scheduler finish its own job, but must wait idly for other > schedulers to complete theirs. > > Hence, we develops a patch to slove this problem, by having a pollset > for each scheduler, so that each scheduler can invoke erts_check_io() > on its own pollset concurrently. After a scheduler complete its tasks, > it can invoke erts_check_io() immediately no matter what state other > schedulers're in. This patch also handles port migration situation, > all used file descriptors in each port're recorded, when a port is > migrated, these > fd 're removed from original scheduler's pollset, and added to new scheduler's. > > Bind port to scheduler together with process is also helpful to > performance, it reduces the cost of thread switches and > synchronization, and bound port won't be migrated between schedulers. > > After apply the two patches, with the same pressure(100 concurrent > clients),epoll_wait is invoked 49332 times per second, and get 3 > revents each time in average, that is to say, our server responds > quicker and become more realtime. > > cpu process times > revents min max avg timeouts > all > 49332 217186 - - 4 3 > [ 2] beam.smp 3219 > 16050 2 7 4 0 > [11] beam.smp 4275 > 16033 1 6 3 0 > [ 8] beam.smp 4240 > 15992 1 6 3 0 > [ 9] beam.smp 4316 > 15964 0 6 3 2 > [10] beam.smp 4139 > 15851 1 6 3 0 > [ 3] beam.smp 4256 > 15816 1 6 3 0 > [ 1] beam.smp 3107 > 15800 2 7 5 0 > [ 0] beam.smp 3727 > 15259 1 6 4 0 > [ 7] beam.smp 2810 > 14722 3 7 5 0 > [13] beam.smp 1981 > 11499 4 7 5 0 > [15] beam.smp 2285 > 10942 3 6 4 0 > [14] beam.smp 2258 > 10866 3 6 4 0 > [ 4] beam.smp 2246 > 10849 3 6 4 0 > [ 6] beam.smp 2206 > 10730 3 6 4 0 > [12] beam.smp 2173 > 10573 3 6 4 0 > [ 5] beam.smp 2093 > 10240 3 6 4 0 > > scheduler_wait no longer take so much time now, > > 169.00 6.2% process_main beam.smp > 55.00 2.0% _spin_lock [kernel] > 45.00 1.7% driver_deliver_term beam.smp > > so is idle CPU time > 04:30:44 PM CPU %usr %nice %sys %iowait %irq %soft > %steal %guest %idle > 04:30:45 PM all 60.34 0.00 21.44 0.00 0.06 16.45 > 0.00 0.00 1.71 > 04:30:46 PM all 60.99 0.00 21.22 0.00 0.00 16.26 > 0.00 0.00 1.52 > > and tcprstat shows, QPS is getting 100K, latency is less than 1 millisecond > > timestamp count max min avg med stddev 95_max 95_avg 95_std 99_max 99_avg 99_std > 1341822689 96078 11592 314 910 817 311 1447 869 228 1777 897 263 > 1341822690 100651 24245 209 914 819 381 1458 870 229 1800 899 263 > > I also write a extreamly simple keep-alive http server(attached at the > end of mail), to compare performance before and after applying the > patches, mearused with apache ab tool(ab -n 1000000 -c 100 -k ), a 30% > performance gain can be get. > > before > Requests per second: 103671.70 [#/sec] (mean) > Time per request: 0.965 [ms] (mean) > > after > Requests per second: 133701.24 [#/sec] (mean) > Time per request: 0.748 [ms] (mean) > > Patches can be found at github, compile with > ./configure CFLAGS=-DERTS_POLLSET_PER_SCHEDULER > > Pollset per scheduler: > > git fetch git://github.com/weicao/otp.git pollset_per_scheduler > > https://github.com/weicao/otp/compare/weicao:master...weicao:pollset_per_scheduler > https://github.com/weicao/otp/compare/weicao:master...weicao:pollset_per_scheduler.patch > > > Bind port to scheduler: > > git fetch git://github.com/weicao/otp.git bind_port_to_scheduler > > https://github.com/weicao/otp/compare/weicao:pollset_per_scheduler...weicao:bind_port_to_scheduler > https://github.com/weicao/otp/compare/weicao:pollset_per_scheduler...weicao:bind_port_to_scheduler.patch > > > Appendix: > > ----------------------------------- > proxy.erl > ------------------------------------ > > -module(proxy). > -compile(export_all). > > -define(RECBUF_SIZE, 8192). > -define(ACCEPT_TIMEOUT, 2000). > > start([MyPortAtom, DestIpAtom, DestPortAtom]) -> > {MyPort, []} = string:to_integer(atom_to_list(MyPortAtom)), > DestIp = atom_to_list(DestIpAtom), > {DestPort, []} = string:to_integer(atom_to_list(DestPortAtom)), > listen(MyPort, DestIp, DestPort), > receive Any -> > io:format("recv ~p~n", [Any]) > end. > > listen(MyPort, DestIp, DestPort) -> > io:format("start proxy on [local] 0.0.0.0:~p -> [remote] ~p:~p~n", > [MyPort, DestIp, DestPort]), > case gen_tcp:listen(MyPort, > [inet, > {ip, {0,0,0,0}}, > binary, > {reuseaddr, true}, > {recbuf, ?RECBUF_SIZE}, > {active, false}, > {nodelay, true} > ]) of > {ok, Listen} -> > N = erlang:system_info(schedulers), > lists:foreach(fun(I) -> accept(Listen, DestIp, DestPort, > I) end, lists:seq(1,N)); > {error, Reason} -> > io:format("error listen ~p~n", [Reason]) > end. > > accept(Listen, DestIp, DestPort, I) -> > spawn_opt(?MODULE, loop, [Listen, DestIp, DestPort, I], [{scheduler, I}]). > > loop(Listen, DestIp, DestPort, I) -> > case gen_tcp:accept(Listen, ?ACCEPT_TIMEOUT) of > {ok, S1} -> > accept(Listen, DestIp, DestPort, I), > case catch gen_tcp:connect(DestIp, DestPort, > [inet, binary, {active, false}, > {reuseaddr, true}, {nodelay, true}]) of > {ok, S2} -> > io:format("new connection~n"), > loop1(S1, S2); > {error, Reason} -> > io:format("error connect ~p~n", [Reason]) > end; > {error, timeout} -> > loop(Listen, DestIp, DestPort, I); > Error -> > io:format("error accept ~p~n", [Error]), > accept(Listen, DestIp, DestPort, I) > end. > > loop1(S1, S2) -> > active(S1, S2), > receive > {tcp, S1, Data} -> > gen_tcp:send(S2, Data), > loop1(S1, S2); > {tcp, S2, Data} -> > gen_tcp:send(S1, Data), > loop1(S1, S2); > {tcp_closed, S1} -> > io:format("S1 close~n"), > gen_tcp:close(S1), > gen_tcp:close(S2); > {tcp_closed, S2} -> > io:format("S2 close~n"), > gen_tcp:close(S1), > gen_tcp:close(S2) > end. > > active(S1,S2)-> > inet:setopts(S1, [{active, once}]), > inet:setopts(S2, [{active, once}]). > > ----------------------------------- > epollwait.stp > ------------------------------------ > #! /usr/bin/env stap > # > # > > global epoll_timeout_flag, epoll_count, epoll_min, epoll_max, > epoll_times, epoll_timeouts > > probe syscall.epoll_wait { > if(timeout > 0) { > epoll_timeout_flag[pid()] = 1 > } > } > > probe syscall.epoll_wait.return { > c = cpu() > p = execname() > > epoll_times[c,p] ++ > epoll_count[c,p] += $return > > if($return == 0 && pid() in epoll_timeout_flag) { > epoll_timeouts[c,p] ++ > delete epoll_timeout_flag[pid()] > } > > if(!([c, p] in epoll_min)) { > epoll_min[c,p] = $return > } else if($return < epoll_min[c,p]) { > epoll_min[c,p] = $return > } > > > if($return > epoll_max[c,p]) { > epoll_max[c,p] = $return > } > } > > probe timer.s($1) { > printf ("%4s %45s %10s %10s %10s %10s %10s %10s\n", "cpu", > "process", "times", "revents", "min", "max", "avg", "timeouts" ) > foreach ([cpu, process] in epoll_count- limit 20) { > all_epoll_times += epoll_times[cpu,process] > all_epoll_count += epoll_count[cpu,process] > all_epoll_timeouts += epoll_timeouts[cpu,process] > } > printf ("%4s %45s %10d %10d %10s %10s %10d %10d\n", > "all", "", all_epoll_times, all_epoll_count, "-", "-", > all_epoll_count == 0? 0:all_epoll_count/all_epoll_times, all_epoll_timeouts) > > foreach ([cpu, process] in epoll_count- limit 20) { > printf ("[%2d] %45s %10d %10d %10d %10d %10d %10d\n", > cpu, process, epoll_times[cpu, process], epoll_count[cpu, process], > epoll_min[cpu, process], epoll_max[cpu, process], > epoll_count[cpu,process]/epoll_times[cpu,process], > epoll_timeouts[cpu,process]) > } > delete epoll_count > delete epoll_min > delete epoll_max > delete epoll_times > delete epoll_timeouts > printf ("--------------------------------------------------------------------------\n\n" > ) > } > > ------------------------------------------------ > ehttpd.erl > ------------------------------------------------- > -module(ehttpd). > -compile(export_all). > > start() -> > start(8888). > start(Port) -> > N = erlang:system_info(schedulers), > listen(Port, N), > io:format("ehttpd ready with ~b schedulers on port ~b~n", [N, Port]), > > register(?MODULE, self()), > receive Any -> io:format("~p~n", [Any]) end. %% to stop: ehttpd!stop. > > listen(Port, N) -> > Opts = [inet, > binary, > {active, false}, > {recbuf, 8192}, > {nodelay,true}, > {reuseaddr, true}], > > {ok, S} = gen_tcp:listen(Port, Opts), > lists:foreach(fun(I)-> spawn_opt(?MODULE, accept, [S, I], > [{scheduler, I}]) end, lists:seq(1, N)). > > accept(S, I) -> > case gen_tcp:accept(S) of > {ok, Socket} -> > spawn_opt(?MODULE, accept, [S, I],[{scheduler,I}] ), > io:format("new connection @~p~n", [I]), > loop(Socket,<<>>); > Error -> erlang:error(Error) > end. > > loop(S,B) -> > inet:setopts(S, [{active, once}]), > receive > {tcp, S, Data} -> > B1 = <>, > case binary:part(B1,{byte_size(B1), -4}) of > <<"\r\n\r\n">> -> > Response = <<"HTTP/1.1 200 OK\r\nContent-Length: > 12\r\nConnection: Keep-Alive\r\n\r\nhello world!">>, > gen_tcp:send(S, Response), > loop(S, <<>>); > _ -> > loop(S, B1) > end; > {tcp_closed, S} -> > io:format("connection closed forced~n"), > gen_tcp:close(S); > Error -> > io:format("unexpected message~p~n", [Error]), > Error > end. > > From zabrane3@REDACTED Tue Jul 10 23:36:56 2012 From: zabrane3@REDACTED (Zabrane Mickael) Date: Tue, 10 Jul 2012 23:36:56 +0200 Subject: [erlang-questions] Improve performance of IO bounded server written in Erlang via having pollset for each scheduler and bind port to scheduler together with process In-Reply-To: <4FFC9B4E.6040603@gmail.com> References: <4FFC9B4E.6040603@gmail.com> Message-ID: <1C75214B-F95F-4A80-8B91-24BA007B1D9A@gmail.com> I dont think so! When I tested Wei's code teh first time, I suppressed all io:format statements and made the inner loop lot simpler. Something like that: [...] loop(S,B) -> inet:setopts(S, [{active, once}]), receive {tcp, S, _Data} -> Response = <<"HTTP/1.1 200 OK\r\nContent-Length: 12\r\nConnection: Keep-Alive\r\n\r\nhello world!">>, gen_tcp:send(S, Response), loop(S, <<>>) [...] didn't change anything in my case. Hope this help! Regards, Zabrane On Jul 10, 2012, at 11:14 PM, erlang wrote: > Hi, all (first time) > on start - i'm sorry for my englisch language. > i think that the problem is on all lines where you use "io:format" > In my system when i start wrote in erlang I used many times io:format for debug. It wos big bottleneck. > > > JanM > > > W dniu 2012-07-09 11:01, Wei Cao pisze: >> Hi, all >> >> We wrote a proxy server in Erlang, the proxy's logic is rather simple, >> it listens on some TCP port, establishes new connection from user >> client, forward packets back and forth between the client and backend >> server after authentication until connection is closed. >> >> It's very easy to write such a proxy in Erlang, fork a process for >> each new user connection and connect to the backend server in the same >> process, the process works like a pipe, sockets from both side is set >> to the active once mode, whenever a tcp packet is received from one >> socket, the packet will be sent to other socket. (A simplified version >> of proxy code is attached at the end of the mail) >> >> However, the performance is not quite satisfying, the proxy can handle >> maximum only 40k requests on our 16 core machine(Intel Xeon L5630, >> 2.13GHz) with heavy stress(100 concurrent clients). We then analyzed >> the behavior of beam.smp use tools like tcprstat, mpstat, perf, and >> SystemTap. >> >> tcprstat shows QPS is about 40k, have average 1.7 millisecond latency >> for each request. >> >> timestamp count max min avg med stddev 95_max 95_avg 95_std 99_max 99_avg 99_std >> 1341813326 39416 17873 953 1718 1519 724 2919 1609 340 3813 1674 462 >> 1341813327 40528 9275 884 1645 1493 491 2777 1559 290 3508 1619 409 >> 1341813328 40009 18105 925 1694 1507 714 2868 1586 328 3753 1650 450 >> >> >> mpstat shows 30% CPU is idle, >> >> 03:30:19 PM CPU %usr %nice %sys %iowait %irq %soft >> %steal %guest %idle >> 03:30:20 PM all 38.69 0.00 21.92 0.00 0.06 7.52 >> 0.00 0.00 31.80 >> 03:30:21 PM all 37.56 0.00 21.99 0.00 0.00 7.50 >> 0.00 0.00 32.95 >> >> and perf top shows, much time is wasted in scheduler_wait, in spin wait I guess. >> >> 9320.00 19.8% scheduler_wait >> /home/mingsong.cw/erlangr16b/lib/erlang/erts-5.10/bin/beam.smp >> 1813.00 3.9% process_main >> /home/mingsong.cw/erlangr16b/lib/erlang/erts-5.10/bin/beam.smp >> 1379.00 2.9% _spin_lock >> /usr/lib/debug/lib/modules/2.6.32-131.21.1.tb477.el6.x86_64/vmlinux >> 1201.00 2.6% schedule >> /home/mingsong.cw/erlangr16b/lib/erlang/erts-5.10/bin/beam.smp >> >> It seems the performance may be associated with scheduler_wait() and >> erts_check_io(), with a SystemTap script(attached at the end of this >> mail), we can find out how many times the system call epoll_wait is >> invoked by beam.smp and each time, how many revents it gets. >> >> cpu process times >> revents min max avg timeouts >> all >> 1754 128042 - - 73 3 >> [14] beam.smp 151 >> 14127 82 97 93 0 >> [ 5] beam.smp 142 >> 13291 83 97 93 0 >> [13] beam.smp 127 >> 11948 86 96 94 0 >> [ 6] beam.smp 127 >> 11836 81 96 93 0 >> [ 4] beam.smp 121 >> 11323 81 96 93 0 >> [15] beam.smp 117 >> 10935 83 96 93 0 >> [12] beam.smp 486 >> 10128 0 96 20 2 >> [ 1] beam.smp 71 >> 6549 71 100 92 0 >> [ 2] beam.smp 62 >> 5695 82 96 91 0 >> [ 7] beam.smp 55 >> 5102 81 95 92 0 >> [11] beam.smp 52 >> 4822 85 95 92 0 >> [ 9] beam.smp 52 >> 4799 85 95 92 0 >> [ 8] beam.smp 51 >> 4680 78 95 91 0 >> [10] beam.smp 49 >> 4508 85 97 92 0 >> [ 3] beam.smp 46 >> 4211 81 95 91 0 >> [ 0] beam.smp 44 >> 4088 83 95 92 0 >> >> The resuls shows, epoll_wait is invoked 1754 times each second, and >> get 73 io events in average. This is unacceptable for writing high >> performance server. Because if epoll_wait is invoked no more than 2k >> times per second, then read/write a packet would cost more than 500ms, >> which causes long delay and affects the throughput finally. >> >> The problem relies on there is only one global pollset in system wide, >> so at a time there is no more than one scheduler can call >> erts_check_io() to obtain pending io tasks from underlying pollset, >> and no scheduler can call erts_check_io() before all pending io >> tasks're processed, so for IO bounded application, it's very likely >> that a scheduler finish its own job, but must wait idly for other >> schedulers to complete theirs. >> >> Hence, we develops a patch to slove this problem, by having a pollset >> for each scheduler, so that each scheduler can invoke erts_check_io() >> on its own pollset concurrently. After a scheduler complete its tasks, >> it can invoke erts_check_io() immediately no matter what state other >> schedulers're in. This patch also handles port migration situation, >> all used file descriptors in each port're recorded, when a port is >> migrated, these >> fd 're removed from original scheduler's pollset, and added to new scheduler's. >> >> Bind port to scheduler together with process is also helpful to >> performance, it reduces the cost of thread switches and >> synchronization, and bound port won't be migrated between schedulers. >> >> After apply the two patches, with the same pressure(100 concurrent >> clients),epoll_wait is invoked 49332 times per second, and get 3 >> revents each time in average, that is to say, our server responds >> quicker and become more realtime. >> >> cpu process times >> revents min max avg timeouts >> all >> 49332 217186 - - 4 3 >> [ 2] beam.smp 3219 >> 16050 2 7 4 0 >> [11] beam.smp 4275 >> 16033 1 6 3 0 >> [ 8] beam.smp 4240 >> 15992 1 6 3 0 >> [ 9] beam.smp 4316 >> 15964 0 6 3 2 >> [10] beam.smp 4139 >> 15851 1 6 3 0 >> [ 3] beam.smp 4256 >> 15816 1 6 3 0 >> [ 1] beam.smp 3107 >> 15800 2 7 5 0 >> [ 0] beam.smp 3727 >> 15259 1 6 4 0 >> [ 7] beam.smp 2810 >> 14722 3 7 5 0 >> [13] beam.smp 1981 >> 11499 4 7 5 0 >> [15] beam.smp 2285 >> 10942 3 6 4 0 >> [14] beam.smp 2258 >> 10866 3 6 4 0 >> [ 4] beam.smp 2246 >> 10849 3 6 4 0 >> [ 6] beam.smp 2206 >> 10730 3 6 4 0 >> [12] beam.smp 2173 >> 10573 3 6 4 0 >> [ 5] beam.smp 2093 >> 10240 3 6 4 0 >> >> scheduler_wait no longer take so much time now, >> >> 169.00 6.2% process_main beam.smp >> 55.00 2.0% _spin_lock [kernel] >> 45.00 1.7% driver_deliver_term beam.smp >> >> so is idle CPU time >> 04:30:44 PM CPU %usr %nice %sys %iowait %irq %soft >> %steal %guest %idle >> 04:30:45 PM all 60.34 0.00 21.44 0.00 0.06 16.45 >> 0.00 0.00 1.71 >> 04:30:46 PM all 60.99 0.00 21.22 0.00 0.00 16.26 >> 0.00 0.00 1.52 >> >> and tcprstat shows, QPS is getting 100K, latency is less than 1 millisecond >> >> timestamp count max min avg med stddev 95_max 95_avg 95_std 99_max 99_avg 99_std >> 1341822689 96078 11592 314 910 817 311 1447 869 228 1777 897 263 >> 1341822690 100651 24245 209 914 819 381 1458 870 229 1800 899 263 >> >> I also write a extreamly simple keep-alive http server(attached at the >> end of mail), to compare performance before and after applying the >> patches, mearused with apache ab tool(ab -n 1000000 -c 100 -k ), a 30% >> performance gain can be get. >> >> before >> Requests per second: 103671.70 [#/sec] (mean) >> Time per request: 0.965 [ms] (mean) >> >> after >> Requests per second: 133701.24 [#/sec] (mean) >> Time per request: 0.748 [ms] (mean) >> >> Patches can be found at github, compile with >> ./configure CFLAGS=-DERTS_POLLSET_PER_SCHEDULER >> >> Pollset per scheduler: >> >> git fetch git://github.com/weicao/otp.git pollset_per_scheduler >> >> https://github.com/weicao/otp/compare/weicao:master...weicao:pollset_per_scheduler >> https://github.com/weicao/otp/compare/weicao:master...weicao:pollset_per_scheduler.patch >> >> >> Bind port to scheduler: >> >> git fetch git://github.com/weicao/otp.git bind_port_to_scheduler >> >> https://github.com/weicao/otp/compare/weicao:pollset_per_scheduler...weicao:bind_port_to_scheduler >> https://github.com/weicao/otp/compare/weicao:pollset_per_scheduler...weicao:bind_port_to_scheduler.patch >> >> >> Appendix: >> >> ----------------------------------- >> proxy.erl >> ------------------------------------ >> >> -module(proxy). >> -compile(export_all). >> >> -define(RECBUF_SIZE, 8192). >> -define(ACCEPT_TIMEOUT, 2000). >> >> start([MyPortAtom, DestIpAtom, DestPortAtom]) -> >> {MyPort, []} = string:to_integer(atom_to_list(MyPortAtom)), >> DestIp = atom_to_list(DestIpAtom), >> {DestPort, []} = string:to_integer(atom_to_list(DestPortAtom)), >> listen(MyPort, DestIp, DestPort), >> receive Any -> >> io:format("recv ~p~n", [Any]) >> end. >> >> listen(MyPort, DestIp, DestPort) -> >> io:format("start proxy on [local] 0.0.0.0:~p -> [remote] ~p:~p~n", >> [MyPort, DestIp, DestPort]), >> case gen_tcp:listen(MyPort, >> [inet, >> {ip, {0,0,0,0}}, >> binary, >> {reuseaddr, true}, >> {recbuf, ?RECBUF_SIZE}, >> {active, false}, >> {nodelay, true} >> ]) of >> {ok, Listen} -> >> N = erlang:system_info(schedulers), >> lists:foreach(fun(I) -> accept(Listen, DestIp, DestPort, >> I) end, lists:seq(1,N)); >> {error, Reason} -> >> io:format("error listen ~p~n", [Reason]) >> end. >> >> accept(Listen, DestIp, DestPort, I) -> >> spawn_opt(?MODULE, loop, [Listen, DestIp, DestPort, I], [{scheduler, I}]). >> >> loop(Listen, DestIp, DestPort, I) -> >> case gen_tcp:accept(Listen, ?ACCEPT_TIMEOUT) of >> {ok, S1} -> >> accept(Listen, DestIp, DestPort, I), >> case catch gen_tcp:connect(DestIp, DestPort, >> [inet, binary, {active, false}, >> {reuseaddr, true}, {nodelay, true}]) of >> {ok, S2} -> >> io:format("new connection~n"), >> loop1(S1, S2); >> {error, Reason} -> >> io:format("error connect ~p~n", [Reason]) >> end; >> {error, timeout} -> >> loop(Listen, DestIp, DestPort, I); >> Error -> >> io:format("error accept ~p~n", [Error]), >> accept(Listen, DestIp, DestPort, I) >> end. >> >> loop1(S1, S2) -> >> active(S1, S2), >> receive >> {tcp, S1, Data} -> >> gen_tcp:send(S2, Data), >> loop1(S1, S2); >> {tcp, S2, Data} -> >> gen_tcp:send(S1, Data), >> loop1(S1, S2); >> {tcp_closed, S1} -> >> io:format("S1 close~n"), >> gen_tcp:close(S1), >> gen_tcp:close(S2); >> {tcp_closed, S2} -> >> io:format("S2 close~n"), >> gen_tcp:close(S1), >> gen_tcp:close(S2) >> end. >> >> active(S1,S2)-> >> inet:setopts(S1, [{active, once}]), >> inet:setopts(S2, [{active, once}]). >> >> ----------------------------------- >> epollwait.stp >> ------------------------------------ >> #! /usr/bin/env stap >> # >> # >> >> global epoll_timeout_flag, epoll_count, epoll_min, epoll_max, >> epoll_times, epoll_timeouts >> >> probe syscall.epoll_wait { >> if(timeout > 0) { >> epoll_timeout_flag[pid()] = 1 >> } >> } >> >> probe syscall.epoll_wait.return { >> c = cpu() >> p = execname() >> >> epoll_times[c,p] ++ >> epoll_count[c,p] += $return >> >> if($return == 0 && pid() in epoll_timeout_flag) { >> epoll_timeouts[c,p] ++ >> delete epoll_timeout_flag[pid()] >> } >> >> if(!([c, p] in epoll_min)) { >> epoll_min[c,p] = $return >> } else if($return < epoll_min[c,p]) { >> epoll_min[c,p] = $return >> } >> >> >> if($return > epoll_max[c,p]) { >> epoll_max[c,p] = $return >> } >> } >> >> probe timer.s($1) { >> printf ("%4s %45s %10s %10s %10s %10s %10s %10s\n", "cpu", >> "process", "times", "revents", "min", "max", "avg", "timeouts" ) >> foreach ([cpu, process] in epoll_count- limit 20) { >> all_epoll_times += epoll_times[cpu,process] >> all_epoll_count += epoll_count[cpu,process] >> all_epoll_timeouts += epoll_timeouts[cpu,process] >> } >> printf ("%4s %45s %10d %10d %10s %10s %10d %10d\n", >> "all", "", all_epoll_times, all_epoll_count, "-", "-", >> all_epoll_count == 0? 0:all_epoll_count/all_epoll_times, all_epoll_timeouts) >> >> foreach ([cpu, process] in epoll_count- limit 20) { >> printf ("[%2d] %45s %10d %10d %10d %10d %10d %10d\n", >> cpu, process, epoll_times[cpu, process], epoll_count[cpu, process], >> epoll_min[cpu, process], epoll_max[cpu, process], >> epoll_count[cpu,process]/epoll_times[cpu,process], >> epoll_timeouts[cpu,process]) >> } >> delete epoll_count >> delete epoll_min >> delete epoll_max >> delete epoll_times >> delete epoll_timeouts >> printf ("--------------------------------------------------------------------------\n\n" >> ) >> } >> >> ------------------------------------------------ >> ehttpd.erl >> ------------------------------------------------- >> -module(ehttpd). >> -compile(export_all). >> >> start() -> >> start(8888). >> start(Port) -> >> N = erlang:system_info(schedulers), >> listen(Port, N), >> io:format("ehttpd ready with ~b schedulers on port ~b~n", [N, Port]), >> >> register(?MODULE, self()), >> receive Any -> io:format("~p~n", [Any]) end. %% to stop: ehttpd!stop. >> >> listen(Port, N) -> >> Opts = [inet, >> binary, >> {active, false}, >> {recbuf, 8192}, >> {nodelay,true}, >> {reuseaddr, true}], >> >> {ok, S} = gen_tcp:listen(Port, Opts), >> lists:foreach(fun(I)-> spawn_opt(?MODULE, accept, [S, I], >> [{scheduler, I}]) end, lists:seq(1, N)). >> >> accept(S, I) -> >> case gen_tcp:accept(S) of >> {ok, Socket} -> >> spawn_opt(?MODULE, accept, [S, I],[{scheduler,I}] ), >> io:format("new connection @~p~n", [I]), >> loop(Socket,<<>>); >> Error -> erlang:error(Error) >> end. >> >> loop(S,B) -> >> inet:setopts(S, [{active, once}]), >> receive >> {tcp, S, Data} -> >> B1 = <>, >> case binary:part(B1,{byte_size(B1), -4}) of >> <<"\r\n\r\n">> -> >> Response = <<"HTTP/1.1 200 OK\r\nContent-Length: >> 12\r\nConnection: Keep-Alive\r\n\r\nhello world!">>, >> gen_tcp:send(S, Response), >> loop(S, <<>>); >> _ -> >> loop(S, B1) >> end; >> {tcp_closed, S} -> >> io:format("connection closed forced~n"), >> gen_tcp:close(S); >> Error -> >> io:format("unexpected message~p~n", [Error]), >> Error >> end. >> >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Wed Jul 11 00:30:26 2012 From: essen@REDACTED (=?ISO-8859-1?Q?Lo=EFc_Hoguin?=) Date: Wed, 11 Jul 2012 00:30:26 +0200 Subject: [erlang-questions] Looking for slides of a lightning talk In-Reply-To: <1709242D-0AD5-4070-A6EC-AB039E62ADFD@dmitriid.com> References: <4FFAAAEA.9060605@ninenines.eu> <0d6c6c5d-2b0f-48da-bae8-f440d232df92@googlegroups.com> <489B3810-49B2-4085-B468-DA037F7DE646@dmitriid.com> <1709242D-0AD5-4070-A6EC-AB039E62ADFD@dmitriid.com> Message-ID: <4FFCAD02.10501@ninenines.eu> Awesome, thanks! On 07/10/2012 08:39 PM, Dmitrii Dimandt wrote: > Ok, just the slides are here: > http://www.slideshare.net/dmitriid/erlang-sucks-euc-2012 (you can > download them there as well) > > A PDF with some notes is > here: http://www.scribd.com/doc/99721085/Erlang-Sucks-EUC-2012 > > Notes isn't as good as talking in from of the audience, but oh well... > > >> >>> Heh. That was me :) >>> >>> I have the slides at home, and I *WILL* forget to send them to you :) >>> So feel free to bug me directly until I do >>> >>> Someone did a lightning talk in Stockholm that was quite funny >>> but also >>> quite true, saying that Erlang isn't ready for the web and doing >>> comparisons with other languages/platforms. Any chance I could >>> get the >>> slides? >>> >>> Thanks. >>> >>> -- >>> Lo?c Hoguin >>> Erlang Cowboy >>> Nine Nines >>> http://ninenines.eu >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >> > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From zabrane3@REDACTED Wed Jul 11 02:24:53 2012 From: zabrane3@REDACTED (Zabrane Mickael) Date: Wed, 11 Jul 2012 02:24:53 +0200 Subject: [erlang-questions] Improve performance of IO bounded server written in Erlang via having pollset for each scheduler and bind port to scheduler together with process In-Reply-To: References: Message-ID: Hi Wei, I did some other tests on our 8-Cores 64-bit machine: before: 157141 rps after: 80245 rps Same behaviour as before ... hope this help. Regards, Zabrane On Jul 10, 2012, at 4:30 AM, Wei Cao wrote: > My tests were all keepalived/persistent connections, and there was > significant performance gain in these situations. but I wrote a http > server with short connection just now, the performance did drop down, > I'll find it out later :). > > 2012/7/10 Zabrane Mickael : >> Hi, >> >> Performance of our HTTP Web Server drops down after applying your patches. >> >> Box: Linux F17, 4GB of RAM: >> $ lscpu >> Architecture: i686 >> CPU op-mode(s): 32-bit, 64-bit >> Byte Order: Little Endian >> CPU(s): 4 >> On-line CPU(s) list: 0-3 >> Thread(s) per core: 1 >> Core(s) per socket: 4 >> Socket(s): 1 >> Vendor ID: GenuineIntel >> CPU family: 6 >> Model: 23 >> Stepping: 7 >> CPU MHz: 2499.772 >> BogoMIPS: 4999.54 >> Virtualization: VT-x >> L1d cache: 32K >> L1i cache: 32K >> L2 cache: 3072K >> >> Bench: >> before: 77787 rps >> after: 53056 rps >> >> Any hint to explain this result Wei ? >> >> Regards, >> Zabrane >> >> >> On Jul 9, 2012, at 11:01 AM, Wei Cao wrote: >> >> Hi, all >> >> We wrote a proxy server in Erlang, the proxy's logic is rather simple, >> it listens on some TCP port, establishes new connection from user >> client, forward packets back and forth between the client and backend >> server after authentication until connection is closed. >> >> It's very easy to write such a proxy in Erlang, fork a process for >> each new user connection and connect to the backend server in the same >> process, the process works like a pipe, sockets from both side is set >> to the active once mode, whenever a tcp packet is received from one >> socket, the packet will be sent to other socket. (A simplified version >> of proxy code is attached at the end of the mail) >> >> However, the performance is not quite satisfying, the proxy can handle >> maximum only 40k requests on our 16 core machine(Intel Xeon L5630, >> 2.13GHz) with heavy stress(100 concurrent clients). We then analyzed >> the behavior of beam.smp use tools like tcprstat, mpstat, perf, and >> SystemTap. >> >> tcprstat shows QPS is about 40k, have average 1.7 millisecond latency >> for each request. >> >> timestamp count max min avg med stddev 95_max 95_avg 95_std 99_max 99_avg >> 99_std >> 1341813326 39416 17873 953 1718 1519 724 2919 1609 340 3813 1674 462 >> 1341813327 40528 9275 884 1645 1493 491 2777 1559 290 3508 1619 409 >> 1341813328 40009 18105 925 1694 1507 714 2868 1586 328 3753 1650 450 >> >> >> mpstat shows 30% CPU is idle, >> >> 03:30:19 PM CPU %usr %nice %sys %iowait %irq %soft >> %steal %guest %idle >> 03:30:20 PM all 38.69 0.00 21.92 0.00 0.06 7.52 >> 0.00 0.00 31.80 >> 03:30:21 PM all 37.56 0.00 21.99 0.00 0.00 7.50 >> 0.00 0.00 32.95 >> >> and perf top shows, much time is wasted in scheduler_wait, in spin wait I >> guess. >> >> 9320.00 19.8% scheduler_wait >> /home/mingsong.cw/erlangr16b/lib/erlang/erts-5.10/bin/beam.smp >> 1813.00 3.9% process_main >> /home/mingsong.cw/erlangr16b/lib/erlang/erts-5.10/bin/beam.smp >> 1379.00 2.9% _spin_lock >> /usr/lib/debug/lib/modules/2.6.32-131.21.1.tb477.el6.x86_64/vmlinux >> 1201.00 2.6% schedule >> /home/mingsong.cw/erlangr16b/lib/erlang/erts-5.10/bin/beam.smp >> >> It seems the performance may be associated with scheduler_wait() and >> erts_check_io(), with a SystemTap script(attached at the end of this >> mail), we can find out how many times the system call epoll_wait is >> invoked by beam.smp and each time, how many revents it gets. >> >> cpu process times >> revents min max avg timeouts >> all >> 1754 128042 - - 73 3 >> [14] beam.smp 151 >> 14127 82 97 93 0 >> [ 5] beam.smp 142 >> 13291 83 97 93 0 >> [13] beam.smp 127 >> 11948 86 96 94 0 >> [ 6] beam.smp 127 >> 11836 81 96 93 0 >> [ 4] beam.smp 121 >> 11323 81 96 93 0 >> [15] beam.smp 117 >> 10935 83 96 93 0 >> [12] beam.smp 486 >> 10128 0 96 20 2 >> [ 1] beam.smp 71 >> 6549 71 100 92 0 >> [ 2] beam.smp 62 >> 5695 82 96 91 0 >> [ 7] beam.smp 55 >> 5102 81 95 92 0 >> [11] beam.smp 52 >> 4822 85 95 92 0 >> [ 9] beam.smp 52 >> 4799 85 95 92 0 >> [ 8] beam.smp 51 >> 4680 78 95 91 0 >> [10] beam.smp 49 >> 4508 85 97 92 0 >> [ 3] beam.smp 46 >> 4211 81 95 91 0 >> [ 0] beam.smp 44 >> 4088 83 95 92 0 >> >> The resuls shows, epoll_wait is invoked 1754 times each second, and >> get 73 io events in average. This is unacceptable for writing high >> performance server. Because if epoll_wait is invoked no more than 2k >> times per second, then read/write a packet would cost more than 500ms, >> which causes long delay and affects the throughput finally. >> >> The problem relies on there is only one global pollset in system wide, >> so at a time there is no more than one scheduler can call >> erts_check_io() to obtain pending io tasks from underlying pollset, >> and no scheduler can call erts_check_io() before all pending io >> tasks're processed, so for IO bounded application, it's very likely >> that a scheduler finish its own job, but must wait idly for other >> schedulers to complete theirs. >> >> Hence, we develops a patch to slove this problem, by having a pollset >> for each scheduler, so that each scheduler can invoke erts_check_io() >> on its own pollset concurrently. After a scheduler complete its tasks, >> it can invoke erts_check_io() immediately no matter what state other >> schedulers're in. This patch also handles port migration situation, >> all used file descriptors in each port're recorded, when a port is >> migrated, these >> fd 're removed from original scheduler's pollset, and added to new >> scheduler's. >> >> Bind port to scheduler together with process is also helpful to >> performance, it reduces the cost of thread switches and >> synchronization, and bound port won't be migrated between schedulers. >> >> After apply the two patches, with the same pressure(100 concurrent >> clients),epoll_wait is invoked 49332 times per second, and get 3 >> revents each time in average, that is to say, our server responds >> quicker and become more realtime. >> >> cpu process times >> revents min max avg timeouts >> all >> 49332 217186 - - 4 3 >> [ 2] beam.smp 3219 >> 16050 2 7 4 0 >> [11] beam.smp 4275 >> 16033 1 6 3 0 >> [ 8] beam.smp 4240 >> 15992 1 6 3 0 >> [ 9] beam.smp 4316 >> 15964 0 6 3 2 >> [10] beam.smp 4139 >> 15851 1 6 3 0 >> [ 3] beam.smp 4256 >> 15816 1 6 3 0 >> [ 1] beam.smp 3107 >> 15800 2 7 5 0 >> [ 0] beam.smp 3727 >> 15259 1 6 4 0 >> [ 7] beam.smp 2810 >> 14722 3 7 5 0 >> [13] beam.smp 1981 >> 11499 4 7 5 0 >> [15] beam.smp 2285 >> 10942 3 6 4 0 >> [14] beam.smp 2258 >> 10866 3 6 4 0 >> [ 4] beam.smp 2246 >> 10849 3 6 4 0 >> [ 6] beam.smp 2206 >> 10730 3 6 4 0 >> [12] beam.smp 2173 >> 10573 3 6 4 0 >> [ 5] beam.smp 2093 >> 10240 3 6 4 0 >> >> scheduler_wait no longer take so much time now, >> >> 169.00 6.2% process_main beam.smp >> 55.00 2.0% _spin_lock [kernel] >> 45.00 1.7% driver_deliver_term beam.smp >> >> so is idle CPU time >> 04:30:44 PM CPU %usr %nice %sys %iowait %irq %soft >> %steal %guest %idle >> 04:30:45 PM all 60.34 0.00 21.44 0.00 0.06 16.45 >> 0.00 0.00 1.71 >> 04:30:46 PM all 60.99 0.00 21.22 0.00 0.00 16.26 >> 0.00 0.00 1.52 >> >> and tcprstat shows, QPS is getting 100K, latency is less than 1 millisecond >> >> timestamp count max min avg med stddev 95_max 95_avg 95_std 99_max 99_avg >> 99_std >> 1341822689 96078 11592 314 910 817 311 1447 869 228 1777 897 263 >> 1341822690 100651 24245 209 914 819 381 1458 870 229 1800 899 263 >> >> I also write a extreamly simple keep-alive http server(attached at the >> end of mail), to compare performance before and after applying the >> patches, mearused with apache ab tool(ab -n 1000000 -c 100 -k ), a 30% >> performance gain can be get. >> >> before >> Requests per second: 103671.70 [#/sec] (mean) >> Time per request: 0.965 [ms] (mean) >> >> after >> Requests per second: 133701.24 [#/sec] (mean) >> Time per request: 0.748 [ms] (mean) >> >> Patches can be found at github, compile with >> ./configure CFLAGS=-DERTS_POLLSET_PER_SCHEDULER >> >> Pollset per scheduler: >> >> git fetch git://github.com/weicao/otp.git pollset_per_scheduler >> >> https://github.com/weicao/otp/compare/weicao:master...weicao:pollset_per_scheduler >> https://github.com/weicao/otp/compare/weicao:master...weicao:pollset_per_scheduler.patch >> >> >> Bind port to scheduler: >> >> git fetch git://github.com/weicao/otp.git bind_port_to_scheduler >> >> https://github.com/weicao/otp/compare/weicao:pollset_per_scheduler...weicao:bind_port_to_scheduler >> https://github.com/weicao/otp/compare/weicao:pollset_per_scheduler...weicao:bind_port_to_scheduler.patch >> >> >> Appendix: >> >> ----------------------------------- >> proxy.erl >> ------------------------------------ >> >> -module(proxy). >> -compile(export_all). >> >> -define(RECBUF_SIZE, 8192). >> -define(ACCEPT_TIMEOUT, 2000). >> >> start([MyPortAtom, DestIpAtom, DestPortAtom]) -> >> {MyPort, []} = string:to_integer(atom_to_list(MyPortAtom)), >> DestIp = atom_to_list(DestIpAtom), >> {DestPort, []} = string:to_integer(atom_to_list(DestPortAtom)), >> listen(MyPort, DestIp, DestPort), >> receive Any -> >> io:format("recv ~p~n", [Any]) >> end. >> >> listen(MyPort, DestIp, DestPort) -> >> io:format("start proxy on [local] 0.0.0.0:~p -> [remote] ~p:~p~n", >> [MyPort, DestIp, DestPort]), >> case gen_tcp:listen(MyPort, >> [inet, >> {ip, {0,0,0,0}}, >> binary, >> {reuseaddr, true}, >> {recbuf, ?RECBUF_SIZE}, >> {active, false}, >> {nodelay, true} >> ]) of >> {ok, Listen} -> >> N = erlang:system_info(schedulers), >> lists:foreach(fun(I) -> accept(Listen, DestIp, DestPort, >> I) end, lists:seq(1,N)); >> {error, Reason} -> >> io:format("error listen ~p~n", [Reason]) >> end. >> >> accept(Listen, DestIp, DestPort, I) -> >> spawn_opt(?MODULE, loop, [Listen, DestIp, DestPort, I], [{scheduler, >> I}]). >> >> loop(Listen, DestIp, DestPort, I) -> >> case gen_tcp:accept(Listen, ?ACCEPT_TIMEOUT) of >> {ok, S1} -> >> accept(Listen, DestIp, DestPort, I), >> case catch gen_tcp:connect(DestIp, DestPort, >> [inet, binary, {active, false}, >> {reuseaddr, true}, {nodelay, true}]) of >> {ok, S2} -> >> io:format("new connection~n"), >> loop1(S1, S2); >> {error, Reason} -> >> io:format("error connect ~p~n", [Reason]) >> end; >> {error, timeout} -> >> loop(Listen, DestIp, DestPort, I); >> Error -> >> io:format("error accept ~p~n", [Error]), >> accept(Listen, DestIp, DestPort, I) >> end. >> >> loop1(S1, S2) -> >> active(S1, S2), >> receive >> {tcp, S1, Data} -> >> gen_tcp:send(S2, Data), >> loop1(S1, S2); >> {tcp, S2, Data} -> >> gen_tcp:send(S1, Data), >> loop1(S1, S2); >> {tcp_closed, S1} -> >> io:format("S1 close~n"), >> gen_tcp:close(S1), >> gen_tcp:close(S2); >> {tcp_closed, S2} -> >> io:format("S2 close~n"), >> gen_tcp:close(S1), >> gen_tcp:close(S2) >> end. >> >> active(S1,S2)-> >> inet:setopts(S1, [{active, once}]), >> inet:setopts(S2, [{active, once}]). >> >> ----------------------------------- >> epollwait.stp >> ------------------------------------ >> #! /usr/bin/env stap >> # >> # >> >> global epoll_timeout_flag, epoll_count, epoll_min, epoll_max, >> epoll_times, epoll_timeouts >> >> probe syscall.epoll_wait { >> if(timeout > 0) { >> epoll_timeout_flag[pid()] = 1 >> } >> } >> >> probe syscall.epoll_wait.return { >> c = cpu() >> p = execname() >> >> epoll_times[c,p] ++ >> epoll_count[c,p] += $return >> >> if($return == 0 && pid() in epoll_timeout_flag) { >> epoll_timeouts[c,p] ++ >> delete epoll_timeout_flag[pid()] >> } >> >> if(!([c, p] in epoll_min)) { >> epoll_min[c,p] = $return >> } else if($return < epoll_min[c,p]) { >> epoll_min[c,p] = $return >> } >> >> >> if($return > epoll_max[c,p]) { >> epoll_max[c,p] = $return >> } >> } >> >> probe timer.s($1) { >> printf ("%4s %45s %10s %10s %10s %10s %10s %10s\n", "cpu", >> "process", "times", "revents", "min", "max", "avg", "timeouts" ) >> foreach ([cpu, process] in epoll_count- limit 20) { >> all_epoll_times += epoll_times[cpu,process] >> all_epoll_count += epoll_count[cpu,process] >> all_epoll_timeouts += epoll_timeouts[cpu,process] >> } >> printf ("%4s %45s %10d %10d %10s %10s %10d %10d\n", >> "all", "", all_epoll_times, all_epoll_count, "-", "-", >> all_epoll_count == 0? 0:all_epoll_count/all_epoll_times, >> all_epoll_timeouts) >> >> foreach ([cpu, process] in epoll_count- limit 20) { >> printf ("[%2d] %45s %10d %10d %10d %10d %10d %10d\n", >> cpu, process, epoll_times[cpu, process], epoll_count[cpu, process], >> epoll_min[cpu, process], epoll_max[cpu, process], >> epoll_count[cpu,process]/epoll_times[cpu,process], >> epoll_timeouts[cpu,process]) >> } >> delete epoll_count >> delete epoll_min >> delete epoll_max >> delete epoll_times >> delete epoll_timeouts >> printf >> ("--------------------------------------------------------------------------\n\n" >> ) >> } >> >> ------------------------------------------------ >> ehttpd.erl >> ------------------------------------------------- >> -module(ehttpd). >> -compile(export_all). >> >> start() -> >> start(8888). >> start(Port) -> >> N = erlang:system_info(schedulers), >> listen(Port, N), >> io:format("ehttpd ready with ~b schedulers on port ~b~n", [N, Port]), >> >> register(?MODULE, self()), >> receive Any -> io:format("~p~n", [Any]) end. %% to stop: ehttpd!stop. >> >> listen(Port, N) -> >> Opts = [inet, >> binary, >> {active, false}, >> {recbuf, 8192}, >> {nodelay,true}, >> {reuseaddr, true}], >> >> {ok, S} = gen_tcp:listen(Port, Opts), >> lists:foreach(fun(I)-> spawn_opt(?MODULE, accept, [S, I], >> [{scheduler, I}]) end, lists:seq(1, N)). >> >> accept(S, I) -> >> case gen_tcp:accept(S) of >> {ok, Socket} -> >> spawn_opt(?MODULE, accept, [S, I],[{scheduler,I}] ), >> io:format("new connection @~p~n", [I]), >> loop(Socket,<<>>); >> Error -> erlang:error(Error) >> end. >> >> loop(S,B) -> >> inet:setopts(S, [{active, once}]), >> receive >> {tcp, S, Data} -> >> B1 = <>, >> case binary:part(B1,{byte_size(B1), -4}) of >> <<"\r\n\r\n">> -> >> Response = <<"HTTP/1.1 200 OK\r\nContent-Length: >> 12\r\nConnection: Keep-Alive\r\n\r\nhello world!">>, >> gen_tcp:send(S, Response), >> loop(S, <<>>); >> _ -> >> loop(S, B1) >> end; >> {tcp_closed, S} -> >> io:format("connection closed forced~n"), >> gen_tcp:close(S); >> Error -> >> io:format("unexpected message~p~n", [Error]), >> Error >> end. >> >> >> -- >> >> Best, >> >> Wei Cao >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> > > > > -- > > Best, > > Wei Cao From ok@REDACTED Wed Jul 11 03:12:37 2012 From: ok@REDACTED (Richard O'Keefe) Date: Wed, 11 Jul 2012 13:12:37 +1200 Subject: [erlang-questions] Local namespaces in case/try In-Reply-To: References: Message-ID: On 10/07/2012, at 8:43 PM, Dmitry Groshev wrote: > Is there any reason why we don't have them? Yes. > It's quite annoying to > have potential error in places like this: > > case do_something() of > {ok, Result} -> Result; > {error, Error} -> Error > end, > case do_another() of > {ok, Result} -> Result; > {error, Error} -> Error > > end, I *really* don't want to have to read code like that. It's a big help to *me* in reading a function if one name has one meaning throughout a clause. Heck, despite having been an Algol fan since 1960, I religiously use -Wshadow in gcc. I've seen (and made!) enough mistakes with two variables with the same name too close together not to. If Erlang did have this "feature" I would be agitating for its removal. If there is an error, why is the first case falling through into the second? Can you give us an example where this actually makes sense? This really has very little to do with 'if', 'case', or 'try'. There was a proposal years ago that begin E1 within E2 end should be like SML's local D1 within D2 end namely that bindings introduced in (E1,D1) are visible in (E2,D2) but not exported, while bindings introduced in (E2,D2) are exported. Failing that, -define(BEGIN, ((fun () ->). -define(END, end)())). ... ?BEGIN case do_something() of {ok, Result} -> Result ; {error, Error} -> Error end ?END, ?BEGIN case do_another() of {ok, Result} -> Result ; {error, Error} -> Error end ?END, should let you do what you want. The version of Erlang I'm using does not open-code (fun (T1,...,Tn) -> B end)(E1, ..., En), but I believe there's no reason why it couldn't, and in the presence of the preprocessor it's a useful thing to do. Even without that, I suspect that the overhead of making a closure and calling it will be hard for you to measure. From ok@REDACTED Wed Jul 11 03:27:53 2012 From: ok@REDACTED (Richard O'Keefe) Date: Wed, 11 Jul 2012 13:27:53 +1200 Subject: [erlang-questions] Local namespaces in case/try In-Reply-To: References: <4FFBEFA5.6030305@gmail.com> Message-ID: On 10/07/2012, at 9:40 PM, Sergei Lebedev wrote:, > > While this indeed *may* be a sign of a problem in your code, for the original example this is hardly the case. Splitting the function apart *only* because of compiler limitations sounds like an ad-hoc hack, not a real solution. I have shown in another message how we can get exactly the intended effect, in the language as it stands with no changes or extensions. The most important point here is that we are ***NOT*** talking about an accidental limitation where a compiler doesn't do something because the authors were too lazy or too incompetent or too resource-limited to do it or anything else like that. There is a software engineering reason: It is simply a bad idea to use the same name for two different variables within a short stretch of code. In my experience in Algol, C, Pascal, &c: if code that does that isn't buggy now, it soon will be. There is a programming language reason: Languages that allow nested variable scopes require some sort of declaration marker than says which variables are new. I can think of a few exceptions (TCL, Icon, Fortran 66) where the default is that any variable in a function is local unless explicitly marked as global, but those languages do not allow nested scopes within a function. Erlang used not to have nested variable scopes at all. When funs were introduced, it got some, but it is still the case that one function-or-fun-clause = one contour. Changing this requires some sort of language change. There is a backwards compatibility reason: Some programmers think that if it is good enough for Dijkstra's notation to say that after if p => x vir:= 1 [] non p => x vir:= 2 fi the variable x is initialised, and if it is good enough for Java to say that after int x; if (p) x = 1; else x = 2; the variable x is initialised, then it is good enough for Erlang to say that after if P -> X = 1 ; true -> X = 2 end the variable X is initialised. This is especially important when the choices need to initialise several variables, and packing them up into an entirely artificial tuple and then unpacking it afterwards is both error-prone and unclear. So what you really need is a way to say WHICH variables are private to an expression. Perhaps local Result, Error in case ... local Result, Error in case ... where 'local' would only count as a keyword if immediately followed by a variable name. As for 'hacks', I remain to be convinced that using the same variable for two different purposes in the same clause is a *SERIOUS* need. From ok@REDACTED Wed Jul 11 03:35:07 2012 From: ok@REDACTED (Richard O'Keefe) Date: Wed, 11 Jul 2012 13:35:07 +1200 Subject: [erlang-questions] Local namespaces in case/try In-Reply-To: References: <4FFBEFA5.6030305@gmail.com> Message-ID: On 10/07/2012, at 10:00 PM, Dmitry Groshev wrote: > Exactly what I was talking about. I can't see any sense in bloating the code with nonsensical functions It isn't the functions that are nonsensical, it is using the same variable with two different meanings which is in the strict literal sense of the word nonsensical. Give us a serious example! Show us some code that is easier to read if you do this! Yes, other functional languages have nested variable scopes within a single function (although the semantics is often defined in terms of nested functions), BUT those languages require you to introduce new bindings with keywords (define (f X) (let ((R) (E)) ;;; HERE! (let ((V (do-something))) (case (car V) ((result) (set! R (cadr V))) ((error) (set! E (cadr V)))))) (let ((R) (E)) ;;; HERE! (case (car V) ((result) (set! R (cadr V))) ((error) (set! E (cadr V))))))) Lisp and Scheme have LET and LET*. ML has let. Haskell has let and where. If you want Lisp-Flavoured Erlang, it exists. From ok@REDACTED Wed Jul 11 03:38:07 2012 From: ok@REDACTED (Richard O'Keefe) Date: Wed, 11 Jul 2012 13:38:07 +1200 Subject: [erlang-questions] Local namespaces in case/try In-Reply-To: <283C9941-B1E9-473B-A557-D18D51E8B56A@gmail.com> References: <4FFBEFA5.6030305@gmail.com> <4FFC0437.3060105@gmail.com> <283C9941-B1E9-473B-A557-D18D51E8B56A@gmail.com> Message-ID: <554EB8CE-5A68-4201-906B-6BBDAA439C59@cs.otago.ac.nz> On 10/07/2012, at 10:52 PM, Sergei Lebedev wrote: > Your argument makes sense, but I wonder why list comprehensions don't follow this design decision? Because they were added later, and because the pattern in a generator *has* to rebind those variables. > Of course, from the implementation point of view the answer is obvious -- list comprehensions are compiled to functions, There isn't the slightest reason why they *need* to be. That's no part of their definition, and avoiding such functions might well be more efficient. From avinash@REDACTED Wed Jul 11 06:42:21 2012 From: avinash@REDACTED (Avinash Dhumane) Date: Wed, 11 Jul 2012 10:12:21 +0530 Subject: [erlang-questions] CTS to Erlang mapping Message-ID: It seems there were attempts made in the late 1990's to create toolkits, with today's Erlang-like capabilities. Continuous Transaction System (CTS) seems to be one such toolkit with C-language bindings for applications. Is there any work available - e.g. papers, sample code, etc. - that shows how CTS programming model could be mapped to Erlang? Thanks From cyg.cao@REDACTED Wed Jul 11 08:44:49 2012 From: cyg.cao@REDACTED (Wei Cao) Date: Wed, 11 Jul 2012 14:44:49 +0800 Subject: [erlang-questions] Improve performance of IO bounded server written in Erlang via having pollset for each scheduler and bind port to scheduler together with process In-Reply-To: References: Message-ID: Find the reason :), please compile with ./configure CFLAGS="-DERTS_POLLSET_PER_SCHEDULER -g -O3 -fomit-frame-pointer" otherwise compiler optimization is disabled. (run ./configure with no CFLAGS set will include " -g -O3 -fomit-frame-pointer" by default) 2012/7/11 Zabrane Mickael : > Hi Wei, > > I did some other tests on our 8-Cores 64-bit machine: > > before: 157141 rps > after: 80245 rps > > Same behaviour as before ... hope this help. > > Regards, > Zabrane > > On Jul 10, 2012, at 4:30 AM, Wei Cao wrote: > >> My tests were all keepalived/persistent connections, and there was >> significant performance gain in these situations. but I wrote a http >> server with short connection just now, the performance did drop down, >> I'll find it out later :). >> >> 2012/7/10 Zabrane Mickael : >>> Hi, >>> >>> Performance of our HTTP Web Server drops down after applying your patches. >>> >>> Box: Linux F17, 4GB of RAM: >>> $ lscpu >>> Architecture: i686 >>> CPU op-mode(s): 32-bit, 64-bit >>> Byte Order: Little Endian >>> CPU(s): 4 >>> On-line CPU(s) list: 0-3 >>> Thread(s) per core: 1 >>> Core(s) per socket: 4 >>> Socket(s): 1 >>> Vendor ID: GenuineIntel >>> CPU family: 6 >>> Model: 23 >>> Stepping: 7 >>> CPU MHz: 2499.772 >>> BogoMIPS: 4999.54 >>> Virtualization: VT-x >>> L1d cache: 32K >>> L1i cache: 32K >>> L2 cache: 3072K >>> >>> Bench: >>> before: 77787 rps >>> after: 53056 rps >>> >>> Any hint to explain this result Wei ? >>> >>> Regards, >>> Zabrane >>> >>> >>> On Jul 9, 2012, at 11:01 AM, Wei Cao wrote: >>> >>> Hi, all >>> >>> We wrote a proxy server in Erlang, the proxy's logic is rather simple, >>> it listens on some TCP port, establishes new connection from user >>> client, forward packets back and forth between the client and backend >>> server after authentication until connection is closed. >>> >>> It's very easy to write such a proxy in Erlang, fork a process for >>> each new user connection and connect to the backend server in the same >>> process, the process works like a pipe, sockets from both side is set >>> to the active once mode, whenever a tcp packet is received from one >>> socket, the packet will be sent to other socket. (A simplified version >>> of proxy code is attached at the end of the mail) >>> >>> However, the performance is not quite satisfying, the proxy can handle >>> maximum only 40k requests on our 16 core machine(Intel Xeon L5630, >>> 2.13GHz) with heavy stress(100 concurrent clients). We then analyzed >>> the behavior of beam.smp use tools like tcprstat, mpstat, perf, and >>> SystemTap. >>> >>> tcprstat shows QPS is about 40k, have average 1.7 millisecond latency >>> for each request. >>> >>> timestamp count max min avg med stddev 95_max 95_avg 95_std 99_max 99_avg >>> 99_std >>> 1341813326 39416 17873 953 1718 1519 724 2919 1609 340 3813 1674 462 >>> 1341813327 40528 9275 884 1645 1493 491 2777 1559 290 3508 1619 409 >>> 1341813328 40009 18105 925 1694 1507 714 2868 1586 328 3753 1650 450 >>> >>> >>> mpstat shows 30% CPU is idle, >>> >>> 03:30:19 PM CPU %usr %nice %sys %iowait %irq %soft >>> %steal %guest %idle >>> 03:30:20 PM all 38.69 0.00 21.92 0.00 0.06 7.52 >>> 0.00 0.00 31.80 >>> 03:30:21 PM all 37.56 0.00 21.99 0.00 0.00 7.50 >>> 0.00 0.00 32.95 >>> >>> and perf top shows, much time is wasted in scheduler_wait, in spin wait I >>> guess. >>> >>> 9320.00 19.8% scheduler_wait >>> /home/mingsong.cw/erlangr16b/lib/erlang/erts-5.10/bin/beam.smp >>> 1813.00 3.9% process_main >>> /home/mingsong.cw/erlangr16b/lib/erlang/erts-5.10/bin/beam.smp >>> 1379.00 2.9% _spin_lock >>> /usr/lib/debug/lib/modules/2.6.32-131.21.1.tb477.el6.x86_64/vmlinux >>> 1201.00 2.6% schedule >>> /home/mingsong.cw/erlangr16b/lib/erlang/erts-5.10/bin/beam.smp >>> >>> It seems the performance may be associated with scheduler_wait() and >>> erts_check_io(), with a SystemTap script(attached at the end of this >>> mail), we can find out how many times the system call epoll_wait is >>> invoked by beam.smp and each time, how many revents it gets. >>> >>> cpu process times >>> revents min max avg timeouts >>> all >>> 1754 128042 - - 73 3 >>> [14] beam.smp 151 >>> 14127 82 97 93 0 >>> [ 5] beam.smp 142 >>> 13291 83 97 93 0 >>> [13] beam.smp 127 >>> 11948 86 96 94 0 >>> [ 6] beam.smp 127 >>> 11836 81 96 93 0 >>> [ 4] beam.smp 121 >>> 11323 81 96 93 0 >>> [15] beam.smp 117 >>> 10935 83 96 93 0 >>> [12] beam.smp 486 >>> 10128 0 96 20 2 >>> [ 1] beam.smp 71 >>> 6549 71 100 92 0 >>> [ 2] beam.smp 62 >>> 5695 82 96 91 0 >>> [ 7] beam.smp 55 >>> 5102 81 95 92 0 >>> [11] beam.smp 52 >>> 4822 85 95 92 0 >>> [ 9] beam.smp 52 >>> 4799 85 95 92 0 >>> [ 8] beam.smp 51 >>> 4680 78 95 91 0 >>> [10] beam.smp 49 >>> 4508 85 97 92 0 >>> [ 3] beam.smp 46 >>> 4211 81 95 91 0 >>> [ 0] beam.smp 44 >>> 4088 83 95 92 0 >>> >>> The resuls shows, epoll_wait is invoked 1754 times each second, and >>> get 73 io events in average. This is unacceptable for writing high >>> performance server. Because if epoll_wait is invoked no more than 2k >>> times per second, then read/write a packet would cost more than 500ms, >>> which causes long delay and affects the throughput finally. >>> >>> The problem relies on there is only one global pollset in system wide, >>> so at a time there is no more than one scheduler can call >>> erts_check_io() to obtain pending io tasks from underlying pollset, >>> and no scheduler can call erts_check_io() before all pending io >>> tasks're processed, so for IO bounded application, it's very likely >>> that a scheduler finish its own job, but must wait idly for other >>> schedulers to complete theirs. >>> >>> Hence, we develops a patch to slove this problem, by having a pollset >>> for each scheduler, so that each scheduler can invoke erts_check_io() >>> on its own pollset concurrently. After a scheduler complete its tasks, >>> it can invoke erts_check_io() immediately no matter what state other >>> schedulers're in. This patch also handles port migration situation, >>> all used file descriptors in each port're recorded, when a port is >>> migrated, these >>> fd 're removed from original scheduler's pollset, and added to new >>> scheduler's. >>> >>> Bind port to scheduler together with process is also helpful to >>> performance, it reduces the cost of thread switches and >>> synchronization, and bound port won't be migrated between schedulers. >>> >>> After apply the two patches, with the same pressure(100 concurrent >>> clients),epoll_wait is invoked 49332 times per second, and get 3 >>> revents each time in average, that is to say, our server responds >>> quicker and become more realtime. >>> >>> cpu process times >>> revents min max avg timeouts >>> all >>> 49332 217186 - - 4 3 >>> [ 2] beam.smp 3219 >>> 16050 2 7 4 0 >>> [11] beam.smp 4275 >>> 16033 1 6 3 0 >>> [ 8] beam.smp 4240 >>> 15992 1 6 3 0 >>> [ 9] beam.smp 4316 >>> 15964 0 6 3 2 >>> [10] beam.smp 4139 >>> 15851 1 6 3 0 >>> [ 3] beam.smp 4256 >>> 15816 1 6 3 0 >>> [ 1] beam.smp 3107 >>> 15800 2 7 5 0 >>> [ 0] beam.smp 3727 >>> 15259 1 6 4 0 >>> [ 7] beam.smp 2810 >>> 14722 3 7 5 0 >>> [13] beam.smp 1981 >>> 11499 4 7 5 0 >>> [15] beam.smp 2285 >>> 10942 3 6 4 0 >>> [14] beam.smp 2258 >>> 10866 3 6 4 0 >>> [ 4] beam.smp 2246 >>> 10849 3 6 4 0 >>> [ 6] beam.smp 2206 >>> 10730 3 6 4 0 >>> [12] beam.smp 2173 >>> 10573 3 6 4 0 >>> [ 5] beam.smp 2093 >>> 10240 3 6 4 0 >>> >>> scheduler_wait no longer take so much time now, >>> >>> 169.00 6.2% process_main beam.smp >>> 55.00 2.0% _spin_lock [kernel] >>> 45.00 1.7% driver_deliver_term beam.smp >>> >>> so is idle CPU time >>> 04:30:44 PM CPU %usr %nice %sys %iowait %irq %soft >>> %steal %guest %idle >>> 04:30:45 PM all 60.34 0.00 21.44 0.00 0.06 16.45 >>> 0.00 0.00 1.71 >>> 04:30:46 PM all 60.99 0.00 21.22 0.00 0.00 16.26 >>> 0.00 0.00 1.52 >>> >>> and tcprstat shows, QPS is getting 100K, latency is less than 1 millisecond >>> >>> timestamp count max min avg med stddev 95_max 95_avg 95_std 99_max 99_avg >>> 99_std >>> 1341822689 96078 11592 314 910 817 311 1447 869 228 1777 897 263 >>> 1341822690 100651 24245 209 914 819 381 1458 870 229 1800 899 263 >>> >>> I also write a extreamly simple keep-alive http server(attached at the >>> end of mail), to compare performance before and after applying the >>> patches, mearused with apache ab tool(ab -n 1000000 -c 100 -k ), a 30% >>> performance gain can be get. >>> >>> before >>> Requests per second: 103671.70 [#/sec] (mean) >>> Time per request: 0.965 [ms] (mean) >>> >>> after >>> Requests per second: 133701.24 [#/sec] (mean) >>> Time per request: 0.748 [ms] (mean) >>> >>> Patches can be found at github, compile with >>> ./configure CFLAGS=-DERTS_POLLSET_PER_SCHEDULER >>> >>> Pollset per scheduler: >>> >>> git fetch git://github.com/weicao/otp.git pollset_per_scheduler >>> >>> https://github.com/weicao/otp/compare/weicao:master...weicao:pollset_per_scheduler >>> https://github.com/weicao/otp/compare/weicao:master...weicao:pollset_per_scheduler.patch >>> >>> >>> Bind port to scheduler: >>> >>> git fetch git://github.com/weicao/otp.git bind_port_to_scheduler >>> >>> https://github.com/weicao/otp/compare/weicao:pollset_per_scheduler...weicao:bind_port_to_scheduler >>> https://github.com/weicao/otp/compare/weicao:pollset_per_scheduler...weicao:bind_port_to_scheduler.patch >>> >>> >>> Appendix: >>> >>> ----------------------------------- >>> proxy.erl >>> ------------------------------------ >>> >>> -module(proxy). >>> -compile(export_all). >>> >>> -define(RECBUF_SIZE, 8192). >>> -define(ACCEPT_TIMEOUT, 2000). >>> >>> start([MyPortAtom, DestIpAtom, DestPortAtom]) -> >>> {MyPort, []} = string:to_integer(atom_to_list(MyPortAtom)), >>> DestIp = atom_to_list(DestIpAtom), >>> {DestPort, []} = string:to_integer(atom_to_list(DestPortAtom)), >>> listen(MyPort, DestIp, DestPort), >>> receive Any -> >>> io:format("recv ~p~n", [Any]) >>> end. >>> >>> listen(MyPort, DestIp, DestPort) -> >>> io:format("start proxy on [local] 0.0.0.0:~p -> [remote] ~p:~p~n", >>> [MyPort, DestIp, DestPort]), >>> case gen_tcp:listen(MyPort, >>> [inet, >>> {ip, {0,0,0,0}}, >>> binary, >>> {reuseaddr, true}, >>> {recbuf, ?RECBUF_SIZE}, >>> {active, false}, >>> {nodelay, true} >>> ]) of >>> {ok, Listen} -> >>> N = erlang:system_info(schedulers), >>> lists:foreach(fun(I) -> accept(Listen, DestIp, DestPort, >>> I) end, lists:seq(1,N)); >>> {error, Reason} -> >>> io:format("error listen ~p~n", [Reason]) >>> end. >>> >>> accept(Listen, DestIp, DestPort, I) -> >>> spawn_opt(?MODULE, loop, [Listen, DestIp, DestPort, I], [{scheduler, >>> I}]). >>> >>> loop(Listen, DestIp, DestPort, I) -> >>> case gen_tcp:accept(Listen, ?ACCEPT_TIMEOUT) of >>> {ok, S1} -> >>> accept(Listen, DestIp, DestPort, I), >>> case catch gen_tcp:connect(DestIp, DestPort, >>> [inet, binary, {active, false}, >>> {reuseaddr, true}, {nodelay, true}]) of >>> {ok, S2} -> >>> io:format("new connection~n"), >>> loop1(S1, S2); >>> {error, Reason} -> >>> io:format("error connect ~p~n", [Reason]) >>> end; >>> {error, timeout} -> >>> loop(Listen, DestIp, DestPort, I); >>> Error -> >>> io:format("error accept ~p~n", [Error]), >>> accept(Listen, DestIp, DestPort, I) >>> end. >>> >>> loop1(S1, S2) -> >>> active(S1, S2), >>> receive >>> {tcp, S1, Data} -> >>> gen_tcp:send(S2, Data), >>> loop1(S1, S2); >>> {tcp, S2, Data} -> >>> gen_tcp:send(S1, Data), >>> loop1(S1, S2); >>> {tcp_closed, S1} -> >>> io:format("S1 close~n"), >>> gen_tcp:close(S1), >>> gen_tcp:close(S2); >>> {tcp_closed, S2} -> >>> io:format("S2 close~n"), >>> gen_tcp:close(S1), >>> gen_tcp:close(S2) >>> end. >>> >>> active(S1,S2)-> >>> inet:setopts(S1, [{active, once}]), >>> inet:setopts(S2, [{active, once}]). >>> >>> ----------------------------------- >>> epollwait.stp >>> ------------------------------------ >>> #! /usr/bin/env stap >>> # >>> # >>> >>> global epoll_timeout_flag, epoll_count, epoll_min, epoll_max, >>> epoll_times, epoll_timeouts >>> >>> probe syscall.epoll_wait { >>> if(timeout > 0) { >>> epoll_timeout_flag[pid()] = 1 >>> } >>> } >>> >>> probe syscall.epoll_wait.return { >>> c = cpu() >>> p = execname() >>> >>> epoll_times[c,p] ++ >>> epoll_count[c,p] += $return >>> >>> if($return == 0 && pid() in epoll_timeout_flag) { >>> epoll_timeouts[c,p] ++ >>> delete epoll_timeout_flag[pid()] >>> } >>> >>> if(!([c, p] in epoll_min)) { >>> epoll_min[c,p] = $return >>> } else if($return < epoll_min[c,p]) { >>> epoll_min[c,p] = $return >>> } >>> >>> >>> if($return > epoll_max[c,p]) { >>> epoll_max[c,p] = $return >>> } >>> } >>> >>> probe timer.s($1) { >>> printf ("%4s %45s %10s %10s %10s %10s %10s %10s\n", "cpu", >>> "process", "times", "revents", "min", "max", "avg", "timeouts" ) >>> foreach ([cpu, process] in epoll_count- limit 20) { >>> all_epoll_times += epoll_times[cpu,process] >>> all_epoll_count += epoll_count[cpu,process] >>> all_epoll_timeouts += epoll_timeouts[cpu,process] >>> } >>> printf ("%4s %45s %10d %10d %10s %10s %10d %10d\n", >>> "all", "", all_epoll_times, all_epoll_count, "-", "-", >>> all_epoll_count == 0? 0:all_epoll_count/all_epoll_times, >>> all_epoll_timeouts) >>> >>> foreach ([cpu, process] in epoll_count- limit 20) { >>> printf ("[%2d] %45s %10d %10d %10d %10d %10d %10d\n", >>> cpu, process, epoll_times[cpu, process], epoll_count[cpu, process], >>> epoll_min[cpu, process], epoll_max[cpu, process], >>> epoll_count[cpu,process]/epoll_times[cpu,process], >>> epoll_timeouts[cpu,process]) >>> } >>> delete epoll_count >>> delete epoll_min >>> delete epoll_max >>> delete epoll_times >>> delete epoll_timeouts >>> printf >>> ("--------------------------------------------------------------------------\n\n" >>> ) >>> } >>> >>> ------------------------------------------------ >>> ehttpd.erl >>> ------------------------------------------------- >>> -module(ehttpd). >>> -compile(export_all). >>> >>> start() -> >>> start(8888). >>> start(Port) -> >>> N = erlang:system_info(schedulers), >>> listen(Port, N), >>> io:format("ehttpd ready with ~b schedulers on port ~b~n", [N, Port]), >>> >>> register(?MODULE, self()), >>> receive Any -> io:format("~p~n", [Any]) end. %% to stop: ehttpd!stop. >>> >>> listen(Port, N) -> >>> Opts = [inet, >>> binary, >>> {active, false}, >>> {recbuf, 8192}, >>> {nodelay,true}, >>> {reuseaddr, true}], >>> >>> {ok, S} = gen_tcp:listen(Port, Opts), >>> lists:foreach(fun(I)-> spawn_opt(?MODULE, accept, [S, I], >>> [{scheduler, I}]) end, lists:seq(1, N)). >>> >>> accept(S, I) -> >>> case gen_tcp:accept(S) of >>> {ok, Socket} -> >>> spawn_opt(?MODULE, accept, [S, I],[{scheduler,I}] ), >>> io:format("new connection @~p~n", [I]), >>> loop(Socket,<<>>); >>> Error -> erlang:error(Error) >>> end. >>> >>> loop(S,B) -> >>> inet:setopts(S, [{active, once}]), >>> receive >>> {tcp, S, Data} -> >>> B1 = <>, >>> case binary:part(B1,{byte_size(B1), -4}) of >>> <<"\r\n\r\n">> -> >>> Response = <<"HTTP/1.1 200 OK\r\nContent-Length: >>> 12\r\nConnection: Keep-Alive\r\n\r\nhello world!">>, >>> gen_tcp:send(S, Response), >>> loop(S, <<>>); >>> _ -> >>> loop(S, B1) >>> end; >>> {tcp_closed, S} -> >>> io:format("connection closed forced~n"), >>> gen_tcp:close(S); >>> Error -> >>> io:format("unexpected message~p~n", [Error]), >>> Error >>> end. >>> >>> >>> -- >>> >>> Best, >>> >>> Wei Cao >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >>> >> >> >> >> -- >> >> Best, >> >> Wei Cao > > > -- Best, Wei Cao From erlangsiri@REDACTED Wed Jul 11 10:35:05 2012 From: erlangsiri@REDACTED (Siri Hansen) Date: Wed, 11 Jul 2012 10:35:05 +0200 Subject: [erlang-questions] Escript archives and the load path In-Reply-To: References: Message-ID: Hi Geoff! There is no good way to do this really. Since rebar is an escript, the code server can not read it as a general archive. The only option you have would be to extract the archive part from it, write it to a file with .ez extension and then add code path to it, e.g. 1> {ok,Stuff} = escript:extract("rebar",[]). {ok,[{shebang,default}, {comment,[]}, {emu_args,"-noshell -noinput"}, {archive,<<80,75,3,4,20,0,0,0,8,0,15,84,235,64,105,178, 62,157,168,18,0,...>>}]} 2> 2> Bin = proplists:get_value(archive,Stuff). <<80,75,3,4,20,0,0,0,8,0,15,84,235,64,105,178,62,157,168, 18,0,0,164,30,0,0,11,0,0,...>> 3> file:write_file("rebar.ez",Bin). ok 4> erl_prim_loader:list_dir("rebar.ez"). {ok,["priv","rebar_xref.beam","rebar_utils.beam", "rebar_upgrade.beam","rebar_templater.beam", "rebar_subdirs.beam","rebar_require_vsn.beam", "rebar_reltool.beam","rebar_rel_utils.beam", "rebar_protobuffs_compiler.beam","rebar_port_compiler.beam", "rebar_otp_app.beam","rebar_neotoma_compiler.beam", "rebar_log.beam","rebar_lfe_compiler.beam", "rebar_file_utils.beam","rebar_eunit.beam", "rebar_escripter.beam","rebar_erlydtl_compiler.beam", "rebar_erlc_compiler.beam","rebar_edoc.beam", "rebar_deps.beam","rebar_ct.beam","rebar_core.beam", "rebar_config.beam","rebar_cleaner.beam", [...]|...]} 5> code:which(rebar). non_existing 6> code:add_path("rebar.ez"). true 7> code:which(rebar). "rebar.ez/rebar.beam" 8> Regards /siri 2012/7/10 Geoff Cant > > Hi all, after looking for a while, I couldn't figure out how to tell > ERTS or the code server that /some/path/rebar was a valid place to load > modules from. > > Is there a magic function / command-line flag I can use to get the > contents of this escript archive into the code path? > > Cheers, > -- > Geoff Cant > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomasl_erlang@REDACTED Wed Jul 11 12:47:39 2012 From: thomasl_erlang@REDACTED (Thomas Lindgren) Date: Wed, 11 Jul 2012 03:47:39 -0700 (PDT) Subject: [erlang-questions] Looking for slides of a lightning talk In-Reply-To: <1709242D-0AD5-4070-A6EC-AB039E62ADFD@dmitriid.com> References: <4FFAAAEA.9060605@ninenines.eu> <0d6c6c5d-2b0f-48da-bae8-f440d232df92@googlegroups.com> <489B3810-49B2-4085-B468-DA037F7DE646@dmitriid.com> <1709242D-0AD5-4070-A6EC-AB039E62ADFD@dmitriid.com> Message-ID: <1342003659.84614.YahooMailNeo@web111404.mail.gq1.yahoo.com> >Ok, just the slides are here:?http://www.slideshare.net/dmitriid/erlang-sucks-euc-2012?(you can download them there as well) > >A PDF with some notes is here:?http://www.scribd.com/doc/99721085/Erlang-Sucks-EUC-2012 > >Notes isn't as good as talking in from of the audience, but oh well... Funny talk, my take aways were these: 1. Too few dedicated erlang web programmers, so still a lot of DIY. This may be a bootstrapping/community issue. Which is nontrivial, by the way. 2. Packages: Let me gripe a bit. At work, we've had endless trouble with Ruby gems, some hair tearing with CPAN, and have spent a couple of man years on packaging for RH and Debian. The whole process is still pretty clunky and hacky. So not a solved problem in the rest of the world either IMO and the erlang way has some advantages. But I agree that?more love is needed to catch up, especially on usability. 3. Image handling: I assume these image processing packages ultimately are wrappers for C/C++. At Diino, our next release (thanks Erik)?will be using RabbitMQ?to a collection of erlang workers that (ahem) invoke PIL through ports to generate thumbnails.?Well, it's erlang much of the way ...? NB: Writing efficient image code in erlang would be a sort of interesting project, even if it's reinventing the wheel. At this stage, probably?mostly inspiration to those interested in compilers though. Best, Thomas From bourinov@REDACTED Wed Jul 11 13:02:21 2012 From: bourinov@REDACTED (Max Bourinov) Date: Wed, 11 Jul 2012 15:02:21 +0400 Subject: [erlang-questions] Tools you use for productive design and coding Message-ID: Hi Erlangers, Let's share our valuable XP with each other. I am sure all of us have some good technics that boost our productivity or productivity in our teams. Anything that worth sharing should be shared! I will start with handy tool we use in out highly distributed team to draw sequence diagrams. It is not free, it is not open source but it is still great and you can use it for free as long as you want! (I am not related to that tool anyhow except I use it every day). So, here is the link: http://www.websequencediagrams.com/ Look forward for you tips and tricks, tools review etc. p.s. I bought this tool, but didn't install yet, because for me it seems perfectly fine to use demo version (it covers all my needs). p.s.s.: I am not promoting to buy the tool. Just sharing good stuff among us. Best regards and happy coding, Max -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Wed Jul 11 13:06:37 2012 From: essen@REDACTED (=?ISO-8859-1?Q?Lo=EFc_Hoguin?=) Date: Wed, 11 Jul 2012 13:06:37 +0200 Subject: [erlang-questions] Tools you use for productive design and coding In-Reply-To: References: Message-ID: <4FFD5E3D.2070606@ninenines.eu> On 07/11/2012 01:02 PM, Max Bourinov wrote: > Hi Erlangers, > > Let's share our valuable XP with each other. I am sure all of us have > some good technics that boost our productivity or productivity in our > teams. Anything that worth sharing should be shared! Brrrrrrraaaiiinnnnsssss. > I will start with handy tool we use in out highly distributed team to > draw sequence diagrams. It is not free, it is not open source but it is > still great and you can use it for free as long as you want! (I am not > related to that tool anyhow except I use it every day). > > So, here is the link: http://www.websequencediagrams.com/ Why not http://plantuml.sourceforge.net/sequence.html ? -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From hm@REDACTED Wed Jul 11 13:34:39 2012 From: hm@REDACTED (=?ISO-8859-1?Q?H=E5kan_Mattsson?=) Date: Wed, 11 Jul 2012 13:34:39 +0200 Subject: [erlang-questions] Escript archives and the load path In-Reply-To: References: Message-ID: The code server can handle one "primary" archive. It is usually used to load code from a running (active) escript but if the system has been started in the classic way (without an escript) the primary archive mechanism could be used to load code from a passive escript instead. See the example below. The "primary archive" mechanism is however only intended to be used in the context of one (active) escript. That's why code:set_primary_archive/3 is undocumented and its parameters may look somewhat strange. But if loading of code from passive escripts is something that is regarded as useful enough, it would not be too hard to generalize the mechanism (in erl_prim_loader and code_server) with one single "primary archive" to handle multiple "escript archives" instead. It's just to contribute to the community... /H?kan Eshell V5.9.1 (abort with ^G) 1> Mod = rebar. rebar 2> code:wh where_is_file/1 where_is_file/2 which/1 2> code:which(Mod). non_existing 3> Escript = "rebar". "rebar" 4> {ok,Props} = escript:extract(Escript,[]). {ok,[{shebang,default}, {comment,[]}, {emu_args,"-noshell -noinput"}, {archive,<<80,75,3,4,20,0,0,0,8,0,129,102,235,64,180, 80,127,93,93,54,0,...>>}]} 5> Bin = proplists:get_value(archive,Props). <<80,75,3,4,20,0,0,0,8,0,129,102,235,64,180,80,127,93,93, 54,0,0,228,65,0,0,11,0,0,...>> 6> {ok, Info} = file:read_file_info(Escript). {ok,{file_info,283118,regular,read_write, {{2012,7,11},{12,52,47}}, {{2012,7,11},{12,52,33}}, {{2012,7,11},{12,52,33}}, 33277,1,2056,0,4853045,1000,1000}} 7> code:set_primary_archive(Escript, Bin, Info). ok 8> code:which(Mod). "/home/hm/repos/rebar/rebar/rebar.beam" 9> On Wed, Jul 11, 2012 at 10:35 AM, Siri Hansen wrote: > Hi Geoff! > > There is no good way to do this really. Since rebar is an escript, the > code server can not read it as a general archive. The only option you have > would be to extract the archive part from it, write it to a file with .ez > extension and then add code path to it, e.g. > > 1> {ok,Stuff} = escript:extract("rebar",[]). > {ok,[{shebang,default}, > {comment,[]}, > {emu_args,"-noshell -noinput"}, > {archive,<<80,75,3,4,20,0,0,0,8,0,15,84,235,64,105,178, > 62,157,168,18,0,...>>}]} > 2> > 2> Bin = proplists:get_value(archive,Stuff). > <<80,75,3,4,20,0,0,0,8,0,15,84,235,64,105,178,62,157,168, > 18,0,0,164,30,0,0,11,0,0,...>> > 3> file:write_file("rebar.ez",Bin). > ok > 4> erl_prim_loader:list_dir("rebar.ez"). > {ok,["priv","rebar_xref.beam","rebar_utils.beam", > "rebar_upgrade.beam","rebar_templater.beam", > "rebar_subdirs.beam","rebar_require_vsn.beam", > "rebar_reltool.beam","rebar_rel_utils.beam", > "rebar_protobuffs_compiler.beam","rebar_port_compiler.beam", > "rebar_otp_app.beam","rebar_neotoma_compiler.beam", > "rebar_log.beam","rebar_lfe_compiler.beam", > "rebar_file_utils.beam","rebar_eunit.beam", > "rebar_escripter.beam","rebar_erlydtl_compiler.beam", > "rebar_erlc_compiler.beam","rebar_edoc.beam", > "rebar_deps.beam","rebar_ct.beam","rebar_core.beam", > "rebar_config.beam","rebar_cleaner.beam", > [...]|...]} > 5> code:which(rebar). > non_existing > 6> code:add_path("rebar.ez"). > true > 7> code:which(rebar). > "rebar.ez/rebar.beam" > 8> > > > Regards > /siri > > 2012/7/10 Geoff Cant > >> >> Hi all, after looking for a while, I couldn't figure out how to tell >> ERTS or the code server that /some/path/rebar was a valid place to load >> modules from. >> >> Is there a magic function / command-line flag I can use to get the >> contents of this escript archive into the code path? >> >> Cheers, >> -- >> Geoff Cant >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hm@REDACTED Wed Jul 11 13:34:40 2012 From: hm@REDACTED (=?ISO-8859-1?Q?H=E5kan_Mattsson?=) Date: Wed, 11 Jul 2012 13:34:40 +0200 Subject: [erlang-questions] Escript archives and the load path In-Reply-To: References: Message-ID: The code server can handle one "primary" archive. It is usually used to load code from a running (active) escript but if the system has been started in the classic way (without an escript) the primary archive mechanism could be used to load code from a passive escript instead. See the example below. The "primary archive" mechanism is however only intended to be used in the context of one (active) escript. That's why code:set_primary_archive/3 is undocumented and its parameters may look somewhat strange. But if loading of code from passive escripts is something that is regarded as useful enough, it would not be too hard to generalize the mechanism (in erl_prim_loader and code_server) with one single "primary archive" to handle multiple "escript archives" instead. It's just to contribute to the community... /H?kan Eshell V5.9.1 (abort with ^G) 1> Mod = rebar. rebar 2> code:wh where_is_file/1 where_is_file/2 which/1 2> code:which(Mod). non_existing 3> Escript = "rebar". "rebar" 4> {ok,Props} = escript:extract(Escript,[]). {ok,[{shebang,default}, {comment,[]}, {emu_args,"-noshell -noinput"}, {archive,<<80,75,3,4,20,0,0,0,8,0,129,102,235,64,180, 80,127,93,93,54,0,...>>}]} 5> Bin = proplists:get_value(archive,Props). <<80,75,3,4,20,0,0,0,8,0,129,102,235,64,180,80,127,93,93, 54,0,0,228,65,0,0,11,0,0,...>> 6> {ok, Info} = file:read_file_info(Escript). {ok,{file_info,283118,regular,read_write, {{2012,7,11},{12,52,47}}, {{2012,7,11},{12,52,33}}, {{2012,7,11},{12,52,33}}, 33277,1,2056,0,4853045,1000,1000}} 7> code:set_primary_archive(Escript, Bin, Info). ok 8> code:which(Mod). "/home/hm/repos/rebar/rebar/rebar.beam" 9> On Wed, Jul 11, 2012 at 10:35 AM, Siri Hansen wrote: > Hi Geoff! > > There is no good way to do this really. Since rebar is an escript, the > code server can not read it as a general archive. The only option you have > would be to extract the archive part from it, write it to a file with .ez > extension and then add code path to it, e.g. > > 1> {ok,Stuff} = escript:extract("rebar",[]). > {ok,[{shebang,default}, > {comment,[]}, > {emu_args,"-noshell -noinput"}, > {archive,<<80,75,3,4,20,0,0,0,8,0,15,84,235,64,105,178, > 62,157,168,18,0,...>>}]} > 2> > 2> Bin = proplists:get_value(archive,Stuff). > <<80,75,3,4,20,0,0,0,8,0,15,84,235,64,105,178,62,157,168, > 18,0,0,164,30,0,0,11,0,0,...>> > 3> file:write_file("rebar.ez",Bin). > ok > 4> erl_prim_loader:list_dir("rebar.ez"). > {ok,["priv","rebar_xref.beam","rebar_utils.beam", > "rebar_upgrade.beam","rebar_templater.beam", > "rebar_subdirs.beam","rebar_require_vsn.beam", > "rebar_reltool.beam","rebar_rel_utils.beam", > "rebar_protobuffs_compiler.beam","rebar_port_compiler.beam", > "rebar_otp_app.beam","rebar_neotoma_compiler.beam", > "rebar_log.beam","rebar_lfe_compiler.beam", > "rebar_file_utils.beam","rebar_eunit.beam", > "rebar_escripter.beam","rebar_erlydtl_compiler.beam", > "rebar_erlc_compiler.beam","rebar_edoc.beam", > "rebar_deps.beam","rebar_ct.beam","rebar_core.beam", > "rebar_config.beam","rebar_cleaner.beam", > [...]|...]} > 5> code:which(rebar). > non_existing > 6> code:add_path("rebar.ez"). > true > 7> code:which(rebar). > "rebar.ez/rebar.beam" > 8> > > > Regards > /siri > > 2012/7/10 Geoff Cant > >> >> Hi all, after looking for a while, I couldn't figure out how to tell >> ERTS or the code server that /some/path/rebar was a valid place to load >> modules from. >> >> Is there a magic function / command-line flag I can use to get the >> contents of this escript archive into the code path? >> >> Cheers, >> -- >> Geoff Cant >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bourinov@REDACTED Wed Jul 11 13:42:35 2012 From: bourinov@REDACTED (Max Bourinov) Date: Wed, 11 Jul 2012 15:42:35 +0400 Subject: [erlang-questions] Tools you use for productive design and coding In-Reply-To: <4FFD5E3D.2070606@ninenines.eu> References: <4FFD5E3D.2070606@ninenines.eu> Message-ID: Hi Lo?c, Didn't know about that tool. Seems that yours is more powerful. Will give it a try soon. Actually this is why I started this thread - I am sure that each of us knows something special and very useful. Something could be shared. You kust pointed me to a great tool. Thank you! :-) Best regards, Max On Wed, Jul 11, 2012 at 3:06 PM, Lo?c Hoguin wrote: > On 07/11/2012 01:02 PM, Max Bourinov wrote: > >> Hi Erlangers, >> >> Let's share our valuable XP with each other. I am sure all of us have >> some good technics that boost our productivity or productivity in our >> teams. Anything that worth sharing should be shared! >> > > Brrrrrrraaaiiinnnnsssss. > > > I will start with handy tool we use in out highly distributed team to >> draw sequence diagrams. It is not free, it is not open source but it is >> still great and you can use it for free as long as you want! (I am not >> related to that tool anyhow except I use it every day). >> >> So, here is the link: http://www.**websequencediagrams.com/ >> > > Why not http://plantuml.sourceforge.**net/sequence.html? > > -- > Lo?c Hoguin > Erlang Cowboy > Nine Nines > http://ninenines.eu > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From arthurbeall@REDACTED Wed Jul 11 15:41:38 2012 From: arthurbeall@REDACTED (Art Beall) Date: Wed, 11 Jul 2012 06:41:38 -0700 (PDT) Subject: [erlang-questions] DETS update or sync locking issue Message-ID: <1342014098.97528.YahooMailNeo@web39302.mail.mud.yahoo.com> Thanks for everybodys contributions to this email list! As Erlang newbies, the folks here have learned so much from reading this digest. Hopefully,?we will be able to contribute as our experience grows. ? We?ve been testing an erlang application that runs as a windows service that collects statistical data and stores timestamped records in a DETS table.?The application takes data from a TCP message and stores historical data from the message into?the table. Each record in the DETS table "grows" in size as the history is collected. The time span is truncated at 9 hours of data down to 6 hours. It continues to grow and truncate as time goes on. During the?test, the data wasnt changing, so the actual size of the table was not growing. The memory usage was way higher than thought. It eventually crashed after a few hours exceeding 1.5 GB.?Before it crashed,?an erlang:garbage_collect() was issued from another Erlang node using remsh, but this did not change anything. We?re using ramfile option with DETS, but sync?ing after every insert. A new process is created to parse each message and store the data. We are depending on the?capability of the DETS table to allow con-current access through many processes. Crash dump viewer indicated there were thousands of processes waiting to update the dets table. As a fix,?the autosave option of dets was changed to 1000ms, and the call to?dets:sync was removed from the callers?process.?This cleared up the memory problem and the?updates are very fast. Memory seems to be stable. We're trying to understand why the dets update or sync?hangs the processes. Any help or explainations would be appreciated! ? Art Beall Virtual Hold Technology, LLC??? From zabrane3@REDACTED Wed Jul 11 21:12:04 2012 From: zabrane3@REDACTED (Zabrane Mickael) Date: Wed, 11 Jul 2012 21:12:04 +0200 Subject: [erlang-questions] Improve performance of IO bounded server written in Erlang via having pollset for each scheduler and bind port to scheduler together with process In-Reply-To: References: <771ADB6B-9CD0-4A7A-A8A1-EC551B84BAD1@gmail.com> Message-ID: <0E26E896-FF0D-4036-A51A-8F3B8C1FA31C@gmail.com> Hi Wei, On Jul 11, 2012, at 2:35 PM, Wei Cao wrote: > sure, the steps is correct I re-installed everything from scratch with your second patch and tested your ehttpd web server example. before: ~55K rps after: ~70K rps but was unable to reach the 100K rps. Anyone courageous enough to help us reach the 100K rps? Regards, Zabrane -------------- next part -------------- An HTML attachment was scrubbed... URL: From fhamilton@REDACTED Wed Jul 11 21:48:32 2012 From: fhamilton@REDACTED (Felix Hamilton) Date: Wed, 11 Jul 2012 12:48:32 -0700 Subject: [erlang-questions] Tools you use for productive design and coding In-Reply-To: References: <4FFD5E3D.2070606@ninenines.eu> Message-ID: In the same vein, I work in distributed teams quite a bit, and recently started using this tool to do diagramming and design stuff collaboratively. I have been really impressed even with the free online only version, its one of those things that just works. And i have used plantUML as well as other UML tools. I am also not connected to these guys in any way, just have had a great experience using their stuff ... http://creately.com/ /Felix On Wed, Jul 11, 2012 at 4:42 AM, Max Bourinov wrote: > Hi Lo?c, > > Didn't know about that tool. Seems that yours is more powerful. Will give > it a try soon. > > Actually this is why I started this thread - I am sure that each of us > knows something special and very useful. Something could be shared. You > kust pointed me to a great tool. Thank you! :-) > > Best regards, > Max > > > > > On Wed, Jul 11, 2012 at 3:06 PM, Lo?c Hoguin wrote: > >> On 07/11/2012 01:02 PM, Max Bourinov wrote: >> >>> Hi Erlangers, >>> >>> Let's share our valuable XP with each other. I am sure all of us have >>> some good technics that boost our productivity or productivity in our >>> teams. Anything that worth sharing should be shared! >>> >> >> Brrrrrrraaaiiinnnnsssss. >> >> >> I will start with handy tool we use in out highly distributed team to >>> draw sequence diagrams. It is not free, it is not open source but it is >>> still great and you can use it for free as long as you want! (I am not >>> related to that tool anyhow except I use it every day). >>> >>> So, here is the link: http://www.**websequencediagrams.com/ >>> >> >> Why not http://plantuml.sourceforge.**net/sequence.html? >> >> >> -- >> Lo?c Hoguin >> >> Erlang Cowboy >> Nine Nines >> http://ninenines.eu >> >> >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ronny.meeus@REDACTED Wed Jul 11 22:35:36 2012 From: ronny.meeus@REDACTED (Ronny Meeus) Date: Wed, 11 Jul 2012 22:35:36 +0200 Subject: [erlang-questions] Improve performance of IO bounded server written in Erlang via having pollset for each scheduler and bind port to scheduler together with process In-Reply-To: <0E26E896-FF0D-4036-A51A-8F3B8C1FA31C@gmail.com> References: <771ADB6B-9CD0-4A7A-A8A1-EC551B84BAD1@gmail.com> <0E26E896-FF0D-4036-A51A-8F3B8C1FA31C@gmail.com> Message-ID: On Wed, Jul 11, 2012 at 9:12 PM, Zabrane Mickael wrote: > Hi Wei, > > On Jul 11, 2012, at 2:35 PM, Wei Cao wrote: > > sure, the steps is correct > > > I re-installed everything from scratch with your second patch and tested > your ehttpd web server example. > > before: ~55K rps > after: ~70K rps > > but was unable to reach the 100K rps. > > Anyone courageous enough to help us reach the 100K rps? > > Regards, > Zabrane > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > I think the feature that certain processes can be bound to a certain CPU at application level would also be useful. The solution implemented here only works (if I understand it well) when a process is using socket communication. There are scenarios where tasks are using just a little load (for example short processing of a message) where the overhead introduced by the scheduler is large so that nothing is gained by switching to a multi-core processor (in fact single core runs much faster). If certain the processes can be grouped on application level and bound to specific cores, the application scales a lot better. The other solution would be to run a separate VM instance per core but I have the feeling that this is more complex to manage and that there is also more overhead if messages need to be sent between the applications running on different VMs. This is a "taskset" like primitive that we know it from the Linux thread world. For example in thread "Strange observation running in an SMP environment." in this mailing list this would certainly be beneficial. Best regards, Ronny From freza@REDACTED Wed Jul 11 22:53:12 2012 From: freza@REDACTED (freza@REDACTED) Date: Wed, 11 Jul 2012 16:53:12 -0400 Subject: [erlang-questions] Tools you use for productive design and coding In-Reply-To: References: Message-ID: <20120711205312.GA22172@circlewave.net> On Wed, Jul 11, 2012 at 03:02:21PM +0400, Max Bourinov wrote: > Let's share our valuable XP with each other. I am sure all of us have some > good technics that boost our productivity or productivity in our teams. > Anything that worth sharing should be shared! LyX for writing pretty documents (when wiki page won't do): http://www.lyx.org/ There's very good MSC package for LaTeX: http://satoss.uni.lu/software/mscpackage/ Badly missing general diagram drawing tool (for supervision hierarchies, connectivity diagrams and such), tried pretty much anything that was open source but no joy so far. :-/ BR, -- Jachym From rtrlists@REDACTED Wed Jul 11 23:31:10 2012 From: rtrlists@REDACTED (Robert Raschke) Date: Wed, 11 Jul 2012 22:31:10 +0100 Subject: [erlang-questions] Tools you use for productive design and coding In-Reply-To: <20120711205312.GA22172@circlewave.net> References: <20120711205312.GA22172@circlewave.net> Message-ID: I sometimes find Graphviz useful for making moderately simple graph or tree diagrams. But it can get tricky when you have a particular layout already in mind. Robby On Jul 11, 2012 10:11 PM, wrote: > On Wed, Jul 11, 2012 at 03:02:21PM +0400, Max Bourinov wrote: > > Let's share our valuable XP with each other. I am sure all of us have > some > > good technics that boost our productivity or productivity in our teams. > > Anything that worth sharing should be shared! > > LyX for writing pretty documents (when wiki page won't do): > > http://www.lyx.org/ > > There's very good MSC package for LaTeX: > > http://satoss.uni.lu/software/mscpackage/ > > Badly missing general diagram drawing tool (for supervision hierarchies, > connectivity diagrams and such), tried pretty much anything that was open > source but no joy so far. :-/ > > BR, > -- Jachym > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hobson42@REDACTED Thu Jul 12 00:30:43 2012 From: hobson42@REDACTED (Ian) Date: Wed, 11 Jul 2012 23:30:43 +0100 Subject: [erlang-questions] Looking for slides of a lightning talk In-Reply-To: <1342003659.84614.YahooMailNeo@web111404.mail.gq1.yahoo.com> References: <4FFAAAEA.9060605@ninenines.eu> <0d6c6c5d-2b0f-48da-bae8-f440d232df92@googlegroups.com> <489B3810-49B2-4085-B468-DA037F7DE646@dmitriid.com> <1709242D-0AD5-4070-A6EC-AB039E62ADFD@dmitriid.com> <1342003659.84614.YahooMailNeo@web111404.mail.gq1.yahoo.com> Message-ID: <4FFDFE93.8040202@gmail.com> On 11/07/2012 11:47, Thomas Lindgren wrote: > 1. Too few dedicated erlang web programmers, so still a lot of DIY. This may be a bootstrapping/community issue. Which is nontrivial, by the way. > > 2. Packages: Let me gripe a bit. At work, we've had endless trouble with Ruby gems, some hair tearing with CPAN, and have spent a couple of man years on packaging for RH and Debian. The whole process is still pretty clunky and hacky. So not a solved problem in the rest of the world either IMO and the erlang way has some advantages. But I agree that more love is needed to catch up, especially on usability. Things need A LOT more love. I want to use cowboy to write some web software. Looks easy enough. There are Erlang drivers for the database I need to use, so all looks OK to proceed. I have just started with Erlang so not too familiar with things yet. This is my experience. 1) Found cowboy needs rebar - and rebar documentation is thin/non-existent/hiding. 2) Find rebar is not available on windows. OK. I'll upgrade a VM I have to Unbuntu 12:04. (which takes 3 hours, fails and needs nursing back to health. After removing and re-installing some packages things are now OK, apart from the occasional crash. Aside - virtual box provides a rather standard environment, so should not be a problem. And I though Linux was supposed not to crash like Windows. Not my experience. Oh well - press on. 3) I discover that the install of rebar into an apt-get install of Erlang will cause all sort of problems. In my book that means that at least one of those installs is not simple minded, or had inadequate checking. It is simply wrong. Maybe both. No matter, decide to install from source. 4) Screw up courage and download and compile Erlang 15B - all goes well. Woo-Hoo on a roll here! 5) Download, and compile rebar from source. That appears to work and it tells me to put it on the path. 6) Eh? Type "path" - gets "not installed" error message. Back to Google. Eventually find the command I need is env. See path is a whole slew of directories. Which should I use? Will the next upgrade to Ubuntu wipe some of them? Don't know. Don't know how to find out. Look in each one and decide that /usr/local/bin is probably a fair choice, because it contains erl. Copy rebar in. 7) Now I can start the instructions at http://blog.pqsdk.org/from-zero-to-deployment-with-cowboy-in-5-minu Note the title includes the sales pitch "In 5 minutes". I have already spent many hours getting the the start position. (At least I hope its the start position). 8) Issue first command and find hg is not an installed command. Apt-get install hg can't find the package. It wasn't the start position! Google turns up that hg is actually called mercurial and gives me the magic incantation to install it. 9) hg installs and I get to clone woody from bitbucket. Step 1 complete. 10) Next command is ./app.sh cowgirl . I think that create the app (or at least its framework). It reports no errors. 11) Then ./build-compile.sh - OK, build and compile. NO errors. Great. 12) Next comes ./build-release.sh This fails saying files exist and were not generated. It adds that I need force=1 on the command line to force it. Why? If forcing is a good idea, the software should just do it. Heck, its only to learn. I can delete and start over if necessary - just press on. Add force=1 to the command line. 13) When I do I get this. ian@REDACTED:~/projects/cowgirl$ ./build-release.sh force=1 ==> rel (create-node) ERROR: One or more files already exist on disk and were not generated: * "reltool.config" * "files/erl" * "files/nodetool" * "files/cowgirl" * "files/app.config" * "files/vm.args" To force overwriting, specify force=1 on the command line. ==> rel (generate) ERROR: generate failed while processing /home/ian/projects/cowgirl/rel: {'EXIT',{{badmatch,{error,"Release \"cowgirl\" uses non existing application cowgirl"}}, [{rebar_reltool,generate,2,[]}, {rebar_core,run_modules,4,[]}, {rebar_core,execute,4,[]}, {rebar_core,process_dir,4,[]}, {rebar_core,process_each,5,[]}, {rebar_core,process_dir,4,[]}, {rebar_core,process_commands,1,[]}, {rebar,main,1,[]}]}} ian@REDACTED:~/projects/cowgirl$ This is copy/pasted from the log, so I can see I did indeed spell force correctly. Where now? Is the pre-existence of those files a problem? I have no idea, and no idea how to find out. However experience tells me to make a real effort to fix the first error, even if later ones can be ignored. Should I delete them all and try again? What does the error in the generate mean? It can't mean the application cowgirl does not exist because hg/mercurial brought it down without errors and app.sh created it without errors. The talk is right. Erlang is just too damn difficult to do the easy stuff. It may have been humorously presented (I wasn't there), but the take-away it so true. Early difficulties are putting off many would-be adopters. (And I'm stuck! Help appreciated! ) Regards Ian From essen@REDACTED Thu Jul 12 01:03:41 2012 From: essen@REDACTED (=?ISO-8859-1?Q?Lo=EFc_Hoguin?=) Date: Thu, 12 Jul 2012 01:03:41 +0200 Subject: [erlang-questions] Looking for slides of a lightning talk In-Reply-To: <4FFDFE93.8040202@gmail.com> References: <4FFAAAEA.9060605@ninenines.eu> <0d6c6c5d-2b0f-48da-bae8-f440d232df92@googlegroups.com> <489B3810-49B2-4085-B468-DA037F7DE646@dmitriid.com> <1709242D-0AD5-4070-A6EC-AB039E62ADFD@dmitriid.com> <1342003659.84614.YahooMailNeo@web111404.mail.gq1.yahoo.com> <4FFDFE93.8040202@gmail.com> Message-ID: <4FFE064D.4000706@ninenines.eu> Hey, Don't worry if I am sounding a bit condescending in some parts of this reply, I'm not rude, I'm just french. :) On 07/12/2012 12:30 AM, Ian wrote: > On 11/07/2012 11:47, Thomas Lindgren wrote: >> 1. Too few dedicated erlang web programmers, so still a lot of DIY. >> This may be a bootstrapping/community issue. Which is nontrivial, by >> the way. >> >> 2. Packages: Let me gripe a bit. At work, we've had endless trouble >> with Ruby gems, some hair tearing with CPAN, and have spent a couple >> of man years on packaging for RH and Debian. The whole process is >> still pretty clunky and hacky. So not a solved problem in the rest of >> the world either IMO and the erlang way has some advantages. But I >> agree that more love is needed to catch up, especially on usability. > Things need A LOT more love. I want to use cowboy to write some web > software. Looks easy enough. There are Erlang drivers for the database I > need to use, so all looks OK to proceed. I have just started with > Erlang so not too familiar with things yet. This is my experience. > > 1) Found cowboy needs rebar - and rebar documentation is > thin/non-existent/hiding. I'll make sure to add instructions about that in the upcoming Cowboy documentation. > 2) Find rebar is not available on windows. OK. I'll upgrade a VM I have > to Unbuntu 12:04. (which takes 3 hours, fails and needs nursing back to > health. After removing and re-installing some packages things are now > OK, apart from the occasional crash. Aside - virtual box provides a > rather standard environment, so should not be a problem. And I though > Linux was supposed not to crash like Windows. Not my experience. Oh > well - press on. Cowboy isn't available on Windows either, AFAIK. None of my software has been tested properly on Windows (or at least I didn't get feedback about it). I think few open-source Erlang developers use Windows and unless they send patches or finance some work towards this it's going to improve very slowly. Not trying to make an apology of it, just explaining the whys. > 3) I discover that the install of rebar into an apt-get install of > Erlang will cause all sort of problems. > > In my book that means that at least one of those installs is not simple > minded, or had inadequate checking. > It is simply wrong. Maybe both. No matter, decide to install from source. That's because Debian and Ubuntu's Erlang packages is completely broken. Can't really blame Erlang for that. > 4) Screw up courage and download and compile Erlang 15B - all goes > well. Woo-Hoo on a roll here! As you noticed it's much easier from source. There's also Ubuntu packages provided by Erlang Solutions, or tool assisted source install using kerl. But these are all third party. > 5) Download, and compile rebar from source. That appears to work and it > tells me to put it on the path. > > 6) Eh? Type "path" - gets "not installed" error message. Back to Google. > Eventually find the command I need is env. See path is a whole slew of > directories. I'll definitely sound condescending with this one, but this is an honest question. Why do you not know about the PATH environment variable? That's generally useful in any kind of development work. Should this also be included in the various projects' documentation? > Which should I use? Will the next upgrade to Ubuntu wipe some of them? > Don't know. Don't know how to find out. Look in each one and decide that > /usr/local/bin is probably a fair choice, because it contains erl. Copy > rebar in. > > 7) Now I can start the instructions at > http://blog.pqsdk.org/from-zero-to-deployment-with-cowboy-in-5-minu > > Note the title includes the sales pitch "In 5 minutes". I have already > spent many hours getting the the start position. (At least I hope its > the start position). Don't think I knew about that one. There's probably various other tutorials here and there not explaining everything properly. I'm open to suggestions for the steps you feel are needed to be put in the documentation. I doubt you can go from 0 knowledge to running Cowboy application in 5 minutes though, Cowboy has a few pre-requisites like knowing rebar or knowing how to design and build an OTP application. Should all this be explained in the documentation though? Anyway Cowboy lacks proper user guides because the API is just about to change a little and I didn't want to duplicate efforts in writing for before and after, as the steps to 'get started' are changed the most. > 8) Issue first command and find hg is not an installed command. Apt-get > install hg can't find the package. > > It wasn't the start position! Google turns up that hg is actually > called mercurial and gives me the magic incantation to install it. > > 9) hg installs and I get to clone woody from bitbucket. Step 1 complete. > > 10) Next command is ./app.sh cowgirl . I think that create the app (or > at least its framework). It reports no errors. > > 11) Then ./build-compile.sh - OK, build and compile. NO errors. Great. > > 12) Next comes ./build-release.sh > > This fails saying files exist and were not generated. It adds that I > need force=1 on the command line to force it. Why? If forcing is a good > idea, the software should just do it. Heck, its only to learn. I can > delete and start over if necessary - just press on. Add force=1 to the > command line. > > 13) When I do I get this. > > ian@REDACTED:~/projects/cowgirl$ ./build-release.sh force=1 > ==> rel (create-node) > ERROR: One or more files already exist on disk and were not generated: > * "reltool.config" > * "files/erl" > * "files/nodetool" > * "files/cowgirl" > * "files/app.config" > * "files/vm.args" > To force overwriting, specify force=1 on the command line. > ==> rel (generate) > ERROR: generate failed while processing /home/ian/projects/cowgirl/rel: > {'EXIT',{{badmatch,{error,"Release \"cowgirl\" uses non existing > application cowgirl"}}, > [{rebar_reltool,generate,2,[]}, > {rebar_core,run_modules,4,[]}, > {rebar_core,execute,4,[]}, > {rebar_core,process_dir,4,[]}, > {rebar_core,process_each,5,[]}, > {rebar_core,process_dir,4,[]}, > {rebar_core,process_commands,1,[]}, > {rebar,main,1,[]}]}} > ian@REDACTED:~/projects/cowgirl$ > > This is copy/pasted from the log, so I can see I did indeed spell force > correctly. All these steps are about that third party tool, which hasn't been updated in a long time, and the rebar you use is probably too recent for it. Things still change a lot in both rebar and Cowboy so it's not surprising to see old guides stop working here and there. > Where now? Is the pre-existence of those files a problem? I have no > idea, and no idea how to find out. However experience tells me to make a > real effort to fix the first error, even if later ones can be ignored. > Should I delete them all and try again? > > What does the error in the generate mean? It can't mean the application > cowgirl does not exist because hg/mercurial brought it down without > errors and app.sh created it without errors. That tool is building a release. Do you know what a release is? It's one of the most advanced topic in Erlang/OTP, a lot of Erlang developers never used releases. That guide/tool is clearly not something useful for beginners. > The talk is right. Erlang is just too damn difficult to do the easy stuff. The problem you point out is general lack of good user guides. Erlang's community has many experts writing amazing software, some of it open source, but most of them do not have the time or skill to properly write good documentation. Writing good documentation is a work on its own and few developers are good at it. On the other hand, these experts are generally here on the mailing list or on IRC where they can provide quick help to solve specific problems. (And don't look at me, I'm no expert.) > It may have been humorously presented (I wasn't there), but the > take-away it so true. > > Early difficulties are putting off many would-be adopters. > > (And I'm stuck! Help appreciated! ) Easiest way to start is to clone the cowboy_examples repository and play around with the existing code: https://github.com/extend/cowboy_examples I've started rewriting examples with one proper application per example and hope to have most examples from misultin ported to Cowboy along with a few others and the user guides ready by the end of summer. Right now unfortunately I'm focusing on my OSCON talk next week so this has to wait a little. But I'm open to any and all suggestions on what you want in the documentation and/or examples for Cowboy or any of my other projects. Hope this helps. -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From tilman.holschuh@REDACTED Thu Jul 12 01:14:42 2012 From: tilman.holschuh@REDACTED (Tilman Holschuh) Date: Wed, 11 Jul 2012 16:14:42 -0700 Subject: [erlang-questions] Looking for slides of a lightning talk In-Reply-To: <1342003659.84614.YahooMailNeo@web111404.mail.gq1.yahoo.com> References: <4FFAAAEA.9060605@ninenines.eu> <0d6c6c5d-2b0f-48da-bae8-f440d232df92@googlegroups.com> <489B3810-49B2-4085-B468-DA037F7DE646@dmitriid.com> <1709242D-0AD5-4070-A6EC-AB039E62ADFD@dmitriid.com> <1342003659.84614.YahooMailNeo@web111404.mail.gq1.yahoo.com> Message-ID: <4FFE08E2.70602@gmail.com> Wouldn't it be nice to have an Erlang "wishlist"? Or does that exist somewhere? For next year's Spawnfest there was a proposal to collect ideas to make it easier for teams to pick a project. Maybe efforts could be unified? Cheers - Tilman On 12-07-11 03:47 AM, Thomas Lindgren wrote: >> Ok, just the slides are here: http://www.slideshare.net/dmitriid/erlang-sucks-euc-2012 (you can download them there as well) >> >> A PDF with some notes is here: http://www.scribd.com/doc/99721085/Erlang-Sucks-EUC-2012 >> >> Notes isn't as good as talking in from of the audience, but oh well... > > > Funny talk, my take aways were these: > > 1. Too few dedicated erlang web programmers, so still a lot of DIY. This may be a bootstrapping/community issue. Which is nontrivial, by the way. > > 2. Packages: Let me gripe a bit. At work, we've had endless trouble with Ruby gems, some hair tearing with CPAN, and have spent a couple of man years on packaging for RH and Debian. The whole process is still pretty clunky and hacky. So not a solved problem in the rest of the world either IMO and the erlang way has some advantages. But I agree that more love is needed to catch up, especially on usability. > > 3. Image handling: I assume these image processing packages ultimately are wrappers for C/C++. At Diino, our next release (thanks Erik) will be using RabbitMQ to a collection of erlang workers that (ahem) invoke PIL through ports to generate thumbnails. Well, it's erlang much of the way ... > > NB: Writing efficient image code in erlang would be a sort of interesting project, even if it's reinventing the wheel. At this stage, probably mostly inspiration to those interested in compilers though. > > Best, > Thomas > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From cyg.cao@REDACTED Thu Jul 12 03:48:09 2012 From: cyg.cao@REDACTED (Wei Cao) Date: Thu, 12 Jul 2012 09:48:09 +0800 Subject: [erlang-questions] Improve performance of IO bounded server written in Erlang via having pollset for each scheduler and bind port to scheduler together with process In-Reply-To: <0E26E896-FF0D-4036-A51A-8F3B8C1FA31C@gmail.com> References: <771ADB6B-9CD0-4A7A-A8A1-EC551B84BAD1@gmail.com> <0E26E896-FF0D-4036-A51A-8F3B8C1FA31C@gmail.com> Message-ID: we can reach 135 rps on a 16 core machine, it's quite reasonable to have 70k rps on a 8 core machine. lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 16 On-line CPU(s) list: 0-15 Thread(s) per core: 2 Core(s) per socket: 4 CPU socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 44 Stepping: 2 CPU MHz: 2134.000 BogoMIPS: 4266.58 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 12288K NUMA node0 CPU(s): 0-3,8-11 NUMA node1 CPU(s): 4-7,12-15 2012/7/12 Zabrane Mickael : > Hi Wei, > > On Jul 11, 2012, at 2:35 PM, Wei Cao wrote: > > sure, the steps is correct > > > I re-installed everything from scratch with your second patch and tested > your ehttpd web server example. > > before: ~55K rps > after: ~70K rps > > but was unable to reach the 100K rps. > > Anyone courageous enough to help us reach the 100K rps? > > Regards, > Zabrane > -- Best, Wei Cao From dcartt@REDACTED Thu Jul 12 03:56:30 2012 From: dcartt@REDACTED (David Cartt) Date: Wed, 11 Jul 2012 18:56:30 -0700 (PDT) Subject: [erlang-questions] Looking for slides of a lightning talk In-Reply-To: <4FFE064D.4000706@ninenines.eu> References: <4FFAAAEA.9060605@ninenines.eu> <0d6c6c5d-2b0f-48da-bae8-f440d232df92@googlegroups.com> <489B3810-49B2-4085-B468-DA037F7DE646@dmitriid.com> <1709242D-0AD5-4070-A6EC-AB039E62ADFD@dmitriid.com> <1342003659.84614.YahooMailNeo@web111404.mail.gq1.yahoo.com> <4FFDFE93.8040202@gmail.com> <4FFE064D.4000706@ninenines.eu> Message-ID: <547e0dd7-d0f4-45b4-9633-b11181c9165b@googlegroups.com> > > 2) Find rebar is not available on windows. OK. I'll upgrade a VM I have > > to Unbuntu 12:04. (which takes 3 hours, fails and needs nursing back to > > health. After removing and re-installing some packages things are now > > OK, apart from the occasional crash. Aside - virtual box provides a > > rather standard environment, so should not be a problem. And I though > > Linux was supposed not to crash like Windows. Not my experience. Oh > > well - press on. > > Cowboy isn't available on Windows either, AFAIK. None of my software has > been tested properly on Windows (or at least I didn't get feedback about > it). I think few open-source Erlang developers use Windows and unless > they send patches or finance some work towards this it's going to > improve very slowly. Not trying to make an apology of it, just > explaining the whys. > > A couple weeks ago I got Rebar and Cowboy working on Windows 7 and Cygwin. It was difficult but only because I wasn't familiar with the process or Rebar, and I learned a lot of value along the way. I had to make only one small change to a Rebar erl file and none to Cowboy. It required some simple Makefile changes too, and some kludged copying of erl and beam files to get dependencies working without errors, but I'm sure that could be done properly by someone with more Erlang and Rebar knowledge. I had to make only one source change, to Rebar, and that was to get the Cowboy tests (mostly) working. "<" is my version, ">" is original. $ diff rebar_ct.erl ../r/src 92,103c92,94 < Msg = case rebar_utils:sh("grep -e \"TEST COMPLETE\" -e \"{error,make_failed}\" \"" ++ RawLog ++ "\"", [{use_stdout, false}, return_on_error]) of < {ok, Msg1} -> Msg1; < {error, {1,[]}} -> "" < end, < --- > {ok, Msg} = > rebar_utils:sh("grep -e 'TEST COMPLETE' -e '{error,make_failed}' " > ++ RawLog, [{use_stdout, false}]), Test run results: Name Ok Failed Skipped(User/Auto) Missing Erlang.app-Cowboy.http_SUITE67 7 4 (4/0) 0 Erlang.app-Cowboy.proper_SUITE1 0 0 (0/0) 0 Erlang.app-Cowboy.ws_SUITE7 0 0 (0/0) 0 Cowboy works great, but there is one thing not working for me in Windows. The cowboy_http_static handler doesn't appear to be detecting file changes. I'll change a .js file, refresh the web page, and Cowboy responds that the file is unchanged. I'm using Etags ({attributes, [filepath, filesize, inode, mtime]}), the Etags are changing when the file changes, but Cowboy never seems to detect that change, according to its response to the browser. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Thu Jul 12 07:37:18 2012 From: ok@REDACTED (Richard O'Keefe) Date: Thu, 12 Jul 2012 17:37:18 +1200 Subject: [erlang-questions] Erlang and bbcode Message-ID: <5C229F26-0DB5-4C67-B8C3-434731D0B233@cs.otago.ac.nz> On reading the slides about "Erlang sucks" I thought, "what is bbcode and how hard can it be to write an Erlang parser for it?" Since having that thought, I've checked half a dozen definitions of bbcode and looked at a parser or two and am little the wiser. BBcode strikes me as truly bizarre. What is the point of entering something that looks pretty much like HTML except for using square brackets instead of angle brackets? But there is worse. - I cannot discover whether any particular character set or encoding is presumed and if so which one. (I'd *guess* Unicode/UTF-8, but a guess is all it would be.) - I cannot discover how you get a plain [ into text. Could it be [[? Could it be [[]? - I cannot discover exactly what is a well-formed tag and what is not. - I cannot discover whether [/*] is legal or not. - I cannot discover whether markup is legal inside a [url]...[/url] or not (it could be stripped out) - Same for [email] and [img] and [youtube] and [gvideo] - I cannot discover whether [size=n] takes n in points, pixels, percentage of default, or anything else (it seems that different systems do different things) - I cannot discover whether [youtube] and [gvideo] allow width/height like [img] or not. - Some descriptions say that :-) is processed as a smiley, and that other emoticons may be processed too, but I cannot find a list; others say [:-)] is a smiley; others say nothing about this. - It is not clear how the author of [quote-author]... should be rendered; I have a strong suspicion it should be locale-dependent. - It appears that different instances of bbcode support different tag sets out of the box and most of them allow some sort of customisation. - It appears to be _expected_ that different bbcode implementations will translate things differently (so [b]xxx[/b] might yield or or or something else), which means that it would be hard to make a test suite. Indeed, I can find no guarantee that [b] [i] and so on won't just be stripped out. If the lexical issues could be sorted out, one could easily enough write a BBcode -> XML value tree parser, and an XML -> XML translator to do things like Y -> Y Y -> Y and then use an existing XML -> text unparser. An non-validating XML parser in Erlang took me 275 lines, so I doubt bbcode would be much harder. From ulf@REDACTED Thu Jul 12 08:16:28 2012 From: ulf@REDACTED (Ulf Wiger) Date: Thu, 12 Jul 2012 08:16:28 +0200 Subject: [erlang-questions] Erlang and bbcode In-Reply-To: <5C229F26-0DB5-4C67-B8C3-434731D0B233@cs.otago.ac.nz> References: <5C229F26-0DB5-4C67-B8C3-434731D0B233@cs.otago.ac.nz> Message-ID: <9257D246-2C46-4692-B88C-9E99C31B357E@feuerlabs.com> On 12 Jul 2012, at 07:37, Richard O'Keefe wrote: > On reading the slides about "Erlang sucks" I thought, > "what is bbcode and how hard can it be to write an > Erlang parser for it?" > > Since having that thought, I've checked half a dozen > definitions of bbcode and looked at a parser or two > and am little the wiser. > > BBcode strikes me as truly bizarre. What is the point > of entering something that looks pretty much like HTML > except for using square brackets instead of angle brackets? > But there is worse. > > [snip worse] I guess this is indicative of a kind of problem that plagues most of the candidate development environments: they all include a variety of helper tools and conventions that work for those who have learned their strengths and idiosynchracies, but which may well look truly bizzarre to someone from the outside. Another part of this is of course that, as developers, we suffer a great deal of frustration on an almost daily (or at least weekly) basis, as things don't quite work the way we expect them, and we have to spend some time (occasionally quite a lot of time) fixing stuff that shouldn't be broken in the first place. When we are in our "home environment", this sort of stuff is par for the course, since we are learning things we consider useful, and also hope to eventually contribute some helpful improvement - or at least knowledge we can share to our fellow community members. As newcomers, we just tend to feel frustrated and alienated. But when spreading into a new domain, a development community of course has the choice of either copying the existing helper tools (complete with possibly bizzarre semantics), or trying to understand them and creating something similar, but more sane. Both approaches have their pros and cons, and either way, friction is inevitable. I am reminded of Todd Proebsting's excellent talk at the LL2 workshop at MIT 2002 - "Disruptive Programming Language Technologies" http://ll2.ai.mit.edu/talks/proebsting.ppt (Powerpoint) https://docs.google.com/viewer?url=http%3A%2F%2Fll2.ai.mit.edu%2Ftalks%2Fproebsting.ppt (Online view of the above) http://web.mit.edu/webcast/ailab/mit-ll2-s2-09nov02-80k.ram (webcast - first talk in the video) In particular, this line: My criteria: technology must Have disadvantages Be mostly ignored by recent PLDI and POPL conferences Alleviate real problems? ?What does it do?? If you pick a disruptive programming technology, expect some pain (by Todd's definition). The initial question then becomes "why use Erlang in the first place?", and, depending on the answer "is the pain worth it?" The answer in many cases may well be "no". BR, Ulf W Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. http://feuerlabs.com From zabrane3@REDACTED Thu Jul 12 08:19:16 2012 From: zabrane3@REDACTED (Zabrane Mickael) Date: Thu, 12 Jul 2012 08:19:16 +0200 Subject: [erlang-questions] Improve performance of IO bounded server written in Erlang via having pollset for each scheduler and bind port to scheduler together with process In-Reply-To: References: <771ADB6B-9CD0-4A7A-A8A1-EC551B84BAD1@gmail.com> <0E26E896-FF0D-4036-A51A-8F3B8C1FA31C@gmail.com> Message-ID: Hi Wei, We already surpassed the 100krps on an 8-cores machine with our HTTP server (~150K rps). My question was: could we reach the 100K rps on a 4-cores machine with ehttpd ? That will be awesome. Regards, Zabrane On Jul 12, 2012, at 3:48 AM, Wei Cao wrote: > we can reach 135 rps on a 16 core machine, it's quite reasonable to > have 70k rps on a 8 core machine. > > lscpu > > Architecture: x86_64 > CPU op-mode(s): 32-bit, 64-bit > Byte Order: Little Endian > CPU(s): 16 > On-line CPU(s) list: 0-15 > Thread(s) per core: 2 > Core(s) per socket: 4 > CPU socket(s): 2 > NUMA node(s): 2 > Vendor ID: GenuineIntel > CPU family: 6 > Model: 44 > Stepping: 2 > CPU MHz: 2134.000 > BogoMIPS: 4266.58 > Virtualization: VT-x > L1d cache: 32K > L1i cache: 32K > L2 cache: 256K > L3 cache: 12288K > NUMA node0 CPU(s): 0-3,8-11 > NUMA node1 CPU(s): 4-7,12-15 > > > > 2012/7/12 Zabrane Mickael : >> Hi Wei, >> >> On Jul 11, 2012, at 2:35 PM, Wei Cao wrote: >> >> sure, the steps is correct >> >> >> I re-installed everything from scratch with your second patch and tested >> your ehttpd web server example. >> >> before: ~55K rps >> after: ~70K rps >> >> but was unable to reach the 100K rps. >> >> Anyone courageous enough to help us reach the 100K rps? >> >> Regards, >> Zabrane >> > > > > -- > > Best, > > Wei Cao -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Thu Jul 12 08:21:17 2012 From: max.lapshin@REDACTED (Max Lapshin) Date: Thu, 12 Jul 2012 10:21:17 +0400 Subject: [erlang-questions] Improve performance of IO bounded server written in Erlang via having pollset for each scheduler and bind port to scheduler together with process In-Reply-To: References: <771ADB6B-9CD0-4A7A-A8A1-EC551B84BAD1@gmail.com> <0E26E896-FF0D-4036-A51A-8F3B8C1FA31C@gmail.com> Message-ID: Can You post code for benchmarks? Or it is here in topic and I've missed it? From dave@REDACTED Thu Jul 12 09:18:28 2012 From: dave@REDACTED (Dave Cottlehuber) Date: Thu, 12 Jul 2012 09:18:28 +0200 Subject: [erlang-questions] Looking for slides of a lightning talk In-Reply-To: <4FFDFE93.8040202@gmail.com> References: <4FFAAAEA.9060605@ninenines.eu> <0d6c6c5d-2b0f-48da-bae8-f440d232df92@googlegroups.com> <489B3810-49B2-4085-B468-DA037F7DE646@dmitriid.com> <1709242D-0AD5-4070-A6EC-AB039E62ADFD@dmitriid.com> <1342003659.84614.YahooMailNeo@web111404.mail.gq1.yahoo.com> <4FFDFE93.8040202@gmail.com> Message-ID: On 12 July 2012 00:30, Ian wrote: > 2) Find rebar is not available on windows. OK. I'll upgrade a VM I have to Erlang & Rebar works pretty well on Windows. Props to basho + OTP team for that. The install might not be obvious, but rebar + git + a recent erlang should be enough. However getting started with Erlang dev on Windows is a little disconcerting to start off with, and particularly for the server-side frameworks, there is a risk that they have some dependencies on things like unix shell scripts or specific C compiler expectations. These are not too hard to fix usually and I've found the upstream support here for addressing these stunningly good. Erlang/OTP rocks but the community is out there mining the asteroid belt already! What's hard to tell for a newcomer is what's new/bleeding edge, being actively developed, or robust & stable. Rebar is new-ish and moving fast, and cowboy also. So keeping a guide up to date is not straightforwards always! I'd gladly do a guide "Erlang on Windows" if I thought there was any interest in it - feel free to bump me offlist please. Understanding OTP & releases is really important in erl-world and I'm not there yet either. LYSE[1] & "OTP in Action" [2] are excellent groundings for this but expect a while before it all sinks in. A+ Dave [1]: http://learnyousomeerlang.com/ [2]: http://www.manning.com/logan/ From bchesneau@REDACTED Thu Jul 12 09:19:07 2012 From: bchesneau@REDACTED (Benoit Chesneau) Date: Thu, 12 Jul 2012 09:19:07 +0200 Subject: [erlang-questions] embedding elixir in erlang code Message-ID: I don't see any example on how to embed Elixir in your own erlang code. Is this possible? How? - beno?t From gordon@REDACTED Thu Jul 12 09:31:47 2012 From: gordon@REDACTED (Gordon Guthrie) Date: Thu, 12 Jul 2012 08:31:47 +0100 Subject: [erlang-questions] Erlang and bbcode In-Reply-To: <5C229F26-0DB5-4C67-B8C3-434731D0B233@cs.otago.ac.nz> References: <5C229F26-0DB5-4C67-B8C3-434731D0B233@cs.otago.ac.nz> Message-ID: I had an interesting experience when I wrote a markdown parser: https://github.com/hypernumbers/erlmarkdown Markdown is widely used and fairly mature with a lot of different implementations. Initially I started writing against the 'spec' but the spec is not so definitive. Then I copied over a chunk of tests from the C# implementation so my spec would be 'like them'. Finally I realised that I was an idiot and that I had it all backwards. In order to be useful it needed to play nicely with the javascript library that was going to preview it in the editor. The js library was pretty mature, widely used and substantially the default. So I then built a test case generator that took sample markedown and generated tests that checked my hand written parser returned the same result at the javascript library. At that point the old 'html doesn't care about whitespace' bug kicked in - some constructions in the js library would return additional spaces or carriage returns and the effort of unpicking my parser to make it match was too much effort. The key takeway is that you can't/shouldn't write any of this sort of stuff without cognisance of the wider environment that they will be used with - particularly with respect to js libraries. We moved away from using markdown and went to an open source WYSIWG html editor in the end. Gordon On 12 July 2012 06:37, Richard O'Keefe wrote: > On reading the slides about "Erlang sucks" I thought, > "what is bbcode and how hard can it be to write an > Erlang parser for it?" > > Since having that thought, I've checked half a dozen > definitions of bbcode and looked at a parser or two > and am little the wiser. > > BBcode strikes me as truly bizarre. What is the point > of entering something that looks pretty much like HTML > except for using square brackets instead of angle brackets? > But there is worse. > > - I cannot discover whether any particular character set > or encoding is presumed and if so which one. (I'd > *guess* Unicode/UTF-8, but a guess is all it would be.) > - I cannot discover how you get a plain [ into text. > Could it be [[? Could it be [[]? > - I cannot discover exactly what is a well-formed tag and > what is not. > - I cannot discover whether [/*] is legal or not. > - I cannot discover whether markup is legal inside > a [url]...[/url] or not (it could be stripped out) > - Same for [email] and [img] and [youtube] and [gvideo] > - I cannot discover whether [size=n] takes n in points, > pixels, percentage of default, or anything else (it > seems that different systems do different things) > - I cannot discover whether [youtube] and [gvideo] > allow width/height like [img] or not. > - Some descriptions say that :-) is processed as a > smiley, and that other emoticons may be processed > too, but I cannot find a list; others say [:-)] is > a smiley; others say nothing about this. > - It is not clear how the author of [quote-author]... > should be rendered; I have a strong suspicion it > should be locale-dependent. > - It appears that different instances of bbcode support > different tag sets out of the box and most of them > allow some sort of customisation. > - It appears to be _expected_ that different bbcode > implementations will translate things differently > (so [b]xxx[/b] might yield or or > or something else), > which means that it would be hard to make a test suite. > Indeed, I can find no guarantee that [b] [i] and so on > won't just be stripped out. > > If the lexical issues could be sorted out, one could easily > enough write a BBcode -> XML value tree parser, and an > XML -> XML translator to do things like > Y -> Y > Y -> Y > and then use an existing XML -> text unparser. > An non-validating XML parser in Erlang took me 275 lines, > so I doubt bbcode would be much harder. > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From zabrane3@REDACTED Thu Jul 12 09:37:08 2012 From: zabrane3@REDACTED (Zabrane Mickael) Date: Thu, 12 Jul 2012 09:37:08 +0200 Subject: [erlang-questions] Improve performance of IO bounded server written in Erlang via having pollset for each scheduler and bind port to scheduler together with process In-Reply-To: References: <771ADB6B-9CD0-4A7A-A8A1-EC551B84BAD1@gmail.com> <0E26E896-FF0D-4036-A51A-8F3B8C1FA31C@gmail.com> Message-ID: Hi Max, On Jul 12, 2012, at 8:21 AM, Max Lapshin wrote: > Can You post code for benchmarks? Or it is here in topic and I've missed it? The ehttpd code was sent by Wei (first email). Here it is again (attached). You also need his patched version of the VM. More precisely, the "bind_port_to_scheduler" patch: git clone git://github.com/weicao/otp.git cd otp git fetch git://github.com/weicao/otp.git bind_port_to_scheduler ./otp_build autoconf make clean CFLAGS="-DERTS_POLLSET_PER_SCHEDULER -g -O3 -fomit-frame-pointer" ./configure --prefix=/SOMEWHERE/usr make && make install Then: export PATH=/SOMEWHERE/usr/bin:$PATH Regards, Zabrane -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ehttpd.erl Type: application/octet-stream Size: 1585 bytes Desc: not available URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: From jose.valim@REDACTED Thu Jul 12 09:51:39 2012 From: jose.valim@REDACTED (=?ISO-8859-1?Q?Jos=E9_Valim?=) Date: Thu, 12 Jul 2012 09:51:39 +0200 Subject: [erlang-questions] embedding elixir in erlang code In-Reply-To: References: Message-ID: Hello Benoit, If you are using rebar, you should be able to use master as a dependency: https://github.com/elixir-lang/elixir.git After using it as a dependency, since Elixir follows a similar structure to OTP, you will need to choose which apps you want to load, there are some instructions inside the rebar.config file: https://github.com/elixir-lang/elixir/blob/master/rebar.config Then you can install Elixir's rebar plugin: https://github.com/yrashk/rebar_elixir_plugin And be able to compile and use Elixir files normally with your rebar workflow. If you are not using rebar, I believe the easiest approach is to clone elixir, run "make" and add the lib directory to ERL_LIBS. I will work on documenting this process and also on adding a readme to rebar_elixir_plugin with the available configuration options. Please let me know if you have any questions or ask away on #elixir-lang. * Jos? Valim www.plataformatec.com.br Founder and Lead Developer * On Thu, Jul 12, 2012 at 9:19 AM, Benoit Chesneau wrote: > I don't see any example on how to embed Elixir in your own erlang > code. Is this possible? How? > > - beno?t > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Thu Jul 12 10:02:05 2012 From: essen@REDACTED (=?ISO-8859-1?Q?Lo=EFc_Hoguin?=) Date: Thu, 12 Jul 2012 10:02:05 +0200 Subject: [erlang-questions] Erlang and bbcode In-Reply-To: <5C229F26-0DB5-4C67-B8C3-434731D0B233@cs.otago.ac.nz> References: <5C229F26-0DB5-4C67-B8C3-434731D0B233@cs.otago.ac.nz> Message-ID: <4FFE847D.3050204@ninenines.eu> Protip: BBCode was introduced by PHP developers who thought entering HTML-alike code converted using regexpes was more secure than entering HTML directly. It's not and it actually brought a lot of security issues on its own. See PHPBB for a more complete history. You're of course right about all this, there's no point in implementing BBCode, HTML works out just fine, we just need an HTML filtering library, perhaps something akin to HTML Purifier (http://htmlpurifier.org/). On 07/12/2012 07:37 AM, Richard O'Keefe wrote: > On reading the slides about "Erlang sucks" I thought, > "what is bbcode and how hard can it be to write an > Erlang parser for it?" > > Since having that thought, I've checked half a dozen > definitions of bbcode and looked at a parser or two > and am little the wiser. > > BBcode strikes me as truly bizarre. What is the point > of entering something that looks pretty much like HTML > except for using square brackets instead of angle brackets? > But there is worse. > > - I cannot discover whether any particular character set > or encoding is presumed and if so which one. (I'd > *guess* Unicode/UTF-8, but a guess is all it would be.) > - I cannot discover how you get a plain [ into text. > Could it be [[? Could it be [[]? > - I cannot discover exactly what is a well-formed tag and > what is not. > - I cannot discover whether [/*] is legal or not. > - I cannot discover whether markup is legal inside > a [url]...[/url] or not (it could be stripped out) > - Same for [email] and [img] and [youtube] and [gvideo] > - I cannot discover whether [size=n] takes n in points, > pixels, percentage of default, or anything else (it > seems that different systems do different things) > - I cannot discover whether [youtube] and [gvideo] > allow width/height like [img] or not. > - Some descriptions say that :-) is processed as a > smiley, and that other emoticons may be processed > too, but I cannot find a list; others say [:-)] is > a smiley; others say nothing about this. > - It is not clear how the author of [quote-author]... > should be rendered; I have a strong suspicion it > should be locale-dependent. > - It appears that different instances of bbcode support > different tag sets out of the box and most of them > allow some sort of customisation. > - It appears to be _expected_ that different bbcode > implementations will translate things differently > (so [b]xxx[/b] might yield or or > or something else), > which means that it would be hard to make a test suite. > Indeed, I can find no guarantee that [b] [i] and so on > won't just be stripped out. > > If the lexical issues could be sorted out, one could easily > enough write a BBcode -> XML value tree parser, and an > XML -> XML translator to do things like > Y -> Y > Y -> Y > and then use an existing XML -> text unparser. > An non-validating XML parser in Erlang took me 275 lines, > so I doubt bbcode would be much harder. > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From marc@REDACTED Thu Jul 12 10:11:56 2012 From: marc@REDACTED (Marc Worrell) Date: Thu, 12 Jul 2012 10:11:56 +0200 Subject: [erlang-questions] Erlang and bbcode In-Reply-To: <4FFE847D.3050204@ninenines.eu> References: <5C229F26-0DB5-4C67-B8C3-434731D0B233@cs.otago.ac.nz> <4FFE847D.3050204@ninenines.eu> Message-ID: On 12 jul. 2012, at 10:02, Lo?c Hoguin wrote: > You're of course right about all this, there's no point in implementing BBCode, HTML works out just fine, we just need an HTML filtering library, perhaps something akin to HTML Purifier (http://htmlpurifier.org/). Actually in Zotonic we have a HTML sanitizer, based on MochiWeb's HTML parser. Maybe we need to split some utils in their own repositories. Anyone wants to help with that? - Marc From gordon@REDACTED Thu Jul 12 10:28:30 2012 From: gordon@REDACTED (Gordon Guthrie) Date: Thu, 12 Jul 2012 09:28:30 +0100 Subject: [erlang-questions] Erlang and bbcode In-Reply-To: References: <5C229F26-0DB5-4C67-B8C3-434731D0B233@cs.otago.ac.nz> <4FFE847D.3050204@ninenines.eu> Message-ID: Marc Speak to the erlware fellas/Eric Merritt: https://github.com/erlware/ https://github.com/afiniate/erfc_parsers They have taken over a couple of my projects (erlmarkdown and RFC 4180 compliant CSV parser). Gordon On 12 July 2012 09:11, Marc Worrell wrote: > > On 12 jul. 2012, at 10:02, Lo?c Hoguin wrote: > > You're of course right about all this, there's no point in implementing > BBCode, HTML works out just fine, we just need an HTML filtering library, > perhaps something akin to HTML Purifier (http://htmlpurifier.org/). > > Actually in Zotonic we have a HTML sanitizer, based on MochiWeb's HTML > parser. > Maybe we need to split some utils in their own repositories. > > Anyone wants to help with that? > > - Marc > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From watson.timothy@REDACTED Thu Jul 12 10:29:54 2012 From: watson.timothy@REDACTED (Tim Watson) Date: Thu, 12 Jul 2012 09:29:54 +0100 Subject: [erlang-questions] Reading comments/annotations inside parse_transforms In-Reply-To: <4FFC806D.7090805@gmail.com> References: <4FFC806D.7090805@gmail.com> Message-ID: <9D5C1044-0E61-4891-9FA9-E66AF76BDE07@gmail.com> On 10 Jul 2012, at 20:20, Richard Carlsson wrote: > On 07/05/2012 11:10 AM, Tim Watson wrote: >> There doesn't appear to be any way of doing this currently. There is >> support in erl_syntax for working with these and there is also an >> erl_comment_scan module in the syntax-tools application which parses >> and returns comments from a file (or whatever). What there doesn't >> appear to be, is a means to get erl_parse to return comments as >> attributes (?) in the AST that the compiler passes to a >> parse_transform. >> >> My question then, is this: what is the best (only!?) way to process >> comments at the same time as source code in a parse_transform? So far, >> given that this seems unlikely to work OOTB, I'm assuming I'll have to >> do something like: >> >> 1. once you hit the 'file' attribute, parse the comments and stash >> these away in a useful place >> 2. process the forms as usual, comparing each line number for each >> form, against the line numbers stored for the comments >> 3. when you get an interleaving of code and comments, you know where >> the comments reside in the source file in relation to the form you're >> currently working with >> >> I *can* do this of course, but it seems like an awful lot of hard >> work. Why can't comments be preserved in the AST passed to a parse >> transform, and then just dropped later on!? Is there an easier way of >> doing this that I'm currently missing? > > The compiler toolchain just isn't targeted at preserving comments; they are treated as whitespace and are discarded already at the tokenization stage (erl_scan), even before the preprocessing stage (epp). After that, there's the parsing stage, and then the parse transforms are called. So, as you said, you'll need to use the -file("...") hints to locate the source files and read them again to extract the comments. > > It would certainly be possible to make a compiler that preserves comments for later passes, but the way it's currently done is the "traditional" way of writing a compiler, and changing it afterwards can be difficult, since all the code that expects the current behaviour has to be updated. > > The good news is that the work of digging out comments and connecting them to the abstract syntax trees has already been done, in the syntax_tools library - EDoc uses exactly this (although not as a parse transform). You can call erl_comment_scan:file/1 to get the comment lines, and then use erl_recomment:recomment_forms/2 to attach the comments to the abstract syntax trees that the parse transform got. The result is a extended abstract syntax tree as defined by the erl_syntax module. You can use the erl_syntax functions to manipulate the tree, and when you're done, you need to call erl_syntax:revert/1 to revert the representation to the form that the compiler understands (this loses the comments again). For a detailed usage example, see edoc_extract:source/3 and friends. > > /Richard Thanks Richard, that's a great deal better than what I had in mind! From erlang@REDACTED Thu Jul 12 10:38:06 2012 From: erlang@REDACTED (Joe Armstrong) Date: Thu, 12 Jul 2012 10:38:06 +0200 Subject: [erlang-questions] Erlang and bbcode In-Reply-To: <9257D246-2C46-4692-B88C-9E99C31B357E@feuerlabs.com> References: <5C229F26-0DB5-4C67-B8C3-434731D0B233@cs.otago.ac.nz> <9257D246-2C46-4692-B88C-9E99C31B357E@feuerlabs.com> Message-ID: On Thu, Jul 12, 2012 at 8:16 AM, Ulf Wiger wrote: > > On 12 Jul 2012, at 07:37, Richard O'Keefe wrote: > >> On reading the slides about "Erlang sucks" I thought, >> "what is bbcode and how hard can it be to write an >> Erlang parser for it?" >> >> Since having that thought, I've checked half a dozen >> definitions of bbcode and looked at a parser or two >> and am little the wiser. >> >> BBcode strikes me as truly bizarre. What is the point >> of entering something that looks pretty much like HTML >> except for using square brackets instead of angle brackets? >> But there is worse. >> >> [snip worse] > > I guess this is indicative of a kind of problem that plagues most > of the candidate development environments: they all include a > variety of helper tools and conventions that work for those who > have learned their strengths and idiosynchracies, but which may > well look truly bizzarre to someone from the outside. > This is why I have finally totally given up on all markup languages. Accuracy of markup is more important than ease of use, I want the results to be correct. I use XML markup and validate the results. I write small custom DTDs and validate against this. That way you get an accurate parse tree with minimal effort. My Erlang book, for example is entirely marked up in XML - the problem with producing text is not the markup - it's the content - that's the tricky bit. Markup that has an unpredictable result is a pain. It's a source of never ending amazement to me that after goodness knows how many year of web development things like gmail (which I'm using now) still can't get quote marks right. "These quotes are wrong," he banged on the keyboard. "They should be '66, 99' quotes." Only a few thousand years of typography down the drain. > Another part of this is of course that, as developers, we suffer a > great deal of frustration on an almost daily (or at least weekly) > basis, as things don't quite work the way we expect them, and > we have to spend some time (occasionally quite a lot of time) > fixing stuff that shouldn't be broken in the first place. Yes - I spend most of my time fixing stuff that shouldn't be broken. > > When we are in our "home environment", this sort of stuff is par > for the course, since we are learning things we consider > useful, and also hope to eventually contribute some helpful > improvement - or at least knowledge we can share to our fellow > community members. > > As newcomers, we just tend to feel frustrated and alienated. And as oldcomers. I still feel frustrated when faced with broken software. /Joe > > But when spreading into a new domain, a development > community of course has the choice of either copying the existing > helper tools (complete with possibly bizzarre semantics), or > trying to understand them and creating something similar, but > more sane. Both approaches have their pros and cons, and > either way, friction is inevitable. > > I am reminded of Todd Proebsting's excellent talk at the LL2 > workshop at MIT 2002 - "Disruptive Programming Language Technologies" > > http://ll2.ai.mit.edu/talks/proebsting.ppt (Powerpoint) > https://docs.google.com/viewer?url=http%3A%2F%2Fll2.ai.mit.edu%2Ftalks%2Fproebsting.ppt (Online view of the above) > http://web.mit.edu/webcast/ailab/mit-ll2-s2-09nov02-80k.ram (webcast - first talk in the video) > > In particular, this line: > My criteria: technology must > > Have disadvantages > Be mostly ignored by recent PLDI > and POPL conferences > Alleviate real problems? > ?What does it do?? > > If you pick a disruptive programming technology, expect some > pain (by Todd's definition). The initial question then becomes > "why use Erlang in the first place?", and, depending on the > answer "is the pain worth it?" > > The answer in many cases may well be "no". > > BR, > Ulf W > > Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. > http://feuerlabs.com > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From cyg.cao@REDACTED Thu Jul 12 10:45:14 2012 From: cyg.cao@REDACTED (Wei Cao) Date: Thu, 12 Jul 2012 16:45:14 +0800 Subject: [erlang-questions] Improve performance of IO bounded server written in Erlang via having pollset for each scheduler and bind port to scheduler together with process In-Reply-To: References: <771ADB6B-9CD0-4A7A-A8A1-EC551B84BAD1@gmail.com> <0E26E896-FF0D-4036-A51A-8F3B8C1FA31C@gmail.com> Message-ID: Hi, 2012/7/12 Zabrane Mickael : > Hi Wei, > > We already surpassed the 100krps on an 8-cores machine with our HTTP server > (~150K rps). Which erlang version did you use to get ~150k rps on 8-cores machine, patched or unpatched? if it was measured on a unpatched erlang version, would you mind measuring it on the patched version and let me know the result? > > My question was: could we reach the 100K rps on a 4-cores machine with > ehttpd ? > That will be awesome. > Today I found a lock bottleneck through SystemTap, trace-cmd and lcnt, after fixing it, ehttpd on my 16-cores can reach 325k rps. RX packets: 326117 TX packets: 326122 RX packets: 326845 TX packets: 326859 RX packets: 327983 TX packets: 327996 RX packets: 326651 TX packets: 326624 This is the upper limit of our Gigabit network card, I run ab on three standalone machines to make enough pressure, I posted the fix to github, have a try ~ > Regards, > Zabrane > > On Jul 12, 2012, at 3:48 AM, Wei Cao wrote: > > we can reach 135 rps on a 16 core machine, it's quite reasonable to > have 70k rps on a 8 core machine. > > lscpu > > Architecture: x86_64 > CPU op-mode(s): 32-bit, 64-bit > Byte Order: Little Endian > CPU(s): 16 > On-line CPU(s) list: 0-15 > Thread(s) per core: 2 > Core(s) per socket: 4 > CPU socket(s): 2 > NUMA node(s): 2 > Vendor ID: GenuineIntel > CPU family: 6 > Model: 44 > Stepping: 2 > CPU MHz: 2134.000 > BogoMIPS: 4266.58 > Virtualization: VT-x > L1d cache: 32K > L1i cache: 32K > L2 cache: 256K > L3 cache: 12288K > NUMA node0 CPU(s): 0-3,8-11 > NUMA node1 CPU(s): 4-7,12-15 > > > > 2012/7/12 Zabrane Mickael : > > Hi Wei, > > > On Jul 11, 2012, at 2:35 PM, Wei Cao wrote: > > > sure, the steps is correct > > > > I re-installed everything from scratch with your second patch and tested > > your ehttpd web server example. > > > before: ~55K rps > > after: ~70K rps > > > but was unable to reach the 100K rps. > > > Anyone courageous enough to help us reach the 100K rps? > > > Regards, > > Zabrane > > > > > > -- > > Best, > > Wei Cao > > > > -- Best, Wei Cao From erlang@REDACTED Thu Jul 12 10:45:41 2012 From: erlang@REDACTED (Joe Armstrong) Date: Thu, 12 Jul 2012 10:45:41 +0200 Subject: [erlang-questions] Erlang and bbcode In-Reply-To: References: <5C229F26-0DB5-4C67-B8C3-434731D0B233@cs.otago.ac.nz> <4FFE847D.3050204@ninenines.eu> Message-ID: On Thu, Jul 12, 2012 at 10:11 AM, Marc Worrell wrote: > > On 12 jul. 2012, at 10:02, Lo?c Hoguin wrote: >> You're of course right about all this, there's no point in implementing BBCode, HTML works out just fine, we just need an HTML filtering library, perhaps something akin to HTML Purifier (http://htmlpurifier.org/). > > Actually in Zotonic we have a HTML sanitizer, based on MochiWeb's HTML parser. > Maybe we need to split some utils in their own repositories. That sounds like a very good idea. It would also be nice to see a move toward simplified HTML that is correct and validated. The trend has been towards more complexity HTML5 is a monster - it has a lot of wonderful things but is still a monster. I'd like to see tiny html

Why tidy

&ldquot;Because ... &rdquot;

 You can get a long way with just

and character entities. The above satisfies 50% of my markup need for text :-) /Joe > > Anyone wants to help with that? > > - Marc > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From zabrane3@REDACTED Thu Jul 12 11:02:20 2012 From: zabrane3@REDACTED (Zabrane Mickael) Date: Thu, 12 Jul 2012 11:02:20 +0200 Subject: [erlang-questions] Improve performance of IO bounded server written in Erlang via having pollset for each scheduler and bind port to scheduler together with process In-Reply-To: References: <771ADB6B-9CD0-4A7A-A8A1-EC551B84BAD1@gmail.com> <0E26E896-FF0D-4036-A51A-8F3B8C1FA31C@gmail.com> Message-ID: Hi Wei, >> We already surpassed the 100krps on an 8-cores machine with our HTTP server >> (~150K rps). > > Which erlang version did you use to get ~150k rps on 8-cores machine, > patched or unpatched? We reach the 150K on the unpatched version. > if it was measured on a unpatched erlang > version, would you mind measuring it on the patched version and let me > know the result? I didn't yet adapted our code to use VM with your patch. I'll keep you informed. > Today I found a lock bottleneck through SystemTap, trace-cmd and lcnt, > after fixing it, ehttpd on my 16-cores can reach 325k rps. > > RX packets: 326117 TX packets: 326122 > RX packets: 326845 TX packets: 326859 > RX packets: 327983 TX packets: 327996 > RX packets: 326651 TX packets: 326624 > > This is the upper limit of our Gigabit network card, I run ab on three > standalone machines to make enough pressure, I posted the fix to > github, have a try ~ That's simply fantastic. Could you share your bottleneck tracking method? Any new VM patch to provide? Regards, Zabrane From cyg.cao@REDACTED Thu Jul 12 11:12:25 2012 From: cyg.cao@REDACTED (Wei Cao) Date: Thu, 12 Jul 2012 17:12:25 +0800 Subject: [erlang-questions] Improve performance of IO bounded server written in Erlang via having pollset for each scheduler and bind port to scheduler together with process In-Reply-To: References: <771ADB6B-9CD0-4A7A-A8A1-EC551B84BAD1@gmail.com> <0E26E896-FF0D-4036-A51A-8F3B8C1FA31C@gmail.com> Message-ID: The fix is git-pushed to patch branches, retrieved by git fetch git://github.com/weicao/otp.git bind_port_to_scheduler git fetch git://github.com/weicao/otp.git pollset_per_scheduler I used git push --force, so it's better to fetch it whole again. 2012/7/12 Zabrane Mickael : > Hi Wei, > >>> We already surpassed the 100krps on an 8-cores machine with our HTTP server >>> (~150K rps). >> >> Which erlang version did you use to get ~150k rps on 8-cores machine, >> patched or unpatched? > > We reach the 150K on the unpatched version. > > >> if it was measured on a unpatched erlang >> version, would you mind measuring it on the patched version and let me >> know the result? > > I didn't yet adapted our code to use VM with your patch. > I'll keep you informed. > >> Today I found a lock bottleneck through SystemTap, trace-cmd and lcnt, >> after fixing it, ehttpd on my 16-cores can reach 325k rps. >> >> RX packets: 326117 TX packets: 326122 >> RX packets: 326845 TX packets: 326859 >> RX packets: 327983 TX packets: 327996 >> RX packets: 326651 TX packets: 326624 >> >> This is the upper limit of our Gigabit network card, I run ab on three >> standalone machines to make enough pressure, I posted the fix to >> github, have a try ~ > > That's simply fantastic. Could you share your bottleneck tracking method? > Any new VM patch to provide? > > Regards, > Zabrane > -- Best, Wei Cao From marc@REDACTED Thu Jul 12 11:13:05 2012 From: marc@REDACTED (Marc Worrell) Date: Thu, 12 Jul 2012 11:13:05 +0200 Subject: [erlang-questions] Erlang and bbcode In-Reply-To: References: <5C229F26-0DB5-4C67-B8C3-434731D0B233@cs.otago.ac.nz> <4FFE847D.3050204@ninenines.eu> Message-ID: <1D101B84-6199-4832-9FB9-876206BFFBCC@worrell.nl> (and now also to Erlang Questions) On 12 jul. 2012, at 10:45, Joe Armstrong wrote: > That sounds like a very good idea. It would also be nice to see a move toward > simplified HTML that is correct and validated. The trend has been towards > more complexity HTML5 is a monster - it has a lot of wonderful things > but is still a monster. > > I'd like to see tiny html > > >

Why tidy

>

&ldquot;Because ... &rdquot;

> > > You can get a long way with just

and character entities. > > The above satisfies 50% of my markup need for text :-) The Zotonic HTML sanitizer is used to remove any "dangerous" HTML. As such it rejects all scripts, funny css extra's, object-tags etc. etc. It also removes anything it doesn't know (ie. white list based). And it rebalances the elements, so that the output is well formed. We use it to process all incoming HTML from tinyMCE. Have to think about anger etc, do they work well with external dependencies (like MochiWeb)? - Marc -- Marc Worrell - marc@REDACTED - http://WhatWebWhat.com/ KvK/Chamber of Commerce 34314912 - Amsterdam - Netherlands -------------- next part -------------- An HTML attachment was scrubbed... URL: From zabrane3@REDACTED Thu Jul 12 11:43:12 2012 From: zabrane3@REDACTED (Zabrane Mickael) Date: Thu, 12 Jul 2012 11:43:12 +0200 Subject: [erlang-questions] Improve performance of IO bounded server written in Erlang via having pollset for each scheduler and bind port to scheduler together with process In-Reply-To: References: <771ADB6B-9CD0-4A7A-A8A1-EC551B84BAD1@gmail.com> <0E26E896-FF0D-4036-A51A-8F3B8C1FA31C@gmail.com> <109FCB08-1C4F-4ABF-A87E-AE5A676C20B2@gmail.com> Message-ID: <18879021-6945-4333-A736-1A9431D9365B@gmail.com> Wei, On Jul 12, 2012, at 11:23 AM, Wei Cao wrote: > Not really, bind_port_to_scheduler is based on pollset_per_scheduler, > you can use bind_port_to_scheduler only, > > git fetch git://github.com/weicao/otp.git bind_port_to_scheduler can't compile the VM wihth your new patch: git clone git://github.com/weicao/otp.git cd otp git fetch git://github.com/weicao/otp.git bind_port_to_scheduler ./otp_build autoconf CFLAGS="-DERTS_POLLSET_PER_SCHEDULER -g -O3 -fomit-frame-pointer" ./configure --prefix=/SOMEWHERE/usr make && make install make clean (WITH OR WITHOUT make clean, it doesn't compile) make [...] gcc -m32 -DERTS_POLLSET_PER_SCHEDULER -g -O3 -fomit-frame-pointer -I/opt/otp/erts/i686-pc-linux-gnu -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -fno-tree-copyrename -D_GNU_SOURCE -DERTS_SMP -DHAVE_CONFIG_H -Wall -Wstrict-prototypes -Wmissing-prototypes -Wdeclaration-after-statement -DUSE_THREADS -D_THREAD_SAFE -D_REENTRANT -DPOSIX_THREADS -D_POSIX_THREAD_SAFE_FUNCTIONS -DERTS_ENABLE_LOCK_COUNT -Ii686-pc-linux-gnu/opt/smp -Ibeam -Isys/unix -Isys/common -Ii686-pc-linux-gnu -Izlib -Ipcre -Ihipe -I../include -I../include/i686-pc-linux-gnu -I../include/internal -I../include/internal/i686-pc-linux-gnu -c beam/erl_process_lock.c -o obj/i686-pc-linux-gnu/opt/smp/erl_process_lock.o beam/erl_process_lock.c: In function 'erts_lcnt_enable_proc_lock_count': beam/erl_process_lock.c:1275:15: error: 'process_tab' undeclared (first use in this function) beam/erl_process_lock.c:1275:15: note: each undeclared identifier is reported only once for each function it appears in make[3]: *** [obj/i686-pc-linux-gnu/opt/smp/erl_process_lock.o] Error 1 make[3]: Leaving directory `/opt/otp/erts/emulator' make[2]: *** [opt] Error 2 make[2]: Leaving directory `/opt/otp/erts/emulator' make[1]: *** [smp] Error 2 make[1]: Leaving directory `/opt/otp/erts' Regards, Zabrane > > > > > 2012/7/12 Zabrane Mickael : >> >> On Jul 12, 2012, at 11:12 AM, Wei Cao wrote: >> >>> The fix is git-pushed to patch branches, retrieved by >>> >>> git fetch git://github.com/weicao/otp.git bind_port_to_scheduler >>> git fetch git://github.com/weicao/otp.git pollset_per_scheduler >>> >>> I used git push --force, so it's better to fetch it whole again. >> >> >> So I need both patches? Right? >> >> Regards, >> Zabrane >> > > > > -- > > Best, > > Wei Cao -------------- next part -------------- An HTML attachment was scrubbed... URL: From cyg.cao@REDACTED Thu Jul 12 11:52:15 2012 From: cyg.cao@REDACTED (Wei Cao) Date: Thu, 12 Jul 2012 17:52:15 +0800 Subject: [erlang-questions] Improve performance of IO bounded server written in Erlang via having pollset for each scheduler and bind port to scheduler together with process In-Reply-To: <18879021-6945-4333-A736-1A9431D9365B@gmail.com> References: <771ADB6B-9CD0-4A7A-A8A1-EC551B84BAD1@gmail.com> <0E26E896-FF0D-4036-A51A-8F3B8C1FA31C@gmail.com> <109FCB08-1C4F-4ABF-A87E-AE5A676C20B2@gmail.com> <18879021-6945-4333-A736-1A9431D9365B@gmail.com> Message-ID: it seems you enable lcnt, do you include --enable-lock-counter in your ./configure command? lcnt is broken in master branch at the time I branched to commit patches, and it's fixed in the lastest otp master branch. 2012/7/12 Zabrane Mickael : > Wei, > > On Jul 12, 2012, at 11:23 AM, Wei Cao wrote: > > Not really, bind_port_to_scheduler is based on pollset_per_scheduler, > you can use bind_port_to_scheduler only, > > git fetch git://github.com/weicao/otp.git bind_port_to_scheduler > > > can't compile the VM wihth your new patch: > > git clone git://github.com/weicao/otp.git > cd otp > git fetch git://github.com/weicao/otp.git bind_port_to_scheduler > ./otp_build autoconf > CFLAGS="-DERTS_POLLSET_PER_SCHEDULER -g -O3 -fomit-frame-pointer" > ./configure --prefix=/SOMEWHERE/usr > make && make install > make clean (WITH OR WITHOUT make clean, it doesn't compile) > make > [...] > gcc -m32 -DERTS_POLLSET_PER_SCHEDULER -g -O3 -fomit-frame-pointer > -I/opt/otp/erts/i686-pc-linux-gnu -D_LARGEFILE_SOURCE > -D_FILE_OFFSET_BITS=64 -fno-tree-copyrename -D_GNU_SOURCE -DERTS_SMP > -DHAVE_CONFIG_H -Wall -Wstrict-prototypes -Wmissing-prototypes > -Wdeclaration-after-statement -DUSE_THREADS -D_THREAD_SAFE -D_REENTRANT > -DPOSIX_THREADS -D_POSIX_THREAD_SAFE_FUNCTIONS -DERTS_ENABLE_LOCK_COUNT > -Ii686-pc-linux-gnu/opt/smp -Ibeam -Isys/unix -Isys/common > -Ii686-pc-linux-gnu -Izlib -Ipcre -Ihipe -I../include > -I../include/i686-pc-linux-gnu -I../include/internal > -I../include/internal/i686-pc-linux-gnu -c beam/erl_process_lock.c -o > obj/i686-pc-linux-gnu/opt/smp/erl_process_lock.o > beam/erl_process_lock.c: In function 'erts_lcnt_enable_proc_lock_count': > beam/erl_process_lock.c:1275:15: error: 'process_tab' undeclared (first use > in this function) > beam/erl_process_lock.c:1275:15: note: each undeclared identifier is > reported only once for each function it appears in > make[3]: *** [obj/i686-pc-linux-gnu/opt/smp/erl_process_lock.o] Error 1 > make[3]: Leaving directory `/opt/otp/erts/emulator' > make[2]: *** [opt] Error 2 > make[2]: Leaving directory `/opt/otp/erts/emulator' > make[1]: *** [smp] Error 2 > make[1]: Leaving directory `/opt/otp/erts' > > Regards, > Zabrane > > > > > > 2012/7/12 Zabrane Mickael : > > > On Jul 12, 2012, at 11:12 AM, Wei Cao wrote: > > > The fix is git-pushed to patch branches, retrieved by > > > git fetch git://github.com/weicao/otp.git bind_port_to_scheduler > > git fetch git://github.com/weicao/otp.git pollset_per_scheduler > > > I used git push --force, so it's better to fetch it whole again. > > > > So I need both patches? Right? > > > Regards, > > Zabrane > > > > > > -- > > Best, > > Wei Cao > > > > -- Best, Wei Cao From cyg.cao@REDACTED Thu Jul 12 11:58:37 2012 From: cyg.cao@REDACTED (Wei Cao) Date: Thu, 12 Jul 2012 17:58:37 +0800 Subject: [erlang-questions] Improve performance of IO bounded server written in Erlang via having pollset for each scheduler and bind port to scheduler together with process In-Reply-To: References: <771ADB6B-9CD0-4A7A-A8A1-EC551B84BAD1@gmail.com> <0E26E896-FF0D-4036-A51A-8F3B8C1FA31C@gmail.com> Message-ID: 2012/7/12 Zabrane Mickael : > Hi Wei, > >>> We already surpassed the 100krps on an 8-cores machine with our HTTP server >>> (~150K rps). >> >> Which erlang version did you use to get ~150k rps on 8-cores machine, >> patched or unpatched? > > We reach the 150K on the unpatched version. > > >> if it was measured on a unpatched erlang >> version, would you mind measuring it on the patched version and let me >> know the result? > > I didn't yet adapted our code to use VM with your patch. > I'll keep you informed. > >> Today I found a lock bottleneck through SystemTap, trace-cmd and lcnt, >> after fixing it, ehttpd on my 16-cores can reach 325k rps. >> >> RX packets: 326117 TX packets: 326122 >> RX packets: 326845 TX packets: 326859 >> RX packets: 327983 TX packets: 327996 >> RX packets: 326651 TX packets: 326624 >> >> This is the upper limit of our Gigabit network card, I run ab on three >> standalone machines to make enough pressure, I posted the fix to >> github, have a try ~ > > That's simply fantastic. Could you share your bottleneck tracking method? > Any new VM patch to provide? through perf top, I see there is a big percentage of time is wasted in kernel _spin_lock 1894.00 16.0% _spin_lock /usr/lib/debug/lib/modules/2.6.32-131.21.1.tb477.el6.x86_64/vmlinux 566.00 4.8% process_main /home/mingsong.cw/erlangpps/lib/erlang/erts-5.10/bin/beam.smp After dumping and doing a statisics of _spin_lock's call stack via trace-cmd, I found most of _spin_lock is called by futex_wake, which is invoked by pthread mutex. Finally, I use lcnt to locate all lock collisions in erlang VM, found the mutex timeofday is the bottleneck. lock location #tries #collisions collisions [%] time [us] duration [%] ----- --------- ------- ------------ --------------- ---------- ------------- timeofday 'beam/erl_time_sup.c':939 895234 551957 61.6551 3185159 23.5296 timeofday 'beam/erl_time_sup.c':971 408006 264498 64.8270 1473816 10.8874 the mutex timeofday is locked each time erts_check_io is invoked to "sync the machine's idea of time", erts_check_io is executed hundreds of thounds of times per second, so it's locked too much times, hence reduce performance. I solved this problem by moving the sync operation into a standalone thread, invoked 1 time per millisecond > > Regards, > Zabrane > -- Best, Wei Cao From zabrane3@REDACTED Thu Jul 12 12:14:18 2012 From: zabrane3@REDACTED (Zabrane Mickael) Date: Thu, 12 Jul 2012 12:14:18 +0200 Subject: [erlang-questions] Improve performance of IO bounded server written in Erlang via having pollset for each scheduler and bind port to scheduler together with process In-Reply-To: References: <771ADB6B-9CD0-4A7A-A8A1-EC551B84BAD1@gmail.com> <0E26E896-FF0D-4036-A51A-8F3B8C1FA31C@gmail.com> <109FCB08-1C4F-4ABF-A87E-AE5A676C20B2@gmail.com> <18879021-6945-4333-A736-1A9431D9365B@gmail.com> Message-ID: <685E2C6D-0002-4DB2-94D6-0D82C7092AA8@gmail.com> Awesome analysis Wei. you were right about LCNT (--enable-lock-counter), now it's compiling just fine. I'll keep you informed. Regards, Zabrane On Jul 12, 2012, at 11:52 AM, Wei Cao wrote: > it seems you enable lcnt, do you include --enable-lock-counter in > your ./configure command? > > lcnt is broken in master branch at the time I branched to commit > patches, and it's fixed in the lastest otp master branch. > > > 2012/7/12 Zabrane Mickael : >> Wei, >> >> On Jul 12, 2012, at 11:23 AM, Wei Cao wrote: >> >> Not really, bind_port_to_scheduler is based on pollset_per_scheduler, >> you can use bind_port_to_scheduler only, >> >> git fetch git://github.com/weicao/otp.git bind_port_to_scheduler >> >> >> can't compile the VM wihth your new patch: >> >> git clone git://github.com/weicao/otp.git >> cd otp >> git fetch git://github.com/weicao/otp.git bind_port_to_scheduler >> ./otp_build autoconf >> CFLAGS="-DERTS_POLLSET_PER_SCHEDULER -g -O3 -fomit-frame-pointer" >> ./configure --prefix=/SOMEWHERE/usr >> make && make install >> make clean (WITH OR WITHOUT make clean, it doesn't compile) >> make >> [...] >> gcc -m32 -DERTS_POLLSET_PER_SCHEDULER -g -O3 -fomit-frame-pointer >> -I/opt/otp/erts/i686-pc-linux-gnu -D_LARGEFILE_SOURCE >> -D_FILE_OFFSET_BITS=64 -fno-tree-copyrename -D_GNU_SOURCE -DERTS_SMP >> -DHAVE_CONFIG_H -Wall -Wstrict-prototypes -Wmissing-prototypes >> -Wdeclaration-after-statement -DUSE_THREADS -D_THREAD_SAFE -D_REENTRANT >> -DPOSIX_THREADS -D_POSIX_THREAD_SAFE_FUNCTIONS -DERTS_ENABLE_LOCK_COUNT >> -Ii686-pc-linux-gnu/opt/smp -Ibeam -Isys/unix -Isys/common >> -Ii686-pc-linux-gnu -Izlib -Ipcre -Ihipe -I../include >> -I../include/i686-pc-linux-gnu -I../include/internal >> -I../include/internal/i686-pc-linux-gnu -c beam/erl_process_lock.c -o >> obj/i686-pc-linux-gnu/opt/smp/erl_process_lock.o >> beam/erl_process_lock.c: In function 'erts_lcnt_enable_proc_lock_count': >> beam/erl_process_lock.c:1275:15: error: 'process_tab' undeclared (first use >> in this function) >> beam/erl_process_lock.c:1275:15: note: each undeclared identifier is >> reported only once for each function it appears in >> make[3]: *** [obj/i686-pc-linux-gnu/opt/smp/erl_process_lock.o] Error 1 >> make[3]: Leaving directory `/opt/otp/erts/emulator' >> make[2]: *** [opt] Error 2 >> make[2]: Leaving directory `/opt/otp/erts/emulator' >> make[1]: *** [smp] Error 2 >> make[1]: Leaving directory `/opt/otp/erts' >> >> Regards, >> Zabrane >> >> >> >> >> >> 2012/7/12 Zabrane Mickael : >> >> >> On Jul 12, 2012, at 11:12 AM, Wei Cao wrote: >> >> >> The fix is git-pushed to patch branches, retrieved by >> >> >> git fetch git://github.com/weicao/otp.git bind_port_to_scheduler >> >> git fetch git://github.com/weicao/otp.git pollset_per_scheduler >> >> >> I used git push --force, so it's better to fetch it whole again. >> >> >> >> So I need both patches? Right? >> >> >> Regards, >> >> Zabrane >> >> >> >> >> >> -- >> >> Best, >> >> Wei Cao >> >> >> >> > > > > -- > > Best, > > Wei Cao From zabrane3@REDACTED Thu Jul 12 13:01:47 2012 From: zabrane3@REDACTED (Zabrane Mickael) Date: Thu, 12 Jul 2012 13:01:47 +0200 Subject: [erlang-questions] Improve performance of IO bounded server written in Erlang via having pollset for each scheduler and bind port to scheduler together with process In-Reply-To: References: <771ADB6B-9CD0-4A7A-A8A1-EC551B84BAD1@gmail.com> <0E26E896-FF0D-4036-A51A-8F3B8C1FA31C@gmail.com> Message-ID: Hi, Good news. With the new (today) patch: old bench: ~70K rps new bench: ~85K rps More than 15K rps handled now !! We're not far from the 100K rps ;-) Well done Wei. Regards, Zabrane On Jul 12, 2012, at 11:58 AM, Wei Cao wrote: > 2012/7/12 Zabrane Mickael : >> Hi Wei, >> >>>> We already surpassed the 100krps on an 8-cores machine with our HTTP server >>>> (~150K rps). >>> >>> Which erlang version did you use to get ~150k rps on 8-cores machine, >>> patched or unpatched? >> >> We reach the 150K on the unpatched version. >> >> >>> if it was measured on a unpatched erlang >>> version, would you mind measuring it on the patched version and let me >>> know the result? >> >> I didn't yet adapted our code to use VM with your patch. >> I'll keep you informed. >> >>> Today I found a lock bottleneck through SystemTap, trace-cmd and lcnt, >>> after fixing it, ehttpd on my 16-cores can reach 325k rps. >>> >>> RX packets: 326117 TX packets: 326122 >>> RX packets: 326845 TX packets: 326859 >>> RX packets: 327983 TX packets: 327996 >>> RX packets: 326651 TX packets: 326624 >>> >>> This is the upper limit of our Gigabit network card, I run ab on three >>> standalone machines to make enough pressure, I posted the fix to >>> github, have a try ~ >> >> That's simply fantastic. Could you share your bottleneck tracking method? >> Any new VM patch to provide? > > through perf top, I see there is a big percentage of time is wasted in > kernel _spin_lock > > 1894.00 16.0% _spin_lock > /usr/lib/debug/lib/modules/2.6.32-131.21.1.tb477.el6.x86_64/vmlinux > 566.00 4.8% process_main > /home/mingsong.cw/erlangpps/lib/erlang/erts-5.10/bin/beam.smp > > After dumping and doing a statisics of _spin_lock's call stack via > trace-cmd, I found most of _spin_lock is called by futex_wake, which > is invoked by pthread mutex. > > Finally, I use lcnt to locate all lock collisions in erlang VM, found > the mutex timeofday is the bottleneck. > > lock > location #tries #collisions collisions [%] time > [us] duration [%] > > ----- --------- ------- ------------ > --------------- ---------- ------------- > > timeofday 'beam/erl_time_sup.c':939 895234 551957 > 61.6551 3185159 23.5296 > > timeofday 'beam/erl_time_sup.c':971 408006 264498 > 64.8270 1473816 10.8874 > > > the mutex timeofday is locked each time erts_check_io is invoked to > "sync the machine's idea of time", erts_check_io is executed hundreds > of thounds of times per second, so it's locked too much times, hence > reduce performance. > > I solved this problem by moving the sync operation into a standalone > thread, invoked 1 time per millisecond > > > >> >> Regards, >> Zabrane >> > > > > -- > > Best, > > Wei Cao From zabrane3@REDACTED Thu Jul 12 13:09:02 2012 From: zabrane3@REDACTED (Zabrane Mickael) Date: Thu, 12 Jul 2012 13:09:02 +0200 Subject: [erlang-questions] Improve performance of IO bounded server written in Erlang via having pollset for each scheduler and bind port to scheduler together with process In-Reply-To: References: <771ADB6B-9CD0-4A7A-A8A1-EC551B84BAD1@gmail.com> <0E26E896-FF0D-4036-A51A-8F3B8C1FA31C@gmail.com> Message-ID: For all using ab or siege, I strongly advice you to move to a better tool. The best one for now which support theadings and the fast libev is weighttp (from Lighttpd webserver project): weighttp use exactly the same ab syntax. So nothing to change in your benchs. Hope this help ! ################################################### # Weighttp: # http://redmine.lighttpd.net/projects/weighttp/wiki ################################################### INSTALL 1. LibEV (http://software.schmorp.de/pkg/libev.html) cvs -z3 -d :pserver:anonymous@REDACTED/schmorpforge co libev cd libev aclocal && automake --add-missing && autoconf libtoolize --copy --force --ltdl sh autogen.sh && ./configure --prefix=/usr && make && make install 2. Weighttp (http://redmine.lighttpd.net/projects/weighttp/wiki) git clone git://git.lighttpd.net/weighttp cd weighttp ./waf configure ./waf build ./waf install Regards, Zabrane On Jul 12, 2012, at 1:01 PM, Zabrane Mickael wrote: > Hi, > > Good news. With the new (today) patch: > > old bench: ~70K rps > new bench: ~85K rps > > More than 15K rps handled now !! > We're not far from the 100K rps ;-) > > Well done Wei. > > Regards, > Zabrane > > On Jul 12, 2012, at 11:58 AM, Wei Cao wrote: > >> 2012/7/12 Zabrane Mickael : >>> Hi Wei, >>> >>>>> We already surpassed the 100krps on an 8-cores machine with our HTTP server >>>>> (~150K rps). >>>> >>>> Which erlang version did you use to get ~150k rps on 8-cores machine, >>>> patched or unpatched? >>> >>> We reach the 150K on the unpatched version. >>> >>> >>>> if it was measured on a unpatched erlang >>>> version, would you mind measuring it on the patched version and let me >>>> know the result? >>> >>> I didn't yet adapted our code to use VM with your patch. >>> I'll keep you informed. >>> >>>> Today I found a lock bottleneck through SystemTap, trace-cmd and lcnt, >>>> after fixing it, ehttpd on my 16-cores can reach 325k rps. >>>> >>>> RX packets: 326117 TX packets: 326122 >>>> RX packets: 326845 TX packets: 326859 >>>> RX packets: 327983 TX packets: 327996 >>>> RX packets: 326651 TX packets: 326624 >>>> >>>> This is the upper limit of our Gigabit network card, I run ab on three >>>> standalone machines to make enough pressure, I posted the fix to >>>> github, have a try ~ >>> >>> That's simply fantastic. Could you share your bottleneck tracking method? >>> Any new VM patch to provide? >> >> through perf top, I see there is a big percentage of time is wasted in >> kernel _spin_lock >> >> 1894.00 16.0% _spin_lock >> /usr/lib/debug/lib/modules/2.6.32-131.21.1.tb477.el6.x86_64/vmlinux >> 566.00 4.8% process_main >> /home/mingsong.cw/erlangpps/lib/erlang/erts-5.10/bin/beam.smp >> >> After dumping and doing a statisics of _spin_lock's call stack via >> trace-cmd, I found most of _spin_lock is called by futex_wake, which >> is invoked by pthread mutex. >> >> Finally, I use lcnt to locate all lock collisions in erlang VM, found >> the mutex timeofday is the bottleneck. >> >> lock >> location #tries #collisions collisions [%] time >> [us] duration [%] >> >> ----- --------- ------- ------------ >> --------------- ---------- ------------- >> >> timeofday 'beam/erl_time_sup.c':939 895234 551957 >> 61.6551 3185159 23.5296 >> >> timeofday 'beam/erl_time_sup.c':971 408006 264498 >> 64.8270 1473816 10.8874 >> >> >> the mutex timeofday is locked each time erts_check_io is invoked to >> "sync the machine's idea of time", erts_check_io is executed hundreds >> of thounds of times per second, so it's locked too much times, hence >> reduce performance. >> >> I solved this problem by moving the sync operation into a standalone >> thread, invoked 1 time per millisecond >> >> >> >>> >>> Regards, >>> Zabrane >>> >> >> >> >> -- >> >> Best, >> >> Wei Cao > > Regards, Zabrane -------------- next part -------------- An HTML attachment was scrubbed... URL: From cgsmcmlxxv@REDACTED Thu Jul 12 13:49:47 2012 From: cgsmcmlxxv@REDACTED (CGS) Date: Thu, 12 Jul 2012 13:49:47 +0200 Subject: [erlang-questions] Looking for slides of a lightning talk In-Reply-To: <4FFDFE93.8040202@gmail.com> References: <4FFAAAEA.9060605@ninenines.eu> <0d6c6c5d-2b0f-48da-bae8-f440d232df92@googlegroups.com> <489B3810-49B2-4085-B468-DA037F7DE646@dmitriid.com> <1709242D-0AD5-4070-A6EC-AB039E62ADFD@dmitriid.com> <1342003659.84614.YahooMailNeo@web111404.mail.gq1.yahoo.com> <4FFDFE93.8040202@gmail.com> Message-ID: Hi, You blame Erlang for an application written in it which doesn't support MS Windows... Well, that reminds me of a case of saying noSQL databases are not good just because that person/team misunderstood the usage of a particular noSQL database. Interesting how people try to generalize their impressions to a category from a wrongly chosen example. Don't get me wrong, I am not that good in Erlang to mock you, but I think you should ask someone more knowledgeable before you to say something like "Erlang is just too damn difficult to do the easy stuff." I do easy stuff in Erlang. It would probably be more difficult to write a proper C code to do those simple things (like proper threading, distributed applications and so on). Now, the problem is what do you need cowboy for? If you need an webserver for MS Windows which is written in Erlang, you can try Yaws (you have installers here: http://yaws.hyber.org/download/). If you need a Erlang-based webserver which you can integrate it in your application, you will need either a Linux or Cygwin (for most of the open source projects, but you can do it without if you use, for example, lhttpc - https://bitbucket.org/etc/lhttpc/src - where you can modify the Makefile to be compatible with MS VS or, for even less, a .bat file). I would avoid installing Linux in a MS Windows box (I'd rather do it oppositely). Erlang under MS Windows works as nice as under Linux (as far as I could play with it under MS Windows), but the installation under MS Windows is much simpler (I would suggest for Linux to compile it from source with the options you need rather than installing it from repositories). I don't know what is your background in programming (and I don't want to know), but there are nice tutorials on web for all the categories of programmers (coming from OOP, functional or whatever else languages). Just get familiarized with Erlang and after that make an impression of your own about the language itself. I am also relatively new in Erlang and I consider I cannot afford to make any statement about how good or not is Erlang. I know only that are some nice things and some not so nice things in Erlang (I leave finding examples to your experience). CGS On Thu, Jul 12, 2012 at 12:30 AM, Ian wrote: > On 11/07/2012 11:47, Thomas Lindgren wrote: > >> 1. Too few dedicated erlang web programmers, so still a lot of DIY. This >> may be a bootstrapping/community issue. Which is nontrivial, by the way. >> >> 2. Packages: Let me gripe a bit. At work, we've had endless trouble with >> Ruby gems, some hair tearing with CPAN, and have spent a couple of man >> years on packaging for RH and Debian. The whole process is still pretty >> clunky and hacky. So not a solved problem in the rest of the world either >> IMO and the erlang way has some advantages. But I agree that more love is >> needed to catch up, especially on usability. >> > Things need A LOT more love. I want to use cowboy to write some web > software. Looks easy enough. There are Erlang drivers for the database I > need to use, so all looks OK to proceed. I have just started with Erlang > so not too familiar with things yet. This is my experience. > > 1) Found cowboy needs rebar - and rebar documentation is > thin/non-existent/hiding. > > 2) Find rebar is not available on windows. OK. I'll upgrade a VM I have > to Unbuntu 12:04. (which takes 3 hours, fails and needs nursing back to > health. After removing and re-installing some packages things are now OK, > apart from the occasional crash. Aside - virtual box provides a rather > standard environment, so should not be a problem. And I though Linux was > supposed not to crash like Windows. Not my experience. Oh well - press on. > > 3) I discover that the install of rebar into an apt-get install of Erlang > will cause all sort of problems. > > In my book that means that at least one of those installs is not simple > minded, or had inadequate checking. > It is simply wrong. Maybe both. No matter, decide to install from source. > > 4) Screw up courage and download and compile Erlang 15B - all goes well. > Woo-Hoo on a roll here! > > 5) Download, and compile rebar from source. That appears to work and it > tells me to put it on the path. > > 6) Eh? Type "path" - gets "not installed" error message. Back to Google. > Eventually find the command I need is env. See path is a whole slew of > directories. > > Which should I use? Will the next upgrade to Ubuntu wipe some of them? > Don't know. Don't know how to find out. Look in each one and decide that > /usr/local/bin is probably a fair choice, because it contains erl. Copy > rebar in. > > 7) Now I can start the instructions at http://blog.pqsdk.org/from-** > zero-to-deployment-with-**cowboy-in-5-minu > > Note the title includes the sales pitch "In 5 minutes". I have already > spent many hours getting the the start position. (At least I hope its the > start position). > > 8) Issue first command and find hg is not an installed command. Apt-get > install hg can't find the package. > > It wasn't the start position! Google turns up that hg is actually called > mercurial and gives me the magic incantation to install it. > > 9) hg installs and I get to clone woody from bitbucket. Step 1 complete. > > 10) Next command is ./app.sh cowgirl . I think that create the app (or > at least its framework). It reports no errors. > > 11) Then ./build-compile.sh - OK, build and compile. NO errors. Great. > > 12) Next comes ./build-release.sh > > This fails saying files exist and were not generated. It adds that I need > force=1 on the command line to force it. Why? If forcing is a good idea, > the software should just do it. Heck, its only to learn. I can delete and > start over if necessary - just press on. Add force=1 to the command line. > > 13) When I do I get this. > > ian@REDACTED:~/projects/cowgirl$ ./build-release.sh force=1 > ==> rel (create-node) > ERROR: One or more files already exist on disk and were not generated: > * "reltool.config" > * "files/erl" > * "files/nodetool" > * "files/cowgirl" > * "files/app.config" > * "files/vm.args" > To force overwriting, specify force=1 on the command line. > ==> rel (generate) > ERROR: generate failed while processing /home/ian/projects/cowgirl/**rel: > {'EXIT',{{badmatch,{error,"**Release \"cowgirl\" uses non existing > application cowgirl"}}, > [{rebar_reltool,generate,2,[]}**, > {rebar_core,run_modules,4,[]}, > {rebar_core,execute,4,[]}, > {rebar_core,process_dir,4,[]}, > {rebar_core,process_each,5,[]}**, > {rebar_core,process_dir,4,[]}, > {rebar_core,process_commands,**1,[]}, > {rebar,main,1,[]}]}} > ian@REDACTED:~/projects/cowgirl$ > > This is copy/pasted from the log, so I can see I did indeed spell force > correctly. > > Where now? Is the pre-existence of those files a problem? I have no idea, > and no idea how to find out. However experience tells me to make a real > effort to fix the first error, even if later ones can be ignored. Should I > delete them all and try again? > > What does the error in the generate mean? It can't mean the application > cowgirl does not exist because hg/mercurial brought it down without errors > and app.sh created it without errors. > > The talk is right. Erlang is just too damn difficult to do the easy stuff. > > It may have been humorously presented (I wasn't there), but the take-away > it so true. > > Early difficulties are putting off many would-be adopters. > > (And I'm stuck! Help appreciated! ) > > Regards > > Ian > > > ______________________________**_________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/**listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cgsmcmlxxv@REDACTED Thu Jul 12 14:46:34 2012 From: cgsmcmlxxv@REDACTED (CGS) Date: Thu, 12 Jul 2012 14:46:34 +0200 Subject: [erlang-questions] lists, binaries or something else? Message-ID: Hi, I am trying to find a balance in between processing speed and RAM consumption for sets of large strings (over 1 M characters per string). To construct such lists is much faster than constructing its binary counterpart. On the other hand, lists are using more RAM than binaries, and that reduces the number of strings I can hold in memory (unless I transform the lists in binaries and call GC after that, but that slows down the processing time). Has anyone had this problem before? What was the solution? Thoughts? A middle way in between lists and binaries is using tuples, but handling them is not as easy as in the case of lists or binaries, especially at variable tuple size. Therefore, working with tuples seems not a good solution. But I might be wrong, so, if anyone used tuples in an efficient way for this case, please, let me know. Any thought would be very much appreciated. Thank you. CGS -------------- next part -------------- An HTML attachment was scrubbed... URL: From eriksoe@REDACTED Thu Jul 12 14:58:31 2012 From: eriksoe@REDACTED (=?ISO-8859-1?Q?Erik_S=F8e_S=F8rensen?=) Date: Thu, 12 Jul 2012 14:58:31 +0200 Subject: [erlang-questions] Lockfree now() Message-ID: When I implemented now() in Erjang, I found a lock-free algorithm for it which has the right "timestamp must be monotonically increasing" property. I did wonder at the time whether it would make sense to suggest using it in BEAM as well. I didn't (although I was proud of the simple solution I'd discovered); mostly because the corresponding BEAM source, with all its smoothing algorithms and support for different platforms, was perhaps a bit intimidating. Now, triggered by the mention on this list that the mutex used in now() is actually being a bottleneck (and having spent the past couple of months ruthlessly refactoring other people's organically grown code), I dare suggest it :-) The algorithm could be used as-is to at least reduce the extent of the timeofday section in get_now(), if used for adjusting for that property after a timestamp has been obtained. (This is assuming that we have true atomic operations of at least 64-bit width; this assumption may of course fail on some platforms. Among the supported (or interesting) platforms, do any have SMP but not such atomic operations?) It would be even better to eliminate the locking in (the common code path of) get_now() altogether, but there are quite a few variants for obtaining the timestamp (3 implementations of get_tolerant_timeofday()). The timer reading and smoothing code seems to have complexity of the "don't change unless you're really sure what you're doing" kind; bearing that in mind, here's what I could imagine for locking-reducing improvements: 0) The code as it is now has two parts: a "read timer, adjust timer value according to state, and adjust state" part, and a "ensure monotonicity" part. 1) Split the former into a lock-less "read+adjust" part, and a locked "adjust state" part which is done seldomly. This can be done if the state for adjusting can be read atomically. 2) The main algorithm for now() is then: // Warning - pseudo-code ahead - I hope it's not too detailed for the purpose { read = adjusted_timer_value(); if (time_to_adjust_timer_state(read)) { lock() adjust_timer_state(read); unlock() time = adjusted_timer_value() // Re-read } else time = read; return ensure_monotonically_increasing_now(time) } 3) Here, time_to_adjust_timer_state() is also lock-free, taking advantage of the fact that it doesn't matter if two threads get a "yes" answer at the same time; if nothing else, the condition can be re-tested in adjust_timer_state(). static erts_smp_atomic_t adjust_time; bool time_to_adjust_timer_state(long read_time) { return erts_smp_atomic_read(adjust_time) - read_time > ADJUST_INTERVAL; } adjust_time is then updated at the end of adjust_timer_state(). 4) Finally, the ensure_monotonically_increasing_now() simply implements the algorithm I discovered for Erjang; here it is in the current Java expression: { // parameter is named 'micros' long prev; while ((prev = latest_now.get()) < micros) { if (latest_now.compareAndSet(prev,micros)) { return micros; } } return latest_now.incrementAndGet(); } So, there it is: ensure that the state needed for the timer correction is readable in a lock-free manner; that it is updated within mutex protection but seldomly; and that timer reading and adjustment itself does not change the global state. Then the common path can use just atomic ops, not locks. The Erjang implementation of now() is quite a bit faster than that of BEAM. It doesn't have the same smoothing logic, of course, but I like to believe that the lack of locking is the major reason. Would the proposed change be worth it, do you think? 2012/7/12 Wei Cao [snip] Finally, I use lcnt to locate all lock collisions in erlang VM, found > the mutex timeofday is the bottleneck. > > lock > location #tries #collisions collisions [%] time > [us] duration [%] > > ----- --------- ------- ------------ > --------------- ---------- ------------- > > timeofday 'beam/erl_time_sup.c':939 895234 551957 > 61.6551 3185159 23.5296 > > timeofday 'beam/erl_time_sup.c':971 408006 264498 > 64.8270 1473816 10.8874 > > > the mutex timeofday is locked each time erts_check_io is invoked to > "sync the machine's idea of time", erts_check_io is executed hundreds > of thounds of times per second, so it's locked too much times, hence > reduce performance. > > I solved this problem by moving the sync operation into a standalone > thread, invoked 1 time per millisecond > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eriksoe@REDACTED Thu Jul 12 15:26:31 2012 From: eriksoe@REDACTED (=?ISO-8859-1?Q?Erik_S=F8e_S=F8rensen?=) Date: Thu, 12 Jul 2012 15:26:31 +0200 Subject: [erlang-questions] lists, binaries or something else? In-Reply-To: References: Message-ID: As so often: "it depends" :-) What are the lifetimes of the strings? If the many large strings have to be alive simultaneously, then that points towards binaries, because of the memory usage. If the strings have rather short lifespans, on the other hand, then their sizes may not matter so much as other things. How are the strings constructed? When you say that they're much faster to construct as lists, does that include constructing the list, then transforming it into a binary? Invoking the GC explicitly should only be necessary in scenarios like if the many strings are constructed and live in separate processes, and these processes afterwards do so little work that it takes a long time until the next GC - that is, scenarios where GC doesn't occur for natural reasons within a reasonable time. Do the strings share substrings? If the strings are built from the same templates, say, or are the results of generating all permutations of lines of the Divine Comedy - then using an iolist would be a good idea. Are the strings built from parts that originally are (or could be) binaries? Another advantage of using iolists is that you can mix lists and binaries; this is good news either if some of the source parts can just as easily be obtained as binaries in the first place, or if you don't want to build it all as a list because of space, nor build it all as a tbinary because of time: you can perhaps build pieces at a time as lists, then convert these pieces to binaries (e.g., build a line at a time), and construct an iolist containing the binary versions of the pieces. Using tuples instead of lists, as a twice-as-compact representation, is of course possible if you're careful - bearing in mind the 64M tuple size limit - though the only efficient way to construct such a thing would be list_to_tuple(). It'd be awkward, though, as well, as unconventional, and you'd have to think the use cases through first. It wouldn't make sense to use tuples for much else than constant strings; the only advantage over binaries might be that you'd avoid encoding or decoding to/from UTF-8 or whatever you'll be using. Hoping this helps. /Erik 2012/7/12 CGS > Hi, > > I am trying to find a balance in between processing speed and RAM > consumption for sets of large strings (over 1 M characters per string). To > construct such lists is much faster than constructing its binary > counterpart. On the other hand, lists are using more RAM than binaries, and > that reduces the number of strings I can hold in memory (unless I transform > the lists in binaries and call GC after that, but that slows down the > processing time). Has anyone had this problem before? What was the > solution? Thoughts? > > A middle way in between lists and binaries is using tuples, but handling > them is not as easy as in the case of lists or binaries, especially at > variable tuple size. Therefore, working with tuples seems not a good > solution. But I might be wrong, so, if anyone used tuples in an efficient > way for this case, please, let me know. > > Any thought would be very much appreciated. Thank you. > > CGS > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From royen@REDACTED Thu Jul 12 17:09:16 2012 From: royen@REDACTED (Dick Oyen) Date: Thu, 12 Jul 2012 11:09:16 -0400 Subject: [erlang-questions] SNMP Notify Target Message-ID: <1E3838671188454484A9D5F6A0C0C9A906982BBF03@EXCH07.vht.virtualhold.com> I am trying to get an SNMP example to send an SNPM trap, but am not succeeding. Any example would do. I am using the tutorial in http://www.trapexit.org/SNMP_Quick_Start because it is easiest for me to understand. The tutorial says to enter: snmpa:send_notification(snmp_master_agent, testMIBMyFirstCoolTrap, no_receiver, "DrpManager", "", []). It returns 'ok', but there is no evidence in Wireshark, the agent's Erlang SNMP log, or the SNMP manager that the trap was actually sent. Using the Erlang debugger, I step through 'snmpa' code and find that the list of notify targets is empty. Please tell me how snmpa should know the notify targets. Here is what I have tried: I tried changing the NotifyName argument from "DrpManager" to "", because "DrpManager" does not appear in anywhere in the conf or mib files and the doc says that empty string should cause it to send to all targets, but my debugger stepping shows that the list is still empty. I tried changing NotifyName to "MyFirstCoolTrap" because notify.conf contains {"MyFirstCoolTrap", "tag1", trap}, but the list is still empty. I wonder if the agent is supposed to know about the manager (before the send_notification/6) because of the manager's call to snmpm:sync_get("simple_user", "otp agent", [[erlNodeName,1]]), which does appear in the agent's SNMP log. -------------- next part -------------- An HTML attachment was scrubbed... URL: From cgsmcmlxxv@REDACTED Thu Jul 12 17:17:03 2012 From: cgsmcmlxxv@REDACTED (CGS) Date: Thu, 12 Jul 2012 17:17:03 +0200 Subject: [erlang-questions] lists, binaries or something else? In-Reply-To: References: Message-ID: On Thu, Jul 12, 2012 at 3:26 PM, Erik S?e S?rensen wrote: > As so often: "it depends" :-) > Yes, I know. :) > > What are the lifetimes of the strings? > If the many large strings have to be alive simultaneously, then that > points towards binaries, because of the memory usage. > If the strings have rather short lifespans, on the other hand, then their > sizes may not matter so much as other things. > Pretty much I don't know. :) There is a dynamic behind those strings by changing them, adding new ones, deleting old ones (that doesn't depend on me). The problem is I may need to "squeeze" simultaneously in memory quite some of them and, depending on the structure I can compute the limitations. > > How are the strings constructed? > When you say that they're much faster to construct as lists, does that > include constructing the list, then transforming it into a binary? > No, I was referring to constructing directly into binaries versus constructing lists and transforming them in binaries. Much more like recursion in strings as binaries versus as lists at string expanding point. On my machine, creating a list of 1 Mchars and transforming it to binary using list_to_binary/1 is much faster than building up directly binaries. And I fear that is a general case and not only on my machine. But, correct me if I am wrong. > Invoking the GC explicitly should only be necessary in scenarios like if > the many strings are constructed and live in separate processes, and these > processes afterwards do so little work that it takes a long time until the > next GC - that is, scenarios where GC doesn't occur for natural reasons > within a reasonable time. > I was referring at constructing a list, transforming it in binary and invoking GC to get rid of the list from the memory. I know GC would occur for natural reasons when the memory becomes insufficient, but then I have no control over how slow the application will be in those moments. Therefore, I would go for calling GC after each transformation. But that is quite inefficiently to do it, I know. > > Do the strings share substrings? > If the strings are built from the same templates, say, or are the results > of generating all permutations of lines of the Divine Comedy - then using > an iolist would be a good idea. > Well, I haven't thought of iolist, it is true. I will have a closer look at iolist. Thank you for suggesting it. > > Are the strings built from parts that originally are (or could be) > binaries? > Another advantage of using iolists is that you can mix lists and binaries; > this is good news either if some of the source parts can just as easily be > obtained as binaries in the first place, or if you don't want to build it > all as a list because of space, nor build it all as a tbinary because of > time: you can perhaps build pieces at a time as lists, then convert these > pieces to binaries (e.g., build a line at a time), and construct an iolist > containing the binary versions of the pieces. > As a low memory space requirement for the application, the strings should be held in binaries as they are the smallest. The iolist, as you said, may be a nice option. > > Using tuples instead of lists, as a twice-as-compact representation, is of > course possible if you're careful - bearing in mind the 64M tuple size > limit - though the only efficient way to construct such a thing would be > list_to_tuple(). It'd be awkward, though, as well, as unconventional, and > you'd have to think the use cases through first. > It wouldn't make sense to use tuples for much else than constant strings; > the only advantage over binaries might be that you'd avoid encoding or > decoding to/from UTF-8 or whatever you'll be using. > About tuples, I had the same feeling. That's why I said they don't seem to be a good solution. Nevertheless, I needed a confirmation that my way of thinking was correct. > Hoping this helps. > Thanks a lot for your thoughts. Of course they help and gave me some new ideas. Feel free to add more thoughts if you have. They will be much appreciated. Regards, CGS > /Erik > > 2012/7/12 CGS > >> Hi, >> >> I am trying to find a balance in between processing speed and RAM >> consumption for sets of large strings (over 1 M characters per string). To >> construct such lists is much faster than constructing its binary >> counterpart. On the other hand, lists are using more RAM than binaries, and >> that reduces the number of strings I can hold in memory (unless I transform >> the lists in binaries and call GC after that, but that slows down the >> processing time). Has anyone had this problem before? What was the >> solution? Thoughts? >> >> A middle way in between lists and binaries is using tuples, but handling >> them is not as easy as in the case of lists or binaries, especially at >> variable tuple size. Therefore, working with tuples seems not a good >> solution. But I might be wrong, so, if anyone used tuples in an efficient >> way for this case, please, let me know. >> >> Any thought would be very much appreciated. Thank you. >> >> CGS >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hobson42@REDACTED Thu Jul 12 17:40:51 2012 From: hobson42@REDACTED (Ian) Date: Thu, 12 Jul 2012 16:40:51 +0100 Subject: [erlang-questions] Looking for slides of a lightning talk In-Reply-To: References: <4FFAAAEA.9060605@ninenines.eu> <0d6c6c5d-2b0f-48da-bae8-f440d232df92@googlegroups.com> <489B3810-49B2-4085-B468-DA037F7DE646@dmitriid.com> <1709242D-0AD5-4070-A6EC-AB039E62ADFD@dmitriid.com> <1342003659.84614.YahooMailNeo@web111404.mail.gq1.yahoo.com> <4FFDFE93.8040202@gmail.com> Message-ID: <4FFEF003.70605@gmail.com> Hi CGS, You are hearing things in my email that were not there when I wrote it. I was reporting my experience, not blaming Erlang for anything. My experience is that the learning curve for the Erlang tools (coupled unfortunately with learning the Linux environment) was extremely steep - and ended up with me getting stuck because the tools failed to do what they claimed (in multiple ways) before I had the skills or knowledge to fix them. Erlang the language is not the problem. The language is never the problem. Once the tool chain is working...... But consider this. Rebar can't be installed in a standard Ubuntu install of Erlang!!! WTF? Would you agree that that means at least one of those installs is inadequate for my needs? Its inadequate for anyone who wishes to use rebar and Erlang on Ubuntu/Debian. Would you agree that giving newbies tools that are inadequate for their needs, does not help them scale their learning curve? This is not meant as a criticism of the work people have done. Its a very hard problem, and I don't think there is an easy solution. PHP isn't there yet. PEAR is not widely used. The comments added to the manual are very helpful. PHP the language is astonishingly inconsistent. Python boasts "batteries included". Whatever you want to do in Python, there are usually 6 modules that claim to do it - all poorly documented, some buggy. So do you spend your time investigating them, or writing a 7th that you know will do what you want? Experienced people have forgotten the misunderstandings they had as newbies - or they had an experienced person to guide them - so they are genuinely puzzled by the problems newbies face. The newbies can't help: they don't know what they don't know. Few technical people can write well, and all are busy with their projects. The result is a distinct lack of excellent, current documentation of the tools that can help newbies start to use Erlang. It comes, slowly, with maturity and stability. Erlang the language is well documented. Its the docs for the supporting cast that is spotty. Which is a shame. On 12/07/2012 12:49, CGS wrote: > Hi, > > You blame Erlang for an application written in it which doesn't > support MS Windows... Well, that reminds me of a case of saying noSQL > databases are not good just because that person/team misunderstood the > usage of a particular noSQL database. Interesting how people try to > generalize their impressions to a category from a wrongly chosen > example. Don't get me wrong, I am not that good in Erlang to mock you, > but I think you should ask someone more knowledgeable before you to > say something like "Erlang is just too damn difficult to do the easy > stuff." I do easy stuff in Erlang. It would probably be more difficult > to write a proper C code to do those simple things (like proper > threading, distributed applications and so on). > > Now, the problem is what do you need cowboy for? If you need an > webserver for MS Windows which is written in Erlang, you can try Yaws > (you have installers here: http://yaws.hyber.org/download/). If you > need a Erlang-based webserver which you can integrate it in your > application, you will need either a Linux or Cygwin (for most of the > open source projects, but you can do it without if you use, for > example, lhttpc - https://bitbucket.org/etc/lhttpc/src - where you can > modify the Makefile to be compatible with MS VS or, for even less, a > .bat file). I would avoid installing Linux in a MS Windows box (I'd > rather do it oppositely). Agreed. Ubuntu host with LVM and software RAID, VBox with a windows guest. Plans are laid :) Your other points are all good stuff. Thanks. I am looking for a fast web server that had a permanent core(1) image that I could program, that was not under any run-away timer (Apache) and that can handle many simultaneous connections (i.e event driven architecture). One of the projects is a special type of chat room where silence might exist for long periods, and I don't want a fast heart-beat or unnecessary writes to disk. Note 1 - Core is RAM to anyone under 50. ;) > > Erlang under MS Windows works as nice as under Linux (as far as I > could play with it under MS Windows), but the installation under MS > Windows is much simpler (I would suggest for Linux to compile it from > source with the options you need rather than installing it from > repositories). I don't know what is your background in programming > (and I don't want to know), but there are nice tutorials on web for > all the categories of programmers (coming from OOP, functional or > whatever else languages). Just get familiarized with Erlang and after > that make an impression of your own about the language itself. I am > also relatively new in Erlang and I consider I cannot afford to make > any statement about how good or not is Erlang. I know only that are > some nice things and some not so nice things in Erlang (I leave > finding examples to your experience). Indeed. Erlang under windows is fine. The target server is Ubuntu, so developing on that platform has advantages. Particularly in view of the planned migration to Ubuntu as my main machine. Regards Ian > > CGS > From cgsmcmlxxv@REDACTED Thu Jul 12 18:06:18 2012 From: cgsmcmlxxv@REDACTED (CGS) Date: Thu, 12 Jul 2012 18:06:18 +0200 Subject: [erlang-questions] lists, binaries or something else? In-Reply-To: References: Message-ID: Hi Joe, The main problem is to find out which strings are read-only and which strings are read-write, and that requires an algorithm for itself (processing time and extra space - I don't know how negligible are at this moment) as I don't know from before which string will be used more frequently and which less frequently. The second problem is I would like to minimize the harddisk usage, so, to try to store as much information as possible in RAM, but without slowing down the overall process. I know, I am an idealist. :) I thought also about working with lists and keep them as binaries when I don't use them, but, as I said before, that implies a lot of garbage to collect which either can be collected immediately after invoking list_to_binary/1, either allowing GC to appear naturally when there is insufficient memory, or to invoke it at certain moments (either at regular interval of time or based on a scheduler triggered by the application usage). I am afraid that all may be quite inefficient, but they may work faster than processing binaries directly. That I have no idea yet. That's why I am asking here for opinions. Nevertheless, I didn't think of trying to split the strings in two categories: read-only and read-write. That definitely is something I should take into account. Thanks a lot for your thoughts and shared experience. Cheers, CGS On Thu, Jul 12, 2012 at 5:17 PM, Joe Armstrong wrote: > As you point out list processing is faster than binary processing. > > I'd keep things as lists as long as possible until you run into memory > problems > If you plot the number of strings against response times (or whatever) > you should see a sudden decrease in performance when you start paging. > At that point you have too much in memory - You could turn the oldest > strings > into binaries to save space. > > I generally keep string as lists when I'm working on them and turn > them into binaries > when I'm finished - sometimes even compressed binaries. > > Then it depends on the access patterns on the strings - random > read-write access is horrible > if you can split them into a read-only part and a write-part, you > could keep the read-only bit > as a binary and the writable bit as a list. > > It's worth spending a lot of effort to save a single disk access. Then > it depends what you do with your strings. If you have a solid state > disk and want read only access to the strings > then you could store them on disk - or at least arrange so that the > constant parts of the strings > are on disk and the variable parts in memory. SSDs are about 10 times > slower than RAM for reading and usually have multiple controllers so > can be very fast - but you need to think a bit first. > > I'd start with a few measurements, try to stress the system and see > where things go wrong. > Plot the results - it's usually easy to see when things go wrong. > > Cheers > > /Joe > > > > On Thu, Jul 12, 2012 at 2:46 PM, CGS wrote: > > Hi, > > > > I am trying to find a balance in between processing speed and RAM > > consumption for sets of large strings (over 1 M characters per string). > To > > construct such lists is much faster than constructing its binary > > counterpart. On the other hand, lists are using more RAM than binaries, > and > > that reduces the number of strings I can hold in memory (unless I > transform > > the lists in binaries and call GC after that, but that slows down the > > processing time). Has anyone had this problem before? What was the > solution? > > Thoughts? > > > > A middle way in between lists and binaries is using tuples, but handling > > them is not as easy as in the case of lists or binaries, especially at > > variable tuple size. Therefore, working with tuples seems not a good > > solution. But I might be wrong, so, if anyone used tuples in an efficient > > way for this case, please, let me know. > > > > Any thought would be very much appreciated. Thank you. > > > > CGS > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wojtek@REDACTED Thu Jul 12 18:17:10 2012 From: wojtek@REDACTED (=?UTF-8?B?V29qdGVrIE5hcmN6ecWEc2tp?=) Date: Thu, 12 Jul 2012 18:17:10 +0200 Subject: [erlang-questions] Improve performance of IO bounded server written in Erlang via having pollset for each scheduler and bind port to scheduler together with process In-Reply-To: References: <771ADB6B-9CD0-4A7A-A8A1-EC551B84BAD1@gmail.com> <0E26E896-FF0D-4036-A51A-8F3B8C1FA31C@gmail.com> Message-ID: <4FFEF886.3060900@power.com.pl> Hello, > I solved this problem by moving the sync operation into a standalone > thread, invoked 1 time per millisecond What about periods, when the machine is idle? Is this thread going to make 1000 syscalls per second anyway? --Regards From wojtek@REDACTED Thu Jul 12 19:08:28 2012 From: wojtek@REDACTED (=?UTF-8?B?V29qdGVrIE5hcmN6ecWEc2tp?=) Date: Thu, 12 Jul 2012 19:08:28 +0200 Subject: [erlang-questions] Lockfree now() In-Reply-To: References: Message-ID: <4FFF048C.5000500@power.com.pl> On 07/12/2012 02:58 PM, Erik S?e S?rensen wrote: > When I implemented now() in Erjang, I found a lock-free algorithm for > it which has the right "timestamp must be monotonically increasing" > property. http://www.erlang.org/doc/man/erlang.html#now-0 " It is also guaranteed that subsequent calls to this BIF returns continuously increasing values." I don't think your solution quarrantees this. It can be implemented with N-bit CAS for N-bit time value. --Regards, Wojtek Narczynski -------------- next part -------------- An HTML attachment was scrubbed... URL: From eriksoe@REDACTED Thu Jul 12 19:23:03 2012 From: eriksoe@REDACTED (=?ISO-8859-1?Q?Erik_S=F8e_S=F8rensen?=) Date: Thu, 12 Jul 2012 19:23:03 +0200 Subject: [erlang-questions] lists, binaries or something else? In-Reply-To: References: Message-ID: Regarding GC, it's still a good question whether you'll have one process holding all the strings (in which case GC will probably occur soon enough), or you've got a lot of processes each holding one or a few strings. In the latter case, you may want to look into gen_server's 'hibernate' feature (which builds on proc_lib:hibernate(), in case your processes aren't gen_servers). gen_fsm also supports 'hibernate'. You can use this for reducing heap usage right after the strings have been constructed. Depending on the nature of the strings in question, and the length of time the strings may be dormant, it might also be relevant to consider compressing the binaries, as Joe suggested. Erlang has nice APIs for this (see zlib). Much depends on which modification operations you need to perform, of course - if any. And on the access patterns in general, and the nature of the strings (similar or not; repetetive or not; ascii/unicode/dna base pairs...). 2012/7/12 CGS > > Hi Joe, > > The main problem is to find out which strings are read-only and which > strings are read-write, and that requires an algorithm for itself > (processing time and extra space - I don't know how negligible are at this > moment) as I don't know from before which string will be used more > frequently and which less frequently. The second problem is I would like to > minimize the harddisk usage, so, to try to store as much information as > possible in RAM, but without slowing down the overall process. I know, I am > an idealist. :) > > I thought also about working with lists and keep them as binaries when I > don't use them, but, as I said before, that implies a lot of garbage to > collect which either can be collected immediately after invoking > list_to_binary/1, either allowing GC to appear naturally when there is > insufficient memory, or to invoke it at certain moments (either at regular > interval of time or based on a scheduler triggered by the application > usage). I am afraid that all may be quite inefficient, but they may work > faster than processing binaries directly. That I have no idea yet. That's > why I am asking here for opinions. > > Nevertheless, I didn't think of trying to split the strings in two > categories: read-only and read-write. That definitely is something I should > take into account. > > Thanks a lot for your thoughts and shared experience. > > Cheers, > CGS > > > > > On Thu, Jul 12, 2012 at 5:17 PM, Joe Armstrong wrote: > >> As you point out list processing is faster than binary processing. >> >> I'd keep things as lists as long as possible until you run into memory >> problems >> If you plot the number of strings against response times (or whatever) >> you should see a sudden decrease in performance when you start paging. >> At that point you have too much in memory - You could turn the oldest >> strings >> into binaries to save space. >> >> I generally keep string as lists when I'm working on them and turn >> them into binaries >> when I'm finished - sometimes even compressed binaries. >> >> Then it depends on the access patterns on the strings - random >> read-write access is horrible >> if you can split them into a read-only part and a write-part, you >> could keep the read-only bit >> as a binary and the writable bit as a list. >> >> It's worth spending a lot of effort to save a single disk access. Then >> it depends what you do with your strings. If you have a solid state >> disk and want read only access to the strings >> then you could store them on disk - or at least arrange so that the >> constant parts of the strings >> are on disk and the variable parts in memory. SSDs are about 10 times >> slower than RAM for reading and usually have multiple controllers so >> can be very fast - but you need to think a bit first. >> >> I'd start with a few measurements, try to stress the system and see >> where things go wrong. >> Plot the results - it's usually easy to see when things go wrong. >> >> Cheers >> >> /Joe >> >> >> >> On Thu, Jul 12, 2012 at 2:46 PM, CGS wrote: >> > Hi, >> > >> > I am trying to find a balance in between processing speed and RAM >> > consumption for sets of large strings (over 1 M characters per string). >> To >> > construct such lists is much faster than constructing its binary >> > counterpart. On the other hand, lists are using more RAM than binaries, >> and >> > that reduces the number of strings I can hold in memory (unless I >> transform >> > the lists in binaries and call GC after that, but that slows down the >> > processing time). Has anyone had this problem before? What was the >> solution? >> > Thoughts? >> > >> > A middle way in between lists and binaries is using tuples, but handling >> > them is not as easy as in the case of lists or binaries, especially at >> > variable tuple size. Therefore, working with tuples seems not a good >> > solution. But I might be wrong, so, if anyone used tuples in an >> efficient >> > way for this case, please, let me know. >> > >> > Any thought would be very much appreciated. Thank you. >> > >> > CGS >> > >> > >> > _______________________________________________ >> > erlang-questions mailing list >> > erlang-questions@REDACTED >> > http://erlang.org/mailman/listinfo/erlang-questions >> > >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eriksoe@REDACTED Thu Jul 12 19:27:33 2012 From: eriksoe@REDACTED (=?ISO-8859-1?Q?Erik_S=F8e_S=F8rensen?=) Date: Thu, 12 Jul 2012 19:27:33 +0200 Subject: [erlang-questions] Lockfree now() In-Reply-To: <4FFF048C.5000500@power.com.pl> References: <4FFF048C.5000500@power.com.pl> Message-ID: 2012/7/12 Wojtek Narczy?ski > On 07/12/2012 02:58 PM, Erik S?e S?rensen wrote: > > When I implemented now() in Erjang, I found a lock-free algorithm for it > which has the right "timestamp must be monotonically increasing" property. > > > http://www.erlang.org/doc/man/erlang.html#now-0 > " It is also guaranteed that subsequent calls to this BIF returns > continuously increasing values." > > I don't think your solution quarrantees this. > Why not? That's certainly the intention of the last part; latest_now should always be updated in a increasing manner, and the return value of ensure_monotonically_increasing_now() should always be equal to the new value of latest_now. Did I make a mistake? > > It can be implemented with N-bit CAS for N-bit time value. > > --Regards, > Wojtek Narczynski > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bob@REDACTED Thu Jul 12 19:29:48 2012 From: bob@REDACTED (Bob Ippolito) Date: Thu, 12 Jul 2012 10:29:48 -0700 Subject: [erlang-questions] lists, binaries or something else? In-Reply-To: References: Message-ID: It would be helpful if you were a lot more specific about how these strings are constructed and what the strings mean. Maybe if you shared some of the code, you'd get better guidance. On Thu, Jul 12, 2012 at 9:06 AM, CGS wrote: > > Hi Joe, > > The main problem is to find out which strings are read-only and which > strings are read-write, and that requires an algorithm for itself > (processing time and extra space - I don't know how negligible are at this > moment) as I don't know from before which string will be used more > frequently and which less frequently. The second problem is I would like to > minimize the harddisk usage, so, to try to store as much information as > possible in RAM, but without slowing down the overall process. I know, I am > an idealist. :) > > I thought also about working with lists and keep them as binaries when I > don't use them, but, as I said before, that implies a lot of garbage to > collect which either can be collected immediately after invoking > list_to_binary/1, either allowing GC to appear naturally when there is > insufficient memory, or to invoke it at certain moments (either at regular > interval of time or based on a scheduler triggered by the application > usage). I am afraid that all may be quite inefficient, but they may work > faster than processing binaries directly. That I have no idea yet. That's > why I am asking here for opinions. > > Nevertheless, I didn't think of trying to split the strings in two > categories: read-only and read-write. That definitely is something I should > take into account. > > Thanks a lot for your thoughts and shared experience. > > Cheers, > CGS > > > > > On Thu, Jul 12, 2012 at 5:17 PM, Joe Armstrong wrote: > >> As you point out list processing is faster than binary processing. >> >> I'd keep things as lists as long as possible until you run into memory >> problems >> If you plot the number of strings against response times (or whatever) >> you should see a sudden decrease in performance when you start paging. >> At that point you have too much in memory - You could turn the oldest >> strings >> into binaries to save space. >> >> I generally keep string as lists when I'm working on them and turn >> them into binaries >> when I'm finished - sometimes even compressed binaries. >> >> Then it depends on the access patterns on the strings - random >> read-write access is horrible >> if you can split them into a read-only part and a write-part, you >> could keep the read-only bit >> as a binary and the writable bit as a list. >> >> It's worth spending a lot of effort to save a single disk access. Then >> it depends what you do with your strings. If you have a solid state >> disk and want read only access to the strings >> then you could store them on disk - or at least arrange so that the >> constant parts of the strings >> are on disk and the variable parts in memory. SSDs are about 10 times >> slower than RAM for reading and usually have multiple controllers so >> can be very fast - but you need to think a bit first. >> >> I'd start with a few measurements, try to stress the system and see >> where things go wrong. >> Plot the results - it's usually easy to see when things go wrong. >> >> Cheers >> >> /Joe >> >> >> >> On Thu, Jul 12, 2012 at 2:46 PM, CGS wrote: >> > Hi, >> > >> > I am trying to find a balance in between processing speed and RAM >> > consumption for sets of large strings (over 1 M characters per string). >> To >> > construct such lists is much faster than constructing its binary >> > counterpart. On the other hand, lists are using more RAM than binaries, >> and >> > that reduces the number of strings I can hold in memory (unless I >> transform >> > the lists in binaries and call GC after that, but that slows down the >> > processing time). Has anyone had this problem before? What was the >> solution? >> > Thoughts? >> > >> > A middle way in between lists and binaries is using tuples, but handling >> > them is not as easy as in the case of lists or binaries, especially at >> > variable tuple size. Therefore, working with tuples seems not a good >> > solution. But I might be wrong, so, if anyone used tuples in an >> efficient >> > way for this case, please, let me know. >> > >> > Any thought would be very much appreciated. Thank you. >> > >> > CGS >> > >> > >> > _______________________________________________ >> > erlang-questions mailing list >> > erlang-questions@REDACTED >> > http://erlang.org/mailman/listinfo/erlang-questions >> > >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mononcqc@REDACTED Thu Jul 12 19:44:31 2012 From: mononcqc@REDACTED (Fred Hebert) Date: Thu, 12 Jul 2012 13:44:31 -0400 Subject: [erlang-questions] Looking for slides of a lightning talk In-Reply-To: <4FFEF003.70605@gmail.com> References: <4FFAAAEA.9060605@ninenines.eu> <0d6c6c5d-2b0f-48da-bae8-f440d232df92@googlegroups.com> <489B3810-49B2-4085-B468-DA037F7DE646@dmitriid.com> <1709242D-0AD5-4070-A6EC-AB039E62ADFD@dmitriid.com> <1342003659.84614.YahooMailNeo@web111404.mail.gq1.yahoo.com> <4FFDFE93.8040202@gmail.com> <4FFEF003.70605@gmail.com> Message-ID: <4FFF0CFF.5000009@ferd.ca> On 12-07-12 11:40 AM, Ian wrote: > Hi CGS, > > You are hearing things in my email that were not there when I wrote it. > > I was reporting my experience, not blaming Erlang for anything. My > experience is that the learning curve for the Erlang tools (coupled > unfortunately with learning the Linux environment) was extremely steep > - and ended up with me getting stuck because the tools failed to do > what they claimed (in multiple ways) before I had the skills or > knowledge to fix them. > > Erlang the language is not the problem. The language is never the > problem. Once the tool chain is working...... > > But consider this. Rebar can't be installed in a standard Ubuntu > install of Erlang!!! WTF? Projects tend to include their own copy of rebar within their own repos. There are a few that don't do it and do assume a globally installed rebar, but I think they're not the majority of repos. Then people either create makefiles or have a README.md that specifies how to build the project. As far as I know, the following ways to build projects are used: - Rebar in the repo - Rebar, where rebar isn't in the repo - Emakefiles - Sinan (usually not in the repo) - Makefiles Moreover, there is a strong culture of getting stuff from source and compiling it ourselves. There is no big central library of precompiled code, and frankly, I'm not sure who would want to maintain one that people would use. There's always the issue of "nobody uses it so nobody is going to use it" to go over. Rebar made itself a good spot because it didn't require anyone to have it installed. It had a little self-executable (as long as Erlang is installed, see escript), and so you needed no one's support to get it going. > > Would you agree that that means at least one of those installs is > inadequate for my needs? Its inadequate for anyone who wishes to use > rebar and Erlang on Ubuntu/Debian. I don't exactly remembers who maintains the Ubuntu repositories, but historically, this has been a problem since they wanted to include CouchDB as part of the default offering /while/ keeping LiveCDs available. An Erlang release was just too big, so they broke it off in different sections that people usually assume are there. Regarding Debian, I haven't looked at it much since they took R14A (or was it R13A?) and declared it stable; that version is a beta version, and because of this (I believe, either this or github), the OTP team at Ericsson stopped distributing the source for A versions on erlang.org. It's been a problem of maintainers; Distros tend to do things one way, and Erlang lives within its own world that sometimes clashes with it. From what I know, package maintainers for distros tended to be more familiar with distros than they were with Erlang, but any maintainer can feel free to correct me. This lead to "compile it yourself and google in case of errors, it's simpler" as a build policy. > > Would you agree that giving newbies tools that are inadequate for > their needs, does not help them scale their learning curve? Agreed. I wrote Learn You Some Erlang and it suprises me that in order to get the structure I wanted, I had to wait more than 5 chapters before showing how to run something from the command line. Hello world as a self-executable is quite something if you don't use escripts. The question I came up with while writing LYSE is "how can we make things easier?" Erlang is complex in itself. People who used it for years tend not to know how the ERL_LIBS variable works (or how it exists) when it's doing something as elementary as finding where libraries are! We have build tools that are somewhat decent enough once their environment is ready. You're new to Erlang, if you can send me a list of the steps you took, I'll keep them in note. (I started working on tend (https://github.com/ferd/tend) with a guy nicknamed orbitz for Spawnfest, and I plan to possibly use it as a tool to download libraries of applications (yes, another one), so help in how to make things easier is always wanted). > > This is not meant as a criticism of the work people have done. Its a > very hard problem, and I don't think there is an easy solution. > > PHP isn't there yet. PEAR is not widely used. The comments added to > the manual are very helpful. PHP the language is astonishingly > inconsistent. > > Python boasts "batteries included". Whatever you want to do in Python, > there are usually 6 modules that claim to do it - all poorly > documented, some buggy. So do you spend your time investigating them, > or writing a 7th that you know will do what you want? > > Experienced people have forgotten the misunderstandings they had as > newbies - or they had an experienced person to guide them - so they > are genuinely puzzled by the problems newbies face. The newbies can't > help: they don't know what they don't know. Few technical people can > write well, and all are busy with their projects. The result is a > distinct lack of excellent, current documentation of the tools that > can help newbies start to use Erlang. As someone who wrote some technical documentation (I am not a technical writer, however), the biggest issue I faced was that Erlang is in a kind of perpetual renewal state when it comes to build tools. Nobody likes them absolutely, everyone has a favorite (and I'm not helping!), and things change so fast it would be very risky to take lots of time to write something that might be invalid in a few weeks or months. > > It comes, slowly, with maturity and stability. Erlang the language is > well documented. Its the docs for the supporting cast that is spotty. > Which is a shame. Erlang/OTP has pretty good docs. There is, however, very little doc when it comes to the code the community produces. I'm not sure how to change that without either having - Someone who goes over a bunch of projects and documents them (how boring!) - Each owner taking responsibility of that (Hard to do!) Earlier this week, Jesper Louis Andersen (jlouis) used one of the libraries a coworker developed at work. He simply took the undocumented code, took a bit of time after figuring it out, and documented how to use it, sent a pull request, and within the hour, it was accepted. I think this is a very good attitude to take. If someone wrote us a library and didn't have the time to document it, basic documentation could be a decent 'pay it back' measure to be given to the authors when we use their code. It's saving us time using the library, let's take a fraction of the time it saved us to pay back with some docs. Projects like rebar would have books written on it at this point if it were the case. I'd do it, but by now I'm getting pretty tired about writing documentation ;) Regards, Fred. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wojtek@REDACTED Thu Jul 12 19:45:09 2012 From: wojtek@REDACTED (=?UTF-8?B?V29qdGVrIE5hcmN6ecWEc2tp?=) Date: Thu, 12 Jul 2012 19:45:09 +0200 Subject: [erlang-questions] Improve performance of IO bounded server written in Erlang via having pollset for each scheduler and bind port to scheduler together with process In-Reply-To: References: <771ADB6B-9CD0-4A7A-A8A1-EC551B84BAD1@gmail.com> <0E26E896-FF0D-4036-A51A-8F3B8C1FA31C@gmail.com> Message-ID: <4FFF0D25.7050601@power.com.pl> On 07/12/2012 11:58 AM, Wei Cao wrote: > the mutex timeofday is locked each time erts_check_io is invoked to > "sync the machine's idea of time", erts_check_io is executed hundreds > of thounds of times per second, so it's locked too much times, hence > reduce performance. > I am trying to figure out how is "the machine's idea of time" relevant to checking for io. I am begining to suspect that the lone call to erts_deliver_time() in erl_check_io.c:1179 might be unnecessary. If do_wait is true, the call to erts_time_remaining(&wait_time) will update the global time. If do_wait is false, which should be the case most of the time on a busy server, as per erts_check_io(!runnable) calls, the timeout to select/poll/epoll whichever is going to be zero anyway. There must be a catch, because this simply looks too good to be true... --Regards, Wojtek Narczynski From wojtek@REDACTED Thu Jul 12 19:52:09 2012 From: wojtek@REDACTED (=?UTF-8?B?V29qdGVrIE5hcmN6ecWEc2tp?=) Date: Thu, 12 Jul 2012 19:52:09 +0200 Subject: [erlang-questions] Lockfree now() In-Reply-To: References: <4FFF048C.5000500@power.com.pl> Message-ID: <4FFF0EC9.5080302@power.com.pl> On 07/12/2012 07:27 PM, Erik S?e S?rensen wrote: > Why not? That's certainly the intention of the last part; latest_now > should always be updated in a increasing manner, and the return value > of ensure_monotonically_increasing_now() should always be equal to the > new value of latest_now. > Did I make a mistake? Well, how do you "ensure" it without taking a lock? Meanwhile the global value can be increased by another thread, to what you calculate locally. --Regards, Wojtek From moxford@REDACTED Thu Jul 12 20:07:53 2012 From: moxford@REDACTED (Mike Oxford) Date: Thu, 12 Jul 2012 11:07:53 -0700 Subject: [erlang-questions] mnesia and majority Message-ID: When facing a split with 'majority' enabled, you end up with a reduction in service based on the non-majority nodes. If the interconnect-network is entirely down, then every node is non-majority and will 'freeze.' If the split is, say, 70/30, then the 30% will not update. But then once the network is back up, does it automatically replay the missing transactions on the 30% from the 70%? In a full-network-down situation where every node halts the write, when the network comes back up will everything re-mesh on its own? Or do I need something like Ulf's "unsplit" handlers? Does mnesia still need to be patched to support "unsplit" or was that merged as well? Thank you! -mox -------------- next part -------------- An HTML attachment was scrubbed... URL: From tsuraan@REDACTED Thu Jul 12 20:08:35 2012 From: tsuraan@REDACTED (tsuraan) Date: Thu, 12 Jul 2012 13:08:35 -0500 Subject: [erlang-questions] Erlang get_memory_data Message-ID: I have a Riak database that frequently gets memsup high watermark warnings and long_gc warnings, but really doesn't seem to be using much RAM. When I run memsup:get_memory_data(), I get this output: {4153327616,3424722944,{<0.7.0>,372152}} Which I think indicates that erlang believes it's using ~3.5GB of RAM (or does it mean something completely different?). If I run "grep ^Vm /proc//status", I get this output: VmPeak: 334560 kB VmSize: 270252 kB VmLck: 0 kB VmHWM: 33384 kB VmRSS: 33084 kB VmData: 225832 kB VmStk: 136 kB VmExe: 2028 kB VmLib: 6792 kB VmPTE: 232 kB VmSwap: 0 kB So, Linux thinks Riak (beam) has never used more than ~350MB of RAM, and that it only has a ~30MB resident set. pmap shows 90MB of writable anonymous pages, and a total of 262MB of mapped data. Where is memsup's 3.5GB of RAM coming from? Or am I way off on my ideas about what memsup is trying to tell me? From tsuraan@REDACTED Thu Jul 12 20:09:13 2012 From: tsuraan@REDACTED (tsuraan) Date: Thu, 12 Jul 2012 13:09:13 -0500 Subject: [erlang-questions] Erlang get_memory_data Message-ID: I have a Riak database that frequently gets memsup high watermark warnings and long_gc warnings, but really doesn't seem to be using much RAM. When I run memsup:get_memory_data(), I get this output: {4153327616,3424722944,{<0.7.0>,372152}} Which I think indicates that erlang believes it's using ~3.5GB of RAM (or does it mean something completely different?). If I run "grep ^Vm /proc//status", I get this output: VmPeak: 334560 kB VmSize: 270252 kB VmLck: 0 kB VmHWM: 33384 kB VmRSS: 33084 kB VmData: 225832 kB VmStk: 136 kB VmExe: 2028 kB VmLib: 6792 kB VmPTE: 232 kB VmSwap: 0 kB So, Linux thinks Riak (beam) has never used more than ~350MB of RAM, and that it only has a ~30MB resident set. pmap shows 90MB of writable anonymous pages, and a total of 262MB of mapped data. Where is memsup's 3.5GB of RAM coming from? Or am I way off on my ideas about what memsup is trying to tell me? From eriksoe@REDACTED Thu Jul 12 20:23:04 2012 From: eriksoe@REDACTED (=?ISO-8859-1?Q?Erik_S=F8e_S=F8rensen?=) Date: Thu, 12 Jul 2012 20:23:04 +0200 Subject: [erlang-questions] Lockfree now() In-Reply-To: <4FFF0EC9.5080302@power.com.pl> References: <4FFF048C.5000500@power.com.pl> <4FFF0EC9.5080302@power.com.pl> Message-ID: Because of atomic operations. I may not have been explicit enough there: latest_now is a java.util.concurrent.AtomicLong, and the two updating accesses to it are atomic. long prev; while ((prev = latest_now.get()) < micros) { if (latest_now.compareAndSet(prev,micros)) { return micros; } } return latest_now.incrementAndGet(); This - and only because of this - means that no lock is necessary. 2012/7/12 Wojtek Narczy?ski > On 07/12/2012 07:27 PM, Erik S?e S?rensen wrote: > >> Why not? That's certainly the intention of the last part; latest_now >> should always be updated in a increasing manner, and the return value of >> ensure_monotonically_**increasing_now() should always be equal to the >> new value of latest_now. >> Did I make a mistake? >> > > Well, how do you "ensure" it without taking a lock? Meanwhile the global > value can be increased by another thread, to what you calculate locally. > > --Regards, > Wojtek > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ulf@REDACTED Thu Jul 12 20:27:18 2012 From: ulf@REDACTED (Ulf Wiger) Date: Thu, 12 Jul 2012 20:27:18 +0200 Subject: [erlang-questions] mnesia and majority In-Reply-To: References: Message-ID: On 12 Jul 2012, at 20:07, Mike Oxford wrote: > When facing a split with 'majority' enabled, you end up with a reduction in service based on the non-majority nodes. > > If the interconnect-network is entirely down, then every node is non-majority and will 'freeze.' > If the split is, say, 70/30, then the 30% will not update. > > But then once the network is back up, does it automatically replay the missing transactions on the 30% from the 70%? In a full-network-down situation where every node halts the write, when the network comes back up will everything re-mesh on its own? Mnesia does no automatic merge. It can detect that a split has happened, and if you have 'majority' enabled, you can trust that you have avoided inconsistency, and can proceed to identify a majority island. Mnesia doesn't give you much help with this, although you can spy into the "unsplit" code for _one_ way to do it. Another could be to have a "backdoor ping", e.g. via UDP, and -kernel dist_auto_connect false. This way, you can exchange information between "islands" and control when they are to reconnect. The standard mnesia-supported way to resolve the situation would then be to restart the nodes in the minority island, after having called mnesia:set_master_nodes(Tables, MajorityNodes). This has always been seen as something of a sledgehammer, but given that the minority nodes are not able to write to the database anyway, it is somewhat less so when combined with the 'majority' option. ;-) > Or do I need something like Ulf's "unsplit" handlers? Does mnesia still need to be patched to support "unsplit" or was that merged as well? That patch was merged in R14B, I believe. BR, Ulf W Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. http://feuerlabs.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From eriksoe@REDACTED Thu Jul 12 20:33:55 2012 From: eriksoe@REDACTED (=?ISO-8859-1?Q?Erik_S=F8e_S=F8rensen?=) Date: Thu, 12 Jul 2012 20:33:55 +0200 Subject: [erlang-questions] Lockfree now() In-Reply-To: References: <4FFF048C.5000500@power.com.pl> <4FFF0EC9.5080302@power.com.pl> Message-ID: 2012/7/12 Erik S?e S?rensen > Because of atomic operations. > I may not have been explicit enough there: latest_now is a > java.util.concurrent.AtomicLong, and the two updating accesses to it are > atomic. > > long prev; > while ((prev = latest_now.get()) < micros) { > if (latest_now.compareAndSet(prev,micros)) { > return micros; > } > } > return latest_now.incrementAndGet(); > > This - and only because of this - means that no lock is necessary. ... and the corresponding Erlang C operations appear to be erts_smp_atomic_cmpxchg() and erts_smp_atomic_inctest(). 2012/7/12 Wojtek Narczy?ski > On 07/12/2012 07:27 PM, Erik S?e S?rensen wrote: > >> Why not? That's certainly the intention of the last part; latest_now >> should always be updated in a increasing manner, and the return value of >> ensure_monotonically_increasing_now() should always be equal to the new >> value of latest_now. >> Did I make a mistake? >> > > Well, how do you "ensure" it without taking a lock? Meanwhile the global > value can be increased by another thread, to what you calculate locally. > > --Regards, > Wojtek > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay@REDACTED Thu Jul 12 20:34:50 2012 From: jay@REDACTED (Jay Nelson) Date: Thu, 12 Jul 2012 11:34:50 -0700 Subject: [erlang-questions] lists, binaries or something else? Message-ID: <697B47FA-DC15-41D9-95EF-E8B4ED4111D0@duomark.com> Processes are private memory spaces. Depending on how you construct the strings, you could spawn a process to create the string convert it to binary and then send the binary as a message. When the process ends, its entire memory space will be reclaimed. The binary will live on the shared binary heap so the sending won't be expensive. The question is the source of the data for the string and whether that can be isolated after the process is spawned. Passing a large data structure on the spawn arguments causes a copy which might blow your memory. jay From wojtek@REDACTED Thu Jul 12 20:49:44 2012 From: wojtek@REDACTED (=?UTF-8?B?V29qdGVrIE5hcmN6ecWEc2tp?=) Date: Thu, 12 Jul 2012 20:49:44 +0200 Subject: [erlang-questions] Lockfree now() In-Reply-To: References: <4FFF048C.5000500@power.com.pl> <4FFF0EC9.5080302@power.com.pl> Message-ID: <4FFF1C48.4050702@power.com.pl> On 07/12/2012 08:23 PM, Erik S?e S?rensen wrote: > Because of atomic operations. > I may not have been explicit enough there: latest_now is a > java.util.concurrent.AtomicLong, and the two updating accesses to it > are atomic. > long prev; > while ((prev = latest_now.get()) < micros) { > if (latest_now.compareAndSet(prev,micros)) { > return micros; > } > } > return latest_now.incrementAndGet(); > > This - and only because of this - means that no lock is necessary. Oh, I think I understand now. You want to have one coarse-grained clock - protected by a lock, and another one fine-grained - updated atomically. Do I understand correctly now? --Regards, Wojtek From cgsmcmlxxv@REDACTED Thu Jul 12 23:45:08 2012 From: cgsmcmlxxv@REDACTED (CGS) Date: Thu, 12 Jul 2012 23:45:08 +0200 Subject: [erlang-questions] Looking for slides of a lightning talk In-Reply-To: <4FFEF003.70605@gmail.com> References: <4FFAAAEA.9060605@ninenines.eu> <0d6c6c5d-2b0f-48da-bae8-f440d232df92@googlegroups.com> <489B3810-49B2-4085-B468-DA037F7DE646@dmitriid.com> <1709242D-0AD5-4070-A6EC-AB039E62ADFD@dmitriid.com> <1342003659.84614.YahooMailNeo@web111404.mail.gq1.yahoo.com> <4FFDFE93.8040202@gmail.com> <4FFEF003.70605@gmail.com> Message-ID: Hi Jan, Few thoughts here. You are hearing things in my email that were not there when I wrote it. > Sorry about that, but stating that "Erlang is just too damn difficult to do the easy stuff" is what I read in your message, not inventing it. And you said that just because of some applications (cowboy and rebar). Use erl with -detach and -noshell options and you have a perfect executable without breaking any sweat to learn how to make an application. All you need is to read "man erl" (you can even google it if you don't have Linux). Of course that's for beginners. Later on, you will learn to make applications with everything you need, so, you will most of the time forget about those options. :) Is that complex? Difficult? I don't suppose it's more difficult than reading about the options of any given compiler. > > I was reporting my experience, not blaming Erlang for anything. My > experience is that the learning curve for the Erlang tools (coupled > unfortunately with learning the Linux environment) was extremely steep - > and ended up with me getting stuck because the tools failed to do what they > claimed (in multiple ways) before I had the skills or knowledge to fix them. > Well, every tool may have the same doom. Those are called bugs and therefore there are versions for each of them. One cannot think of every possibility, can he/she? That doesn't mean the tool is stopping you in learning about what is applied to. Moreover, in the Linux world, until one tool is accepted as satisfactory (so, further tools are no more developed in that field), there are a lot of alternative ones. You don't need to get stubborn in having one and only that tool, especially when you are new in that field. > > Erlang the language is not the problem. The language is never the problem. > Once the tool chain is working...... > Search for the tool which suites your needs or ask if it is possible to get new features which may support your needs in a given time. I tell you that from my own experience because not once it happened to get the wrong tool for my needs and after I managed to learn about that, I discovered it doesn't do what others were thinking it may do. One example is SpiderMonkey for CouchDB. CouchDB and SpiderMonkey are both great tools, but, in the past, CouchDB had quite few problems in getting it work because of SpiderMonkey. So, I googled it and I found a repository where I could find a version of SpiderMonkey which was compatible with that version of CouchDB. Now, CouchDB has no longer that problem anymore. But the conclusion remains: search for the correct version of what you need. > > But consider this. Rebar can't be installed in a standard Ubuntu install > of Erlang!!! WTF? > As far as I know (I am not using rebar because I am creating my own installation scripts which is much faster than thinking how to do it using rebar), rebar doesn't need any installation. I usually take it from the same repository as the source I need to compile. Strangely enough, those projects which do not provide rebar, they work with the latest version of rebar. Never met your problem, but there are many problems I haven't met, so, I don't deny your's. Indeed, bad experience for a newbie. > > Would you agree that that means at least one of those installs is > inadequate for my needs? Its inadequate for anyone who wishes to use rebar > and Erlang on Ubuntu/Debian. > I work under Ubuntu most of the time. It's my favorite Linux flavor (CentOS is becoming the second, but I wouldn't recommend it at starting point). Few times I had problems with it, but I do tend to install my own versions instead of those from repos. They old and providing many crashes in the new versions of applications which require them. Fedora may cope with this problem, but you may have some other surprises there which may be more troublesome if you don't know about Linux. > > Would you agree that giving newbies tools that are inadequate for their > needs, does not help them scale their learning curve? > I do not understand your point. A software is not for a newbie or for a veteran. A software is created to fill an empty spot in a certain field. There are many other tools which you can start with and I don't suppose somebody gave you the tool, but you picked it. All you can do is to ask specific questions about the tool until you learn how to use it properly for your needs. In this community (and quite a lot of others) there will be no answer like RTFM or learn by yourself. There are many persons here who can answer questions in a wide range of knowledge (from where can I find any documentation about Erlang? to, I don't know, "why not using frames in Erlang?" just to give an example which covers a subject way over my head). > > This is not meant as a criticism of the work people have done. Its a very > hard problem, and I don't think there is an easy solution. > In my opinion, there are no easy or difficult solutions. There are only smart solutions or using the brute force. :) When I am not smart enough, I use brute force (it seems quite a natural choice at the beginning and I am still using it in Erlang ;) ). > > PHP isn't there yet. PEAR is not widely used. The comments added to the > manual are very helpful. PHP the language is astonishingly inconsistent. > I wouldn't know that. I know that PHP has an insanely large documentation which I will never be able to know it by heart. :) > > Python boasts "batteries included". Whatever you want to do in Python, > there are usually 6 modules that claim to do it - all poorly documented, > some buggy. So do you spend your time investigating them, or writing a 7th > that you know will do what you want? > Yeah, agree. I usually opt for the last solution as well if I feel smart enough to do that. :) > > Experienced people have forgotten the misunderstandings they had as > newbies - or they had an experienced person to guide them - so they are > genuinely puzzled by the problems newbies face. The newbies can't help: > they don't know what they don't know. Few technical people can write well, > and all are busy with their projects. The result is a distinct lack of > excellent, current documentation of the tools that can help newbies start > to use Erlang. > Therefore there are a lot of others writing some beginner's guides. I remember one of my starting documentations for Erlang was "Learn You Some Erlang" (yes, Fred, I haven't had the opportunity to thank you for that, but I do it now). Just try few of them. And if you want some code, I would recommend to start with trapexit. > > It comes, slowly, with maturity and stability. Erlang the language is well > documented. Its the docs for the supporting cast that is spotty. Which is > a shame. Just don't get discouraged. You still have the chance to criticize and hate Erlang. :D > You blame Erlang for an application written in it which doesn't support MS >> Windows... Well, that reminds me of a case of saying noSQL databases are >> not good just because that person/team misunderstood the usage of a >> particular noSQL database. Interesting how people try to generalize their >> impressions to a category from a wrongly chosen example. Don't get me >> wrong, I am not that good in Erlang to mock you, but I think you should ask >> someone more knowledgeable before you to say something like "Erlang is just >> too damn difficult to do the easy stuff." I do easy stuff in Erlang. It >> would probably be more difficult to write a proper C code to do those >> simple things (like proper threading, distributed applications and so on). >> >> Now, the problem is what do you need cowboy for? If you need an webserver >> for MS Windows which is written in Erlang, you can try Yaws (you have >> installers here: http://yaws.hyber.org/**download/). >> If you need a Erlang-based webserver which you can integrate it in your >> application, you will need either a Linux or Cygwin (for most of the open >> source projects, but you can do it without if you use, for example, lhttpc >> - https://bitbucket.org/etc/**lhttpc/src- where you can modify the Makefile to be compatible with MS VS or, for >> even less, a .bat file). I would avoid installing Linux in a MS Windows box >> (I'd rather do it oppositely). >> > Agreed. Ubuntu host with LVM and software RAID, VBox with a windows guest. > Plans are laid :) > > Your other points are all good stuff. Thanks. > > I am looking for a fast web server that had a permanent core(1) image that > I could program, that was not under any run-away timer (Apache) and that > can handle many simultaneous connections (i.e event driven architecture). > One of the projects is a special type of chat room where silence might > exist for long periods, and I don't want a fast heart-beat or unnecessary > writes to disk. > I may recommend BOSH technique (http://xmpp.org/extensions/xep-0124.html/) rather than long polling. :) Best of luck! CGS > > Note 1 - Core is RAM to anyone under 50. ;) > > > >> Erlang under MS Windows works as nice as under Linux (as far as I could >> play with it under MS Windows), but the installation under MS Windows is >> much simpler (I would suggest for Linux to compile it from source with the >> options you need rather than installing it from repositories). I don't know >> what is your background in programming (and I don't want to know), but >> there are nice tutorials on web for all the categories of programmers >> (coming from OOP, functional or whatever else languages). Just get >> familiarized with Erlang and after that make an impression of your own >> about the language itself. I am also relatively new in Erlang and I >> consider I cannot afford to make any statement about how good or not is >> Erlang. I know only that are some nice things and some not so nice things >> in Erlang (I leave finding examples to your experience). >> > Indeed. Erlang under windows is fine. The target server is Ubuntu, so > developing on that platform has advantages. Particularly in view of the > planned migration to Ubuntu as my main machine. > > Regards > > Ian > >> >> CGS >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eriksoe@REDACTED Thu Jul 12 23:55:18 2012 From: eriksoe@REDACTED (=?ISO-8859-1?Q?Erik_S=F8e_S=F8rensen?=) Date: Thu, 12 Jul 2012 23:55:18 +0200 Subject: [erlang-questions] Lockfree now() In-Reply-To: <4FFF1C48.4050702@power.com.pl> References: <4FFF048C.5000500@power.com.pl> <4FFF0EC9.5080302@power.com.pl> <4FFF1C48.4050702@power.com.pl> Message-ID: 2012/7/12 Wojtek Narczy?ski > On 07/12/2012 08:23 PM, Erik S?e S?rensen wrote: > >> Because of atomic operations. >> I may not have been explicit enough there: latest_now is a >> java.util.concurrent.**AtomicLong, and the two updating accesses to it >> are atomic. >> long prev; >> while ((prev = latest_now.get()) < micros) { >> if (latest_now.compareAndSet(**prev,micros)) { >> return micros; >> } >> } >> return latest_now.incrementAndGet(); >> >> This - and only because of this - means that no lock is necessary. >> > Oh, I think I understand now. > > You want to have one coarse-grained clock - protected by a lock, and > another one fine-grained - updated atomically. > > Do I understand correctly now? > Indeed :-) Roughly speaking, Erlang calculates now() from a monotonic, hi-res time source plus a lower-res absolute-time source. The now-time is the monotonic time plus a certain offset, which may change over time in a controlled fashion, and which is adjusted only when the monotonic source has made a certain amount of progress. This is more or less what already happens, at https://github.com/erlang/otp/blob/master/erts/emulator/beam/erl_time_sup.c line 172. The variant at line 262 and forth is somewhat more complex to follow; I'm afraid it doesn't follow the pattern as things are. -------------- next part -------------- An HTML attachment was scrubbed... URL: From cgsmcmlxxv@REDACTED Fri Jul 13 00:16:16 2012 From: cgsmcmlxxv@REDACTED (CGS) Date: Fri, 13 Jul 2012 00:16:16 +0200 Subject: [erlang-questions] lists, binaries or something else? In-Reply-To: References: Message-ID: Unfortunately, the project is still in the planning stage, so, no real code was written yet. Nevertheless, I plan some open source projects for some parts of the project. About each string, it is constructed from chunks of fixed size, usually, much smaller than the string itself, hopefully. On Thu, Jul 12, 2012 at 7:29 PM, Bob Ippolito wrote: > It would be helpful if you were a lot more specific about how these > strings are constructed and what the strings mean. Maybe if you shared some > of the code, you'd get better guidance. > > > On Thu, Jul 12, 2012 at 9:06 AM, CGS wrote: > >> >> Hi Joe, >> >> The main problem is to find out which strings are read-only and which >> strings are read-write, and that requires an algorithm for itself >> (processing time and extra space - I don't know how negligible are at this >> moment) as I don't know from before which string will be used more >> frequently and which less frequently. The second problem is I would like to >> minimize the harddisk usage, so, to try to store as much information as >> possible in RAM, but without slowing down the overall process. I know, I am >> an idealist. :) >> >> I thought also about working with lists and keep them as binaries when I >> don't use them, but, as I said before, that implies a lot of garbage to >> collect which either can be collected immediately after invoking >> list_to_binary/1, either allowing GC to appear naturally when there is >> insufficient memory, or to invoke it at certain moments (either at regular >> interval of time or based on a scheduler triggered by the application >> usage). I am afraid that all may be quite inefficient, but they may work >> faster than processing binaries directly. That I have no idea yet. That's >> why I am asking here for opinions. >> >> Nevertheless, I didn't think of trying to split the strings in two >> categories: read-only and read-write. That definitely is something I should >> take into account. >> >> Thanks a lot for your thoughts and shared experience. >> >> Cheers, >> CGS >> >> >> >> >> On Thu, Jul 12, 2012 at 5:17 PM, Joe Armstrong wrote: >> >>> As you point out list processing is faster than binary processing. >>> >>> I'd keep things as lists as long as possible until you run into memory >>> problems >>> If you plot the number of strings against response times (or whatever) >>> you should see a sudden decrease in performance when you start paging. >>> At that point you have too much in memory - You could turn the oldest >>> strings >>> into binaries to save space. >>> >>> I generally keep string as lists when I'm working on them and turn >>> them into binaries >>> when I'm finished - sometimes even compressed binaries. >>> >>> Then it depends on the access patterns on the strings - random >>> read-write access is horrible >>> if you can split them into a read-only part and a write-part, you >>> could keep the read-only bit >>> as a binary and the writable bit as a list. >>> >>> It's worth spending a lot of effort to save a single disk access. Then >>> it depends what you do with your strings. If you have a solid state >>> disk and want read only access to the strings >>> then you could store them on disk - or at least arrange so that the >>> constant parts of the strings >>> are on disk and the variable parts in memory. SSDs are about 10 times >>> slower than RAM for reading and usually have multiple controllers so >>> can be very fast - but you need to think a bit first. >>> >>> I'd start with a few measurements, try to stress the system and see >>> where things go wrong. >>> Plot the results - it's usually easy to see when things go wrong. >>> >>> Cheers >>> >>> /Joe >>> >>> >>> >>> On Thu, Jul 12, 2012 at 2:46 PM, CGS wrote: >>> > Hi, >>> > >>> > I am trying to find a balance in between processing speed and RAM >>> > consumption for sets of large strings (over 1 M characters per >>> string). To >>> > construct such lists is much faster than constructing its binary >>> > counterpart. On the other hand, lists are using more RAM than >>> binaries, and >>> > that reduces the number of strings I can hold in memory (unless I >>> transform >>> > the lists in binaries and call GC after that, but that slows down the >>> > processing time). Has anyone had this problem before? What was the >>> solution? >>> > Thoughts? >>> > >>> > A middle way in between lists and binaries is using tuples, but >>> handling >>> > them is not as easy as in the case of lists or binaries, especially at >>> > variable tuple size. Therefore, working with tuples seems not a good >>> > solution. But I might be wrong, so, if anyone used tuples in an >>> efficient >>> > way for this case, please, let me know. >>> > >>> > Any thought would be very much appreciated. Thank you. >>> > >>> > CGS >>> > >>> > >>> > _______________________________________________ >>> > erlang-questions mailing list >>> > erlang-questions@REDACTED >>> > http://erlang.org/mailman/listinfo/erlang-questions >>> > >>> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cgsmcmlxxv@REDACTED Fri Jul 13 00:20:16 2012 From: cgsmcmlxxv@REDACTED (CGS) Date: Fri, 13 Jul 2012 00:20:16 +0200 Subject: [erlang-questions] lists, binaries or something else? In-Reply-To: <697B47FA-DC15-41D9-95EF-E8B4ED4111D0@duomark.com> References: <697B47FA-DC15-41D9-95EF-E8B4ED4111D0@duomark.com> Message-ID: Nice idea about splitting in storage and working processes. Thanks a lot to both of you, Jay and Erik, for suggestion. CGS On Thu, Jul 12, 2012 at 8:34 PM, Jay Nelson wrote: > Processes are private memory spaces. Depending on how you > construct the strings, you could spawn a process to create the > string convert it to binary and then send the binary as a message. > When the process ends, its entire memory space will be reclaimed. > > The binary will live on the shared binary heap so the sending > won't be expensive. The question is the source of the data for > the string and whether that can be isolated after the process > is spawned. Passing a large data structure on the spawn > arguments causes a copy which might blow your memory. > > jay > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Fri Jul 13 00:34:37 2012 From: ok@REDACTED (Richard O'Keefe) Date: Fri, 13 Jul 2012 10:34:37 +1200 Subject: [erlang-questions] Erlang and bbcode In-Reply-To: References: <5C229F26-0DB5-4C67-B8C3-434731D0B233@cs.otago.ac.nz> <9257D246-2C46-4692-B88C-9E99C31B357E@feuerlabs.com> Message-ID: <17869BBE-97E3-4A98-A15B-9C04D0E6479B@cs.otago.ac.nz> I wonder if other people are getting the same take-home lesson from bbcode that I am? It's - we've had adequate notations for describing syntax since 1960, so why, more than 50 years later, are we using notations presented in an informal and incomplete way? - we've known about the importance of testing for nearly as long, so why, in the 21st century, are things like Markdown and BBcode promulgated without a common set of test cases? From ulf@REDACTED Fri Jul 13 00:43:58 2012 From: ulf@REDACTED (Ulf Wiger) Date: Fri, 13 Jul 2012 00:43:58 +0200 Subject: [erlang-questions] mnesia and majority In-Reply-To: References: Message-ID: <24EF4467-AE52-49E9-A354-B61476DBA9F2@feuerlabs.com> One way is to look at mnesia:system_info(running_db_nodes) - see e.g. https://github.com/uwiger/unsplit/blob/master/src/unsplit_server.erl#L138 But since the 'majority' option is table-specific, I'd go for mnesia:table_info(Tab, where_to_wlock), which returns {ActiveNodes, Majority} - that is, how many replicas it sees, and whether the table has the 'majority' option enabled. The undocumented function mnesia_lib:have_majority(Tab, ActiveNodes) can also be helpful. BR, Ulf W Ulf Wiger, Feuerlabs, Inc. http://www.feuerlabs.com 12 jul 2012 kl. 23:42 skrev Mike Oxford : > Interesting, thanks. So how does one identify, when the network comes back up, that "node X was in the majority" so that you can set the master node appropriately? > > -mox > > On Thu, Jul 12, 2012 at 11:27 AM, Ulf Wiger wrote: > > On 12 Jul 2012, at 20:07, Mike Oxford wrote: > >> When facing a split with 'majority' enabled, you end up with a reduction in service based on the non-majority nodes. >> >> If the interconnect-network is entirely down, then every node is non-majority and will 'freeze.' >> If the split is, say, 70/30, then the 30% will not update. >> >> But then once the network is back up, does it automatically replay the missing transactions on the 30% from the 70%? In a full-network-down situation where every node halts the write, when the network comes back up will everything re-mesh on its own? > > Mnesia does no automatic merge. It can detect that a split has happened, and if you have 'majority' enabled, you can trust that you have avoided inconsistency, and can proceed to identify a majority island. > > Mnesia doesn't give you much help with this, although you can spy into the "unsplit" code for _one_ way to do it. Another could be to have a "backdoor ping", e.g. via UDP, and -kernel dist_auto_connect false. This way, you can exchange information between "islands" and control when they are to reconnect. > > The standard mnesia-supported way to resolve the situation would then be to restart the nodes in the minority island, after having called mnesia:set_master_nodes(Tables, MajorityNodes). > > This has always been seen as something of a sledgehammer, but given that the minority nodes are not able to write to the database anyway, it is somewhat less so when combined with the 'majority' option. ;-) > >> Or do I need something like Ulf's "unsplit" handlers? Does mnesia still need to be patched to support "unsplit" or was that merged as well? > > That patch was merged in R14B, I believe. > > BR, > Ulf W > > Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. > http://feuerlabs.com > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wojtek@REDACTED Fri Jul 13 02:08:20 2012 From: wojtek@REDACTED (=?UTF-8?B?V29qdGVrIE5hcmN6ecWEc2tp?=) Date: Fri, 13 Jul 2012 02:08:20 +0200 Subject: [erlang-questions] Lockfree now() In-Reply-To: References: <4FFF048C.5000500@power.com.pl> <4FFF0EC9.5080302@power.com.pl> <4FFF1C48.4050702@power.com.pl> Message-ID: <4FFF66F4.5000901@power.com.pl> On 2012-07-12 23:55, Erik S?e S?rensen wrote: > 2012/7/12 Wojtek Narczy?ski > > > On 07/12/2012 08:23 PM, Erik S?e S?rensen wrote: > > Because of atomic operations. > I may not have been explicit enough there: latest_now is a > java.util.concurrent.AtomicLong, and the two updating accesses > to it are atomic. > long prev; > while ((prev = latest_now.get()) < micros) { > if > (latest_now.compareAndSet(prev,micros)) { > return micros; > } > } > return latest_now.incrementAndGet(); > > This - and only because of this - means that no lock is necessary. > > Oh, I think I understand now. > > You want to have one coarse-grained clock - protected by a lock, > and another one fine-grained - updated atomically. > > Do I understand correctly now? > > > Indeed :-) I looked into the Erjang repo, and indeed now_unique_micros() looks good, so I honorably retreat. > This is more or less what already happens, at > https://github.com/erlang/otp/blob/master/erts/emulator/beam/erl_time_sup.c > line 172. The variant at line 262 and forth is somewhat more complex > to follow; I'm afraid it doesn't follow the pattern as things are. I suspect that there are some serious differences between platforms, because on Unix the POSIX API (clock_gettime and friends) is a clean one - you have a choice of different clocks with different characteristics. Erlang exposes only one clock, the most expensive one possible - strictly increasing [0, 1, 2, 3, ...], basically a clock and a counter in one. Using atomic operations instead of locks would definitely be faster. But still slower compared with an increasing clock [0, 1, 1, 2, ...], and other clocks. Certainly, it would be nice to have that POSIX choices in erlang. I have been surprised once by erlang:now() performance on multiclore, but still I do not think this is big issue. --Regards, Wojtek -------------- next part -------------- An HTML attachment was scrubbed... URL: From bob@REDACTED Fri Jul 13 02:32:12 2012 From: bob@REDACTED (Bob Ippolito) Date: Thu, 12 Jul 2012 17:32:12 -0700 Subject: [erlang-questions] lists, binaries or something else? In-Reply-To: References: Message-ID: Sorry, but that's not enough information. Where do these chunks come from: source code, some other process, ets? How many chunks are in a string? Do you compose strings out of other strings, or just these chunks? Are you constructing them from tail to head like you would a list? Is the string constructed all at once, or over some time? It sounds like you have some code that's doing something with these strings, because you say that it's faster to use lists than binaries. Maybe if you post whatever it is you used to determine that, someone can help you find a better algorithm and/or data structure for that benchmark. On Thursday, July 12, 2012, CGS wrote: > Unfortunately, the project is still in the planning stage, so, no real > code was written yet. Nevertheless, I plan some open source projects for > some parts of the project. > > About each string, it is constructed from chunks of fixed size, usually, > much smaller than the string itself, hopefully. > > > > On Thu, Jul 12, 2012 at 7:29 PM, Bob Ippolito wrote: > > It would be helpful if you were a lot more specific about how these > strings are constructed and what the strings mean. Maybe if you shared some > of the code, you'd get better guidance. > > > On Thu, Jul 12, 2012 at 9:06 AM, CGS wrote: > > > Hi Joe, > > The main problem is to find out which strings are read-only and which > strings are read-write, and that requires an algorithm for itself > (processing time and extra space - I don't know how negligible are at this > moment) as I don't know from before which string will be used more > frequently and which less frequently. The second problem is I would like to > minimize the harddisk usage, so, to try to store as much information as > possible in RAM, but without slowing down the overall process. I know, I am > an idealist. :) > > I thought also about working with lists and keep them as binaries when I > don't use them, but, as I said before, that implies a lot of garbage to > collect which either can be collected immediately after invoking > list_to_binary/1, either allowing GC to appear naturally when there is > insufficient memory, or to invoke it at certain moments (either at regular > interval of time or based on a scheduler triggered by the application > usage). I am afraid that all may be quite inefficient, but they may work > faster than processing binaries directly. That I have no idea yet. That's > why I am asking here for opinions. > > Nevertheless, I didn't think of trying to split the strings in two > categories: read-only and read-write. That definitely is something I should > take into account. > > Thanks a lot for your thoughts and shared experience. > > Cheers, > CGS > > > > > On Thu, Jul 12, 2012 at 5:17 PM, Joe Armstrong wrote: > > As you point out list processing is faster than binary processing. > > I'd keep things as lists as long as possible until you run into memory > problems > If you plot the number of strings against response times (or whatever) > you should see a sudden decrease in performance when you start paging. > At that point you have too much in memory - You could turn the oldest > strings > into binaries to save space. > > I generally keep string as lists when I'm working on them and turn > them into binaries > when I'm finished - sometimes even compressed binaries. > > Then it depends on the access patterns on the strings - random > read-write access is horrible > if you can split them into a read-only part and a write-part, you > could keep the read-only bit > as a binary and the writable bit as a list. > > It's worth spending a lot of effort to save a single disk access. Then > it depends what you do with your strings. If you have a solid state > disk and want read only access to the strings > then you could store them on disk - or at least arrange so that the > constant parts of the strings > are on disk and the variable parts in memory. SSDs are about 10 times > slower than RAM for reading and usually have multiple controllers so > can be very fast - but you need to think a bit first. > > I'd start with a few measurements, try to stress the system and see > where things go wrong. > Plot the results - it's usually easy to see when things go wrong. > > Cheers > > /Joe > > > > On Thu, Jul 12, 2012 at 2:46 PM, CGS wrote: > > Hi, > > > > I am trying to find a balance in between processing speed and RAM > > consumption for sets > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bob@REDACTED Fri Jul 13 02:43:27 2012 From: bob@REDACTED (Bob Ippolito) Date: Thu, 12 Jul 2012 17:43:27 -0700 Subject: [erlang-questions] Erlang and bbcode In-Reply-To: <17869BBE-97E3-4A98-A15B-9C04D0E6479B@cs.otago.ac.nz> References: <5C229F26-0DB5-4C67-B8C3-434731D0B233@cs.otago.ac.nz> <9257D246-2C46-4692-B88C-9E99C31B357E@feuerlabs.com> <17869BBE-97E3-4A98-A15B-9C04D0E6479B@cs.otago.ac.nz> Message-ID: Sounds like an education problem to me. If I had to guess, I'd say that markdown was done this way because it was somehow easier to come up with a ball of regexes than make a complete grammar that did something useful. Testing seems a lot more mainstream now than it was when either of these things were developed. On Thursday, July 12, 2012, Richard O'Keefe wrote: > I wonder if other people are getting the same take-home lesson from bbcode > that I am? It's > > - we've had adequate notations for describing syntax since > 1960, so why, more than 50 years later, are we using notations > presented in an informal and incomplete way? > > - we've known about the importance of testing for nearly as long, > so why, in the 21st century, are things like Markdown and BBcode > promulgated without a common set of test cases? > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cyg.cao@REDACTED Fri Jul 13 04:10:49 2012 From: cyg.cao@REDACTED (Wei Cao) Date: Fri, 13 Jul 2012 10:10:49 +0800 Subject: [erlang-questions] Improve performance of IO bounded server written in Erlang via having pollset for each scheduler and bind port to scheduler together with process In-Reply-To: <4FFF0D25.7050601@power.com.pl> References: <771ADB6B-9CD0-4A7A-A8A1-EC551B84BAD1@gmail.com> <0E26E896-FF0D-4036-A51A-8F3B8C1FA31C@gmail.com> <4FFF0D25.7050601@power.com.pl> Message-ID: I think periodly calls to erts_deliver_time() is necessary.Because at first I tried to comment out the call to erts_deliver_time() from erts_check_io(), it would cause erlang VM hang out. erts_deliver_time() does more than erts_time_remaining(), it updates a clock counter named do_time which is used inside timer wheel. 2012/7/13 Wojtek Narczy??ski : > On 07/12/2012 11:58 AM, Wei Cao wrote: >> >> the mutex timeofday is locked each time erts_check_io is invoked to >> "sync the machine's idea of time", erts_check_io is executed hundreds >> of thounds of times per second, so it's locked too much times, hence >> reduce performance. >> > I am trying to figure out how is "the machine's idea of time" relevant to > checking for io. I am begining to suspect that the lone call to > erts_deliver_time() in erl_check_io.c:1179 might be unnecessary. If do_wait > is true, the call to erts_time_remaining(&wait_time) will update the global > time. If do_wait is false, which should be the case most of the time on a > busy server, as per erts_check_io(!runnable) calls, the timeout to > select/poll/epoll whichever is going to be zero anyway. There must be a > catch, because this simply looks too good to be true... > > --Regards, > Wojtek Narczynski -- Best, Wei Cao From ok@REDACTED Fri Jul 13 05:31:55 2012 From: ok@REDACTED (Richard O'Keefe) Date: Fri, 13 Jul 2012 15:31:55 +1200 Subject: [erlang-questions] Erlang and bbcode In-Reply-To: References: <5C229F26-0DB5-4C67-B8C3-434731D0B233@cs.otago.ac.nz> <9257D246-2C46-4692-B88C-9E99C31B357E@feuerlabs.com> <17869BBE-97E3-4A98-A15B-9C04D0E6479B@cs.otago.ac.nz> Message-ID: <4E57FFCE-2CFE-4C9E-A9A1-94E9957BD5D9@cs.otago.ac.nz> On 13/07/2012, at 12:43 PM, Bob Ippolito wrote: > Sounds like an education problem to me. If I had to guess, I'd say that markdown was done this way because it was somehow easier to come up with a ball of regexes than make a complete grammar that did something useful. Testing seems a lot more mainstream now than it was when either of these things were developed. When _were_ they developed? Testing has been around for a _long_ time. From bob@REDACTED Fri Jul 13 05:38:36 2012 From: bob@REDACTED (Bob Ippolito) Date: Thu, 12 Jul 2012 20:38:36 -0700 Subject: [erlang-questions] Erlang and bbcode In-Reply-To: <4E57FFCE-2CFE-4C9E-A9A1-94E9957BD5D9@cs.otago.ac.nz> References: <5C229F26-0DB5-4C67-B8C3-434731D0B233@cs.otago.ac.nz> <9257D246-2C46-4692-B88C-9E99C31B357E@feuerlabs.com> <17869BBE-97E3-4A98-A15B-9C04D0E6479B@cs.otago.ac.nz> <4E57FFCE-2CFE-4C9E-A9A1-94E9957BD5D9@cs.otago.ac.nz> Message-ID: On Thu, Jul 12, 2012 at 8:31 PM, Richard O'Keefe wrote: > > On 13/07/2012, at 12:43 PM, Bob Ippolito wrote: > > > Sounds like an education problem to me. If I had to guess, I'd say that > markdown was done this way because it was somehow easier to come up with a > ball of regexes than make a complete grammar that did something useful. > Testing seems a lot more mainstream now than it was when either of these > things were developed. > > When _were_ they developed? Testing has been around for a _long_ time. > > Looks like Markdown was 2003. From memory, comprehensive unit testing of small open source projects wasn't as popular then as it is now. -bob -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.sovetov@REDACTED Fri Jul 13 09:30:41 2012 From: victor.sovetov@REDACTED (Viktor Sovietov) Date: Fri, 13 Jul 2012 00:30:41 -0700 (PDT) Subject: [erlang-questions] Ling: Erlang over Xen, no OS required Message-ID: <9b565382-35da-472d-8315-0a8d7b35ace8@n5g2000vbb.googlegroups.com> Ling is highly compatible (with R15B) Erlang VM that is capable to run over bare virtual iron on Xen hypervisor without the need in underlying operating system. Learn more on : http://erlangonxen.org/ There are similar projects for OCaml http://www.openmirage.org/ and Haskell http://halvm.org/ From olopierpa@REDACTED Fri Jul 13 09:51:18 2012 From: olopierpa@REDACTED (Pierpaolo Bernardi) Date: Fri, 13 Jul 2012 09:51:18 +0200 Subject: [erlang-questions] Ling: Erlang over Xen, no OS required In-Reply-To: <9b565382-35da-472d-8315-0a8d7b35ace8@n5g2000vbb.googlegroups.com> References: <9b565382-35da-472d-8315-0a8d7b35ace8@n5g2000vbb.googlegroups.com> Message-ID: That's incredibly cool! On Fri, Jul 13, 2012 at 9:30 AM, Viktor Sovietov wrote: > Ling is highly compatible (with R15B) Erlang VM that is capable to run > over bare virtual iron on Xen hypervisor without the need in > underlying operating system. > > Learn more on : http://erlangonxen.org/ > > There are similar projects for OCaml http://www.openmirage.org/ > and Haskell http://halvm.org/ > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From victor.sovetov@REDACTED Fri Jul 13 10:19:02 2012 From: victor.sovetov@REDACTED (Viktor Sovietov) Date: Fri, 13 Jul 2012 01:19:02 -0700 (PDT) Subject: [erlang-questions] Ling: Erlang over Xen, no OS required In-Reply-To: References: <9b565382-35da-472d-8315-0a8d7b35ace8@n5g2000vbb.googlegroups.com> Message-ID: <7be84695-4af3-4d0f-ab92-d4a149805c7a@b1g2000vbb.googlegroups.com> Probably, it is, the next step is implementing really elastic scalable service. On 13 ???, 10:51, Pierpaolo Bernardi wrote: > That's incredibly cool! > > On Fri, Jul 13, 2012 at 9:30 AM, Viktor Sovietov wrote: > > Ling is highly compatible (with R15B) Erlang VM that is capable to run > > over bare virtual iron on Xen hypervisor without the need in > > underlying operating system. > > > Learn more on :http://erlangonxen.org/ > > > There are similar projects for OCaml ?http://www.openmirage.org/ > > and Haskellhttp://halvm.org/ > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questi...@REDACTED > >http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questi...@REDACTED://erlang.org/mailman/listinfo/erlang-questions From eriksoe@REDACTED Fri Jul 13 10:35:48 2012 From: eriksoe@REDACTED (=?ISO-8859-1?Q?Erik_S=F8e_S=F8rensen?=) Date: Fri, 13 Jul 2012 10:35:48 +0200 Subject: [erlang-questions] Lockfree now() In-Reply-To: <4FFF66F4.5000901@power.com.pl> References: <4FFF048C.5000500@power.com.pl> <4FFF0EC9.5080302@power.com.pl> <4FFF1C48.4050702@power.com.pl> <4FFF66F4.5000901@power.com.pl> Message-ID: Oh, but you do have a choice between more than one time source in Erlang: now() - as known, for microsecond-precision and uniqueness erlang:universaltime() and friends - for second-precision and YMD-HMS format. os:timestamp() - for microsecond-precision but without uniqueness and time correction. For many places where you could use now() but simply need a timestamp for logging or the like, os:timestamp() might be more appropriate and a bit more lightweight. Its return type is compatible with that of now(). 2012/7/13 Wojtek Narczy?ski > On 2012-07-12 23:55, Erik S?e S?rensen wrote: > > 2012/7/12 Wojtek Narczy?ski > >> On 07/12/2012 08:23 PM, Erik S?e S?rensen wrote: >> >>> Because of atomic operations. >>> I may not have been explicit enough there: latest_now is a >>> java.util.concurrent.AtomicLong, and the two updating accesses to it are >>> atomic. >>> long prev; >>> while ((prev = latest_now.get()) < micros) { >>> if (latest_now.compareAndSet(prev,micros)) { >>> return micros; >>> } >>> } >>> return latest_now.incrementAndGet(); >>> >>> This - and only because of this - means that no lock is necessary. >>> >> Oh, I think I understand now. >> >> You want to have one coarse-grained clock - protected by a lock, and >> another one fine-grained - updated atomically. >> >> Do I understand correctly now? >> > > Indeed :-) > > > I looked into the Erjang repo, and indeed now_unique_micros() looks good, > so I honorably retreat. > > > This is more or less what already happens, at > > https://github.com/erlang/otp/blob/master/erts/emulator/beam/erl_time_sup.c > line 172. The variant at line 262 and forth is somewhat more complex to > follow; I'm afraid it doesn't follow the pattern as things are. > > > I suspect that there are some serious differences between platforms, > because on Unix the POSIX API (clock_gettime and friends) is a clean one - > you have a choice of different clocks with different characteristics. > Erlang exposes only one clock, the most expensive one possible - strictly > increasing [0, 1, 2, 3, ...], basically a clock and a counter in one. Using > atomic operations instead of locks would definitely be faster. But still > slower compared with an increasing clock [0, 1, 1, 2, ...], and other > clocks. Certainly, it would be nice to have that POSIX choices in erlang. I > have been surprised once by erlang:now() performance on multiclore, but > still I do not think this is big issue. > > --Regards, > Wojtek > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gustav.simonsson@REDACTED Fri Jul 13 11:22:03 2012 From: gustav.simonsson@REDACTED (Gustav Simonsson) Date: Fri, 13 Jul 2012 10:22:03 +0100 (BST) Subject: [erlang-questions] Erlang get_memory_data In-Reply-To: Message-ID: This is because the second value in the tuple returned by memsup:get_memory_data/0 does not represent "used" memory, it represents memory Linux has allocated for use while no applications require it [1]. On my machine the function returns {3169628160,3058540544,{<0.6.0>,213104}} and 'free -m' reports I have 2.2 GB free for applications in the "-/+ buffers/cache" row. I looked briefly at the memsup source code, and it looks like the calculation that decides whether a high watermark is set or not is based on the allocated value [2], and not how much memory is in fact available for applications under Linux. 1. http://www.linuxatemyram.com/ 2. https://github.com/erlang/otp/blob/maint/lib/os_mon/src/memsup.erl#L353 It looks like this needs to be revised for when memsup is running on Linux, so it does not create false high memory alarms. // Gustav Simonsson Sent from my PC ----- Original Message ----- > From: "tsuraan" > To: erlang-questions@REDACTED > Sent: Thursday, 12 July, 2012 8:09:13 PM > Subject: [erlang-questions] Erlang get_memory_data > > I have a Riak database that frequently gets memsup high watermark > warnings and long_gc warnings, but really doesn't seem to be using > much RAM. When I run memsup:get_memory_data(), I get this output: > > {4153327616,3424722944,{<0.7.0>,372152}} > > Which I think indicates that erlang believes it's using ~3.5GB of RAM > (or does it mean something completely different?). If I run "grep > ^Vm > /proc//status", I get this output: > > VmPeak: 334560 kB > VmSize: 270252 kB > VmLck: 0 kB > VmHWM: 33384 kB > VmRSS: 33084 kB > VmData: 225832 kB > VmStk: 136 kB > VmExe: 2028 kB > VmLib: 6792 kB > VmPTE: 232 kB > VmSwap: 0 kB > > So, Linux thinks Riak (beam) has never used more than ~350MB of RAM, > and that it only has a ~30MB resident set. pmap shows 90MB of > writable anonymous pages, and a total of 262MB of mapped data. Where > is memsup's 3.5GB of RAM coming from? Or am I way off on my ideas > about what memsup is trying to tell me? > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From erlang@REDACTED Fri Jul 13 13:04:26 2012 From: erlang@REDACTED (Joe Armstrong) Date: Fri, 13 Jul 2012 13:04:26 +0200 Subject: [erlang-questions] Erlang and bbcode In-Reply-To: <17869BBE-97E3-4A98-A15B-9C04D0E6479B@cs.otago.ac.nz> References: <5C229F26-0DB5-4C67-B8C3-434731D0B233@cs.otago.ac.nz> <9257D246-2C46-4692-B88C-9E99C31B357E@feuerlabs.com> <17869BBE-97E3-4A98-A15B-9C04D0E6479B@cs.otago.ac.nz> Message-ID: On Fri, Jul 13, 2012 at 12:34 AM, Richard O'Keefe wrote: > I wonder if other people are getting the same take-home lesson from bbcode > that I am? It's > > - we've had adequate notations for describing syntax since > 1960, so why, more than 50 years later, are we using notations > presented in an informal and incomplete way? > > - we've known about the importance of testing for nearly as long, > so why, in the 21st century, are things like Markdown and BBcode > promulgated without a common set of test cases? > But tests are used to see if the implementation corresponds to the specification. Since they have no specification, their can't really be any tests. Or worse, in the absence of a specification, the tests *become* the specification. Thus if the first implementation of XXX was stupid all re-implementations would also have to be stupid (as is the case) /Joe > From mahesh@REDACTED Fri Jul 13 13:29:58 2012 From: mahesh@REDACTED (Mahesh Paolini-Subramanya) Date: Fri, 13 Jul 2012 07:29:58 -0400 Subject: [erlang-questions] Erlang and bbcode In-Reply-To: References: <5C229F26-0DB5-4C67-B8C3-434731D0B233@cs.otago.ac.nz> <9257D246-2C46-4692-B88C-9E99C31B357E@feuerlabs.com> <17869BBE-97E3-4A98-A15B-9C04D0E6479B@cs.otago.ac.nz> Message-ID: I'd also like to point out that a some (most?) of these also belonged to the "Make it up as you go along" camp. Which isn't necessarily bad, but in some (most?) of these cases, they were being made up by people who were really not all that experienced in the field. cheers p.s. Do note that "lack of expertise" is not necessarily A Bad Thing... Mahesh Paolini-Subramanya That Tall Bald Indian Guy... Blog | Twitter | Google+ On Jul 13, 2012, at 7:04 AM, Joe Armstrong wrote: > On Fri, Jul 13, 2012 at 12:34 AM, Richard O'Keefe wrote: >> I wonder if other people are getting the same take-home lesson from bbcode >> that I am? It's >> >> - we've had adequate notations for describing syntax since >> 1960, so why, more than 50 years later, are we using notations >> presented in an informal and incomplete way? >> >> - we've known about the importance of testing for nearly as long, >> so why, in the 21st century, are things like Markdown and BBcode >> promulgated without a common set of test cases? >> > > But tests are used to see if the implementation corresponds to the > specification. > Since they have no specification, their can't really be any tests. > > Or worse, in the absence of a specification, the tests *become* the > specification. > > Thus if the first implementation of XXX was stupid all re-implementations > would also have to be stupid (as is the case) > > /Joe > >> > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From cgsmcmlxxv@REDACTED Fri Jul 13 13:48:02 2012 From: cgsmcmlxxv@REDACTED (CGS) Date: Fri, 13 Jul 2012 13:48:02 +0200 Subject: [erlang-questions] lists, binaries or something else? In-Reply-To: References: Message-ID: Hi Bob, On Fri, Jul 13, 2012 at 2:32 AM, Bob Ippolito wrote: > Sorry, but that's not enough information. Sorry, but I didn't know what you meant by that. Here are the answers. > > Where do these chunks come from: source code, some other process, ets? > Mostly from other processes. > How many chunks are in a string? > That is computed from the string size and chunk size (to be decided later). > Do you compose strings out of other strings, or just these chunks? > Just chunks inserted into existing/new strings. > Are you constructing them from tail to head like you would a list? > Unfortunately, not all the time. > Is the string constructed all at once, or over some time? > If you mean by that the string will be fully given in the same message or whatever by other processes, the answer is no. Over some time may be the answer, but with the remark that I have no idea what means "over some time" as period of time (can get chunks one after the other for the same string or for different strings). > > It sounds like you have some code that's doing something with these > strings, because you say that it's faster to use lists than binaries. Maybe > if you post whatever it is you used to determine that, someone can help you > find a better algorithm and/or data structure for that benchmark. > That is simple. For benchmark I just took two simple loops (only for insertion): -export([loop_list/1, loop_binary/2]). loop_list(0) -> []; loop_list(N) -> [107 | loop_list(N)]. loop_binary(0,B) -> B; loop_binary(N,B) -> loop_binary(N-1,<<107,B/binary>>). If you go in powers of tens and measure the execution time, you will see the difference (you will also notice the drop in efficiency for binaries when you need fragmentation for the same string, which is not visible in the case of lists - or at least I couldn't notice). That is not a fully conclusive test for my case, but it is quite a conclusive test for processing speed in the two cases (simple expansion of the string by adding a char at the beginning of the string), in which, for a 10^5 chars string, the list gains at least one order of magnitude in processing time than its binary counterpart (1) (2) (3). (1) Even if you change the list loop to have an accumulator in which the list exists, there is still one order of magnitude difference. (2) The results are specific for the machine on which the tests are done. (3) That is just an example. CGS > > > On Thursday, July 12, 2012, CGS wrote: > >> Unfortunately, the project is still in the planning stage, so, no real >> code was written yet. Nevertheless, I plan some open source projects for >> some parts of the project. >> >> About each string, it is constructed from chunks of fixed size, usually, >> much smaller than the string itself, hopefully. >> >> >> >> On Thu, Jul 12, 2012 at 7:29 PM, Bob Ippolito wrote: >> >> It would be helpful if you were a lot more specific about how these >> strings are constructed and what the strings mean. Maybe if you shared some >> of the code, you'd get better guidance. >> >> >> On Thu, Jul 12, 2012 at 9:06 AM, CGS wrote: >> >> >> Hi Joe, >> >> The main problem is to find out which strings are read-only and which >> strings are read-write, and that requires an algorithm for itself >> (processing time and extra space - I don't know how negligible are at this >> moment) as I don't know from before which string will be used more >> frequently and which less frequently. The second problem is I would like to >> minimize the harddisk usage, so, to try to store as much information as >> possible in RAM, but without slowing down the overall process. I know, I am >> an idealist. :) >> >> I thought also about working with lists and keep them as binaries when I >> don't use them, but, as I said before, that implies a lot of garbage to >> collect which either can be collected immediately after invoking >> list_to_binary/1, either allowing GC to appear naturally when there is >> insufficient memory, or to invoke it at certain moments (either at regular >> interval of time or based on a scheduler triggered by the application >> usage). I am afraid that all may be quite inefficient, but they may work >> faster than processing binaries directly. That I have no idea yet. That's >> why I am asking here for opinions. >> >> Nevertheless, I didn't think of trying to split the strings in two >> categories: read-only and read-write. That definitely is something I should >> take into account. >> >> Thanks a lot for your thoughts and shared experience. >> >> Cheers, >> CGS >> >> >> >> >> On Thu, Jul 12, 2012 at 5:17 PM, Joe Armstrong wrote: >> >> As you point out list processing is faster than binary processing. >> >> I'd keep things as lists as long as possible until you run into memory >> problems >> If you plot the number of strings against response times (or whatever) >> you should see a sudden decrease in performance when you start paging. >> At that point you have too much in memory - You could turn the oldest >> strings >> into binaries to save space. >> >> I generally keep string as lists when I'm working on them and turn >> them into binaries >> when I'm finished - sometimes even compressed binaries. >> >> Then it depends on the access patterns on the strings - random >> read-write access is horrible >> if you can split them into a read-only part and a write-part, you >> could keep the read-only bit >> as a binary and the writable bit as a list. >> >> It's worth spending a lot of effort to save a single disk access. Then >> it depends what you do with your strings. If you have a solid state >> disk and want read only access to the strings >> then you could store them on disk - or at least arrange so that the >> constant parts of the strings >> are on disk and the variable parts in memory. SSDs are about 10 times >> slower than RAM for reading and usually have multiple controllers so >> can be very fast - but you need to think a bit first. >> >> I'd start with a few measurements, try to stress the system and see >> where things go wrong. >> Plot the results - it's usually easy to see when things go wrong. >> >> Cheers >> >> /Joe >> >> >> >> On Thu, Jul 12, 2012 at 2:46 PM, CGS wrote: >> > Hi, >> > >> > I am trying to find a balance in between processing speed and RAM >> > consumption for sets >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From wojtek@REDACTED Fri Jul 13 14:00:33 2012 From: wojtek@REDACTED (=?UTF-8?B?V29qdGVrIE5hcmN6ecWEc2tp?=) Date: Fri, 13 Jul 2012 14:00:33 +0200 Subject: [erlang-questions] Lockfree now() In-Reply-To: References: <4FFF048C.5000500@power.com.pl> <4FFF0EC9.5080302@power.com.pl> <4FFF1C48.4050702@power.com.pl> <4FFF66F4.5000901@power.com.pl> Message-ID: <50000DE1.3030506@power.com.pl> On 07/13/2012 10:35 AM, Erik S?e S?rensen wrote: > Oh, but you do have a choice between more than one time source in Erlang: > now() - as known, for microsecond-precision and uniqueness > erlang:universaltime() and friends - for second-precision and YMD-HMS > format. > os:timestamp() - for microsecond-precision but without uniqueness and > time correction. > I didn't know that. Not that I ever looked thoroughly... > For many places where you could use now() but simply need a timestamp > for logging or the like, os:timestamp() might be more appropriate and > a bit more lightweight. Its return type is compatible with that of now(). I just replaced calendar:now_to_datetime(erlang:now()) with erlang:universaltime() in logging code and gained 13% performance. And that machine is only dual-core. --Regards, Wojtek From ulf@REDACTED Fri Jul 13 14:11:18 2012 From: ulf@REDACTED (Ulf Wiger) Date: Fri, 13 Jul 2012 14:11:18 +0200 Subject: [erlang-questions] Erlang and bbcode In-Reply-To: References: <5C229F26-0DB5-4C67-B8C3-434731D0B233@cs.otago.ac.nz> <9257D246-2C46-4692-B88C-9E99C31B357E@feuerlabs.com> <17869BBE-97E3-4A98-A15B-9C04D0E6479B@cs.otago.ac.nz> Message-ID: <1FAE142E-806A-4A59-BDA4-B2D49A68A702@feuerlabs.com> On 13 Jul 2012, at 13:04, Joe Armstrong wrote: > Since they have no specification, their can't really be any tests. > > Or worse, in the absence of a specification, the tests *become* the > specification. Well, in general, I'd have to take exception to this description ;-), but in the case of a markup language, pushing through without a specification is obviously asking for trouble. With common commercial development, there is very seldom a specification in any useful sense of the word. What there is, is a customer's more or less vague idea of what they want, and the reality any implementation of their idea will eventually face. In such situations, the tests *do* become the specification, because a specification *cannot* be written! If you try to write one anyway, it will seriously delay the project, and be obsolete as soon as it's finished, if it ever had a chance of being correct (it probably didn't). In other words, don't blame the designer who creates a test suite in those cases, rather than brow-beating the poor customer until they come up with a workable specification. The practice of using cue cards to write user stories (and the acceptance criteria on the back) evolved very much because it was a workable way of discussing the requirements with users, whether or not they had any engineering or science background. http://www.allaboutagile.com/user-story-example/ It seems ridiculously vague and limited, but in practice, it actually helps to catch *some* of the information needed to arrive at some mutually understood acceptance criteria. This wasn't created because people didn't understand what formal specifications, or thorough testing, looked like - it was created in order to achieve at least *some* level of improvement with *real* projects and real customers. Besides, when did you last see a good specification of how a Web GUI should behave - or a good way to test it? * Another example: When I gave an invited talk at NWPT'99 [1] in Uppsala, Sweden, a member of the audience (probably a PdD student), reacted to my description of the 3-month test period before release of the AXD 301, saying "That's ridiculous! If you had written it using a language that could be formally verified, you wouldn't have to test at all!". I mumbled something about not knowing if that would have been at all possible, and Thomas Arts jumped in and offered to argue the point with him in the break. It was a very vivid discussion, as I recall. [1] http://www.it.uu.se/research/publications/reports/1999-008/nwpt99/proceedings/ Obviously at that scale (for all except for that PhD student), there is no chance of arriving at a coherent formal specification, since the standards we were supposed to comply with were not themselves formally specified**, and many of the customers didn't necessarily want *exactly* what was in the standard anyway. Furthermore, many of the non-functional requirements were not specified at all, except as a more or less vague list of "goodness criteria", trying to encapsulate what it means for a system to be robust. What we ended up with was a large pile of informal requirements, and a huge set of test cases (nearly a million lines of code). At that point, the test suites *were* indeed the real specification, albeit an extremely bulky and inaccessible one. BR, Ulf W * One of the best is probably Cucumber (http://cukes.info/), which is based on cue cards, and relies on quite an ambitious mapping to the underlying web stack (e.g. Rails). Without such a mapping, it is less useful, although it still has its sweet spots. ** ?although some would disagree. They are certainly specified in a bureaucratic way, but that's not the same thing. Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. http://feuerlabs.com From mfidelman@REDACTED Fri Jul 13 14:14:19 2012 From: mfidelman@REDACTED (Miles Fidelman) Date: Fri, 13 Jul 2012 08:14:19 -0400 Subject: [erlang-questions] Ling: Erlang over Xen, no OS required In-Reply-To: References: <9b565382-35da-472d-8315-0a8d7b35ace8@n5g2000vbb.googlegroups.com> Message-ID: <5000111B.7010109@meetinghouse.net> +1 Pierpaolo Bernardi wrote: > That's incredibly cool! > > > On Fri, Jul 13, 2012 at 9:30 AM, Viktor Sovietov > wrote: >> Ling is highly compatible (with R15B) Erlang VM that is capable to run >> over bare virtual iron on Xen hypervisor without the need in >> underlying operating system. >> >> Learn more on : http://erlangonxen.org/ >> >> There are similar projects for OCaml http://www.openmirage.org/ >> and Haskell http://halvm.org/ >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- In theory, there is no difference between theory and practice. In practice, there is. .... Yogi Berra From jbothma@REDACTED Fri Jul 13 14:20:43 2012 From: jbothma@REDACTED (JD Bothma) Date: Fri, 13 Jul 2012 14:20:43 +0200 Subject: [erlang-questions] Ling: Erlang over Xen, no OS required In-Reply-To: <9b565382-35da-472d-8315-0a8d7b35ace8@n5g2000vbb.googlegroups.com> References: <9b565382-35da-472d-8315-0a8d7b35ace8@n5g2000vbb.googlegroups.com> Message-ID: Probably a silly question but... How does networking work without an OS? Do you just assign an interface in Xen that Ling knows how to manage directly? Is Ling 'basically' the standard VM with this and other facilities that the OS would provide hacked on? Why does Ling mean "no more out of memory crashes"? How does Ling deal with using up its allocated memory? Sounds cool! Is it open source? JD On 13 July 2012 09:30, Viktor Sovietov wrote: > Ling is highly compatible (with R15B) Erlang VM that is capable to run > over bare virtual iron on Xen hypervisor without the need in > underlying operating system. > > Learn more on : http://erlangonxen.org/ > > There are similar projects for OCaml http://www.openmirage.org/ > and Haskell http://halvm.org/ > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From gordon@REDACTED Fri Jul 13 14:33:22 2012 From: gordon@REDACTED (Gordon Guthrie) Date: Fri, 13 Jul 2012 13:33:22 +0100 Subject: [erlang-questions] Erlang and bbcode In-Reply-To: <1FAE142E-806A-4A59-BDA4-B2D49A68A702@feuerlabs.com> References: <5C229F26-0DB5-4C67-B8C3-434731D0B233@cs.otago.ac.nz> <9257D246-2C46-4692-B88C-9E99C31B357E@feuerlabs.com> <17869BBE-97E3-4A98-A15B-9C04D0E6479B@cs.otago.ac.nz> <1FAE142E-806A-4A59-BDA4-B2D49A68A702@feuerlabs.com> Message-ID: If we go back to 1999 - so I was the Chief Technical Architect at if.comand we spend $100m on technology. We didn't have ANY tests at all. What we did have was a testing team of I suppose 35 full time testers and varyingly dozens to hundreds on secondment. If you were in the business of consuming software (unlike Ericsson) you didn't have kit lying around. If you split out your code base into components that are pure functions (unit testable) and components that have side effects (system testable) you can build good test case coverage into your dev process. But if you are an n-tier architecture in imperative/OO languages with side-effects in almost all modules you can only do system tests. If you are a n-tier application (web server/Java mid-tier/mainframe backend) no developer has access to a machine that is capable of running 'the system' - so automatic system testing is simply impossible - it is not possible write them, let alone run them many times a day. So what you end up with is a monolithic user testing process (at if.com/Halifax it was 12 weeks end to end MINIMUM from developers desktop to production) with 6 major testing gates and you beat the software into submission. If you look at us now at Vixo (formerly Hypernumbers) we have a BFN* of tests and we can release from dev to production many times a day. The reason for this is that: * most of our code is side-effect free and is built backwards from tests - you write a test suite and tinker about until the module passes them all * everyone has full access to a machine which has near production performance (ie a laptop) We rent boxes for a couple of hundred dollars a month that we paid capital costs of ?1.5m the pair in '99 Testing may well have been invented years ago but it is still fairly recent in bespoke commercial software development. Gordon * BFN is the correct mathematical term for a Big Number On 13 July 2012 13:11, Ulf Wiger wrote: > > On 13 Jul 2012, at 13:04, Joe Armstrong wrote: > > > Since they have no specification, their can't really be any tests. > > > > Or worse, in the absence of a specification, the tests *become* the > > specification. > > Well, in general, I'd have to take exception to this description ;-), > but in the case of a markup language, pushing through without > a specification is obviously asking for trouble. > > With common commercial development, there is very > seldom a specification in any useful sense of the word. What > there is, is a customer's more or less vague idea of what they > want, and the reality any implementation of their idea will > eventually face. > > In such situations, the tests *do* become the specification, > because a specification *cannot* be written! If you try to write > one anyway, it will seriously delay the project, and be obsolete > as soon as it's finished, if it ever had a chance of being correct > (it probably didn't). > > In other words, don't blame the designer who creates a test > suite in those cases, rather than brow-beating the poor customer > until they come up with a workable specification. The practice > of using cue cards to write user stories (and the acceptance > criteria on the back) evolved very much because it was a > workable way of discussing the requirements with users, > whether or not they had any engineering or science background. > > http://www.allaboutagile.com/user-story-example/ > > It seems ridiculously vague and limited, but in practice, it actually > helps to catch *some* of the information needed to arrive at some > mutually understood acceptance criteria. This wasn't created > because people didn't understand what formal specifications, > or thorough testing, looked like - it was created in order to achieve > at least *some* level of improvement with *real* projects and > real customers. > > Besides, when did you last see a good specification of how > a Web GUI should behave - or a good way to test it? * > > Another example: When I gave an invited talk at NWPT'99 [1] in > Uppsala, Sweden, a member of the audience (probably a PdD > student), reacted to my description of the 3-month test period > before release of the AXD 301, saying "That's ridiculous! If you > had written it using a language that could be formally verified, > you wouldn't have to test at all!". > > I mumbled something about not knowing if that would have been > at all possible, and Thomas Arts jumped in and offered to argue > the point with him in the break. It was a very vivid discussion, as I > recall. > > [1] > http://www.it.uu.se/research/publications/reports/1999-008/nwpt99/proceedings/ > > Obviously at that scale (for all except for that PhD student), there > is no chance of arriving at a coherent formal specification, since > the standards we were supposed to comply with were not > themselves formally specified**, and many of the customers didn't > necessarily want *exactly* what was in the standard anyway. > Furthermore, many of the non-functional requirements were not > specified at all, except as a more or less vague list of "goodness > criteria", trying to encapsulate what it means for a system to be > robust. > > What we ended up with was a large pile of informal requirements, > and a huge set of test cases (nearly a million lines of code). > At that point, the test suites *were* indeed the real specification, > albeit an extremely bulky and inaccessible one. > > BR, > Ulf W > > * One of the best is probably Cucumber (http://cukes.info/), which > is based on cue cards, and relies on quite an ambitious mapping > to the underlying web stack (e.g. Rails). Without such a mapping, > it is less useful, although it still has its sweet spots. > > ** ?although some would disagree. They are certainly specified in > a bureaucratic way, but that's not the same thing. > > Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. > http://feuerlabs.com > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.sovetov@REDACTED Fri Jul 13 15:26:46 2012 From: victor.sovetov@REDACTED (Viktor Sovietov) Date: Fri, 13 Jul 2012 06:26:46 -0700 (PDT) Subject: [erlang-questions] Ling: Erlang over Xen, no OS required In-Reply-To: References: <9b565382-35da-472d-8315-0a8d7b35ace8@n5g2000vbb.googlegroups.com> Message-ID: On 13 ???, 15:20, JD Bothma wrote: > Probably a silly question but... > > How does networking work without an OS? Do you just assign an > interface in Xen that Ling knows how to manage directly? Is Ling Both TCP/IP stack and filesystem are accessible by 9P protocol over Xen's virtual Ethernet. It's quite efficient, since it uses zero-copy scheme to pass pages to dom0. In dom0 we run resource re-export server that maps network and filesystems to Ling instances. That part is still far away from being optimal, but it'll improve in near future. > 'basically' the standard VM with this and other facilities that the OS > would provide hacked on? Ling uses only basic mechanisms of resource management which are necessary to compile and run Erlang code. All of the are much easier than their analogs in major OSes, because there is no need to share resources with other processes. Everything which can be externalized is served by dom0 services and the hypervisor itself. > > Why does Ling mean "no more out of memory crashes"? How does Ling deal > with using up its allocated memory? It's well-known BEAM's disadvantage to behave inadequately when OOM happened. Ling has predictable behavior when all memory is gone and able to sort out things even in such tough conditions. > > Sounds cool! Is it open source? That's still under consideration. In any case, binaries will be accessible soon. > > JD > > On 13 July 2012 09:30, Viktor Sovietov wrote:> Ling is highly compatible (with R15B) Erlang VM that is capable to run > > over bare virtual iron on Xen hypervisor without the need in > > underlying operating system. > > > Learn more on :http://erlangonxen.org/ > > > There are similar projects for OCaml ?http://www.openmirage.org/ > > and Haskellhttp://halvm.org/ > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questi...@REDACTED > >http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questi...@REDACTED://erlang.org/mailman/listinfo/erlang-questions From wojtek@REDACTED Fri Jul 13 15:41:53 2012 From: wojtek@REDACTED (=?GB2312?B?V29qdGVrIE5hcmN6eai9c2tp?=) Date: Fri, 13 Jul 2012 15:41:53 +0200 Subject: [erlang-questions] Improve performance of IO bounded server written in Erlang via having pollset for each scheduler and bind port to scheduler together with process In-Reply-To: References: <771ADB6B-9CD0-4A7A-A8A1-EC551B84BAD1@gmail.com> <0E26E896-FF0D-4036-A51A-8F3B8C1FA31C@gmail.com> <4FFF0D25.7050601@power.com.pl> Message-ID: <500025A1.7040201@power.com.pl> On 07/13/2012 04:10 AM, Wei Cao wrote: > I think periodly calls to erts_deliver_time() is necessary.Because at > first I tried to comment out the call to erts_deliver_time() from > erts_check_io(), it would cause erlang VM hang out. Well, that's hard evidence. > erts_deliver_time() does more than erts_time_remaining(), it updates > a clock counter named do_time which is used inside timer wheel. > Okay, I see that now. Maybe it would be enough to call erts_deliver_time() only if do_wait is true? Or if that fails, create a new mutex, say erts_timedelivery_mtx, use erts_smp_mtx_trylock() on it, and only proceed with the time delivery, from the thread that succeeds in obtaining the lock. This should keep do_time relatively fresh. Hope you don't mind me throwing my silly ideas at you. That is because I am excited about your great work. I would like erlang in sockets heavy pumping applications to outperform verything. I think, I'll try to build and test otp on my own, too. After all, how hard can it be ;-) --Regards, Wojtek Narczynski From anders.nygren@REDACTED Fri Jul 13 16:51:52 2012 From: anders.nygren@REDACTED (Anders Nygren) Date: Fri, 13 Jul 2012 09:51:52 -0500 Subject: [erlang-questions] lists, binaries or something else? In-Reply-To: References: Message-ID: For some cases appending to a binary is much faster, see http://www.erlang.org/doc/efficiency_guide/binaryhandling.html#id65923 So Your simple benchmark is not fair, You are comparing the best case for lists with the worst case for binaries. If You change it to appending instead of prepending You will get very different results. So as one of the first replies said "it depends". On my machine: For small N prepending to a list is ~2x faster than appending to a binary. When N=1E6 appending to a binary is ~2x faster than prepending to a list. /Anders On Fri, Jul 13, 2012 at 6:48 AM, CGS wrote: > Hi Bob, > > On Fri, Jul 13, 2012 at 2:32 AM, Bob Ippolito wrote: >> >> Sorry, but that's not enough information. > > > Sorry, but I didn't know what you meant by that. Here are the answers. > >> >> >> Where do these chunks come from: source code, some other process, ets? > > > Mostly from other processes. > >> >> How many chunks are in a string? > > > That is computed from the string size and chunk size (to be decided later). > >> >> Do you compose strings out of other strings, or just these chunks? > > > Just chunks inserted into existing/new strings. > >> >> Are you constructing them from tail to head like you would a list? > > > Unfortunately, not all the time. > >> >> Is the string constructed all at once, or over some time? > > > If you mean by that the string will be fully given in the same message or > whatever by other processes, the answer is no. Over some time may be the > answer, but with the remark that I have no idea what means "over some time" > as period of time (can get chunks one after the other for the same string or > for different strings). > >> >> >> It sounds like you have some code that's doing something with these >> strings, because you say that it's faster to use lists than binaries. Maybe >> if you post whatever it is you used to determine that, someone can help you >> find a better algorithm and/or data structure for that benchmark. > > > That is simple. For benchmark I just took two simple loops (only for > insertion): > > -export([loop_list/1, loop_binary/2]). > > loop_list(0) -> []; > loop_list(N) -> [107 | loop_list(N)]. > > loop_binary(0,B) -> B; > loop_binary(N,B) -> loop_binary(N-1,<<107,B/binary>>). > > If you go in powers of tens and measure the execution time, you will see the > difference (you will also notice the drop in efficiency for binaries when > you need fragmentation for the same string, which is not visible in the case > of lists - or at least I couldn't notice). That is not a fully conclusive > test for my case, but it is quite a conclusive test for processing speed in > the two cases (simple expansion of the string by adding a char at the > beginning of the string), in which, for a 10^5 chars string, the list gains > at least one order of magnitude in processing time than its binary > counterpart (1) (2) (3). > > (1) Even if you change the list loop to have an accumulator in which the > list exists, there is still one order of magnitude difference. > (2) The results are specific for the machine on which the tests are done. > (3) That is just an example. > > CGS > > >> >> >> >> On Thursday, July 12, 2012, CGS wrote: >>> >>> Unfortunately, the project is still in the planning stage, so, no real >>> code was written yet. Nevertheless, I plan some open source projects for >>> some parts of the project. >>> >>> About each string, it is constructed from chunks of fixed size, usually, >>> much smaller than the string itself, hopefully. >>> >>> >>> >>> On Thu, Jul 12, 2012 at 7:29 PM, Bob Ippolito wrote: >>> >>> It would be helpful if you were a lot more specific about how these >>> strings are constructed and what the strings mean. Maybe if you shared some >>> of the code, you'd get better guidance. >>> >>> >>> On Thu, Jul 12, 2012 at 9:06 AM, CGS wrote: >>> >>> >>> Hi Joe, >>> >>> The main problem is to find out which strings are read-only and which >>> strings are read-write, and that requires an algorithm for itself >>> (processing time and extra space - I don't know how negligible are at this >>> moment) as I don't know from before which string will be used more >>> frequently and which less frequently. The second problem is I would like to >>> minimize the harddisk usage, so, to try to store as much information as >>> possible in RAM, but without slowing down the overall process. I know, I am >>> an idealist. :) >>> >>> I thought also about working with lists and keep them as binaries when I >>> don't use them, but, as I said before, that implies a lot of garbage to >>> collect which either can be collected immediately after invoking >>> list_to_binary/1, either allowing GC to appear naturally when there is >>> insufficient memory, or to invoke it at certain moments (either at regular >>> interval of time or based on a scheduler triggered by the application >>> usage). I am afraid that all may be quite inefficient, but they may work >>> faster than processing binaries directly. That I have no idea yet. That's >>> why I am asking here for opinions. >>> >>> Nevertheless, I didn't think of trying to split the strings in two >>> categories: read-only and read-write. That definitely is something I should >>> take into account. >>> >>> Thanks a lot for your thoughts and shared experience. >>> >>> Cheers, >>> CGS >>> >>> >>> >>> >>> On Thu, Jul 12, 2012 at 5:17 PM, Joe Armstrong wrote: >>> >>> As you point out list processing is faster than binary processing. >>> >>> I'd keep things as lists as long as possible until you run into memory >>> problems >>> If you plot the number of strings against response times (or whatever) >>> you should see a sudden decrease in performance when you start paging. >>> At that point you have too much in memory - You could turn the oldest >>> strings >>> into binaries to save space. >>> >>> I generally keep string as lists when I'm working on them and turn >>> them into binaries >>> when I'm finished - sometimes even compressed binaries. >>> >>> Then it depends on the access patterns on the strings - random >>> read-write access is horrible >>> if you can split them into a read-only part and a write-part, you >>> could keep the read-only bit >>> as a binary and the writable bit as a list. >>> >>> It's worth spending a lot of effort to save a single disk access. Then >>> it depends what you do with your strings. If you have a solid state >>> disk and want read only access to the strings >>> then you could store them on disk - or at least arrange so that the >>> constant parts of the strings >>> are on disk and the variable parts in memory. SSDs are about 10 times >>> slower than RAM for reading and usually have multiple controllers so >>> can be very fast - but you need to think a bit first. >>> >>> I'd start with a few measurements, try to stress the system and see >>> where things go wrong. >>> Plot the results - it's usually easy to see when things go wrong. >>> >>> Cheers >>> >>> /Joe >>> >>> >>> >>> On Thu, Jul 12, 2012 at 2:46 PM, CGS wrote: >>> > Hi, >>> > >>> > I am trying to find a balance in between processing speed and RAM >>> > consumption for sets > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From cgsmcmlxxv@REDACTED Fri Jul 13 17:57:17 2012 From: cgsmcmlxxv@REDACTED (CGS) Date: Fri, 13 Jul 2012 17:57:17 +0200 Subject: [erlang-questions] lists, binaries or something else? In-Reply-To: References: Message-ID: Yes, you are right. When I did the test, I thought also about that, so, I made a third loop which should have appended the char to the binary string. Unfortunately, I had a typo there and I ended up having the same loop for binaries (and therefore the same numbers). Your message made me check it again and I found that typo. I didn't find quite those ratios you got, but, yes, they are comparable with your ratios. Thanks a lot. On Fri, Jul 13, 2012 at 4:51 PM, Anders Nygren wrote: > For some cases appending to a binary is much faster, see > http://www.erlang.org/doc/efficiency_guide/binaryhandling.html#id65923 > > So Your simple benchmark is not fair, You are comparing the best case > for lists with the worst case for binaries. If You change it to > appending instead of prepending You will get very different results. > So as one of the first replies said "it depends". > > On my machine: > For small N prepending to a list is ~2x faster than appending to a binary. > When N=1E6 appending to a binary is ~2x faster than prepending to a list. > > /Anders > > > On Fri, Jul 13, 2012 at 6:48 AM, CGS wrote: > > Hi Bob, > > > > On Fri, Jul 13, 2012 at 2:32 AM, Bob Ippolito wrote: > >> > >> Sorry, but that's not enough information. > > > > > > Sorry, but I didn't know what you meant by that. Here are the answers. > > > >> > >> > >> Where do these chunks come from: source code, some other process, ets? > > > > > > Mostly from other processes. > > > >> > >> How many chunks are in a string? > > > > > > That is computed from the string size and chunk size (to be decided > later). > > > >> > >> Do you compose strings out of other strings, or just these chunks? > > > > > > Just chunks inserted into existing/new strings. > > > >> > >> Are you constructing them from tail to head like you would a list? > > > > > > Unfortunately, not all the time. > > > >> > >> Is the string constructed all at once, or over some time? > > > > > > If you mean by that the string will be fully given in the same message or > > whatever by other processes, the answer is no. Over some time may be the > > answer, but with the remark that I have no idea what means "over some > time" > > as period of time (can get chunks one after the other for the same > string or > > for different strings). > > > >> > >> > >> It sounds like you have some code that's doing something with these > >> strings, because you say that it's faster to use lists than binaries. > Maybe > >> if you post whatever it is you used to determine that, someone can help > you > >> find a better algorithm and/or data structure for that benchmark. > > > > > > That is simple. For benchmark I just took two simple loops (only for > > insertion): > > > > -export([loop_list/1, loop_binary/2]). > > > > loop_list(0) -> []; > > loop_list(N) -> [107 | loop_list(N)]. > > > > loop_binary(0,B) -> B; > > loop_binary(N,B) -> loop_binary(N-1,<<107,B/binary>>). > > > > If you go in powers of tens and measure the execution time, you will see > the > > difference (you will also notice the drop in efficiency for binaries when > > you need fragmentation for the same string, which is not visible in the > case > > of lists - or at least I couldn't notice). That is not a fully conclusive > > test for my case, but it is quite a conclusive test for processing speed > in > > the two cases (simple expansion of the string by adding a char at the > > beginning of the string), in which, for a 10^5 chars string, the list > gains > > at least one order of magnitude in processing time than its binary > > counterpart (1) (2) (3). > > > > (1) Even if you change the list loop to have an accumulator in which the > > list exists, there is still one order of magnitude difference. > > (2) The results are specific for the machine on which the tests are done. > > (3) That is just an example. > > > > CGS > > > > > >> > >> > >> > >> On Thursday, July 12, 2012, CGS wrote: > >>> > >>> Unfortunately, the project is still in the planning stage, so, no real > >>> code was written yet. Nevertheless, I plan some open source projects > for > >>> some parts of the project. > >>> > >>> About each string, it is constructed from chunks of fixed size, > usually, > >>> much smaller than the string itself, hopefully. > >>> > >>> > >>> > >>> On Thu, Jul 12, 2012 at 7:29 PM, Bob Ippolito wrote: > >>> > >>> It would be helpful if you were a lot more specific about how these > >>> strings are constructed and what the strings mean. Maybe if you shared > some > >>> of the code, you'd get better guidance. > >>> > >>> > >>> On Thu, Jul 12, 2012 at 9:06 AM, CGS wrote: > >>> > >>> > >>> Hi Joe, > >>> > >>> The main problem is to find out which strings are read-only and which > >>> strings are read-write, and that requires an algorithm for itself > >>> (processing time and extra space - I don't know how negligible are at > this > >>> moment) as I don't know from before which string will be used more > >>> frequently and which less frequently. The second problem is I would > like to > >>> minimize the harddisk usage, so, to try to store as much information as > >>> possible in RAM, but without slowing down the overall process. I know, > I am > >>> an idealist. :) > >>> > >>> I thought also about working with lists and keep them as binaries when > I > >>> don't use them, but, as I said before, that implies a lot of garbage to > >>> collect which either can be collected immediately after invoking > >>> list_to_binary/1, either allowing GC to appear naturally when there is > >>> insufficient memory, or to invoke it at certain moments (either at > regular > >>> interval of time or based on a scheduler triggered by the application > >>> usage). I am afraid that all may be quite inefficient, but they may > work > >>> faster than processing binaries directly. That I have no idea yet. > That's > >>> why I am asking here for opinions. > >>> > >>> Nevertheless, I didn't think of trying to split the strings in two > >>> categories: read-only and read-write. That definitely is something I > should > >>> take into account. > >>> > >>> Thanks a lot for your thoughts and shared experience. > >>> > >>> Cheers, > >>> CGS > >>> > >>> > >>> > >>> > >>> On Thu, Jul 12, 2012 at 5:17 PM, Joe Armstrong > wrote: > >>> > >>> As you point out list processing is faster than binary processing. > >>> > >>> I'd keep things as lists as long as possible until you run into memory > >>> problems > >>> If you plot the number of strings against response times (or whatever) > >>> you should see a sudden decrease in performance when you start paging. > >>> At that point you have too much in memory - You could turn the oldest > >>> strings > >>> into binaries to save space. > >>> > >>> I generally keep string as lists when I'm working on them and turn > >>> them into binaries > >>> when I'm finished - sometimes even compressed binaries. > >>> > >>> Then it depends on the access patterns on the strings - random > >>> read-write access is horrible > >>> if you can split them into a read-only part and a write-part, you > >>> could keep the read-only bit > >>> as a binary and the writable bit as a list. > >>> > >>> It's worth spending a lot of effort to save a single disk access. Then > >>> it depends what you do with your strings. If you have a solid state > >>> disk and want read only access to the strings > >>> then you could store them on disk - or at least arrange so that the > >>> constant parts of the strings > >>> are on disk and the variable parts in memory. SSDs are about 10 times > >>> slower than RAM for reading and usually have multiple controllers so > >>> can be very fast - but you need to think a bit first. > >>> > >>> I'd start with a few measurements, try to stress the system and see > >>> where things go wrong. > >>> Plot the results - it's usually easy to see when things go wrong. > >>> > >>> Cheers > >>> > >>> /Joe > >>> > >>> > >>> > >>> On Thu, Jul 12, 2012 at 2:46 PM, CGS wrote: > >>> > Hi, > >>> > > >>> > I am trying to find a balance in between processing speed and RAM > >>> > consumption for sets > > > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eriksoe@REDACTED Fri Jul 13 18:42:51 2012 From: eriksoe@REDACTED (=?ISO-8859-1?Q?Erik_S=F8e_S=F8rensen?=) Date: Fri, 13 Jul 2012 18:42:51 +0200 Subject: [erlang-questions] Lockfree now() In-Reply-To: <50000DE1.3030506@power.com.pl> References: <4FFF048C.5000500@power.com.pl> <4FFF0EC9.5080302@power.com.pl> <4FFF1C48.4050702@power.com.pl> <4FFF66F4.5000901@power.com.pl> <50000DE1.3030506@power.com.pl> Message-ID: Den 13/07/2012 14.00 skrev "Wojtek Narczy?ski" : > > On 07/13/2012 10:35 AM, Erik S?e S?rensen wrote: >> >> Oh, but you do have a choice between more than one time source in Erlang: >> now() - as known, for microsecond-precision and uniqueness >> erlang:universaltime() and friends - for second-precision and YMD-HMS format. >> os:timestamp() - for microsecond-precision but without uniqueness and time correction. >> > > I didn't know that. Not that I ever looked thoroughly... I used to wonder if it existed... until someone told me. It's probably just not a module you start looking in for that kind of functionality (at least when you already know about now()). I guess a crosslink in the module docs would be an idea? >> For many places where you could use now() but simply need a timestamp for logging or the like, os:timestamp() might be more appropriate and a bit more lightweight. Its return type is compatible with that of now(). > > > I just replaced calendar:now_to_datetime(erlang:now()) with erlang:universaltime() in logging code and gained 13% performance. And that machine is only dual-core. Nice! :-) (Topper: That's nothing, though - try using gregorian-seconds-to-datetime on a huge number, like MAX_DOUBLE - that can be quite slow...) Time conversion is one of those things you don't want to do more than strictly necessary. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bob@REDACTED Fri Jul 13 18:43:25 2012 From: bob@REDACTED (Bob Ippolito) Date: Fri, 13 Jul 2012 09:43:25 -0700 Subject: [erlang-questions] lists, binaries or something else? In-Reply-To: References: Message-ID: If the chunks are big enough you will benefit from sending them as binary since there won't be any copying. As others have already said, you probably will benefit most from either an iolist or all binary scheme. Benchmarking adding characters at a time isn't really equivalent to adding chunks at a time, especially for large chunk sizes. Of course, adding a chunk (or even a list if chunks) to either side of an iolist is O(1), so if you're mostly just building these things you're really better off leaving it like that as long as possible. On Friday, July 13, 2012, CGS wrote: > Yes, you are right. When I did the test, I thought also about that, so, I > made a third loop which should have appended the char to the binary string. > Unfortunately, I had a typo there and I ended up having the same loop for > binaries (and therefore the same numbers). Your message made me check it > again and I found that typo. I didn't find quite those ratios you got, but, > yes, they are comparable with your ratios. Thanks a lot. > > > > On Fri, Jul 13, 2012 at 4:51 PM, Anders Nygren wrote: > > For some cases appending to a binary is much faster, see > http://www.erlang.org/doc/efficiency_guide/binaryhandling.html#id65923 > > So Your simple benchmark is not fair, You are comparing the best case > for lists with the worst case for binaries. If You change it to > appending instead of prepending You will get very different results. > So as one of the first replies said "it depends". > > On my machine: > For small N prepending to a list is ~2x faster than appending to a binary. > When N=1E6 appending to a binary is ~2x faster than prepending to a list. > > /Anders > > > On Fri, Jul 13, 2012 at 6:48 AM, CGS wrote: > > Hi Bob, > > > > On Fri, Jul 13, 2012 at 2:32 AM, Bob Ippolito wrote: > >> > >> Sorry, but that's not enough information. > > > > > > Sorry, but I didn't know what you meant by that. Here are the answers. > > > >> > >> > >> Where do these chunks come from: source code, some other process, ets? > > > > > > Mostly from other processes. > > > >> > >> How many chunks are in a string? > > > > > > That is computed from the string size and chunk size (to be decided > later). > > > >> > >> Do you compose strings out of other strings, or just these chunks? > > > > > > Just chunks inserted into existing/new strings. > > > >> > >> Are you constructing them from tail to head like you would a list? > > > > > > Unfortunately, not all the time. > > > >> > >> Is the string constructed all at once, or over some time? > > > > > > If you mean by that the string will be fully given in the same message or > > whatever by other processes, the answer is no. Over some time may be the > > answer, but with the remark that I have no idea what means "over some > time" > > as period of time (can get chunks one after the other for the same > string or > > for different strings). > > > >> > >> > >> It sounds like you have some code that's doing something with these > >> strings, because you say that it's faster to use lists than binaries. > Maybe > >> if you post whatever it is you used to determine that, someone can help > you > >> find a better algorithm and/or data structure for that benchmark. > > > > > > That is simple. For benchmark I just took two simple loops (only for > > insertion): > > > > -export([loop_list/1, loop_binary/2]). > > > > loop_list(0) -> []; > > loop_list(N) -> [107 | loop_list(N)]. > > > > loop_binary(0,B) -> B; > > loop_binary(N,B) -> loop_binary(N-1,<<107,B/binary>>). > > > > If you go in powers of tens and measure the execution time, you will see > the > > difference (you will also notice the drop in efficiency for binaries when > > you need fragmentation for the same string, which is not visible in the > case > > of lists - or at least I couldn't notice). That is not a fully conclusive > > test for my case, but it is quite a conclusive test for processing speed > in > > the two cases (simple expansion of the string by adding a char at the > > beginning of the string), in which, for a 10^5 chars string, the list > gains > > at least one order of magnitude in processing time than its binary > > counterpart (1) (2) (3). > > > > (1) Even if you change the list loop to have an accumulator in which the > > list exists, there is still one order of magnitude difference. > > (2) The results are specific for the machine on which the tests are done. > > (3) That is just an example. > > > > CGS > > > > > >> > >> > >> > >> On Thursday, July 12, 2012, CGS wrote: > >>> > >>> Unfortunately, the project is still in the planning stage, so, no real > >>> code was written yet. Nevertheless, I plan some open source projects > for > >>> some parts of the project. > >>> > >>> About each string, it is constructed from chunks of fixe > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.joseph.davis@REDACTED Fri Jul 13 20:29:36 2012 From: paul.joseph.davis@REDACTED (Paul Davis) Date: Fri, 13 Jul 2012 13:29:36 -0500 Subject: [erlang-questions] Selective receive optimization for NIFs In-Reply-To: References: <4FFAA4C7.4040800@erix.ericsson.se> Message-ID: Just to round this off I've managed to hack in the recv_set/recv_mark instructions by editing the Erlang assembler output by hand and then building the beam from there. The basic steps are: 1> compile:file("filename.erl", 'S'). % Edit filename.S to add recv_set/recv_mark instructions 2> compile:file("filename.S", [asm, no_postopt]). Obviously this is horribly dangerous for a whole host of reasons but it'll work for the short term. At some point I'll have to try diving into the compiler code again to see if I can't figure out how to make this work automatically. On Mon, Jul 9, 2012 at 2:07 PM, Paul Davis wrote: > On Mon, Jul 9, 2012 at 9:05 AM, Dave Smith wrote: >> In the past, I think I solved this problem on a NIF by just passing >> the ref in. It's a bit more of a pain, but should do the trick. >> >> D. >> > > That was the method I tried first and it didn't have any effect. I'll > spend some time reading the generated beam code for both versions to > see if I can't figure out a way around it. > >> On Mon, Jul 9, 2012 at 7:52 AM, Paul Davis wrote: >>> Sverker, >>> >>> Ah! That would explain a lot of my confusion. I spent a lot of time >>> reading through code thinking the magic was that refs were tagged with >>> when they were created or had some special flag when they got copied >>> out of a process. If I have time later this week I'll take some time >>> to start reading through decompiled beam to try and get a better >>> understanding and see what I can come up with. >>> >>> Thanks, >>> Paul >>> >>> On Mon, Jul 9, 2012 at 4:30 AM, Sverker Eriksson >>> wrote: >>>> Hi Paul >>>> >>>> The OTP-8623 selective receive optimization was achieved partly with static >>>> code analysis done by the Erlang compiler. >>>> >>>> In your case the NIF is creating the ref. To do the same optimization for >>>> that, would need some sort of dynamic analysis done in runtime. >>>> >>>> >>>> /Sverker, Erlang/OTP >>>> >>>> >>>> >>>> Paul Davis wrote: >>>>> >>>>> I've got a bit of a curious question in relation to OTP-8623 (which is >>>>> the selective receive for a newly created ref skips messages in the >>>>> inbox). Apparently this doesn't appear to apply to messages generated >>>>> from NIFs. Or at least, not in the code I've got. >>>>> >>>>> I did spend some time trawling around erts internals but the few >>>>> references I found weren't enough for the whole thing to click. I'll >>>>> paste the sample test code I've been using in case I'm doing something >>>>> supremely silly. >>>>> >>>>> Thanks, >>>>> Paul Davis >>>>> >>>>> https://gist.github.com/3073295 >>>>> _______________________________________________ >>>>> erlang-questions mailing list >>>>> erlang-questions@REDACTED >>>>> http://erlang.org/mailman/listinfo/erlang-questions >>>>> >>>>> >>>> >>>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions From dmkolesnikov@REDACTED Fri Jul 13 20:44:23 2012 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Fri, 13 Jul 2012 21:44:23 +0300 Subject: [erlang-questions] sasl, lager or something else? Message-ID: <1704D579-9127-4E65-8B78-248E5F94EB20@gmail.com> Hello, I've been using SASL for error logger within applications. Now, I need to adjust them for hosting by non-erlang folks. It means that I have to adjust logging... Multiple alternatives exists: a) make text extension for SASL b) use lagger c) use some other frameworks: alogger, log4erl, etc I am very closed to jump into option b). What is you opinion about lagger? Are there any better alternatives? Thanks in advanced! - Dmitry From roe.adrian@REDACTED Fri Jul 13 21:16:52 2012 From: roe.adrian@REDACTED (Adrian Roe) Date: Fri, 13 Jul 2012 20:16:52 +0100 Subject: [erlang-questions] sasl, lager or something else? In-Reply-To: <1704D579-9127-4E65-8B78-248E5F94EB20@gmail.com> References: <1704D579-9127-4E65-8B78-248E5F94EB20@gmail.com> Message-ID: <50007424.5070208@gmail.com> Lager gets a big thumbs up from me - low(ish) learning curve and for us at least "it just works". Even writing a custom logger is straightforward if the default doesn't do quite what you want. Do read the docs pretty thoroughly up front to get familiar with its config etc and make sure lager is the first thing that is compiled in your project (or the lager parse transforms will fail in modules that try to use it - rebar (and others I'm sure) has easy support for this). The way it can trap error_logger calls is nice - meaning you can get usefully up an running without having to first change all of your logging code. Adrian *Director id3as * On 13/07/12 19:44, Dmitry Kolesnikov wrote: > Hello, > > I've been using SASL for error logger within applications. > Now, I need to adjust them for hosting by non-erlang folks. It means that I have to adjust logging... > Multiple alternatives exists: > a) make text extension for SASL > b) use lagger > c) use some other frameworks: alogger, log4erl, etc > > I am very closed to jump into option b). > What is you opinion about lagger? > Are there any better alternatives? > > Thanks in advanced! > > - Dmitry > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From mrtndimitrov@REDACTED Fri Jul 13 22:00:26 2012 From: mrtndimitrov@REDACTED (Martin Dimitrov) Date: Fri, 13 Jul 2012 23:00:26 +0300 Subject: [erlang-questions] XSL transformations Message-ID: <50007E5A.9000903@gmail.com> Is it possible to do XSL transformations with xmerl or is there other Erlang library capable of doing this? Thanks, Martin From freza@REDACTED Fri Jul 13 23:08:33 2012 From: freza@REDACTED (freza@REDACTED) Date: Fri, 13 Jul 2012 17:08:33 -0400 Subject: [erlang-questions] Ling: Erlang over Xen, no OS required In-Reply-To: <9b565382-35da-472d-8315-0a8d7b35ace8@n5g2000vbb.googlegroups.com> References: <9b565382-35da-472d-8315-0a8d7b35ace8@n5g2000vbb.googlegroups.com> Message-ID: <20120713210833.GA16632@circlewave.net> On Fri, Jul 13, 2012 at 12:30:41AM -0700, Viktor Sovietov wrote: > Ling is highly compatible (with R15B) Erlang VM that is capable to run > over bare virtual iron on Xen hypervisor without the need in > underlying operating system. > > Learn more on : http://erlangonxen.org/ Interesting -- it's looking like it's not a port of BEAM? Is source code available somewhere? How do you handle TCP/IP networking + persistent storage without an OS? BR, -- Jachym From dialtone@REDACTED Fri Jul 13 23:38:08 2012 From: dialtone@REDACTED (Valentino Volonghi) Date: Fri, 13 Jul 2012 14:38:08 -0700 Subject: [erlang-questions] Base64 decode binaries with ets bug? Message-ID: If you take the code at the bottom and run it you'll get the following outputs: 1> wtf:bad(). Db: [{{test,<<"aHR0cDovL2hvb3RzdWl0ZS5jb20vYy9wcm8tYWRyb2xsLWFi">>}, <<"/usr/local/Cellar/erlang/R15B01/lib/">>}] ok 2> wtf:good(). Value: <<"http://hootsuite.com/c/pro-adroll-ab">> Db: [{{test,<<"aHR0cDovL2hvb3RzdWl0ZS5jb20vYy9wcm8tYWRyb2xsLWFi">>}, <<"http://hootsuite.com/c/pro-adroll-ab">>}] ok 3> wtf:good2(). Db: [{{test,<<"aHR0cDovL2hvb3RzdWl0ZS5jb20vYy9wcm8tYWRyb2xsLWFi">>}, "http://hootsuite.com/c/pro-adroll-ab"}] ok 4> wtf:good3(). Db: [{{test,<<"aHR0cDovL2hvb3RzdWl0ZS5jb20vYy9wcm8tYWRyb2xsLWFi">>}, <<"http://hootsuite.com/c/pro-adroll-ab">>}] What's funny is that this doesn't appear to break when you take out the last 8 characters in the base64 encoded string so: These 2 don't work: aHR0cDovL2hvb3RzdWl0ZS5jb20vYy9wcm8tYWRyb2xsLWFi aHR0cDovL2hvb3RzdWl0ZS5jb20vYy9wcm8tYWRyb2xs This works: aHR0cDovL2hvb3RzdWl0ZS5jb20vYy9wcm8tYWRy Are we doing something that we shouldn't or is this a bug somewhere in the VM/library/ets? % % Code that exposes the problem from here down. % -module(wtf). -compile(export_all). %% %% 1) base64:decode %% 2) ets:insert %% bad() -> Key = {test, <<"aHR0cDovL2hvb3RzdWl0ZS5jb20vYy9wcm8tYWRyb2xsLWFi">>}, Value = base64:decode(<<"aHR0cDovL2hvb3RzdWl0ZS5jb20vYy9wcm8tYWRyb2xsLWFi">>), Db = ets:new(undefined, [set, protected, {read_concurrency, true}, compressed]), ets:insert(Db, {Key, Value}), io:format("Db: ~p~n", [ets:lookup(Db, Key)]). %% %% 1) base64:decode_to_string %% 2) io:format %% 3) ets:insert %% good() -> Key = {test, <<"aHR0cDovL2hvb3RzdWl0ZS5jb20vYy9wcm8tYWRyb2xsLWFi">>}, Value = base64:decode(<<"aHR0cDovL2hvb3RzdWl0ZS5jb20vYy9wcm8tYWRyb2xsLWFi">>), io:format("Value: ~p~n", [Value]), Db = ets:new(undefined, [set, protected, {read_concurrency, true}, compressed]), ets:insert(Db, {Key, Value}), io:format("Db: ~p~n", [ets:lookup(Db, Key)]). %% %% 1) base64:decode_to_string %% 2) ets:insert %% good2() -> Key = {test, <<"aHR0cDovL2hvb3RzdWl0ZS5jb20vYy9wcm8tYWRyb2xsLWFi">>}, Value = base64:decode_to_string(<<"aHR0cDovL2hvb3RzdWl0ZS5jb20vYy9wcm8tYWRyb2xsLWFi">>), Db = ets:new(undefined, [set, protected, {read_concurrency, true}, compressed]), ets:insert(Db, {Key, Value}), io:format("Db: ~p~n", [ets:lookup(Db, Key)]). %% %% 1) base64:decode_to_string %% 2) list_to_binary %% 3) ets:insert %% good3() -> Key = {test, <<"aHR0cDovL2hvb3RzdWl0ZS5jb20vYy9wcm8tYWRyb2xsLWFi">>}, Value = list_to_binary(base64:decode_to_string(<<"aHR0cDovL2hvb3RzdWl0ZS5jb20vYy9wcm8tYWRyb2xsLWFi">>)), Db = ets:new(undefined, [set, protected, {read_concurrency, true}, compressed]), ets:insert(Db, {Key, Value}), io:format("Db: ~p~n", [ets:lookup(Db, Key)]). -- Valentino Volonghi http://www.adroll.com From dialtone@REDACTED Fri Jul 13 23:46:40 2012 From: dialtone@REDACTED (Valentino Volonghi) Date: Fri, 13 Jul 2012 14:46:40 -0700 Subject: [erlang-questions] Base64 decode binaries with ets bug? In-Reply-To: References: Message-ID: <16814DC1-B287-48A4-A779-A09D28FA42C8@gmail.com> On 13 Jul 2012, at 14:38, Valentino Volonghi wrote: > Are we doing something that we shouldn't or is this a bug somewhere in the VM/library/ets? After investigating a bit more and noticing that sometimes we had padded 0 in front or in the back we played with the ets compression and it appears that it's ets compression that makes this bug pop up, so I guess this is related to ets rather than the rest. -- Valentino Volonghi http://www.adroll.com From jayson.barley@REDACTED Sat Jul 14 04:59:39 2012 From: jayson.barley@REDACTED (Jayson Barley) Date: Fri, 13 Jul 2012 19:59:39 -0700 Subject: [erlang-questions] Framework advise Message-ID: I need to show a POC on a rewrite/redesign of an existing Java/CPP code base. From my work with Erlang I believe it is the best choice for the job. I am seeking some advise on what frameworks would be best to start with that will meet my goals. See below for the details. I am willing to listen to any feedback on which frameworks would be best or even if Erlang isn't the best tool for the job. Requirements 1. Web standards based - Must support HTML 5, JS, CSS, etc. I am thinking 2. Authentication - Users will need to be able to move from one server to another without losing their session. 3. Template-able - I will need to be able to perform CRUD and retrieve data as HTML, XML, JSON, and possibly other formats in the future. 4. Scalable - It must support at least 10,000 requests per second. It could be higher but based on the next requirement I am starting out small. 5. Capable of streaming large amounts of data - It is possible that a REST request could contain millions of rows of data. This is rare but it is a use case that the existing Tomcat/IIS/Java Frankenstein currently does. 6. MS SQL connectivity - Our existing database is MS SQL and that won't be easy to change so it needs to be able to connect to it. 7. Testable - I will need to be able to write and run Unit, Integration tests. I am guessing that eunit and common test will cover this. 8. Easily deployable - A new server will need to be deployed and configured in minutes. Something like Chef would be ideal since we are already using that in other areas. 9. In place upgrades - When a new build is ready to deployed we should be able to deploy it, leave the previous version in place to roll back should an issue be found. 10. Easy logging. I think that is everything I can think of. If anyone has any questions or recommendations on how to make this clearer please let me know. Thank you in advance, Jayson -------------- next part -------------- An HTML attachment was scrubbed... URL: From paperless@REDACTED Sat Jul 14 06:33:57 2012 From: paperless@REDACTED (Tim McNamara) Date: Sat, 14 Jul 2012 16:33:57 +1200 Subject: [erlang-questions] Framework advise In-Reply-To: References: Message-ID: Good luck developing the prototype Jayson. You may want to give some reasons why the Frankenstein middleware isn't performing to get the most helpful responses. Are you just want to creating an authenticated front-end for a database? Without knowning more, I would say that facilitating single sign-on will be the trickest part. Creating a database front end will be pretty easy. You might want to look at Chicago Boss's Boss_DB project (https://github.com/evanmiller/boss_db), which will allow you to easily create an adapter for MS SQL if one doesn't already work for you. On 14 July 2012 14:59, Jayson Barley wrote: > I need to show a POC on a rewrite/redesign of an existing Java/CPP code > base. From my work with Erlang I believe it is the best choice for the job. > I am seeking some advise on what frameworks would be best to start with that > will meet my goals. See below for the details. I am willing to listen to any > feedback on which frameworks would be best or even if Erlang isn't the best > tool for the job. > > Requirements > > Web standards based - Must support HTML 5, JS, CSS, etc. I am thinking > Authentication - Users will need to be able to move from one server to > another without losing their session. > Template-able - I will need to be able to perform CRUD and retrieve data as > HTML, XML, JSON, and possibly other formats in the future. > Scalable - It must support at least 10,000 requests per second. It could be > higher but based on the next requirement I am starting out small. > Capable of streaming large amounts of data - It is possible that a REST > request could contain millions of rows of data. This is rare but it is a use > case that the existing Tomcat/IIS/Java Frankenstein currently does. > MS SQL connectivity - Our existing database is MS SQL and that won't be easy > to change so it needs to be able to connect to it. > Testable - I will need to be able to write and run Unit, Integration tests. > I am guessing that eunit and common test will cover this. > Easily deployable - A new server will need to be deployed and configured in > minutes. Something like Chef would be ideal since we are already using that > in other areas. > In place upgrades - When a new build is ready to deployed we should be able > to deploy it, leave the previous version in place to roll back should an > issue be found. > Easy logging. > > I think that is everything I can think of. If anyone has any questions or > recommendations on how to make this clearer please let me know. > > Thank you in advance, > Jayson > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From baliulia@REDACTED Sat Jul 14 10:25:55 2012 From: baliulia@REDACTED (=?UTF-8?B?SWduYXMgVnnFoW5pYXVza2Fz?=) Date: Sat, 14 Jul 2012 11:25:55 +0300 Subject: [erlang-questions] XSL transformations In-Reply-To: <50007E5A.9000903@gmail.com> References: <50007E5A.9000903@gmail.com> Message-ID: <50012D13.9060707@gmail.com> On 2012.07.13 23:00, Martin Dimitrov wrote: > Is it possible to do XSL transformations with xmerl or is there other > Erlang library capable of doing this? I think xmerl_xs[1] might help you. Some examples here[2]. [1]: http://www.erlang.org/doc/man/xmerl_xs.html [2]: http://erlang.org/doc/apps/xmerl/xmerl_xs_examples.html Ignas From mrtndimitrov@REDACTED Sat Jul 14 13:37:16 2012 From: mrtndimitrov@REDACTED (Martin Dimitrov) Date: Sat, 14 Jul 2012 14:37:16 +0300 Subject: [erlang-questions] XSL transformations In-Reply-To: <50012D13.9060707@gmail.com> References: <50007E5A.9000903@gmail.com> <50012D13.9060707@gmail.com> Message-ID: <500159EC.7030405@gmail.com> What I need is to produce another XML document through the XSL transformation. I really don't see how I can use xmerl_xs to accomplish this. Thanks, Martin On 7/14/2012 11:25 AM, Ignas Vy?niauskas wrote: > On 2012.07.13 23:00, Martin Dimitrov wrote: >> Is it possible to do XSL transformations with xmerl or is there other >> Erlang library capable of doing this? > > I think xmerl_xs[1] might help you. Some examples here[2]. > > [1]: http://www.erlang.org/doc/man/xmerl_xs.html > [2]: http://erlang.org/doc/apps/xmerl/xmerl_xs_examples.html > > Ignas From paul.james.barry@REDACTED Sat Jul 14 14:06:06 2012 From: paul.james.barry@REDACTED (Paul Barry) Date: Sat, 14 Jul 2012 13:06:06 +0100 Subject: [erlang-questions] Framework advise In-Reply-To: References: Message-ID: +1 from me on Chicago Boss. It's worth spending some time with - and I think it hits most of the items on your list (as it is built on Erlang/OTP and "inherits" a lot of the application characteristics that are important to you. Chicago Boss has a very friendly user community and mailing list (just like Erlang's). :-) Paul. P.S. I'm just starting to go through Zachary Kessin's recent book (http://www.amazon.com/Building-Web-Applications-Erlang-Working/dp/1449309968/) which may be worth a look also. It's not an expensive purchase, and touches on some of your issues. On 14 July 2012 05:33, Tim McNamara wrote: > Good luck developing the prototype Jayson. You may want to give some > reasons why the Frankenstein middleware isn't performing to get the > most helpful responses. > > Are you just want to creating an authenticated front-end for a > database? Without knowning more, I would say that facilitating single > sign-on will be the trickest part. Creating a database front end will > be pretty easy. You might want to look at Chicago Boss's Boss_DB > project (https://github.com/evanmiller/boss_db), which will allow you > to easily create an adapter for MS SQL if one doesn't already work for > you. > > > On 14 July 2012 14:59, Jayson Barley wrote: >> I need to show a POC on a rewrite/redesign of an existing Java/CPP code >> base. From my work with Erlang I believe it is the best choice for the job. >> I am seeking some advise on what frameworks would be best to start with that >> will meet my goals. See below for the details. I am willing to listen to any >> feedback on which frameworks would be best or even if Erlang isn't the best >> tool for the job. >> >> Requirements >> >> Web standards based - Must support HTML 5, JS, CSS, etc. I am thinking >> Authentication - Users will need to be able to move from one server to >> another without losing their session. >> Template-able - I will need to be able to perform CRUD and retrieve data as >> HTML, XML, JSON, and possibly other formats in the future. >> Scalable - It must support at least 10,000 requests per second. It could be >> higher but based on the next requirement I am starting out small. >> Capable of streaming large amounts of data - It is possible that a REST >> request could contain millions of rows of data. This is rare but it is a use >> case that the existing Tomcat/IIS/Java Frankenstein currently does. >> MS SQL connectivity - Our existing database is MS SQL and that won't be easy >> to change so it needs to be able to connect to it. >> Testable - I will need to be able to write and run Unit, Integration tests. >> I am guessing that eunit and common test will cover this. >> Easily deployable - A new server will need to be deployed and configured in >> minutes. Something like Chef would be ideal since we are already using that >> in other areas. >> In place upgrades - When a new build is ready to deployed we should be able >> to deploy it, leave the previous version in place to roll back should an >> issue be found. >> Easy logging. >> >> I think that is everything I can think of. If anyone has any questions or >> recommendations on how to make this clearer please let me know. >> >> Thank you in advance, >> Jayson >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- Paul Barry, w: http://paulbarry.itcarlow.ie - e: paul.barry@REDACTED Lecturer, Computer Networking: Institute of Technology, Carlow, Ireland. From gumm@REDACTED Sat Jul 14 15:36:34 2012 From: gumm@REDACTED (Jesse Gumm) Date: Sat, 14 Jul 2012 08:36:34 -0500 Subject: [erlang-questions] Framework advise In-Reply-To: References: Message-ID: As far as Erlang web frameworks go, you're basically gonna have Chicago Boss (MVC), Nitrogen (event driven), and Zotonic (CMS with some event driven stuff). You can also develop a little "closer to the metal" using yaws, cowboy, mochiweb, webmachine, or the new fandangled hybrid mochicow. Both Nitrogen and Boss use simple_bridge to facilitate whichever underlying erlang server you'd want to use, so you can tap into the various streaming capacities of the underlying server of your choice. As I'm familiar with Nitrogen moreso than the others, I can at least address how it can handle your requirements: The multiserver (multi domain?) one is interesting. Nitrogen supports session sharing across a cluster, however it's default identification is with a cookie, so that's worth considering with regard to multiple domains. I can imagine a custom session handler combined with a custom security handler solving the session sharing problem. I may put together a single sign on POC just to satisfy my own curiosity. In place upgrade with a hot swappable rollback sounds more like something you'd want to do by using multiple nodes and a reverse proxy like nginx. As for mssql connectivity goes, nitrogen is db agnostic (provides no direct facilities for db integration), but using the erlang drivers for databases is no problem at all. The easiest method of accessing mssql from erlang might be odbc. With those caveats mentioned, Nitrogen can handle the others just fine. I'm sure Boss and Zotonic would do just as well. Using Chicago Boss would likely require a custom BossDB adapter for mssql. The big difference, however, will be in how you prefer to develop. Spend a few hours tinkering with each and see which development model you prefer. -- Jesse Gumm Owner, Sigma Star Systems 414.940.4866 || sigma-star.com || @jessegumm On Jul 14, 2012 7:06 AM, "Paul Barry" wrote: > +1 from me on Chicago Boss. It's worth spending some time with - and > I think it hits most of the items on your list (as it is built on > Erlang/OTP and "inherits" a lot of the application characteristics > that are important to you. Chicago Boss has a very friendly user > community and mailing list (just like Erlang's). :-) > > Paul. > > P.S. I'm just starting to go through Zachary Kessin's recent book > ( > http://www.amazon.com/Building-Web-Applications-Erlang-Working/dp/1449309968/ > ) > which may be worth a look also. It's not an expensive purchase, and > touches on some of your issues. > > > On 14 July 2012 05:33, Tim McNamara wrote: > > Good luck developing the prototype Jayson. You may want to give some > > reasons why the Frankenstein middleware isn't performing to get the > > most helpful responses. > > > > Are you just want to creating an authenticated front-end for a > > database? Without knowning more, I would say that facilitating single > > sign-on will be the trickest part. Creating a database front end will > > be pretty easy. You might want to look at Chicago Boss's Boss_DB > > project (https://github.com/evanmiller/boss_db), which will allow you > > to easily create an adapter for MS SQL if one doesn't already work for > > you. > > > > > > On 14 July 2012 14:59, Jayson Barley wrote: > >> I need to show a POC on a rewrite/redesign of an existing Java/CPP code > >> base. From my work with Erlang I believe it is the best choice for the > job. > >> I am seeking some advise on what frameworks would be best to start with > that > >> will meet my goals. See below for the details. I am willing to listen > to any > >> feedback on which frameworks would be best or even if Erlang isn't the > best > >> tool for the job. > >> > >> Requirements > >> > >> Web standards based - Must support HTML 5, JS, CSS, etc. I am thinking > >> Authentication - Users will need to be able to move from one server to > >> another without losing their session. > >> Template-able - I will need to be able to perform CRUD and retrieve > data as > >> HTML, XML, JSON, and possibly other formats in the future. > >> Scalable - It must support at least 10,000 requests per second. It > could be > >> higher but based on the next requirement I am starting out small. > >> Capable of streaming large amounts of data - It is possible that a REST > >> request could contain millions of rows of data. This is rare but it is > a use > >> case that the existing Tomcat/IIS/Java Frankenstein currently does. > >> MS SQL connectivity - Our existing database is MS SQL and that won't be > easy > >> to change so it needs to be able to connect to it. > >> Testable - I will need to be able to write and run Unit, Integration > tests. > >> I am guessing that eunit and common test will cover this. > >> Easily deployable - A new server will need to be deployed and > configured in > >> minutes. Something like Chef would be ideal since we are already using > that > >> in other areas. > >> In place upgrades - When a new build is ready to deployed we should be > able > >> to deploy it, leave the previous version in place to roll back should an > >> issue be found. > >> Easy logging. > >> > >> I think that is everything I can think of. If anyone has any questions > or > >> recommendations on how to make this clearer please let me know. > >> > >> Thank you in advance, > >> Jayson > >> > >> > >> _______________________________________________ > >> erlang-questions mailing list > >> erlang-questions@REDACTED > >> http://erlang.org/mailman/listinfo/erlang-questions > >> > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > > -- > Paul Barry, w: http://paulbarry.itcarlow.ie - e: paul.barry@REDACTED > Lecturer, Computer Networking: Institute of Technology, Carlow, Ireland. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From emmiller@REDACTED Sat Jul 14 15:41:13 2012 From: emmiller@REDACTED (Evan Miller) Date: Sat, 14 Jul 2012 09:41:13 -0400 Subject: [erlang-questions] Erlang and bbcode In-Reply-To: <5C229F26-0DB5-4C67-B8C3-434731D0B233@cs.otago.ac.nz> References: <5C229F26-0DB5-4C67-B8C3-434731D0B233@cs.otago.ac.nz> Message-ID: <06E3FEC3-C3A1-4CEE-AB69-1529F0E7317C@gmail.com> https://github.com/evanmiller/jerome It is far from perfect but Jerome's bbcode scanner is 92 lines and the grammar is 74 lines. The intermediate format can then be used to generate HTML, RTF, or Textile Sent from my iPhone On Jul 12, 2012, at 1:37, "Richard O'Keefe" wrote: > On reading the slides about "Erlang sucks" I thought, > "what is bbcode and how hard can it be to write an > Erlang parser for it?" > > Since having that thought, I've checked half a dozen > definitions of bbcode and looked at a parser or two > and am little the wiser. > > BBcode strikes me as truly bizarre. What is the point > of entering something that looks pretty much like HTML > except for using square brackets instead of angle brackets? > But there is worse. > > - I cannot discover whether any particular character set > or encoding is presumed and if so which one. (I'd > *guess* Unicode/UTF-8, but a guess is all it would be.) > - I cannot discover how you get a plain [ into text. > Could it be [[? Could it be [[]? > - I cannot discover exactly what is a well-formed tag and > what is not. > - I cannot discover whether [/*] is legal or not. > - I cannot discover whether markup is legal inside > a [url]...[/url] or not (it could be stripped out) > - Same for [email] and [img] and [youtube] and [gvideo] > - I cannot discover whether [size=n] takes n in points, > pixels, percentage of default, or anything else (it > seems that different systems do different things) > - I cannot discover whether [youtube] and [gvideo] > allow width/height like [img] or not. > - Some descriptions say that :-) is processed as a > smiley, and that other emoticons may be processed > too, but I cannot find a list; others say [:-)] is > a smiley; others say nothing about this. > - It is not clear how the author of [quote-author]... > should be rendered; I have a strong suspicion it > should be locale-dependent. > - It appears that different instances of bbcode support > different tag sets out of the box and most of them > allow some sort of customisation. > - It appears to be _expected_ that different bbcode > implementations will translate things differently > (so [b]xxx[/b] might yield or or > or something else), > which means that it would be hard to make a test suite. > Indeed, I can find no guarantee that [b] [i] and so on > won't just be stripped out. > > If the lexical issues could be sorted out, one could easily > enough write a BBcode -> XML value tree parser, and an > XML -> XML translator to do things like > Y -> Y > Y -> Y > and then use an existing XML -> text unparser. > An non-validating XML parser in Erlang took me 275 lines, > so I doubt bbcode would be much harder. > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From erlang@REDACTED Sat Jul 14 18:27:49 2012 From: erlang@REDACTED (Joe Armstrong) Date: Sat, 14 Jul 2012 18:27:49 +0200 Subject: [erlang-questions] client in erlang server in objective C Message-ID: I want the following: - Client in erlang (easy) - Server in objective C (I'm thinking Mac OS-X lion - xcode 4.2 - cocoa) - socket communication [[ I'm a complete beginner with xcode/objective C ]] Has anybody written a simple objective C client that opens a port waits for a message then executes a callback. This should be non-blocking and be integrated with the "standard" dispatcher top-loop. I'm confused by the profusion of classes that almost do this. Has anybody implemented this, or have pointers to a good place to start. Cheers /Joe From bob@REDACTED Sat Jul 14 18:56:09 2012 From: bob@REDACTED (Bob Ippolito) Date: Sat, 14 Jul 2012 09:56:09 -0700 Subject: [erlang-questions] client in erlang server in objective C In-Reply-To: References: Message-ID: The best way to do this now is to use Grand Central Dispatch. Your listening socket and the connected sockets are just a dispatch sources that can call blocks on a queue when stuff happens. I believe there's a wrapper for this on github called AsyncSocket that abstracts things a bit (on my phone so I don't have all of my notes). On Saturday, July 14, 2012, Joe Armstrong wrote: > I want the following: > > - Client in erlang (easy) > - Server in objective C (I'm thinking Mac OS-X lion - xcode 4.2 - cocoa) > - socket communication > > [[ I'm a complete beginner with xcode/objective C ]] > > Has anybody written a simple objective C client that opens a port > waits for a message then executes a callback. This should be non-blocking > and be integrated with the "standard" dispatcher top-loop. > > I'm confused by the profusion of classes that almost do this. > > Has anybody implemented this, or have pointers to a good place to start. > > Cheers > > /Joe > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.virding@REDACTED Sun Jul 15 03:01:39 2012 From: robert.virding@REDACTED (Robert Virding) Date: Sun, 15 Jul 2012 02:01:39 +0100 (BST) Subject: [erlang-questions] Local namespaces in case/try In-Reply-To: Message-ID: Sorry for not getting into this discussion earlier but I am vegetating in the country (also called being on holidays) at the moment and the my internet connection is not what it should be. Go Joe! I agree with what you and both Richards have written. Just some personal comments: - If you were to introduce a let with variable scoping then you would have to do the same for if/case/receive as well otherwise you would get a very schizophrenic language. One which would definitely be much more confusing for beginners than it is today. It would also mean that let/if/case/receive wouldn't export variables, only return values, which would mean that the original code would be useless anyway. - Even if funs/comprehensions behave differently from the rest their behaviour is quite logical within the erlang context. For a fun to *export* a variable would be completely crazy as it is defined in one place and executed in another; where and when would you see the exported value? For a comprehension which value of a variable should you see? - Having a variable referring to the same value in the whole function clause body maybe different but it is actually very simple to grasp; actually much simpler than having scopes. - As has been pointed out it is not a compiler problem, having scopes is actually easier. Robert ----- Original Message ----- > On Tue, Jul 10, 2012 at 12:00 PM, Dmitry Groshev > wrote: > > Exactly what I was talking about. I can't see any sense in bloating > > the code > > with nonsensical functions just to get something let-like -- AFAIK, > > every > > functional language out there has a local namespace for such > > constructs. > > Moreover, there is a huge inconsistency right now with list > > comprehensions > > that have their own namespace because of their transformation to > > functions: > > > > test1() -> > > [begin > > _A = X, > > ok > > end || X <- [1, 2, 3]], > > _A = 1. > > > > test2() -> > > case 2 of > > _A -> _A > > end, > > _A = 1. > > > > I think it's a sign of a bigger problem with Erlang as a language: > > we have > > an amazing BEAM, some brilliant libraries, a lot of godlike > > developers, but > > Erlang as a language evolves extremely slow. One reason that I can > > think > > about is a lack of ultimate authority that can decide what is right > > and what > > is wrong ? theoretically there is a EEP process (and some proposals > > are out > > of EEPs scope, like an e2 thing), but in practice some EEPs are > > here for > > years (like Frames proposal or JSON BIFs) and nothing really moves > > on. > > No that's not the reason. > > There is no ultimate authority. Change is driven by the interplay of > several factors: > > - consensus (ie general agreement that something is good) > - payment (the people who finance Erlang pay to have something > done) > - voluntary effort (individuals make changes, which are then > adopted) > > As regards the changes you suggest: > > 1) Millions of lines of code have been written with the assumption > that if you see a variable twice > in the same lexical scope it has the same value. Changing this would > mean that all old code > must be retested and modified. > > 2) When funs and list comprehensions where introduce the lexical > scoping rules where changed, > but there were no requirements for backwards compatibility > > 3) A change like this invalidate all existing Erlang books and all > books in the pipeline. > Lack of good documentation is probably our single greatest problem, > so > we don't like changes that > invalidate all old documents. > > 4) Something like frames and JSON bifs is being implemented and will > be released > as soon as it is stable. And yes there is slow progress on things > that are > not financed. The priorities for development are set by the people > who > pay for the development. > This means in practice that stuff that is need for Ericsson products > gets prioritized. > > 5) Slow development of a language is a sign of maturity - languages > that are over twenty years old don't > change quickly. Most effort has not gone into language development > but > into the run-time system. > As each new generation of multicores has come along the system has > evolved to accommodate them > We want to keep the language constant, while adapting the run-time > system to modern architectures. > This takes priority over language changes. > > 6) If you made a change like this and distributed the code somebody > who read the code later would not > know what the code meant. You can't have the same syntax for > something > in a program that means two different things > at two different times. > > 7) I'm not against change as such - but we do have to take backwards > comparability seriously for many reasons. > We want to improve the system, subject to the condition that we don't > break the existing stuff. Even small change > require a lot of work, it's not just a question of changing the > parser > and compiler. All third-part tools have to be changed > the dialyzer, quickcheck, eclipse IDE, refactoring tools etc. > > 8) Erlang could do with some major changes - I proposed erl2 a while > back but these changes are too > dramatic to be accommodated in a incremental and backwards compatible > manner. I perceive erl2 as a code generator > that produces regular erlang programs. > > /Joe > > > > > On Tuesday, July 10, 2012 1:40:17 PM UTC+4, Sergei Lebedev wrote: > >> > >> Richard, > >> > >> While this indeed *may* be a sign of a problem in your code, for > >> the > >> original example this is hardly the case. Splitting the function > >> apart > >> *only* because of compiler limitations sounds like an ad-hoc hack, > >> not a > >> real solution. > >> > >> -- Sergei > >> > >> On Jul 10, 2012, at 1:02 PM, Richard Carlsson wrote: > >> > >> > On 07/10/2012 10:43 AM, Dmitry Groshev wrote: > >> >> case do_something() of > >> >> {ok, Result} -> Result; > >> >> {error, Error} -> Error > >> >> end, > >> >> case do_another() of > >> >> {ok, Result} -> Result; > >> >> {error, Error} -> Error > >> >> end, > >> >> > >> >> Result and Error are bound in first case and we will probably > >> >> have a > >> >> match failure in second one. Compiler warns about this, but > >> >> it's still > >> >> very unwieldy to fix it with names like Error1, Error2, etc. > >> > > >> > Take it as a sign that you should break out those case > >> > expressions into > >> > separate functions, or restructure the code in some other way. > >> > > >> > /Richard > >> > _______________________________________________ > >> > erlang-questions mailing list > >> > erlang-questions@REDACTED > >> > http://erlang.org/mailman/listinfo/erlang-questions > >> > >> _______________________________________________ > >> erlang-questions mailing list > >> erlang-questions@REDACTED > >> http://erlang.org/mailman/listinfo/erlang-questions > > > > > > On Tuesday, July 10, 2012 1:40:17 PM UTC+4, Sergei Lebedev wrote: > >> > >> Richard, > >> > >> While this indeed *may* be a sign of a problem in your code, for > >> the > >> original example this is hardly the case. Splitting the function > >> apart > >> *only* because of compiler limitations sounds like an ad-hoc hack, > >> not a > >> real solution. > >> > >> -- Sergei > >> > >> On Jul 10, 2012, at 1:02 PM, Richard Carlsson wrote: > >> > >> > On 07/10/2012 10:43 AM, Dmitry Groshev wrote: > >> >> case do_something() of > >> >> {ok, Result} -> Result; > >> >> {error, Error} -> Error > >> >> end, > >> >> case do_another() of > >> >> {ok, Result} -> Result; > >> >> {error, Error} -> Error > >> >> end, > >> >> > >> >> Result and Error are bound in first case and we will probably > >> >> have a > >> >> match failure in second one. Compiler warns about this, but > >> >> it's still > >> >> very unwieldy to fix it with names like Error1, Error2, etc. > >> > > >> > Take it as a sign that you should break out those case > >> > expressions into > >> > separate functions, or restructure the code in some other way. > >> > > >> > /Richard > >> > _______________________________________________ > >> > erlang-questions mailing list > >> > erlang-questions@REDACTED > >> > http://erlang.org/mailman/listinfo/erlang-questions > >> > >> _______________________________________________ > >> erlang-questions mailing list > >> erlang-questions@REDACTED > >> http://erlang.org/mailman/listinfo/erlang-questions > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From vychodil.hynek@REDACTED Sun Jul 15 11:06:36 2012 From: vychodil.hynek@REDACTED (Hynek Vychodil) Date: Sun, 15 Jul 2012 11:06:36 +0200 Subject: [erlang-questions] Improve performance of IO bounded server written in Erlang via having pollset for each scheduler and bind port to scheduler together with process In-Reply-To: References: <771ADB6B-9CD0-4A7A-A8A1-EC551B84BAD1@gmail.com> <0E26E896-FF0D-4036-A51A-8F3B8C1FA31C@gmail.com> Message-ID: On Wed, Jul 11, 2012 at 10:35 PM, Ronny Meeus wrote: > > On Wed, Jul 11, 2012 at 9:12 PM, Zabrane Mickael > wrote: > > Hi Wei, > > > > On Jul 11, 2012, at 2:35 PM, Wei Cao wrote: > > > > sure, the steps is correct > > > > > > I re-installed everything from scratch with your second patch and tested > > your ehttpd web server example. > > > > before: ~55K rps > > after: ~70K rps > > > > but was unable to reach the 100K rps. > > > > Anyone courageous enough to help us reach the 100K rps? > > > > Regards, > > Zabrane > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > > I think the feature that certain processes can be bound to a certain > CPU at application level would also be useful. > The solution implemented here only works (if I understand it well) > when a process is using socket communication. > > There are scenarios where tasks are using just a little load (for > example short processing of a message) where the overhead introduced > by the scheduler is large so that nothing is gained by switching to a > multi-core processor (in fact single core runs much faster). > If certain the processes can be grouped on application level and bound > to specific cores, the application scales a lot better. > The other solution would be to run a separate VM instance per core but > I have the feeling that this is more complex to manage and that there > is also more overhead if messages need to be sent between the > applications running on different VMs. > You can bind process to specific core in Erlang. First you have to bind schedulers to cores using +sbt emulator option (I'm using default binding +sbt db) and than you can bind process to specific scheduler using spawn option {scheduler, N} when N is id of scheduler starting from 1 (0 binds to same scheduler may be - it is not documented what I could be able find). You can also get current scheduler id using erlang:system_info(scheduler_id). [spawn_opt(fun() -> io:format("~p -> Scheduler: ~p~n", [X, erlang:system_info(scheduler_id)]) end, [{scheduler, X}]) || X <- lists:seq(1,erlang:system_info(schedulers))]. > This is a "taskset" like primitive that we know it from the Linux thread > world. > For example in thread "Strange observation running in an SMP > environment." in this mailing list this would certainly be beneficial. > > Best regards, > Ronny > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From lambdadmitry@REDACTED Sun Jul 15 11:50:16 2012 From: lambdadmitry@REDACTED (Dmitry Groshev) Date: Sun, 15 Jul 2012 02:50:16 -0700 (PDT) Subject: [erlang-questions] Local namespaces in case/try In-Reply-To: References: <4FFBEFA5.6030305@gmail.com> Message-ID: <5e56d3fd-2919-4538-bb4e-c0c3571aaeca@googlegroups.com> I wasn't talking about breaking backward compatibility, I was talking about some sort of -compile() option. Moreover, I doubt that anything will break even if this feature will become standart, because right now compiler throws an error/warning if one use a variable like that. On Tuesday, July 10, 2012 4:41:54 PM UTC+4, Joe Armstrong wrote: > > On Tue, Jul 10, 2012 at 12:00 PM, Dmitry Groshev > wrote: > > Exactly what I was talking about. I can't see any sense in bloating the > code > > with nonsensical functions just to get something let-like -- AFAIK, > every > > functional language out there has a local namespace for such constructs. > > Moreover, there is a huge inconsistency right now with list > comprehensions > > that have their own namespace because of their transformation to > functions: > > > > test1() -> > > [begin > > _A = X, > > ok > > end || X <- [1, 2, 3]], > > _A = 1. > > > > test2() -> > > case 2 of > > _A -> _A > > end, > > _A = 1. > > > > I think it's a sign of a bigger problem with Erlang as a language: we > have > > an amazing BEAM, some brilliant libraries, a lot of godlike developers, > but > > Erlang as a language evolves extremely slow. One reason that I can think > > about is a lack of ultimate authority that can decide what is right and > what > > is wrong ? theoretically there is a EEP process (and some proposals are > out > > of EEPs scope, like an e2 thing), but in practice some EEPs are here for > > years (like Frames proposal or JSON BIFs) and nothing really moves on. > > No that's not the reason. > > There is no ultimate authority. Change is driven by the interplay of > several factors: > > - consensus (ie general agreement that something is good) > - payment (the people who finance Erlang pay to have something done) > - voluntary effort (individuals make changes, which are then > adopted) > > As regards the changes you suggest: > > 1) Millions of lines of code have been written with the assumption > that if you see a variable twice > in the same lexical scope it has the same value. Changing this would > mean that all old code > must be retested and modified. > > 2) When funs and list comprehensions where introduce the lexical > scoping rules where changed, > but there were no requirements for backwards compatibility > > 3) A change like this invalidate all existing Erlang books and all > books in the pipeline. > Lack of good documentation is probably our single greatest problem, so > we don't like changes that > invalidate all old documents. > > 4) Something like frames and JSON bifs is being implemented and will be > released > as soon as it is stable. And yes there is slow progress on things that are > not financed. The priorities for development are set by the people who > pay for the development. > This means in practice that stuff that is need for Ericsson products > gets prioritized. > > 5) Slow development of a language is a sign of maturity - languages > that are over twenty years old don't > change quickly. Most effort has not gone into language development but > into the run-time system. > As each new generation of multicores has come along the system has > evolved to accommodate them > We want to keep the language constant, while adapting the run-time > system to modern architectures. > This takes priority over language changes. > > 6) If you made a change like this and distributed the code somebody > who read the code later would not > know what the code meant. You can't have the same syntax for something > in a program that means two different things > at two different times. > > 7) I'm not against change as such - but we do have to take backwards > comparability seriously for many reasons. > We want to improve the system, subject to the condition that we don't > break the existing stuff. Even small change > require a lot of work, it's not just a question of changing the parser > and compiler. All third-part tools have to be changed > the dialyzer, quickcheck, eclipse IDE, refactoring tools etc. > > 8) Erlang could do with some major changes - I proposed erl2 a while > back but these changes are too > dramatic to be accommodated in a incremental and backwards compatible > manner. I perceive erl2 as a code generator > that produces regular erlang programs. > > /Joe > > > > > On Tuesday, July 10, 2012 1:40:17 PM UTC+4, Sergei Lebedev wrote: > >> > >> Richard, > >> > >> While this indeed *may* be a sign of a problem in your code, for the > >> original example this is hardly the case. Splitting the function apart > >> *only* because of compiler limitations sounds like an ad-hoc hack, not > a > >> real solution. > >> > >> -- Sergei > >> > >> On Jul 10, 2012, at 1:02 PM, Richard Carlsson wrote: > >> > >> > On 07/10/2012 10:43 AM, Dmitry Groshev wrote: > >> >> case do_something() of > >> >> {ok, Result} -> Result; > >> >> {error, Error} -> Error > >> >> end, > >> >> case do_another() of > >> >> {ok, Result} -> Result; > >> >> {error, Error} -> Error > >> >> end, > >> >> > >> >> Result and Error are bound in first case and we will probably have a > >> >> match failure in second one. Compiler warns about this, but it's > still > >> >> very unwieldy to fix it with names like Error1, Error2, etc. > >> > > >> > Take it as a sign that you should break out those case expressions > into > >> > separate functions, or restructure the code in some other way. > >> > > >> > /Richard > >> > _______________________________________________ > >> > erlang-questions mailing list > >> > erlang-questions@REDACTED > >> > http://erlang.org/mailman/listinfo/erlang-questions > >> > >> _______________________________________________ > >> erlang-questions mailing list > >> erlang-questions@REDACTED > >> http://erlang.org/mailman/listinfo/erlang-questions > > > > > > On Tuesday, July 10, 2012 1:40:17 PM UTC+4, Sergei Lebedev wrote: > >> > >> Richard, > >> > >> While this indeed *may* be a sign of a problem in your code, for the > >> original example this is hardly the case. Splitting the function apart > >> *only* because of compiler limitations sounds like an ad-hoc hack, not > a > >> real solution. > >> > >> -- Sergei > >> > >> On Jul 10, 2012, at 1:02 PM, Richard Carlsson wrote: > >> > >> > On 07/10/2012 10:43 AM, Dmitry Groshev wrote: > >> >> case do_something() of > >> >> {ok, Result} -> Result; > >> >> {error, Error} -> Error > >> >> end, > >> >> case do_another() of > >> >> {ok, Result} -> Result; > >> >> {error, Error} -> Error > >> >> end, > >> >> > >> >> Result and Error are bound in first case and we will probably have a > >> >> match failure in second one. Compiler warns about this, but it's > still > >> >> very unwieldy to fix it with names like Error1, Error2, etc. > >> > > >> > Take it as a sign that you should break out those case expressions > into > >> > separate functions, or restructure the code in some other way. > >> > > >> > /Richard > >> > _______________________________________________ > >> > erlang-questions mailing list > >> > erlang-questions@REDACTED > >> > http://erlang.org/mailman/listinfo/erlang-questions > >> > >> _______________________________________________ > >> erlang-questions mailing list > >> erlang-questions@REDACTED > >> http://erlang.org/mailman/listinfo/erlang-questions > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lambdadmitry@REDACTED Sun Jul 15 12:01:08 2012 From: lambdadmitry@REDACTED (Dmitry Groshev) Date: Sun, 15 Jul 2012 03:01:08 -0700 (PDT) Subject: [erlang-questions] Local namespaces in case/try In-Reply-To: References: Message-ID: <76444ea4-4c69-480b-82b2-ea2c290ddbd8@googlegroups.com> Robert, language *is* schizophrenic right now thanks to list comprehensions. [A || {ok, A} <- B] %% there is no A outside this LC case hd(B) of {ok, A} -> A end %% and now we have A bound. Feels like asylum In my opinion, there are only two ways to make it less so: -do some magic and rework list comprehensions so they export their variables ? it's hard with current LC implementation, but it's doable. It will break some code for sure; -rework case/if/receive statements to introduce their own namespaces. It will not break any code that compiles without warnings, because reuse of variables outside of case/if/receive emits a warning right now. The second one is better, I think. On Sunday, July 15, 2012 5:01:39 AM UTC+4, Robert Virding wrote: > > Sorry for not getting into this discussion earlier but I am vegetating in > the country (also called being on holidays) at the moment and the my > internet connection is not what it should be. > > Go Joe! I agree with what you and both Richards have written. Just some > personal comments: > > - If you were to introduce a let with variable scoping then you would have > to do the same for if/case/receive as well otherwise you would get a very > schizophrenic language. One which would definitely be much more confusing > for beginners than it is today. It would also mean that let/if/case/receive > wouldn't export variables, only return values, which would mean that the > original code would be useless anyway. > > - Even if funs/comprehensions behave differently from the rest their > behaviour is quite logical within the erlang context. For a fun to *export* > a variable would be completely crazy as it is defined in one place and > executed in another; where and when would you see the exported value? For a > comprehension which value of a variable should you see? > > - Having a variable referring to the same value in the whole function > clause body maybe different but it is actually very simple to grasp; > actually much simpler than having scopes. > > - As has been pointed out it is not a compiler problem, having scopes is > actually easier. > > Robert > > ----- Original Message ----- > > On Tue, Jul 10, 2012 at 12:00 PM, Dmitry Groshev > > wrote: > > > Exactly what I was talking about. I can't see any sense in bloating > > > the code > > > with nonsensical functions just to get something let-like -- AFAIK, > > > every > > > functional language out there has a local namespace for such > > > constructs. > > > Moreover, there is a huge inconsistency right now with list > > > comprehensions > > > that have their own namespace because of their transformation to > > > functions: > > > > > > test1() -> > > > [begin > > > _A = X, > > > ok > > > end || X <- [1, 2, 3]], > > > _A = 1. > > > > > > test2() -> > > > case 2 of > > > _A -> _A > > > end, > > > _A = 1. > > > > > > I think it's a sign of a bigger problem with Erlang as a language: > > > we have > > > an amazing BEAM, some brilliant libraries, a lot of godlike > > > developers, but > > > Erlang as a language evolves extremely slow. One reason that I can > > > think > > > about is a lack of ultimate authority that can decide what is right > > > and what > > > is wrong ? theoretically there is a EEP process (and some proposals > > > are out > > > of EEPs scope, like an e2 thing), but in practice some EEPs are > > > here for > > > years (like Frames proposal or JSON BIFs) and nothing really moves > > > on. > > > > No that's not the reason. > > > > There is no ultimate authority. Change is driven by the interplay of > > several factors: > > > > - consensus (ie general agreement that something is good) > > - payment (the people who finance Erlang pay to have something > > done) > > - voluntary effort (individuals make changes, which are then > > adopted) > > > > As regards the changes you suggest: > > > > 1) Millions of lines of code have been written with the assumption > > that if you see a variable twice > > in the same lexical scope it has the same value. Changing this would > > mean that all old code > > must be retested and modified. > > > > 2) When funs and list comprehensions where introduce the lexical > > scoping rules where changed, > > but there were no requirements for backwards compatibility > > > > 3) A change like this invalidate all existing Erlang books and all > > books in the pipeline. > > Lack of good documentation is probably our single greatest problem, > > so > > we don't like changes that > > invalidate all old documents. > > > > 4) Something like frames and JSON bifs is being implemented and will > > be released > > as soon as it is stable. And yes there is slow progress on things > > that are > > not financed. The priorities for development are set by the people > > who > > pay for the development. > > This means in practice that stuff that is need for Ericsson products > > gets prioritized. > > > > 5) Slow development of a language is a sign of maturity - languages > > that are over twenty years old don't > > change quickly. Most effort has not gone into language development > > but > > into the run-time system. > > As each new generation of multicores has come along the system has > > evolved to accommodate them > > We want to keep the language constant, while adapting the run-time > > system to modern architectures. > > This takes priority over language changes. > > > > 6) If you made a change like this and distributed the code somebody > > who read the code later would not > > know what the code meant. You can't have the same syntax for > > something > > in a program that means two different things > > at two different times. > > > > 7) I'm not against change as such - but we do have to take backwards > > comparability seriously for many reasons. > > We want to improve the system, subject to the condition that we don't > > break the existing stuff. Even small change > > require a lot of work, it's not just a question of changing the > > parser > > and compiler. All third-part tools have to be changed > > the dialyzer, quickcheck, eclipse IDE, refactoring tools etc. > > > > 8) Erlang could do with some major changes - I proposed erl2 a while > > back but these changes are too > > dramatic to be accommodated in a incremental and backwards compatible > > manner. I perceive erl2 as a code generator > > that produces regular erlang programs. > > > > /Joe > > > > > > > > On Tuesday, July 10, 2012 1:40:17 PM UTC+4, Sergei Lebedev wrote: > > >> > > >> Richard, > > >> > > >> While this indeed *may* be a sign of a problem in your code, for > > >> the > > >> original example this is hardly the case. Splitting the function > > >> apart > > >> *only* because of compiler limitations sounds like an ad-hoc hack, > > >> not a > > >> real solution. > > >> > > >> -- Sergei > > >> > > >> On Jul 10, 2012, at 1:02 PM, Richard Carlsson wrote: > > >> > > >> > On 07/10/2012 10:43 AM, Dmitry Groshev wrote: > > >> >> case do_something() of > > >> >> {ok, Result} -> Result; > > >> >> {error, Error} -> Error > > >> >> end, > > >> >> case do_another() of > > >> >> {ok, Result} -> Result; > > >> >> {error, Error} -> Error > > >> >> end, > > >> >> > > >> >> Result and Error are bound in first case and we will probably > > >> >> have a > > >> >> match failure in second one. Compiler warns about this, but > > >> >> it's still > > >> >> very unwieldy to fix it with names like Error1, Error2, etc. > > >> > > > >> > Take it as a sign that you should break out those case > > >> > expressions into > > >> > separate functions, or restructure the code in some other way. > > >> > > > >> > /Richard > > >> > _______________________________________________ > > >> > erlang-questions mailing list > > >> > erlang-questions@REDACTED > > >> > http://erlang.org/mailman/listinfo/erlang-questions > > >> > > >> _______________________________________________ > > >> erlang-questions mailing list > > >> erlang-questions@REDACTED > > >> http://erlang.org/mailman/listinfo/erlang-questions > > > > > > > > > On Tuesday, July 10, 2012 1:40:17 PM UTC+4, Sergei Lebedev wrote: > > >> > > >> Richard, > > >> > > >> While this indeed *may* be a sign of a problem in your code, for > > >> the > > >> original example this is hardly the case. Splitting the function > > >> apart > > >> *only* because of compiler limitations sounds like an ad-hoc hack, > > >> not a > > >> real solution. > > >> > > >> -- Sergei > > >> > > >> On Jul 10, 2012, at 1:02 PM, Richard Carlsson wrote: > > >> > > >> > On 07/10/2012 10:43 AM, Dmitry Groshev wrote: > > >> >> case do_something() of > > >> >> {ok, Result} -> Result; > > >> >> {error, Error} -> Error > > >> >> end, > > >> >> case do_another() of > > >> >> {ok, Result} -> Result; > > >> >> {error, Error} -> Error > > >> >> end, > > >> >> > > >> >> Result and Error are bound in first case and we will probably > > >> >> have a > > >> >> match failure in second one. Compiler warns about this, but > > >> >> it's still > > >> >> very unwieldy to fix it with names like Error1, Error2, etc. > > >> > > > >> > Take it as a sign that you should break out those case > > >> > expressions into > > >> > separate functions, or restructure the code in some other way. > > >> > > > >> > /Richard > > >> > _______________________________________________ > > >> > erlang-questions mailing list > > >> > erlang-questions@REDACTED > > >> > http://erlang.org/mailman/listinfo/erlang-questions > > >> > > >> _______________________________________________ > > >> erlang-questions mailing list > > >> erlang-questions@REDACTED > > >> http://erlang.org/mailman/listinfo/erlang-questions > > > > > > > > > _______________________________________________ > > > erlang-questions mailing list > > > erlang-questions@REDACTED > > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jose.valim@REDACTED Sun Jul 15 12:30:01 2012 From: jose.valim@REDACTED (=?ISO-8859-1?Q?Jos=E9_Valim?=) Date: Sun, 15 Jul 2012 12:30:01 +0200 Subject: [erlang-questions] Local namespaces in case/try In-Reply-To: References: Message-ID: > > - Even if funs/comprehensions behave differently from the rest their > behaviour is quite logical within the erlang context. For a fun to *export* > a variable would be completely crazy as it is defined in one place and > executed in another; where and when would you see the exported value? Agreed. Languages like python and ruby change the context of where the function/lambda is defined, which introduces race conditions when the function is run in a thread. This is actually one of the ways such languages use to communicate between threads, *shrugs*. > For a comprehension which value of a variable should you see? > Agreed as well, I wouldn't expect comprehensions to set a variable in the outer context. To add to your puzzle, if the list is empty, what is the value of the variable? -------------- next part -------------- An HTML attachment was scrubbed... URL: From jose.valim@REDACTED Sun Jul 15 12:35:18 2012 From: jose.valim@REDACTED (=?ISO-8859-1?Q?Jos=E9_Valim?=) Date: Sun, 15 Jul 2012 12:35:18 +0200 Subject: [erlang-questions] Local namespaces in case/try In-Reply-To: <76444ea4-4c69-480b-82b2-ea2c290ddbd8@googlegroups.com> References: <76444ea4-4c69-480b-82b2-ea2c290ddbd8@googlegroups.com> Message-ID: > > -rework case/if/receive statements to introduce their own namespaces. It > will not break any code that compiles without warnings, because reuse of > variables outside of case/if/receive emits a warning right now. > Please don't. I would actually prefer if we didn't emit a warning in the first place (in fact I frequently disable it). This is a convenient feature and the compiler is smart enough to let us know if the variable is assigned in just one branch, which would introduce runtime errors. -------------- next part -------------- An HTML attachment was scrubbed... URL: From lambdadmitry@REDACTED Sun Jul 15 12:40:23 2012 From: lambdadmitry@REDACTED (Dmitry Groshev) Date: Sun, 15 Jul 2012 03:40:23 -0700 (PDT) Subject: [erlang-questions] Local namespaces in case/try In-Reply-To: References: <4FFBEFA5.6030305@gmail.com> Message-ID: Richard, don't you think about case as some sort of imperative construction? Because I can't see why "case" can't be that "keyword to introduce new variable" ? case is isolated in some sense. It was already done with list comprehensions, after all. Moreover, this is the case in Haskell that you've mentioned: Prelude> let a = [1, 2, 3] Prelude> case a of [b, c, d] -> "ok" "ok" Prelude> b :6:1: Not in scope: `b' So does OCaml: # let a = [1; 2; 3] ;; val a : int list = [1; 2; 3] # match a with [b; c; d] -> b | _ -> 0 ;; - : int = 1 # b ;; Error: Unbound value b So it's more like some obscure "feature" of Erlang for newcomers, which is no good. Speaking about better example: test(Input) -> Result = case test1(Input) of {ok, BetterInput} -> case test2(BetterInput) of {ok, BestInput} -> do_something(BestInput); {error, _Reason}=Error -> Error end; {error, _Reson}=Error -> Error end, case Result of {ok, _}=Ok -> Ok; {error, _Reason}=Error -> %% OOPS! do_something_on_error(Error) end. I can come up with a few solutions to this compile-time error, but I can't understand why I should. Correct indentation solves a problem of reasoning about nested constructions. On Wednesday, July 11, 2012 5:35:07 AM UTC+4, Richard O'Keefe wrote: > > > On 10/07/2012, at 10:00 PM, Dmitry Groshev wrote: > > > Exactly what I was talking about. I can't see any sense in bloating the > code with nonsensical functions > > It isn't the functions that are nonsensical, it is > using the same variable with two different meanings > which is in the strict literal sense of the word nonsensical. > > Give us a serious example! > Show us some code that is easier to read if you do this! > > Yes, other functional languages have nested variable scopes within a > single function (although the semantics is often defined in terms of > nested functions), BUT those languages require you to introduce new > bindings with keywords > > (define (f X) > (let ((R) (E)) ;;; HERE! > (let ((V (do-something))) > (case (car V) > ((result) (set! R (cadr V))) > ((error) (set! E (cadr V)))))) > (let ((R) (E)) ;;; HERE! > (case (car V) > ((result) (set! R (cadr V))) > ((error) (set! E (cadr V))))))) > > Lisp and Scheme have LET and LET*. > ML has let. > Haskell has let and where. > > If you want Lisp-Flavoured Erlang, it exists. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lambdadmitry@REDACTED Sun Jul 15 12:46:49 2012 From: lambdadmitry@REDACTED (Dmitry Groshev) Date: Sun, 15 Jul 2012 03:46:49 -0700 (PDT) Subject: [erlang-questions] Local namespaces in case/try In-Reply-To: References: <76444ea4-4c69-480b-82b2-ea2c290ddbd8@googlegroups.com> Message-ID: So you are against changing context (as follows from your previous message), and in the same time you prefer to change context of matching in non-obvious (without carefully observing all case's branches) way? Shiny! Don't you think that it's *way* less readable than explicit return of bunch of variables (through tuples, for example) from case? On Sunday, July 15, 2012 2:35:18 PM UTC+4, Jos? Valim wrote: > > -rework case/if/receive statements to introduce their own namespaces. It >> will not break any code that compiles without warnings, because reuse of >> variables outside of case/if/receive emits a warning right now. >> > > Please don't. I would actually prefer if we didn't emit a warning in the > first place (in fact I frequently disable it). This is a convenient feature > and the compiler is smart enough to let us know if the variable is assigned > in just one branch, which would introduce runtime errors. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jose.valim@REDACTED Sun Jul 15 13:20:24 2012 From: jose.valim@REDACTED (=?ISO-8859-1?Q?Jos=E9_Valim?=) Date: Sun, 15 Jul 2012 13:20:24 +0200 Subject: [erlang-questions] Local namespaces in case/try In-Reply-To: References: <76444ea4-4c69-480b-82b2-ea2c290ddbd8@googlegroups.com> Message-ID: > > So you are against changing context (as follows from your previous > message), and in the same time you prefer to change context of matching in > non-obvious (without carefully observing all case's branches) way? There are **technical** good reasons for not changing the context in functions and comprehensions. Erlang is made of very few constructs and you quickly learn which ones change and which ones do not change the context. That said, I prefer to learn how to best use these constructs and use them accordingly than to be painted into a corner. Yes, things can get nasty if you are using such feature in large/complex `case` (which, as you said, would require careful observation) but any feature gets ugly if abused or not used properly. -------------- next part -------------- An HTML attachment was scrubbed... URL: From chandrashekhar.mullaparthi@REDACTED Sun Jul 15 21:23:17 2012 From: chandrashekhar.mullaparthi@REDACTED (Chandru) Date: Sun, 15 Jul 2012 20:23:17 +0100 Subject: [erlang-questions] Erlang and Solaris SMF's In-Reply-To: References: Message-ID: On 28 June 2012 21:24, Heinz N. Gies wrote: > I am sure someone has asked this before but I googled for the last hour > and have found nothing but a discussion that didn't provide a answer. > > My problem is that I want to run a erlang otp release as a SMF service, > now the monitoring does not work properly, what I discovered from the > discussion mentioned is that it seems the problem that solaris things the > release is still running since the epmd is still up and good even so the > application might not. > > What would be the best solution for this? My first guess was to create a > empd service but that kind of gets ugly once you've more then one erlang > release running. > > We do exactly that. We have an installation script which installs epmd as a separate SMF service, and all other erlang nodes on the machine depend on it. When installing an erlang node, just check whether epmd is installed as a service or not, and if not then install it. cheers Chandru -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.sovetov@REDACTED Sun Jul 15 22:31:24 2012 From: victor.sovetov@REDACTED (Viktor Sovietov) Date: Sun, 15 Jul 2012 13:31:24 -0700 (PDT) Subject: [erlang-questions] Ling: Erlang over Xen, no OS required In-Reply-To: <20120713210833.GA16632@circlewave.net> References: <9b565382-35da-472d-8315-0a8d7b35ace8@n5g2000vbb.googlegroups.com> <20120713210833.GA16632@circlewave.net> Message-ID: On 14 ???, 00:08, fr...@REDACTED wrote: > On Fri, Jul 13, 2012 at 12:30:41AM -0700, Viktor Sovietov wrote: > > Ling is highly compatible (with R15B) Erlang VM that is capable to run > > over bare virtual iron on Xen hypervisor without the need in > > underlying operating system. > > > Learn more on :http://erlangonxen.org/ > > Interesting -- it's looking like it's not a port of BEAM? Is source code You're correct, it's different VM developed from the scratch. > available somewhere? We didn't publish sources yet for number of reasons - the business model of this product isn't defined yet, VM itself is under heavy development, it's quite specific piece of code which requires deep knowledge of BEAM, etc, etc, etc. > How do you handle TCP/IP networking + persistent > storage without an OS? It had been done through 9P client in VM and re-export server in dom0 that shares filesystems and network access for Ling instances. > > BR, > ? ? ? ? -- Jachym > _______________________________________________ > erlang-questions mailing list > erlang-questi...@REDACTED://erlang.org/mailman/listinfo/erlang-questions From ok@REDACTED Mon Jul 16 00:53:38 2012 From: ok@REDACTED (Richard O'Keefe) Date: Mon, 16 Jul 2012 10:53:38 +1200 Subject: [erlang-questions] lists, binaries or something else? In-Reply-To: References: Message-ID: <33FAA0D4-1581-428D-8BAC-08B928E4CEC9@cs.otago.ac.nz> On 13/07/2012, at 12:46 AM, CGS wrote: > I am trying to find a balance in between processing speed and RAM consumption for sets of large strings (over 1 M characters per string). I've been watching this thread with some interest. Just last week I told a 3rd-year software engineering class about the importance of making sure you are solving the right problem. Are all the strings over 1M characters? Can you characterise the size distribution more clearly? How many of these strings do you have? Do you have fixed sets, or do you have them flooding in and out again? Where do they come from? Are you doing anything to these strings, or just holding them and passing them on? Smalltalk systems keep track of the source code of every method, but they do this by keeping the characters in any of several files and internally just keeping what-and-where tuples; I used that successfully in a Prolog program once (think CLOBs). If you might be editing the strings, have you considered using a tree of binaries? (I'm thinking of "piece tables" and AVL DAGs.) Haskell's lazy ByteStrings http://www.haskell.org/ghc/docs/7.0-latest/html/libraries/bytestring-0.9.1.10/Data-ByteString-Lazy.html are in effect lists of binaries (default chunk size = 64k). > About each string, it is constructed from chunks of fixed size, > usually, much smaller than the string itself, hopefully. This sounds a lot like the list-of-chunk representation. Are all the chunks the same size? (Not that it matters much. Erlang iodata is basically list-of-chunk.) Do the contents of these strings have any structure? Are strings (however represented) *really* the right tool for the job? I just have such a hard time believing in 1M-char strings that are *just* strings and not something structured that has been represented as a string. Do they have shareable parts? (When I want to process XML or SGML, I have a thing I call the "Document Value Model", which uses hash consing in C. Even plain text files usually save at least a few percent.) Could the chunks be shareable, or are they just an artefact of packetising? Are the data already compressed? Are they of a kind that might benefit from compression? gzip, which is not state of the art, gets text down to about 30%. Heck, even random decimal digit sequences compress to less than 50%. The zlib module may be helpful. Some of the chunks in a string could be compressed and others not. What do the strings signify? Why are they so big? What is the purpose of the Erlang program? What is the *real* problem? From ok@REDACTED Mon Jul 16 01:47:13 2012 From: ok@REDACTED (Richard O'Keefe) Date: Mon, 16 Jul 2012 11:47:13 +1200 Subject: [erlang-questions] Local namespaces in case/try In-Reply-To: References: <4FFBEFA5.6030305@gmail.com> Message-ID: <571306AB-6159-44BA-98B3-A5CB30DF0135@cs.otago.ac.nz> On 15/07/2012, at 10:40 PM, Dmitry Groshev wrote: > don't you think about case as some sort of imperative construction? HELL NO! I don't think of 'case' in SML as imperative. I don't think of 'case' in Haskell as imperative. Because they aren't. So why should I think of 'case' in Erlang as imperative? Because it isn't. Yes, of course you can wrap 'case' around some code that _is_ imperative, but if I put a tiger in a wooden box, that doesn't mean the *box* has teeth and claws. And in any case, what has "imperative" to do with "variable scope"? > Because I can't see why "case" can't be that "keyword to introduce new variable" It _could_ be in a new language. But it isn't in Erlang, and there are good reasons why it wasn't. > ? case is isolated in some sense. Why yes: in the sense that there is nothing like 'case' except, oh, 'if', and, 'receive'. They are all exactly the same in this respect. Where do you get 'isolated'? > It was already done with list comprehensions, after all. There is a HUGE difference. In the alternative-choice constructs of Erlang, if execution continues after them, exactly one of the alternatives will have been executed exactly once, so any variable which is bound in every alternative will certainly be available with an unambiguous binding. List comprehensions and funs share an important characteristic which is relevant: the bindings in them might never be executed at all, or if executed might be executed very many times. They cannot produce an unambiguous binding. So they do not. > Moreover, this is the case in Haskell that you've mentioned: > > Prelude> let a = [1, 2, 3] > Prelude> case a of [b, c, d] -> "ok" > "ok" > Prelude> b > :6:1: Not in scope: `b' Haskell is not Erlang; many aspects of Haskell differ from Erlang, starting with the syntax of identifiers ("a" is a variable in Haskell but a constant in Erlang; Haskell allows "'" in identifiers but forbids "." while Erlang does it the other way). In particular, Haskell and Erlang have different scope rules. So what else is new? That does NOT mean that Haskell is the only way to do things. (Although I do love Haskell.) > So does OCaml: which is the runner-up for the title of "world's ugliest functional language". The prize-winner is of course XSLT. I keep trying to use O'Caml for the excellence of its compiler, and keep being driven away by its ugliness. The important point, of course, is that these are different languages with different rules about practically everything. > > So it's more like some obscure "feature" of Erlang for newcomers, which is no good. Erlang is designed for people who are willing to RTFM. > Speaking about better example: > > test(Input) -> > Result = case test1(Input) of > {ok, BetterInput} -> > case test2(BetterInput) of > {ok, BestInput} -> > do_something(BestInput); > {error, _Reason}=Error -> > Error > end; > {error, _Reson}=Error -> > Error > end, > case Result of > {ok, _}=Ok -> Ok; > {error, _Reason}=Error -> %% OOPS! > do_something_on_error(Error) > end. But this is NOT ***REAL*** CODE. It is an example made up for the occasion. I have no reason whatsoever to imagine for one instant that this is anything like code that anyone *really* needs to write. Amongst other things, I find this tangled code quite unnecessarily hard to read. Let's try to figure out what it is supposed to do test1(Input) = {ok,BetterInput} and test2(BetterInput) = {ok,BestInput} and do_something(BestInput) = {ok,Answer} => {ok,Answer} Anything along the way = {error,Reason} => do_something_on_error({error,Reason}) Anything else => crash. Ah hah! Just test(Input) -> case case test1(Input) of {ok,Better_Input} -> case test2(Better_Input) of {ok,Best_Input} -> do_something(Best_Input) ; E -> E end ; E -> E end of OK = {ok,_} -> OK ; ERR = {error,_} -> do_something_on_error(ERR) end. Now it's a _little_ clearer. This actually looks like a case where throwing exceptions instead of returning {error,Reason} would be better: test(Input) -> try do_something(test2(test1(Input))) catch {error,E} -> do_something_on_error(E) end. I repeat my earlier request, with added emphasis. > Give us a SERIOUS example! > Show us some code that is easier to read if you do this! From physacco@REDACTED Mon Jul 16 10:02:21 2012 From: physacco@REDACTED (=?UTF-8?B?6b6Z56ys5Lmd5a2Q?=) Date: Mon, 16 Jul 2012 16:02:21 +0800 Subject: [erlang-questions] Memory issue: big swap usage Message-ID: Hi all, Recently I wrote a service-monitoring program in Erlang. It uses httpc (the HTTP client shipped with Erlang/OTP) to fetch web server statistics data, does some calculations, and then sends results to another service. It works fine, however, the memory usage is very strange. While VmRSS stays at a reasonable level, but VmSize and VmData keep increasing (about 40m per day). Please look at the following data: System memory usage: $ free -m total used free shared buffers cached Mem: 2012 1486 525 0 216 713 -/+ buffers/cache: 557 1455 Swap: 2047 5 2041 `top` shows that the beam.smp process consumes 51m physical memory :) and 857m swap memory :( PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ SWAP COMMAND 27100 root 20 0 909m 51m 2288 S 2 2.6 149:54.65 857m beam.smp Current VM status of the process: $ cat /proc/27100/status|grep Vm VmPeak: 932260 kB VmSize: 931616 kB VmLck: 0 kB VmHWM: 54624 kB VmRSS: 52968 kB VmData: 907556 kB VmStk: 88 kB VmExe: 1860 kB VmLib: 2412 kB VmPTE: 208 kB And Erlang reported memory usage (seems good): 15> erlang:memory(). [{total,46487136}, {processes,1283144}, {processes_used,1251480}, {system,45203992}, {atom,509577}, {atom_used,497728}, {binary,169912}, {code,4395539}, {ets,323224}] I use Erlang R14B03 (erts-5.8.4) compiled with GCC 4.4.5 on Debian 6 Linux 2.6.32-5-amd64. I'm not sure if there is a memory-leaking bug. The problem is too hard for me and I have no choice but to restart the process. Could someone be kind enough to help me out? Please. From eriksoe@REDACTED Mon Jul 16 10:33:56 2012 From: eriksoe@REDACTED (=?ISO-8859-1?Q?Erik_S=F8e_S=F8rensen?=) Date: Mon, 16 Jul 2012 10:33:56 +0200 Subject: [erlang-questions] Memory issue: big swap usage In-Reply-To: References: Message-ID: Either the problem is in Erlang code, or it's a leak at a lower level. The erlang:memory() numbers seem to indicate the latter. If it's a low-level issue, then a leak detector like libumem(?) or Valgrind might be of some use. It may be an idea to check if the number of ports is stable, and whether it helps matters if you do the work in a short-lived process. Other than that, I guess the issue may be hard to tackle from within Erlang itself. Den 16/07/2012 10.02 skrev "????" : > Hi all, > > Recently I wrote a service-monitoring program in Erlang. It uses httpc > (the HTTP client shipped with Erlang/OTP) to fetch web server > statistics data, does some calculations, and then sends results to > another service. It works fine, however, the memory usage is very > strange. While VmRSS stays at a reasonable level, but VmSize and > VmData keep increasing (about 40m per day). Please look at the > following data: > > System memory usage: > > $ free -m > total used free shared buffers > cached > Mem: 2012 1486 525 0 216 > 713 > -/+ buffers/cache: 557 1455 > Swap: 2047 5 2041 > > > `top` shows that the beam.smp process consumes 51m physical memory :) > and 857m swap memory :( > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ SWAP > COMMAND > 27100 root 20 0 909m 51m 2288 S 2 2.6 149:54.65 857m > beam.smp > > > Current VM status of the process: > > $ cat /proc/27100/status|grep Vm > VmPeak: 932260 kB > VmSize: 931616 kB > VmLck: 0 kB > VmHWM: 54624 kB > VmRSS: 52968 kB > VmData: 907556 kB > VmStk: 88 kB > VmExe: 1860 kB > VmLib: 2412 kB > VmPTE: 208 kB > > > And Erlang reported memory usage (seems good): > > 15> erlang:memory(). > [{total,46487136}, > {processes,1283144}, > {processes_used,1251480}, > {system,45203992}, > {atom,509577}, > {atom_used,497728}, > {binary,169912}, > {code,4395539}, > {ets,323224}] > > > I use Erlang R14B03 (erts-5.8.4) compiled with GCC 4.4.5 on Debian 6 > Linux 2.6.32-5-amd64. I'm not sure if there is a memory-leaking bug. > The problem is too hard for me and I have no choice but to restart the > process. Could someone be kind enough to help me out? Please. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cgsmcmlxxv@REDACTED Mon Jul 16 11:03:57 2012 From: cgsmcmlxxv@REDACTED (CGS) Date: Mon, 16 Jul 2012 11:03:57 +0200 Subject: [erlang-questions] lists, binaries or something else? In-Reply-To: References: Message-ID: Agree. Thanks a lot, Bob! CGS On Fri, Jul 13, 2012 at 6:43 PM, Bob Ippolito wrote: > If the chunks are big enough you will benefit from sending them as binary > since there won't be any copying. As others have already said, you probably > will benefit most from either an iolist or all binary scheme. > > Benchmarking adding characters at a time isn't really equivalent to adding > chunks at a time, especially for large chunk sizes. Of course, adding a > chunk (or even a list if chunks) to either side of an iolist is O(1), so if > you're mostly just building these things you're really better off leaving > it like that as long as possible. > > > On Friday, July 13, 2012, CGS wrote: > >> Yes, you are right. When I did the test, I thought also about that, so, I >> made a third loop which should have appended the char to the binary string. >> Unfortunately, I had a typo there and I ended up having the same loop for >> binaries (and therefore the same numbers). Your message made me check it >> again and I found that typo. I didn't find quite those ratios you got, but, >> yes, they are comparable with your ratios. Thanks a lot. >> >> >> >> On Fri, Jul 13, 2012 at 4:51 PM, Anders Nygren wrote: >> >> For some cases appending to a binary is much faster, see >> http://www.erlang.org/doc/efficiency_guide/binaryhandling.html#id65923 >> >> So Your simple benchmark is not fair, You are comparing the best case >> for lists with the worst case for binaries. If You change it to >> appending instead of prepending You will get very different results. >> So as one of the first replies said "it depends". >> >> On my machine: >> For small N prepending to a list is ~2x faster than appending to a binary. >> When N=1E6 appending to a binary is ~2x faster than prepending to a list. >> >> /Anders >> >> >> On Fri, Jul 13, 2012 at 6:48 AM, CGS wrote: >> > Hi Bob, >> > >> > On Fri, Jul 13, 2012 at 2:32 AM, Bob Ippolito wrote: >> >> >> >> Sorry, but that's not enough information. >> > >> > >> > Sorry, but I didn't know what you meant by that. Here are the answers. >> > >> >> >> >> >> >> Where do these chunks come from: source code, some other process, ets? >> > >> > >> > Mostly from other processes. >> > >> >> >> >> How many chunks are in a string? >> > >> > >> > That is computed from the string size and chunk size (to be decided >> later). >> > >> >> >> >> Do you compose strings out of other strings, or just these chunks? >> > >> > >> > Just chunks inserted into existing/new strings. >> > >> >> >> >> Are you constructing them from tail to head like you would a list? >> > >> > >> > Unfortunately, not all the time. >> > >> >> >> >> Is the string constructed all at once, or over some time? >> > >> > >> > If you mean by that the string will be fully given in the same message >> or >> > whatever by other processes, the answer is no. Over some time may be the >> > answer, but with the remark that I have no idea what means "over some >> time" >> > as period of time (can get chunks one after the other for the same >> string or >> > for different strings). >> > >> >> >> >> >> >> It sounds like you have some code that's doing something with these >> >> strings, because you say that it's faster to use lists than binaries. >> Maybe >> >> if you post whatever it is you used to determine that, someone can >> help you >> >> find a better algorithm and/or data structure for that benchmark. >> > >> > >> > That is simple. For benchmark I just took two simple loops (only for >> > insertion): >> > >> > -export([loop_list/1, loop_binary/2]). >> > >> > loop_list(0) -> []; >> > loop_list(N) -> [107 | loop_list(N)]. >> > >> > loop_binary(0,B) -> B; >> > loop_binary(N,B) -> loop_binary(N-1,<<107,B/binary>>). >> > >> > If you go in powers of tens and measure the execution time, you will >> see the >> > difference (you will also notice the drop in efficiency for binaries >> when >> > you need fragmentation for the same string, which is not visible in the >> case >> > of lists - or at least I couldn't notice). That is not a fully >> conclusive >> > test for my case, but it is quite a conclusive test for processing >> speed in >> > the two cases (simple expansion of the string by adding a char at the >> > beginning of the string), in which, for a 10^5 chars string, the list >> gains >> > at least one order of magnitude in processing time than its binary >> > counterpart (1) (2) (3). >> > >> > (1) Even if you change the list loop to have an accumulator in which the >> > list exists, there is still one order of magnitude difference. >> > (2) The results are specific for the machine on which the tests are >> done. >> > (3) That is just an example. >> > >> > CGS >> > >> > >> >> >> >> >> >> >> >> On Thursday, July 12, 2012, CGS wrote: >> >>> >> >>> Unfortunately, the project is still in the planning stage, so, no real >> >>> code was written yet. Nevertheless, I plan some open source projects >> for >> >>> some parts of the project. >> >>> >> >>> About each string, it is constructed from chunks of fixe >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang@REDACTED Mon Jul 16 11:07:49 2012 From: erlang@REDACTED (Joe Armstrong) Date: Mon, 16 Jul 2012 11:07:49 +0200 Subject: [erlang-questions] Looking for slides of a lightning talk In-Reply-To: References: <4FFAAAEA.9060605@ninenines.eu> <0d6c6c5d-2b0f-48da-bae8-f440d232df92@googlegroups.com> <489B3810-49B2-4085-B468-DA037F7DE646@dmitriid.com> <1709242D-0AD5-4070-A6EC-AB039E62ADFD@dmitriid.com> <1342003659.84614.YahooMailNeo@web111404.mail.gq1.yahoo.com> <4FFDFE93.8040202@gmail.com> <4FFEF003.70605@gmail.com> Message-ID: On Thu, Jul 12, 2012 at 11:45 PM, CGS wrote: > Hi Jan, > > Few thoughts here. > >> You are hearing things in my email that were not there when I wrote it. > > > Sorry about that, but stating that "Erlang is just too damn difficult to do > the easy stuff" is what I read in your message, not inventing it. And you > said that just because of some applications (cowboy and rebar). Use erl with > -detach and -noshell options and you have a perfect executable without > breaking any sweat to learn how to make an application. All you need is to > read "man erl" (you can even google it if you don't have Linux). Of course > that's for beginners. Later on, you will learn to make applications with > everything you need, so, you will most of the time forget about those > options. :) Is that complex? Difficult? I don't suppose it's more difficult > than reading about the options of any given compiler. > I think the point is that in any language it's easy to do the things that that language was designed to do and difficult otherwise. Erlang was designed for (among other things) distributed programming - thus distributed programming is easy. Pid ! Message and receive Pattern1 -> Action1; ... end Are sufficient to write distributed programs (and a small amount of setup) This is ridiculously easy - I'm fighting with Objective C trying to do the same thing - I need to know about Grand Central Dispatch (GCD) and sockets and goodness knows what just to send a message between two machines. So in Objective C it's "really difficult to do easy stuff" On the other hand in Objective C it appears to be really easy to make a window throw in a few buttons and get them to do things when you click on them (which is really difficult to do in Erlang) I accept that Erlang is not good at processing images and things like that _ but it was never intended to be good at things like that ... /Joe >> >> >> I was reporting my experience, not blaming Erlang for anything. My >> experience is that the learning curve for the Erlang tools (coupled >> unfortunately with learning the Linux environment) was extremely steep - and >> ended up with me getting stuck because the tools failed to do what they >> claimed (in multiple ways) before I had the skills or knowledge to fix them. > > > Well, every tool may have the same doom. Those are called bugs and therefore > there are versions for each of them. One cannot think of every possibility, > can he/she? That doesn't mean the tool is stopping you in learning about > what is applied to. Moreover, in the Linux world, until one tool is accepted > as satisfactory (so, further tools are no more developed in that field), > there are a lot of alternative ones. You don't need to get stubborn in > having one and only that tool, especially when you are new in that field. > >> >> >> Erlang the language is not the problem. The language is never the problem. >> Once the tool chain is working...... > > > Search for the tool which suites your needs or ask if it is possible to get > new features which may support your needs in a given time. I tell you that > from my own experience because not once it happened to get the wrong tool > for my needs and after I managed to learn about that, I discovered it > doesn't do what others were thinking it may do. One example is SpiderMonkey > for CouchDB. CouchDB and SpiderMonkey are both great tools, but, in the > past, CouchDB had quite few problems in getting it work because of > SpiderMonkey. So, I googled it and I found a repository where I could find a > version of SpiderMonkey which was compatible with that version of CouchDB. > Now, CouchDB has no longer that problem anymore. But the conclusion remains: > search for the correct version of what you need. > >> >> >> But consider this. Rebar can't be installed in a standard Ubuntu install >> of Erlang!!! WTF? > > > As far as I know (I am not using rebar because I am creating my own > installation scripts which is much faster than thinking how to do it using > rebar), rebar doesn't need any installation. I usually take it from the same > repository as the source I need to compile. Strangely enough, those projects > which do not provide rebar, they work with the latest version of rebar. > Never met your problem, but there are many problems I haven't met, so, I > don't deny your's. Indeed, bad experience for a newbie. > >> >> >> Would you agree that that means at least one of those installs is >> inadequate for my needs? Its inadequate for anyone who wishes to use rebar >> and Erlang on Ubuntu/Debian. > > > I work under Ubuntu most of the time. It's my favorite Linux flavor (CentOS > is becoming the second, but I wouldn't recommend it at starting point). Few > times I had problems with it, but I do tend to install my own versions > instead of those from repos. They old and providing many crashes in the new > versions of applications which require them. Fedora may cope with this > problem, but you may have some other surprises there which may be more > troublesome if you don't know about Linux. > >> >> >> Would you agree that giving newbies tools that are inadequate for their >> needs, does not help them scale their learning curve? > > > I do not understand your point. A software is not for a newbie or for a > veteran. A software is created to fill an empty spot in a certain field. > There are many other tools which you can start with and I don't suppose > somebody gave you the tool, but you picked it. All you can do is to ask > specific questions about the tool until you learn how to use it properly for > your needs. In this community (and quite a lot of others) there will be no > answer like RTFM or learn by yourself. There are many persons here who can > answer questions in a wide range of knowledge (from where can I find any > documentation about Erlang? to, I don't know, "why not using frames in > Erlang?" just to give an example which covers a subject way over my head). > >> >> >> This is not meant as a criticism of the work people have done. Its a very >> hard problem, and I don't think there is an easy solution. > > > In my opinion, there are no easy or difficult solutions. There are only > smart solutions or using the brute force. :) When I am not smart enough, I > use brute force (it seems quite a natural choice at the beginning and I am > still using it in Erlang ;) ). > >> >> >> PHP isn't there yet. PEAR is not widely used. The comments added to the >> manual are very helpful. PHP the language is astonishingly inconsistent. > > > I wouldn't know that. I know that PHP has an insanely large documentation > which I will never be able to know it by heart. :) > >> >> >> Python boasts "batteries included". Whatever you want to do in Python, >> there are usually 6 modules that claim to do it - all poorly documented, >> some buggy. So do you spend your time investigating them, or writing a 7th >> that you know will do what you want? > > > Yeah, agree. I usually opt for the last solution as well if I feel smart > enough to do that. :) > >> >> >> Experienced people have forgotten the misunderstandings they had as >> newbies - or they had an experienced person to guide them - so they are >> genuinely puzzled by the problems newbies face. The newbies can't help: they >> don't know what they don't know. Few technical people can write well, and >> all are busy with their projects. The result is a distinct lack of >> excellent, current documentation of the tools that can help newbies start to >> use Erlang. > > > Therefore there are a lot of others writing some beginner's guides. I > remember one of my starting documentations for Erlang was "Learn You Some > Erlang" (yes, Fred, I haven't had the opportunity to thank you for that, but > I do it now). Just try few of them. And if you want some code, I would > recommend to start with trapexit. > >> >> >> It comes, slowly, with maturity and stability. Erlang the language is well >> documented. Its the docs for the supporting cast that is spotty. Which is a >> shame. > > > Just don't get discouraged. You still have the chance to criticize and hate > Erlang. :D > >>> >>> You blame Erlang for an application written in it which doesn't support >>> MS Windows... Well, that reminds me of a case of saying noSQL databases are >>> not good just because that person/team misunderstood the usage of a >>> particular noSQL database. Interesting how people try to generalize their >>> impressions to a category from a wrongly chosen example. Don't get me wrong, >>> I am not that good in Erlang to mock you, but I think you should ask someone >>> more knowledgeable before you to say something like "Erlang is just too damn >>> difficult to do the easy stuff." I do easy stuff in Erlang. It would >>> probably be more difficult to write a proper C code to do those simple >>> things (like proper threading, distributed applications and so on). >>> >>> Now, the problem is what do you need cowboy for? If you need an webserver >>> for MS Windows which is written in Erlang, you can try Yaws (you have >>> installers here: http://yaws.hyber.org/download/). If you need a >>> Erlang-based webserver which you can integrate it in your application, you >>> will need either a Linux or Cygwin (for most of the open source projects, >>> but you can do it without if you use, for example, lhttpc - >>> https://bitbucket.org/etc/lhttpc/src - where you can modify the Makefile to >>> be compatible with MS VS or, for even less, a .bat file). I would avoid >>> installing Linux in a MS Windows box (I'd rather do it oppositely). >> >> Agreed. Ubuntu host with LVM and software RAID, VBox with a windows guest. >> Plans are laid :) >> >> Your other points are all good stuff. Thanks. >> >> I am looking for a fast web server that had a permanent core(1) image that >> I could program, that was not under any run-away timer (Apache) and that can >> handle many simultaneous connections (i.e event driven architecture). One of >> the projects is a special type of chat room where silence might exist for >> long periods, and I don't want a fast heart-beat or unnecessary writes to >> disk. > > > I may recommend BOSH technique (http://xmpp.org/extensions/xep-0124.html/) > rather than long polling. :) > > Best of luck! > > CGS > >> >> >> Note 1 - Core is RAM to anyone under 50. ;) >> >> >>> >>> Erlang under MS Windows works as nice as under Linux (as far as I could >>> play with it under MS Windows), but the installation under MS Windows is >>> much simpler (I would suggest for Linux to compile it from source with the >>> options you need rather than installing it from repositories). I don't know >>> what is your background in programming (and I don't want to know), but there >>> are nice tutorials on web for all the categories of programmers (coming from >>> OOP, functional or whatever else languages). Just get familiarized with >>> Erlang and after that make an impression of your own about the language >>> itself. I am also relatively new in Erlang and I consider I cannot afford to >>> make any statement about how good or not is Erlang. I know only that are >>> some nice things and some not so nice things in Erlang (I leave finding >>> examples to your experience). >> >> Indeed. Erlang under windows is fine. The target server is Ubuntu, so >> developing on that platform has advantages. Particularly in view of the >> planned migration to Ubuntu as my main machine. >> >> Regards >> >> Ian >>> >>> >>> CGS >>> >> >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From cgsmcmlxxv@REDACTED Mon Jul 16 11:37:13 2012 From: cgsmcmlxxv@REDACTED (CGS) Date: Mon, 16 Jul 2012 11:37:13 +0200 Subject: [erlang-questions] Looking for slides of a lightning talk In-Reply-To: References: <4FFAAAEA.9060605@ninenines.eu> <0d6c6c5d-2b0f-48da-bae8-f440d232df92@googlegroups.com> <489B3810-49B2-4085-B468-DA037F7DE646@dmitriid.com> <1709242D-0AD5-4070-A6EC-AB039E62ADFD@dmitriid.com> <1342003659.84614.YahooMailNeo@web111404.mail.gq1.yahoo.com> <4FFDFE93.8040202@gmail.com> <4FFEF003.70605@gmail.com> Message-ID: Hi Joe, I couldn't agree more with your message. After all, choosing the right tool for the problem at hand is part of solving the problem. CGS On Mon, Jul 16, 2012 at 11:07 AM, Joe Armstrong wrote: > On Thu, Jul 12, 2012 at 11:45 PM, CGS wrote: > > Hi Jan, > > > > Few thoughts here. > > > >> You are hearing things in my email that were not there when I wrote it. > > > > > > Sorry about that, but stating that "Erlang is just too damn difficult to > do > > the easy stuff" is what I read in your message, not inventing it. And you > > said that just because of some applications (cowboy and rebar). Use erl > with > > -detach and -noshell options and you have a perfect executable without > > breaking any sweat to learn how to make an application. All you need is > to > > read "man erl" (you can even google it if you don't have Linux). Of > course > > that's for beginners. Later on, you will learn to make applications with > > everything you need, so, you will most of the time forget about those > > options. :) Is that complex? Difficult? I don't suppose it's more > difficult > > than reading about the options of any given compiler. > > > > I think the point is that in any language it's easy to do the things that > that language was designed to do and difficult otherwise. > > Erlang was designed for (among other things) distributed programming - > thus distributed programming is easy. > > Pid ! Message > > and > > receive > Pattern1 -> > Action1; > ... > end > > Are sufficient to write distributed programs (and a small amount of setup) > > This is ridiculously easy - I'm fighting with Objective C trying to do > the same thing - I need to > know about Grand Central Dispatch (GCD) and sockets and goodness knows > what just > to send a message between two machines. So in Objective C it's "really > difficult to do easy stuff" > On the other hand in Objective C it appears to be really easy to make > a window throw in a few > buttons and get them to do things when you click on them (which is > really difficult to do in Erlang) > > I accept that Erlang is not good at processing images and things like > that _ but it was never intended to be good at things like that ... > > /Joe > > >> > >> > >> I was reporting my experience, not blaming Erlang for anything. My > >> experience is that the learning curve for the Erlang tools (coupled > >> unfortunately with learning the Linux environment) was extremely steep > - and > >> ended up with me getting stuck because the tools failed to do what they > >> claimed (in multiple ways) before I had the skills or knowledge to fix > them. > > > > > > Well, every tool may have the same doom. Those are called bugs and > therefore > > there are versions for each of them. One cannot think of every > possibility, > > can he/she? That doesn't mean the tool is stopping you in learning about > > what is applied to. Moreover, in the Linux world, until one tool is > accepted > > as satisfactory (so, further tools are no more developed in that field), > > there are a lot of alternative ones. You don't need to get stubborn in > > having one and only that tool, especially when you are new in that field. > > > >> > >> > >> Erlang the language is not the problem. The language is never the > problem. > >> Once the tool chain is working...... > > > > > > Search for the tool which suites your needs or ask if it is possible to > get > > new features which may support your needs in a given time. I tell you > that > > from my own experience because not once it happened to get the wrong tool > > for my needs and after I managed to learn about that, I discovered it > > doesn't do what others were thinking it may do. One example is > SpiderMonkey > > for CouchDB. CouchDB and SpiderMonkey are both great tools, but, in the > > past, CouchDB had quite few problems in getting it work because of > > SpiderMonkey. So, I googled it and I found a repository where I could > find a > > version of SpiderMonkey which was compatible with that version of > CouchDB. > > Now, CouchDB has no longer that problem anymore. But the conclusion > remains: > > search for the correct version of what you need. > > > >> > >> > >> But consider this. Rebar can't be installed in a standard Ubuntu install > >> of Erlang!!! WTF? > > > > > > As far as I know (I am not using rebar because I am creating my own > > installation scripts which is much faster than thinking how to do it > using > > rebar), rebar doesn't need any installation. I usually take it from the > same > > repository as the source I need to compile. Strangely enough, those > projects > > which do not provide rebar, they work with the latest version of rebar. > > Never met your problem, but there are many problems I haven't met, so, I > > don't deny your's. Indeed, bad experience for a newbie. > > > >> > >> > >> Would you agree that that means at least one of those installs is > >> inadequate for my needs? Its inadequate for anyone who wishes to use > rebar > >> and Erlang on Ubuntu/Debian. > > > > > > I work under Ubuntu most of the time. It's my favorite Linux flavor > (CentOS > > is becoming the second, but I wouldn't recommend it at starting point). > Few > > times I had problems with it, but I do tend to install my own versions > > instead of those from repos. They old and providing many crashes in the > new > > versions of applications which require them. Fedora may cope with this > > problem, but you may have some other surprises there which may be more > > troublesome if you don't know about Linux. > > > >> > >> > >> Would you agree that giving newbies tools that are inadequate for their > >> needs, does not help them scale their learning curve? > > > > > > I do not understand your point. A software is not for a newbie or for a > > veteran. A software is created to fill an empty spot in a certain field. > > There are many other tools which you can start with and I don't suppose > > somebody gave you the tool, but you picked it. All you can do is to ask > > specific questions about the tool until you learn how to use it properly > for > > your needs. In this community (and quite a lot of others) there will be > no > > answer like RTFM or learn by yourself. There are many persons here who > can > > answer questions in a wide range of knowledge (from where can I find any > > documentation about Erlang? to, I don't know, "why not using frames in > > Erlang?" just to give an example which covers a subject way over my > head). > > > >> > >> > >> This is not meant as a criticism of the work people have done. Its a > very > >> hard problem, and I don't think there is an easy solution. > > > > > > In my opinion, there are no easy or difficult solutions. There are only > > smart solutions or using the brute force. :) When I am not smart enough, > I > > use brute force (it seems quite a natural choice at the beginning and I > am > > still using it in Erlang ;) ). > > > >> > >> > >> PHP isn't there yet. PEAR is not widely used. The comments added to the > >> manual are very helpful. PHP the language is astonishingly inconsistent. > > > > > > I wouldn't know that. I know that PHP has an insanely large documentation > > which I will never be able to know it by heart. :) > > > >> > >> > >> Python boasts "batteries included". Whatever you want to do in Python, > >> there are usually 6 modules that claim to do it - all poorly documented, > >> some buggy. So do you spend your time investigating them, or writing a > 7th > >> that you know will do what you want? > > > > > > Yeah, agree. I usually opt for the last solution as well if I feel smart > > enough to do that. :) > > > >> > >> > >> Experienced people have forgotten the misunderstandings they had as > >> newbies - or they had an experienced person to guide them - so they are > >> genuinely puzzled by the problems newbies face. The newbies can't help: > they > >> don't know what they don't know. Few technical people can write well, > and > >> all are busy with their projects. The result is a distinct lack of > >> excellent, current documentation of the tools that can help newbies > start to > >> use Erlang. > > > > > > Therefore there are a lot of others writing some beginner's guides. I > > remember one of my starting documentations for Erlang was "Learn You Some > > Erlang" (yes, Fred, I haven't had the opportunity to thank you for that, > but > > I do it now). Just try few of them. And if you want some code, I would > > recommend to start with trapexit. > > > >> > >> > >> It comes, slowly, with maturity and stability. Erlang the language is > well > >> documented. Its the docs for the supporting cast that is spotty. Which > is a > >> shame. > > > > > > Just don't get discouraged. You still have the chance to criticize and > hate > > Erlang. :D > > > >>> > >>> You blame Erlang for an application written in it which doesn't support > >>> MS Windows... Well, that reminds me of a case of saying noSQL > databases are > >>> not good just because that person/team misunderstood the usage of a > >>> particular noSQL database. Interesting how people try to generalize > their > >>> impressions to a category from a wrongly chosen example. Don't get me > wrong, > >>> I am not that good in Erlang to mock you, but I think you should ask > someone > >>> more knowledgeable before you to say something like "Erlang is just > too damn > >>> difficult to do the easy stuff." I do easy stuff in Erlang. It would > >>> probably be more difficult to write a proper C code to do those simple > >>> things (like proper threading, distributed applications and so on). > >>> > >>> Now, the problem is what do you need cowboy for? If you need an > webserver > >>> for MS Windows which is written in Erlang, you can try Yaws (you have > >>> installers here: http://yaws.hyber.org/download/). If you need a > >>> Erlang-based webserver which you can integrate it in your application, > you > >>> will need either a Linux or Cygwin (for most of the open source > projects, > >>> but you can do it without if you use, for example, lhttpc - > >>> https://bitbucket.org/etc/lhttpc/src - where you can modify the > Makefile to > >>> be compatible with MS VS or, for even less, a .bat file). I would avoid > >>> installing Linux in a MS Windows box (I'd rather do it oppositely). > >> > >> Agreed. Ubuntu host with LVM and software RAID, VBox with a windows > guest. > >> Plans are laid :) > >> > >> Your other points are all good stuff. Thanks. > >> > >> I am looking for a fast web server that had a permanent core(1) image > that > >> I could program, that was not under any run-away timer (Apache) and > that can > >> handle many simultaneous connections (i.e event driven architecture). > One of > >> the projects is a special type of chat room where silence might exist > for > >> long periods, and I don't want a fast heart-beat or unnecessary writes > to > >> disk. > > > > > > I may recommend BOSH technique ( > http://xmpp.org/extensions/xep-0124.html/) > > rather than long polling. :) > > > > Best of luck! > > > > CGS > > > >> > >> > >> Note 1 - Core is RAM to anyone under 50. ;) > >> > >> > >>> > >>> Erlang under MS Windows works as nice as under Linux (as far as I could > >>> play with it under MS Windows), but the installation under MS Windows > is > >>> much simpler (I would suggest for Linux to compile it from source with > the > >>> options you need rather than installing it from repositories). I don't > know > >>> what is your background in programming (and I don't want to know), but > there > >>> are nice tutorials on web for all the categories of programmers > (coming from > >>> OOP, functional or whatever else languages). Just get familiarized with > >>> Erlang and after that make an impression of your own about the language > >>> itself. I am also relatively new in Erlang and I consider I cannot > afford to > >>> make any statement about how good or not is Erlang. I know only that > are > >>> some nice things and some not so nice things in Erlang (I leave finding > >>> examples to your experience). > >> > >> Indeed. Erlang under windows is fine. The target server is Ubuntu, so > >> developing on that platform has advantages. Particularly in view of the > >> planned migration to Ubuntu as my main machine. > >> > >> Regards > >> > >> Ian > >>> > >>> > >>> CGS > >>> > >> > >> > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gustav.simonsson@REDACTED Mon Jul 16 12:06:26 2012 From: gustav.simonsson@REDACTED (Gustav Simonsson) Date: Mon, 16 Jul 2012 11:06:26 +0100 (BST) Subject: [erlang-questions] Memory issue: big swap usage In-Reply-To: Message-ID: Hi, The swap column in top does not represent the current swap usage of processes. Rather, it represents how much swap would be necessary if the process was swapped out. If you want to find out how much swap your BEAM process is really using, look at the smaps file of the process and sum up the "Swap:" fields. For example: ps -e | grep beam 4726 pts/1 00:46:14 beam.smp grep "Swap:" /proc/4726/smaps | awk '{swap = swap + $2} END {print swap}' The output is in kilobytes. man proc | grep -A 15 smaps Regards, Gustav Simonsson Sent from my PC ----- Original Message ----- > From: "????" > To: "Erlang-Questions" > Sent: Monday, 16 July, 2012 10:02:21 AM > Subject: [erlang-questions] Memory issue: big swap usage > > Hi all, > > Recently I wrote a service-monitoring program in Erlang. It uses > httpc > (the HTTP client shipped with Erlang/OTP) to fetch web server > statistics data, does some calculations, and then sends results to > another service. It works fine, however, the memory usage is very > strange. While VmRSS stays at a reasonable level, but VmSize and > VmData keep increasing (about 40m per day). Please look at the > following data: > > System memory usage: > > $ free -m > total used free shared buffers > cached > Mem: 2012 1486 525 0 216 > 713 > -/+ buffers/cache: 557 1455 > Swap: 2047 5 2041 > > > `top` shows that the beam.smp process consumes 51m physical memory :) > and 857m swap memory :( > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ SWAP > COMMAND > 27100 root 20 0 909m 51m 2288 S 2 2.6 149:54.65 857m > beam.smp > > > Current VM status of the process: > > $ cat /proc/27100/status|grep Vm > VmPeak: 932260 kB > VmSize: 931616 kB > VmLck: 0 kB > VmHWM: 54624 kB > VmRSS: 52968 kB > VmData: 907556 kB > VmStk: 88 kB > VmExe: 1860 kB > VmLib: 2412 kB > VmPTE: 208 kB > > > And Erlang reported memory usage (seems good): > > 15> erlang:memory(). > [{total,46487136}, > {processes,1283144}, > {processes_used,1251480}, > {system,45203992}, > {atom,509577}, > {atom_used,497728}, > {binary,169912}, > {code,4395539}, > {ets,323224}] > > > I use Erlang R14B03 (erts-5.8.4) compiled with GCC 4.4.5 on Debian 6 > Linux 2.6.32-5-amd64. I'm not sure if there is a memory-leaking bug. > The problem is too hard for me and I have no choice but to restart > the > process. Could someone be kind enough to help me out? Please. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From cgsmcmlxxv@REDACTED Mon Jul 16 12:40:09 2012 From: cgsmcmlxxv@REDACTED (CGS) Date: Mon, 16 Jul 2012 12:40:09 +0200 Subject: [erlang-questions] lists, binaries or something else? In-Reply-To: <33FAA0D4-1581-428D-8BAC-08B928E4CEC9@cs.otago.ac.nz> References: <33FAA0D4-1581-428D-8BAC-08B928E4CEC9@cs.otago.ac.nz> Message-ID: Hi Richard, Let me see if I understood your points correctly: 1. Is string the right tool? Alternatives: list/iolist (even binary tree) of hashes for the chunks. 2. To use compressed data as Joe suggested some messages ago. If that is the case, here are few thoughts: 1. a) Using binary tree structure is something I definitely took into account whenever is possible. That is why I was asking for opinions about processing lists vs. binaries vs. something else. Once the basic storage type is set, the rest is coming accordingly. My only question was about that storage type because my experience with Erlang has a gap here (as I said, I am relatively new in using Erlang) and I wanted some opinions from those who used them more extensively. Until now I got nice opinions and I thank everyone who shared them here. 1. b) Using hashes for the chunks may be a solution worth thinking of. I do not share too much enthusiasm on that as converting chunks into hashes may slow down the overall process. But since I have no idea about how much such a solution slows down the overall processing time, I definitely should take this solution into account. 2. As I said, this is something worth taking into account even if I need to find a way to define "read-only" and "read-write" characteristic for each string. Of course, until I have some test cases, this is only a pure discussion. So, the first step now should be to think of some test cases, I suppose. Thanks a lot for your input. It definitely made me think twice about how to approach the problem. CGS On Mon, Jul 16, 2012 at 12:53 AM, Richard O'Keefe wrote: > > On 13/07/2012, at 12:46 AM, CGS wrote: > > I am trying to find a balance in between processing speed and RAM > consumption for sets of large strings (over 1 M characters per string). > > I've been watching this thread with some interest. > > Just last week I told a 3rd-year software engineering class about the > importance of making sure you are solving the right problem. > > Are all the strings over 1M characters? > Can you characterise the size distribution more clearly? > How many of these strings do you have? > Do you have fixed sets, or do you have them flooding in and > out again? > Where do they come from? > Are you doing anything to these strings, > or just holding them and passing them on? > > Smalltalk systems keep track of the source code of every method, > but they do this by keeping the characters in any of several files > and internally just keeping what-and-where tuples; I used that > successfully in a Prolog program once (think CLOBs). > > If you might be editing the strings, have you considered using > a tree of binaries? (I'm thinking of "piece tables" and AVL DAGs.) > > Haskell's lazy ByteStrings > > http://www.haskell.org/ghc/docs/7.0-latest/html/libraries/bytestring-0.9.1.10/Data-ByteString-Lazy.html > are in effect lists of binaries (default chunk size = 64k). > > About each string, it is constructed from chunks of fixed size, > > usually, much smaller than the string itself, hopefully. > This sounds a lot like the list-of-chunk representation. > Are all the chunks the same size? (Not that it matters much. > Erlang iodata is basically list-of-chunk.) > > Do the contents of these strings have any structure? > Are strings (however represented) *really* the right tool for the job? > > I just have such a hard time believing in 1M-char strings that > are *just* strings and not something structured that has been > represented as a string. > > Do they have shareable parts? > (When I want to process XML or SGML, I have a thing I call the > "Document Value Model", which uses hash consing in C. Even plain > text files usually save at least a few percent.) > Could the chunks be shareable, or are they just an artefact of > packetising? > > Are the data already compressed? Are they of a kind that might benefit > from compression? gzip, which is not state of the art, gets text down > to about 30%. Heck, even random decimal digit sequences compress to > less than 50%. The zlib module may be helpful. Some of the chunks in > a string could be compressed and others not. > > What do the strings signify? Why are they so big? > What is the purpose of the Erlang program? > What is the *real* problem? > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From physacco@REDACTED Mon Jul 16 13:07:28 2012 From: physacco@REDACTED (=?UTF-8?B?6b6Z56ys5Lmd5a2Q?=) Date: Mon, 16 Jul 2012 19:07:28 +0800 Subject: [erlang-questions] Memory issue: big swap usage In-Reply-To: References: Message-ID: Thanks for your replies. I'll refactor my code to reduce some long living processes. Just now I tried Gustav's method as follow: $ grep "Swap:" /proc/27100/smaps | awk '{swap = swap + $2} END {print swap}' 776 This indicated the beam process is using 776K swap memory. Not a big number. It seems that the process doesn't use too much memory. (51m rss + 776K swap) But I still don't understand why VmSize and VmData are so large, and are growing larger everyday... Kind regards On Mon, Jul 16, 2012 at 6:06 PM, Gustav Simonsson wrote: > > Hi, > > The swap column in top does not represent the current swap usage of processes. > Rather, it represents how much swap would be necessary if the process was swapped out. > > If you want to find out how much swap your BEAM process is really using, > look at the smaps file of the process and sum up the "Swap:" fields. > > For example: > > ps -e | grep beam > 4726 pts/1 00:46:14 beam.smp > > grep "Swap:" /proc/4726/smaps | awk '{swap = swap + $2} END {print swap}' > > The output is in kilobytes. > > man proc | grep -A 15 smaps > > Regards, > Gustav Simonsson > > Sent from my PC > > ----- Original Message ----- >> From: "????" >> To: "Erlang-Questions" >> Sent: Monday, 16 July, 2012 10:02:21 AM >> Subject: [erlang-questions] Memory issue: big swap usage >> >> Hi all, >> >> Recently I wrote a service-monitoring program in Erlang. It uses >> httpc >> (the HTTP client shipped with Erlang/OTP) to fetch web server >> statistics data, does some calculations, and then sends results to >> another service. It works fine, however, the memory usage is very >> strange. While VmRSS stays at a reasonable level, but VmSize and >> VmData keep increasing (about 40m per day). Please look at the >> following data: >> >> System memory usage: >> >> $ free -m >> total used free shared buffers >> cached >> Mem: 2012 1486 525 0 216 >> 713 >> -/+ buffers/cache: 557 1455 >> Swap: 2047 5 2041 >> >> >> `top` shows that the beam.smp process consumes 51m physical memory :) >> and 857m swap memory :( >> >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ SWAP >> COMMAND >> 27100 root 20 0 909m 51m 2288 S 2 2.6 149:54.65 857m >> beam.smp >> >> >> Current VM status of the process: >> >> $ cat /proc/27100/status|grep Vm >> VmPeak: 932260 kB >> VmSize: 931616 kB >> VmLck: 0 kB >> VmHWM: 54624 kB >> VmRSS: 52968 kB >> VmData: 907556 kB >> VmStk: 88 kB >> VmExe: 1860 kB >> VmLib: 2412 kB >> VmPTE: 208 kB >> >> >> And Erlang reported memory usage (seems good): >> >> 15> erlang:memory(). >> [{total,46487136}, >> {processes,1283144}, >> {processes_used,1251480}, >> {system,45203992}, >> {atom,509577}, >> {atom_used,497728}, >> {binary,169912}, >> {code,4395539}, >> {ets,323224}] >> >> >> I use Erlang R14B03 (erts-5.8.4) compiled with GCC 4.4.5 on Debian 6 >> Linux 2.6.32-5-amd64. I'm not sure if there is a memory-leaking bug. >> The problem is too hard for me and I have no choice but to restart >> the >> process. Could someone be kind enough to help me out? Please. >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- ???????????????????? From wojtek@REDACTED Mon Jul 16 16:27:08 2012 From: wojtek@REDACTED (=?GB2312?B?V29qdGVrIE5hcmN6eai9c2tp?=) Date: Mon, 16 Jul 2012 16:27:08 +0200 Subject: [erlang-questions] Improve performance of IO bounded server written in Erlang via having pollset for each scheduler and bind port to scheduler together with process In-Reply-To: <500025A1.7040201@power.com.pl> References: <771ADB6B-9CD0-4A7A-A8A1-EC551B84BAD1@gmail.com> <0E26E896-FF0D-4036-A51A-8F3B8C1FA31C@gmail.com> <4FFF0D25.7050601@power.com.pl> <500025A1.7040201@power.com.pl> Message-ID: <500424BC.8060904@power.com.pl> On 07/13/2012 03:41 PM, Wojtek Narczy??ski wrote: > Maybe it would be enough to call erts_deliver_time() only if do_wait is > true? I checked, it would not. > Or if that fails, create a new mutex, say erts_timedelivery_mtx, use > erts_smp_mtx_trylock() on it, and only proceed with the time delivery, > from the thread that succeeds in obtaining the lock. This should keep > do_time relatively fresh. This did not work either. Neither calling erts_deliver_time() only if the poll call returned zero workable descriptors. And, because there are quite a few variables related to time, atomic operations are not an easy solution either. Perhaps something like seqlock, invented exactly in the context of timekeeping, would do the job. --Regards, Wojtek From siraaj@REDACTED Mon Jul 16 16:46:40 2012 From: siraaj@REDACTED (Siraaj Khandkar) Date: Mon, 16 Jul 2012 10:46:40 -0400 Subject: [erlang-questions] Local namespaces in case/try In-Reply-To: <571306AB-6159-44BA-98B3-A5CB30DF0135@cs.otago.ac.nz> References: <4FFBEFA5.6030305@gmail.com> <571306AB-6159-44BA-98B3-A5CB30DF0135@cs.otago.ac.nz> Message-ID: On Jul 15, 2012, at 19:47, "Richard O'Keefe" wrote: > On 15/07/2012, at 10:40 PM, Dmitry Groshev wrote: > >> So does OCaml: > > which is the runner-up for the title of "world's ugliest functional > language". The prize-winner is of course XSLT. > > I keep trying to use O'Caml for the excellence of its compiler, > and keep being driven away by its ugliness. Could you give some real code examples that you consider ugly in OCaml and beautiful in Erlang? -- Siraaj Khandkar .o. ..o ooo From wojtek@REDACTED Mon Jul 16 17:17:17 2012 From: wojtek@REDACTED (=?UTF-8?B?V29qdGVrIE5hcmN6ecWEc2tp?=) Date: Mon, 16 Jul 2012 17:17:17 +0200 Subject: [erlang-questions] Framework advise In-Reply-To: References: Message-ID: <5004307D.2000208@power.com.pl> Jayson, > Web standards based - Must support HTML 5, JS, CSS, etc. I am thinking ChicagoBoss + twitter bootstrap worked well for us. You can code in such a way, that the client is the client, and the server is the server (serving JSON). > MS SQL connectivity - Our existing database is MS SQL and that won't be easy to change so it needs to be able to connect to it. You can connect with ODBC, which runs as an external process, which might be fast enough, or not. > Testable - I will need to be able to write and run Unit, Integration tests. I am guessing that eunit and common test will cover this. The testing framework is much less important than the ability to isolate components. In erlang you can test functions, but you can also relatively easily isolate and test whole subsystems, because of erlang message-passing nature. > Easily deployable - A new server will need to be deployed and configured in minutes. Something like Chef would be ideal since we are already using that in other areas. > In place upgrades - When a new build is ready to deployed we should be able to deploy it, leave the previous version in place to roll back should an issue be found. If you mean adding capacity, virtualization is difficult to beat. Erlang hot code updates are useful for patching. For a major roll-out of a new version, where the database schema changes, there is no silver bullet. > Easy logging. It depends on your requirements for the logging subsystem, but in logging in erlang is pretty hard. --Regards, Wojtek Narczynski From jesper.louis.andersen@REDACTED Mon Jul 16 17:19:43 2012 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Mon, 16 Jul 2012 17:19:43 +0200 Subject: [erlang-questions] Local namespaces in case/try In-Reply-To: <571306AB-6159-44BA-98B3-A5CB30DF0135@cs.otago.ac.nz> References: <4FFBEFA5.6030305@gmail.com> <571306AB-6159-44BA-98B3-A5CB30DF0135@cs.otago.ac.nz> Message-ID: <6D4E0716-C2BC-4EDB-857A-257A20E83B32@erlang-solutions.com> On Jul 16, 2012, at 1:47 AM, Richard O'Keefe wrote: > test(Input) -> > try do_something(test2(test1(Input))) > catch {error,E} -> do_something_on_error(E) > end. This is *the* way to structure this code. The observation is that failure of one pattern leads to failure of the them all. This tells you that your code lives in an error-monad. The way to get hold of such a monad in Erlang is to use a try?catch construct because it is really an exceptional structure. If you are really lucky, any failure will just lead to the process exiting and you don't have to handle that specifically. Then something is to be said for just structuring the code like R = do_something(test2(test1(Input))). And let the crash be handled directly. Usually the pattern reveals itself when you have deeply nested levels of case constructs where the error is just passed on if it happens. The trick is to rewrite it into a try-catch (and be precise about what error classes you catch. Don't catch them all - i.e., do Pokemon? coding). Another solution is to utilize the fact that case is an expression, not a statement. So you can fold functions over the accumulated result: lists:foldl(fun(F, {error, E}) -> {error, E}; (F, {ok, V}) -> F(V) end, {ok, Input}, [fun test1/1, fun test2/1]). (not tested). Essentially, this is the same thing as the try..catch. Going further, one could seek to add enough syntactic sugar to support something like the Haskell do-notation - but it won't be as neat as in Haskell because we have no type information in type classes to implicitly choose the right monad to operate in[1] J. [1] Aside: This is really what the power of Haskell is about. Using the type system to *avoid* having to write boring code. And use the type system to *substitute* the code executed automatically so save you time when your program changes. From gabre@REDACTED Mon Jul 16 21:39:43 2012 From: gabre@REDACTED (gabre@REDACTED) Date: Mon, 16 Jul 2012 21:39:43 +0200 Subject: [erlang-questions] SSH SCP Message-ID: <20120716213943.17894ve1kldrmqdb@webmail.elte.hu> Hello, I d like to ask about the SSH module found in OTP. I would like to use the SCP command (which uses ssh) via Erlang (SSH module). Is there anybody, who can explain me, how to do it? I have googled and searched million times, but nothing... (I have also tried it using os:cmd/1 and a seld-made shell loop) Thank you in advance! Gabor H. From bchesneau@REDACTED Tue Jul 17 01:28:15 2012 From: bchesneau@REDACTED (Benoit Chesneau) Date: Tue, 17 Jul 2012 01:28:15 +0200 Subject: [erlang-questions] [ANN] hackney 0.1 Message-ID: Hi all, Just a quick message to let you know I released the first version of hackney, a simple HTTP client in Erlang. Some part of the code reuse the cowboy HTTP parser and follow similar concepts. Main features are: - no message passing: response is directly streamed to the current process and a state is kept in a #client{} record. binary streams - SSL support - Keepalive handling - basic authentication - stream the response - Can send files using the sendfile API - Chunked encoding support - Optionnal socket pool - Used parse transform for shorcut methods calls:hackney:get("https://friendpaste.com") This a WIP, current TODO is available on github [1]. Full doc and source are available on github [2]. Let me know what you think about it, - beno?t [1] https://github.com/benoitc/hackney/blob/master/TODO.md [2] https://github.com/benoitc/hackney From kenji.rikitake@REDACTED Tue Jul 17 01:31:00 2012 From: kenji.rikitake@REDACTED (Kenji Rikitake) Date: Tue, 17 Jul 2012 08:31:00 +0900 Subject: [erlang-questions] Tokyo Erlang Workshop #6 27-JUL-2012 1830-2030JST Message-ID: Details: http://connpass.com/event/790/ Regards, Kenji Rikitake From dialtone@REDACTED Tue Jul 17 02:15:36 2012 From: dialtone@REDACTED (Valentino Volonghi) Date: Mon, 16 Jul 2012 17:15:36 -0700 Subject: [erlang-questions] Base64 decode binaries with ets bug? In-Reply-To: References: Message-ID: <747EA411-ECF2-48FF-A3FF-D908C7DE4EB2@gmail.com> On 13 Jul 2012, at 14:38, Valentino Volonghi wrote: > If you take the code at the bottom and run it you'll get the following outputs: Anybody can reproduce the issue in the original email? I'm running Erlang R15B01 on OSX (but it happens on linux too) 64bit. -- Valentino Volonghi http://www.adroll.com From overminddl1@REDACTED Tue Jul 17 02:22:42 2012 From: overminddl1@REDACTED (OvermindDL1) Date: Mon, 16 Jul 2012 18:22:42 -0600 Subject: [erlang-questions] Base64 decode binaries with ets bug? In-Reply-To: <747EA411-ECF2-48FF-A3FF-D908C7DE4EB2@gmail.com> References: <747EA411-ECF2-48FF-A3FF-D908C7DE4EB2@gmail.com> Message-ID: On Mon, Jul 16, 2012 at 6:15 PM, Valentino Volonghi wrote: > On 13 Jul 2012, at 14:38, Valentino Volonghi wrote: > > > If you take the code at the bottom and run it you'll get the following > outputs: > > > Anybody can reproduce the issue in the original email? > > I'm running Erlang R15B01 on OSX (but it happens on linux too) 64bit. > Replying to the list this time: I have not looked at the issue, but I did go ahead and run the code and I get identical results on R15B01 on Linux 64-bit too. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steven.charles.davis@REDACTED Tue Jul 17 02:28:10 2012 From: steven.charles.davis@REDACTED (Steve Davis) Date: Mon, 16 Jul 2012 17:28:10 -0700 (PDT) Subject: [erlang-questions] Local namespaces in case/try In-Reply-To: <6D4E0716-C2BC-4EDB-857A-257A20E83B32@erlang-solutions.com> References: <4FFBEFA5.6030305@gmail.com> <571306AB-6159-44BA-98B3-A5CB30DF0135@cs.otago.ac.nz> <6D4E0716-C2BC-4EDB-857A-257A20E83B32@erlang-solutions.com> Message-ID: <330ced82-b554-458a-bb14-0824d543f7ea@googlegroups.com> Given: X = a(b(c()) then I definitely like and usually use (but don't necessarily proscribe) the "=" It's not the end of the story here, since I remember a great lesson taught to me (by Mr Virding?) that the application code should expect valid values and that any exception (try...catch) should therefore be caught at the system boundary i.e. at c(). Which leads me to say that you should never have come to this place in your application. /s On Monday, July 16, 2012 10:19:43 AM UTC-5, Jesper Louis Andersen wrote: > > R = do_something(test2(test1(Input))). > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Tue Jul 17 03:53:50 2012 From: ok@REDACTED (Richard O'Keefe) Date: Tue, 17 Jul 2012 13:53:50 +1200 Subject: [erlang-questions] lists, binaries or something else? In-Reply-To: References: <33FAA0D4-1581-428D-8BAC-08B928E4CEC9@cs.otago.ac.nz> Message-ID: <0F993E1B-BC96-4161-BAE8-058F2FC1AB04@cs.otago.ac.nz> On 16/07/2012, at 10:40 PM, CGS wrote: > Hi Richard, > > Let me see if I understood your points correctly: > 1. Is string the right tool? Alternatives: list/iolist (even binary tree) of hashes for the chunks. > 2. To use compressed data as Joe suggested some messages ago. You're missing point 0. 0. What is the *real* problem that is to be solved? The first time anyone asked me for some statistics advice, they asked "How do I use SPSS to analyse percentage data?" The answer turned out to be "In the name of all you hold sacred, don't turn your measurements into percentages in the first place!" In the same way, when someone asks "How do I handle very large strings in language X?" I *automatically* ask "That's not your real problem. Your real problem is the *reason* that you want to handle very large strings. If you tell us what your real problem is there may be a way never to pass very large strings around in the first place." So "is string the right tool [for representing extraordinarily large character sequences" is a good question to ask, but ONLY after it has been very clearly established that passing extraordinarily large character sequences around in _some_ form is actually a good thing to do. > 1. a) Using binary tree structure is something I definitely took into account whenever is possible. Did I say "binary" tree? I know I mentioned AVL dags (which are binary but not quite trees); I also mentioned piece tables (not binary trees) and Haskell's lazy ByteStrings (who knows what inside?). Other kinds of tree are possible. If they're _that_ big, perhaps some sort of B-tree with B not too small might work. Imagine holding a tree as a 32-way B-tree of 1kB chunks and getting to any byte in a small number of memory accesses. > 2. As I said, this is something worth taking into account even if I need to find a way to define "read-only" and "read-write" characteristic for each string. > > Of course, until I have some test cases, this is only a pure discussion. So, the first step now should be to think of some test cases, I suppose. By the time you have test cases, you have already made the big choices that will limit you. I keep coming back to the question, "why move this much data around?" Might it make sense to leave the strings where they are, and send the computations back to the strings instead of sending the strings to the computations? And when I ask whether strings are the right data type, I am *NOT* asking whether list-of-character-code is the right concrete data type, I am asking whether sequence-of-character-represented-anyhow is the right *abstract* data type, whether you should be sending (parsed) XML or JSON or some sort of abstract syntax graph instead. Maybe (abstract) strings are right, but since we do not yet know what the REAL problem is, how can we tell? From siraaj@REDACTED Tue Jul 17 09:46:59 2012 From: siraaj@REDACTED (Siraaj Khandkar) Date: Tue, 17 Jul 2012 03:46:59 -0400 Subject: [erlang-questions] SSH SCP In-Reply-To: <20120716213943.17894ve1kldrmqdb@webmail.elte.hu> References: <20120716213943.17894ve1kldrmqdb@webmail.elte.hu> Message-ID: <0A16E23D-FDDD-43DA-B24B-D56BE405022E@khandkar.net> On Jul 16, 2012, at 3:39 PM, gabre@REDACTED wrote: > Hello, > > I d like to ask about the SSH module found in OTP. I would like to use > the SCP command (which uses ssh) via Erlang (SSH module). Is there > anybody, who can explain me, how to do it? I have googled and searched > million times, but nothing... (I have also tried it using os:cmd/1 and a > seld-made shell loop) I haven't gotten around to file transfers (with OTP ssh app) yet, but I did manage to mostly figure-out the basics of using the SSH app, and can give you a jump start. For the simplest example of doing something useful, lets say we want a an escript that sort-of mimics an ssh command. First, make sure you have an SSH directory (~/.ssh) and a pub-priv key pair (~/.ssh/id_dsa and ~/.ssh/id_dsa.pub) that is NOT password protected. This last part is important, because at this time the ssh app does not yet support password-protected keys [1]. After you have the key pair, the simplest implementation could look something like this: #! /usr/bin/env escript %%% ---------------------------------------------------------------------- %%% file: ssh.erl %%% ---------------------------------------------------------------------- main([User, Address, Port, Command]) -> ok = crypto:start(), ok = ssh:start(), Timeout = 5000, SSHDirectory = filename:join(os:getenv("HOME"), ".ssh"), ConnectOptions = [ {user, User}, {connect_timeout, Timeout}, {silently_accept_hosts, true}, {user_interaction, true}, {user_dir, SSHDirectory} ], case ssh:connect(Address, list_to_integer(Port), ConnectOptions) of {ok, ConnRef} -> case ssh_connection:session_channel(ConnRef, Timeout) of {ok, ChannId} -> ssh_connection:exec(ConnRef, ChannId, Command, Timeout), Data = collect_data(), io:format("~s~n", [Data]); {error, Reason} -> io:format("~p~n", [Reason]) end; {error, Reason} -> io:format("~p~n", [Reason]) end; main(_) -> io:format("USAGE: ssh.erl

~n"). collect_data() -> collect_data([]). collect_data(Data) -> receive {ssh_cm, _, {data, _, _, Datum}} -> collect_data([Datum | Data]); {ssh_cm, _, {closed, _}} -> lists:reverse(Data); {ssh_cm, _} -> collect_data(Data) end. For a more extended example of the use of the above code, see my Cluster Commander project [2], particularly commander_worker_otp.erl module. I also took advantage of OS's ssh/scp commands (as an alternative to OTP ssh and as the only, for now, way to do scp), see commander_worker_os.erl module and os_cmd/1 function in commander_lib.erl module. The above uses ssh_connection:exec/4 to just send a command string to the remote host and receive the resulting data, but, for more interesting uses, you can also use ssh_connection:send/3-5 and push any data you want to the remote host (assuming it knows how to interpret it). For a more interesting example, see Kenji Rikitake's sshrpc [3] [4] and my (uber-early and half-baked) attempts at implementing the ssh_channel behavior for my needs [5]. [1] - http://erlang.org/pipermail/erlang-questions/2010-April/050637.html [2] - https://github.com/ibnfirnas/cluster-commander [3] - https://github.com/jj1bdx/sshrpc [4] - http://www.erlang-factory.com/conference/SFBay2010/speakers/kenjirikitake [5] - https://github.com/ibnfirnas/data-mill -- Siraaj Khandkar .o. ..o ooo From zabrane3@REDACTED Tue Jul 17 14:56:40 2012 From: zabrane3@REDACTED (Zabrane Mickael) Date: Tue, 17 Jul 2012 14:56:40 +0200 Subject: [erlang-questions] [ANN] hackney 0.1 In-Reply-To: References: Message-ID: Thanks for sharing Hackney Benoit. > Let me know what you think about it, Just a simple question: what was your first motivation when you decided to start hackney? There's a couple of well tested alternatives out there: ibrowse, lhttpc, httpc ... (and for sure, others less known) Isn't it better to help to improve these well established alternatives first? Anyway, thanks for sharing. Regards, Zabrane From gordon@REDACTED Tue Jul 17 16:57:15 2012 From: gordon@REDACTED (Gordon Guthrie) Date: Tue, 17 Jul 2012 15:57:15 +0100 Subject: [erlang-questions] Long polling Message-ID: Folks I am building a softphone which needs to register with the server to say "I am available" for incoming calls. So it makes an HTTP POST request to the server and that notifies the softphone gen_srv that is is available. At that point I leave the TCP request hanging on a receive waiting for a notification that the socket has been torn down (and which point it unregisters the phone). My question is: how long can I leave the request up? Infinity? or do I need to have a timeout/reregister cycle setup? Gordon -------------- next part -------------- An HTML attachment was scrubbed... URL: From gleber.p@REDACTED Tue Jul 17 17:01:06 2012 From: gleber.p@REDACTED (Gleb Peregud) Date: Tue, 17 Jul 2012 17:01:06 +0200 Subject: [erlang-questions] Long polling In-Reply-To: References: Message-ID: On Tue, Jul 17, 2012 at 4:57 PM, Gordon Guthrie wrote: > Folks > > I am building a softphone which needs to register with the server to say "I > am available" for incoming calls. > > So it makes an HTTP POST request to the server and that notifies the > softphone gen_srv that is is available. > > At that point I leave the TCP request hanging on a receive waiting for a > notification that the socket has been torn down (and which point it > unregisters the phone). > > My question is: how long can I leave the request up? Infinity? or do I need > to have a timeout/reregister cycle setup? You have to cycle it. Some proxies timeout connections after 30 seconds, some after 45 some after 60. It is very tricky to get it right for all browsers and all platforms. Generally if you want a robust solution use something like SockJS or Socket.IO or Bullet. From kaiduanx@REDACTED Tue Jul 17 17:34:46 2012 From: kaiduanx@REDACTED (Kaiduan Xie) Date: Tue, 17 Jul 2012 11:34:46 -0400 Subject: [erlang-questions] Long polling In-Reply-To: References: Message-ID: I assume that you are talking about the TCP NAT keep-alive time. Please check the following paper, http://nutss.gforge.cis.cornell.edu/stunt.php http://saikat.guha.cc/stunt-results.php? Please look at the Timer section in the survey. Why you need to use HTTP long poll? You can just use a persistent TCP/TLS connection and periodically send keep-alive message to keep the TCP/TLS connection alive. Or you can use Websocket, it is designed to replace long poll. /Kaiduan On Tue, Jul 17, 2012 at 11:01 AM, Gleb Peregud wrote: > On Tue, Jul 17, 2012 at 4:57 PM, Gordon Guthrie wrote: >> Folks >> >> I am building a softphone which needs to register with the server to say "I >> am available" for incoming calls. >> >> So it makes an HTTP POST request to the server and that notifies the >> softphone gen_srv that is is available. >> >> At that point I leave the TCP request hanging on a receive waiting for a >> notification that the socket has been torn down (and which point it >> unregisters the phone). >> >> My question is: how long can I leave the request up? Infinity? or do I need >> to have a timeout/reregister cycle setup? > > You have to cycle it. Some proxies timeout connections after 30 > seconds, some after 45 some after 60. It is very tricky to get it > right for all browsers and all platforms. Generally if you want a > robust solution use something like SockJS or Socket.IO or Bullet. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From bchesneau@REDACTED Tue Jul 17 18:35:26 2012 From: bchesneau@REDACTED (Benoit Chesneau) Date: Tue, 17 Jul 2012 18:35:26 +0200 Subject: [erlang-questions] [ANN] hackney 0.1 In-Reply-To: References: Message-ID: On Tue, Jul 17, 2012 at 2:56 PM, Zabrane Mickael wrote: > Thanks for sharing Hackney Benoit. > >> Let me know what you think about it, > > Just a simple question: what was your first motivation when you decided to start hackney? > There's a couple of well tested alternatives out there: ibrowse, lhttpc, httpc ... (and for sure, others less known) > > Isn't it better to help to improve these well established alternatives first? > I have played the idea of writing an http client with a simple API and more support support of the possibilities offered by the HTTP (multipart, websockets, ...) since a long time. As a developer of CouchDB and other things around I have a good experience of all the HTTP clients around. But I wanted something new: no hidden state (no message passing from a process somewhere), optional socket pool, full binary parsing (it doesn't event use erlang:decode_packet/3), support for multipart in both way, sendfile API, restish, support for websockets & other coming protocols ... . So it's simpler to start a new one at this point. - beno?t From zabrane3@REDACTED Tue Jul 17 19:31:08 2012 From: zabrane3@REDACTED (Zabrane Mickael) Date: Tue, 17 Jul 2012 19:31:08 +0200 Subject: [erlang-questions] [ANN] hackney 0.1 In-Reply-To: References: Message-ID: <2CAE5C5F-D51D-43A3-8CD1-4D6EA8480FA4@gmail.com> Perfect Benoit. Good luck with Hackney, Zabrane On Jul 17, 2012, at 6:35 PM, Benoit Chesneau wrote: > On Tue, Jul 17, 2012 at 2:56 PM, Zabrane Mickael wrote: >> Thanks for sharing Hackney Benoit. >> >>> Let me know what you think about it, >> >> Just a simple question: what was your first motivation when you decided to start hackney? >> There's a couple of well tested alternatives out there: ibrowse, lhttpc, httpc ... (and for sure, others less known) >> >> Isn't it better to help to improve these well established alternatives first? >> > > I have played the idea of writing an http client with a simple API and > more support support of the possibilities offered by the HTTP > (multipart, websockets, ...) since a long time. > > As a developer of CouchDB and other things around I have a good > experience of all the HTTP clients around. But I wanted something new: > no hidden state (no message passing from a process somewhere), > optional socket pool, full binary parsing (it doesn't event use > erlang:decode_packet/3), support for multipart in both way, sendfile > API, restish, support for websockets & other coming protocols ... . > > So it's simpler to start a new one at this point. > > - beno?t From aleksandr.vin@REDACTED Tue Jul 17 20:09:16 2012 From: aleksandr.vin@REDACTED (Aleksandr Vinokurov) Date: Tue, 17 Jul 2012 22:09:16 +0400 Subject: [erlang-questions] No backtrace on errors in cowboy Message-ID: Hello all, I need to see a backtrace of a cowboy_http_protocol:request/2 and to get it I've added some code in the function (marked strong): request(_Any, State) -> %%%%%%%%% * io:format("~n~p~n", [{xxxxx161, State, _Any}]),* * try* * erlang:error(xxxxx161_error)* * catch* * error:X ->* * io:format("~nxxxxx161_error stacktrace~n~p~n", [erlang:get_stacktrace()]),* * erlang:error(X)* * end,* *%%%%%%%%%%%* error_terminate(400, State). After recompilation and reloading I see only one line in backtrace list: (web@REDACTED)1> {xxxxx161, {state,<0.85.0>,#Port<0.3809>,cowboy_tcp_transport, [{'_', [{[<<"token">>], ps_oauth_2_api_external_token_endpoint, [{config,off,off, {{127,0,0,1},8080}, {{127,0,0,1},8180}, {{127,0,0,1},8443}, {{127,0,0,1},8543}, [{"jay","123"},"bob",{"matt","321"}], "http://127.0.0.1:8881/client-storage", [{<<"apt1">>,{seconds,600,"600"}}, {<<"apt2">>,{seconds,6000,"6000"}}], {seconds,3600,"3600"}, {seconds,600,"600"}, undefined, {bytes,64}, {bytes,256}, {bytes,1024}}, {external_api_applications,ps_oauth_2_ext_api_applications}]}, {'_',ps_oauth_2_index,[external,#Fun]}]}], {ps_oauth_2_api_external_token_endpoint, [{config,off,off, {{127,0,0,1},8080}, {{127,0,0,1},8180}, {{127,0,0,1},8443}, {{127,0,0,1},8543}, [{"jay","123"},"bob",{"matt","321"}], "http://127.0.0.1:8881/client-storage", [{<<"apt1">>,{seconds,600,"600"}}, {<<"apt2">>,{seconds,6000,"6000"}}], {seconds,3600,"3600"}, {seconds,600,"600"}, undefined, {bytes,64}, {bytes,256}, {bytes,1024}}, {external_api_applications,ps_oauth_2_ext_api_applications}]}, undefined,undefined, {#Fun,crash}, 0,5,213,infinity,4096,5000, <<"authorization: Basic YXBwMToxMjM=\r\nconnection: keep-alive\r\n\r\ngrant_type=authorization_code&code=1pD8tsKZTFXw3sbbOPbLtCU0oSg5CWx%2FohR8AC%2BAuTROzsx1DQC6LLm4h6iaOvupV5cOs3cAqBqsWB%2BtzfxVtg%3D%3D">>, false,infinity,undefined}, {http_error,<<"7.0.0.1:8080\r\n">>}} (web@REDACTED)1> xxxxx161_error stacktrace [{cowboy_http_protocol,request,2, [{file,"src/cowboy_http_protocol.erl"},{line,172}]}] (web@REDACTED)1> =ERROR REPORT==== 17-Jul-2012::21:51:26 === Error in process <0.1373.0> on node 'web@REDACTED' with exit value: {xxxxx161_error,[{cowboy_http_protocol,request,2,[{file,"src/cowboy_http_protocol.erl"},{line,176}]}]} =SUPERVISOR REPORT==== 17-Jul-2012::21:51:26 === Supervisor: {<0.86.0>,cowboy_requests_sup} Context: child_terminated Reason: {xxxxx161_error, [{cowboy_http_protocol,request,2, [{file,"src/cowboy_http_protocol.erl"}, {line,176}]}]} Offender: [{pid,<0.1373.0>}, {name,cowboy_requests_sup}, {mfargs,{cowboy_requests_sup,start_request,undefined}}, {restart_type,temporary}, {shutdown,brutal_kill}, {child_type,worker}] If I call that code in shell I see a full backtrace like that: (web@REDACTED)4> try erlang:error(xxxxxREQ_error) catch error:_ -> io:format("~nxxxxxREQ_error stacktrace~n~p~n", [erlang:get_stacktrace()]) end. xxxxxREQ_error stacktrace [{erl_eval,do_apply,6,[{file,"erl_eval.erl"},{line,576}]}, {erl_eval,try_clauses,8,[{file,"erl_eval.erl"},{line,767}]}, {shell,exprs,7,[{file,"shell.erl"},{line,668}]}, {shell,eval_exprs,7,[{file,"shell.erl"},{line,623}]}, {shell,eval_loop,3,[{file,"shell.erl"},{line,608}]}] ok (web@REDACTED)5> I don't understand what am I doing wrong. -- ????????? ????????? +7 (921) 982-21-43 @aleksandrvin -------------- next part -------------- An HTML attachment was scrubbed... URL: From anthonym@REDACTED Tue Jul 17 23:48:24 2012 From: anthonym@REDACTED (Anthony Molinaro) Date: Tue, 17 Jul 2012 14:48:24 -0700 Subject: [erlang-questions] client in erlang server in objective C In-Reply-To: References: Message-ID: <20120717214824.GA41548@alumni.caltech.edu> What about thrift http://thrift.apache.org/ It's RPC style so you describe function calls, erlang clients are easy, the objective-c server would be generated for you (other than the body of the function). I've been using thrift to talk java->erlang and erlang->java, so talking erlang->objective-c should be straightforward. Some examples http://wiki.apache.org/thrift/ThriftUsageObjectiveC -Anthony On Sat, Jul 14, 2012 at 06:27:49PM +0200, Joe Armstrong wrote: > I want the following: > > - Client in erlang (easy) > - Server in objective C (I'm thinking Mac OS-X lion - xcode 4.2 - cocoa) > - socket communication > > [[ I'm a complete beginner with xcode/objective C ]] > > Has anybody written a simple objective C client that opens a port > waits for a message then executes a callback. This should be non-blocking > and be integrated with the "standard" dispatcher top-loop. > > I'm confused by the profusion of classes that almost do this. > > Has anybody implemented this, or have pointers to a good place to start. > > Cheers > > /Joe > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- ------------------------------------------------------------------------ Anthony Molinaro From mononcqc@REDACTED Wed Jul 18 04:34:57 2012 From: mononcqc@REDACTED (Fred Hebert) Date: Tue, 17 Jul 2012 22:34:57 -0400 Subject: [erlang-questions] No backtrace on errors in cowboy In-Reply-To: References: Message-ID: <500620D1.7000708@ferd.ca> First of all, you have to understand that when a request comes in cowboy, there goes a chain of command based on how you described your listener in a call to cowboy:start_listener. That chain does things like accepting requests, handling socket/port ownership, and dispatching stuff. One of the steps in there is to start a process based on the protocol you picked. In this case, the protocol is handled by cowboy_http_protocol, and the function is request. That's where your stack trace begins. When a call is made in Erlang, and that call is the last thing to happen in a function's body (it's a tail call), the stack trace for the current one will be dropped. This is to say, if I have: a() -> b(), ok. b() -> hello, c(). c() -> ok. and I'm in b(), the stack trace will be [a, b]. a() is there because b() isn't happening in last place. We thus need to store that we're part of 'a' in order to return to it in the future and execute the 'ok' expression. Then b() calls c(). Because b() ends with a call to c(), and we know that b() returns to a(), we can replace b in [a,b] and change it to [a,c]. This is good, because when we do things like recursing forever in a server loop, it keeps us from growing the stack until it crashes (stack overflow, or OOM). We drop whatever stack trace information we can afford to drop. So when your error happens, it's on line 172. This goes through: request -> parse_header -> wait_header parse_header and wait_header both happen in a tail position. This explains why your stack trace shows *nothing* but request. They're each removed from the stack trace before. That explains why you get: [{cowboy_http_protocol,request,2, [{file,"src/cowboy_http_protocol.erl"},{line,172}]}] Consider yourself lucky we now have line numbers to help! Now for the shell, when you get an error there, it looks like: [{erl_eval,do_apply,6,[{file,"erl_eval.erl"},{line,576}]}, {erl_eval,try_clauses,8,[{file,"erl_eval.erl"},{line,767}]}, {shell,exprs,7,[{file,"shell.erl"},{line,668}]}, {shell,eval_exprs,7,[{file,"shell.erl"},{line,623}]}, {shell,eval_loop,3,[{file,"shell.erl"},{line,608}]}] This is simply because shell evaluation and whatnot, for all these functions, is *not* making these calls in a tail position. This is part of how the shell works, and explains why you'll get a longer stack trace there than with compiled code. You can see part of the variation there: 1> spawn(fun() -> try 1> erlang:error(xxxxxREQ_error) 1> catch 1> error:_ -> 1> io:format("~nxxxxxREQ_error stacktrace~n~p~n", [erlang:get_stacktrace()]) 1> end 1> end). xxxxxREQ_error stacktrace [{erl_eval,do_apply,6,[{file,"erl_eval.erl"},{line,576}]}, {erl_eval,try_clauses,8,[{file,"erl_eval.erl"},{line,767}]}] <0.71.0> You'll see that for that one, part of the evaluation was done beforehand. When the new function is spawned and starts running, the 'shell' module is no longer involved, and the stack trace changes. New processes do not carry old stack information as they need not to return anywhere. So it's not surprising that you get barely nothing from cowboy when the function you get is pretty much the only one that can be allowed in the stack trace by these rules. It's brand new and the error happens after a chain of tail calls. I hope this helps. On 12-07-17 2:09 PM, Aleksandr Vinokurov wrote: > > > Hello all, > > I need to see a backtrace of a cowboy_http_protocol:request/2 and to > get it I've added some code in the function (marked strong): > > request(_Any, State) -> > %%%%%%%%% > *io:format("~n~p~n", [{xxxxx161, State, _Any}]),* > * try* > * erlang:error(xxxxx161_error)* > * catch* > * error:X ->* > * io:format("~nxxxxx161_error stacktrace~n~p~n", > [erlang:get_stacktrace()]),* > * erlang:error(X)* > * end,* > *%%%%%%%%%%%* > error_terminate(400, State). > > > After recompilation and reloading I see only one line in backtrace list: > > (web@REDACTED)1> > {xxxxx161, > {state,<0.85.0>,#Port<0.3809>,cowboy_tcp_transport, > [{'_', > [{[<<"token">>], > ps_oauth_2_api_external_token_endpoint, > [{config,off,off, > {{127,0,0,1},8080}, > {{127,0,0,1},8180}, > {{127,0,0,1},8443}, > {{127,0,0,1},8543}, > [{"jay","123"},"bob",{"matt","321"}], > "http://127.0.0.1:8881/client-storage", > [{<<"apt1">>,{seconds,600,"600"}}, > {<<"apt2">>,{seconds,6000,"6000"}}], > {seconds,3600,"3600"}, > {seconds,600,"600"}, > undefined, > {bytes,64}, > {bytes,256}, > {bytes,1024}}, > {external_api_applications,ps_oauth_2_ext_api_applications}]}, > {'_',ps_oauth_2_index,[external,#Fun]}]}], > {ps_oauth_2_api_external_token_endpoint, > [{config,off,off, > {{127,0,0,1},8080}, > {{127,0,0,1},8180}, > {{127,0,0,1},8443}, > {{127,0,0,1},8543}, > [{"jay","123"},"bob",{"matt","321"}], > "http://127.0.0.1:8881/client-storage", > [{<<"apt1">>,{seconds,600,"600"}}, > {<<"apt2">>,{seconds,6000,"6000"}}], > {seconds,3600,"3600"}, > {seconds,600,"600"}, > undefined, > {bytes,64}, > {bytes,256}, > {bytes,1024}}, > {external_api_applications,ps_oauth_2_ext_api_applications}]}, > undefined,undefined, > {#Fun,crash}, > 0,5,213,infinity,4096,5000, > <<"authorization: Basic YXBwMToxMjM=\r\nconnection: > keep-alive\r\n\r\ngrant_type=authorization_code&code=1pD8tsKZTFXw3sbbOPbLtCU0oSg5CWx%2FohR8AC%2BAuTROzsx1DQC6LLm4h6iaOvupV5cOs3cAqBqsWB%2BtzfxVtg%3D%3D">>, > false,infinity,undefined}, > {http_error,<<"7.0.0.1:8080 \r\n">>}} > > (web@REDACTED)1> > xxxxx161_error stacktrace > [{cowboy_http_protocol,request,2, > [{file,"src/cowboy_http_protocol.erl"},{line,172}]}] > (web@REDACTED)1> > > =ERROR REPORT==== 17-Jul-2012::21:51:26 === > Error in process <0.1373.0> on node 'web@REDACTED' with exit value: > {xxxxx161_error,[{cowboy_http_protocol,request,2,[{file,"src/cowboy_http_protocol.erl"},{line,176}]}]} > > =SUPERVISOR REPORT==== 17-Jul-2012::21:51:26 === > Supervisor: {<0.86.0>,cowboy_requests_sup} > Context: child_terminated > Reason: {xxxxx161_error, > [{cowboy_http_protocol,request,2, > [{file,"src/cowboy_http_protocol.erl"}, > {line,176}]}]} > Offender: [{pid,<0.1373.0>}, > {name,cowboy_requests_sup}, > {mfargs,{cowboy_requests_sup,start_request,undefined}}, > {restart_type,temporary}, > {shutdown,brutal_kill}, > {child_type,worker}] > > If I call that code in shell I see a full backtrace like that: > > (web@REDACTED)4> try > erlang:error(xxxxxREQ_error) > catch > error:_ -> > io:format("~nxxxxxREQ_error stacktrace~n~p~n", > [erlang:get_stacktrace()]) > end. > > xxxxxREQ_error stacktrace > [{erl_eval,do_apply,6,[{file,"erl_eval.erl"},{line,576}]}, > {erl_eval,try_clauses,8,[{file,"erl_eval.erl"},{line,767}]}, > {shell,exprs,7,[{file,"shell.erl"},{line,668}]}, > {shell,eval_exprs,7,[{file,"shell.erl"},{line,623}]}, > {shell,eval_loop,3,[{file,"shell.erl"},{line,608}]}] > ok > (web@REDACTED)5> > > I don't understand what am I doing wrong. > > -- > ????????? ????????? > +7 (921) 982-21-43 > @aleksandrvin > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From aleksandr.vin@REDACTED Wed Jul 18 05:37:00 2012 From: aleksandr.vin@REDACTED (Aleksandr Vinokurov) Date: Wed, 18 Jul 2012 07:37:00 +0400 Subject: [erlang-questions] No backtrace on errors in cowboy In-Reply-To: <500620D1.7000708@ferd.ca> References: <500620D1.7000708@ferd.ca> Message-ID: <2BEC2110-2C70-47E1-9095-CA3A717BEFB7@gmail.com> Thanks, Fred That describe everything. Wish I suggested tail recursion yesterday. -- Aleksandr 18.07.2012, ? 6:34, Fred Hebert ???????(?): > First of all, you have to understand that when a request comes in cowboy, there goes a chain of command based on how you described your listener in a call to cowboy:start_listener. That chain does things like accepting requests, handling socket/port ownership, and dispatching stuff. > > One of the steps in there is to start a process based on the protocol you picked. In this case, the protocol is handled by cowboy_http_protocol, and the function is request. > > That's where your stack trace begins. > > When a call is made in Erlang, and that call is the last thing to happen in a function's body (it's a tail call), the stack trace for the current one will be dropped. This is to say, if I have: > > a() -> b(), ok. > b() -> hello, c(). > c() -> ok. > > and I'm in b(), the stack trace will be [a, b]. a() is there because b() isn't happening in last place. We thus need to store that we're part of 'a' in order to return to it in the future and execute the 'ok' expression. Then b() calls c(). > > Because b() ends with a call to c(), and we know that b() returns to a(), we can replace b in [a,b] and change it to [a,c]. This is good, because when we do things like recursing forever in a server loop, it keeps us from growing the stack until it crashes (stack overflow, or OOM). We drop whatever stack trace information we can afford to drop. > > So when your error happens, it's on line 172. > > This goes through: > > request -> parse_header -> wait_header > > parse_header and wait_header both happen in a tail position. This explains why your stack trace shows nothing but request. They're each removed from the stack trace before. That explains why you get: > > [{cowboy_http_protocol,request,2, > [{file,"src/cowboy_http_protocol.erl"},{line,172}]}] > > Consider yourself lucky we now have line numbers to help! > > Now for the shell, when you get an error there, it looks like: > > [{erl_eval,do_apply,6,[{file,"erl_eval.erl"},{line,576}]}, > {erl_eval,try_clauses,8,[{file,"erl_eval.erl"},{line,767}]}, > {shell,exprs,7,[{file,"shell.erl"},{line,668}]}, > {shell,eval_exprs,7,[{file,"shell.erl"},{line,623}]}, > {shell,eval_loop,3,[{file,"shell.erl"},{line,608}]}] > > This is simply because shell evaluation and whatnot, for all these functions, is not making these calls in a tail position. This is part of how the shell works, and explains why you'll get a longer stack trace there than with compiled code. > > You can see part of the variation there: > > 1> spawn(fun() -> try > 1> erlang:error(xxxxxREQ_error) > 1> catch > 1> error:_ -> > 1> io:format("~nxxxxxREQ_error stacktrace~n~p~n", [erlang:get_stacktrace()]) > 1> end > 1> end). > > xxxxxREQ_error stacktrace > [{erl_eval,do_apply,6,[{file,"erl_eval.erl"},{line,576}]}, > {erl_eval,try_clauses,8,[{file,"erl_eval.erl"},{line,767}]}] > <0.71.0> > > You'll see that for that one, part of the evaluation was done beforehand. When the new function is spawned and starts running, the 'shell' module is no longer involved, and the stack trace changes. New processes do not carry old stack information as they need not to return anywhere. > > So it's not surprising that you get barely nothing from cowboy when the function you get is pretty much the only one that can be allowed in the stack trace by these rules. It's brand new and the error happens after a chain of tail calls. > > I hope this helps. > > On 12-07-17 2:09 PM, Aleksandr Vinokurov wrote: >> >> >> Hello all, >> >> I need to see a backtrace of a cowboy_http_protocol:request/2 and to get it I've added some code in the function (marked strong): >> >> request(_Any, State) -> >> %%%%%%%%% >> io:format("~n~p~n", [{xxxxx161, State, _Any}]), >> try >> erlang:error(xxxxx161_error) >> catch >> error:X -> >> io:format("~nxxxxx161_error stacktrace~n~p~n", [erlang:get_stacktrace()]), >> erlang:error(X) >> end, >> %%%%%%%%%%% >> error_terminate(400, State). >> >> >> After recompilation and reloading I see only one line in backtrace list: >> >> (web@REDACTED)1> >> {xxxxx161, >> {state,<0.85.0>,#Port<0.3809>,cowboy_tcp_transport, >> [{'_', >> [{[<<"token">>], >> ps_oauth_2_api_external_token_endpoint, >> [{config,off,off, >> {{127,0,0,1},8080}, >> {{127,0,0,1},8180}, >> {{127,0,0,1},8443}, >> {{127,0,0,1},8543}, >> [{"jay","123"},"bob",{"matt","321"}], >> "http://127.0.0.1:8881/client-storage", >> [{<<"apt1">>,{seconds,600,"600"}}, >> {<<"apt2">>,{seconds,6000,"6000"}}], >> {seconds,3600,"3600"}, >> {seconds,600,"600"}, >> undefined, >> {bytes,64}, >> {bytes,256}, >> {bytes,1024}}, >> {external_api_applications,ps_oauth_2_ext_api_applications}]}, >> {'_',ps_oauth_2_index,[external,#Fun]}]}], >> {ps_oauth_2_api_external_token_endpoint, >> [{config,off,off, >> {{127,0,0,1},8080}, >> {{127,0,0,1},8180}, >> {{127,0,0,1},8443}, >> {{127,0,0,1},8543}, >> [{"jay","123"},"bob",{"matt","321"}], >> "http://127.0.0.1:8881/client-storage", >> [{<<"apt1">>,{seconds,600,"600"}}, >> {<<"apt2">>,{seconds,6000,"6000"}}], >> {seconds,3600,"3600"}, >> {seconds,600,"600"}, >> undefined, >> {bytes,64}, >> {bytes,256}, >> {bytes,1024}}, >> {external_api_applications,ps_oauth_2_ext_api_applications}]}, >> undefined,undefined, >> {#Fun,crash}, >> 0,5,213,infinity,4096,5000, >> <<"authorization: Basic YXBwMToxMjM=\r\nconnection: keep-alive\r\n\r\ngrant_type=authorization_code&code=1pD8tsKZTFXw3sbbOPbLtCU0oSg5CWx%2FohR8AC%2BAuTROzsx1DQC6LLm4h6iaOvupV5cOs3cAqBqsWB%2BtzfxVtg%3D%3D">>, >> false,infinity,undefined}, >> {http_error,<<"7.0.0.1:8080\r\n">>}} >> >> (web@REDACTED)1> >> xxxxx161_error stacktrace >> [{cowboy_http_protocol,request,2, >> [{file,"src/cowboy_http_protocol.erl"},{line,172}]}] >> (web@REDACTED)1> >> >> =ERROR REPORT==== 17-Jul-2012::21:51:26 === >> Error in process <0.1373.0> on node 'web@REDACTED' with exit value: {xxxxx161_error,[{cowboy_http_protocol,request,2,[{file,"src/cowboy_http_protocol.erl"},{line,176}]}]} >> >> =SUPERVISOR REPORT==== 17-Jul-2012::21:51:26 === >> Supervisor: {<0.86.0>,cowboy_requests_sup} >> Context: child_terminated >> Reason: {xxxxx161_error, >> [{cowboy_http_protocol,request,2, >> [{file,"src/cowboy_http_protocol.erl"}, >> {line,176}]}]} >> Offender: [{pid,<0.1373.0>}, >> {name,cowboy_requests_sup}, >> {mfargs,{cowboy_requests_sup,start_request,undefined}}, >> {restart_type,temporary}, >> {shutdown,brutal_kill}, >> {child_type,worker}] >> >> If I call that code in shell I see a full backtrace like that: >> >> (web@REDACTED)4> try >> erlang:error(xxxxxREQ_error) >> catch >> error:_ -> >> io:format("~nxxxxxREQ_error stacktrace~n~p~n", [erlang:get_stacktrace()]) >> end. >> >> xxxxxREQ_error stacktrace >> [{erl_eval,do_apply,6,[{file,"erl_eval.erl"},{line,576}]}, >> {erl_eval,try_clauses,8,[{file,"erl_eval.erl"},{line,767}]}, >> {shell,exprs,7,[{file,"shell.erl"},{line,668}]}, >> {shell,eval_exprs,7,[{file,"shell.erl"},{line,623}]}, >> {shell,eval_loop,3,[{file,"shell.erl"},{line,608}]}] >> ok >> (web@REDACTED)5> >> >> I don't understand what am I doing wrong. >> >> -- >> ????????? ????????? >> +7 (921) 982-21-43 >> @aleksandrvin >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eduard.sergeev@REDACTED Wed Jul 18 07:26:49 2012 From: eduard.sergeev@REDACTED (Eduard Sergeev) Date: Tue, 17 Jul 2012 22:26:49 -0700 (PDT) Subject: [erlang-questions] Reltool error: "Module potentially included by two different applications" In-Reply-To: References: Message-ID: <792292ca-3755-443e-bab2-c5f7a5faa0c0@googlegroups.com> Hi Siri, Thank you for responding to my post! Since you haven't included your reltool config file > Fair enough. To simplify things I've now created a simple project which illustrates my first post with a real code: https://github.com/EduardSergeev/potentially_included/tree/error If you have *rebar* installed you can build the entire thing with: *rebar clear compile *then *cd* to *./rel* and do *rebar generate -f* (runs reltool) which will demonstrate the original error. This project basically contains only two custom apps 'main' and 'library' where 'main' uses (depends) on 'library' and 'library' also includes dummy module 'eunit_lib' - to simulate a duplication with the original module from the system app 'eunit'. Apart from that 'main' also includes some other standard apps to make it more 'real' (but *not* 'eunit'). > I'm just guessing that it looks (maybe simplified) something like this: > {sys,[{lib_dirs,[""]}, > {incl_cond,derived}, > {app,,[{incl_cond,include}]} > ]}. > Pretty much so: https://github.com/EduardSergeev/potentially_included/blob/error/rel/reltool.config (with some rebar flavour in the form of '{target_dir, "main"}') > and that exists in . > Not exactly. The is only happened to be installed on my local 'target system' but is not a part of this project, so it is not in , but it is apparently included by reltool from my `code:root_dir/0` due to default settings {incl_cond, derived} on 'sys' level. At least this is my understanding. > If this is close to correct, then what you ask reltool to do is > > "Take my erlang installation and also look in and figure out > which applications and modules I need to include in my release so that all > dependencies of are taken care of." > > And to be very strict - it could(!!) be that you want to use the > from and not from itself... but I > realize that it is not very likely. At least not if is the > only relation between and ... So the question is if > this is ALWAYS the case?? > Yes, now after your explanation I think I understand that if the application from both `{sys,[{lib_dirs,[""]} ..]}` and from target system (`code:root_dir/0`) has the same 'priorities' there is no way reltool can be sure which app to include from "lib_dirs" or from the target system. The current error definitelly makes sense here. > A simple workaround for the above config file would be to skip the system > level lib_dirs parameter, and specify directory per application instead: > > {sys,[{incl_cond,derived}, > {app,,[{incl_cond,include},{lib_dir,"/"]} > ]}. > > This will of course only work as long as there are no other dependecies > from towards other applications in ... > Hm. First of all this option seems to available only from R15B01 (the latest release as of today) and we are yet to switch to this version. But anyway, I did some experiments with this approach and probably I misunderstood you suggestion but I failed to make reltool to build the release via this workaround... You can see my understanding of your suggestion here: https://github.com/EduardSergeev/potentially_included/blob/proposed_solution/rel/reltool.config But when I try to build this configuration it throws the error: *{'EXIT',{{badmatch,{error,"library: Application version clash. Multiple directories contains version \"0.2.0\"."}},* Not sure why. So I am still see only two ways to solve this problem described in my first post: 1. The bad one: https://github.com/EduardSergeev/potentially_included/blob/first_solution/rel/reltool.config 2. And the verbose one: https://github.com/EduardSergeev/potentially_included/blob/second_solution/rel/reltool.config So unless I got your workaround wrong (please correct my 'reltool.config' if this is the case!) I think that the only option for me is still 2. - to explicitly specify all dependent apps. I guess the feature which I would ideally like to see in reltool is an ability to somehow *prioritise* the over target system: i.e. when searching to an app which includes a specific , first look in and only then if not found in there look in 'target system' (`code:root_dir/0`) dir. Exactly as now, if a given is included in more then one app from - throw the error but if there is only one app in which has this just use this app even if another from 'target system' includes this too (well, unless this is included for any reason). Apart from being obviously more complicated than the current process wouldn't this approach work? Or still maybe there is a simpler way to solve the original problem? Thanks! Eduard PS By the way, it looks that the lines 907-909 of the current revision of 'reltool_server' will cause `bad argument` should they ever be called: https://github.com/erlang/otp/blob/maint/lib/reltool/src/reltool_server.erl#L907-909 -------------- next part -------------- An HTML attachment was scrubbed... URL: From cloudzen@REDACTED Wed Jul 18 08:27:38 2012 From: cloudzen@REDACTED (skyman) Date: Wed, 18 Jul 2012 14:27:38 +0800 (CST) Subject: [erlang-questions] prim_inet:async_recv() received '{inet_async, Socket, Ref, {error, etimedout}}' message Message-ID: <65050f6c.1ed90.13898c539c2.Coremail.cloudzen@163.com> Hi all, Below is the code snippet of receive and parse packets from client: async_recv(Sock, Timeout) when is_port(Sock) -> case prim_inet:async_recv(Sock, 0, Timeout) of {error, Reason} -> throw({Reason}); {ok, Res} -> Res; Res -> Res end. do_parse_packet(Socket, Client) -> Ref = async_recv(Socket, 60000), receive {inet_async, Socket, Ref, {ok, <>}} -> ... {inet_async, Socket, Ref, {error,etimedout}} -> io:format("Error: etimedout~n") end. Sometimes it receives '{inet_async, Socket, Ref, {error,etimedout}} ' message, what does '{error,etimedout}' mean? Thanks in advance! -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang@REDACTED Wed Jul 18 10:10:39 2012 From: erlang@REDACTED (Joe Armstrong) Date: Wed, 18 Jul 2012 10:10:39 +0200 Subject: [erlang-questions] client in erlang server in objective C In-Reply-To: <20120717214824.GA41548@alumni.caltech.edu> References: <20120717214824.GA41548@alumni.caltech.edu> Message-ID: That's a very interesting idea. I haven't used thrift - but if the server side code exists in Objective C this would be very interesting. (I tried to build thrift but the build failed :-) I've now actually got an erlang client talking to an objective C server - using AsnycSocket that Bob suggested but I can only exchange strings. Now I need to figure out how to do build dynamic method calls in objective C. In objective C I'd write [button setTitle:@"click me"] In Erlang I'd like to encode this as a string send it to objective C decode it and evaluate it. I'd like to do something like send("button", "setTitle", [{string,"click me"}]) In erlang or to encode [foo this:123 that:@"yea"] as send("foo", "this:that", [{integer,123},{string,"yea"}]) Then I'd like to serialize this as a string (in Erlang) and decode it and evaluate it in Objective C Any ideas how to do this? /Joe On Tue, Jul 17, 2012 at 11:48 PM, Anthony Molinaro wrote: > What about thrift http://thrift.apache.org/ > > It's RPC style so you describe function calls, erlang clients are easy, > the objective-c server would be generated for you (other than the > body of the function). > > I've been using thrift to talk java->erlang and erlang->java, so > talking erlang->objective-c should be straightforward. > > Some examples http://wiki.apache.org/thrift/ThriftUsageObjectiveC > > -Anthony > > On Sat, Jul 14, 2012 at 06:27:49PM +0200, Joe Armstrong wrote: >> I want the following: >> >> - Client in erlang (easy) >> - Server in objective C (I'm thinking Mac OS-X lion - xcode 4.2 - cocoa) >> - socket communication >> >> [[ I'm a complete beginner with xcode/objective C ]] >> >> Has anybody written a simple objective C client that opens a port >> waits for a message then executes a callback. This should be non-blocking >> and be integrated with the "standard" dispatcher top-loop. >> >> I'm confused by the profusion of classes that almost do this. >> >> Has anybody implemented this, or have pointers to a good place to start. >> >> Cheers >> >> /Joe >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > -- > ------------------------------------------------------------------------ > Anthony Molinaro From dm.klionsky@REDACTED Wed Jul 18 10:42:30 2012 From: dm.klionsky@REDACTED (Dmitry Klionsky) Date: Wed, 18 Jul 2012 11:42:30 +0300 Subject: [erlang-questions] client in erlang server in objective C In-Reply-To: References: <20120717214824.GA41548@alumni.caltech.edu> Message-ID: <500676F6.6020207@gmail.com> Hi Joe, NSSelectorFromString, NSClassFromString and others are your friends here. For more detail have a look at: Objective-C Runtime Programming Guide https://developer.apple.com/library/mac/#documentation/Cocoa/Conceptual/ObjCRuntimeGuide/Introduction/Introduction.html Objective-C Programming Language https://developer.apple.com/library/mac/#documentation/Cocoa/Conceptual/ObjectiveC/Introduction/introObjectiveC.html Also I would also suggest you to evaluate the protobuffs. In my opinion the protobuffs are more concise and clear. There a couple of Objective-C implementations available: http://code.google.com/p/metasyntactic/wiki/ProtocolBuffers https://github.com/booyah/protobuf-objc Best regards, Dmitry Klionsky On 07/18/2012 11:10 AM, Joe Armstrong wrote: > That's a very interesting idea. I haven't used thrift - but if the server > side code exists in Objective C this would be very interesting. > > (I tried to build thrift but the build failed :-) > > > I've now actually got an erlang client talking to an objective C > server - using AsnycSocket that Bob suggested but I can only exchange strings. > > Now I need to figure out how to do build dynamic method calls in objective C. > > In objective C I'd write > > [button setTitle:@"click me"] > > In Erlang I'd like to encode this as a string > send it to objective C decode it and evaluate it. > > I'd like to do something like > > send("button", "setTitle", [{string,"click me"}]) > > In erlang > > or to encode [foo this:123 that:@"yea"] as > > send("foo", "this:that", [{integer,123},{string,"yea"}]) > > Then I'd like to serialize this as a string (in Erlang) and decode it > and evaluate it in Objective C > > Any ideas how to do this? > > /Joe > > > On Tue, Jul 17, 2012 at 11:48 PM, Anthony Molinaro > wrote: >> What about thrift http://thrift.apache.org/ >> >> It's RPC style so you describe function calls, erlang clients are easy, >> the objective-c server would be generated for you (other than the >> body of the function). >> >> I've been using thrift to talk java->erlang and erlang->java, so >> talking erlang->objective-c should be straightforward. >> >> Some examples http://wiki.apache.org/thrift/ThriftUsageObjectiveC >> >> -Anthony >> >> On Sat, Jul 14, 2012 at 06:27:49PM +0200, Joe Armstrong wrote: >>> I want the following: >>> >>> - Client in erlang (easy) >>> - Server in objective C (I'm thinking Mac OS-X lion - xcode 4.2 - cocoa) >>> - socket communication >>> >>> [[ I'm a complete beginner with xcode/objective C ]] >>> >>> Has anybody written a simple objective C client that opens a port >>> waits for a message then executes a callback. This should be non-blocking >>> and be integrated with the "standard" dispatcher top-loop. >>> >>> I'm confused by the profusion of classes that almost do this. >>> >>> Has anybody implemented this, or have pointers to a good place to start. >>> >>> Cheers >>> >>> /Joe >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >> -- >> ------------------------------------------------------------------------ >> Anthony Molinaro > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- Best regards, Dmitry Klionsky From dm.klionsky@REDACTED Wed Jul 18 12:07:55 2012 From: dm.klionsky@REDACTED (Dmitry Klionsky) Date: Wed, 18 Jul 2012 13:07:55 +0300 Subject: [erlang-questions] client in erlang server in objective C In-Reply-To: <500676F6.6020207@gmail.com> References: <20120717214824.GA41548@alumni.caltech.edu> <500676F6.6020207@gmail.com> Message-ID: <50068AFB.5070103@gmail.com> Decided to help you a little further: The problem is you can't just send("button", ...), because the "button" is not a symbol, but a NSButton's class (or something like this) instance. I'm not sure because I've never done any MacOX programming, but iOS. And in order to send a message to an instance you need to know it. So, in general, your workflow should be like this: Erlang side: %% create button send {create_instance, {string, "NSButton"}} receive {ok, Instance} %% set button's title send {perform, {integer, Button} {string, "setTitle:"}, {string, "click me"}} receive ok Objective-C side On receiving {create_instance, {string, Clazz}}: // note that `Class' is a reserved word id instance = [[NSClassFromString(Clazz) alloc] init]; if (instance) { reply {ok, instance}; } else { reply {error, @"No such class"} } On receiving {perform, {integer, Instance}, {string, Selector}, {string, Object}}: SEL sel = NSSelectorFromString(Selector); if (sel) { if ([Instance respondsToSelector(sel)]) { [Instance performSelector:sel withObject:Object]; reply ok; } else { reply {error, @"No such selector"}; } } else { reply {error, @"Invalid selector"} } This was a little messy, but hopefully you will get the idea. On 07/18/2012 11:42 AM, Dmitry Klionsky wrote: > Hi Joe, > > NSSelectorFromString, NSClassFromString and others are your friends here. > > For more detail have a look at: > > Objective-C Runtime Programming Guide > https://developer.apple.com/library/mac/#documentation/Cocoa/Conceptual/ObjCRuntimeGuide/Introduction/Introduction.html > > > Objective-C Programming Language > https://developer.apple.com/library/mac/#documentation/Cocoa/Conceptual/ObjectiveC/Introduction/introObjectiveC.html > > > Also I would also suggest you to evaluate the protobuffs. In my > opinion the protobuffs are more concise and clear. > There a couple of Objective-C implementations available: > http://code.google.com/p/metasyntactic/wiki/ProtocolBuffers > https://github.com/booyah/protobuf-objc > > > Best regards, > Dmitry Klionsky > > > On 07/18/2012 11:10 AM, Joe Armstrong wrote: >> That's a very interesting idea. I haven't used thrift - but if the >> server >> side code exists in Objective C this would be very interesting. >> >> (I tried to build thrift but the build failed :-) >> >> >> I've now actually got an erlang client talking to an objective C >> server - using AsnycSocket that Bob suggested but I can only exchange >> strings. >> >> Now I need to figure out how to do build dynamic method calls in >> objective C. >> >> In objective C I'd write >> >> [button setTitle:@"click me"] >> >> In Erlang I'd like to encode this as a string >> send it to objective C decode it and evaluate it. >> >> I'd like to do something like >> >> send("button", "setTitle", [{string,"click me"}]) >> >> In erlang >> >> or to encode [foo this:123 that:@"yea"] as >> >> send("foo", "this:that", [{integer,123},{string,"yea"}]) >> >> Then I'd like to serialize this as a string (in Erlang) and decode it >> and evaluate it in Objective C >> >> Any ideas how to do this? >> >> /Joe >> >> >> On Tue, Jul 17, 2012 at 11:48 PM, Anthony Molinaro >> wrote: >>> What about thrift http://thrift.apache.org/ >>> >>> It's RPC style so you describe function calls, erlang clients are easy, >>> the objective-c server would be generated for you (other than the >>> body of the function). >>> >>> I've been using thrift to talk java->erlang and erlang->java, so >>> talking erlang->objective-c should be straightforward. >>> >>> Some examples http://wiki.apache.org/thrift/ThriftUsageObjectiveC >>> >>> -Anthony >>> >>> On Sat, Jul 14, 2012 at 06:27:49PM +0200, Joe Armstrong wrote: >>>> I want the following: >>>> >>>> - Client in erlang (easy) >>>> - Server in objective C (I'm thinking Mac OS-X lion - xcode 4.2 >>>> - cocoa) >>>> - socket communication >>>> >>>> [[ I'm a complete beginner with xcode/objective C ]] >>>> >>>> Has anybody written a simple objective C client that opens a port >>>> waits for a message then executes a callback. This should be >>>> non-blocking >>>> and be integrated with the "standard" dispatcher top-loop. >>>> >>>> I'm confused by the profusion of classes that almost do this. >>>> >>>> Has anybody implemented this, or have pointers to a good place to >>>> start. >>>> >>>> Cheers >>>> >>>> /Joe >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>> -- >>> ------------------------------------------------------------------------ >>> >>> Anthony Molinaro >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > -- Best regards, Dmitry Klionsky From erlang@REDACTED Wed Jul 18 14:16:42 2012 From: erlang@REDACTED (Joe Armstrong) Date: Wed, 18 Jul 2012 14:16:42 +0200 Subject: [erlang-questions] client in erlang server in objective C In-Reply-To: <50068AFB.5070103@gmail.com> References: <20120717214824.GA41548@alumni.caltech.edu> <500676F6.6020207@gmail.com> <50068AFB.5070103@gmail.com> Message-ID: On Wed, Jul 18, 2012 at 12:07 PM, Dmitry Klionsky wrote: > Decided to help you a little further: Thanks - I appreciate this ... I have a few more questions (inline) > > The problem is you can't just send("button", ...), because the "button" is > not a symbol, but a NSButton's class (or something like this) instance. > I'm not sure because I've never done any MacOX programming, but iOS. > And in order to send a message to an instance you need to know it. So, in > general, your workflow should be like this: > > Erlang side: > %% create button > send > {create_instance, {string, "NSButton"}} > receive > {ok, Instance} question 1: Is Instance an integer here? (see question 3) > > %% set button's title > send > {perform, {integer, Button} {string, "setTitle:"}, {string, "click me"}} Button is the integer in "Instance" above I assume > receive > ok > > > Objective-C side > > On receiving {create_instance, {string, Clazz}}: > > // note that `Class' is a reserved word > id instance = [[NSClassFromString(Clazz) alloc] init]; > if (instance) { What is instance? is this "really" a 32 bit pointer? How do I serialize it to send it back to erlang? > reply {ok, instance}; > } else { > reply {error, @"No such class"} > } > > On receiving {perform, {integer, Instance}, {string, Selector}, {string, > Object}}: > SEL sel = NSSelectorFromString(Selector); > if (sel) { > if ([Instance respondsToSelector(sel)]) { > [Instance performSelector:sel withObject:Object]; > reply ok; > } else { > reply {error, @"No such selector"}; > } > } else { > reply {error, @"Invalid selector"} > } Same question here. On the "wire" I'll see an encoding of {perform, {integer, Instance}, ... and so on How do I make an Objective C id or NSString from a sequence of bytes that I read from a socket. Also do I have to worry about strings (for example) being garbage collected. For example If I say (in objective C) NSString s = @"hello" I create a literal string in the variable s But if I receive a string "hello" somewhere in a buffer from a socket I need to create a NSString and put it somewhere so it doesn't get garbage collected away. Something like: NSMutableArray *stringtable = [[NSMutableArray alloc] init]; NSString s = [[NSString alloc] initWithString: Data] [stringtable addObject:s]; (Pardon my Objective C here - I'm a total beginner here) > This was a little messy, but hopefully you will get the idea. Yes - I get the idea, the problem is the nitty gritty details, not the idea. The idea is easy. Allocate some memory to store the stuff you received in create some dynamic objects and call some methods - problem is how. Thanks - I'll try this later /Joe > > > > On 07/18/2012 11:42 AM, Dmitry Klionsky wrote: >> >> Hi Joe, >> >> NSSelectorFromString, NSClassFromString and others are your friends here. >> >> For more detail have a look at: >> >> Objective-C Runtime Programming Guide >> >> https://developer.apple.com/library/mac/#documentation/Cocoa/Conceptual/ObjCRuntimeGuide/Introduction/Introduction.html >> >> Objective-C Programming Language >> >> https://developer.apple.com/library/mac/#documentation/Cocoa/Conceptual/ObjectiveC/Introduction/introObjectiveC.html >> >> Also I would also suggest you to evaluate the protobuffs. In my opinion >> the protobuffs are more concise and clear. >> There a couple of Objective-C implementations available: >> http://code.google.com/p/metasyntactic/wiki/ProtocolBuffers >> https://github.com/booyah/protobuf-objc >> >> >> Best regards, >> Dmitry Klionsky >> >> >> On 07/18/2012 11:10 AM, Joe Armstrong wrote: >>> >>> That's a very interesting idea. I haven't used thrift - but if the server >>> side code exists in Objective C this would be very interesting. >>> >>> (I tried to build thrift but the build failed :-) >>> >>> >>> I've now actually got an erlang client talking to an objective C >>> server - using AsnycSocket that Bob suggested but I can only exchange >>> strings. >>> >>> Now I need to figure out how to do build dynamic method calls in >>> objective C. >>> >>> In objective C I'd write >>> >>> [button setTitle:@"click me"] >>> >>> In Erlang I'd like to encode this as a string >>> send it to objective C decode it and evaluate it. >>> >>> I'd like to do something like >>> >>> send("button", "setTitle", [{string,"click me"}]) >>> >>> In erlang >>> >>> or to encode [foo this:123 that:@"yea"] as >>> >>> send("foo", "this:that", [{integer,123},{string,"yea"}]) >>> >>> Then I'd like to serialize this as a string (in Erlang) and decode it >>> and evaluate it in Objective C >>> >>> Any ideas how to do this? >>> >>> /Joe >>> >>> >>> On Tue, Jul 17, 2012 at 11:48 PM, Anthony Molinaro >>> wrote: >>>> >>>> What about thrift http://thrift.apache.org/ >>>> >>>> It's RPC style so you describe function calls, erlang clients are easy, >>>> the objective-c server would be generated for you (other than the >>>> body of the function). >>>> >>>> I've been using thrift to talk java->erlang and erlang->java, so >>>> talking erlang->objective-c should be straightforward. >>>> >>>> Some examples http://wiki.apache.org/thrift/ThriftUsageObjectiveC >>>> >>>> -Anthony >>>> >>>> On Sat, Jul 14, 2012 at 06:27:49PM +0200, Joe Armstrong wrote: >>>>> >>>>> I want the following: >>>>> >>>>> - Client in erlang (easy) >>>>> - Server in objective C (I'm thinking Mac OS-X lion - xcode 4.2 - >>>>> cocoa) >>>>> - socket communication >>>>> >>>>> [[ I'm a complete beginner with xcode/objective C ]] >>>>> >>>>> Has anybody written a simple objective C client that opens a port >>>>> waits for a message then executes a callback. This should be >>>>> non-blocking >>>>> and be integrated with the "standard" dispatcher top-loop. >>>>> >>>>> I'm confused by the profusion of classes that almost do this. >>>>> >>>>> Has anybody implemented this, or have pointers to a good place to >>>>> start. >>>>> >>>>> Cheers >>>>> >>>>> /Joe >>>>> _______________________________________________ >>>>> erlang-questions mailing list >>>>> erlang-questions@REDACTED >>>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> >>>> -- >>>> ------------------------------------------------------------------------ >>>> Anthony Molinaro >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> > > > -- > Best regards, > Dmitry Klionsky > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From mononcqc@REDACTED Wed Jul 18 14:26:28 2012 From: mononcqc@REDACTED (Fred Hebert) Date: Wed, 18 Jul 2012 08:26:28 -0400 Subject: [erlang-questions] UDP sockets and gen_server are hypocritical and it makes me mad Message-ID: <5006AB74.80804@ferd.ca> I'm having the weirdest of issues right now. If you bear with me for the read, you'll know what I mean. I recently pushed an experimental version of vmstats online on our production servers, and it mostly works fine. The code in question is a gen_server sending itself messages here: https://github.com/ferd/vmstats/blob/master/src/vmstats_server.erl And it sends through UDP sockets with the help of https://github.com/lpgauth/statsderl/blob/master/src/statsderl.erl Now I said it 'mostly' works fine. The problem is that after a few hours, two of the servers' statistics related to vmstats go dead. By that, I mean that in our reports, all I see is a flat line (see http://i.imgur.com/LIJ1J.png, where h019 stops sending data near 19:16) There is no error log, no exception, everything else runs fine. ngrep shows nothing coming from vmstats on UDP, but other parts of the server stack keep sending data through the UDP port fine. This is where I would suspect things to be related to the server being locked up, but, here's the weird part. As soon as I connect to the virtual machine (^G -> r 'nodename@REDACTED' ) to inspect the vmstats_server, data starts pouring in again (this is what happens on the previous image at 19:28). Any mere attempt to connect to the node to understand what the problem is causes it to be resolved. I had the bug happen to me again a bit past 4 this morning. When I unstuck it around 8, I got the following data on the next few loops of the server (caught with "ngrep 'vmstats.scheduler_wall_time.1\.' udp"): ##### 1 vmstats.scheduler_wall_time.1.active: 189374595 vmstats.scheduler_wall_time.1.total: 1002022817 ##### 2 vmstats.scheduler_wall_time.1.active:2293308912394 vmstats.scheduler_wall_time.1.total:12460747343114 ##### 3 vmstats.scheduler_wall_time.1.active:186326615 vmstats.scheduler_wall_time.1.total:1004246720 The interesting thing with the numbers above is that the server loops every second or so. The numbers are coming from a subtractions in scheduler_wall_time values as of R15B01 between slices of 1 second for each loop. Thus, the values '1002022817' and '1004246720' are averaging the equivalent of one second (the Erlang VM says the unit is undefined). If I make a ratio of them with '12460747343114' (the second value), I get 12435 seconds, or roughly 3h45, equivalent to the time the server sent nothing. I sadly have no absolute timestamp. This tells me that when things get stuck, the gen_server itself stops polling (or at least pushing data), up until the point I connect to the virtual machine, and it gets back to work. This wouldn't be related to UDP. Hypocritical thing. It takes breaks until I notice it then goes back to work. I'm running R15B01, only 2 out of 6 servers we deployed it to show this issue (so far), it happens a couple of times a day in unpredictable manners. Can anyone spot an obvious bug in the gen_server? Can this be some weird scheduling behavior? Or maybe a bug with erlang:statistics(scheduler_wall_time)? Anyone else ever had similar issues? (should I have sent this to erlang-bugs?) Thanks for your help, Fred. From gianfranco.alongi@REDACTED Wed Jul 18 14:29:23 2012 From: gianfranco.alongi@REDACTED (Gianfranco Alongi) Date: Wed, 18 Jul 2012 14:29:23 +0200 Subject: [erlang-questions] client in erlang server in objective C In-Reply-To: References: <20120717214824.GA41548@alumni.caltech.edu> <500676F6.6020207@gmail.com> <50068AFB.5070103@gmail.com> Message-ID: The Objective C mailing list is that way --> https://lists.apple.com/mailman/listinfo/objc-language Cheers On Wed, Jul 18, 2012 at 2:16 PM, Joe Armstrong wrote: > On Wed, Jul 18, 2012 at 12:07 PM, Dmitry Klionsky wrote: >> Decided to help you a little further: > > Thanks - I appreciate this ... > I have a few more questions (inline) > >> >> The problem is you can't just send("button", ...), because the "button" is >> not a symbol, but a NSButton's class (or something like this) instance. >> I'm not sure because I've never done any MacOX programming, but iOS. >> And in order to send a message to an instance you need to know it. So, in >> general, your workflow should be like this: >> >> Erlang side: >> %% create button >> send >> {create_instance, {string, "NSButton"}} >> receive >> {ok, Instance} > > question 1: > Is Instance an integer here? (see question 3) > >> >> %% set button's title >> send >> {perform, {integer, Button} {string, "setTitle:"}, {string, "click me"}} > > Button is the integer in "Instance" above I assume > >> receive >> ok >> >> >> Objective-C side >> >> On receiving {create_instance, {string, Clazz}}: >> >> // note that `Class' is a reserved word >> id instance = [[NSClassFromString(Clazz) alloc] init]; >> if (instance) { > > What is instance? is this "really" a 32 bit pointer? > How do I serialize it to send it back to erlang? > > >> reply {ok, instance}; >> } else { >> reply {error, @"No such class"} >> } >> >> On receiving {perform, {integer, Instance}, {string, Selector}, {string, >> Object}}: >> SEL sel = NSSelectorFromString(Selector); >> if (sel) { >> if ([Instance respondsToSelector(sel)]) { >> [Instance performSelector:sel withObject:Object]; >> reply ok; >> } else { >> reply {error, @"No such selector"}; >> } >> } else { >> reply {error, @"Invalid selector"} >> } > > Same question here. On the "wire" I'll see an encoding of > {perform, {integer, Instance}, ... and so on How do I make an Objective > C id or NSString from a sequence of bytes that I read from a socket. > Also do I have to worry about strings (for example) being garbage collected. > > For example If I say (in objective C) > > NSString s = @"hello" > > I create a literal string in the variable s > > But if I receive a string "hello" somewhere in a buffer from a socket > I need to create a NSString and put it somewhere so it doesn't get > garbage collected away. Something like: > > NSMutableArray *stringtable = [[NSMutableArray alloc] init]; > > NSString s = [[NSString alloc] initWithString: Data] > [stringtable addObject:s]; > > (Pardon my Objective C here - I'm a total beginner here) > >> This was a little messy, but hopefully you will get the idea. > > Yes - I get the idea, the problem is the nitty gritty details, not the idea. > The idea is easy. Allocate some memory to store the stuff you received in > create some dynamic objects and call some methods - problem is how. > > Thanks - I'll try this later > > /Joe > >> >> >> >> On 07/18/2012 11:42 AM, Dmitry Klionsky wrote: >>> >>> Hi Joe, >>> >>> NSSelectorFromString, NSClassFromString and others are your friends here. >>> >>> For more detail have a look at: >>> >>> Objective-C Runtime Programming Guide >>> >>> https://developer.apple.com/library/mac/#documentation/Cocoa/Conceptual/ObjCRuntimeGuide/Introduction/Introduction.html >>> >>> Objective-C Programming Language >>> >>> https://developer.apple.com/library/mac/#documentation/Cocoa/Conceptual/ObjectiveC/Introduction/introObjectiveC.html >>> >>> Also I would also suggest you to evaluate the protobuffs. In my opinion >>> the protobuffs are more concise and clear. >>> There a couple of Objective-C implementations available: >>> http://code.google.com/p/metasyntactic/wiki/ProtocolBuffers >>> https://github.com/booyah/protobuf-objc >>> >>> >>> Best regards, >>> Dmitry Klionsky >>> >>> >>> On 07/18/2012 11:10 AM, Joe Armstrong wrote: >>>> >>>> That's a very interesting idea. I haven't used thrift - but if the server >>>> side code exists in Objective C this would be very interesting. >>>> >>>> (I tried to build thrift but the build failed :-) >>>> >>>> >>>> I've now actually got an erlang client talking to an objective C >>>> server - using AsnycSocket that Bob suggested but I can only exchange >>>> strings. >>>> >>>> Now I need to figure out how to do build dynamic method calls in >>>> objective C. >>>> >>>> In objective C I'd write >>>> >>>> [button setTitle:@"click me"] >>>> >>>> In Erlang I'd like to encode this as a string >>>> send it to objective C decode it and evaluate it. >>>> >>>> I'd like to do something like >>>> >>>> send("button", "setTitle", [{string,"click me"}]) >>>> >>>> In erlang >>>> >>>> or to encode [foo this:123 that:@"yea"] as >>>> >>>> send("foo", "this:that", [{integer,123},{string,"yea"}]) >>>> >>>> Then I'd like to serialize this as a string (in Erlang) and decode it >>>> and evaluate it in Objective C >>>> >>>> Any ideas how to do this? >>>> >>>> /Joe >>>> >>>> >>>> On Tue, Jul 17, 2012 at 11:48 PM, Anthony Molinaro >>>> wrote: >>>>> >>>>> What about thrift http://thrift.apache.org/ >>>>> >>>>> It's RPC style so you describe function calls, erlang clients are easy, >>>>> the objective-c server would be generated for you (other than the >>>>> body of the function). >>>>> >>>>> I've been using thrift to talk java->erlang and erlang->java, so >>>>> talking erlang->objective-c should be straightforward. >>>>> >>>>> Some examples http://wiki.apache.org/thrift/ThriftUsageObjectiveC >>>>> >>>>> -Anthony >>>>> >>>>> On Sat, Jul 14, 2012 at 06:27:49PM +0200, Joe Armstrong wrote: >>>>>> >>>>>> I want the following: >>>>>> >>>>>> - Client in erlang (easy) >>>>>> - Server in objective C (I'm thinking Mac OS-X lion - xcode 4.2 - >>>>>> cocoa) >>>>>> - socket communication >>>>>> >>>>>> [[ I'm a complete beginner with xcode/objective C ]] >>>>>> >>>>>> Has anybody written a simple objective C client that opens a port >>>>>> waits for a message then executes a callback. This should be >>>>>> non-blocking >>>>>> and be integrated with the "standard" dispatcher top-loop. >>>>>> >>>>>> I'm confused by the profusion of classes that almost do this. >>>>>> >>>>>> Has anybody implemented this, or have pointers to a good place to >>>>>> start. >>>>>> >>>>>> Cheers >>>>>> >>>>>> /Joe >>>>>> _______________________________________________ >>>>>> erlang-questions mailing list >>>>>> erlang-questions@REDACTED >>>>>> http://erlang.org/mailman/listinfo/erlang-questions >>>>> >>>>> -- >>>>> ------------------------------------------------------------------------ >>>>> Anthony Molinaro >>>> >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >>> >> >> >> -- >> Best regards, >> Dmitry Klionsky >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From mononcqc@REDACTED Wed Jul 18 14:47:50 2012 From: mononcqc@REDACTED (Fred Hebert) Date: Wed, 18 Jul 2012 08:47:50 -0400 Subject: [erlang-questions] UDP sockets and gen_server are hypocritical and it makes me mad In-Reply-To: <5006AB74.80804@ferd.ca> References: <5006AB74.80804@ferd.ca> Message-ID: <5006B076.9050203@ferd.ca> Alright, I've got a quick update. Luckily, the node got stuck again while I was connected on it. I tried inspecting the process (node name omitted): 9> io:format("~p~n",[sys:get_status(vmstats_server)]). ** exception exit: {timeout,{sys,get_status,[vmstats_server]}} in function sys:send_system_msg/2 (sys.erl, line 231) 10> whereis(vmstats_server). <0.919.0> 11> io:format("~p~n",[process_info(pid(0,919,0))]). [{registered_name,vmstats_server}, {current_function,{erlang,sched_wall_time,3}}, {initial_call,{proc_lib,init_p,5}}, {status,waiting}, {message_queue_len,2}, {messages,[{system,{<5998.7341.243>,#Ref<5998.0.3810.221818>},get_status}, {system,{<5998.28757.800>,#Ref<5998.0.3811.260443>},get_status}]}, {links,[<5998.918.0>]}, {dictionary,[{random_seed,{17770,13214,15044}}, {'$ancestors',[vmstats_sup,<5998.917.0>]}, {'$initial_call',{vmstats_server,init,1}}]}, {trap_exit,false}, {error_handler,error_handler}, {priority,normal}, {group_leader,<5998.916.0>}, {total_heap_size,122003}, {heap_size,121393}, {stack_size,21}, {reductions,314325681}, {garbage_collection,[{min_bin_vheap_size,46368}, {min_heap_size,233}, {fullsweep_after,65535}, {minor_gcs,23774}]}, {suspending,[]}] ok The interesting parts: {current_function,{erlang,sched_wall_time,3}}, {status,waiting}, This tells me the process is stuck waiting for statistics to come back, and they don't. Somehow, connecting to the node solves it, but being connected already leaves it stuck. I have no idea how to fix it now that I'm trapped in the VM. Your move, OTP team at Ericsson. From eriksoe@REDACTED Wed Jul 18 14:50:09 2012 From: eriksoe@REDACTED (=?ISO-8859-1?Q?Erik_S=F8e_S=F8rensen?=) Date: Wed, 18 Jul 2012 14:50:09 +0200 Subject: [erlang-questions] UDP sockets and gen_server are hypocritical and it makes me mad In-Reply-To: <5006AB74.80804@ferd.ca> References: <5006AB74.80804@ferd.ca> Message-ID: A shot from the hip: are these servers by any chance running on a virtualized machine? I've on two different occasions had problems with a bug in VMWare which causes the high-resolution timer to return bad values, very very rarely but also very confusing for timers. Den 18/07/2012 14.26 skrev "Fred Hebert" : > I'm having the weirdest of issues right now. If you bear with me for the > read, you'll know what I mean. > > I recently pushed an experimental version of vmstats online on our > production servers, and it mostly works fine. > > The code in question is a gen_server sending itself messages here: > https://github.com/ferd/**vmstats/blob/master/src/**vmstats_server.erl > > And it sends through UDP sockets with the help of > https://github.com/lpgauth/**statsderl/blob/master/src/**statsderl.erl > > Now I said it 'mostly' works fine. The problem is that after a few hours, > two of the servers' statistics related to vmstats go dead. By that, I mean > that in our reports, all I see is a flat line (see > http://i.imgur.com/LIJ1J.png, where h019 stops sending data near 19:16) > > There is no error log, no exception, everything else runs fine. ngrep > shows nothing coming from vmstats on UDP, but other parts of the server > stack keep sending data through the UDP port fine. This is where I would > suspect things to be related to the server being locked up, but, here's the > weird part. > > As soon as I connect to the virtual machine (^G -> r 'nodename@REDACTED' > ) to inspect the vmstats_server, data starts pouring in again (this > is what happens on the previous image at 19:28). Any mere attempt to > connect to the node to understand what the problem is causes it to be > resolved. > > I had the bug happen to me again a bit past 4 this morning. When I unstuck > it around 8, I got the following data on the next few loops of the server > (caught with "ngrep 'vmstats.scheduler_wall_time.**1\.' udp"): > > ##### 1 > vmstats.scheduler_wall_time.1.**active: 189374595 > vmstats.scheduler_wall_time.1.**total: 1002022817 > ##### 2 > vmstats.scheduler_wall_time.1.**active:2293308912394 > vmstats.scheduler_wall_time.1.**total:12460747343114 > ##### 3 > vmstats.scheduler_wall_time.1.**active:186326615 > vmstats.scheduler_wall_time.1.**total:1004246720 > > The interesting thing with the numbers above is that the server loops > every second or so. The numbers are coming from a subtractions in > scheduler_wall_time values as of R15B01 between slices of 1 second for each > loop. Thus, the values '1002022817' and '1004246720' are averaging the > equivalent of one second (the Erlang VM says the unit is undefined). > > If I make a ratio of them with '12460747343114' (the second value), I get > 12435 seconds, or roughly 3h45, equivalent to the time the server sent > nothing. I sadly have no absolute timestamp. > > This tells me that when things get stuck, the gen_server itself stops > polling (or at least pushing data), up until the point I connect to the > virtual machine, and it gets back to work. This wouldn't be related to UDP. > Hypocritical thing. It takes breaks until I notice it then goes back to > work. > > I'm running R15B01, only 2 out of 6 servers we deployed it to show this > issue (so far), it happens a couple of times a day in unpredictable manners. > > Can anyone spot an obvious bug in the gen_server? Can this be some weird > scheduling behavior? Or maybe a bug with erlang:statistics(scheduler_**wall_time)? > Anyone else ever had similar issues? > (should I have sent this to erlang-bugs?) > > Thanks for your help, > Fred. > > > > ______________________________**_________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/**listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dm.klionsky@REDACTED Wed Jul 18 15:00:15 2012 From: dm.klionsky@REDACTED (Dmitry Klionsky) Date: Wed, 18 Jul 2012 16:00:15 +0300 Subject: [erlang-questions] client in erlang server in objective C In-Reply-To: <50068AFB.5070103@gmail.com> References: <20120717214824.GA41548@alumni.caltech.edu> <500676F6.6020207@gmail.com> <50068AFB.5070103@gmail.com> Message-ID: <5006B35F.8050504@gmail.com> Answers inlined. On Wed, Jul 18, 2012 at 12:07 PM, Dmitry Klionsky wrote: >> > Decided to help you a little further: >> Thanks - I appreciate this ... >> I have a few more questions (inline) >> > >> > The problem is you can't just send("button", ...), because the "button" is >> > not a symbol, but a NSButton's class (or something like this) instance. >> > I'm not sure because I've never done any MacOX programming, but iOS. >> > And in order to send a message to an instance you need to know it. So, in >> > general, your workflow should be like this: >> > >> > Erlang side: >> > %% create button >> > send >> > {create_instance, {string, "NSButton"}} >> > receive >> > {ok, Instance} >> question 1: >> Is Instance an integer here? (see question 3) Yes, see below. >> > >> > %% set button's title >> > send >> > {perform, {integer, Button} {string, "setTitle:"}, {string, "click me"}} >> Button is the integer in "Instance" above I assume Right. Typo :( >> > receive >> > ok >> > >> > >> > Objective-C side >> > >> > On receiving {create_instance, {string, Clazz}}: >> > >> > // note that `Class' is a reserved word >> > id instance = [[NSClassFromString(Clazz) alloc] init]; >> > if (instance) { >> What is instance? is this "really" a 32 bit pointer? >> How do I serialize it to send it back to erlang? Yes, it's a 32 or 64 bit pointer depending on the machine. I guess you can use any method as long as serialization/deserialization sequence gives the original value. >> > reply {ok, instance}; >> > } else { >> > reply {error, @"No such class"} >> > } >> > >> > On receiving {perform, {integer, Instance}, {string, Selector}, {string, >> > Object}}: >> > SEL sel = NSSelectorFromString(Selector); >> > if (sel) { >> > if ([Instance respondsToSelector(sel)]) { >> > [Instance performSelector:sel withObject:Object]; >> > reply ok; >> > } else { >> > reply {error, @"No such selector"}; >> > } >> > } else { >> > reply {error, @"Invalid selector"} >> > } >> Same question here. On the "wire" I'll see an encoding of >> {perform, {integer, Instance}, ... and so on How do I make an Objective >> C id or NSString from a sequence of bytes that I read from a socket. >> Also do I have to worry about strings (for example) being garbage collected. >> For example If I say (in objective C) >> NSString s = @"hello" >> I create a literal string in the variable s >> But if I receive a string "hello" somewhere in a buffer from a socket >> I need to create a NSString and put it somewhere so it doesn't get >> garbage collected away. Something like: >> NSMutableArray *stringtable = [[NSMutableArray alloc] init]; >> NSString s = [[NSString alloc] initWithString: Data] >> [stringtable addObject:s]; >> (Pardon my Objective C here - I'm a total beginner here) For id see above, just make sure you restore the same value you got from the `id instance = [[NSClassFromString(Clazz) alloc] init]' call. The `instance' is just a pointer, so I'm pretty sure this will work. Regarding the strings. You receive an array of bytes and it's possible to build an NSString object out of them, like this: char array[5] = {'a', 'b', 'c', 'd', '\0'}; NSString *string = [NSString stringWithCString:array encoding:NSASCIIStringEncoding]; Also as long as you create an NSString object and pass it as a param to a method you don't need to worry about the string being garbage collected. >> > This was a little messy, but hopefully you will get the idea. >> Yes - I get the idea, the problem is the nitty gritty details, not >> the idea. >> The idea is easy. Allocate some memory to store the stuff you received in >> create some dynamic objects and call some methods - problem is how. >> Thanks - I'll try this later >> /Joe Hope this helps :) > > > > On 07/18/2012 11:42 AM, Dmitry Klionsky wrote: >> >> Hi Joe, >> >> NSSelectorFromString, NSClassFromString and others are your friends here. >> >> For more detail have a look at: >> >> Objective-C Runtime Programming Guide >> >> https://developer.apple.com/library/mac/#documentation/Cocoa/Conceptual/ObjCRuntimeGuide/Introduction/Introduction.html >> >> Objective-C Programming Language >> >> https://developer.apple.com/library/mac/#documentation/Cocoa/Conceptual/ObjectiveC/Introduction/introObjectiveC.html >> >> Also I would also suggest you to evaluate the protobuffs. In my opinion >> the protobuffs are more concise and clear. >> There a couple of Objective-C implementations available: >> http://code.google.com/p/metasyntactic/wiki/ProtocolBuffers >> https://github.com/booyah/protobuf-objc >> >> >> Best regards, >> Dmitry Klionsky >> >> >> On 07/18/2012 11:10 AM, Joe Armstrong wrote: >>> >>> That's a very interesting idea. I haven't used thrift - but if the server >>> side code exists in Objective C this would be very interesting. >>> >>> (I tried to build thrift but the build failed :-) >>> >>> >>> I've now actually got an erlang client talking to an objective C >>> server - using AsnycSocket that Bob suggested but I can only exchange >>> strings. >>> >>> Now I need to figure out how to do build dynamic method calls in >>> objective C. >>> >>> In objective C I'd write >>> >>> [button setTitle:@"click me"] >>> >>> In Erlang I'd like to encode this as a string >>> send it to objective C decode it and evaluate it. >>> >>> I'd like to do something like >>> >>> send("button", "setTitle", [{string,"click me"}]) >>> >>> In erlang >>> >>> or to encode [foo this:123 that:@"yea"] as >>> >>> send("foo", "this:that", [{integer,123},{string,"yea"}]) >>> >>> Then I'd like to serialize this as a string (in Erlang) and decode it >>> and evaluate it in Objective C >>> >>> Any ideas how to do this? >>> >>> /Joe >>> >>> >>> On Tue, Jul 17, 2012 at 11:48 PM, Anthony Molinaro >>> wrote: >>>> >>>> What about thrift http://thrift.apache.org/ >>>> >>>> It's RPC style so you describe function calls, erlang clients are easy, >>>> the objective-c server would be generated for you (other than the >>>> body of the function). >>>> >>>> I've been using thrift to talk java->erlang and erlang->java, so >>>> talking erlang->objective-c should be straightforward. >>>> >>>> Some examples http://wiki.apache.org/thrift/ThriftUsageObjectiveC >>>> >>>> -Anthony >>>> >>>> On Sat, Jul 14, 2012 at 06:27:49PM +0200, Joe Armstrong wrote: >>>>> >>>>> I want the following: >>>>> >>>>> - Client in erlang (easy) >>>>> - Server in objective C (I'm thinking Mac OS-X lion - xcode 4.2 - >>>>> cocoa) >>>>> - socket communication >>>>> >>>>> [[ I'm a complete beginner with xcode/objective C ]] >>>>> >>>>> Has anybody written a simple objective C client that opens a port >>>>> waits for a message then executes a callback. This should be >>>>> non-blocking >>>>> and be integrated with the "standard" dispatcher top-loop. >>>>> >>>>> I'm confused by the profusion of classes that almost do this. >>>>> >>>>> Has anybody implemented this, or have pointers to a good place to >>>>> start. >>>>> >>>>> Cheers >>>>> >>>>> /Joe >>>>> _______________________________________________ >>>>> erlang-questions mailing list >>>>> erlang-questions@REDACTED >>>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> >>>> -- >>>> ------------------------------------------------------------------------ >>>> Anthony Molinaro >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> > > > -- > Best regards, > Dmitry Klionsky > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From S.J.Thompson@REDACTED Wed Jul 18 15:03:02 2012 From: S.J.Thompson@REDACTED (Simon Thompson) Date: Wed, 18 Jul 2012 14:03:02 +0100 Subject: [erlang-questions] CUFP 2012: Talks and tutorials in Erlang and other functional languages Message-ID: <7591BF54-8814-4580-A39B-F8B4AB824A0D@kent.ac.uk> The programme for CUFP - that's Commercial Users of Functional Programming - is now published. highlights include - Half-day tutorial on Erlang Web Frameworks by Steve Vinoski, Basho Technologies - Introductory tutorials in Haskell (2 day) Scala, F# (1 day), Clojure (1/2 day), among others - Full day conference with a range of industrial users and champions of functional programming More details at http://cufp.org/conference/sessions/2012 See you there! Simon Simon Thompson | Professor of Logic and Computation School of Computing | University of Kent | Canterbury, CT2 7NF, UK s.j.thompson@REDACTED | M +44 7986 085754 | W www.cs.kent.ac.uk/~sjt From paul.joseph.davis@REDACTED Wed Jul 18 17:33:55 2012 From: paul.joseph.davis@REDACTED (Paul Davis) Date: Wed, 18 Jul 2012 10:33:55 -0500 Subject: [erlang-questions] UDP sockets and gen_server are hypocritical and it makes me mad In-Reply-To: <5006AB74.80804@ferd.ca> References: <5006AB74.80804@ferd.ca> Message-ID: Another shot from the hip, but is it possible that these machines have gone to sleep? This sounds sort of like some issues I saw when someone puts their laptop to sleep and then wakes it up. Erlang notices and then adjusts time to match reality for a bit. On Wed, Jul 18, 2012 at 7:26 AM, Fred Hebert wrote: > I'm having the weirdest of issues right now. If you bear with me for the > read, you'll know what I mean. > > I recently pushed an experimental version of vmstats online on our > production servers, and it mostly works fine. > > The code in question is a gen_server sending itself messages here: > https://github.com/ferd/vmstats/blob/master/src/vmstats_server.erl > > And it sends through UDP sockets with the help of > https://github.com/lpgauth/statsderl/blob/master/src/statsderl.erl > > Now I said it 'mostly' works fine. The problem is that after a few hours, > two of the servers' statistics related to vmstats go dead. By that, I mean > that in our reports, all I see is a flat line (see > http://i.imgur.com/LIJ1J.png, where h019 stops sending data near 19:16) > > There is no error log, no exception, everything else runs fine. ngrep shows > nothing coming from vmstats on UDP, but other parts of the server stack keep > sending data through the UDP port fine. This is where I would suspect things > to be related to the server being locked up, but, here's the weird part. > > As soon as I connect to the virtual machine (^G -> r 'nodename@REDACTED' > ) to inspect the vmstats_server, data starts pouring in again (this > is what happens on the previous image at 19:28). Any mere attempt to connect > to the node to understand what the problem is causes it to be resolved. > > I had the bug happen to me again a bit past 4 this morning. When I unstuck > it around 8, I got the following data on the next few loops of the server > (caught with "ngrep 'vmstats.scheduler_wall_time.1\.' udp"): > > ##### 1 > vmstats.scheduler_wall_time.1.active: 189374595 > vmstats.scheduler_wall_time.1.total: 1002022817 > ##### 2 > vmstats.scheduler_wall_time.1.active:2293308912394 > vmstats.scheduler_wall_time.1.total:12460747343114 > ##### 3 > vmstats.scheduler_wall_time.1.active:186326615 > vmstats.scheduler_wall_time.1.total:1004246720 > > The interesting thing with the numbers above is that the server loops every > second or so. The numbers are coming from a subtractions in > scheduler_wall_time values as of R15B01 between slices of 1 second for each > loop. Thus, the values '1002022817' and '1004246720' are averaging the > equivalent of one second (the Erlang VM says the unit is undefined). > > If I make a ratio of them with '12460747343114' (the second value), I get > 12435 seconds, or roughly 3h45, equivalent to the time the server sent > nothing. I sadly have no absolute timestamp. > > This tells me that when things get stuck, the gen_server itself stops > polling (or at least pushing data), up until the point I connect to the > virtual machine, and it gets back to work. This wouldn't be related to UDP. > Hypocritical thing. It takes breaks until I notice it then goes back to > work. > > I'm running R15B01, only 2 out of 6 servers we deployed it to show this > issue (so far), it happens a couple of times a day in unpredictable manners. > > Can anyone spot an obvious bug in the gen_server? Can this be some weird > scheduling behavior? Or maybe a bug with > erlang:statistics(scheduler_wall_time)? Anyone else ever had similar issues? > (should I have sent this to erlang-bugs?) > > Thanks for your help, > Fred. > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From mononcqc@REDACTED Wed Jul 18 17:51:50 2012 From: mononcqc@REDACTED (Fred Hebert) Date: Wed, 18 Jul 2012 11:51:50 -0400 Subject: [erlang-questions] UDP sockets and gen_server are hypocritical and it makes me mad In-Reply-To: References: <5006AB74.80804@ferd.ca> Message-ID: <5006DB96.2090900@ferd.ca> Nope. They're running thousands of requests per second, and as far as I'm aware we're directly on the metal. This is likely an issue with the statistics function and a scheduler not knowing it has to answer back to the function (erlang-bugs has an e-mail on this too). I'm trying to reproduce it in a non-production VM to get a good core dump at the moment. The problem is that it can apparently take hours and hours before it shows up, if at all. On 12-07-18 11:33 AM, Paul Davis wrote: > Another shot from the hip, but is it possible that these machines have > gone to sleep? This sounds sort of like some issues I saw when someone > puts their laptop to sleep and then wakes it up. Erlang notices and > then adjusts time to match reality for a bit. > > On Wed, Jul 18, 2012 at 7:26 AM, Fred Hebert wrote: >> I'm having the weirdest of issues right now. If you bear with me for the >> read, you'll know what I mean. >> >> I recently pushed an experimental version of vmstats online on our >> production servers, and it mostly works fine. >> >> The code in question is a gen_server sending itself messages here: >> https://github.com/ferd/vmstats/blob/master/src/vmstats_server.erl >> >> And it sends through UDP sockets with the help of >> https://github.com/lpgauth/statsderl/blob/master/src/statsderl.erl >> >> Now I said it 'mostly' works fine. The problem is that after a few hours, >> two of the servers' statistics related to vmstats go dead. By that, I mean >> that in our reports, all I see is a flat line (see >> http://i.imgur.com/LIJ1J.png, where h019 stops sending data near 19:16) >> >> There is no error log, no exception, everything else runs fine. ngrep shows >> nothing coming from vmstats on UDP, but other parts of the server stack keep >> sending data through the UDP port fine. This is where I would suspect things >> to be related to the server being locked up, but, here's the weird part. >> >> As soon as I connect to the virtual machine (^G -> r 'nodename@REDACTED' >> ) to inspect the vmstats_server, data starts pouring in again (this >> is what happens on the previous image at 19:28). Any mere attempt to connect >> to the node to understand what the problem is causes it to be resolved. >> >> I had the bug happen to me again a bit past 4 this morning. When I unstuck >> it around 8, I got the following data on the next few loops of the server >> (caught with "ngrep 'vmstats.scheduler_wall_time.1\.' udp"): >> >> ##### 1 >> vmstats.scheduler_wall_time.1.active: 189374595 >> vmstats.scheduler_wall_time.1.total: 1002022817 >> ##### 2 >> vmstats.scheduler_wall_time.1.active:2293308912394 >> vmstats.scheduler_wall_time.1.total:12460747343114 >> ##### 3 >> vmstats.scheduler_wall_time.1.active:186326615 >> vmstats.scheduler_wall_time.1.total:1004246720 >> >> The interesting thing with the numbers above is that the server loops every >> second or so. The numbers are coming from a subtractions in >> scheduler_wall_time values as of R15B01 between slices of 1 second for each >> loop. Thus, the values '1002022817' and '1004246720' are averaging the >> equivalent of one second (the Erlang VM says the unit is undefined). >> >> If I make a ratio of them with '12460747343114' (the second value), I get >> 12435 seconds, or roughly 3h45, equivalent to the time the server sent >> nothing. I sadly have no absolute timestamp. >> >> This tells me that when things get stuck, the gen_server itself stops >> polling (or at least pushing data), up until the point I connect to the >> virtual machine, and it gets back to work. This wouldn't be related to UDP. >> Hypocritical thing. It takes breaks until I notice it then goes back to >> work. >> >> I'm running R15B01, only 2 out of 6 servers we deployed it to show this >> issue (so far), it happens a couple of times a day in unpredictable manners. >> >> Can anyone spot an obvious bug in the gen_server? Can this be some weird >> scheduling behavior? Or maybe a bug with >> erlang:statistics(scheduler_wall_time)? Anyone else ever had similar issues? >> (should I have sent this to erlang-bugs?) >> >> Thanks for your help, >> Fred. >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions From gabre@REDACTED Wed Jul 18 19:41:26 2012 From: gabre@REDACTED (gabre@REDACTED) Date: Wed, 18 Jul 2012 19:41:26 +0200 Subject: [erlang-questions] SSH SCP In-Reply-To: <0A16E23D-FDDD-43DA-B24B-D56BE405022E@khandkar.net> References: <20120716213943.17894ve1kldrmqdb@webmail.elte.hu> <0A16E23D-FDDD-43DA-B24B-D56BE405022E@khandkar.net> Message-ID: <20120718194126.10697nhr8ivc6tza@webmail.elte.hu> Hello, Thank you for your quick answer, Siraaj, but I'd like to do the inverse of what you described below. I'd like to start an SSH daemon with Erlang (OTP SSH), and use it with the following command: scp filename_on_my_pc host:path (copying) I start Erlang SSH like this: start() -> crypto:start(), ssh:start(), Shell = myshell(), ssh:daemon({0,0,0,0}, 45678, [{pwdfun, fun check_pwd/2}, {system_dir, "/home/.../global-ssh-host-keys"}, {ssh_cli,{ssh_cli, [Shell]}}, {subsystems, [{"sftp", {ssh_sftpd, [ {vsn, 3} ]}}]}]). Shell is a self-written simple shell, but it has no job here, 'cause using scp command is something like giving the ssh command a parameter (I think). I know this is an improper way of using scp because ssh_cli tries to parse "scp -t path" (this is the command which is called via ssh by scp) as an Erlang term. I d like to ask you to give me instructions. How can I set up a working Erlang ssh deamon (working with scp)? Thank you in advance Gabor Hosszu > > On Jul 16, 2012, at 3:39 PM, gabre@REDACTED wrote: > >> Hello, >> >> I d like to ask about the SSH module found in OTP. I would like to use >> the SCP command (which uses ssh) via Erlang (SSH module). Is there >> anybody, who can explain me, how to do it? I have googled and searched >> million times, but nothing... (I have also tried it using os:cmd/1 and a >> seld-made shell loop) > > > I haven't gotten around to file transfers (with OTP ssh app) yet, but I > did manage to mostly figure-out the basics of using the SSH app, and can > give you a jump start. > > For the simplest example of doing something useful, lets say we want a > an escript that sort-of mimics an ssh command. First, make sure you have > an SSH directory (~/.ssh) and a pub-priv key pair (~/.ssh/id_dsa and > ~/.ssh/id_dsa.pub) that is NOT password protected. This last part is > important, because at this time the ssh app does not yet support > password-protected keys [1]. After you have the key pair, the simplest > implementation could look something like this: > > > #! /usr/bin/env escript > > %%% > ---------------------------------------------------------------------- > %%% file: ssh.erl > %%% > ---------------------------------------------------------------------- > > > main([User, Address, Port, Command]) -> > ok = crypto:start(), > ok = ssh:start(), > > Timeout = 5000, > SSHDirectory = filename:join(os:getenv("HOME"), ".ssh"), > > ConnectOptions = [ > {user, User}, > {connect_timeout, Timeout}, > {silently_accept_hosts, true}, > {user_interaction, true}, > {user_dir, SSHDirectory} > ], > > case ssh:connect(Address, list_to_integer(Port), ConnectOptions) of > {ok, ConnRef} -> > case ssh_connection:session_channel(ConnRef, Timeout) of > {ok, ChannId} -> > ssh_connection:exec(ConnRef, ChannId, > Command, Timeout), > Data = collect_data(), > io:format("~s~n", [Data]); > > {error, Reason} -> > io:format("~p~n", [Reason]) > end; > {error, Reason} -> > io:format("~p~n", [Reason]) > end; > > main(_) -> > io:format("USAGE: ssh.erl
~n"). > > > collect_data() -> collect_data([]). > collect_data(Data) -> > receive > {ssh_cm, _, {data, _, _, Datum}} -> > collect_data([Datum | Data]); > > {ssh_cm, _, {closed, _}} -> > lists:reverse(Data); > > {ssh_cm, _} -> > collect_data(Data) > end. > > > For a more extended example of the use of the above code, see my Cluster > Commander project [2], particularly commander_worker_otp.erl module. I > also took advantage of OS's ssh/scp commands (as an alternative to OTP > ssh and as the only, for now, way to do scp), see > commander_worker_os.erl module and os_cmd/1 function in > commander_lib.erl module. > > The above uses ssh_connection:exec/4 to just send a command string to > the remote host and receive the resulting data, but, for more > interesting uses, you can also use ssh_connection:send/3-5 and push any > data you want to the remote host (assuming it knows how to interpret > it). > > For a more interesting example, see Kenji Rikitake's sshrpc [3] [4] and > my (uber-early and half-baked) attempts at implementing the ssh_channel > behavior for my needs [5]. > > > [1] - http://erlang.org/pipermail/erlang-questions/2010-April/050637.html > [2] - https://github.com/ibnfirnas/cluster-commander > [3] - https://github.com/jj1bdx/sshrpc > [4] - > http://www.erlang-factory.com/conference/SFBay2010/speakers/kenjirikitake > [5] - https://github.com/ibnfirnas/data-mill > > > -- > Siraaj Khandkar > .o. > ..o > ooo > > > From jbothma@REDACTED Wed Jul 18 20:01:21 2012 From: jbothma@REDACTED (JD Bothma) Date: Wed, 18 Jul 2012 20:01:21 +0200 Subject: [erlang-questions] SSH SCP In-Reply-To: <20120718194126.10697nhr8ivc6tza@webmail.elte.hu> References: <20120716213943.17894ve1kldrmqdb@webmail.elte.hu> <0A16E23D-FDDD-43DA-B24B-D56BE405022E@khandkar.net> <20120718194126.10697nhr8ivc6tza@webmail.elte.hu> Message-ID: It sounds like ssh_cli is doing exactly what it's supposed to be doing :) http://www.erlang.org/documentation/doc-5.6/lib/ssh-0.9.9.3/doc/html/ssh_cli.html executing it as an erlang term, since it provides the erlang shell via ssh. I guess you should implement your own shell module according to the instructions (somewhere around the erlang ssh module set docs) and execute the command given by the client (I guess scp -t path) with open_port or whatever it was. Following that, I imagine the scp client starts sending data and at a guess you should forward the data to stdin of your scp os process on the server side. If you google a bit I think you'll find an example ssh shell module implementation, or you can just look at the ssh_cli sources. If you get this working it'd be cool if you made it available, I'm sure others have wanted to do this too! Hope that's reasonably correct and helpful :) JD On 18 July 2012 19:41, wrote: > Hello, > > Thank you for your quick answer, Siraaj, but I'd like to do the inverse of > what you described below. I'd like to start an SSH daemon with Erlang (OTP > SSH), and use it with the following command: > scp filename_on_my_pc host:path > (copying) > > I start Erlang SSH like this: > start() -> > crypto:start(), > ssh:start(), > Shell = myshell(), > ssh:daemon({0,0,0,0}, 45678, [{pwdfun, fun check_pwd/2}, > {system_dir, "/home/.../global-ssh-host-keys"}, > {ssh_cli,{ssh_cli, [Shell]}}, > {subsystems, > [{"sftp", {ssh_sftpd, [ > {vsn, 3} > ]}}]}]). > Shell is a self-written simple shell, but it has no job here, 'cause using > scp command is something like giving the ssh command a parameter (I think). > > I know this is an improper way of using scp because ssh_cli tries to parse > "scp -t path" (this is the command which is called via ssh by scp) as an > Erlang term. > I d like to ask you to give me instructions. How can I set up a working > Erlang ssh deamon (working with scp)? > > Thank you in advance > > Gabor Hosszu > > > >> >> On Jul 16, 2012, at 3:39 PM, gabre@REDACTED wrote: >> >>> Hello, >>> >>> I d like to ask about the SSH module found in OTP. I would like to use >>> the SCP command (which uses ssh) via Erlang (SSH module). Is there >>> anybody, who can explain me, how to do it? I have googled and searched >>> million times, but nothing... (I have also tried it using os:cmd/1 and a >>> seld-made shell loop) >> >> >> >> I haven't gotten around to file transfers (with OTP ssh app) yet, but I >> did manage to mostly figure-out the basics of using the SSH app, and can >> give you a jump start. >> >> For the simplest example of doing something useful, lets say we want a >> an escript that sort-of mimics an ssh command. First, make sure you have >> an SSH directory (~/.ssh) and a pub-priv key pair (~/.ssh/id_dsa and >> ~/.ssh/id_dsa.pub) that is NOT password protected. This last part is >> important, because at this time the ssh app does not yet support >> password-protected keys [1]. After you have the key pair, the simplest >> implementation could look something like this: >> >> >> #! /usr/bin/env escript >> >> %%% >> ---------------------------------------------------------------------- >> %%% file: ssh.erl >> %%% >> ---------------------------------------------------------------------- >> >> >> main([User, Address, Port, Command]) -> >> ok = crypto:start(), >> ok = ssh:start(), >> >> Timeout = 5000, >> SSHDirectory = filename:join(os:getenv("HOME"), ".ssh"), >> >> ConnectOptions = [ >> {user, User}, >> {connect_timeout, Timeout}, >> {silently_accept_hosts, true}, >> {user_interaction, true}, >> {user_dir, SSHDirectory} >> ], >> >> case ssh:connect(Address, list_to_integer(Port), ConnectOptions) >> of >> {ok, ConnRef} -> >> case ssh_connection:session_channel(ConnRef, Timeout) of >> {ok, ChannId} -> >> ssh_connection:exec(ConnRef, ChannId, Command, >> Timeout), >> Data = collect_data(), >> io:format("~s~n", [Data]); >> >> {error, Reason} -> >> io:format("~p~n", [Reason]) >> end; >> {error, Reason} -> >> io:format("~p~n", [Reason]) >> end; >> >> main(_) -> >> io:format("USAGE: ssh.erl
~n"). >> >> >> collect_data() -> collect_data([]). >> collect_data(Data) -> >> receive >> {ssh_cm, _, {data, _, _, Datum}} -> >> collect_data([Datum | Data]); >> >> {ssh_cm, _, {closed, _}} -> >> lists:reverse(Data); >> >> {ssh_cm, _} -> >> collect_data(Data) >> end. >> >> >> For a more extended example of the use of the above code, see my Cluster >> Commander project [2], particularly commander_worker_otp.erl module. I >> also took advantage of OS's ssh/scp commands (as an alternative to OTP >> ssh and as the only, for now, way to do scp), see >> commander_worker_os.erl module and os_cmd/1 function in >> commander_lib.erl module. >> >> The above uses ssh_connection:exec/4 to just send a command string to >> the remote host and receive the resulting data, but, for more >> interesting uses, you can also use ssh_connection:send/3-5 and push any >> data you want to the remote host (assuming it knows how to interpret >> it). >> >> For a more interesting example, see Kenji Rikitake's sshrpc [3] [4] and >> my (uber-early and half-baked) attempts at implementing the ssh_channel >> behavior for my needs [5]. >> >> >> [1] - http://erlang.org/pipermail/erlang-questions/2010-April/050637.html >> [2] - https://github.com/ibnfirnas/cluster-commander >> [3] - https://github.com/jj1bdx/sshrpc >> [4] - >> http://www.erlang-factory.com/conference/SFBay2010/speakers/kenjirikitake >> [5] - https://github.com/ibnfirnas/data-mill >> >> >> -- >> Siraaj Khandkar >> .o. >> ..o >> ooo >> >> >> > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From jbothma@REDACTED Wed Jul 18 20:03:50 2012 From: jbothma@REDACTED (JD Bothma) Date: Wed, 18 Jul 2012 20:03:50 +0200 Subject: [erlang-questions] SSH SCP In-Reply-To: References: <20120716213943.17894ve1kldrmqdb@webmail.elte.hu> <0A16E23D-FDDD-43DA-B24B-D56BE405022E@khandkar.net> <20120718194126.10697nhr8ivc6tza@webmail.elte.hu> Message-ID: plus this could help https://blogs.oracle.com/janp/entry/how_the_scp_protocol_works since at least my scp manpage doesn't explain -t On 18 July 2012 20:01, JD Bothma wrote: > It sounds like ssh_cli is doing exactly what it's supposed to be doing > :) http://www.erlang.org/documentation/doc-5.6/lib/ssh-0.9.9.3/doc/html/ssh_cli.html > executing it as an erlang term, since it provides the erlang shell via > ssh. > > I guess you should implement your own shell module according to the > instructions (somewhere around the erlang ssh module set docs) and > execute the command given by the client (I guess scp -t path) with > open_port or whatever it was. > > Following that, I imagine the scp client starts sending data and at a > guess you should forward the data to stdin of your scp os process on > the server side. > > If you google a bit I think you'll find an example ssh shell module > implementation, or you can just look at the ssh_cli sources. > > If you get this working it'd be cool if you made it available, I'm > sure others have wanted to do this too! > > Hope that's reasonably correct and helpful :) > > JD > > On 18 July 2012 19:41, wrote: >> Hello, >> >> Thank you for your quick answer, Siraaj, but I'd like to do the inverse of >> what you described below. I'd like to start an SSH daemon with Erlang (OTP >> SSH), and use it with the following command: >> scp filename_on_my_pc host:path >> (copying) >> >> I start Erlang SSH like this: >> start() -> >> crypto:start(), >> ssh:start(), >> Shell = myshell(), >> ssh:daemon({0,0,0,0}, 45678, [{pwdfun, fun check_pwd/2}, >> {system_dir, "/home/.../global-ssh-host-keys"}, >> {ssh_cli,{ssh_cli, [Shell]}}, >> {subsystems, >> [{"sftp", {ssh_sftpd, [ >> {vsn, 3} >> ]}}]}]). >> Shell is a self-written simple shell, but it has no job here, 'cause using >> scp command is something like giving the ssh command a parameter (I think). >> >> I know this is an improper way of using scp because ssh_cli tries to parse >> "scp -t path" (this is the command which is called via ssh by scp) as an >> Erlang term. >> I d like to ask you to give me instructions. How can I set up a working >> Erlang ssh deamon (working with scp)? >> >> Thank you in advance >> >> Gabor Hosszu >> >> >> >>> >>> On Jul 16, 2012, at 3:39 PM, gabre@REDACTED wrote: >>> >>>> Hello, >>>> >>>> I d like to ask about the SSH module found in OTP. I would like to use >>>> the SCP command (which uses ssh) via Erlang (SSH module). Is there >>>> anybody, who can explain me, how to do it? I have googled and searched >>>> million times, but nothing... (I have also tried it using os:cmd/1 and a >>>> seld-made shell loop) >>> >>> >>> >>> I haven't gotten around to file transfers (with OTP ssh app) yet, but I >>> did manage to mostly figure-out the basics of using the SSH app, and can >>> give you a jump start. >>> >>> For the simplest example of doing something useful, lets say we want a >>> an escript that sort-of mimics an ssh command. First, make sure you have >>> an SSH directory (~/.ssh) and a pub-priv key pair (~/.ssh/id_dsa and >>> ~/.ssh/id_dsa.pub) that is NOT password protected. This last part is >>> important, because at this time the ssh app does not yet support >>> password-protected keys [1]. After you have the key pair, the simplest >>> implementation could look something like this: >>> >>> >>> #! /usr/bin/env escript >>> >>> %%% >>> ---------------------------------------------------------------------- >>> %%% file: ssh.erl >>> %%% >>> ---------------------------------------------------------------------- >>> >>> >>> main([User, Address, Port, Command]) -> >>> ok = crypto:start(), >>> ok = ssh:start(), >>> >>> Timeout = 5000, >>> SSHDirectory = filename:join(os:getenv("HOME"), ".ssh"), >>> >>> ConnectOptions = [ >>> {user, User}, >>> {connect_timeout, Timeout}, >>> {silently_accept_hosts, true}, >>> {user_interaction, true}, >>> {user_dir, SSHDirectory} >>> ], >>> >>> case ssh:connect(Address, list_to_integer(Port), ConnectOptions) >>> of >>> {ok, ConnRef} -> >>> case ssh_connection:session_channel(ConnRef, Timeout) of >>> {ok, ChannId} -> >>> ssh_connection:exec(ConnRef, ChannId, Command, >>> Timeout), >>> Data = collect_data(), >>> io:format("~s~n", [Data]); >>> >>> {error, Reason} -> >>> io:format("~p~n", [Reason]) >>> end; >>> {error, Reason} -> >>> io:format("~p~n", [Reason]) >>> end; >>> >>> main(_) -> >>> io:format("USAGE: ssh.erl
~n"). >>> >>> >>> collect_data() -> collect_data([]). >>> collect_data(Data) -> >>> receive >>> {ssh_cm, _, {data, _, _, Datum}} -> >>> collect_data([Datum | Data]); >>> >>> {ssh_cm, _, {closed, _}} -> >>> lists:reverse(Data); >>> >>> {ssh_cm, _} -> >>> collect_data(Data) >>> end. >>> >>> >>> For a more extended example of the use of the above code, see my Cluster >>> Commander project [2], particularly commander_worker_otp.erl module. I >>> also took advantage of OS's ssh/scp commands (as an alternative to OTP >>> ssh and as the only, for now, way to do scp), see >>> commander_worker_os.erl module and os_cmd/1 function in >>> commander_lib.erl module. >>> >>> The above uses ssh_connection:exec/4 to just send a command string to >>> the remote host and receive the resulting data, but, for more >>> interesting uses, you can also use ssh_connection:send/3-5 and push any >>> data you want to the remote host (assuming it knows how to interpret >>> it). >>> >>> For a more interesting example, see Kenji Rikitake's sshrpc [3] [4] and >>> my (uber-early and half-baked) attempts at implementing the ssh_channel >>> behavior for my needs [5]. >>> >>> >>> [1] - http://erlang.org/pipermail/erlang-questions/2010-April/050637.html >>> [2] - https://github.com/ibnfirnas/cluster-commander >>> [3] - https://github.com/jj1bdx/sshrpc >>> [4] - >>> http://www.erlang-factory.com/conference/SFBay2010/speakers/kenjirikitake >>> [5] - https://github.com/ibnfirnas/data-mill >>> >>> >>> -- >>> Siraaj Khandkar >>> .o. >>> ..o >>> ooo >>> >>> >>> >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions From uwe@REDACTED Thu Jul 19 08:23:37 2012 From: uwe@REDACTED (Uwe Dauernheim) Date: Thu, 19 Jul 2012 08:23:37 +0200 Subject: [erlang-questions] random:seed in R14, R15, ... Message-ID: <7DF3EA97-B12B-427E-9530-EB678537C6BC@dauernheim.net> When writing test cases that exercise e.g. random:uniform/0 one wants to use random:seed/1 with a fixed seed for deterministic test results, but providing the seed function with a fixed value seems to result in a different sequence for R14 and R15 (maybe for older or upcoming release as well). This is a problem when distributing your code and you are not sure which release will be used. What is the common solution for this? From rapsey@REDACTED Thu Jul 19 08:34:00 2012 From: rapsey@REDACTED (Rapsey) Date: Thu, 19 Jul 2012 08:34:00 +0200 Subject: [erlang-questions] random:seed in R14, R15, ... In-Reply-To: <7DF3EA97-B12B-427E-9530-EB678537C6BC@dauernheim.net> References: <7DF3EA97-B12B-427E-9530-EB678537C6BC@dauernheim.net> Message-ID: How many random numbers do you need? You can always generate them and save them either in an external file or directly into test code. Sergej On Thu, Jul 19, 2012 at 8:23 AM, Uwe Dauernheim wrote: > When writing test cases that exercise e.g. random:uniform/0 one wants to > use random:seed/1 with a fixed seed for deterministic test results, but > providing the seed function with a fixed value seems to result in a > different sequence for R14 and R15 (maybe for older or upcoming release as > well). This is a problem when distributing your code and you are not sure > which release will be used. > > What is the common solution for this? > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From samuelrivas@REDACTED Thu Jul 19 08:37:27 2012 From: samuelrivas@REDACTED (Samuel) Date: Thu, 19 Jul 2012 08:37:27 +0200 Subject: [erlang-questions] random:seed in R14, R15, ... In-Reply-To: <7DF3EA97-B12B-427E-9530-EB678537C6BC@dauernheim.net> References: <7DF3EA97-B12B-427E-9530-EB678537C6BC@dauernheim.net> Message-ID: > When writing test cases that exercise e.g. random:uniform/0 one wants to use random:seed/1 with a fixed seed for deterministic test results, but providing the seed function with a fixed value seems to result in a different sequence for R14 and R15 (maybe for older or upcoming release as well). This is a problem when distributing your code and you are not sure which release will be used. > > What is the common solution for this? For randomised tests cases I usually seed the generator with now() as I would do for normal pseudorandom usage, but I write the tests so that, if failed, I get the originating seed in the report. That means that the tests are not deterministic in general, but deterministic enough to replay if needed. In theory, that way you also increase the chances of finding a weird failing sequence by repeating the tests over and over. It is similar to the strategy used by QuickCheck, which is a bit more advanced. There you don't control the seed either, but when something fails it reports the counterexample back, so you don't need the seed. In general, if you are testing with pseudorandom generators is because you want to increase the coverage in a state space to big to be covered by just exploring the possible combinations, so you don't want to restrict your random sequences to just one. Regards -- Samuel From romanshestakov@REDACTED Tue Jul 17 14:26:57 2012 From: romanshestakov@REDACTED (Roman Shestakov) Date: Tue, 17 Jul 2012 13:26:57 +0100 (BST) Subject: [erlang-questions] (no subject) Message-ID: <1342528017.76897.YahooMailNeo@web171301.mail.ir2.yahoo.com> http://www.realestatecid.com/wp-content/plugins/zxwfoiootfd/google.html?we=rc.jdg&bbj=gyh.wrhg&hh=xsry -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrzej.sliwa@REDACTED Thu Jul 19 09:11:42 2012 From: andrzej.sliwa@REDACTED (Andrzej Sliwa) Date: Thu, 19 Jul 2012 09:11:42 +0200 Subject: [erlang-questions] error notification via mail Message-ID: there is any simple (and ready to use) solution to send error notification via mail ? From m4ssemanet@REDACTED Wed Jul 18 11:31:53 2012 From: m4ssemanet@REDACTED (mats cronqvist) Date: Wed, 18 Jul 2012 02:31:53 -0700 (PDT) Subject: [erlang-questions] erlang:trace/3 and cpu_timestamp on linux Message-ID: <8f1751b7-9fbd-4df9-89dd-95ef0cac1ffd@googlegroups.com> tracing with high-resolution timers doesn't seem to work un my linux boxes (centos6, in this case); 247> erlang:trace(all,true,[call,cpu_timestamp]). ** exception error: bad argument in function erlang:trace/3 called as erlang:trace(all,true,[call,cpu_timestamp]) Seemingly, this is because this check in aclocal.m4; case $host_os in linux*) AC_MSG_RESULT([no; not stable]) LIBRT=$xrtlib ;; Indeed, commenting out the above and rebuilding beam works; 1> erlang:trace(all,true,[call,cpu_timestamp]). 24 Is there any reason not to enable this? mats -------------- next part -------------- An HTML attachment was scrubbed... URL: From gleber.p@REDACTED Thu Jul 19 09:13:49 2012 From: gleber.p@REDACTED (Gleb Peregud) Date: Thu, 19 Jul 2012 09:13:49 +0200 Subject: [erlang-questions] error notification via mail In-Reply-To: References: Message-ID: Amazon SQS with erlcloud On Thu, Jul 19, 2012 at 9:11 AM, Andrzej Sliwa wrote: > there is any simple (and ready to use) solution to send error notification via mail ? > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From a.shneyderman@REDACTED Thu Jul 19 09:32:49 2012 From: a.shneyderman@REDACTED (Alex Shneyderman) Date: Thu, 19 Jul 2012 09:32:49 +0200 Subject: [erlang-questions] error notification via mail In-Reply-To: References: Message-ID: On Thu, Jul 19, 2012 at 9:13 AM, Gleb Peregud wrote: > Amazon SQS with erlcloud How does that send a mail? I guess a more appropriate service would be SNS, although erlcloud does not appear to have an adapter for it. From gleber.p@REDACTED Thu Jul 19 09:39:29 2012 From: gleber.p@REDACTED (Gleb Peregud) Date: Thu, 19 Jul 2012 09:39:29 +0200 Subject: [erlang-questions] error notification via mail In-Reply-To: References: Message-ID: On Thu, Jul 19, 2012 at 9:32 AM, Alex Shneyderman wrote: > How does that send a mail? I guess a more appropriate service would be SNS, > although erlcloud does not appear to have an adapter for it. True. A result of sleepy me sending mails at too early hour. Sorry for noise. From carlsson.richard@REDACTED Thu Jul 19 09:43:44 2012 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Thu, 19 Jul 2012 09:43:44 +0200 Subject: [erlang-questions] error notification via mail In-Reply-To: References: Message-ID: <5007BAB0.6030103@gmail.com> On 07/19/2012 09:11 AM, Andrzej Sliwa wrote: > there is any simple (and ready to use) solution to send error notification via mail ? https://github.com/richcarl/sendmail /Richard From bengt.kleberg@REDACTED Thu Jul 19 09:39:56 2012 From: bengt.kleberg@REDACTED (Bengt Kleberg) Date: Thu, 19 Jul 2012 09:39:56 +0200 Subject: [erlang-questions] beginner: Alternative ways to start ssh application? Message-ID: <1342683596.4752.12.camel@seasc1027.dyn.rnd.as.sw.ericsson.se> Greetings, Is there more ways than ssh:start/0 to start ssh? I want to use SFTP. When I try: ssh_sftp:start_channel("137.58...", [{user, "jpT..."}, {password, "..."}] ). I get: {error,ssh_not_started} So I try to do: ssh:start(). but then I get {error,{already_started,ssh}} So, perhaps there is an alternative way to start ssh, that will please ssh_sftp? Or an alternative way to use ssh_sftp that will use the existing ssh? bengt From max.lapshin@REDACTED Thu Jul 19 09:49:23 2012 From: max.lapshin@REDACTED (Max Lapshin) Date: Thu, 19 Jul 2012 11:49:23 +0400 Subject: [erlang-questions] error notification via mail In-Reply-To: <5007BAB0.6030103@gmail.com> References: <5007BAB0.6030103@gmail.com> Message-ID: You should definitely look at hoptoad (airbrake.io). When you serve many clients, error may repeat up to several thousands times per second and it is impossible to send such amount of email. This is why you should launch agent, that collects errors, aggregates them and send only cumulative error message. I advise to take a look at hoptoad protocol: there are agents for many languages. From gleber.p@REDACTED Thu Jul 19 09:52:04 2012 From: gleber.p@REDACTED (Gleb Peregud) Date: Thu, 19 Jul 2012 09:52:04 +0200 Subject: [erlang-questions] error notification via mail In-Reply-To: References: <5007BAB0.6030103@gmail.com> Message-ID: On Thu, Jul 19, 2012 at 9:49 AM, Max Lapshin wrote: > I advise to take a look at hoptoad protocol: there are agents for many > languages. Including Erlang - https://github.com/kenpratt/erlbrake From uwe@REDACTED Thu Jul 19 10:21:56 2012 From: uwe@REDACTED (Uwe Dauernheim) Date: Thu, 19 Jul 2012 10:21:56 +0200 Subject: [erlang-questions] random:seed in R14, R15, ... In-Reply-To: References: <7DF3EA97-B12B-427E-9530-EB678537C6BC@dauernheim.net> Message-ID: <1FA1AAD9-D782-4879-8C74-BF75A02A1BAE@dauernheim.net> @Rapsey For now I don't use the seeding and just give predefined fixture data, thanks. @Samuel The problem is that given the same seed, you get different sequences: $ PATH=/Users/uwe/dev/OTP/R14B03/bin erl Erlang R14B03 ... 1> random:seed({1,2,3}). undefined 2> random:uniform(). 1.9796325776202811e-4 3> random:uniform(). 0.03381877363047378 4> random:uniform(). 0.7775418875596665 $ PATH=/Users/uwe/dev/OTP/R15B01/bin erl Erlang R15B01 ... 1> random:seed({1,2,3}). undefined 2> random:uniform(). 0.05074967983013061 3> random:uniform(). 0.6727957987976656 4> random:uniform(). 0.16422626735554258 But it was my bad, the documentation actually now states that: "The implementation changed in R15. Upgrading to R15 will break applications that expect a specific output for a given seed. The output is still deterministic number series, but different compared to releases older than R15." On Jul 19, 2012, at 8:37 AM, Samuel wrote: >> When writing test cases that exercise e.g. random:uniform/0 one wants to use random:seed/1 with a fixed seed for deterministic test results, but providing the seed function with a fixed value seems to result in a different sequence for R14 and R15 (maybe for older or upcoming release as well). This is a problem when distributing your code and you are not sure which release will be used. >> >> What is the common solution for this? > > For randomised tests cases I usually seed the generator with now() as > I would do for normal pseudorandom usage, but I write the tests so > that, if failed, I get the originating seed in the report. That means > that the tests are not deterministic in general, but deterministic > enough to replay if needed. In theory, that way you also increase the > chances of finding a weird failing sequence by repeating the tests > over and over. > > It is similar to the strategy used by QuickCheck, which is a bit more > advanced. There you don't control the seed either, but when something > fails it reports the counterexample back, so you don't need the seed. > > In general, if you are testing with pseudorandom generators is because > you want to increase the coverage in a state space to big to be > covered by just exploring the possible combinations, so you don't want > to restrict your random sequences to just one. > > Regards > -- > Samuel From andrzej.sliwa@REDACTED Thu Jul 19 10:55:52 2012 From: andrzej.sliwa@REDACTED (Andrzej Sliwa) Date: Thu, 19 Jul 2012 10:55:52 +0200 Subject: [erlang-questions] error notification via mail In-Reply-To: References: <5007BAB0.6030103@gmail.com> Message-ID: <23F44D94-97DD-4140-9BB5-E7D8CB02B00D@i-tool.eu> yep, I just using it ? with AirBrake ? but want to switch to something cost free? I just testing errbit on heroku - https://github.com/errbit/errbit/ On Jul 19, 2012, at 9:52 AM, Gleb Peregud wrote: > On Thu, Jul 19, 2012 at 9:49 AM, Max Lapshin wrote: >> I advise to take a look at hoptoad protocol: there are agents for many >> languages. > > Including Erlang - https://github.com/kenpratt/erlbrake > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From andrzej.sliwa@REDACTED Thu Jul 19 11:01:24 2012 From: andrzej.sliwa@REDACTED (Andrzej Sliwa) Date: Thu, 19 Jul 2012 11:01:24 +0200 Subject: [erlang-questions] error notification via mail In-Reply-To: <23F44D94-97DD-4140-9BB5-E7D8CB02B00D@i-tool.eu> References: <5007BAB0.6030103@gmail.com> <23F44D94-97DD-4140-9BB5-E7D8CB02B00D@i-tool.eu> Message-ID: anyway error notifier (via mail) is so basic tool in Ruby/Rails or Python/Django world, and I wonder how Erlang community surviving without free, simple & ready to use solution like this. I can bet that you have it, but you keep it only for self ? you selfish erlangers ;) c'mon open source don't hurt :D On Jul 19, 2012, at 10:55 AM, Andrzej Sliwa wrote: > yep, I just using it ? with AirBrake ? > but want to switch to something cost free? > > I just testing errbit on heroku - https://github.com/errbit/errbit/ > > On Jul 19, 2012, at 9:52 AM, Gleb Peregud wrote: > >> On Thu, Jul 19, 2012 at 9:49 AM, Max Lapshin wrote: >>> I advise to take a look at hoptoad protocol: there are agents for many >>> languages. >> >> Including Erlang - https://github.com/kenpratt/erlbrake >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > From max.lapshin@REDACTED Thu Jul 19 12:03:23 2012 From: max.lapshin@REDACTED (Max Lapshin) Date: Thu, 19 Jul 2012 14:03:23 +0400 Subject: [erlang-questions] error notification via mail In-Reply-To: References: <5007BAB0.6030103@gmail.com> <23F44D94-97DD-4140-9BB5-E7D8CB02B00D@i-tool.eu> Message-ID: You can try to use redmine hoptoad plugin, that accepts incoming bugs and posts them as tickets. From roe.adrian@REDACTED Thu Jul 19 12:38:39 2012 From: roe.adrian@REDACTED (Adrian Roe) Date: Thu, 19 Jul 2012 11:38:39 +0100 Subject: [erlang-questions] error notification via mail In-Reply-To: <23F44D94-97DD-4140-9BB5-E7D8CB02B00D@i-tool.eu> References: <5007BAB0.6030103@gmail.com> <23F44D94-97DD-4140-9BB5-E7D8CB02B00D@i-tool.eu> Message-ID: <5007E3AF.9020304@gmail.com> I've attached 4 files that together form a remarkably simple email sender: smtp.erl - a very simple SMTP protocol that happily talks to GMail (you fill in your account details in the smtp_info record) smtp.hrl - container for the smtp_info record mail_sender.erl - not really needed, but an equally simple gen server that provides fire and forget access to smtp.erl i_convert.erl - utility data conversion routines It's one of the first things I ever wrote in Erlang so it's not pretty, but it has been in production for over a year and does appear to "just work" (no warranties, feel free to make whatever changes you want, please don't sue me...). Backoff etc (not sending too many emails in too short a time) we have always implemented higher up the food chain - we've functions with names like "maybe_send_email" that do simple checks as to whether we are needlessly filling inboxes with support messages. Those functions are just a few lines long and in our case are tightly coupled to the business logic that requires the email to be sent so we've never seen the point in trying to make that generic. Feel free to have a play - it should be trivial to get going, but do drop me an email if you have any questions. If you find it useful I might even put it up as a simple github project with docs (surely not) and an example or two... Here's how you fill in the record (for GMail). #smtp_info{ name = whatever_you_want, %% The name (atom) of the gen_server (if you use mail_sender.erl) from = <<"your_account@REDACTED">>, username = <<"your_account@REDACTED">>, password = <<"somev3rysecretthing">>, server = <<"smtp.gmail.com">>, port = 465}. To send an email (directly - the gen_server is self explanatory) you send_mail(SmtpInfo, To, %% <<"erlang-questions@REDACTED">> ReplyTo, %% Binary - the "from" in the email - should really be smtp_info.from Subject, %% Body, Attachments, %% List of attachments to send with the email, Type %% text or html). Read the code - it's simple enough to be documentation for now ;) Adrian *Dr Adrian Roe Director id3as* On 19/07/12 09:55, Andrzej Sliwa wrote: > yep, I just using it ? with AirBrake ? > but want to switch to something cost free? > > I just testing errbit on heroku - https://github.com/errbit/errbit/ > > On Jul 19, 2012, at 9:52 AM, Gleb Peregud wrote: > >> On Thu, Jul 19, 2012 at 9:49 AM, Max Lapshin wrote: >>> I advise to take a look at hoptoad protocol: there are agents for many >>> languages. >> Including Erlang - https://github.com/kenpratt/erlbrake >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: i_convert.erl Type: text/x-erlang Size: 7301 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: mail_sender.erl Type: text/x-erlang Size: 3577 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smtp.erl Type: text/x-erlang Size: 4933 bytes Desc: not available URL: -------------- next part -------------- -record(smtp_info, {name, from, username, password, server, port}). From diego.llarrull@REDACTED Thu Jul 19 16:13:49 2012 From: diego.llarrull@REDACTED (Diego Llarrull) Date: Thu, 19 Jul 2012 11:13:49 -0300 Subject: [erlang-questions] Cover compiling of .beam files inside .ez archives, is it possible? Message-ID: <5008161D.1060205@tecso.coop> Good day to everyone, I'm building a riak-core based application which relies con rebar + gnu make to generate deployment nodes (i.e: an embedded erts plus all source code and dependencies). Each app is stored in an .ez file inside the "lib" folder. No unzipping of the files in a special directory occurs in runtime because the code server is able to load .beam files from archives (tough experimentally, as stated in http://www.erlang.org/doc/man/code.html). The cover server, instead, can't access such files, which makes it impossible to perform cover analysis in runtime (something we require because the modules to analyse rely on servers which are spawned on demand). My question would be the following: 1) Is it possible to cover compile the binaries at compile-time (i.e., before generating the node releases) ? 2) Is there any way to access the .ez files without unzipping them, so as to pass a reference to that file to cover:compile_beam() ? (Note to rebar users: rebar eunit doesn't do the job for me, because it performs offline, static, coverage analysis). Any help would be greatly appreciated. Thanks in advance Diego Llarrull From gordon@REDACTED Thu Jul 19 22:34:19 2012 From: gordon@REDACTED (Gordon Guthrie) Date: Thu, 19 Jul 2012 21:34:19 +0100 Subject: [erlang-questions] Long polling In-Reply-To: References: Message-ID: Kaiduanx Essentially, is this testing how long different bits of kit will tolerate long-open connections? We already have one long remoting connection but this pushes updates to the front-end and the front-end polls it - and is fairly straightforward and works well. For what I am doing now I need to get a response back from the browser, so I think I will need another approach. Gordon On 17 July 2012 16:34, Kaiduan Xie wrote: > I assume that you are talking about the TCP NAT keep-alive time. > Please check the following paper, > > http://nutss.gforge.cis.cornell.edu/stunt.php > http://saikat.guha.cc/stunt-results.php? > > Please look at the Timer section in the survey. > > Why you need to use HTTP long poll? You can just use a persistent > TCP/TLS connection and periodically send keep-alive message to keep > the TCP/TLS connection alive. > > Or you can use Websocket, it is designed to replace long poll. > > /Kaiduan > > On Tue, Jul 17, 2012 at 11:01 AM, Gleb Peregud wrote: >> On Tue, Jul 17, 2012 at 4:57 PM, Gordon Guthrie wrote: >>> Folks >>> >>> I am building a softphone which needs to register with the server to say "I >>> am available" for incoming calls. >>> >>> So it makes an HTTP POST request to the server and that notifies the >>> softphone gen_srv that is is available. >>> >>> At that point I leave the TCP request hanging on a receive waiting for a >>> notification that the socket has been torn down (and which point it >>> unregisters the phone). >>> >>> My question is: how long can I leave the request up? Infinity? or do I need >>> to have a timeout/reregister cycle setup? >> >> You have to cycle it. Some proxies timeout connections after 30 >> seconds, some after 45 some after 60. It is very tricky to get it >> right for all browsers and all platforms. Generally if you want a >> robust solution use something like SockJS or Socket.IO or Bullet. >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions From gordon@REDACTED Thu Jul 19 23:02:42 2012 From: gordon@REDACTED (Gordon Guthrie) Date: Thu, 19 Jul 2012 22:02:42 +0100 Subject: [erlang-questions] Mochiwebsockets was Re: Long polling Message-ID: Gleb The app is built on mochiweb - I inspect the request and route it to a handler. So I would need to bind the server part of the library to the that particular HTTP request. Bullet is designed to work in Cowboy, Socket.io has its own Erlang server SockJS is designed to be web-framework agnostic but the examples are all in Cowboy As I understand it - these socket solutions have a code the server side and client side which ping traffic backwards and forwards and keep a 'permanent' connection open - and the higher level libraries can then write to the socket as if there was a stateful synchronous connection. So to use one of these I would need to pick a particular client JS library and then write my own handler to use on the socket in mochiweb (or cut one out from one of the existing servers...) Has anyone done stuff on getting it into mochiweb? Gordon On 17 July 2012 16:01, Gleb Peregud wrote: > On Tue, Jul 17, 2012 at 4:57 PM, Gordon Guthrie wrote: >> Folks >> >> I am building a softphone which needs to register with the server to say "I >> am available" for incoming calls. >> >> So it makes an HTTP POST request to the server and that notifies the >> softphone gen_srv that is is available. >> >> At that point I leave the TCP request hanging on a receive waiting for a >> notification that the socket has been torn down (and which point it >> unregisters the phone). >> >> My question is: how long can I leave the request up? Infinity? or do I need >> to have a timeout/reregister cycle setup? > > You have to cycle it. Some proxies timeout connections after 30 > seconds, some after 45 some after 60. It is very tricky to get it > right for all browsers and all platforms. Generally if you want a > robust solution use something like SockJS or Socket.IO or Bullet. From ok@REDACTED Fri Jul 20 00:12:20 2012 From: ok@REDACTED (Richard O'Keefe) Date: Fri, 20 Jul 2012 10:12:20 +1200 Subject: [erlang-questions] random:seed in R14, R15, ... In-Reply-To: <7DF3EA97-B12B-427E-9530-EB678537C6BC@dauernheim.net> References: <7DF3EA97-B12B-427E-9530-EB678537C6BC@dauernheim.net> Message-ID: <97336266-8375-43DE-A156-F61EFC34C3DA@cs.otago.ac.nz> On 19/07/2012, at 6:23 PM, Uwe Dauernheim wrote: > When writing test cases that exercise e.g. random:uniform/0 one wants to use random:seed/1 with a fixed seed for deterministic test results, but providing the seed function with a fixed value seems to result in a different sequence for R14 and R15 (maybe for older or upcoming release as well). This is a problem when distributing your code and you are not sure which release will be used. > > What is the common solution for this? The solution that is common across programming languages is to use your own random number generator. For example, here's Marsaglia's KISS in Erlang: initial_state() -> { 123456789, 362436000, 521288629, 7654321 }. kiss({X0,Y0,Z0,C0}) -> X = 69069 * X0 + 12345, Y1 = ((Y0 band 524287) bsl 13) bxor Y0, Y2 = (Y1 bsr 17) band Y1, Y = ((Y2 band 134217727) bsl 5) bxor Y2, T = Z0 * 698769069 + C0, Z = T band 4294967295, C = T bsr 32, {(X + Y + Z) band 4294967295, {X,Y,Z,C}}. This generates a stream of 32-bit integers "with period > 2^125" according to Marsaglia. (I hope I've transcribed it correctly.) The initial state is any four 32-bit integers. kissf(State0) -> {I, State} = kiss(State), {(I+1)/4294967297.0, State}. This gives you 32 bits worth of random floats in the open interval (0.0,1.0). With trickier code one could do better. From essen@REDACTED Fri Jul 20 00:24:40 2012 From: essen@REDACTED (=?ISO-8859-1?Q?Lo=EFc_Hoguin?=) Date: Fri, 20 Jul 2012 00:24:40 +0200 Subject: [erlang-questions] Mochiwebsockets was Re: Long polling In-Reply-To: References: Message-ID: <50088928.6000506@ninenines.eu> Or you can replace Mochiweb by Cowboy+Mochicow (https://github.com/benoitc/mochicow) and use Cowboy's routing to redirect to your websocket handler or your mochiweb code. On 07/19/2012 11:02 PM, Gordon Guthrie wrote: > Gleb > > The app is built on mochiweb - I inspect the request and route it to a > handler. So I would need to bind the server part of the library to the > that particular HTTP request. > > Bullet is designed to work in Cowboy, > Socket.io has its own Erlang server > SockJS is designed to be web-framework agnostic but the examples are > all in Cowboy > > As I understand it - these socket solutions have a code the server > side and client side which ping traffic backwards and forwards and > keep a 'permanent' connection open - and the higher level libraries > can then write to the socket as if there was a stateful synchronous > connection. > > So to use one of these I would need to pick a particular client JS > library and then write my own handler to use on the socket in mochiweb > (or cut one out from one of the existing servers...) > > Has anyone done stuff on getting it into mochiweb? > > Gordon > > > On 17 July 2012 16:01, Gleb Peregud wrote: >> On Tue, Jul 17, 2012 at 4:57 PM, Gordon Guthrie wrote: >>> Folks >>> >>> I am building a softphone which needs to register with the server to say "I >>> am available" for incoming calls. >>> >>> So it makes an HTTP POST request to the server and that notifies the >>> softphone gen_srv that is is available. >>> >>> At that point I leave the TCP request hanging on a receive waiting for a >>> notification that the socket has been torn down (and which point it >>> unregisters the phone). >>> >>> My question is: how long can I leave the request up? Infinity? or do I need >>> to have a timeout/reregister cycle setup? >> >> You have to cycle it. Some proxies timeout connections after 30 >> seconds, some after 45 some after 60. It is very tricky to get it >> right for all browsers and all platforms. Generally if you want a >> robust solution use something like SockJS or Socket.IO or Bullet. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From gordon@REDACTED Fri Jul 20 00:42:40 2012 From: gordon@REDACTED (Gordon Guthrie) Date: Thu, 19 Jul 2012 23:42:40 +0100 Subject: [erlang-questions] Mochiwebsockets was Re: Long polling In-Reply-To: <50088928.6000506@ninenines.eu> References: <50088928.6000506@ninenines.eu> Message-ID: Lo?c So I would then have 3 things running my application - mochiweb, cowboy and a socket server... I am a big fan of avoiding code at all costs :( Given that I have a simple understood use case, mebbies I don't need a fully featured socket server, mebbies I need to put a simple client-driven registration system... Gordon On 19 July 2012 23:24, Lo?c Hoguin wrote: > Or you can replace Mochiweb by Cowboy+Mochicow > (https://github.com/benoitc/mochicow) and use Cowboy's routing to redirect > to your websocket handler or your mochiweb code. > > > On 07/19/2012 11:02 PM, Gordon Guthrie wrote: >> >> Gleb >> >> The app is built on mochiweb - I inspect the request and route it to a >> handler. So I would need to bind the server part of the library to the >> that particular HTTP request. >> >> Bullet is designed to work in Cowboy, >> Socket.io has its own Erlang server >> SockJS is designed to be web-framework agnostic but the examples are >> all in Cowboy >> >> As I understand it - these socket solutions have a code the server >> side and client side which ping traffic backwards and forwards and >> keep a 'permanent' connection open - and the higher level libraries >> can then write to the socket as if there was a stateful synchronous >> connection. >> >> So to use one of these I would need to pick a particular client JS >> library and then write my own handler to use on the socket in mochiweb >> (or cut one out from one of the existing servers...) >> >> Has anyone done stuff on getting it into mochiweb? >> >> Gordon >> >> >> On 17 July 2012 16:01, Gleb Peregud wrote: >>> >>> On Tue, Jul 17, 2012 at 4:57 PM, Gordon Guthrie wrote: >>>> >>>> Folks >>>> >>>> I am building a softphone which needs to register with the server to say >>>> "I >>>> am available" for incoming calls. >>>> >>>> So it makes an HTTP POST request to the server and that notifies the >>>> softphone gen_srv that is is available. >>>> >>>> At that point I leave the TCP request hanging on a receive waiting for a >>>> notification that the socket has been torn down (and which point it >>>> unregisters the phone). >>>> >>>> My question is: how long can I leave the request up? Infinity? or do I >>>> need >>>> to have a timeout/reregister cycle setup? >>> >>> >>> You have to cycle it. Some proxies timeout connections after 30 >>> seconds, some after 45 some after 60. It is very tricky to get it >>> right for all browsers and all platforms. Generally if you want a >>> robust solution use something like SockJS or Socket.IO or Bullet. >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > > -- > Lo?c Hoguin > Erlang Cowboy > Nine Nines > http://ninenines.eu > > From essen@REDACTED Fri Jul 20 00:46:56 2012 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Fri, 20 Jul 2012 00:46:56 +0200 Subject: [erlang-questions] Mochiwebsockets was Re: Long polling In-Reply-To: References: <50088928.6000506@ninenines.eu> Message-ID: <50088E60.7010309@ninenines.eu> Just Cowboy. Mochicow is plugged on top of Cowboy and replaces Mochiweb entirely. You don't need a "socket server", whatever that is, you can just use websockets directly in Cowboy, or if you need a long polling fallback there's Bullet available, which is just a library of one module + one js file really. Either way they don't try to anticipate your needs, they just transport data. On 07/20/2012 12:42 AM, Gordon Guthrie wrote: > Lo?c > > So I would then have 3 things running my application - mochiweb, > cowboy and a socket server... > > I am a big fan of avoiding code at all costs :( > > Given that I have a simple understood use case, mebbies I don't need a > fully featured socket server, mebbies I need to put a simple > client-driven registration system... > > Gordon > > On 19 July 2012 23:24, Lo?c Hoguin wrote: >> Or you can replace Mochiweb by Cowboy+Mochicow >> (https://github.com/benoitc/mochicow) and use Cowboy's routing to redirect >> to your websocket handler or your mochiweb code. >> >> >> On 07/19/2012 11:02 PM, Gordon Guthrie wrote: >>> >>> Gleb >>> >>> The app is built on mochiweb - I inspect the request and route it to a >>> handler. So I would need to bind the server part of the library to the >>> that particular HTTP request. >>> >>> Bullet is designed to work in Cowboy, >>> Socket.io has its own Erlang server >>> SockJS is designed to be web-framework agnostic but the examples are >>> all in Cowboy >>> >>> As I understand it - these socket solutions have a code the server >>> side and client side which ping traffic backwards and forwards and >>> keep a 'permanent' connection open - and the higher level libraries >>> can then write to the socket as if there was a stateful synchronous >>> connection. >>> >>> So to use one of these I would need to pick a particular client JS >>> library and then write my own handler to use on the socket in mochiweb >>> (or cut one out from one of the existing servers...) >>> >>> Has anyone done stuff on getting it into mochiweb? >>> >>> Gordon >>> >>> >>> On 17 July 2012 16:01, Gleb Peregud wrote: >>>> >>>> On Tue, Jul 17, 2012 at 4:57 PM, Gordon Guthrie wrote: >>>>> >>>>> Folks >>>>> >>>>> I am building a softphone which needs to register with the server to say >>>>> "I >>>>> am available" for incoming calls. >>>>> >>>>> So it makes an HTTP POST request to the server and that notifies the >>>>> softphone gen_srv that is is available. >>>>> >>>>> At that point I leave the TCP request hanging on a receive waiting for a >>>>> notification that the socket has been torn down (and which point it >>>>> unregisters the phone). >>>>> >>>>> My question is: how long can I leave the request up? Infinity? or do I >>>>> need >>>>> to have a timeout/reregister cycle setup? >>>> >>>> >>>> You have to cycle it. Some proxies timeout connections after 30 >>>> seconds, some after 45 some after 60. It is very tricky to get it >>>> right for all browsers and all platforms. Generally if you want a >>>> robust solution use something like SockJS or Socket.IO or Bullet. >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >> >> >> -- >> Lo?c Hoguin >> Erlang Cowboy >> Nine Nines >> http://ninenines.eu >> >> -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From samuelrivas@REDACTED Fri Jul 20 08:36:21 2012 From: samuelrivas@REDACTED (Samuel) Date: Fri, 20 Jul 2012 08:36:21 +0200 Subject: [erlang-questions] random:seed in R14, R15, ... In-Reply-To: <1FA1AAD9-D782-4879-8C74-BF75A02A1BAE@dauernheim.net> References: <7DF3EA97-B12B-427E-9530-EB678537C6BC@dauernheim.net> <1FA1AAD9-D782-4879-8C74-BF75A02A1BAE@dauernheim.net> Message-ID: > @Samuel The problem is that given the same seed, you get different sequences: Yes, but my approach is that I let the tests work in different sequences anyway, as I am generating the seed for each run. When something fails I just recover the seed and replay the test with it. As I will replay in the same platform it failed, I will get the same sequence. I understand your problem is that you want to have the same sequences in both releases, my approach was to avoid that need more than to solve your exact problem :) -- Samuel From gleber.p@REDACTED Fri Jul 20 08:39:57 2012 From: gleber.p@REDACTED (Gleb Peregud) Date: Fri, 20 Jul 2012 08:39:57 +0200 Subject: [erlang-questions] Mochiwebsockets was Re: Long polling In-Reply-To: <50088E60.7010309@ninenines.eu> References: <50088928.6000506@ninenines.eu> <50088E60.7010309@ninenines.eu> Message-ID: On Fri, Jul 20, 2012 at 12:46 AM, Lo?c Hoguin wrote: > You don't need a "socket server", whatever that is, you can just use > websockets directly in Cowboy, or if you need a long polling fallback > there's Bullet available, which is just a library of one module + one js > file really. Either way they don't try to anticipate your needs, they just > transport data. There are few basic questions which are very important here to decide which approach to use: 1) Do you plan to use it cross-domain or just on one domain? 2) Do you want to have as low latency as possible on every browser? Or not? 3) Are you interested in just one-directional server-to-client push? Or in both directions? 4) Do you need fast and robust connection error detection? 5) Do you use jQuery already? How large JS library will you tolerate? 6) How large server-side library will you tolerate? If you don't care about latency that much and you are dealing with just one domain - probably Cowboy + Bullet would be easiest to use. If you care about latency in all browser and you have to work cross-domain - probably SockJS would be the best solution. If you want to port server-side part of SockJS logic into your app, it should be pretty straightforward and I can help you with it. IMO in a long run it is probably a good idea to switch to Cowboy (and Mochicow) would help you a lot with the switch. From samuelrivas@REDACTED Fri Jul 20 08:43:04 2012 From: samuelrivas@REDACTED (Samuel) Date: Fri, 20 Jul 2012 08:43:04 +0200 Subject: [erlang-questions] beginner: Alternative ways to start ssh application? In-Reply-To: <1342683596.4752.12.camel@seasc1027.dyn.rnd.as.sw.ericsson.se> References: <1342683596.4752.12.camel@seasc1027.dyn.rnd.as.sw.ericsson.se> Message-ID: > Is there more ways than ssh:start/0 to start ssh? Start the ssh application? 1> application:start(crypto). ok 2> application:start(ssh). ok 3> ssh_sftp:start_channel("192.168.1.1", [{user, "foo"}, {password, "bar"}]). ... -- Samuel From essen@REDACTED Fri Jul 20 09:11:00 2012 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Fri, 20 Jul 2012 09:11:00 +0200 Subject: [erlang-questions] Mochiwebsockets was Re: Long polling In-Reply-To: References: <50088928.6000506@ninenines.eu> <50088E60.7010309@ninenines.eu> Message-ID: <50090484.8030207@ninenines.eu> On 07/20/2012 08:39 AM, Gleb Peregud wrote: > On Fri, Jul 20, 2012 at 12:46 AM, Lo?c Hoguin wrote: >> You don't need a "socket server", whatever that is, you can just use >> websockets directly in Cowboy, or if you need a long polling fallback >> there's Bullet available, which is just a library of one module + one js >> file really. Either way they don't try to anticipate your needs, they just >> transport data. > > There are few basic questions which are very important here to decide > which approach to use: > > 1) Do you plan to use it cross-domain or just on one domain? > 2) Do you want to have as low latency as possible on every browser? Or not? > 3) Are you interested in just one-directional server-to-client push? > Or in both directions? > 4) Do you need fast and robust connection error detection? > 5) Do you use jQuery already? How large JS library will you tolerate? > 6) How large server-side library will you tolerate? > > If you don't care about latency that much and you are dealing with > just one domain - probably Cowboy + Bullet would be easiest to use. > > If you care about latency in all browser and you have to work > cross-domain - probably SockJS would be the best solution. Why? I don't see any reason why SockJS would have better latency. And for cross-domain, it's just a matter of sending one header. > If you want > to port server-side part of SockJS logic into your app, it should be > pretty straightforward and I can help you with it. > > IMO in a long run it is probably a good idea to switch to Cowboy (and > Mochicow) would help you a lot with the switch. > -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From gleber.p@REDACTED Fri Jul 20 09:16:50 2012 From: gleber.p@REDACTED (Gleb Peregud) Date: Fri, 20 Jul 2012 09:16:50 +0200 Subject: [erlang-questions] Mochiwebsockets was Re: Long polling In-Reply-To: <50090484.8030207@ninenines.eu> References: <50088928.6000506@ninenines.eu> <50088E60.7010309@ninenines.eu> <50090484.8030207@ninenines.eu> Message-ID: On Fri, Jul 20, 2012 at 9:11 AM, Lo?c Hoguin wrote: > Why? I don't see any reason why SockJS would have better latency. Because many browsers doesn't support websockets and because streaming transports (xhrstreaming, htmlfile streaming, eventsource streaming, etc.) have better latency than long polling trasnport. > And for cross-domain, it's just a matter of sending one header. Because it works only in most recent browser. It doesn't work for most older browsers, which are, unfortunately, still in use. Bullet is useless in cross-domain setup in very many browsers, which doesn't have cross domain xhr request support. From bengt.kleberg@REDACTED Fri Jul 20 09:19:10 2012 From: bengt.kleberg@REDACTED (Bengt Kleberg) Date: Fri, 20 Jul 2012 09:19:10 +0200 Subject: [erlang-questions] beginner: Alternative ways to start ssh application? In-Reply-To: References: <1342683596.4752.12.camel@seasc1027.dyn.rnd.as.sw.ericsson.se> Message-ID: <1342768750.4931.23.camel@seasc1027.dyn.rnd.as.sw.ericsson.se> application:start(ssh) gives the same error as ssh:start(). Ie {error,{already_started,ssh}} So, the question remains, if ssh_sftp:start_channel/2 says {error,ssh_not_started} how do I start ssh, if it is already started? bengt And On Fri, 2012-07-20 at 08:43 +0200, Samuel wrote: > > Is there more ways than ssh:start/0 to start ssh? > > Start the ssh application? > > 1> application:start(crypto). > ok > 2> application:start(ssh). > ok > 3> ssh_sftp:start_channel("192.168.1.1", [{user, "foo"}, {password, "bar"}]). > ... > From gleber.p@REDACTED Fri Jul 20 09:28:25 2012 From: gleber.p@REDACTED (Gleb Peregud) Date: Fri, 20 Jul 2012 09:28:25 +0200 Subject: [erlang-questions] Mochiwebsockets was Re: Long polling In-Reply-To: References: <50088928.6000506@ninenines.eu> <50088E60.7010309@ninenines.eu> Message-ID: On Fri, Jul 20, 2012 at 8:39 AM, Gleb Peregud wrote: > There are few basic questions which are very important here to decide > which approach to use: Oh, and obviously one more: 0) Which browsers do you want to support? From essen@REDACTED Fri Jul 20 09:30:15 2012 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Fri, 20 Jul 2012 09:30:15 +0200 Subject: [erlang-questions] Mochiwebsockets was Re: Long polling In-Reply-To: References: <50088928.6000506@ninenines.eu> <50088E60.7010309@ninenines.eu> <50090484.8030207@ninenines.eu> Message-ID: <50090907.9080209@ninenines.eu> On 07/20/2012 09:16 AM, Gleb Peregud wrote: > On Fri, Jul 20, 2012 at 9:11 AM, Lo?c Hoguin wrote: >> Why? I don't see any reason why SockJS would have better latency. > > Because many browsers doesn't support websockets and because streaming > transports (xhrstreaming, htmlfile streaming, eventsource streaming, > etc.) have better latency than long polling trasnport. Probably different depending on the location, but I've observed personally that browsers not able to do websockets were the minority. We have around 70% of webkit/gecko, it's only really IE that could use improved latency. What's the best solution for IE? I could add it to Bullet. >> And for cross-domain, it's just a matter of sending one header. > > Because it works only in most recent browser. It doesn't work for most > older browsers, which are, unfortunately, still in use. > > Bullet is useless in cross-domain setup in very many browsers, which > doesn't have cross domain xhr request support. You don't need cross-domain though, you can just include the files from anywhere and use $.bullet(url) inside your page. Or event mylib(url) which subsequently calls bullet. The only requirement is that the origin comes from the same domain. This doesn't solve calling an entirely different application you don't control but that one's very unusual. -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From gleber.p@REDACTED Fri Jul 20 09:36:47 2012 From: gleber.p@REDACTED (Gleb Peregud) Date: Fri, 20 Jul 2012 09:36:47 +0200 Subject: [erlang-questions] Mochiwebsockets was Re: Long polling In-Reply-To: <50090907.9080209@ninenines.eu> References: <50088928.6000506@ninenines.eu> <50088E60.7010309@ninenines.eu> <50090484.8030207@ninenines.eu> <50090907.9080209@ninenines.eu> Message-ID: On Fri, Jul 20, 2012 at 9:30 AM, Lo?c Hoguin wrote: >> Because many browsers doesn't support websockets and because streaming >> transports (xhrstreaming, htmlfile streaming, eventsource streaming, >> etc.) have better latency than long polling trasnport. > > Probably different depending on the location, but I've observed personally > that browsers not able to do websockets were the minority. We have around > 70% of webkit/gecko, it's only really IE that could use improved latency. > What's the best solution for IE? I could add it to Bullet. Take a look at here for the best available transports per browser (cross domain): https://github.com/sockjs/sockjs-client#supported-transports-by-browser-html-served-from-http-or-https > You don't need cross-domain though, Gordon didn't mention if he needs it or not, so I'm presenting the most general information on the topic. > you can just include the files from > anywhere and use $.bullet(url) inside your page. Or event mylib(url) which > subsequently calls bullet. The only requirement is that the origin comes > from the same domain. It's worth noting on Bullet's web page that it assumes single domain setup in general. From bchesneau@REDACTED Fri Jul 20 09:41:07 2012 From: bchesneau@REDACTED (Benoit Chesneau) Date: Fri, 20 Jul 2012 09:41:07 +0200 Subject: [erlang-questions] Mochiwebsockets was Re: Long polling In-Reply-To: References: <50088928.6000506@ninenines.eu> <50088E60.7010309@ninenines.eu> <50090484.8030207@ninenines.eu> Message-ID: On Fri, Jul 20, 2012 at 9:16 AM, Gleb Peregud wrote: > Because it works only in most recent browser. It doesn't work for most > older browsers, which are, unfortunately, still in use. > > Bullet is useless in cross-domain setup in very many browsers, which > doesn't have cross domain xhr request support. I guess adding CORS support to bullet is quite easy and would solve the problem. - beno?t From essen@REDACTED Fri Jul 20 09:43:12 2012 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Fri, 20 Jul 2012 09:43:12 +0200 Subject: [erlang-questions] Mochiwebsockets was Re: Long polling In-Reply-To: References: <50088928.6000506@ninenines.eu> <50088E60.7010309@ninenines.eu> <50090484.8030207@ninenines.eu> <50090907.9080209@ninenines.eu> Message-ID: <50090C10.6070401@ninenines.eu> On 07/20/2012 09:36 AM, Gleb Peregud wrote: >> you can just include the files from >> anywhere and use $.bullet(url) inside your page. Or event mylib(url) which >> subsequently calls bullet. The only requirement is that the origin comes >> from the same domain. > > It's worth noting on Bullet's web page that it assumes single domain > setup in general. It's supposed to follow the same rules as Websockets. If it doesn't, it's a bug, and you should open a ticket. :) -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From gleber.p@REDACTED Fri Jul 20 09:51:41 2012 From: gleber.p@REDACTED (Gleb Peregud) Date: Fri, 20 Jul 2012 09:51:41 +0200 Subject: [erlang-questions] Mochiwebsockets was Re: Long polling In-Reply-To: <50090C10.6070401@ninenines.eu> References: <50088928.6000506@ninenines.eu> <50088E60.7010309@ninenines.eu> <50090484.8030207@ninenines.eu> <50090907.9080209@ninenines.eu> <50090C10.6070401@ninenines.eu> Message-ID: On Fri, Jul 20, 2012 at 9:43 AM, Lo?c Hoguin wrote: > It's supposed to follow the same rules as Websockets. If it doesn't, it's a > bug, and you should open a ticket. :) I won't, since I don't use Bullet :) SockJS provides more complete solution for my needs (cross-domain setup and possibly best latency in all browsers). SockJS-client provides all necessary documentation to be able to figure out what Bullet is still missing to satisfy my needs. Most probably Gordon has more moderate requirements and Bullet will work nicely in his case :) From gordon@REDACTED Fri Jul 20 10:24:36 2012 From: gordon@REDACTED (Gordon Guthrie) Date: Fri, 20 Jul 2012 09:24:36 +0100 Subject: [erlang-questions] Mochiwebsockets was Re: Long polling In-Reply-To: References: <50088928.6000506@ninenines.eu> <50088E60.7010309@ninenines.eu> Message-ID: Gleb There are a couple of complications - we already use one long-poll for server-to-client updates - in that case the client posts changes and the server works on them and then streams out updates on a separate long-poll. Lets put that aside and focus on this use case. For the softphone: > 1) Do you plan to use it cross-domain or just on one domain? Single domain > 2) Do you want to have as low latency as possible on every browser? Or not? It is a single registration event - is this softphone available for me to get Twilio to push a phone call to it - the actual phone call is handled peer-to-peer in the browser - so latency is not so important - I have a call phone call setup time anyway > 3) Are you interested in just one-directional server-to-client push? > Or in both directions? It's the server saying "are you still there?" > 4) Do you need fast and robust connection error detection? Needs to be reliable. > 5) Do you use jQuery already? How large JS library will you tolerate? Yeah, lots of JQuery - given that the phone itself doesn't ship with much javascript it should be OK... > 6) How large server-side library will you tolerate? I will tolerate any size of code base if it is encapsulated nicely. Its just when the suggestion is that I rip out mochiweb, fire in this and that, I start thinking, erk! > IMO in a long run it is probably a good idea to switch to Cowboy (and > Mochicow) would help you a lot with the switch. What would the benefits of Cowboy over Mochiweb be? Gordon On 20 July 2012 07:39, Gleb Peregud wrote: > On Fri, Jul 20, 2012 at 12:46 AM, Lo?c Hoguin wrote: >> You don't need a "socket server", whatever that is, you can just use >> websockets directly in Cowboy, or if you need a long polling fallback >> there's Bullet available, which is just a library of one module + one js >> file really. Either way they don't try to anticipate your needs, they just >> transport data. > > There are few basic questions which are very important here to decide > which approach to use: > > 1) Do you plan to use it cross-domain or just on one domain? > 2) Do you want to have as low latency as possible on every browser? Or not? > 3) Are you interested in just one-directional server-to-client push? > Or in both directions? > 4) Do you need fast and robust connection error detection? > 5) Do you use jQuery already? How large JS library will you tolerate? > 6) How large server-side library will you tolerate? > > If you don't care about latency that much and you are dealing with > just one domain - probably Cowboy + Bullet would be easiest to use. > > If you care about latency in all browser and you have to work > cross-domain - probably SockJS would be the best solution. If you want > to port server-side part of SockJS logic into your app, it should be > pretty straightforward and I can help you with it. > > IMO in a long run it is probably a good idea to switch to Cowboy (and > Mochicow) would help you a lot with the switch. From gordon@REDACTED Fri Jul 20 10:33:19 2012 From: gordon@REDACTED (Gordon Guthrie) Date: Fri, 20 Jul 2012 09:33:19 +0100 Subject: [erlang-questions] Mochiwebsockets was Re: Long polling In-Reply-To: <50088E60.7010309@ninenines.eu> References: <50088928.6000506@ninenines.eu> <50088E60.7010309@ninenines.eu> Message-ID: Loic > Mochicow is plugged on top of Cowboy and replaces Mochiweb entirely. At the moment Mochiweb does it stuff and passes a MochiRequest into my handler. We then parse that MochiRequest, do a load of authentication on it and pass it into essentially some pattern matchers that handle POST's, GET's, content negotiation, blah-blah-blah. Are you saying that Cowboy would take an incoming request, parse it, decide - 'this is a long poll - pass it to bullet', 'this isn't pass it to mochicow' and the mochicow would give me an identically formed MochiRequest 'just like mochiweb'? We also make a lot of use of the mochijson/mochihex libraries to handle our JSON, hex and stuff - I could probably just pull those libraries out... Gordon On 19 July 2012 23:46, Lo?c Hoguin wrote: > Just Cowboy. > > Mochicow is plugged on top of Cowboy and replaces Mochiweb entirely. > > You don't need a "socket server", whatever that is, you can just use > websockets directly in Cowboy, or if you need a long polling fallback > there's Bullet available, which is just a library of one module + one js > file really. Either way they don't try to anticipate your needs, they just > transport data. > > > On 07/20/2012 12:42 AM, Gordon Guthrie wrote: >> >> Lo?c >> >> So I would then have 3 things running my application - mochiweb, >> cowboy and a socket server... >> >> I am a big fan of avoiding code at all costs :( >> >> Given that I have a simple understood use case, mebbies I don't need a >> fully featured socket server, mebbies I need to put a simple >> client-driven registration system... >> >> Gordon >> >> On 19 July 2012 23:24, Lo?c Hoguin wrote: >>> >>> Or you can replace Mochiweb by Cowboy+Mochicow >>> (https://github.com/benoitc/mochicow) and use Cowboy's routing to >>> redirect >>> to your websocket handler or your mochiweb code. >>> >>> >>> On 07/19/2012 11:02 PM, Gordon Guthrie wrote: >>>> >>>> >>>> Gleb >>>> >>>> The app is built on mochiweb - I inspect the request and route it to a >>>> handler. So I would need to bind the server part of the library to the >>>> that particular HTTP request. >>>> >>>> Bullet is designed to work in Cowboy, >>>> Socket.io has its own Erlang server >>>> SockJS is designed to be web-framework agnostic but the examples are >>>> all in Cowboy >>>> >>>> As I understand it - these socket solutions have a code the server >>>> side and client side which ping traffic backwards and forwards and >>>> keep a 'permanent' connection open - and the higher level libraries >>>> can then write to the socket as if there was a stateful synchronous >>>> connection. >>>> >>>> So to use one of these I would need to pick a particular client JS >>>> library and then write my own handler to use on the socket in mochiweb >>>> (or cut one out from one of the existing servers...) >>>> >>>> Has anyone done stuff on getting it into mochiweb? >>>> >>>> Gordon >>>> >>>> >>>> On 17 July 2012 16:01, Gleb Peregud wrote: >>>>> >>>>> >>>>> On Tue, Jul 17, 2012 at 4:57 PM, Gordon Guthrie >>>>> wrote: >>>>>> >>>>>> >>>>>> Folks >>>>>> >>>>>> I am building a softphone which needs to register with the server to >>>>>> say >>>>>> "I >>>>>> am available" for incoming calls. >>>>>> >>>>>> So it makes an HTTP POST request to the server and that notifies the >>>>>> softphone gen_srv that is is available. >>>>>> >>>>>> At that point I leave the TCP request hanging on a receive waiting for >>>>>> a >>>>>> notification that the socket has been torn down (and which point it >>>>>> unregisters the phone). >>>>>> >>>>>> My question is: how long can I leave the request up? Infinity? or do I >>>>>> need >>>>>> to have a timeout/reregister cycle setup? >>>>> >>>>> >>>>> >>>>> You have to cycle it. Some proxies timeout connections after 30 >>>>> seconds, some after 45 some after 60. It is very tricky to get it >>>>> right for all browsers and all platforms. Generally if you want a >>>>> robust solution use something like SockJS or Socket.IO or Bullet. >>>> >>>> >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> >>> >>> >>> -- >>> Lo?c Hoguin >>> Erlang Cowboy >>> Nine Nines >>> http://ninenines.eu >>> >>> > > > -- > Lo?c Hoguin > > Erlang Cowboy > Nine Nines > http://ninenines.eu > > From gleber.p@REDACTED Fri Jul 20 10:36:31 2012 From: gleber.p@REDACTED (Gleb Peregud) Date: Fri, 20 Jul 2012 10:36:31 +0200 Subject: [erlang-questions] Mochiwebsockets was Re: Long polling In-Reply-To: References: <50088928.6000506@ninenines.eu> <50088E60.7010309@ninenines.eu> Message-ID: On Fri, Jul 20, 2012 at 10:24 AM, Gordon Guthrie wrote: > There are a couple of complications - we already use one long-poll for > server-to-client updates - in that case the client posts changes and > the server works on them and then streams out updates on a separate > long-poll. Lets put that aside and focus on this use case. It can be important. Some browsers limit number of concurrent connections opened to a single host to two. This means that if you would have two long-polling connections opened at the same time, you won't be able to do a simple AJAX request on those browser (it will queue after long-polling connections and possibly timeout). Afair older IE had this problem, I don't recall in more details which one was it. > Single domain > It is a single registration event - ... > It's the server saying "are you still there?" > Needs to be reliable. ... > Yeah, lots of JQuery - ... > I will tolerate any size of code base if it is encapsulated nicely. ... So it seems that you don't need any complex solution. Most probably a simple long-polling XmlHttpRequest with a 30 seconds connection cycling (to work through proxies) should work well for you. If you don't care about proxies ~50 seconds will be better choice to preserve bandwidth and CPU on server. Essentially you could take parts of Bullet which does cycle connections, while jQuery handles most intricacies related to error handling. > What would the benefits of Cowboy over Mochiweb be? Since it seems that you don't need websockets after all, if Mochiweb works well there is no real need to switch. Cheers, Gleb From gordon@REDACTED Fri Jul 20 10:59:36 2012 From: gordon@REDACTED (Gordon Guthrie) Date: Fri, 20 Jul 2012 09:59:36 +0100 Subject: [erlang-questions] Mochiwebsockets was Re: Long polling In-Reply-To: References: <50088928.6000506@ninenines.eu> <50088E60.7010309@ninenines.eu> Message-ID: Gleb > It can be important. Some browsers limit number of concurrent > connections opened to a single host to two. This means that if you > would have two long-polling connections opened at the same time, you > won't be able to do a simple AJAX request on those browser (it will That's from a different page so it won't affect the softphone. > So it seems that you don't need any complex solution. Most probably a > simple long-polling XmlHttpRequest with a 30 seconds connection > cycling (to work through proxies) should work well for you. If you > don't care about proxies ~50 seconds will be better choice to preserve > bandwidth and CPU on server. That's kinda what I was thinking, open a connection, hang it on a receive, if the server needs to talk to the phone get the connection to return {msg: "are you there?"} client sends back {msg: "yup"} then job's a good un. I need to write the 'keep alive' case when the connection breaks down at the same time as the inbound call asks for it, but that's OK. So 30 secs is the right time for a long connection - at the moment we use 600 secs and get intermittent disconnection problems - usually when using a mobile internet connection - mebbies I should dial it down to 30 secs. Gordon On 20 July 2012 09:36, Gleb Peregud wrote: > On Fri, Jul 20, 2012 at 10:24 AM, Gordon Guthrie wrote: >> There are a couple of complications - we already use one long-poll for >> server-to-client updates - in that case the client posts changes and >> the server works on them and then streams out updates on a separate >> long-poll. Lets put that aside and focus on this use case. > > It can be important. Some browsers limit number of concurrent > connections opened to a single host to two. This means that if you > would have two long-polling connections opened at the same time, you > won't be able to do a simple AJAX request on those browser (it will > queue after long-polling connections and possibly timeout). Afair > older IE had this problem, I don't recall in more details which one > was it. > >> Single domain >> It is a single registration event - ... >> It's the server saying "are you still there?" >> Needs to be reliable. ... >> Yeah, lots of JQuery - ... >> I will tolerate any size of code base if it is encapsulated nicely. ... > > So it seems that you don't need any complex solution. Most probably a > simple long-polling XmlHttpRequest with a 30 seconds connection > cycling (to work through proxies) should work well for you. If you > don't care about proxies ~50 seconds will be better choice to preserve > bandwidth and CPU on server. > > Essentially you could take parts of Bullet which does cycle > connections, while jQuery handles most intricacies related to error > handling. > >> What would the benefits of Cowboy over Mochiweb be? > > Since it seems that you don't need websockets after all, if Mochiweb > works well there is no real need to switch. > > Cheers, > Gleb From gleber.p@REDACTED Fri Jul 20 11:54:30 2012 From: gleber.p@REDACTED (Gleb Peregud) Date: Fri, 20 Jul 2012 11:54:30 +0200 Subject: [erlang-questions] Mochiwebsockets was Re: Long polling In-Reply-To: References: <50088928.6000506@ninenines.eu> <50088E60.7010309@ninenines.eu> Message-ID: On Fri, Jul 20, 2012 at 10:59 AM, Gordon Guthrie wrote: > So 30 secs is the right time for a long connection - at the moment we > use 600 secs and get intermittent disconnection problems - usually > when using a mobile internet connection - mebbies I should dial it > down to 30 secs. Alternatively you could leave it at 600 secs and employ reconnect mechanism, which tries to reconnect few times (and adjusting timeout value for next attempt adaptively) before giving up and returning an error. It may be better solution for 90% of your users, but will still work for users with bad proxies/connection. But be aware that some proxies (or bad internet connection) may drop those connections silently and reconnect mechanism won't work since broken connection won't be detected. So it's a tradeoff between robust solution for everyone with high bandwidth consumption and low bandwidth consumption for 99% of users and broken service for 1% of users. In my setup where I'm using SockJS I am employing a heartbeat mechanism over streaming connections. I.e. SockJS sets up a persistent connection and server sends one byte of information to the user very 30 seconds (in case of polling connections it is of course much higher than 1 byte, but for streaming connections, which are prevalent it's just one byte) and client checks if he got heartbeat to detect silently dropped connection in at most 30 seconds. Oh, and mobile users are often connected via some sort of proxy (Blackberry, Opera Mini/Mobile, some ISPs use transparent caching as well). From cloudzen@REDACTED Fri Jul 20 14:55:03 2012 From: cloudzen@REDACTED (skyman) Date: Fri, 20 Jul 2012 20:55:03 +0800 (CST) Subject: [erlang-questions] prim_inet:async_recv/3 received {error, etimedout} message Message-ID: <6f860986.245df.138a474a144.Coremail.cloudzen@163.com> Hi all, I'd like to know when prim_inet:async_recv/3 received {error,etimedout} message, is the Socket still usable? can server still communicate with client using this Socket? Thanks in advance! -------------- next part -------------- An HTML attachment was scrubbed... URL: From hobson42@REDACTED Fri Jul 20 16:05:20 2012 From: hobson42@REDACTED (Ian) Date: Fri, 20 Jul 2012 15:05:20 +0100 Subject: [erlang-questions] How can I debug this? Message-ID: <500965A0.5030500@gmail.com> Hi All, I am attempting to work through the Chicago Boss quickstart tutorial at https://github.com/evanmiller/ChicagoBoss/wiki/Quickstart I am using Windows 7 - 64 bit, and the R15B release of Erlang. New to Erlang. First a minor bug - I think I installed the 32 bit version, removed it, and installed the 64 bit version. Then I discovered that the 64 bit version's bin directory is C:\Program Files\erl5.9.1\bin but the directory in the path was C:\Program Files (x86)\erl5.9.1\bin. That was why nothing worked. After that was corrected, I created the project and set up the file in Step 4, before starting the server with start-server.bat. The werl window flashed up an error message for a second or so - and vanished before I could read it! A few tries later I was able to catch it in a screen dump. The error message was: {"init terminating in do_boot",{undef,[{boss,start,[],[],{init,start_it,1,[{file ,"init.erl"},[line,1041}]},{init,start_em,1,[{file,"init.erl"},{line,1022}]}]}} (copied by hand) I know this means the server did not start up. But why, and what have I done wrong/missed out? I have not found a init.erl file. My project is called erlian and is in d:\erlian . The file D:\erlian\src\controller\erlian_greeting_controller.erl contains: -module(erlian_greeting_controller, [Req]). -compile(export_all). hello('GET', []) -> {output, "Erlian says hello!"}. (copy/pasted) Help much appreciated. Ian From hobson42@REDACTED Fri Jul 20 17:18:06 2012 From: hobson42@REDACTED (Ian) Date: Fri, 20 Jul 2012 16:18:06 +0100 Subject: [erlang-questions] How can I debug this? In-Reply-To: <500965A0.5030500@gmail.com> References: <500965A0.5030500@gmail.com> Message-ID: <500976AE.2040505@gmail.com> Further information: It is something to do with Windows and/or my windows setup. I tried the same tutorial in Ubuntu, in a VM and it worked as advertised. Regards Ian From essen@REDACTED Fri Jul 20 17:53:25 2012 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Fri, 20 Jul 2012 17:53:25 +0200 Subject: [erlang-questions] Mochiwebsockets was Re: Long polling In-Reply-To: References: <50088928.6000506@ninenines.eu> <50088E60.7010309@ninenines.eu> Message-ID: <50097EF5.4030503@ninenines.eu> On 07/20/2012 10:33 AM, Gordon Guthrie wrote: > Are you saying that Cowboy would take an incoming request, parse it, > decide - 'this is a long poll - pass it to bullet', 'this isn't pass > it to mochicow' and the mochicow would give me an identically formed > MochiRequest 'just like mochiweb'? Yep. I haven't used Mochicow myself but Benoit and others successfully replaced Mochiweb with it without changing the Mochiweb request handlers. -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From thomas.elsgaard@REDACTED Fri Jul 20 20:52:45 2012 From: thomas.elsgaard@REDACTED (Thomas Elsgaard) Date: Fri, 20 Jul 2012 20:52:45 +0200 Subject: [erlang-questions] Problem with Emacs/Distel Message-ID: Hi Everybody After i re-installed my mac (Lion) i installed latest Erlang (R15) Emacs (24.1.1) and Distel (4.03), but now it seems i have a hard time getting Emacs working with Distel... Basically i cannot get Emacs/Distel to connect to my erlang node (running on same laptop) I have started my erlang node via C-c C-k (emacs@REDACTED)2> (emacs@REDACTED)2> Connect's the node emacs@REDACTED via C-c C-d n Node: emacs@REDACTED Verifying the connection with: C-c C-d g And i am getting:EXIT: <0.5.0> [emacs-error "(error \"valhalla/4369 node name nor servname provided, or not known\")"] Any hints? What have i forgot? ///Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinoski@REDACTED Fri Jul 20 21:03:14 2012 From: vinoski@REDACTED (Steve Vinoski) Date: Fri, 20 Jul 2012 15:03:14 -0400 Subject: [erlang-questions] Problem with Emacs/Distel In-Reply-To: References: Message-ID: On Fri, Jul 20, 2012 at 2:52 PM, Thomas Elsgaard wrote: > Hi Everybody > > After i re-installed my mac (Lion) i installed latest Erlang (R15) Emacs > (24.1.1) and Distel (4.03), but now it seems i have a hard time getting > Emacs working with Distel... I can't answer your Distel problem but I can say that on Lion I've tried multiple emacs 24 builds retrieved from different places, including one I built myself, and found them all to be unstable, crashing all the time. I went back to 23.4.1 and have had no more crashes since. --steve From thomas.elsgaard@REDACTED Fri Jul 20 21:25:03 2012 From: thomas.elsgaard@REDACTED (Thomas Elsgaard) Date: Fri, 20 Jul 2012 21:25:03 +0200 Subject: [erlang-questions] Problem with Emacs/Distel [SOLVED] Message-ID: > > After i re-installed my mac (Lion) i installed latest Erlang (R15) Emacs > > (24.1.1) and Distel (4.03), but now it seems i have a hard time getting > > Emacs working with Distel... > > I can't answer your Distel problem but I can say that on Lion I've > tried multiple emacs 24 builds retrieved from different places, > including one I built myself, and found them all to be unstable, > crashing all the time. I went back to 23.4.1 and have had no more > crashes since. > > --steve > Hi All I downgraded Emacs to 23.4.1 and installed Distel from: massemanet-distel-ecc8581.tar instead of distel-4.03.tar and now i got it working again ;-) Thanks a lot for the help ///Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave@REDACTED Sat Jul 21 21:36:06 2012 From: dave@REDACTED (Dave Cottlehuber) Date: Sat, 21 Jul 2012 21:36:06 +0200 Subject: [erlang-questions] How can I debug this? In-Reply-To: <500976AE.2040505@gmail.com> References: <500965A0.5030500@gmail.com> <500976AE.2040505@gmail.com> Message-ID: On 20 July 2012 17:18, Ian wrote: > Further information: > > It is something to do with Windows and/or my windows setup. > > I tried the same tutorial in Ubuntu, in a VM and it worked as advertised. > > Regards > > > Ian > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions Hi Ian, I've tested & get the same issue here as well. On my side, the issue is that for some reason, boss isn't being compiled correctly. I was able to build it manually but can't see immediately why this doesn't work within the rebar structure. You can edit start-server.bat and replace werl for erl, this at least means you can see the error messages in the command prompt. Let's move this to the Chicago Boss list, can you re-post there with the output from your original windows-make script perhaps? A+ Dave From 7stud@REDACTED Sun Jul 22 10:52:58 2012 From: 7stud@REDACTED (7stud) Date: Sun, 22 Jul 2012 04:52:58 -0400 Subject: [erlang-questions] guard subexpressions resulting in a runtime error Message-ID: <20120722045258.5296@web002.roc2.bluetie.com> $ erl -v Erlang R15B01 (erts-5.9.1) [source] [smp:2:2] [async-threads:0] [hipe] [kernel-poll:false] According to "Erlang Programming" p.51, "Guard subexpressions resulting in a runtime error are treated as returning false" But with this function definition: -module(t1). -export([guard1/1]). guard1(N) when ((N/0 == 0) or N==1) -> hello. I get the following runtime error: $ erl Erlang R15B01 (erts-5.9.1) [source] [smp:2:2] [async-threads:0] [hipe] [kernel-poll:false] Eshell V5.9.1 (abort with ^G) 1> c(t1). {ok,t1} 2> t1:guard1(1). ** exception error: no function clause matching t1:guard1(1) (t1.erl, line 4) 3> I expected the guard subexpression (N/0 == 0) to generate a runtime error and therefore evaluate to false, and then (false or true) to evaluate to true, and therefore the atom 'hello' to be displayed. From jose.valim@REDACTED Sun Jul 22 11:03:06 2012 From: jose.valim@REDACTED (=?ISO-8859-1?Q?Jos=E9_Valim?=) Date: Sun, 22 Jul 2012 11:03:06 +0200 Subject: [erlang-questions] guard subexpressions resulting in a runtime error In-Reply-To: <20120722045258.5296@web002.roc2.bluetie.com> References: <20120722045258.5296@web002.roc2.bluetie.com> Message-ID: > > "Guard subexpressions resulting in a runtime error are treated as > returning false" > > But with this function definition: > > -module(t1). > -export([guard1/1]). > > guard1(N) when ((N/0 == 0) or N==1) -> > hello. > > In this example, the guard subexpression is the whole ((N/0 == 0) or N==1). You can use , (meaning and) or ; (meaning or) to provide subexpressions: guard1(N) when N/0 == 0; N==1 -> hello. The example above should give you the expected result. -------------- next part -------------- An HTML attachment was scrubbed... URL: From gleber.p@REDACTED Sun Jul 22 13:08:15 2012 From: gleber.p@REDACTED (Gleb Peregud) Date: Sun, 22 Jul 2012 13:08:15 +0200 Subject: [erlang-questions] Makefile detecting / downloading rebar Message-ID: Hello all Just an idea. Do you think this is a good approach to put the following into a makefile of an open source library? all: rebar ./rebar compile rebar: which rebar && ln -sf `which rebar` || true test -x ./rebar || (wget http://cloud.github.com/downloads/basho/rebar/rebar && chmod u+x rebar) This allows user who has rebar installed globally to still use it, but for users who prefer local rebar binaries to download it automatically. Can this script be improved? Cheers, Gleb From ok@REDACTED Sun Jul 22 13:10:26 2012 From: ok@REDACTED (ok@REDACTED) Date: Sun, 22 Jul 2012 23:10:26 +1200 Subject: [erlang-questions] guard subexpressions resulting in a runtime error In-Reply-To: <20120722045258.5296@web002.roc2.bluetie.com> References: <20120722045258.5296@web002.roc2.bluetie.com> Message-ID: <01c269a946a8bdd993bc62e3e6e6070f.squirrel@chasm.otago.ac.nz> > $ erl -v > Erlang R15B01 (erts-5.9.1) [source] [smp:2:2] [async-threads:0] [hipe] > [kernel-poll:false] > > According to "Erlang Programming" p.51, > > "Guard subexpressions resulting in a runtime error are treated as > returning false" Right. > > But with this function definition: > guard1(N) when ((N/0 == 0) or N==1) -> > hello. This is a perfect example of why it was EVIL to allow Boolean operators in guards instead of sticking with , and ;. If this were guard1(N) when N/0 == 0 ; N == 1 -> hello. it would do what you expected. 1> F = fun (N) when N/0 == 0 ; N == 1 -> hello end. #Fun 2> F(1). hello But 'or' is a plain old operator that evaluates both of its arguments, and when one of its operands erred, the *whole* guard expression failed. You had only one guard, so when it failed, there was nothing left to test. Using 'orelse' doesn't help either: N/0 == 0 orelse N == 1 is still ONE expression, so ONE guard test, and if any part of it errs, the whole of it fails. This is a straightforward inference from the documentation if you are familiar with the Good Old Ways and expect these new-fangled operators to do something stupid, but if you are a trusting soul who isn't aware that Boolean operators belong to the Holocene period of Erlang, it would be helpful if the Erlang manual were a bit more explicit about this. From lemenkov@REDACTED Sun Jul 22 13:17:55 2012 From: lemenkov@REDACTED (Peter Lemenkov) Date: Sun, 22 Jul 2012 15:17:55 +0400 Subject: [erlang-questions] Makefile detecting / downloading rebar In-Reply-To: References: Message-ID: Hello. 2012/7/22 Gleb Peregud : > Hello all > > Just an idea. Do you think this is a good approach to put the > following into a makefile of an open source library? > > all: rebar > ./rebar compile > > rebar: > which rebar && ln -sf `which rebar` || true > test -x ./rebar || (wget > http://cloud.github.com/downloads/basho/rebar/rebar && chmod u+x > rebar) > > This allows user who has rebar installed globally to still use it, but > for users who prefer local rebar binaries to download it > automatically. Can this script be improved? Your script requires wget which isn't installed by default on some Linux boxes (sad but true). So you need to check for wget first which adds even more complexity. Why not just to bundle your own rebar copy and check whether target box has system-wide installed version first? So my proposal is to simply include your own rebar copy into the project's source tree and add the following line to the Makefile (should be topmost line): REBAR ?= $(shell which rebar 2>/dev/null || which ./rebar) Also you need to replace all 'rebar' or './rebar' invocations with $(REBAR). -- With best regards, Peter Lemenkov. From dmkolesnikov@REDACTED Sun Jul 22 13:21:12 2012 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Sun, 22 Jul 2012 14:21:12 +0300 Subject: [erlang-questions] Makefile detecting / downloading rebar In-Reply-To: References: Message-ID: <499142D7-D002-433E-B3F4-20BCC627FBE1@gmail.com> Hello, Yes, I am using same approach with an exception I am ignoring a global rebar installation. just keep in mind that wget might not be present. It should be possible to choose between curl & wget. I believe distclean target should also remove downloaded rebar. - Dmitry On Jul 22, 2012, at 2:08 PM, Gleb Peregud wrote: > Hello all > > Just an idea. Do you think this is a good approach to put the > following into a makefile of an open source library? > > all: rebar > ./rebar compile > > rebar: > which rebar && ln -sf `which rebar` || true > test -x ./rebar || (wget > http://cloud.github.com/downloads/basho/rebar/rebar && chmod u+x > rebar) > > This allows user who has rebar installed globally to still use it, but > for users who prefer local rebar binaries to download it > automatically. Can this script be improved? > > Cheers, > Gleb > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From michael.santos@REDACTED Sun Jul 22 14:17:07 2012 From: michael.santos@REDACTED (Michael Santos) Date: Sun, 22 Jul 2012 08:17:07 -0400 Subject: [erlang-questions] Makefile detecting / downloading rebar In-Reply-To: References: Message-ID: <20120722121707.GA18388@ioctl> On Sun, Jul 22, 2012 at 01:08:15PM +0200, Gleb Peregud wrote: > Hello all > > Just an idea. Do you think this is a good approach to put the > following into a makefile of an open source library? > > all: rebar > ./rebar compile > > rebar: > which rebar && ln -sf `which rebar` || true > test -x ./rebar || (wget > http://cloud.github.com/downloads/basho/rebar/rebar && chmod u+x > rebar) > > This allows user who has rebar installed globally to still use it, but > for users who prefer local rebar binaries to download it > automatically. Can this script be improved? You could download rebar using erlang rather than wget: # shell is GNU makefile only syntax REBAR=$(shell which rebar || echo ./rebar) ./rebar: erl -noshell -s inets start -s ssl start \ -eval 'httpc:request(get, {"https://github.com/downloads/basho/rebar/rebar", []}, [], [{stream, "./rebar"}])' \ -s inets stop -s init stop chmod +x ./rebar From gleber.p@REDACTED Sun Jul 22 14:25:25 2012 From: gleber.p@REDACTED (Gleb Peregud) Date: Sun, 22 Jul 2012 14:25:25 +0200 Subject: [erlang-questions] Makefile detecting / downloading rebar In-Reply-To: <20120722121707.GA18388@ioctl> References: <20120722121707.GA18388@ioctl> Message-ID: On Sun, Jul 22, 2012 at 2:17 PM, Michael Santos wrote: > You could download rebar using erlang rather than wget: > > # shell is GNU makefile only syntax > REBAR=$(shell which rebar || echo ./rebar) > > ./rebar: > erl -noshell -s inets start -s ssl start \ > -eval 'httpc:request(get, {"https://github.com/downloads/basho/rebar/rebar", []}, [], [{stream, "./rebar"}])' \ > -s inets stop -s init stop > chmod +x ./rebar I have written an reasonably short curl-or-wget downloading script but approach using Erlang seems the nicest out of all of them. Erlang has to be present, so download will work. And it is nicer than my initial approach, since it doesn't do links. I'll borrow this approach :) My curl-or-wget script looks like this: REBAR_URL=http://cloud.github.com/downloads/basho/rebar/rebar DOWN=$(shell (which wget && echo -O rebar) || (which curl && echo -o rebar)) # detect curl or wget rebar: @which rebar && ln -sf `which rebar` || true >/dev/null 2>&1 @test -x ./rebar || ($(DOWN) $(REBAR_URL) >/dev/null 2>&1 && chmod u+x rebar) From gleber.p@REDACTED Sun Jul 22 14:32:14 2012 From: gleber.p@REDACTED (Gleb Peregud) Date: Sun, 22 Jul 2012 14:32:14 +0200 Subject: [erlang-questions] Makefile detecting / downloading rebar In-Reply-To: References: <20120722121707.GA18388@ioctl> Message-ID: IMO "best" solution is: REBAR=$(shell which rebar || echo ./rebar) REBAR_URL=http://cloud.github.com/downloads/basho/rebar/rebar all: $(REBAR) $(REBAR) compile ./rebar: erl -noshell -s inets -s ssl \ -eval 'httpc:request(get, {"$(REBAR_URL)", []}, [], [{stream, "./rebar"}])' \ -s init stop chmod +x ./rebar distclean: rm -f ./rebar Thanks Michael! From ttom.kelly@REDACTED Sun Jul 22 17:16:06 2012 From: ttom.kelly@REDACTED (tom kelly) Date: Sun, 22 Jul 2012 17:16:06 +0200 Subject: [erlang-questions] How can I debug this? In-Reply-To: <500B0E37.5000003@gmail.com> References: <500965A0.5030500@gmail.com> <500B0E37.5000003@gmail.com> Message-ID: Hi Ian, I'm CC'ing the list again. It was probably a typo not including them on your mail? and I think it's important that more people see you points. If we want Erlang usage to grow it's important that we address issues like this as much as possible. I'm happy to hear my input helped, and I find your comments as a newbie interesting. I didn't see that thread your referenced last week (going to search for it as soon as I hit send) but I'm sure we can all associate with the frustrations of struggling with unfamiliar tools or environments. I can only speak for myself but will try to smooth things for newbies, even if it's only by replying to similar questions here more often. //TTom On Sat, Jul 21, 2012 at 10:16 PM, Ian wrote: > Hi Tom, > > Your right - it cannot find boss.beam. There is no such file on my > system. > > After some investigation on the Ubuntu system, I discover that it should > be in D:\ChicagoBoss\bin with a lot of other .beam files. > > In fact the ChicargoBoss\bin directory has no .beam files at all. So I > tried to compile them. That crashed. > > After some more investigation it appears that the file > D:\ChicagoBoss\deps\proper\include\compile_flags.hrl did not exist. > > What should it contain? Back to the Unbuntu system and found that it is > empty! After "touching" it, the compile worked and I now have a running > system under windows. Thank you for your help - it was just the information > I needed. > > I wrote last week in another thread that I agreed with Dimitii in his talk > in Stockholm that it was too difficult to do the easy stuff in Erlang. It > has taken two days and help from kind people to get me over these initial > teething problems. > > But note that I was doing real basic newbie stuff. I downloaded the > STANDARD install of erlang for my operating system, and the STANDARD > install of Chicago Boss, installed them into the STANDARD locations, and > attempted to set up a web server to serve "Hello World" following > instructions included with those downloads. > > But look at what went wrong. > The install of Erlang did not set it up my path. > Chicago Boss assumed I had a suitable Make on my machine. > Chicago Boss failed to create a file that was necessary. > The error message was incomprehensible to a newbie. > > Trivial problems for the experienced Erlang buff, I'm sure, but for a > newbie (or a manager who is authorising the investigation) they are > stoppers. These problems must be slowing the adoption of Erlang. That is a > pity, because Erlang is very strongly positioned for major growth because > it is "the way" to take advantage of multiple cores. Actors, message > passing, no shared state, no slouch, reliable and easy to scale to multiple > cores/cpus - what's not to like? > > > Thanks again Tom, your information was all I needed. > > Ian > > > On 21/07/2012 14:51, tom kelly wrote: > > Hi, > I've never worked with Chicago boss but it looks like your start script > doesn't have the path to boss.beam. > > Your error message: > {"init terminating in > do_boot",{undef,[{boss,start,[],[],{init,start_it,1,[{file,"init.erl"},[line,1041}]},{init,start_em,1,[{file,"init.erl"},{line,1022}]}]}} > > > says that during startup it tried to call "boss:start()." but that this is > undefined, probably because it didn't have boss.beam in its path. > You should locate boss.beam on your system and check that its location is > added. > HTH, > //TTom. > > > On Fri, Jul 20, 2012 at 3:05 PM, Ian wrote: > >> Hi All, >> >> I am attempting to work through the Chicago Boss quickstart tutorial at >> https://github.com/evanmiller/ChicagoBoss/wiki/Quickstart >> >> I am using Windows 7 - 64 bit, and the R15B release of Erlang. New to >> Erlang. >> >> First a minor bug - I think I installed the 32 bit version, removed it, >> and installed the 64 bit version. Then I discovered that the 64 bit >> version's bin directory is C:\Program Files\erl5.9.1\bin but the directory >> in the path was C:\Program Files (x86)\erl5.9.1\bin. That was why nothing >> worked. >> >> After that was corrected, I created the project and set up the file in >> Step 4, before starting the server with start-server.bat. The werl window >> flashed up an error message for a second or so - and vanished before I >> could read it! >> >> A few tries later I was able to catch it in a screen dump. The error >> message was: >> >> {"init terminating in >> do_boot",{undef,[{boss,start,[],[],{init,start_it,1,[{file >> ,"init.erl"},[line,1041}]},{init,start_em,1,[{file,"init.erl"},{line,1022}]}]}} >> >> >> (copied by hand) >> >> I know this means the server did not start up. But why, and what have I >> done wrong/missed out? I have not found a init.erl file. >> >> My project is called erlian and is in d:\erlian . The file >> D:\erlian\src\controller\erlian_greeting_controller.erl >> contains: >> >> -module(erlian_greeting_controller, [Req]). >> -compile(export_all). >> >> hello('GET', []) -> >> {output, "Erlian says hello!"}. >> >> (copy/pasted) >> >> Help much appreciated. >> >> Ian >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gentibogu@REDACTED Mon Jul 23 00:53:30 2012 From: gentibogu@REDACTED (Genti Bogu) Date: Sun, 22 Jul 2012 23:53:30 +0100 (BST) Subject: [erlang-questions] Mnesia Reads and Writes In-Reply-To: <1342997087.75885.YahooMailNeo@web171305.mail.ir2.yahoo.com> References: <1342997087.75885.YahooMailNeo@web171305.mail.ir2.yahoo.com> Message-ID: <1342997610.99278.YahooMailNeo@web171305.mail.ir2.yahoo.com> Hello everybody, This is my first Erlang project and I was evaluating Mnesia's performance. I wrote a simple erlang module for doing read and write tests to a disc_copies mnesia database. So the data are stored in the disk but the database is loaded in memory. ?A write test consists of a number of records inserted per second in the Database, and a read test consists on a number of records that are read per second from the db.?Each read or write operation is done inside a transaction. The read and write tests are performed separately. Based on Mnesia's performance I define the appropriate load (number of reads/sec or number of writes/sec) that mnesia can handle. Since transactional reads require only read locks (other reads can be done without requiring a lock), and transactional writes require write locks (exclusive access), I was expecting for mnesia to handle more load for read tests than for write tests. However, the supported load is the same in both the cases. Another observation of mine is the CPU usage, which is around 70% in the read tests (7 times higher than the write tests). Does anybody know why it can happen? Since, I do the read tests separately from the write ones, and I have only one db node, I tried doing mnesia reads without using transactions at all. I got a read time that was more than 20 times faster than the time obtained when doing transactional reads. Why is it so huge the difference in performance in such a case? We have just read locks which allow simultaneous read requests to be performed in parallel without any exclusive lock. How does the locking mechanism work for transactional reads and writes in Mnesia, and how are these operation handled? I would be glad if somebody could help me understand some basic stuff about Mnesia IO operations. Thanks, Genti -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto@REDACTED Mon Jul 23 04:15:42 2012 From: roberto@REDACTED (Roberto Ostinelli) Date: Sun, 22 Jul 2012 19:15:42 -0700 Subject: [erlang-questions] meck Message-ID: dear all, i'm trying to use the meck framework (https://github.com/eproxus/meck) and i can't seem to find a way to test that a function gets called with the appropriate values. for instance: %%%%%%%%%%%%%%%%%%%% -module(example). -compile(export_all). get_config() -> mocked:myfun("test.config"). -ifdef(TEST). -include_lib("eunit/include/eunit.hrl"). get_config_test_() -> meck:new(mocked), meck:expect(mocked, myfun, fun(Filename) -> ok end), ?_assertEqual(ok, get_config()), ?_assert(meck:validate(mocked)). -endif. %%%%%%%%%%%%%%%%%%%% i'd like to test that mocked:myfun/1 gets called with "test.config" as parameter. is this possible? r. -------------- next part -------------- An HTML attachment was scrubbed... URL: From YurinVV@REDACTED Mon Jul 23 06:37:05 2012 From: YurinVV@REDACTED (Slava Yurin) Date: Mon, 23 Jul 2012 11:37:05 +0700 Subject: [erlang-questions] meck In-Reply-To: References: Message-ID: <2103441343018225@web28g.yandex.ru> An HTML attachment was scrubbed... URL: From vances@REDACTED Mon Jul 23 07:40:14 2012 From: vances@REDACTED (Vance Shipley) Date: Mon, 23 Jul 2012 11:10:14 +0530 Subject: [erlang-questions] Loading Object Code Contained in a Binary Message-ID: <20120723054000.GA48523@aluminum.motivity.ca> I am dynamically contructing abstract forms representing modules and loading them into the code server using code:load_binary/3. I find that if I load a new version of the module multiple times code:load_binary/3 does not complain about previous versions not being purged. If I use erlang:load_module/2 it does complain. The documentation warns that this BIF is intended for the code servers and should not be used elsewhere. What kind of trouble am I likely to get into doing this? Should I use erlang:load_module/2 instead of code:load_binary/3? -- -Vance From samuelrivas@REDACTED Mon Jul 23 09:09:21 2012 From: samuelrivas@REDACTED (Samuel) Date: Mon, 23 Jul 2012 09:09:21 +0200 Subject: [erlang-questions] Loading Object Code Contained in a Binary In-Reply-To: <20120723054000.GA48523@aluminum.motivity.ca> References: <20120723054000.GA48523@aluminum.motivity.ca> Message-ID: > I find that if I load a new version of the module multiple times > code:load_binary/3 does not complain about previous versions not > being purged. If I use erlang:load_module/2 it does complain. > The documentation warns that this BIF is intended for the code > servers and should not be used elsewhere. > > What kind of trouble am I likely to get into doing this? The usual problem of getting processes killed if a new version of the code is loaded when they are running old code. Suppose I write a module lopp (I mistyped the name and was too lazy to correct it :) ) -module(lopp). -compile(export_all). start_link() -> spawn_link(fun() -> loop() end). loop() -> timer:sleep(1000), io:format("."), loop(). foo() -> bar. And then start the shell and read the beam into Lopp: 9> lopp:start_link(). <0.101.0> .......40> code:load_binary(lopp, "no_file", Lopp). {module,lopp} ..41> code:load_binary(lopp, "no_file", Lopp). ** exception exit: killed You can be a bit more careful and purge the code yourself before attempting to load the new version: 42> lopp:start_link(). <0.106.0> ......43> code:soft_purge(lopp). true .....44> code:delete(lopp). true .....45> code:load_binary(lopp, "no_file", Lopp). {module,lopp} ...46> code:soft_purge(lopp). false %% Here we know there is old code that cannot be replaced .....47> code:delete(lopp). =ERROR REPORT==== 23-Jul-2012::08:53:27 === Module lopp must be purged before loading false If you use code:soft_purge you should never kill processes that are running old code. But if you want to, you can then use code:purge. A small caveat is that the code can be reloaded between your call to code:delete and code:load_binary, which can be a problem if you are not running in embedded mode: 65> lopp:start_link(). <0.145.0> .....66> code:soft_purge(lopp). true ......67> code:delete(lopp). true ...68> lopp:foo(). bar ..........69> code:load_binary(lopp, "no_file", Lopp). ** exception exit: killed I don't know a way of doing that atomically, I would like a version of code:load_binary that refuses to load when there is code already running in the system. Best -- Samuel From desired.mta@REDACTED Mon Jul 23 10:07:59 2012 From: desired.mta@REDACTED (=?UTF-8?Q?Motiejus_Jak=C5=A1tys?=) Date: Mon, 23 Jul 2012 10:07:59 +0200 Subject: [erlang-questions] meck In-Reply-To: References: Message-ID: On Mon, Jul 23, 2012 at 4:15 AM, Roberto Ostinelli wrote: > dear all, > > for instance: > > %%%%%%%%%%%%%%%%%%%% > > -module(example). > -compile(export_all). > > get_config() -> > mocked:myfun("test.config"). > > -ifdef(TEST). > -include_lib("eunit/include/eunit.hrl"). > > get_config_test_() -> > meck:new(mocked), > meck:expect(mocked, myfun, fun(Filename) -> ok end), > ?_assertEqual(ok, get_config()), > ?_assert(meck:validate(mocked)). > -endif. Hi, you are mixing eunit generators with simple test cases. Did you mean this? get_config_test() -> meck:new(mocked), meck:expect(mocked, myfun, fun(Filename) -> ok end), ?assertEqual(ok, get_config()), ?assert(meck:called(mocked, myfun, ["test.config"])), meck:unload(mocked). Normally, you should split meck:new and meck:unload to separate setup/cleanup functions. Because if a test fails and the module is not cleaned, it might unexpectedly do ill for your other test cases. -- Motiejus Jak?tys From msegalis@REDACTED Mon Jul 23 11:58:54 2012 From: msegalis@REDACTED (Morgan Segalis) Date: Mon, 23 Jul 2012 11:58:54 +0200 Subject: [erlang-questions] SSL Server Bad record mac error Message-ID: Hi everyone, I'm currently trying to find out, where could come from this error : =ERROR REPORT==== 23-Jul-2012::11:03:56 === SSL: certify: ssl_record.erl:654:Fatal error: bad record mac The SSL function where it is coming from is decipher: ---------------------------------- ssl_record.erl --------------------------------------------- decipher(TLS=#ssl_tls{type=Type, version=Version, fragment=Fragment}, CS0) -> SP = CS0#connection_state.security_parameters, BCA = SP#security_parameters.bulk_cipher_algorithm, HashSz = SP#security_parameters.hash_size, CipherS0 = CS0#connection_state.cipher_state, case ssl_cipher:decipher(BCA, HashSz, CipherS0, Fragment, Version) of {T, Mac, CipherS1} -> CS1 = CS0#connection_state{cipher_state = CipherS1}, TLength = size(T), {MacHash, CS2} = hash_and_bump_seqno(CS1, Type, Version, TLength, T), case is_correct_mac(Mac, MacHash) of true -> {TLS#ssl_tls{fragment = T}, CS2}; false -> ?ALERT_REC(?FATAL, ?BAD_RECORD_MAC) %%<-------- HERE end; #alert{} = Alert -> Alert end. ---------------------------------------------------------------------------------------------------- I'm not really sure if this error is happening at handshake, or read. The thing is, the error is only coming only from users connecting from a BlackBerry device (Can't know which OS version, I suppose it comes from rather old devices). If there is nothing I can't do server-side, does anyone could point out how to handle the error, in order to close cleanly the connection ? Regards, Morgan. -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg@REDACTED Mon Jul 23 21:50:54 2012 From: greg@REDACTED (Greg Martin) Date: Mon, 23 Jul 2012 12:50:54 -0700 Subject: [erlang-questions] Recursive Heap Allocation Message-ID: I'm just learning erlang using Erlang R15B01 (erts-5.9.1) on Windows. I'm wondering why ipv4_addrs:test(ipv4_addrs:first()) leads to the crash below and how it can be called so that it doesn't. Thanks. {1,2,22,127} {1,2,22,128} {1,2,22,129} {1,2,22,130} {1,2,22,131} {1,2,22,132} {1,2,22,133} {1,2,22,134} Crash dump was written to: erl_crash.dump eheap_alloc: Cannot allocate 373662860 bytes of memory (of type "heap"). Abnormal termination -module(ipv4_addrs). -export([first/0, next/1, test/1]). first()-> { 1,1,0,0 }. next( { 255, 255, 255, 255 } ) -> done; next( { 9, 255, 255, 255 } ) -> { 11, 0, 0, 0 }; next( { 172, 15, 255, 255 } ) -> { 172, 32, 0, 0 }; next( { 192, 167, 255, 255 } ) -> { 192, 169, 0, 0 }; next( { First, 255, 255, 255 } ) -> { First + 1, 0, 0, 0 }; next( { First, Second, 255, 255 } ) -> { First, Second + 1, 0, 0 }; next( { First, Second, Third, 255 } ) -> { First, Second, Third + 1, 0 }; next( { First, Second, Third, Fourth } ) -> { First, Second, Third, Fourth + 1 }. test(done) -> done; test(Addr) -> io:format("~p~n", [Addr]), Next = next(Addr), test(Next). From bob@REDACTED Mon Jul 23 22:04:26 2012 From: bob@REDACTED (Bob Ippolito) Date: Mon, 23 Jul 2012 13:04:26 -0700 Subject: [erlang-questions] Recursive Heap Allocation In-Reply-To: References: Message-ID: You're probably generating output too fast, there's no flow control in io:format/2. You can generate things to print much faster than they get printed, which results in the io server accumulating so many messages in its mailbox that you eventually run out of memory and/or the machine grinds to a halt due to swapping. On Mon, Jul 23, 2012 at 12:50 PM, Greg Martin wrote: > I'm just learning erlang using Erlang R15B01 (erts-5.9.1) on Windows. I'm > wondering why ipv4_addrs:test(ipv4_addrs:first()) leads to the crash below > and how it can be called so that it doesn't. Thanks. > > {1,2,22,127} > {1,2,22,128} > {1,2,22,129} > {1,2,22,130} > {1,2,22,131} > {1,2,22,132} > {1,2,22,133} > {1,2,22,134} > > Crash dump was written to: erl_crash.dump > eheap_alloc: Cannot allocate 373662860 bytes of memory (of type "heap"). > > > Abnormal termination > > > -module(ipv4_addrs). > > -export([first/0, next/1, test/1]). > > first()-> > { 1,1,0,0 }. > > next( { 255, 255, 255, 255 } ) -> done; > next( { 9, 255, 255, 255 } ) -> { 11, 0, 0, 0 }; > next( { 172, 15, 255, 255 } ) -> { 172, 32, 0, 0 }; > next( { 192, 167, 255, 255 } ) -> { 192, 169, 0, 0 }; > next( { First, 255, 255, 255 } ) -> { First + 1, 0, 0, 0 }; > next( { First, Second, 255, 255 } ) -> { First, Second + 1, 0, 0 }; > next( { First, Second, Third, 255 } ) -> { First, Second, Third + 1, 0 }; > next( { First, Second, Third, Fourth } ) -> { First, Second, Third, Fourth > + > 1 }. > > test(done) -> done; > test(Addr) -> > io:format("~p~n", [Addr]), > Next = next(Addr), > test(Next). > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mharrell-keyword-erlang.a034fe@REDACTED Mon Jul 23 22:26:37 2012 From: mharrell-keyword-erlang.a034fe@REDACTED (Matthew Harrell) Date: Mon, 23 Jul 2012 16:26:37 -0400 Subject: [erlang-questions] SSL key / password problems Message-ID: <20120723202637.GA25591@bittwiddlers.com> I searched but all the information I found was very dated and didn't seem to help any. Most of these examples are just a few modifications on what is found here http://www.erlang.org/doc/apps/ssl/using_ssl.html First, only TLS v1 is supported at the moment in R15B01, right? Not 1.1 or 1.2? I have a key pair with protected with the password "password" called client.crt and client.key. When I try to start up a client connection using that pair I get ssl:start(). {ok, Socket} = ssl:connect("localhost", 9950, [{certfile, "client.crt"}, {keyfile, "client.key"}], infinity). ** exception error: no match of right hand side value {error,ekeyfile} which is fine because it can't open the private key. But when I try {ok, Socket} = ssl:connect("localhost", 9950, [{certfile, "example/client.crt"}, {keyfile, "example/client.key"}, {password, "password"}], infinity). ** exception error: no match of right hand side value {error,ekeyfile} I get the same error. What am I doing wrong? Isn't that the point of the password option? When I try things with openssl like the following it works fine openssl s_client -cert example/client.crt -key example/client.key \ -CAfile example/ca.pem -pass pass:password -state -connect 127.0.0.1:9950 Also if I try to load the CA files I get messages about them not being decoded properly {ok, Socket} = ssl:connect("localhost", 9950, [{cacertfile, "/etc/ssl/certs/ca-certificates.crt"}, {certfile, "example/client.crt"}, {keyfile, "example/client.key"}, {password, "password"}], infinity). =INFO REPORT==== 23-Jul-2012::15:36:56 === SSL WARNING: Ignoring a CA cert as it could not be correctly decoded. I get the same message on my own ca.crt file with it's one key but thought I would try the system one to see whether it differed Finally, on the server side if I do the following using server keys (without passwords) and the openssl client line above ssl:start(). {ok, ListenSocket} = ssl:listen ( 9950, [{active, true}, {reuseaddr, true}, {keyfile, "example/server.key"}, {certfile, "example/server.crt"}, {backlog, 30}] ). {ok, Socket} = ssl:transport_accept ( ListenSocket ). ssl:ssl_accept ( Socket ). ssl:setopts ( Socket, [{active, true}] ). then an SSL connection seems to start up fine according to the messages on the openssl side. If I change this to ssl:start(). {ok, ListenSocket} = ssl:listen ( 9950, [{active, true}, {reuseaddr, true}, {verify, verify_peer}, {depth, 2}, {cacertfile, "example/ca.pem"}, {keyfile, "example/server.key"}, {certfile, "example/server.crt"}, {backlog, 30}] ). {ok, Socket} = ssl:transport_accept ( ListenSocket ). ssl:ssl_accept ( Socket ). ssl:setopts ( Socket, [{active, true}] ). where example/ca.pem is the one CA certificate I get =INFO REPORT==== 23-Jul-2012::16:12:59 === SSL WARNING: Ignoring a CA cert as it could not be correctly decoded. ** exception exit: {{{badmatch, {error, {asn1, {'Type not compatible with table constraint', {{component,'Type'}, {value,{5,<<>>}}, {unique_name_and_value,id,{1,3,14,3,2,29}}}}}}}, [{public_key,pkix_decode_cert,2, [{file,"public_key.erl"},{line,215}]}, {ssl_certificate,trusted_cert_and_path,3, [{file,"ssl_certificate.erl"},{line,58}]}, {ssl_handshake,certify,7, [{file,"ssl_handshake.erl"},{line,216}]}, {ssl_connection,certify,2, [{file,"ssl_connection.erl"},{line,514}]}, {ssl_connection,next_state,4, [{file,"ssl_connection.erl"},{line,1929}]}, {gen_fsm,handle_msg,7,[{file,"gen_fsm.erl"},{line,494}]}, {proc_lib,init_p_do_apply,3, [{file,"proc_lib.erl"},{line,227}]}]}, {gen_fsm,sync_send_all_state_event, [<0.50.0>,start,infinity]}} in function gen_fsm:sync_send_all_state_event/3 (gen_fsm.erl, line 240) in call from ssl_connection:sync_send_all_state_event/3 (ssl_connection.erl, line 1195) in call from ssl_connection:handshake/2 (ssl_connection.erl, line 167) What does that mean? Thanks for any help From ronny.meeus@REDACTED Mon Jul 23 22:59:35 2012 From: ronny.meeus@REDACTED (Ronny Meeus) Date: Mon, 23 Jul 2012 22:59:35 +0200 Subject: [erlang-questions] Recursive Heap Allocation In-Reply-To: References: Message-ID: Hello I think there is an issue in the logic of your code: next( { First, Second, Third, 255 } ) -> { First, Second, Third + 1, 0 }; next( { First, Second, Third, Fourth } ) -> { First, Second, Third, Fourth + 1 }. If nothing matches, the last clause will do and the last number will be incremented until 255 is reached. If I understand you well, I would think that the looping continues with the last but 1 clause but here is the problem: you pass 0 instead of 255 so there is no match and you end up in the last clause again. Result is an endless loop. I think the code should be corrected like this: next( { First, Second, Third, 255 } ) -> { First, Second, Third + 1, 255 }; next( { First, Second, Third, Fourth } ) -> { First, Second, Third, Fourth + 1 }. --- Best regards, Ronny On Mon, Jul 23, 2012 at 10:04 PM, Bob Ippolito wrote: > You're probably generating output too fast, there's no flow control in > io:format/2. You can generate things to print much faster than they get > printed, which results in the io server accumulating so many messages in its > mailbox that you eventually run out of memory and/or the machine grinds to a > halt due to swapping. > > > On Mon, Jul 23, 2012 at 12:50 PM, Greg Martin wrote: >> >> I'm just learning erlang using Erlang R15B01 (erts-5.9.1) on Windows. I'm >> wondering why ipv4_addrs:test(ipv4_addrs:first()) leads to the crash below >> and how it can be called so that it doesn't. Thanks. >> >> {1,2,22,127} >> {1,2,22,128} >> {1,2,22,129} >> {1,2,22,130} >> {1,2,22,131} >> {1,2,22,132} >> {1,2,22,133} >> {1,2,22,134} >> >> Crash dump was written to: erl_crash.dump >> eheap_alloc: Cannot allocate 373662860 bytes of memory (of type "heap"). >> >> >> Abnormal termination >> >> >> -module(ipv4_addrs). >> >> -export([first/0, next/1, test/1]). >> >> first()-> >> { 1,1,0,0 }. >> >> next( { 255, 255, 255, 255 } ) -> done; >> next( { 9, 255, 255, 255 } ) -> { 11, 0, 0, 0 }; >> next( { 172, 15, 255, 255 } ) -> { 172, 32, 0, 0 }; >> next( { 192, 167, 255, 255 } ) -> { 192, 169, 0, 0 }; >> next( { First, 255, 255, 255 } ) -> { First + 1, 0, 0, 0 }; >> next( { First, Second, 255, 255 } ) -> { First, Second + 1, 0, 0 }; >> next( { First, Second, Third, 255 } ) -> { First, Second, Third + 1, 0 }; >> next( { First, Second, Third, Fourth } ) -> { First, Second, Third, Fourth >> + >> 1 }. >> >> test(done) -> done; >> test(Addr) -> >> io:format("~p~n", [Addr]), >> Next = next(Addr), >> test(Next). >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From greg@REDACTED Mon Jul 23 23:50:31 2012 From: greg@REDACTED (Greg Martin) Date: Mon, 23 Jul 2012 14:50:31 -0700 Subject: [erlang-questions] Recursive Heap Allocation In-Reply-To: References: Message-ID: <945E4A18C0214F36BB397894926D8164@my.domain> I think you're misunderstanding. It's like a 256 based number scheme. 0.255 is followed by 1.0 not 1.255. -----Original Message----- From: Ronny Meeus [mailto:ronny.meeus@REDACTED] Sent: Monday, July 23, 2012 2:00 PM To: Bob Ippolito Cc: Greg Martin; erlang-questions@REDACTED Subject: Re: [erlang-questions] Recursive Heap Allocation Hello I think there is an issue in the logic of your code: next( { First, Second, Third, 255 } ) -> { First, Second, Third + 1, 0 }; next( { First, Second, Third, Fourth } ) -> { First, Second, Third, Fourth + 1 }. If nothing matches, the last clause will do and the last number will be incremented until 255 is reached. If I understand you well, I would think that the looping continues with the last but 1 clause but here is the problem: you pass 0 instead of 255 so there is no match and you end up in the last clause again. Result is an endless loop. I think the code should be corrected like this: next( { First, Second, Third, 255 } ) -> { First, Second, Third + 1, 255 }; next( { First, Second, Third, Fourth } ) -> { First, Second, Third, Fourth + 1 }. --- Best regards, Ronny On Mon, Jul 23, 2012 at 10:04 PM, Bob Ippolito wrote: > You're probably generating output too fast, there's no flow control in > io:format/2. You can generate things to print much faster than they > get printed, which results in the io server accumulating so many > messages in its mailbox that you eventually run out of memory and/or > the machine grinds to a halt due to swapping. > > > On Mon, Jul 23, 2012 at 12:50 PM, Greg Martin wrote: >> >> I'm just learning erlang using Erlang R15B01 (erts-5.9.1) on Windows. >> I'm wondering why ipv4_addrs:test(ipv4_addrs:first()) leads to the >> crash below and how it can be called so that it doesn't. Thanks. >> >> {1,2,22,127} >> {1,2,22,128} >> {1,2,22,129} >> {1,2,22,130} >> {1,2,22,131} >> {1,2,22,132} >> {1,2,22,133} >> {1,2,22,134} >> >> Crash dump was written to: erl_crash.dump >> eheap_alloc: Cannot allocate 373662860 bytes of memory (of type "heap"). >> >> >> Abnormal termination >> >> >> -module(ipv4_addrs). >> >> -export([first/0, next/1, test/1]). >> >> first()-> >> { 1,1,0,0 }. >> >> next( { 255, 255, 255, 255 } ) -> done; next( { 9, 255, 255, 255 } ) >> -> { 11, 0, 0, 0 }; next( { 172, 15, 255, 255 } ) -> { 172, 32, 0, 0 >> }; next( { 192, 167, 255, 255 } ) -> { 192, 169, 0, 0 }; next( { >> First, 255, 255, 255 } ) -> { First + 1, 0, 0, 0 }; next( { First, >> Second, 255, 255 } ) -> { First, Second + 1, 0, 0 }; next( { First, >> Second, Third, 255 } ) -> { First, Second, Third + 1, 0 }; next( { >> First, Second, Third, Fourth } ) -> { First, Second, Third, Fourth >> + >> 1 }. >> >> test(done) -> done; >> test(Addr) -> >> io:format("~p~n", [Addr]), >> Next = next(Addr), >> test(Next). >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From cgsmcmlxxv@REDACTED Tue Jul 24 06:25:29 2012 From: cgsmcmlxxv@REDACTED (CGS) Date: Tue, 24 Jul 2012 06:25:29 +0200 Subject: [erlang-questions] Recursive Heap Allocation In-Reply-To: <945E4A18C0214F36BB397894926D8164@my.domain> References: <945E4A18C0214F36BB397894926D8164@my.domain> Message-ID: Hi, I managed to reproduce the error and the problem is coming from io:format/2 (I suppose writing is too slow for the loop itself, so, the io buffer gets overflowed). Try this replacing ipv4_addrs:next/1 with this: test(done) -> done; test(Addr) -> test(next(Addrs)). CGS On Mon, Jul 23, 2012 at 11:50 PM, Greg Martin wrote: > I think you're misunderstanding. It's like a 256 based number scheme. > 0.255 > is followed by 1.0 not 1.255. > > -----Original Message----- > From: Ronny Meeus [mailto:ronny.meeus@REDACTED] > Sent: Monday, July 23, 2012 2:00 PM > To: Bob Ippolito > Cc: Greg Martin; erlang-questions@REDACTED > Subject: Re: [erlang-questions] Recursive Heap Allocation > > Hello > > I think there is an issue in the logic of your code: > > next( { First, Second, Third, 255 } ) -> { First, Second, Third + 1, 0 }; > next( { First, Second, Third, Fourth } ) -> { First, Second, Third, Fourth > + > 1 }. > > If nothing matches, the last clause will do and the last number will be > incremented until 255 is reached. > If I understand you well, I would think that the looping continues with the > last but 1 clause but here is the problem: you pass 0 instead of 255 so > there is no match and you end up in the last clause again. > Result is an endless loop. > > I think the code should be corrected like this: > next( { First, Second, Third, 255 } ) -> { First, Second, Third + 1, 255 }; > next( { First, Second, Third, Fourth } ) -> { First, Second, Third, Fourth > + > 1 }. > > --- > Best regards, > Ronny > > On Mon, Jul 23, 2012 at 10:04 PM, Bob Ippolito wrote: > > You're probably generating output too fast, there's no flow control in > > io:format/2. You can generate things to print much faster than they > > get printed, which results in the io server accumulating so many > > messages in its mailbox that you eventually run out of memory and/or > > the machine grinds to a halt due to swapping. > > > > > > On Mon, Jul 23, 2012 at 12:50 PM, Greg Martin > wrote: > >> > >> I'm just learning erlang using Erlang R15B01 (erts-5.9.1) on Windows. > >> I'm wondering why ipv4_addrs:test(ipv4_addrs:first()) leads to the > >> crash below and how it can be called so that it doesn't. Thanks. > >> > >> {1,2,22,127} > >> {1,2,22,128} > >> {1,2,22,129} > >> {1,2,22,130} > >> {1,2,22,131} > >> {1,2,22,132} > >> {1,2,22,133} > >> {1,2,22,134} > >> > >> Crash dump was written to: erl_crash.dump > >> eheap_alloc: Cannot allocate 373662860 bytes of memory (of type "heap"). > >> > >> > >> Abnormal termination > >> > >> > >> -module(ipv4_addrs). > >> > >> -export([first/0, next/1, test/1]). > >> > >> first()-> > >> { 1,1,0,0 }. > >> > >> next( { 255, 255, 255, 255 } ) -> done; next( { 9, 255, 255, 255 } ) > >> -> { 11, 0, 0, 0 }; next( { 172, 15, 255, 255 } ) -> { 172, 32, 0, 0 > >> }; next( { 192, 167, 255, 255 } ) -> { 192, 169, 0, 0 }; next( { > >> First, 255, 255, 255 } ) -> { First + 1, 0, 0, 0 }; next( { First, > >> Second, 255, 255 } ) -> { First, Second + 1, 0, 0 }; next( { First, > >> Second, Third, 255 } ) -> { First, Second, Third + 1, 0 }; next( { > >> First, Second, Third, Fourth } ) -> { First, Second, Third, Fourth > >> + > >> 1 }. > >> > >> test(done) -> done; > >> test(Addr) -> > >> io:format("~p~n", [Addr]), > >> Next = next(Addr), > >> test(Next). > >> > >> _______________________________________________ > >> erlang-questions mailing list > >> erlang-questions@REDACTED > >> http://erlang.org/mailman/listinfo/erlang-questions > > > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cgsmcmlxxv@REDACTED Tue Jul 24 06:40:43 2012 From: cgsmcmlxxv@REDACTED (CGS) Date: Tue, 24 Jul 2012 06:40:43 +0200 Subject: [erlang-questions] Recursive Heap Allocation In-Reply-To: References: <945E4A18C0214F36BB397894926D8164@my.domain> Message-ID: Sorry, I meant ipv4_addrs:test/1, not ipv4_addrs:next/1. As for how to avoid this crash, I suppose it becomes obvious not to rely on writing the addresses to the console, but to devise a test within ipv4_addrs:test/1 (either count the addresses, or check and report only errors which are coming from reaching undesired values). Good luck! CGS On Tue, Jul 24, 2012 at 6:25 AM, CGS wrote: > Hi, > > I managed to reproduce the error and the problem is coming from > io:format/2 (I suppose writing is too slow for the loop itself, so, the io > buffer gets overflowed). Try this replacing ipv4_addrs:next/1 with this: > > test(done) -> done; > test(Addr) -> test(next(Addrs)). > > CGS > > > > > On Mon, Jul 23, 2012 at 11:50 PM, Greg Martin wrote: > >> I think you're misunderstanding. It's like a 256 based number scheme. >> 0.255 >> is followed by 1.0 not 1.255. >> >> -----Original Message----- >> From: Ronny Meeus [mailto:ronny.meeus@REDACTED] >> Sent: Monday, July 23, 2012 2:00 PM >> To: Bob Ippolito >> Cc: Greg Martin; erlang-questions@REDACTED >> Subject: Re: [erlang-questions] Recursive Heap Allocation >> >> Hello >> >> I think there is an issue in the logic of your code: >> >> next( { First, Second, Third, 255 } ) -> { First, Second, Third + 1, 0 }; >> next( { First, Second, Third, Fourth } ) -> { First, Second, Third, >> Fourth + >> 1 }. >> >> If nothing matches, the last clause will do and the last number will be >> incremented until 255 is reached. >> If I understand you well, I would think that the looping continues with >> the >> last but 1 clause but here is the problem: you pass 0 instead of 255 so >> there is no match and you end up in the last clause again. >> Result is an endless loop. >> >> I think the code should be corrected like this: >> next( { First, Second, Third, 255 } ) -> { First, Second, Third + 1, 255 >> }; >> next( { First, Second, Third, Fourth } ) -> { First, Second, Third, >> Fourth + >> 1 }. >> >> --- >> Best regards, >> Ronny >> >> On Mon, Jul 23, 2012 at 10:04 PM, Bob Ippolito wrote: >> > You're probably generating output too fast, there's no flow control in >> > io:format/2. You can generate things to print much faster than they >> > get printed, which results in the io server accumulating so many >> > messages in its mailbox that you eventually run out of memory and/or >> > the machine grinds to a halt due to swapping. >> > >> > >> > On Mon, Jul 23, 2012 at 12:50 PM, Greg Martin >> wrote: >> >> >> >> I'm just learning erlang using Erlang R15B01 (erts-5.9.1) on Windows. >> >> I'm wondering why ipv4_addrs:test(ipv4_addrs:first()) leads to the >> >> crash below and how it can be called so that it doesn't. Thanks. >> >> >> >> {1,2,22,127} >> >> {1,2,22,128} >> >> {1,2,22,129} >> >> {1,2,22,130} >> >> {1,2,22,131} >> >> {1,2,22,132} >> >> {1,2,22,133} >> >> {1,2,22,134} >> >> >> >> Crash dump was written to: erl_crash.dump >> >> eheap_alloc: Cannot allocate 373662860 bytes of memory (of type >> "heap"). >> >> >> >> >> >> Abnormal termination >> >> >> >> >> >> -module(ipv4_addrs). >> >> >> >> -export([first/0, next/1, test/1]). >> >> >> >> first()-> >> >> { 1,1,0,0 }. >> >> >> >> next( { 255, 255, 255, 255 } ) -> done; next( { 9, 255, 255, 255 } ) >> >> -> { 11, 0, 0, 0 }; next( { 172, 15, 255, 255 } ) -> { 172, 32, 0, 0 >> >> }; next( { 192, 167, 255, 255 } ) -> { 192, 169, 0, 0 }; next( { >> >> First, 255, 255, 255 } ) -> { First + 1, 0, 0, 0 }; next( { First, >> >> Second, 255, 255 } ) -> { First, Second + 1, 0, 0 }; next( { First, >> >> Second, Third, 255 } ) -> { First, Second, Third + 1, 0 }; next( { >> >> First, Second, Third, Fourth } ) -> { First, Second, Third, Fourth >> >> + >> >> 1 }. >> >> >> >> test(done) -> done; >> >> test(Addr) -> >> >> io:format("~p~n", [Addr]), >> >> Next = next(Addr), >> >> test(Next). >> >> >> >> _______________________________________________ >> >> erlang-questions mailing list >> >> erlang-questions@REDACTED >> >> http://erlang.org/mailman/listinfo/erlang-questions >> > >> > >> > >> > _______________________________________________ >> > erlang-questions mailing list >> > erlang-questions@REDACTED >> > http://erlang.org/mailman/listinfo/erlang-questions >> > >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From siraaj@REDACTED Tue Jul 24 16:01:42 2012 From: siraaj@REDACTED (Siraaj Khandkar) Date: Tue, 24 Jul 2012 10:01:42 -0400 Subject: [erlang-questions] SSH SCP In-Reply-To: <20120718194126.10697nhr8ivc6tza@webmail.elte.hu> References: <20120716213943.17894ve1kldrmqdb@webmail.elte.hu> <0A16E23D-FDDD-43DA-B24B-D56BE405022E@khandkar.net> <20120718194126.10697nhr8ivc6tza@webmail.elte.hu> Message-ID: <3228EED7-7CD2-4546-AD96-6A562D9D313C@khandkar.net> Restoring back to inline ;) See bottom. On Jul 18, 2012, at 1:41 PM, gabre@REDACTED wrote: >> On Jul 16, 2012, at 3:39 PM, gabre@REDACTED wrote: >> >>> Hello, >>> >>> I d like to ask about the SSH module found in OTP. I would like to use >>> the SCP command (which uses ssh) via Erlang (SSH module). Is there >>> anybody, who can explain me, how to do it? I have googled and searched >>> million times, but nothing... (I have also tried it using os:cmd/1 and a >>> seld-made shell loop) >> >> >> I haven't gotten around to file transfers (with OTP ssh app) yet, but I >> did manage to mostly figure-out the basics of using the SSH app, and can >> give you a jump start. >> >> For the simplest example of doing something useful, lets say we want a >> an escript that sort-of mimics an ssh command. First, make sure you have >> an SSH directory (~/.ssh) and a pub-priv key pair (~/.ssh/id_dsa and >> ~/.ssh/id_dsa.pub) that is NOT password protected. This last part is >> important, because at this time the ssh app does not yet support >> password-protected keys [1]. After you have the key pair, the simplest >> implementation could look something like this: >> >> >> #! /usr/bin/env escript >> >> %%% ---------------------------------------------------------------------- >> %%% file: ssh.erl >> %%% ---------------------------------------------------------------------- >> >> >> main([User, Address, Port, Command]) -> >> ok = crypto:start(), >> ok = ssh:start(), >> >> Timeout = 5000, >> SSHDirectory = filename:join(os:getenv("HOME"), ".ssh"), >> >> ConnectOptions = [ >> {user, User}, >> {connect_timeout, Timeout}, >> {silently_accept_hosts, true}, >> {user_interaction, true}, >> {user_dir, SSHDirectory} >> ], >> >> case ssh:connect(Address, list_to_integer(Port), ConnectOptions) of >> {ok, ConnRef} -> >> case ssh_connection:session_channel(ConnRef, Timeout) of >> {ok, ChannId} -> >> ssh_connection:exec(ConnRef, ChannId, Command, Timeout), >> Data = collect_data(), >> io:format("~s~n", [Data]); >> >> {error, Reason} -> >> io:format("~p~n", [Reason]) >> end; >> {error, Reason} -> >> io:format("~p~n", [Reason]) >> end; >> >> main(_) -> >> io:format("USAGE: ssh.erl
~n"). >> >> >> collect_data() -> collect_data([]). >> collect_data(Data) -> >> receive >> {ssh_cm, _, {data, _, _, Datum}} -> >> collect_data([Datum | Data]); >> >> {ssh_cm, _, {closed, _}} -> >> lists:reverse(Data); >> >> {ssh_cm, _} -> >> collect_data(Data) >> end. >> >> >> For a more extended example of the use of the above code, see my Cluster >> Commander project [2], particularly commander_worker_otp.erl module. I >> also took advantage of OS's ssh/scp commands (as an alternative to OTP >> ssh and as the only, for now, way to do scp), see >> commander_worker_os.erl module and os_cmd/1 function in >> commander_lib.erl module. >> >> The above uses ssh_connection:exec/4 to just send a command string to >> the remote host and receive the resulting data, but, for more >> interesting uses, you can also use ssh_connection:send/3-5 and push any >> data you want to the remote host (assuming it knows how to interpret >> it). >> >> For a more interesting example, see Kenji Rikitake's sshrpc [3] [4] and >> my (uber-early and half-baked) attempts at implementing the ssh_channel >> behavior for my needs [5]. >> >> >> [1] - http://erlang.org/pipermail/erlang-questions/2010-April/050637.html >> [2] - https://github.com/ibnfirnas/cluster-commander >> [3] - https://github.com/jj1bdx/sshrpc >> [4] - http://www.erlang-factory.com/conference/SFBay2010/speakers/kenjirikitake >> [5] - https://github.com/ibnfirnas/data-mill >> > Hello, > > Thank you for your quick answer, Siraaj, but I'd like to do the inverse of > what you described below. I'd like to start an SSH daemon with Erlang (OTP > SSH), and use it with the following command: > scp filename_on_my_pc host:path > (copying) > > I start Erlang SSH like this: > start() -> > crypto:start(), > ssh:start(), > Shell = myshell(), > ssh:daemon({0,0,0,0}, 45678, [{pwdfun, fun check_pwd/2}, > {system_dir, "/home/.../global-ssh-host-keys"}, > {ssh_cli,{ssh_cli, [Shell]}}, > {subsystems, > [{"sftp", {ssh_sftpd, [ > {vsn, 3} > ]}}]}]). > Shell is a self-written simple shell, but it has no job here, 'cause using > scp command is something like giving the ssh command a parameter (I think). > > I know this is an improper way of using scp because ssh_cli tries to parse > "scp -t path" (this is the command which is called via ssh by scp) as an > Erlang term. I d like to ask you to give me instructions. How can I set up a > working Erlang ssh deamon (working with scp)? I see. I'd be interested in getting that to work as well, but haven't gotten all the way there last time I played with it :) This is where I left off, hopefully it'll be of use to you: https://github.com/ibnfirnas/data-mill/blob/master/apps/datamill_reclaimer/src/datamill_reclaimer_ssh_server.erl I'm still interested in completing this and want to revisit it at some point, its just low on my priority queue at this time. Please let me know if you get further along! -- Siraaj Khandkar .o. ..o ooo From attila.r.nohl@REDACTED Tue Jul 24 17:11:31 2012 From: attila.r.nohl@REDACTED (Attila Rajmund Nohl) Date: Tue, 24 Jul 2012 17:11:31 +0200 Subject: [erlang-questions] Memory-effective queue handling Message-ID: Hello! I have a data structure where I manage "historical data": a list of timestamps when events happened. I only need to know how many events happened in the last 24 hours. When an event happens, I put the timestamp into the queue (I use the queue module in OTP). When I check how many values are in the queue, I also remove the entries that are older than 24 hours. My problem is: when I remove the old elements, I can do it one by one and at each step the (possibly big) queue gets copied and I run out of memory. Is there a cleverer way to remove elements from queue? Or maybe I shall use an ordered_set ETS table instead of a queue? From dmkolesnikov@REDACTED Tue Jul 24 18:11:19 2012 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Tue, 24 Jul 2012 19:11:19 +0300 Subject: [erlang-questions] Memory-effective queue handling In-Reply-To: References: Message-ID: <2347870F-33E8-4FB2-B809-A65150E4BF19@gmail.com> Hello, I do not think that this is a good idea to keep whole 24h statistic is same queue. For "cleanup" efficiency you could have a queue per time slot (e.g. per hour, per 30min, etc) Instead of queue, I am using patched gb_trees exactly for same purposes but I am doing both read/write. - Dmitry On Jul 24, 2012, at 6:11 PM, Attila Rajmund Nohl wrote: > Hello! > > I have a data structure where I manage "historical data": a list of > timestamps when events happened. I only need to know how many events > happened in the last 24 hours. When an event happens, I put the > timestamp into the queue (I use the queue module in OTP). When I check > how many values are in the queue, I also remove the entries that are > older than 24 hours. My problem is: when I remove the old elements, I > can do it one by one and at each step the (possibly big) queue gets > copied and I run out of memory. Is there a cleverer way to remove > elements from queue? Or maybe I shall use an ordered_set ETS table > instead of a queue? > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From eriksoe@REDACTED Tue Jul 24 20:36:43 2012 From: eriksoe@REDACTED (=?ISO-8859-1?Q?Erik_S=F8e_S=F8rensen?=) Date: Tue, 24 Jul 2012 20:36:43 +0200 Subject: [erlang-questions] Memory-effective queue handling In-Reply-To: References: Message-ID: Den 24/07/2012 17.11 skrev "Attila Rajmund Nohl" : > > Hello! > > I have a data structure where I manage "historical data": a list of > timestamps when events happened. I only need to know how many events > happened in the last 24 hours. When an event happens, I put the > timestamp into the queue (I use the queue module in OTP). When I check > how many values are in the queue, I also remove the entries that are > older than 24 hours. My problem is: when I remove the old elements, I > can do it one by one and at each step the (possibly big) queue gets > copied and I run out of memory. I am curious: how do you manage to do this? Can we see the code? (Certainly it shouldn't be the case that much copying should be done at each step.) > Is there a cleverer way to remove > elements from queue? Or maybe I shall use an ordered_set ETS table > instead of a queue? > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From ronny.meeus@REDACTED Tue Jul 24 20:52:07 2012 From: ronny.meeus@REDACTED (Ronny Meeus) Date: Tue, 24 Jul 2012 20:52:07 +0200 Subject: [erlang-questions] Memory-effective queue handling In-Reply-To: References: Message-ID: A possibility is to count the number of elements that have to be present in the new list by just iterating over it. Once you know the number of elements, the queue can be split in 2. ---- Ronny On Tue, Jul 24, 2012 at 8:36 PM, Erik S?e S?rensen wrote: > > Den 24/07/2012 17.11 skrev "Attila Rajmund Nohl" : > > >> >> Hello! >> >> I have a data structure where I manage "historical data": a list of >> timestamps when events happened. I only need to know how many events >> happened in the last 24 hours. When an event happens, I put the >> timestamp into the queue (I use the queue module in OTP). When I check >> how many values are in the queue, I also remove the entries that are >> older than 24 hours. My problem is: when I remove the old elements, I >> can do it one by one and at each step the (possibly big) queue gets >> copied and I run out of memory. > I am curious: how do you manage to do this? Can we see the code? > (Certainly it shouldn't be the case that much copying should be done at each > step.) > >> Is there a cleverer way to remove >> elements from queue? Or maybe I shall use an ordered_set ETS table >> instead of a queue? >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From knutin@REDACTED Tue Jul 24 23:09:05 2012 From: knutin@REDACTED (Knut Nesheim) Date: Tue, 24 Jul 2012 23:09:05 +0200 Subject: [erlang-questions] Memory-effective queue handling In-Reply-To: References: Message-ID: On Tue, Jul 24, 2012 at 5:11 PM, Attila Rajmund Nohl wrote: > Hello! > > I have a data structure where I manage "historical data": a list of > timestamps when events happened. I only need to know how many events > happened in the last 24 hours. Using an ordered_set ETS-table and iterating over the table to count and delete events will be very fast and not explode your memory usage. You can also enable compression on the table. ETS allows concurrent writes which removes a potential disaster if you are using a single process in charge of the queue (with many events, this process might get overloaded). However, if you are only interested in the count of events, maybe you don't need to store all events in full? If you are ok with losing some granularity, you can use one "bucket" per minute or second or whatever for the last 24 hours. In the bucket you keep the count in that time period, incrementing it for every event. Finding the current bucket can be done using "Now rem NumBuckets" where Now has the granularity you need. The count for the last 24 hours is the sum of all the elements in the table. If you use an ETS table to keep the count (ets:update_counter/3 is very nice for this), you can reset the count by doing a decrement of the current bucket every minute or second by fetching the current count, then decrementing with that value (this would work even with interleaved operations.) If you are able to use a single process in charge of getting the count polling-style, you could copy the buckets to a second table at every polling interval. The diff between the two tables would be the number of events since last poll. With this solution, the space required is determined by the granularity. Finding the sum is a traversal away. Pruning can be fiddly, but requires no CPU or memory intensive work. To me it sounds good on paper, maybe there are some obstacles I didn't think of. Maybe there would be very high write lock contention on the single element to update, maybe ets:update_counter/3 still performs very well. For problems like these, pushing the requirements might be very helpful. If you could do a good sample of the events, you can still get an accurate answer when you ask for the count and keep your current approach. Regards Knut From ok@REDACTED Wed Jul 25 02:58:34 2012 From: ok@REDACTED (Richard O'Keefe) Date: Wed, 25 Jul 2012 12:58:34 +1200 Subject: [erlang-questions] Memory-effective queue handling In-Reply-To: References: Message-ID: You have a collection of {Time_Stamp, Data} items covering the last 24 hours. You want to add such pairs, and you also want to remove old pairs of this kind. Some kind of balanced search tree such as an AVL tree or 2-3 tree can do this in O(lg n) time per updated, where n is the number of pairs in your collection. A very simple hack would be to divide a sequence into some number of buckets where the first and last are special, so we have {Old,Middling,New} New is a list running from newest to oldest within the current bucket. To add a new event: add_event(When, What, {Old,Middling,New}) -> {Old,Middling,[{When,What}|New]}. Old is a list running from oldest to newest within the oldest bucket. To remove old events: remove_old_events(Now, {Old,Middling,New}) -> Bound = <| however you calculate Now - 24 hours |>, {remove_expired(Bound, Old),Middling,New}. remove_expired(Bound, [{When,What}|Old]) when When =< Bound -> remove_expired(Bound, Old); remove_expired(_, Old) -> Old. One extra step is to shift the buckets when the hour strikes: hourly_shift({_,{A,B,C,D,E,F,G,H,I,J,K,L,M,N, O,P,Q,R,S,T,U,V,W,X},New}) -> {A,{B,C,D,E,F,G,H,I,J,K,L,M,N, O,P,Q,R,S,T,U,V,W,X,lists:reverse(New)},[]}. To walk over the elements in increasing order: foldl(Fun, Acc, {Old,{A,B,C,D,E,F,G,I,J,K,L,M,N, O,P,Q,R,S,T,U,V,W,X},New}) -> Acc_0 = foldl(Fun, Acc, Old), Acc_1 = foldl(Fun, Acc_1, A), ... Acc24 = foldl(Fun, Acc23, X), foldl(Fun, Acc24, lists:reverse(New)). Whether this is a good idea depends on what else you want to do. It also matters how many events you expect to be in the data structure. Oh yes, to keep track of how many events there are, use {Count,Old,Middling,New} with the other changes to the code being obvious. From roberto@REDACTED Wed Jul 25 07:30:55 2012 From: roberto@REDACTED (Roberto Ostinelli) Date: Tue, 24 Jul 2012 22:30:55 -0700 Subject: [erlang-questions] meck In-Reply-To: References: Message-ID: > Hi, > you are mixing eunit generators with simple test cases. Did you mean this? > > get_config_test() -> > meck:new(mocked), > meck:expect(mocked, myfun, fun(Filename) -> ok end), > ?assertEqual(ok, get_config()), > ?assert(meck:called(mocked, myfun, ["test.config"])), > meck:unload(mocked). > > Normally, you should split meck:new and meck:unload to separate > setup/cleanup functions. Because if a test fails and the module is not > cleaned, it might unexpectedly do ill for your other test cases. > can't i use this in generators? the reason why i didn't include unload is because of this: https://github.com/eproxus/meck/issues/72 r. -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang@REDACTED Wed Jul 25 09:36:02 2012 From: erlang@REDACTED (Joe Armstrong) Date: Wed, 25 Jul 2012 09:36:02 +0200 Subject: [erlang-questions] Recursive Heap Allocation In-Reply-To: References: Message-ID: It's a bug in the smp handling of io:format. This program has similar behaviour -module(bug1). -compile(export_all). test(N) -> io:format("~p~n", [N]), test(N+1). I did this: 1 > spawn(fun() -> etop:start() end). 2> bug1:test(0). You can see that user_drv global process is accumulating messages faster than it can handle them. It's in usr_drv:io_request/3. As far as I'm aware io:format did have flow control so this shouldn't happen I then changed the program to the following: test(N) -> Len = process_info(whereis(user_drv),[message_queue_len]), io:format("~p ~p~n", [N, Len]), test(N+1). The printout from this is weird - the length stays at zero for long periods - then climbs linearly up to some number like 180 or 169 then instantaneously goes back to zero Now I ran with "erl -smp disable" The counter stays at zero forever. I think the I/O handling for smp erlang is buggy On Mon, Jul 23, 2012 at 9:50 PM, Greg Martin wrote: > I'm just learning erlang using Erlang R15B01 (erts-5.9.1) on Windows. I'm > wondering why ipv4_addrs:test(ipv4_addrs:first()) leads to the crash below > and how it can be called so that it doesn't. Thanks. > > {1,2,22,127} > {1,2,22,128} > {1,2,22,129} > {1,2,22,130} > {1,2,22,131} > {1,2,22,132} > {1,2,22,133} > {1,2,22,134} > > Crash dump was written to: erl_crash.dump > eheap_alloc: Cannot allocate 373662860 bytes of memory (of type "heap"). > > > Abnormal termination > > > -module(ipv4_addrs). > > -export([first/0, next/1, test/1]). > > first()-> > { 1,1,0,0 }. > > next( { 255, 255, 255, 255 } ) -> done; > next( { 9, 255, 255, 255 } ) -> { 11, 0, 0, 0 }; > next( { 172, 15, 255, 255 } ) -> { 172, 32, 0, 0 }; > next( { 192, 167, 255, 255 } ) -> { 192, 169, 0, 0 }; > next( { First, 255, 255, 255 } ) -> { First + 1, 0, 0, 0 }; > next( { First, Second, 255, 255 } ) -> { First, Second + 1, 0, 0 }; > next( { First, Second, Third, 255 } ) -> { First, Second, Third + 1, 0 }; > next( { First, Second, Third, Fourth } ) -> { First, Second, Third, Fourth + > 1 }. > > test(done) -> done; > test(Addr) -> > io:format("~p~n", [Addr]), > Next = next(Addr), > test(Next). > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From carlsson.richard@REDACTED Wed Jul 25 10:02:48 2012 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Wed, 25 Jul 2012 10:02:48 +0200 Subject: [erlang-questions] Recursive Heap Allocation In-Reply-To: References: Message-ID: <500FA828.3030104@gmail.com> On 07/25/2012 09:36 AM, Joe Armstrong wrote: > This program has similar behaviour > > -module(bug1). > -compile(export_all). > > test(N) -> > io:format("~p~n", [N]), > test(N+1). > > I did this: > > 1 > spawn(fun() -> etop:start() end). > 2> bug1:test(0). > > You can see that user_drv global process is accumulating messages > faster than it can handle them. > It's in usr_drv:io_request/3. > > As far as I'm aware io:format did have flow control so this shouldn't happen The I/O protocol has no built-in flow control. > Now I ran with "erl -smp disable" > > The counter stays at zero forever. With only a single scheduler, it will be alternating between running the producer process and the consumer processes, but they won't be running at the same time, so the consumer has a better chance of keeping up. The number of reductions used per message is the deciding factor, and if I recall correctly, there's a built-in penalty in reductions when you pass a message to a process that already has a long queue. With multiple schedulers, the producer and the consumer will run on separate schedulers, i.e., they run on separate OS threads and reduction count doesn't matter, only actual time per message. The only thing keeping them back a little is the synchronization needed for the mailbox. It's not so surprising that it's easier to overload the consumer in this case. (Also note that the I/O protocol causes the formatting to be done on the receiver side, so if you're printing complex terms, you're making the consumer do a lot more work per message than the producer.) /Richard From erlang@REDACTED Wed Jul 25 10:25:18 2012 From: erlang@REDACTED (Joe Armstrong) Date: Wed, 25 Jul 2012 10:25:18 +0200 Subject: [erlang-questions] How can I debug this? In-Reply-To: <500965A0.5030500@gmail.com> References: <500965A0.5030500@gmail.com> Message-ID: > {"init terminating in > do_boot",{undef,[{boss,start,[],[],{init,start_it,1,[{file > ,"init.erl"},[line,1041}]},{init,start_em,1,[{file,"init.erl"},{line,1022}]}]}} This error message means Erlang cannot find the file boss.beam - some code has tried to evaluate the function boss:start() (That's what the cryptic error message means) and the module boss is not loaded. When this happens the system tries to find a file called boss.beam - this must be one of the files in the code load path. To debug this - start erlang (If you can't start erlang at all) you're screwed. then in the erlang shell give the command and code:get_path() (Or io:format("~p~n",[code:get_path()]). When I do this I see this: 1 > io:format("~p~n",[code:get_path()]). [".","/usr/local/lib/erlang/lib/kernel-2.15/ebin", "/usr/local/lib/erlang/lib/stdlib-1.18/ebin", ... "/home/joe/projects/doc42/ebin","/home/joe/projects/validate", "/home/joe/nobackup/erlang_imports/deps/meck/ebin", "/home/joe/nobackup/erlang_imports/deps/cowboy/ebin", "/home/joe/nobackup/erlang_imports/deps/bitcask/ebin"] One of these directories should contain the file boss.beam Since boss.beam is not in your path you have to find and add it to your path. This can be done in (at least) three ways: If boss.beam is in the directory DIR then you can evaluate this: > code:add_pathz(Dir). in the shell Better is to add it to the erlang startup file. On a unix or mac you just make a file called ".erlang" which is either on ${HOME}/.erlang or the directory you start erlang from. I tried to do this on windows but I have no evironment variable HOME and I do now have write permissions under Program Files (or whatever) (( Actually I'd like to keep an up-to-date install guide on the erlang web site so if anybody could tell me how to make a .erlang file on windows I would be appreciate this)) Once we know how to make a .erlang file I can fill in the details of how to get your boss example running. Just keep mailing back to this thread every time you have a problem and we'll get it fixed. Cheers /Joe On Fri, Jul 20, 2012 at 4:05 PM, Ian wrote: > Hi All, > > I am attempting to work through the Chicago Boss quickstart tutorial at > https://github.com/evanmiller/ChicagoBoss/wiki/Quickstart > > I am using Windows 7 - 64 bit, and the R15B release of Erlang. New to > Erlang. > > First a minor bug - I think I installed the 32 bit version, removed it, and > installed the 64 bit version. Then I discovered that the 64 bit version's > bin directory is C:\Program Files\erl5.9.1\bin but the directory in the path > was C:\Program Files (x86)\erl5.9.1\bin. That was why nothing worked. > > After that was corrected, I created the project and set up the file in Step > 4, before starting the server with start-server.bat. The werl window flashed > up an error message for a second or so - and vanished before I could read > it! > > A few tries later I was able to catch it in a screen dump. The error message > was: > > {"init terminating in > do_boot",{undef,[{boss,start,[],[],{init,start_it,1,[{file > ,"init.erl"},[line,1041}]},{init,start_em,1,[{file,"init.erl"},{line,1022}]}]}} > > (copied by hand) > > I know this means the server did not start up. But why, and what have I done > wrong/missed out? I have not found a init.erl file. > > My project is called erlian and is in d:\erlian . The file > D:\erlian\src\controller\erlian_greeting_controller.erl > contains: > > -module(erlian_greeting_controller, [Req]). > -compile(export_all). > > hello('GET', []) -> > {output, "Erlian says hello!"}. > > (copy/pasted) > > Help much appreciated. > > Ian > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From czinkos@REDACTED Wed Jul 25 10:44:38 2012 From: czinkos@REDACTED (Zsolt Czinkos) Date: Wed, 25 Jul 2012 10:44:38 +0200 Subject: [erlang-questions] solr client in erlang? Message-ID: Hi all, Is anybody working with Erlang and solr? I've found esolr[1], but does not seem to be maintained. Thanks for any pointer. Zsolt [1] http://forum.trapexit.org/viewtopic.php?t=13059 From ulf@REDACTED Wed Jul 25 10:59:45 2012 From: ulf@REDACTED (Ulf Wiger) Date: Wed, 25 Jul 2012 10:59:45 +0200 Subject: [erlang-questions] solr client in erlang? In-Reply-To: References: Message-ID: Well, there's always riak_search, https://github.com/basho/riak_search/tree/master/src but the solr interface is embedded into the application. I don't know how reusable it is. BR, Ulf W On 25 Jul 2012, at 10:44, Zsolt Czinkos wrote: > Hi all, > > Is anybody working with Erlang and solr? I've found esolr[1], but does > not seem to be maintained. > > Thanks for any pointer. > > Zsolt > > > [1] http://forum.trapexit.org/viewtopic.php?t=13059 > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. http://feuerlabs.com From hobson42@REDACTED Wed Jul 25 11:49:29 2012 From: hobson42@REDACTED (Ian) Date: Wed, 25 Jul 2012 10:49:29 +0100 Subject: [erlang-questions] How can I debug this? In-Reply-To: References: <500965A0.5030500@gmail.com> Message-ID: <500FC129.1080709@gmail.com> Hi Joe, Thanks for the reply! On 25/07/2012 09:25, Joe Armstrong wrote: > On a unix or mac you just make a file called ".erlang" which is either > on ${HOME}/.erlang or the directory you start erlang from. I tried to > do this on windows but I have no evironment variable HOME and I do now > have write permissions under Program Files (or whatever) (( Actually > I'd like to keep an up-to-date install guide on the erlang web site so > if anybody could tell me how to make a .erlang file on windows I would > be appreciate this The equivalent of HOME is %USERPROFILE%. The more tricky problem with the windows install was that the install script does not change the path variable. I guess because there might be more than one version of erlang installed. The more satisfactory solution is to add the /bin to the path. Setting up a erl.bat and .werl.bat files is not enough because escript remains broken, and that leads to further problems. > )) Once we know how to make a .erlang file I can fill in the details > of how to get your boss example running. Just keep mailing back to > this thread every time you have a problem and we'll get it fixed. > Cheers /Joe Thanks for the offer. I'll try not to waste people's time. Ian From attila.r.nohl@REDACTED Wed Jul 25 11:57:14 2012 From: attila.r.nohl@REDACTED (Attila Rajmund Nohl) Date: Wed, 25 Jul 2012 11:57:14 +0200 Subject: [erlang-questions] Memory-effective queue handling In-Reply-To: References: Message-ID: 2012/7/24 Erik S?e S?rensen : > > Den 24/07/2012 17.11 skrev "Attila Rajmund Nohl" : > > >> >> Hello! >> >> I have a data structure where I manage "historical data": a list of >> timestamps when events happened. I only need to know how many events >> happened in the last 24 hours. When an event happens, I put the >> timestamp into the queue (I use the queue module in OTP). When I check >> how many values are in the queue, I also remove the entries that are >> older than 24 hours. My problem is: when I remove the old elements, I >> can do it one by one and at each step the (possibly big) queue gets >> copied and I run out of memory. > I am curious: how do you manage to do this? Can we see the code? > (Certainly it shouldn't be the case that much copying should be done at each > step.) I can show an excerpt from the code: increase_counter(#ch{queue=Q, counter_value=CV}=CH, Value) -> Date = ?MODULE:get_date(), NewQueue=queue:in({Date, Value}, Q), CH#ch{queue=NewQueue, counter_value=CV+Value}. get_value(#ch{hist_len=HL}=CH) -> % get the values from the queue until we reach a first element in the queue % which is not older then the hist_len (e.g. not older than 15 mins) Date = ?MODULE:get_date(), get_younger_than(CH, Date-HL). get_younger_than(#ch{queue=Q, counter_value=CV}=CH, TargetDate) -> case queue:out(Q) of {empty, _} -> 0 = CV, % this is an assert, the value shall be 0 {CH, 0}; {{value, {Date, Value}}, NewQ} when Date < TargetDate -> get_younger_than(CH#ch{queue=NewQ, counter_value=CV-Value}, TargetDate); {{value, _}, _} -> % the currently removed entry is too young, so leave that in the % queue (i.e. use the parameter queue). {CH, CV} end. The copying is done in the queue:out call, because it returns not just the first value, but the rest of the queu (newQ) too. From aschultz@REDACTED Wed Jul 25 12:14:33 2012 From: aschultz@REDACTED (Andreas Schultz) Date: Wed, 25 Jul 2012 12:14:33 +0200 (CEST) Subject: [erlang-questions] SSL Server Bad record mac error In-Reply-To: Message-ID: <127222070.301766.1343211273642.JavaMail.root@tpip.net> Hi Morgan, The connection should be closing cleanly already. The alert should be sent by the ssl state machine over the ssl connection and both sides will close. To diagnose this further, you could try to capture the broken ssl session in a controlled setup. The selected crypto suite is in the unecrypted part of the handshake. If you disable all EDH ciphers, wireshark or ssldump can be used to decipher the encrypted part of the exchange. That should give you a hint what excatly is going on. Andreas ----- Original Message ----- > > Hi everyone, > > > I'm currently trying to find out, where could come from this error : > > > > =ERROR REPORT==== 23-Jul-2012::11:03:56 === > SSL: certify: ssl_record.erl:654:Fatal error: bad record mac > > > The SSL function where it is coming from is decipher: > > > ---------------------------------- ssl_record.erl > --------------------------------------------- > > > > decipher(TLS=#ssl_tls{type=Type, version=Version, fragment=Fragment}, > CS0) -> > SP = CS0#connection_state.security_parameters, > BCA = SP#security_parameters.bulk_cipher_algorithm, > HashSz = SP#security_parameters.hash_size, > CipherS0 = CS0#connection_state.cipher_state, > case ssl_cipher:decipher(BCA, HashSz, CipherS0, Fragment, Version) of > {T, Mac, CipherS1} -> > CS1 = CS0#connection_state{cipher_state = CipherS1}, > TLength = size(T), > {MacHash, CS2} = hash_and_bump_seqno(CS1, Type, Version, TLength, T), > case is_correct_mac(Mac, MacHash) of > true -> > {TLS#ssl_tls{fragment = T}, CS2}; > false -> > ?ALERT_REC(?FATAL, ?BAD_RECORD_MAC) %%<-------- HERE > end; > #alert{} = Alert -> > Alert > end. > ---------------------------------------------------------------------------------------------------- > > > I'm not really sure if this error is happening at handshake, or read. > The thing is, the error is only coming only from users connecting > from a BlackBerry device (Can't know which OS version, I suppose it > comes from rather old devices). > > > If there is nothing I can't do server-side, does anyone could point > out how to handle the error, in order to close cleanly the > connection ? > > > Regards, > > > Morgan. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- -- Dipl. Inform. Andreas Schultz email: as@REDACTED phone: +49-391-819099-224 mobil: +49-170-2226073 ------------------ managed broadband access ------------------ Travelping GmbH phone: +49-391-8190990 Roentgenstr. 13 fax: +49-391-819099299 D-39108 Magdeburg email: info@REDACTED GERMANY web: http://www.travelping.com Company Registration: HRB21276 Handelsregistergericht Chemnitz Geschaeftsfuehrer: Holger Winkelmann | VAT ID No.: DE236673780 -------------------------------------------------------------- From dave@REDACTED Wed Jul 25 14:14:34 2012 From: dave@REDACTED (Dave Cottlehuber) Date: Wed, 25 Jul 2012 14:14:34 +0200 Subject: [erlang-questions] How can I debug this? In-Reply-To: References: <500965A0.5030500@gmail.com> Message-ID: On 25 July 2012 10:25, Joe Armstrong wrote: > Better is to add it to the erlang startup file. > > On a unix or mac you just make a file called ".erlang" which is either > on ${HOME}/.erlang > or the directory you start erlang from. > > I tried to do this on windows but I have no evironment variable HOME > and I do now have write permissions > under Program Files (or whatever) > > (( Actually I'd like to keep an up-to-date install guide on the erlang > web site so if anybody > could tell me how to make a .erlang file on windows I would be appreciate this)) On Windows, the HOME env var trick doesn't work, you need to use the current working directory of the werl/erl process instead. @Joe what format (or where in OTP source) is best for documenting this? I've got a few other things that would be useful too. ## Creating a working .erlang file: - open notepad or , and paste this in, or whatever you prefer: c:pwd(), io:format("running .erlang~n", []). - file -> save as ".erlang" including the double quotes. This tells the dialog box NOT to add .txt, just to use the provided filename. ## Create a shortcut to use this new startup file: - right-click on desktop - create new shortcut - enter path c:\erlang\bin\werl.exe (or where-ever you installed the appropriate version) - name it whatever & click finish - right-click on new icon & select "properties" - change "Start In" field to the directory you want to load your .erlang file from - click OK ## Alternatively, from command line start /d c:\home c:\erlang\bin\werl.exe Or: - change directory to the same folder as your .erlang file - run c:\erlang\bin\werl.exe (or erl.exe) BTW Ian's original issue is that Chicago Boss uses proper as a rebar dependency, and proper uses some shell scripts which don't work under native windows. This prevents Boss from compiling successfully depending on what unix-like shell you are using, if any. A+ Dave From tristan.sloughter@REDACTED Wed Jul 25 14:16:47 2012 From: tristan.sloughter@REDACTED (Tristan Sloughter) Date: Wed, 25 Jul 2012 07:16:47 -0500 Subject: [erlang-questions] solr client in erlang? In-Reply-To: References: Message-ID: And Elastic Search: https://github.com/tsloughter/erlastic_search http://elasticsearch.com/ Tristan On Wed, Jul 25, 2012 at 3:59 AM, Ulf Wiger wrote: > > Well, there's always riak_search, > > https://github.com/basho/riak_search/tree/master/src > > but the solr interface is embedded into the application. I don't know how > reusable it is. > > BR, > Ulf W > > On 25 Jul 2012, at 10:44, Zsolt Czinkos wrote: > > > Hi all, > > > > Is anybody working with Erlang and solr? I've found esolr[1], but does > > not seem to be maintained. > > > > Thanks for any pointer. > > > > Zsolt > > > > > > [1] http://forum.trapexit.org/viewtopic.php?t=13059 > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. > http://feuerlabs.com > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlsson.richard@REDACTED Wed Jul 25 14:38:07 2012 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Wed, 25 Jul 2012 14:38:07 +0200 Subject: [erlang-questions] Memory-effective queue handling In-Reply-To: References: Message-ID: <500FE8AF.90308@gmail.com> On 07/25/2012 11:57 AM, Attila Rajmund Nohl wrote: >>> I have a data structure where I manage "historical data": a list of >>> timestamps when events happened. I only need to know how many events >>> happened in the last 24 hours. When an event happens, I put the >>> timestamp into the queue (I use the queue module in OTP). When I check >>> how many values are in the queue, I also remove the entries that are >>> older than 24 hours. My problem is: when I remove the old elements, I >>> can do it one by one and at each step the (possibly big) queue gets >>> copied and I run out of memory. >> I am curious: how do you manage to do this? Can we see the code? >> (Certainly it shouldn't be the case that much copying should be done at each >> step.) > > I can show an excerpt from the code: > > increase_counter(#ch{queue=Q, counter_value=CV}=CH, Value) -> > Date = ?MODULE:get_date(), > NewQueue=queue:in({Date, Value}, Q), > CH#ch{queue=NewQueue, counter_value=CV+Value}. > > get_value(#ch{hist_len=HL}=CH) -> > % get the values from the queue until we reach a first element in the queue > % which is not older then the hist_len (e.g. not older than 15 mins) > Date = ?MODULE:get_date(), > get_younger_than(CH, Date-HL). > > get_younger_than(#ch{queue=Q, counter_value=CV}=CH, TargetDate) -> > case queue:out(Q) of > {empty, _} -> > 0 = CV, % this is an assert, the value shall be 0 > {CH, 0}; > {{value, {Date, Value}}, NewQ} when Date < TargetDate -> > get_younger_than(CH#ch{queue=NewQ, counter_value=CV-Value}, TargetDate); > {{value, _}, _} -> > % the currently removed entry is too young, so leave that in the > % queue (i.e. use the parameter queue). > {CH, CV} > end. > > The copying is done in the queue:out call, because it returns not just > the first value, but the rest of the queu (newQ) too. You'd only run out of space if the queue is so long that you don't have memory enough to call lists:reverse() on the current in-half of the queue (and after that, the old queue will immediately become garbage). Can that really be the case? But you should be able to improve the behaviour of the above code by using queue:peek_r(Q) instead of queue:out(Q), and then call queue:drop_r(Q) in the second case clause only, so no rewriting of the queue representation is done unless necessary. /Richard From eriksoe@REDACTED Wed Jul 25 15:01:32 2012 From: eriksoe@REDACTED (=?ISO-8859-1?Q?Erik_S=F8e_S=F8rensen?=) Date: Wed, 25 Jul 2012 15:01:32 +0200 Subject: [erlang-questions] Memory-effective queue handling In-Reply-To: References: Message-ID: Den 25/07/2012 11.57 skrev "Attila Rajmund Nohl" : > > 2012/7/24 Erik S?e S?rensen : > > > > Den 24/07/2012 17.11 skrev "Attila Rajmund Nohl" < attila.r.nohl@REDACTED>: > > > > > >> > >> Hello! > >> > >> I have a data structure where I manage "historical data": a list of > >> timestamps when events happened. I only need to know how many events > >> happened in the last 24 hours. When an event happens, I put the > >> timestamp into the queue (I use the queue module in OTP). When I check > >> how many values are in the queue, I also remove the entries that are > >> older than 24 hours. My problem is: when I remove the old elements, I > >> can do it one by one and at each step the (possibly big) queue gets > >> copied and I run out of memory. > > I am curious: how do you manage to do this? Can we see the code? > > (Certainly it shouldn't be the case that much copying should be done at each > > step.) > > I can show an excerpt from the code: > > increase_counter(#ch{queue=Q, counter_value=CV}=CH, Value) -> > Date = ?MODULE:get_date(), > NewQueue=queue:in({Date, Value}, Q), > CH#ch{queue=NewQueue, counter_value=CV+Value}. > > get_value(#ch{hist_len=HL}=CH) -> > % get the values from the queue until we reach a first element in the queue > % which is not older then the hist_len (e.g. not older than 15 mins) > Date = ?MODULE:get_date(), > get_younger_than(CH, Date-HL). > > get_younger_than(#ch{queue=Q, counter_value=CV}=CH, TargetDate) -> > case queue:out(Q) of > {empty, _} -> > 0 = CV, % this is an assert, the value shall be 0 > {CH, 0}; > {{value, {Date, Value}}, NewQ} when Date < TargetDate -> > get_younger_than(CH#ch{queue=NewQ, counter_value=CV-Value}, TargetDate); > {{value, _}, _} -> > % the currently removed entry is too young, so leave that in the > % queue (i.e. use the parameter queue). > {CH, CV} > end. > > The copying is done in the queue:out call, because it returns not just > the first value, but the rest of the queu (newQ) too. That doesn't copy the entire queue. That is, occasionally it recreates the *spine* of a new list the length of the entire queue, but normally it does way less than that. The problem is that the way your code works, that worst-case is every time, and the throwing-away of old entries is also done from the start each time: your get_value() does not return the pruned queue. A further consequence of that is that Richard's statement doesn't hold: because you keep a reference (apparently) to the original, unpruned queue, the memory will have to hold both list spines simultaneously. So I'd suggest you change the return value of get_value. -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlsson.richard@REDACTED Wed Jul 25 15:21:00 2012 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Wed, 25 Jul 2012 15:21:00 +0200 Subject: [erlang-questions] Memory-effective queue handling In-Reply-To: References: Message-ID: <500FF2BC.7010801@gmail.com> On 07/25/2012 03:01 PM, Erik S?e S?rensen wrote: > > increase_counter(#ch{queue=Q, counter_value=CV}=CH, Value) -> > > Date = ?MODULE:get_date(), > > NewQueue=queue:in({Date, Value}, Q), > > CH#ch{queue=NewQueue, counter_value=CV+Value}. > > > > get_value(#ch{hist_len=HL}=CH) -> > > % get the values from the queue until we reach a first element in > the queue > > % which is not older then the hist_len (e.g. not older than 15 mins) > > Date = ?MODULE:get_date(), > > get_younger_than(CH, Date-HL). > > > > get_younger_than(#ch{queue=Q, counter_value=CV}=CH, TargetDate) -> > > case queue:out(Q) of > > {empty, _} -> > > 0 = CV, % this is an assert, the value shall be 0 > > {CH, 0}; > > {{value, {Date, Value}}, NewQ} when Date < TargetDate -> > > get_younger_than(CH#ch{queue=NewQ, > counter_value=CV-Value}, TargetDate); > > {{value, _}, _} -> > > % the currently removed entry is too young, so leave that > in the > > % queue (i.e. use the parameter queue). > > {CH, CV} > > end. > > > > The copying is done in the queue:out call, because it returns not just > > the first value, but the rest of the queu (newQ) too. > That doesn't copy the entire queue. That is, occasionally it recreates > the *spine* of a new list the length of the entire queue, but normally > it does way less than that. > The problem is that the way your code works, that worst-case is every > time, and the throwing-away of old entries is also done from the start > each time: your get_value() does not return the pruned queue. > A further consequence of that is that Richard's statement doesn't hold: > because you keep a reference (apparently) to the original, unpruned > queue, the memory will have to hold both list spines simultaneously. > So I'd suggest you change the return value of get_value. I might be a bit tired, but where does he still hold a reference to the original queue (unless somewhere outside the shown code)? And if so, is the problem really that he can't hold 2 copies of the queue at the same time? - In that case he's pushing the limits anyway. As far as I can see, get_value() tail calls to get_younger_than(), so no references are kept there, and get_younger_than() keeps tailcalling itself with the updated queue in its #ch-record, dropping the previous queue. The final #ch-record gets returned to the caller. Only the result of the last call to queue:out() does any wasted rewriting. /Richard From attila.r.nohl@REDACTED Wed Jul 25 15:44:35 2012 From: attila.r.nohl@REDACTED (Attila Rajmund Nohl) Date: Wed, 25 Jul 2012 15:44:35 +0200 Subject: [erlang-questions] Memory-effective queue handling In-Reply-To: <500FE8AF.90308@gmail.com> References: <500FE8AF.90308@gmail.com> Message-ID: 2012/7/25 Richard Carlsson : [...] > You'd only run out of space if the queue is so long that you don't have > memory enough to call lists:reverse() on the current in-half of the queue > (and after that, the old queue will immediately become garbage). Can that > really be the case? It might happen. According to the crash dump the process in question uses Stack+heap: 228065700 words of memory. I store a tuple of bigint and a small integer in the queue, and the queue is essentially a list (actually two lists, but doesn't matter), so one entry in the queue costs 2+3(?)+1+1=7 words. The process has some other stuff in its state, but that should be negligible. The erlang VM run for less than 13 hours. If I calculate correctly, this means that there should be around 600 events per second though 13 hours to reach this amount of memory. After some internal conversation I think it is possible that due to an unrelated bug I do get this much events. > But you should be able to improve the behaviour of the above code by using > queue:peek_r(Q) instead of queue:out(Q), and then call queue:drop_r(Q) in > the second case clause only, so no rewriting of the queue representation is > done unless necessary. Thanks. From erlang@REDACTED Wed Jul 25 16:17:04 2012 From: erlang@REDACTED (Joe Armstrong) Date: Wed, 25 Jul 2012 16:17:04 +0200 Subject: [erlang-questions] How can I debug this? In-Reply-To: References: <500965A0.5030500@gmail.com> Message-ID: On Wed, Jul 25, 2012 at 2:14 PM, Dave Cottlehuber wrote: > On 25 July 2012 10:25, Joe Armstrong wrote: >> Better is to add it to the erlang startup file. >> >> On a unix or mac you just make a file called ".erlang" which is either >> on ${HOME}/.erlang >> or the directory you start erlang from. >> >> I tried to do this on windows but I have no evironment variable HOME >> and I do now have write permissions >> under Program Files (or whatever) >> >> (( Actually I'd like to keep an up-to-date install guide on the erlang >> web site so if anybody >> could tell me how to make a .erlang file on windows I would be appreciate this)) > > On Windows, the HOME env var trick doesn't work, you need to use the > current working directory of the werl/erl process instead. > @Joe what format (or where in OTP source) is best for documenting > this? I've got a few other things that would be useful too. Plain text is fine - I'll reformat if necessary - it's the content that's the important bit > > ## Creating a working .erlang file: > > - open notepad or , and paste this in, > or whatever you prefer: > > c:pwd(), io:format("running .erlang~n", []). Or io:format("running .erlang in ~p~n", [file:get_cwd()]). > > - file -> save as ".erlang" including the double quotes. This tells > the dialog box NOT to add .txt, just to use the provided filename. > > ## Create a shortcut to use this new startup file: > > - right-click on desktop > - create new shortcut > - enter path c:\erlang\bin\werl.exe (or where-ever you installed the > appropriate version) > - name it whatever & click finish > - right-click on new icon & select "properties" > - change "Start In" field to the directory you want to load your > .erlang file from > - click OK It worked - thanks > > ## Alternatively, from command line > > start /d c:\home c:\erlang\bin\werl.exe > > Or: > > - change directory to the same folder as your .erlang file > - run c:\erlang\bin\werl.exe (or erl.exe) > > BTW Ian's original issue is that Chicago Boss uses proper as a rebar > dependency, and proper uses some shell scripts which don't work under > native windows. This prevents Boss from compiling successfully > depending on what unix-like shell you are using, if any. Sigh - then we'll have to compile chicago boss manually (or fix the scripts) Manual compilation is something like .. $ cd ~/Downloads/ChicagoBoss-0.7.5/src/boss $ erlc .o ../../ebin -I ../../include *.erl You'll have to do this for each directory that has .erl files in it /Joe > cd > > A+ > Dave > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From carlsson.richard@REDACTED Wed Jul 25 16:18:06 2012 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Wed, 25 Jul 2012 16:18:06 +0200 Subject: [erlang-questions] Memory-effective queue handling In-Reply-To: References: <500FE8AF.90308@gmail.com> Message-ID: <5010001E.50401@gmail.com> On 07/25/2012 03:44 PM, Attila Rajmund Nohl wrote: > 2012/7/25 Richard Carlsson : > [...] >> You'd only run out of space if the queue is so long that you don't have >> memory enough to call lists:reverse() on the current in-half of the queue >> (and after that, the old queue will immediately become garbage). Can that >> really be the case? > > It might happen. According to the crash dump the process in question uses > Stack+heap: 228065700 > > words of memory. I store a tuple of bigint and a small integer in the > queue, and the queue is essentially a list (actually two lists, but > doesn't matter), so one entry in the queue costs 2+3(?)+1+1=7 words. > The process has some other stuff in its state, but that should be > negligible. The erlang VM run for less than 13 hours. If I calculate > correctly, this means that there should be around 600 events per > second though 13 hours to reach this amount of memory. After some > internal conversation I think it is possible that due to an unrelated > bug I do get this much events. Probably what you should use for a case like this is an ordered_set ets table, with the date as key and ets:last() to pick the oldest entry. The queue module is nondestructive and is nice in many ways, but it relies on amortized behaviour, and huge queues like yours make the intermittent periods of heavy internal work stick out like a sore thumb. The reversing of enormous lists isn't any good for responsiveness either, even if you had the memory. If you want to stick with a functional implementation, the gb_trees module will work pretty well, using gb_trees:smallest() to peek at the oldest entry. /Richard From ignatovs@REDACTED Wed Jul 25 17:22:25 2012 From: ignatovs@REDACTED (Sergey Ignatov) Date: Wed, 25 Jul 2012 19:22:25 +0400 Subject: [erlang-questions] Erlang plugin for IntelliJ IDEA Message-ID: Hi all, Today I'm happy to announce the pre-alpha version of Erlang support plugin for IntelliJ Platform based products, e.g. IntelliJ IDEA, RubyMine etc. At the moment there are such basic features: - Syntax and errors highlighting - References resolving - Code completion for functions, records and variables - Keyword code completion - Rename refactoring for modules, functions, records and variables - Safe delete refactoring - Structure view - Find usages - Code commenting/uncommenting - Brace matching - Basic code formatter Plugin page in IntelliJ repository: http://plugins.jetbrains.com/plugin/?pluginId=7083 Source code: https://github.com/ignatov/intellij-erlang If you are interesting in this pluging, feel free to use the issue tracker: https://github.com/ignatov/intellij-erlang/issues Stay tuned. Cheers, Sergey Ignatov From janburse@REDACTED Wed Jul 25 18:30:21 2012 From: janburse@REDACTED (Jan Burse) Date: Wed, 25 Jul 2012 18:30:21 +0200 Subject: [erlang-questions] Erlang plugin for IntelliJ IDEA In-Reply-To: References: Message-ID: <50101F1D.9050201@fastmail.fm> Sergey Ignatov schrieb: > Hi all, > > Today I'm happy to announce the pre-alpha version of Erlang support > plugin for IntelliJ Platform based products, e.g. IntelliJ IDEA, > RubyMine etc. > > At the moment there are such basic features: > - Syntax and errors highlighting > - References resolving > - Code completion for functions, records and variables > - Keyword code completion > - Rename refactoring for modules, functions, records and variables > - Safe delete refactoring > - Structure view > - Find usages > - Code commenting/uncommenting > - Brace matching > - Basic code formatter > > Plugin page in IntelliJ repository: > http://plugins.jetbrains.com/plugin/?pluginId=7083 > Source code: https://github.com/ignatov/intellij-erlang > > If you are interesting in this pluging, feel free to use the issue > tracker: https://github.com/ignatov/intellij-erlang/issues > > Stay tuned. > > Cheers, > Sergey Ignatov > Cool, what is the license of the plugin? From robert.virding@REDACTED Thu Jul 26 01:20:49 2012 From: robert.virding@REDACTED (Robert Virding) Date: Thu, 26 Jul 2012 00:20:49 +0100 (BST) Subject: [erlang-questions] [erlang-bugs] Recursive Heap Allocation In-Reply-To: <500FA828.3030104@gmail.com> Message-ID: There is limited flow-control. An io request is sent to an io-server, in this case the default io-server group.erl. The function called by the user waits for a reply from the io-server. The io-server processes the output call, in this case the formatted output, sends it to the port and then replies back to the calling function/process. It does not wait until the output has been sent all the way to user so it can definitely overload the io output. Robert ----- Original Message ----- > On 07/25/2012 09:36 AM, Joe Armstrong wrote: > > This program has similar behaviour > > > > -module(bug1). > > -compile(export_all). > > > > test(N) -> > > io:format("~p~n", [N]), > > test(N+1). > > > > I did this: > > > > 1 > spawn(fun() -> etop:start() end). > > 2> bug1:test(0). > > > > You can see that user_drv global process is accumulating messages > > faster than it can handle them. > > It's in usr_drv:io_request/3. > > > > As far as I'm aware io:format did have flow control so this > > shouldn't happen > > The I/O protocol has no built-in flow control. > > > Now I ran with "erl -smp disable" > > > > The counter stays at zero forever. > > With only a single scheduler, it will be alternating between running > the > producer process and the consumer processes, but they won't be > running > at the same time, so the consumer has a better chance of keeping up. > The > number of reductions used per message is the deciding factor, and if > I > recall correctly, there's a built-in penalty in reductions when you > pass > a message to a process that already has a long queue. > > With multiple schedulers, the producer and the consumer will run on > separate schedulers, i.e., they run on separate OS threads and > reduction > count doesn't matter, only actual time per message. The only thing > keeping them back a little is the synchronization needed for the > mailbox. It's not so surprising that it's easier to overload the > consumer in this case. (Also note that the I/O protocol causes the > formatting to be done on the receiver side, so if you're printing > complex terms, you're making the consumer do a lot more work per > message > than the producer.) > > /Richard > > _______________________________________________ > erlang-bugs mailing list > erlang-bugs@REDACTED > http://erlang.org/mailman/listinfo/erlang-bugs > From roberto@REDACTED Thu Jul 26 03:43:31 2012 From: roberto@REDACTED (Roberto Ostinelli) Date: Wed, 25 Jul 2012 18:43:31 -0700 Subject: [erlang-questions] Erlang plugin for IntelliJ IDEA In-Reply-To: References: Message-ID: may i ask you why you decided to start another plugin, instead of contributing to one of the existing projects that is? do you plan on supporting testings? that's the reason that made me start sublimerl. https://github.com/ostinelli/sublimerl r. On Wed, Jul 25, 2012 at 8:22 AM, Sergey Ignatov wrote: > Hi all, > > Today I'm happy to announce the pre-alpha version of Erlang support > plugin for IntelliJ Platform based products, e.g. IntelliJ IDEA, > RubyMine etc. > > At the moment there are such basic features: > - Syntax and errors highlighting > - References resolving > - Code completion for functions, records and variables > - Keyword code completion > - Rename refactoring for modules, functions, records and variables > - Safe delete refactoring > - Structure view > - Find usages > - Code commenting/uncommenting > - Brace matching > - Basic code formatter > > Plugin page in IntelliJ repository: > http://plugins.jetbrains.com/plugin/?pluginId=7083 > Source code: https://github.com/ignatov/intellij-erlang > > If you are interesting in this pluging, feel free to use the issue > tracker: https://github.com/ignatov/intellij-erlang/issues > > Stay tuned. > > Cheers, > Sergey Ignatov > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From meetprashant007@REDACTED Thu Jul 26 06:54:40 2012 From: meetprashant007@REDACTED (Prashant Sharma) Date: Thu, 26 Jul 2012 10:24:40 +0530 Subject: [erlang-questions] Erlang plugin for IntelliJ IDEA In-Reply-To: References: Message-ID: Robert , Do we have something of that sort for emacs? (code completion etc..) On Thu, Jul 26, 2012 at 7:13 AM, Roberto Ostinelli wrote: > may i ask you why you decided to start another plugin, instead of > contributing to one of the existing projects that is? > > do you plan on supporting testings? that's the reason that made me start > sublimerl. > https://github.com/ostinelli/sublimerl > > r. > > > On Wed, Jul 25, 2012 at 8:22 AM, Sergey Ignatov wrote: >> >> Hi all, >> >> Today I'm happy to announce the pre-alpha version of Erlang support >> plugin for IntelliJ Platform based products, e.g. IntelliJ IDEA, >> RubyMine etc. >> >> At the moment there are such basic features: >> - Syntax and errors highlighting >> - References resolving >> - Code completion for functions, records and variables >> - Keyword code completion >> - Rename refactoring for modules, functions, records and variables >> - Safe delete refactoring >> - Structure view >> - Find usages >> - Code commenting/uncommenting >> - Brace matching >> - Basic code formatter >> >> Plugin page in IntelliJ repository: >> http://plugins.jetbrains.com/plugin/?pluginId=7083 >> Source code: https://github.com/ignatov/intellij-erlang >> >> If you are interesting in this pluging, feel free to use the issue >> tracker: https://github.com/ignatov/intellij-erlang/issues >> >> Stay tuned. >> >> Cheers, >> Sergey Ignatov >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From watson.timothy@REDACTED Thu Jul 26 07:40:18 2012 From: watson.timothy@REDACTED (Tim Watson) Date: Thu, 26 Jul 2012 06:40:18 +0100 Subject: [erlang-questions] Erlang plugin for IntelliJ IDEA In-Reply-To: References: Message-ID: On 26 Jul 2012, at 05:54, Prashant Sharma wrote: > Robert , > > Do we have something of that sort for emacs? (code completion etc..) > Yes. Look at distel. It can be extended easily too. > > On Thu, Jul 26, 2012 at 7:13 AM, Roberto Ostinelli wrote: >> may i ask you why you decided to start another plugin, instead of >> contributing to one of the existing projects that is? >> >> do you plan on supporting testings? that's the reason that made me start >> sublimerl. >> https://github.com/ostinelli/sublimerl >> >> r. >> >> >> On Wed, Jul 25, 2012 at 8:22 AM, Sergey Ignatov wrote: >>> >>> Hi all, >>> >>> Today I'm happy to announce the pre-alpha version of Erlang support >>> plugin for IntelliJ Platform based products, e.g. IntelliJ IDEA, >>> RubyMine etc. >>> >>> At the moment there are such basic features: >>> - Syntax and errors highlighting >>> - References resolving >>> - Code completion for functions, records and variables >>> - Keyword code completion >>> - Rename refactoring for modules, functions, records and variables >>> - Safe delete refactoring >>> - Structure view >>> - Find usages >>> - Code commenting/uncommenting >>> - Brace matching >>> - Basic code formatter >>> >>> Plugin page in IntelliJ repository: >>> http://plugins.jetbrains.com/plugin/?pluginId=7083 >>> Source code: https://github.com/ignatov/intellij-erlang >>> >>> If you are interesting in this pluging, feel free to use the issue >>> tracker: https://github.com/ignatov/intellij-erlang/issues >>> >>> Stay tuned. >>> >>> Cheers, >>> Sergey Ignatov >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From watson.timothy@REDACTED Thu Jul 26 07:47:57 2012 From: watson.timothy@REDACTED (Tim Watson) Date: Thu, 26 Jul 2012 06:47:57 +0100 Subject: [erlang-questions] Erlang plugin for IntelliJ IDEA In-Reply-To: References: Message-ID: <16954DDF-7FB4-434F-BBDA-25241FDD8AF0@gmail.com> On 26 Jul 2012, at 02:43, Roberto Ostinelli wrote: > may i ask you why you decided to start another plugin, instead of contributing to one of the existing projects that is? > I suspect it was because the IntelliJ extension framework provides a very rich environment for binding, indexing and working with complex metadata, linking this to the editor and providing superlative navigation and refactoring capabilities. A lot of the donkey work is done for you already with IntelliJ. Personally what I would love to see is some standard Apis emerge for useful operations that all the editors require. This is probably the most realistic sharing of code between different environments being developed in parallel. Die hard emacs users are unlikely to move because they do everything in emacs. The same is likely true for vi users. Having a common framework for working with otp code and metadata based on syntax tools et al is probably the best way for all the IDE efforts to help one another. > do you plan on supporting testings? that's the reason that made me start sublimerl. > https://github.com/ostinelli/sublimerl > > r. > > > On Wed, Jul 25, 2012 at 8:22 AM, Sergey Ignatov wrote: > Hi all, > > Today I'm happy to announce the pre-alpha version of Erlang support > plugin for IntelliJ Platform based products, e.g. IntelliJ IDEA, > RubyMine etc. > > At the moment there are such basic features: > - Syntax and errors highlighting > - References resolving > - Code completion for functions, records and variables > - Keyword code completion > - Rename refactoring for modules, functions, records and variables > - Safe delete refactoring > - Structure view > - Find usages > - Code commenting/uncommenting > - Brace matching > - Basic code formatter > > Plugin page in IntelliJ repository: > http://plugins.jetbrains.com/plugin/?pluginId=7083 > Source code: https://github.com/ignatov/intellij-erlang > > If you are interesting in this pluging, feel free to use the issue > tracker: https://github.com/ignatov/intellij-erlang/issues > > Stay tuned. > > Cheers, > Sergey Ignatov > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From damienuk@REDACTED Thu Jul 26 08:10:47 2012 From: damienuk@REDACTED (Damienuk Davis) Date: Wed, 25 Jul 2012 23:10:47 -0700 (PDT) Subject: [erlang-questions] run_erl and fsync Message-ID: <1343283047.62266.YahooMailNeo@web124906.mail.ne1.yahoo.com> Hi all, In one of my projects I used run_erl to launch Erlang VM in daemon mode, rotate logs, etc. But I discovered unexpected performance problem running production application with run_erl. Application caused high iowait. After some investigation I found that run_erl does fsync on every log entry, causing disk IO load. Do we need fsync on every log entry. Appreciate your valuable comments. Thanks Damien -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto@REDACTED Thu Jul 26 08:32:32 2012 From: roberto@REDACTED (Roberto Ostinelli) Date: Wed, 25 Jul 2012 23:32:32 -0700 Subject: [erlang-questions] Erlang plugin for IntelliJ IDEA In-Reply-To: <16954DDF-7FB4-434F-BBDA-25241FDD8AF0@gmail.com> References: <16954DDF-7FB4-434F-BBDA-25241FDD8AF0@gmail.com> Message-ID: Right now, this plugin is still providing basic editor functionalities, and there are many out there doing the same thing. Don't get me wrong, though. I do like IntelliJ: I use RubyMine on a daily basis and I'd love to see the same thing for Erlang. But what I really need is the ability to run *tests* in a seamless manner as in RubyMine. This is why I've implemented SublimErl. Therefore my question: does the author plan to support testing in the way RubyMine is? That';s something I'd be interested in hearing about. r. On Wed, Jul 25, 2012 at 10:47 PM, Tim Watson wrote: > > On 26 Jul 2012, at 02:43, Roberto Ostinelli wrote: > > may i ask you why you decided to start another plugin, instead of > contributing to one of the existing projects that is? > > > I suspect it was because the IntelliJ extension framework provides a very > rich environment for binding, indexing and working with complex metadata, > linking this to the editor and providing superlative navigation and > refactoring capabilities. A lot of the donkey work is done for you already > with IntelliJ. > > Personally what I would love to see is some standard Apis emerge for > useful operations that all the editors require. This is probably the most > realistic sharing of code between different environments being developed in > parallel. > > Die hard emacs users are unlikely to move because they do everything in > emacs. The same is likely true for vi users. Having a common framework for > working with otp code and metadata based on syntax tools et al is probably > the best way for all the IDE efforts to help one another. > > do you plan on supporting testings? that's the reason that made me start > sublimerl. > https://github.com/ostinelli/sublimerl > > r. > > > On Wed, Jul 25, 2012 at 8:22 AM, Sergey Ignatov wrote: > >> Hi all, >> >> Today I'm happy to announce the pre-alpha version of Erlang support >> plugin for IntelliJ Platform based products, e.g. IntelliJ IDEA, >> RubyMine etc. >> >> At the moment there are such basic features: >> - Syntax and errors highlighting >> - References resolving >> - Code completion for functions, records and variables >> - Keyword code completion >> - Rename refactoring for modules, functions, records and variables >> - Safe delete refactoring >> - Structure view >> - Find usages >> - Code commenting/uncommenting >> - Brace matching >> - Basic code formatter >> >> Plugin page in IntelliJ repository: >> http://plugins.jetbrains.com/plugin/?pluginId=7083 >> Source code: https://github.com/ignatov/intellij-erlang >> >> If you are interesting in this pluging, feel free to use the issue >> tracker: https://github.com/ignatov/intellij-erlang/issues >> >> Stay tuned. >> >> Cheers, >> Sergey Ignatov >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrzej.sliwa@REDACTED Thu Jul 26 08:47:05 2012 From: andrzej.sliwa@REDACTED (Andrzej Sliwa) Date: Thu, 26 Jul 2012 08:47:05 +0200 Subject: [erlang-questions] Erlang plugin for IntelliJ IDEA In-Reply-To: References: Message-ID: maybe because not all of us like sublime text? maybe because some of use using rubymine or intellij and wants to have only one tool? c'mon could be a 1000 good reasons same question question for you? why you decide to make own one for sublime instead improve existing already erlang-mode with distel in emacs :P On Jul 26, 2012, at 3:43 AM, Roberto Ostinelli wrote: > may i ask you why you decided to start another plugin, instead of contributing to one of the existing projects that is? > > do you plan on supporting testings? that's the reason that made me start sublimerl. > https://github.com/ostinelli/sublimerl > > r. > > > On Wed, Jul 25, 2012 at 8:22 AM, Sergey Ignatov wrote: > Hi all, > > Today I'm happy to announce the pre-alpha version of Erlang support > plugin for IntelliJ Platform based products, e.g. IntelliJ IDEA, > RubyMine etc. > > At the moment there are such basic features: > - Syntax and errors highlighting > - References resolving > - Code completion for functions, records and variables > - Keyword code completion > - Rename refactoring for modules, functions, records and variables > - Safe delete refactoring > - Structure view > - Find usages > - Code commenting/uncommenting > - Brace matching > - Basic code formatter > > Plugin page in IntelliJ repository: > http://plugins.jetbrains.com/plugin/?pluginId=7083 > Source code: https://github.com/ignatov/intellij-erlang > > If you are interesting in this pluging, feel free to use the issue > tracker: https://github.com/ignatov/intellij-erlang/issues > > Stay tuned. > > Cheers, > Sergey Ignatov > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto@REDACTED Thu Jul 26 09:19:55 2012 From: roberto@REDACTED (Roberto Ostinelli) Date: Thu, 26 Jul 2012 00:19:55 -0700 Subject: [erlang-questions] Erlang plugin for IntelliJ IDEA In-Reply-To: References: Message-ID: > > same question question for you? > why you decide to make own one for sublime instead improve existing > already erlang-mode with distel in emacs :P > because i haven't found a single editor where on a single key press you can *run tests* within the editor itself. i said this three times already. 1000 reasons don't make up for concentrating open source efforts on fewer projects and have a single *good* one. same story, different repos (yes, i'm talking about misultin & co). r. -------------- next part -------------- An HTML attachment was scrubbed... URL: From watson.timothy@REDACTED Thu Jul 26 09:23:10 2012 From: watson.timothy@REDACTED (Tim Watson) Date: Thu, 26 Jul 2012 08:23:10 +0100 Subject: [erlang-questions] Erlang plugin for IntelliJ IDEA In-Reply-To: References: <16954DDF-7FB4-434F-BBDA-25241FDD8AF0@gmail.com> Message-ID: On 26 Jul 2012, at 07:32, Roberto Ostinelli wrote: > Right now, this plugin is still providing basic editor functionalities, and there are many out there doing the same thing. > > Don't get me wrong, though. I do like IntelliJ: I use RubyMine on a daily basis and I'd love to see the same thing for Erlang. But what I really need is the ability to run *tests* in a seamless manner as in RubyMine. This is why I've implemented SublimErl. > > Therefore my question: does the author plan to support testing in the way RubyMine is? That';s something I'd be interested in hearing about. > Yeah be too. I might even chip in and contribute to the plugin if the author's amenable. > r. > > On Wed, Jul 25, 2012 at 10:47 PM, Tim Watson wrote: > > On 26 Jul 2012, at 02:43, Roberto Ostinelli wrote: > >> may i ask you why you decided to start another plugin, instead of contributing to one of the existing projects that is? >> > > I suspect it was because the IntelliJ extension framework provides a very rich environment for binding, indexing and working with complex metadata, linking this to the editor and providing superlative navigation and refactoring capabilities. A lot of the donkey work is done for you already with IntelliJ. > > Personally what I would love to see is some standard Apis emerge for useful operations that all the editors require. This is probably the most realistic sharing of code between different environments being developed in parallel. > > Die hard emacs users are unlikely to move because they do everything in emacs. The same is likely true for vi users. Having a common framework for working with otp code and metadata based on syntax tools et al is probably the best way for all the IDE efforts to help one another. > >> do you plan on supporting testings? that's the reason that made me start sublimerl. >> https://github.com/ostinelli/sublimerl >> >> r. >> >> >> On Wed, Jul 25, 2012 at 8:22 AM, Sergey Ignatov wrote: >> Hi all, >> >> Today I'm happy to announce the pre-alpha version of Erlang support >> plugin for IntelliJ Platform based products, e.g. IntelliJ IDEA, >> RubyMine etc. >> >> At the moment there are such basic features: >> - Syntax and errors highlighting >> - References resolving >> - Code completion for functions, records and variables >> - Keyword code completion >> - Rename refactoring for modules, functions, records and variables >> - Safe delete refactoring >> - Structure view >> - Find usages >> - Code commenting/uncommenting >> - Brace matching >> - Basic code formatter >> >> Plugin page in IntelliJ repository: >> http://plugins.jetbrains.com/plugin/?pluginId=7083 >> Source code: https://github.com/ignatov/intellij-erlang >> >> If you are interesting in this pluging, feel free to use the issue >> tracker: https://github.com/ignatov/intellij-erlang/issues >> >> Stay tuned. >> >> Cheers, >> Sergey Ignatov >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eriksoe@REDACTED Thu Jul 26 10:10:33 2012 From: eriksoe@REDACTED (=?ISO-8859-1?Q?Erik_S=F8e_S=F8rensen?=) Date: Thu, 26 Jul 2012 10:10:33 +0200 Subject: [erlang-questions] Memory-effective queue handling In-Reply-To: <500FF2BC.7010801@gmail.com> References: <500FF2BC.7010801@gmail.com> Message-ID: Den 25/07/2012 15.21 skrev "Richard Carlsson" : > > On 07/25/2012 03:01 PM, Erik S?e S?rensen wrote: >> >> > increase_counter(#ch{queue=Q, counter_value=CV}=CH, Value) -> >> > Date = ?MODULE:get_date(), >> > NewQueue=queue:in({Date, Value}, Q), >> > CH#ch{queue=NewQueue, counter_value=CV+Value}. >> > >> > get_value(#ch{hist_len=HL}=CH) -> >> > % get the values from the queue until we reach a first element in >> the queue >> > % which is not older then the hist_len (e.g. not older than 15 mins) >> > Date = ?MODULE:get_date(), >> > get_younger_than(CH, Date-HL). >> > >> > get_younger_than(#ch{queue=Q, counter_value=CV}=CH, TargetDate) -> >> > case queue:out(Q) of >> > {empty, _} -> >> > 0 = CV, % this is an assert, the value shall be 0 >> > {CH, 0}; >> > {{value, {Date, Value}}, NewQ} when Date < TargetDate -> >> > get_younger_than(CH#ch{queue=NewQ, >> counter_value=CV-Value}, TargetDate); >> > {{value, _}, _} -> >> > % the currently removed entry is too young, so leave that >> in the >> > % queue (i.e. use the parameter queue). >> > {CH, CV} >> > end. >> > >> > The copying is done in the queue:out call, because it returns not just >> > the first value, but the rest of the queu (newQ) too. >> That doesn't copy the entire queue. That is, occasionally it recreates >> the *spine* of a new list the length of the entire queue, but normally >> it does way less than that. >> The problem is that the way your code works, that worst-case is every >> time, and the throwing-away of old entries is also done from the start >> each time: your get_value() does not return the pruned queue. >> A further consequence of that is that Richard's statement doesn't hold: >> because you keep a reference (apparently) to the original, unpruned >> queue, the memory will have to hold both list spines simultaneously. >> So I'd suggest you change the return value of get_value. > > > I might be a bit tired, but where does he still hold a reference to the original queue (unless somewhere outside the shown code)? And I may be assuming too much by thinking that get_value is called by a server loop which keeps holding on to the queue. Sorry I wasn't explicit about that; if it wasn't so, then what I said didn't make much sense. (It's vacation time and email-by-phone time, so my editing facilities are a bit limited...) > And if so, is the problem really that he can't hold 2 copies of the queue at the same time? - In that case he's pushing the limits anyway. As far as I can see, get_value() tail calls to get_younger_than(), so no references are kept there, and get_younger_than() keeps tailcalling itself with the updated queue in its #ch-record, dropping the previous queue. The final #ch-record gets returned to the caller. Only the result of the last call to queue:out() does any wasted rewriting. > > /Richard > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ignatovs@REDACTED Thu Jul 26 10:26:17 2012 From: ignatovs@REDACTED (Sergey Ignatov) Date: Thu, 26 Jul 2012 12:26:17 +0400 Subject: [erlang-questions] Erlang plugin for IntelliJ IDEA In-Reply-To: References: Message-ID: Hi guys, Thanks for discussion. Yesterday's release had the main goal: to get feedback from users. So, I got it. Tim said very good words about tools development: > I suspect it was because the IntelliJ extension framework provides a very rich environment for binding, indexing and working with complex metadata, linking this to the editor and providing superlative navigation and refactoring capabilities. A lot of the donkey work is done for you already with IntelliJ. > Personally what I would love to see is some standard Apis emerge for useful operations that all the editors require. This is probably the most realistic sharing of code between different environments being developed in parallel. > Die hard emacs users are unlikely to move because they do everything in emacs. The same is likely true for vi users. Having a common framework for working with otp code and metadata based on syntax tools et al is probably the best way for all the IDE efforts to help one another. I started Erlang plugin for IntelliJ because it's much easy to create a good language support than in another platform. And sure, I prefer IntelliJ for product development. Roberto, I probably got your point: provides a unit tests runner like IndelliJ and RubyMine does, right? Some words about future plans: I would like to get feedback from users to issue tracker, sort issues by priority and begin to implement them as far as possible. But I'm limited in my time/resources because this plugin is my pet project, not a full time job. Possibly, I'll develop Erlang plugin as `20% project` at my work. Once again, If you are interested in my project, please create issues with bugs or feature requests in the issue tracker: https://github.com/ignatov/intellij-erlang/issues I beleive that plugin can grow into a full-fledged Erlang language support. Sergey Ignatov On 26 July 2012 10:47, Andrzej Sliwa wrote: > maybe because not all of us like sublime text? > maybe because some of use using rubymine or intellij and wants to have only > one tool? > > c'mon could be a 1000 good reasons > > same question question for you? > why you decide to make own one for sublime instead improve existing already > erlang-mode with distel in emacs :P > > On Jul 26, 2012, at 3:43 AM, Roberto Ostinelli wrote: > > may i ask you why you decided to start another plugin, instead of > contributing to one of the existing projects that is? > > do you plan on supporting testings? that's the reason that made me start > sublimerl. > https://github.com/ostinelli/sublimerl > > r. > > > On Wed, Jul 25, 2012 at 8:22 AM, Sergey Ignatov wrote: >> >> Hi all, >> >> Today I'm happy to announce the pre-alpha version of Erlang support >> plugin for IntelliJ Platform based products, e.g. IntelliJ IDEA, >> RubyMine etc. >> >> At the moment there are such basic features: >> - Syntax and errors highlighting >> - References resolving >> - Code completion for functions, records and variables >> - Keyword code completion >> - Rename refactoring for modules, functions, records and variables >> - Safe delete refactoring >> - Structure view >> - Find usages >> - Code commenting/uncommenting >> - Brace matching >> - Basic code formatter >> >> Plugin page in IntelliJ repository: >> http://plugins.jetbrains.com/plugin/?pluginId=7083 >> Source code: https://github.com/ignatov/intellij-erlang >> >> If you are interesting in this pluging, feel free to use the issue >> tracker: https://github.com/ignatov/intellij-erlang/issues >> >> Stay tuned. >> >> Cheers, >> Sergey Ignatov >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > From russelldb@REDACTED Thu Jul 26 10:54:45 2012 From: russelldb@REDACTED (Russell Brown) Date: Thu, 26 Jul 2012 09:54:45 +0100 Subject: [erlang-questions] Included applications in releases : differences between r14 and r15 Message-ID: Hi, I have an application that wants to supervise another application. Looking at http://www.erlang.org/doc/design_principles/included_applications.html it seemed that an included application was the way to go. I added the included app like this {included_applications, [the_app]}, in the primary applications .app file. When I build a release with rebar under r15 the boot script does _NOT_ start the included app, as hoped and expected. However, the exact same code and config on r14 results in this line appearing in the boost script {apply,{application,start_boot,[the_app,permanent]}}, Which means the app is started on boot, and my subsequent call to start the top level supervisor of the included app fails with `already_started`. What is the correct way to include an app so that it is loaded, but not started, in r14, so that I may have my primary app call the top level supervisor of the included application, please? Also, in Learn You some Erlang, there is a section that says "don't do this" (http://learnyousomeerlang.com/the-count-of-applications#included-applications.) Is there a better way to achieve what I am after? Cheers Russell -------------- next part -------------- An HTML attachment was scrubbed... URL: From russelldb@REDACTED Thu Jul 26 11:25:47 2012 From: russelldb@REDACTED (Russell Brown) Date: Thu, 26 Jul 2012 10:25:47 +0100 Subject: [erlang-questions] Included applications in releases : differences between r14 and r15 In-Reply-To: References: Message-ID: <2907DDFD-3445-45EF-8A50-B38DD63B18BB@basho.com> Going to answer my own question here, sorry. {my_app, load} in the rel_app list in the `rel` tuple in my retool.config fixes the problem. Is that the correct way? Cheers Russell On 26 Jul 2012, at 09:54, Russell Brown wrote: > Hi, > I have an application that wants to supervise another application. Looking at http://www.erlang.org/doc/design_principles/included_applications.html it seemed that an included application was the way to go. > > I added the included app like this > > {included_applications, [the_app]}, > > in the primary applications .app file. > > When I build a release with rebar under r15 the boot script does _NOT_ start the included app, as hoped and expected. However, the exact same code and config on r14 results in this line appearing in the boost script > > {apply,{application,start_boot,[the_app,permanent]}}, > > Which means the app is started on boot, and my subsequent call to start the top level supervisor of the included app fails with `already_started`. > > What is the correct way to include an app so that it is loaded, but not started, in r14, so that I may have my primary app call the top level supervisor of the included application, please? > > Also, in Learn You some Erlang, there is a section that says "don't do this" (http://learnyousomeerlang.com/the-count-of-applications#included-applications.) Is there a better way to achieve what I am after? > > Cheers > > Russell > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From msegalis@REDACTED Thu Jul 26 11:51:48 2012 From: msegalis@REDACTED (Morgan Segalis) Date: Thu, 26 Jul 2012 11:51:48 +0200 Subject: [erlang-questions] SSL Server Bad record mac error In-Reply-To: <127222070.301766.1343211273642.JavaMail.root@tpip.net> References: <127222070.301766.1343211273642.JavaMail.root@tpip.net> Message-ID: <151B50BA-6F6E-4EA9-9DC4-ED7A6606D96B@gmail.com> Hi Andreas, thanks for the answer, I have cooked up a little info function, that gives me some info about processes in my Erlang server : Here's what it gives me on the server dedicated to blackberries (The one that gives mac errors) {{[[{{gen_server_pool_proxy,init,1}, [<0.74.0>,<0.73.0>,<0.41.0>]}| 1]], [],[],[], [[undefined|6]], [], [[{{ssl_connection,init,1}, [ssl_connection_sup,ssl_sup,<0.55.0>]}| 38770]], [],[],[], [[{unknown,unknown}|2]], [[{{supervisor,muserver_sup,1},[muserver,<0.41.0>]}|1]], [],[],[], [[{{supervisor,muserver,1},[<0.41.0>]}|1], [{{muserver_serv,init,1},[muserver_sup,muserver,<0.41.0>]}|26556]]}}} The muserver_serv gives me the exact number of connected clients 26556 (cross-checked with a netstat) However I do get way more ssl_connection_sup than I should 38770. That gives me 12214 ssl processes that should not exist, if as you said, the connection should be closing cleanly. On servers that do not get mac errors, the SSL and TCP number of processes are roughly the same with more or less 10 processes. Best regards, Morgan. Le 25 juil. 2012 ? 12:14, Andreas Schultz a ?crit : > Hi Morgan, > > The connection should be closing cleanly already. The alert > should be sent by the ssl state machine over the ssl connection > and both sides will close. > > To diagnose this further, you could try to capture the broken > ssl session in a controlled setup. The selected crypto suite > is in the unecrypted part of the handshake. > If you disable all EDH ciphers, wireshark or ssldump can be > used to decipher the encrypted part of the exchange. That > should give you a hint what excatly is going on. > > Andreas > > ----- Original Message ----- >> >> Hi everyone, >> >> >> I'm currently trying to find out, where could come from this error : >> >> >> >> =ERROR REPORT==== 23-Jul-2012::11:03:56 === >> SSL: certify: ssl_record.erl:654:Fatal error: bad record mac >> >> >> The SSL function where it is coming from is decipher: >> >> >> ---------------------------------- ssl_record.erl >> --------------------------------------------- >> >> >> >> decipher(TLS=#ssl_tls{type=Type, version=Version, fragment=Fragment}, >> CS0) -> >> SP = CS0#connection_state.security_parameters, >> BCA = SP#security_parameters.bulk_cipher_algorithm, >> HashSz = SP#security_parameters.hash_size, >> CipherS0 = CS0#connection_state.cipher_state, >> case ssl_cipher:decipher(BCA, HashSz, CipherS0, Fragment, Version) of >> {T, Mac, CipherS1} -> >> CS1 = CS0#connection_state{cipher_state = CipherS1}, >> TLength = size(T), >> {MacHash, CS2} = hash_and_bump_seqno(CS1, Type, Version, TLength, T), >> case is_correct_mac(Mac, MacHash) of >> true -> >> {TLS#ssl_tls{fragment = T}, CS2}; >> false -> >> ?ALERT_REC(?FATAL, ?BAD_RECORD_MAC) %%<-------- HERE >> end; >> #alert{} = Alert -> >> Alert >> end. >> ---------------------------------------------------------------------------------------------------- >> >> >> I'm not really sure if this error is happening at handshake, or read. >> The thing is, the error is only coming only from users connecting >> from a BlackBerry device (Can't know which OS version, I suppose it >> comes from rather old devices). >> >> >> If there is nothing I can't do server-side, does anyone could point >> out how to handle the error, in order to close cleanly the >> connection ? >> >> >> Regards, >> >> >> Morgan. >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > -- > -- > Dipl. Inform. > Andreas Schultz > > email: as@REDACTED > phone: +49-391-819099-224 > mobil: +49-170-2226073 > > ------------------ managed broadband access ------------------ > > Travelping GmbH phone: +49-391-8190990 > Roentgenstr. 13 fax: +49-391-819099299 > D-39108 Magdeburg email: info@REDACTED > GERMANY web: http://www.travelping.com > > Company Registration: HRB21276 Handelsregistergericht Chemnitz > Geschaeftsfuehrer: Holger Winkelmann | VAT ID No.: DE236673780 > -------------------------------------------------------------- From dmkolesnikov@REDACTED Thu Jul 26 14:14:50 2012 From: dmkolesnikov@REDACTED (dmitry kolesnikov) Date: Thu, 26 Jul 2012 15:14:50 +0300 Subject: [erlang-questions] run_erl and fsync In-Reply-To: <1343283047.62266.YahooMailNeo@web124906.mail.ne1.yahoo.com> References: <1343283047.62266.YahooMailNeo@web124906.mail.ne1.yahoo.com> Message-ID: <-3577860171847931767@unknownmsgid> Does fsync block any VM scheduler? Or is it executed within async thread? Or none of them :-) I do not think that flush log is required for majority of Web systems. I believe this is a TeleCo legacy... Should it be configurable at least build time? Best Regards, Dmitry >-|-|-*> On 26.7.2012, at 9.10, Damienuk Davis wrote: Hi all, In one of my projects I used run_erl to launch Erlang VM in daemon mode, rotate logs, etc. But I discovered unexpected performance problem running production application with run_erl. Application caused high iowait. After some investigation I found that run_erl does fsync on every log entry, causing disk IO load. Do we need fsync on every log entry. Appreciate your valuable comments. Thanks Damien _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From damienuk@REDACTED Thu Jul 26 14:36:35 2012 From: damienuk@REDACTED (Damienuk Davis) Date: Thu, 26 Jul 2012 05:36:35 -0700 (PDT) Subject: [erlang-questions] run_erl and fsync In-Reply-To: <-3577860171847931767@unknownmsgid> References: <1343283047.62266.YahooMailNeo@web124906.mail.ne1.yahoo.com> <-3577860171847931767@unknownmsgid> Message-ID: <1343306195.69768.YahooMailNeo@web124905.mail.ne1.yahoo.com> In my project there are fsms that communicates with c via erl driver (port). There are io:fwrite's in the erlang side. This program along works fine. But when you run some other process with high disk access, it causes erlang program to halt even if there is?enough memory and processing. If I comment the fsync is run_erl.c file then program works fine even with higher load on?hard disk.? static void write_to_log(int* lfd, int* log_num, char* buf, int len) { ? int size; ? /* Decide if new logfile needed, and open if so */ ? size = lseek(*lfd,0,SEEK_END); ? if(size+len > log_maxsize) { ? ? close(*lfd); ? ? *log_num = next_log(*log_num); ? ? *lfd = open_log(*log_num, O_RDWR|O_CREAT|O_TRUNC|O_SYNC); ? } ? /* Write to log file */ ? if (write_all(*lfd, buf, len) < 0) { ? ? status("Error in writing to log.\n"); ? } #if USE_FSYNC ? fsync(*lfd);//after commenting this line it works fine? #endif } > > > >Thanks >Damien ________________________________ From: dmitry kolesnikov To: Damienuk Davis Cc: "erlang-questions@REDACTED" Sent: Thursday, July 26, 2012 5:44 PM Subject: Re: [erlang-questions] run_erl and fsync Does fsync block any VM scheduler? Or is it executed within async thread? Or none of them :-) I do not think that flush log is required for majority of Web systems. I believe this is a TeleCo legacy... Should it be configurable at least build time? Best Regards, Dmitry >-|-|-*> On 26.7.2012, at 9.10, Damienuk Davis wrote: Hi all, > > >In one of my projects I used run_erl to launch Erlang VM in daemon mode, rotate logs, etc. >But I discovered unexpected performance problem running production application with run_erl. Application caused high iowait. >After some investigation I found that run_erl does fsync on every log entry, causing disk IO load. Do we need fsync on every log entry. >Appreciate your valuable comments. > > >Thanks >Damien _______________________________________________ >erlang-questions mailing list >erlang-questions@REDACTED >http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From f@REDACTED Thu Jul 26 16:50:20 2012 From: f@REDACTED (Francesco Mazzoli) Date: Thu, 26 Jul 2012 15:50:20 +0100 Subject: [erlang-questions] Erlang plugin for IntelliJ IDEA In-Reply-To: References: Message-ID: <87ipdabmqb.wl%f@mazzo.li> At Thu, 26 Jul 2012 00:19:55 -0700, Roberto Ostinelli wrote: > because i haven't found a single editor where on a single key press you can > *run tests* within the editor itself. i said this three times already. You can do that in any decent extensible editor, given enough effort. And it's probably really easy in emacs, given that you already have distel. There are far more complex programs written in elisp that executing some commands to run some tests. -- Francesco * Often in error, never in doubt From roberto@REDACTED Thu Jul 26 17:04:43 2012 From: roberto@REDACTED (Roberto Ostinelli) Date: Thu, 26 Jul 2012 08:04:43 -0700 Subject: [erlang-questions] Erlang plugin for IntelliJ IDEA In-Reply-To: <87ipdabmqb.wl%f@mazzo.li> References: <87ipdabmqb.wl%f@mazzo.li> Message-ID: On Thu, Jul 26, 2012 at 7:50 AM, Francesco Mazzoli wrote: > At Thu, 26 Jul 2012 00:19:55 -0700, > Roberto Ostinelli wrote: > > because i haven't found a single editor where on a single key press you > can > > *run tests* within the editor itself. i said this three times already. > > You can do that in any decent extensible editor, given enough effort. And > it's > probably really easy in emacs, given that you already have distel. There > are > far more complex programs written in elisp that executing some commands to > run > some tests. > let me summarize this: 1. no 'key press' test support *existed* in *any* editor, out of the box 2. i chose one and added it. you are telling me that "you can do that in any decent extensible editor, given enough effort". what is your point exactly? r. -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto@REDACTED Thu Jul 26 17:05:18 2012 From: roberto@REDACTED (Roberto Ostinelli) Date: Thu, 26 Jul 2012 08:05:18 -0700 Subject: [erlang-questions] Erlang plugin for IntelliJ IDEA In-Reply-To: References: Message-ID: > Roberto, I probably got your point: provides a unit tests runner like > IndelliJ and RubyMine does, right? indeed ^^_ r. -------------- next part -------------- An HTML attachment was scrubbed... URL: From f@REDACTED Thu Jul 26 17:14:19 2012 From: f@REDACTED (Francesco Mazzoli) Date: Thu, 26 Jul 2012 16:14:19 +0100 Subject: [erlang-questions] Erlang plugin for IntelliJ IDEA In-Reply-To: References: <87ipdabmqb.wl%f@mazzo.li> Message-ID: <87hasublmc.wl%f@mazzo.li> At Thu, 26 Jul 2012 08:04:43 -0700, Roberto Ostinelli wrote: > 1. no 'key press' test support *existed* in *any* editor, out of the box The key press part is trivial - once you have the functionality in place. But that's not the point. > 2. i chose one and added it. > > you are telling me that "you can do that in any decent extensible > editor,?given enough effort". > > what is your point exactly? You asked the author of the IntelliJ plugin why he would do that and not contribute to existing projects. The same can be asked to you, since you decided to extend SublimeText while a lot of groundwork was already there for emacs (which is a far more widespread editor, and more importantly a free one). -- Francesco * Often in error, never in doubt From dmkolesnikov@REDACTED Thu Jul 26 17:18:47 2012 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Thu, 26 Jul 2012 18:18:47 +0300 Subject: [erlang-questions] run_erl and fsync In-Reply-To: <1343306195.69768.YahooMailNeo@web124905.mail.ne1.yahoo.com> References: <1343283047.62266.YahooMailNeo@web124906.mail.ne1.yahoo.com> <-3577860171847931767@unknownmsgid> <1343306195.69768.YahooMailNeo@web124905.mail.ne1.yahoo.com> Message-ID: I am curious what platform you are build the staff and what is the config? fsync is behind USE_FSYNC but #ifndef O_SYNC #define O_SYNC 0 #define USE_FSYNC 1 #endif - Dmitry On Jul 26, 2012, at 3:36 PM, Damienuk Davis wrote: >> In my project there are fsms that communicates with c via erl driver (port). There are io:fwrite's in the erlang side. This program along works fine. But when you run some other process with high disk access, it causes erlang program to halt even if there is enough memory and processing. If I comment the fsync is run_erl.c file then program works fine even with higher load on hard disk. > static void write_to_log(int* lfd, int* log_num, char* buf, int len) > { > int size; > > /* Decide if new logfile needed, and open if so */ > > size = lseek(*lfd,0,SEEK_END); > if(size+len > log_maxsize) { > close(*lfd); > *log_num = next_log(*log_num); > *lfd = open_log(*log_num, O_RDWR|O_CREAT|O_TRUNC|O_SYNC); > } > > /* Write to log file */ > > if (write_all(*lfd, buf, len) < 0) { > status("Error in writing to log.\n"); > } > > #if USE_FSYNC > fsync(*lfd); //after commenting this line it works fine > #endif > } >> >> >> Thanks >> Damien > > > From: dmitry kolesnikov > To: Damienuk Davis > Cc: "erlang-questions@REDACTED" > Sent: Thursday, July 26, 2012 5:44 PM > Subject: Re: [erlang-questions] run_erl and fsync > > Does fsync block any VM scheduler? Or is it executed within async thread? > Or none of them :-) > > I do not think that flush log is required for majority of Web systems. I believe this is a TeleCo legacy... Should it be configurable at least build time? > > Best Regards, > Dmitry >-|-|-*> > > > On 26.7.2012, at 9.10, Damienuk Davis wrote: > >> Hi all, >> >> In one of my projects I used run_erl to launch Erlang VM in daemon mode, rotate logs, etc. >> But I discovered unexpected performance problem running production application with run_erl. Application caused high iowait. >> After some investigation I found that run_erl does fsync on every log entry, causing disk IO load. Do we need fsync on every log entry. >> Appreciate your valuable comments. >> >> Thanks >> Damien >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From watson.timothy@REDACTED Thu Jul 26 17:21:03 2012 From: watson.timothy@REDACTED (Tim Watson) Date: Thu, 26 Jul 2012 16:21:03 +0100 Subject: [erlang-questions] Erlang plugin for IntelliJ IDEA In-Reply-To: References: Message-ID: <119DF4D4-6C35-4DDD-9345-B8BA44F48DFA@gmail.com> On 26 Jul 2012, at 09:26, Sergey Ignatov wrote: > > Once again, If you are interested in my project, please create issues > with bugs or feature requests in the issue tracker: > https://github.com/ignatov/intellij-erlang/issues > > I beleive that plugin can grow into a full-fledged Erlang language support. > Will you take pull requests? > Sergey Ignatov > > > On 26 July 2012 10:47, Andrzej Sliwa wrote: >> maybe because not all of us like sublime text? >> maybe because some of use using rubymine or intellij and wants to have only >> one tool? >> >> c'mon could be a 1000 good reasons >> >> same question question for you? >> why you decide to make own one for sublime instead improve existing already >> erlang-mode with distel in emacs :P >> >> On Jul 26, 2012, at 3:43 AM, Roberto Ostinelli wrote: >> >> may i ask you why you decided to start another plugin, instead of >> contributing to one of the existing projects that is? >> >> do you plan on supporting testings? that's the reason that made me start >> sublimerl. >> https://github.com/ostinelli/sublimerl >> >> r. >> >> >> On Wed, Jul 25, 2012 at 8:22 AM, Sergey Ignatov wrote: >>> >>> Hi all, >>> >>> Today I'm happy to announce the pre-alpha version of Erlang support >>> plugin for IntelliJ Platform based products, e.g. IntelliJ IDEA, >>> RubyMine etc. >>> >>> At the moment there are such basic features: >>> - Syntax and errors highlighting >>> - References resolving >>> - Code completion for functions, records and variables >>> - Keyword code completion >>> - Rename refactoring for modules, functions, records and variables >>> - Safe delete refactoring >>> - Structure view >>> - Find usages >>> - Code commenting/uncommenting >>> - Brace matching >>> - Basic code formatter >>> >>> Plugin page in IntelliJ repository: >>> http://plugins.jetbrains.com/plugin/?pluginId=7083 >>> Source code: https://github.com/ignatov/intellij-erlang >>> >>> If you are interesting in this pluging, feel free to use the issue >>> tracker: https://github.com/ignatov/intellij-erlang/issues >>> >>> Stay tuned. >>> >>> Cheers, >>> Sergey Ignatov >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> From ignatovs@REDACTED Thu Jul 26 17:43:55 2012 From: ignatovs@REDACTED (Sergey Ignatov) Date: Thu, 26 Jul 2012 19:43:55 +0400 Subject: [erlang-questions] Erlang plugin for IntelliJ IDEA In-Reply-To: <119DF4D4-6C35-4DDD-9345-B8BA44F48DFA@gmail.com> References: <119DF4D4-6C35-4DDD-9345-B8BA44F48DFA@gmail.com> Message-ID: On 26 July 2012 19:21, Tim Watson wrote: > Will you take pull requests? Sure, I will. Sergey Ignatov From roberto@REDACTED Thu Jul 26 18:16:05 2012 From: roberto@REDACTED (Roberto Ostinelli) Date: Thu, 26 Jul 2012 09:16:05 -0700 Subject: [erlang-questions] Erlang plugin for IntelliJ IDEA In-Reply-To: <87hasublmc.wl%f@mazzo.li> References: <87ipdabmqb.wl%f@mazzo.li> <87hasublmc.wl%f@mazzo.li> Message-ID: > > You asked the author of the IntelliJ plugin why he would do that and not > contribute to existing projects. > > The same can be asked to you, since you decided to extend SublimeText > while a > lot of groundwork was already there for emacs (which is a far more > widespread > editor, and more importantly a free one). i asked the author "why have you started a project similar to many existing ones". you now ask me "why did you pick sublime text as your baseground to implement new stuff". these are not the 'same' question. :) that being said, i am not interested in pursuing this debate. i love intellij and will be providing feedback to the author, and see where we get from there. r. -------------- next part -------------- An HTML attachment was scrubbed... URL: From damienuk@REDACTED Thu Jul 26 18:27:56 2012 From: damienuk@REDACTED (Damienuk Davis) Date: Thu, 26 Jul 2012 09:27:56 -0700 (PDT) Subject: [erlang-questions] run_erl and fsync In-Reply-To: References: <1343283047.62266.YahooMailNeo@web124906.mail.ne1.yahoo.com> <-3577860171847931767@unknownmsgid> <1343306195.69768.YahooMailNeo@web124905.mail.ne1.yahoo.com> Message-ID: <1343320076.43713.YahooMailNeo@web124903.mail.ne1.yahoo.com> It's linux (Redhat ES 5.2) platform with following server configuration DL370 with?2*6core and 12GB RAM? Is it safe to define O_SYNC (this will cause to bybass fsync call)? What can be the impact to the system? ________________________________ From: Dmitry Kolesnikov To: Damienuk Davis Cc: "erlang-questions@REDACTED" Sent: Thursday, July 26, 2012 8:48 PM Subject: Re: [erlang-questions] run_erl and fsync I am curious what platform you are build the staff and what is the config? fsync is behind?USE_FSYNC?but? #ifndef O_SYNC #define O_SYNC 0 #define USE_FSYNC 1 #endif - Dmitry On Jul 26, 2012, at 3:36 PM, Damienuk Davis wrote: In my project there are fsms that communicates with c via erl driver (port). There are io:fwrite's in the erlang side. This program along works fine. But when you run some other process with high disk access, it causes erlang program to halt even if there is?enough memory and processing. If I comment the fsync is run_erl.c file then program works fine even with higher load on?hard disk.? >static void write_to_log(int* lfd, int* log_num, char* buf, int len) >{ >? int size; > > >? /* Decide if new logfile needed, and open if so */ > > >? size = lseek(*lfd,0,SEEK_END); >? if(size+len > log_maxsize) { >? ? close(*lfd); >? ? *log_num = next_log(*log_num); >? ? *lfd = open_log(*log_num, O_RDWR|O_CREAT|O_TRUNC|O_SYNC); >? } > > >? /* Write to log file */ > > >? if (write_all(*lfd, buf, len) < 0) { >? ? status("Error in writing to log.\n"); >? } > > >#if USE_FSYNC >? fsync(*lfd);//after commenting this line it works fine? >#endif >} > >> >> >> >>Thanks >>Damien > > > > > >________________________________ > From: dmitry kolesnikov >To: Damienuk Davis >Cc: "erlang-questions@REDACTED" >Sent: Thursday, July 26, 2012 5:44 PM >Subject: Re: [erlang-questions] run_erl and fsync > > >Does fsync block any VM scheduler? Or is it executed within async thread? >Or none of them :-) > > >I do not think that flush log is required for majority of Web systems. I believe this is a TeleCo legacy... Should it be configurable at least build time? > >Best Regards, >Dmitry >-|-|-*> > > > >On 26.7.2012, at 9.10, Damienuk Davis wrote: > > >Hi all, >> >> >>In one of my projects I used run_erl to launch Erlang VM in daemon mode, rotate logs, etc. >>But I discovered unexpected performance problem running production application with run_erl. Application caused high iowait. >>After some investigation I found that run_erl does fsync on every log entry, causing disk IO load. Do we need fsync on every log entry. >>Appreciate your valuable comments. >> >> >>Thanks >>Damien >_______________________________________________ >>erlang-questions mailing list >>erlang-questions@REDACTED >>http://erlang.org/mailman/listinfo/erlang-questions >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From royen@REDACTED Thu Jul 26 22:12:14 2012 From: royen@REDACTED (Dick Oyen) Date: Thu, 26 Jul 2012 16:12:14 -0400 Subject: [erlang-questions] SNMP Notify Target Message-ID: <1E3838671188454484A9D5F6A0C0C9A906B1797B26@EXCH07.vht.virtualhold.com> Unfortunately there were no responses to my original question, but I have been using debugger to step through Erlang snmp code to see what is happening. The snmpa:send_notification/6 (indirectly) calls snmpa_target_cache:get_targets/1, which returns []. That seems to be because start:application(snmp) decided not to read target_addr.conf or target_params.conf; the reason for its decision was that snmp_target_mib:configure/2 finds that the target params table already exists. It exists because start:application(snmp) had created, but not populated it. QUESTION: Can someone tell me if there is some option and where I should use the option to get the target_addr.conf to be read into the table? Next I experimented with forcing the file to be read into the table by running the following from my agent erl command window: snmp_target_mib:reconfigure("snmp/agent/conf"). The effects of this showed up two tables in the Table Viewer of 'observer': First, snmpa_target_cache contained a single record shown as a 2-tuple of column values: {state, valid} Second, snmp_local_db_shadow contained (among 26 rows) the following 4 rows shown as 2-tuples of column values below: {{snmpTargetAddrTable,first}, {undef,"testMIBMyFirstCoolTrap","testMIBMyFirstCoolTrap"}} {{snmpTargetAddrTable,"testMIBMyFirstCoolTrap"}, {{"testMIBMyFirstCoolTrap",[1,3,6,1,2,1,100,1,1],[127,0,0,1,0,162],5000,3,"tag1","MyCoolTrapParams",3,1,"agent's engine",[],2048},first,first}} {{snmpTargetParamsTable,first}, {undef,"MyCoolTrapParams","MyCoolTrapParams"}} {{snmpTargetParamsTable,"MyCoolTrapParams"}, {{"MyCoolTrapParams",1,2,"initial",1,3,1},first,first}} Wow, that's great, so I called snmpa:send_notification/6 again. Unfortunately, snmpa_target_cache:get_targets/1 still returns [] because ets:match(?CACHE, {{'_','$1'},'$2'} is looking at the table that contains only {state,valid}. Thus, the information from target_addr.conf and target_params.conf is in some rows in a table, but not the table that the code seems to be looking at. QUESTION: Can someone tell me what I am doing wrong? >I am trying to get an SNMP example to send an SNPM trap, but am not succeeding. Any example would do. I am using the tutorial in >http://www.trapexit.org/SNMP_Quick_Start because it is easiest for me to understand. The tutorial says to enter: > >snmpa:send_notification(snmp_master_agent, testMIBMyFirstCoolTrap, no_receiver, "DrpManager", "", []). > >It returns 'ok', but there is no evidence in Wireshark, the agent's Erlang SNMP log, or the SNMP manager that the trap was actually sent. > >Using the Erlang debugger, I step through 'snmpa' code and find that the list of notify targets is empty. > >Please tell me how snmpa should know the notify targets. > >Here is what I have tried: > >I tried changing the NotifyName argument from "DrpManager" to "", because "DrpManager" does not appear in anywhere in the conf or mib files and the doc says that empty string should cause it to send to all targets, but my debugger stepping shows that the list is still empty. > >I tried changing NotifyName to "MyFirstCoolTrap" because notify.conf contains {"MyFirstCoolTrap", "tag1", trap}, but the list is still empty. > >I wonder if the agent is supposed to know about the manager (before the send_notification/6) because of the manager's call to >snmpm:sync_get("simple_user", "otp agent", [[erlNodeName,1]]), which does appear in the agent's SNMP log. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jacob.vorreuter@REDACTED Thu Jul 26 22:52:39 2012 From: jacob.vorreuter@REDACTED (Jacob Vorreuter) Date: Thu, 26 Jul 2012 13:52:39 -0700 Subject: [erlang-questions] [job listing] Senior Erlang Infrastructure Dev @Heroku Message-ID: <0991B54A-C1F6-4257-8373-7393A90194E1@gmail.com> Hi all, We're expanding our Erlang team at Heroku and are looking for either an experienced Erlang developer or someone interested in working with Erlang full time. Here's the listing: http://heroku.theresumator.com/apply/VAmX9t/Senior-Erlang-Infrastructure-Developer.html Heroku is one of the world's largest cloud application platforms. We have extensive infrastructure built out in Erlang and plan on continuing down that path. We're located in San Francisco. We'd prefer someone local or someone willing to relocate to SF, but we'll also consider remote candidates with solid Erlang experience. Please follow the directions in the job posting to apply. Submissions will come directly to me. Questions are welcome. Thanks! Jake -------------- next part -------------- An HTML attachment was scrubbed... URL: From toby@REDACTED Fri Jul 27 04:09:50 2012 From: toby@REDACTED (Toby Thain) Date: Thu, 26 Jul 2012 22:09:50 -0400 Subject: [erlang-questions] Erlang plugin for IntelliJ IDEA In-Reply-To: References: Message-ID: <5011F86E.8080809@telegraphics.com.au> On 25/07/12 11:22 AM, Sergey Ignatov wrote: > Hi all, > > Today I'm happy to announce the pre-alpha version of Erlang support > plugin for IntelliJ Platform based products, e.g. IntelliJ IDEA, > RubyMine etc. > This sounds great - can it backport to IDEA 9.0.4 (OS X PowerPC)? regards --Toby From watson.timothy@REDACTED Fri Jul 27 08:31:11 2012 From: watson.timothy@REDACTED (Tim Watson) Date: Fri, 27 Jul 2012 07:31:11 +0100 Subject: [erlang-questions] Erlang plugin for IntelliJ IDEA In-Reply-To: References: <119DF4D4-6C35-4DDD-9345-B8BA44F48DFA@gmail.com> Message-ID: <7E1FEDCA-E87D-4891-AE6C-E760A9C6B313@gmail.com> Cool. :) On 26 Jul 2012, at 16:43, Sergey Ignatov wrote: > On 26 July 2012 19:21, Tim Watson wrote: >> Will you take pull requests? > Sure, I will. > > Sergey Ignatov From desired.mta@REDACTED Fri Jul 27 09:32:18 2012 From: desired.mta@REDACTED (=?UTF-8?Q?Motiejus_Jak=C5=A1tys?=) Date: Fri, 27 Jul 2012 10:32:18 +0300 Subject: [erlang-questions] Fwd: meck In-Reply-To: References: Message-ID: Forwarding to list. ---------- Forwarded message ---------- From: Motiejus Jak?tys Date: Wed, Jul 25, 2012 at 9:31 AM Subject: Re: [erlang-questions] meck To: Roberto Ostinelli On Wed, Jul 25, 2012 at 7:30 AM, Roberto Ostinelli wrote: > > can't i use this in generators? You can, but another way. Generator must return a set of test cases. For instance: arith_test_() -> [ ?_assertEqual(4, 2*2), ?_assertEqual(0, 1*0) ]. >From eunit manual[1]: A function with a name ending in ..._test_() (note the final underscore) is recognized by EUnit as a test generator function. Test generators return a representation of a set of tests to be executed by EUnit. Stick with the simple test case instead (which ends with test()). > the reason why i didn't include unload is because of this: > https://github.com/eproxus/meck/issues/72 As said, try simple test case first. [1]: http://www.erlang.org/doc/apps/eunit/chapter.html#EUnit_macros -- Motiejus Jak?tys -- Motiejus Jak?tys From adam@REDACTED Fri Jul 27 12:47:31 2012 From: adam@REDACTED (Adam Warski) Date: Fri, 27 Jul 2012 12:47:31 +0200 Subject: [erlang-questions] Replicated messaging in Erlang? Message-ID: Hello, I've been looking at the Erlang ecosystem, and coming from a Java world, I am wondering what is the "Erlang way" to implement the following. The use case is very simple: I want to receive requests from users, and later process them asynchronously. However I want to be sure that if a request is received, it will be safe from single-node crashes, in other words, I want requests to be replicated across my cluster. In Java I would implement it using a replicated message queue. However RabbitMQ doesn't have replicated queues, only the meta-data is clustered (as far as I know). Mnesia has replication, but then it is a database, not a queue (although you could probably implement a mq ontop od that, but I didn't see anybody doing that). So how would you implememt this in Erlang? :) Regards, Adam Warski From f@REDACTED Fri Jul 27 13:19:30 2012 From: f@REDACTED (Francesco Mazzoli) Date: Fri, 27 Jul 2012 12:19:30 +0100 Subject: [erlang-questions] Replicated messaging in Erlang? In-Reply-To: References: Message-ID: <87ipd930zh.wl%f@mazzo.li> At Fri, 27 Jul 2012 12:47:31 +0200, Adam Warski wrote: > However RabbitMQ doesn't have replicated queues, only the meta-data is > clustered (as far as I know). RabbitMQ *does* have replicated (High Availability) queues: http://www.rabbitmq.com/ha.html . -- Francesco * Often in error, never in doubt From cgsmcmlxxv@REDACTED Fri Jul 27 13:24:29 2012 From: cgsmcmlxxv@REDACTED (CGS) Date: Fri, 27 Jul 2012 13:24:29 +0200 Subject: [erlang-questions] Replicated messaging in Erlang? In-Reply-To: References: Message-ID: Hi Adam, If I understood correctly your question, there are two points there: 1. how you can replicate a message; 2. how you can be sure that the message is received. If that is the case, here are few simple answers (hopefully, more skilled Erlang users will come with better ideas/solutions): 1. Take a look at gproc or any equivalent implementation. 2. That depends on how the message is sent, but the general solution is to have a reply from the receiver. Here are two simple solutions for doing that: a) Message sent via threads messaging (Pid ! MyMessage): -> sender: ReceiverPid ! {self(), MyMessage} -> receiver listener (basics only): receive Message -> {SenderPid, SenderMessage} = Message, SenderPid ! received ... end or you can add SenderPid in the group of receivers. b) Message sent via TCP connection (for the receiver only; the sender should have also a listener to be able to receive the answer): listen_socket(Socket) -> case gen_tcp:recv(Socket, 0) of {ok,Data} -> gen_tcp:send(Socket,<<"received">>), listen_socket(Socket); ... end. These are only few basic options. Of course, there are many other options (e.g., monitoring your receiver). I resume my answer to a simple one. Take care of the length of the queue (if you need a personalized queue, I suggest gen_fsm). I hope this answer helps and if I misunderstood your problem, I kindly ask you to ignore my answer. CGS On Fri, Jul 27, 2012 at 12:47 PM, Adam Warski wrote: > Hello, > > I've been looking at the Erlang ecosystem, and coming from a Java world, I > am wondering what is the "Erlang way" to implement the following. > > The use case is very simple: I want to receive requests from users, and > later process them asynchronously. However I want to be sure that if a > request is received, it will be safe from single-node crashes, in other > words, I want requests to be replicated across my cluster. > > In Java I would implement it using a replicated message queue. However > RabbitMQ doesn't have replicated queues, only the meta-data is clustered > (as far as I know). Mnesia has replication, but then it is a database, not > a queue (although you could probably implement a mq ontop od that, but I > didn't see anybody doing that). > > So how would you implememt this in Erlang? :) > > Regards, > Adam Warski > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.santos@REDACTED Fri Jul 27 14:34:23 2012 From: michael.santos@REDACTED (Michael Santos) Date: Fri, 27 Jul 2012 08:34:23 -0400 Subject: [erlang-questions] run_erl and fsync In-Reply-To: <1343320076.43713.YahooMailNeo@web124903.mail.ne1.yahoo.com> References: <1343283047.62266.YahooMailNeo@web124906.mail.ne1.yahoo.com> <-3577860171847931767@unknownmsgid> <1343306195.69768.YahooMailNeo@web124905.mail.ne1.yahoo.com> <1343320076.43713.YahooMailNeo@web124903.mail.ne1.yahoo.com> Message-ID: <20120727123423.GA31973@ioctl> On Thu, Jul 26, 2012 at 09:27:56AM -0700, Damienuk Davis wrote: > It's linux (Redhat ES 5.2) platform with following server configuration > DL370 with?2*6core and 12GB RAM? > Is it safe to define O_SYNC (this will cause to bybass fsync call)? According to the man page, fsync() will result in 2 writes (data + mtime). So, O_SYNC should be faster. > What can be the impact to the system? Some of the log might be lost in a crash if sync writes are disabled. Here's an untested patch to run_erl that gives the option of disabling sync writes. It'd be nice to add an option to use syslog too. diff --git a/erts/etc/unix/run_erl.c b/erts/etc/unix/run_erl.c index 6b350e8..e2de798 100644 --- a/erts/etc/unix/run_erl.c +++ b/erts/etc/unix/run_erl.c @@ -169,6 +169,7 @@ static int log_generations = DEFAULT_LOG_GENERATIONS; static int log_maxsize = DEFAULT_LOG_MAXSIZE; static int log_alive_minutes = DEFAULT_LOG_ALIVE_MINUTES; static int log_activity_minutes = DEFAULT_LOG_ACTIVITY_MINUTES; +static int log_sync_on_write = 1; static int log_alive_in_gmt = 0; static char log_alive_format[ALIVE_BUFFSIZ+1]; static int run_daemon = 0; @@ -296,6 +297,9 @@ int main(int argc, char **argv) if (log_maxsize < LOG_MIN_MAXSIZE) ERROR1(LOG_ERR,"Minimum RUN_ERL_LOG_MAXSIZE is %d", LOG_MIN_MAXSIZE); } + if ((p = getenv("RUN_ERL_DISABLE_SYNC_ON_WRITE"))) { + log_sync_on_write = 0; + } /* * Create FIFOs and open them @@ -480,6 +484,7 @@ static void pass_on(pid_t childpid) int maxfd; int ready; int got_some = 0; /* from to_erl */ + int flags = O_RDWR|O_APPEND|O_CREAT; /* Open the to_erl pipe for reading. * We can't open the writing side because nobody is reading and @@ -495,9 +500,11 @@ static void pass_on(pid_t childpid) #endif /* Open the log file */ + if (log_sync_on_write) + flags |= O_SYNC; lognum = find_next_log_num(); - lfd = open_log(lognum, O_RDWR|O_APPEND|O_CREAT|O_SYNC); + lfd = open_log(lognum, flags); /* Enter the work loop */ @@ -842,7 +849,8 @@ static int open_log(int log_num, int flags) status("Error in writing to log.\n"); #if USE_FSYNC - fsync(lfd); + if (log_sync_on_write) + fsync(lfd); #endif return lfd; @@ -855,14 +863,19 @@ static int open_log(int log_num, int flags) static void write_to_log(int* lfd, int* log_num, char* buf, int len) { int size; + int flags = O_RDWR|O_CREAT|O_TRUNC|O_SYNC; /* Decide if new logfile needed, and open if so */ size = lseek(*lfd,0,SEEK_END); if(size+len > log_maxsize) { + if (!log_sync_on_write) { + fsync(*lfd); + flags &= ~O_SYNC; + } sf_close(*lfd); *log_num = next_log(*log_num); - *lfd = open_log(*log_num, O_RDWR|O_CREAT|O_TRUNC|O_SYNC); + *lfd = open_log(*log_num, flags); } /* Write to log file */ @@ -872,7 +885,8 @@ static void write_to_log(int* lfd, int* log_num, char* buf, int len) } #if USE_FSYNC - fsync(*lfd); + if (log_sync_on_write) + fsync(*lfd); #endif } From desired.mta@REDACTED Fri Jul 27 14:40:48 2012 From: desired.mta@REDACTED (Motiejus =?utf-8?Q?Jak=C5=A1tys?=) Date: Fri, 27 Jul 2012 15:40:48 +0300 Subject: [erlang-questions] [ANN] erlualib 0.1 - seamlessly implement *any* Erlang behaviour in Lua Message-ID: <20120727124048.GA20335@localhost> Hi, I am excited to announce erlualib 0.1, which enables us to implement arbitrary Erlang behavious in Lua. This library allows to embed Lua code to Erlang codebase very easily and transparently. Why do we do it? In Spil games, we have game developers, who are wonderful flash programmers. We want to create some server-side game processing actions, and want game devs to create these. They are also not really much into Erlang. So they write the game rules in Lua, adhering to an Erlang behaviour, and we easily plug the module in. Because of transparency of this library, we (backend) will not even be aware if the game is written in Lua or Erlang. How to do it: 1. Create an Erlang module which you want to implement in Lua 2. Add 4 lines: 1. `-module(my_mod).` 2. `-behaviour(abitrary_behaviour).` 3. `-implemented_in({priv, "/module_impl.lua"}). % where to forward calls` 4. `-compile({parse_transform, lua_behaviour}). % this does the hard work` 3. Compile and use `my_mod` as if it was written in pure Erlang. Below is an example of simple key-value name server. It has 2 operations: * `{add_name, Key :: binary(), Value :: binary()} -> ok.` * `{get_addr, Key :: binary()} -> 'error' | Value :: binary().` ==== `name_server.erl` ==== -module(name_server). -behaviour(gen_server). -implemented_in({priv, "/name_server.lua"}). -compile({parse_transform, lua_behaviour}). ==== `priv/name_server.lua` ==== function init() return erlang.atom("ok"), {} -- This empty table will be our state end -- Forwards the call to function which is specified in req[1]. Returns -- {reply, Return, NewTable}. Return and NewTable are values from the -- forwarded module. function handle_call(req, from, tbl) return erlang.atom("reply"), call(tbl, req) end -- Adds name to State. Returns {ok, Table}. function add_name(tbl, name, address) tbl[name] = address return erlang.atom("ok"), tbl end -- Gets name table and current name. Returns: { 'error' | Value, Table} function get_addr(tbl, name) return tbl[name] or erlang.atom("error"), tbl end -- Call req[1](tbl, req[2], req[3], ...) function call(tbl, req) return _G[req[1]](tbl, unpack(req, 2)) end That's it! Compile `name_server.erl` and call it. Alternatively, download the [example][erlualib_examples] and `make test`. Performance =========== `luam:one_call/3` (the "do all" function) consists of 3 parts: 1. `lua:new_state/0`, takes ~220-250?s. 2. `luam:call/4`, takes ~12-15?s. 3. `lua:close/1`, negligible. For `lua_behaviour`, new Lua state is created on every request, which adds significant overhead. In `gen_(server|fsm)` case (as well as many others), it will be possible to reuse the state, and performance will be much, much better. I just need to create `gen_(server|fsm)` specific `parse_transform` for it, which is planned for near future. Type conversions Lua -> Erlang ============================== As you already noticed in the Lua example, "erlang" lua library is available! Now it has only one method `atom`, which takes a string and returns atom. Analogue `tuple` method is planned (now, when you return an indexed table in Lua, it is treated as a proplist with numeric indices, so there is no way to return nested tuples). Some words about erlualib ========================= erlualib is a library for embedding Lua into Erlang. It provides a simple interface that is very similar to the Lua C API, as well as some very useful high-level functions. This is a fork of Ray Morgan's Erl-Lua library with the following changes: * High test coverage (and PropEr tests) * New low-level commands * Strings in Lua are Binaries in Erlang (instead of lists of numbers) * Many bugfixes * Dialyzer is happy about this project * Rebarized * `luam:call/4`. * arbitrary erlang behaviours in Lua Erlualib is a nice example how and when to PropErly test things. Tests for `parse_transforms` are coming soon. Library is very fresh, and in heavy development. Bug reports, patches, and comments are very welcome. [erlualib_examples]: https://github.com/Motiejus/erlualib_examples [erlualib]: https://github.com/Motiejus/erlualib Motiejus -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: Digital signature URL: From royen@REDACTED Fri Jul 27 14:42:46 2012 From: royen@REDACTED (Dick Oyen) Date: Fri, 27 Jul 2012 08:42:46 -0400 Subject: [erlang-questions] SNMP Notify Target In-Reply-To: References: <1E3838671188454484A9D5F6A0C0C9A906B1797B26@EXCH07.vht.virtualhold.com> Message-ID: <1E3838671188454484A9D5F6A0C0C9A906B1797C62@EXCH07.vht.virtualhold.com> The original mail is at http://erlang.org/pipermail/erlang-questions/2012-July/067981.html and is included in this reply. Thanks for the advice on the {force_load, true} option in agent.config. It got the target address loaded. I am continuing to investigate why the snmpa:send_notification/6 tries to get the target address from snmpa_target_cache (which contains only {state, invalid}) rather than a table that contains the target address (observer Table Viewer shows that table snmp_local_db1_shadow contains the target address and params). From: DefacedVR [mailto:vladislav.chugunov@REDACTED] Sent: Friday, July 27, 2012 6:35 AM To: erlang-programming@REDACTED Cc: Dick Oyen; erlang-questions@REDACTED; Dick Oyen Subject: Re: [erlang-questions] SNMP Notify Target ???????, 27 ???? 2012 ?., 0:12:14 UTC+4 ???????????? Dick Oyen ???????: >Unfortunately there were no responses to my original question, but I have been Can you provide link to the previous mail? using debugger to step through Erlang snmp code to see what is happening. The snmpa:send_notification/6 (indirectly) calls snmpa_target_cache:get_targets/1, which returns []. That seems to be because start:application(snmp) decided not to read target_addr.conf or target_params.conf; the reason for its decision was that snmp_target_mib:configure/2 finds that the target params table already exists. It exists because start:application(snmp) had created, but not populated it. QUESTION: Can someone tell me if there is some option and where I should use the option to get the target_addr.conf to be read into the table? There is option {force_load, true} from http://www.erlang.org/doc/man/snmp_app.html Next I experimented with forcing the file to be read into the table by running the following from my agent erl command window: snmp_target_mib:reconfigure("snmp/agent/conf"). The effects of this showed up two tables in the Table Viewer of 'observer': First, snmpa_target_cache contained a single record shown as a 2-tuple of column values: {state, valid} Second, snmp_local_db_shadow contained (among 26 rows) the following 4 rows shown as 2-tuples of column values below: {{snmpTargetAddrTable,first}, {undef,"testMIBMyFirstCoolTrap","testMIBMyFirstCoolTrap"}} {{snmpTargetAddrTable,"testMIBMyFirstCoolTrap"}, {{"testMIBMyFirstCoolTrap",[1,3,6,1,2,1,100,1,1],[127,0,0,1,0,162],5000,3,"tag1","MyCoolTrapParams",3,1,"agent's engine",[],2048},first,first}} {{snmpTargetParamsTable,first}, {undef,"MyCoolTrapParams","MyCoolTrapParams"}} {{snmpTargetParamsTable,"MyCoolTrapParams"}, {{"MyCoolTrapParams",1,2,"initial",1,3,1},first,first}} Wow, that's great, so I called snmpa:send_notification/6 again. Unfortunately, snmpa_target_cache:get_targets/1 still returns [] because ets:match(?CACHE, {{'_','$1'},'$2'} is looking at the table that contains only {state,valid}. Thus, the information from target_addr.conf and target_params.conf is in some rows in a table, but not the table that the code seems to be looking at. QUESTION: Can someone tell me what I am doing wrong? >I am trying to get an SNMP example to send an SNPM trap, but am not succeeding. Any example would do. I am using the tutorial in >http://www.trapexit.org/SNMP_Quick_Start because it is easiest for me to understand. The tutorial says to enter: > >snmpa:send_notification(snmp_master_agent, testMIBMyFirstCoolTrap, no_receiver, "DrpManager", "", []). > >It returns 'ok', but there is no evidence in Wireshark, the agent's Erlang SNMP log, or the SNMP manager that the trap was actually sent. > >Using the Erlang debugger, I step through 'snmpa' code and find that the list of notify targets is empty. > >Please tell me how snmpa should know the notify targets. > >Here is what I have tried: > >I tried changing the NotifyName argument from "DrpManager" to "", because "DrpManager" does not appear in anywhere in the conf or mib files and the doc says that empty string should cause it to send to all targets, but my debugger stepping shows that the list is still empty. > >I tried changing NotifyName to "MyFirstCoolTrap" because notify.conf contains {"MyFirstCoolTrap", "tag1", trap}, but the list is still empty. > >I wonder if the agent is supposed to know about the manager (before the send_notification/6) because of the manager's call to >snmpm:sync_get("simple_user", "otp agent", [[erlNodeName,1]]), which does appear in the agent's SNMP log. -------------- next part -------------- An HTML attachment was scrubbed... URL: From damienuk@REDACTED Fri Jul 27 14:45:07 2012 From: damienuk@REDACTED (Damienuk Davis) Date: Fri, 27 Jul 2012 05:45:07 -0700 (PDT) Subject: [erlang-questions] run_erl and fsync In-Reply-To: <20120727123423.GA31973@ioctl> References: <1343283047.62266.YahooMailNeo@web124906.mail.ne1.yahoo.com> <-3577860171847931767@unknownmsgid> <1343306195.69768.YahooMailNeo@web124905.mail.ne1.yahoo.com> <1343320076.43713.YahooMailNeo@web124903.mail.ne1.yahoo.com> <20120727123423.GA31973@ioctl> Message-ID: <1343393107.83138.YahooMailNeo@web124903.mail.ne1.yahoo.com> thanks for the information. So disable fsync may not cause huge issue in logs, but will reduce the io wait. Thanks Damien ________________________________ From: Michael Santos To: Damienuk Davis Cc: Dmitry Kolesnikov ; "erlang-questions@REDACTED" Sent: Friday, July 27, 2012 6:04 PM Subject: Re: [erlang-questions] run_erl and fsync On Thu, Jul 26, 2012 at 09:27:56AM -0700, Damienuk Davis wrote: > It's linux (Redhat ES 5.2) platform with following server configuration > DL370 with?2*6core and 12GB RAM? > Is it safe to define O_SYNC (this will cause to bybass fsync call)? According to the man page, fsync() will result in 2 writes (data + mtime). So, O_SYNC should be faster. > What can be the impact to the system? Some of the log might be lost in a crash if sync writes are disabled. Here's an untested patch to run_erl that gives the option of disabling sync writes. It'd be nice to add an option to use syslog too. diff --git a/erts/etc/unix/run_erl.c b/erts/etc/unix/run_erl.c index 6b350e8..e2de798 100644 --- a/erts/etc/unix/run_erl.c +++ b/erts/etc/unix/run_erl.c @@ -169,6 +169,7 @@ static int log_generations = DEFAULT_LOG_GENERATIONS; static int log_maxsize? ? = DEFAULT_LOG_MAXSIZE; static int log_alive_minutes = DEFAULT_LOG_ALIVE_MINUTES; static int log_activity_minutes = DEFAULT_LOG_ACTIVITY_MINUTES; +static int log_sync_on_write = 1; static int log_alive_in_gmt = 0; static char log_alive_format[ALIVE_BUFFSIZ+1]; static int run_daemon = 0; @@ -296,6 +297,9 @@ int main(int argc, char **argv) ? ? if (log_maxsize < LOG_MIN_MAXSIZE) ? ? ? ERROR1(LOG_ERR,"Minimum RUN_ERL_LOG_MAXSIZE is %d", LOG_MIN_MAXSIZE); ? } +? if ((p = getenv("RUN_ERL_DISABLE_SYNC_ON_WRITE"))) { +? ? log_sync_on_write = 0; +? } ? /* ? ? * Create FIFOs and open them @@ -480,6 +484,7 @@ static void pass_on(pid_t childpid) ? ? int maxfd; ? ? int ready; ? ? int got_some = 0; /* from to_erl */ +? ? int flags = O_RDWR|O_APPEND|O_CREAT; ? ? ? ? /* Open the to_erl pipe for reading. ? ? ? * We can't open the writing side because nobody is reading and @@ -495,9 +500,11 @@ static void pass_on(pid_t childpid) #endif ? ? ? ? /* Open the log file */ +? ? if (log_sync_on_write) +? ? ? ? flags |= O_SYNC; ? ? ? ? lognum = find_next_log_num(); -? ? lfd = open_log(lognum, O_RDWR|O_APPEND|O_CREAT|O_SYNC); +? ? lfd = open_log(lognum, flags); ? ? ? ? /* Enter the work loop */ ? ? @@ -842,7 +849,8 @@ static int open_log(int log_num, int flags) ? ? ? status("Error in writing to log.\n"); #if USE_FSYNC -? fsync(lfd); +? if (log_sync_on_write) +? ? ? fsync(lfd); #endif ? return lfd; @@ -855,14 +863,19 @@ static int open_log(int log_num, int flags) static void write_to_log(int* lfd, int* log_num, char* buf, int len) { ? int size; +? int flags = O_RDWR|O_CREAT|O_TRUNC|O_SYNC; ? /* Decide if new logfile needed, and open if so */ ? ? size = lseek(*lfd,0,SEEK_END); ? if(size+len > log_maxsize) { +? ? if (!log_sync_on_write) { +? ? ? ? fsync(*lfd); +? ? ? ? flags &= ~O_SYNC; +? ? } ? ? sf_close(*lfd); ? ? *log_num = next_log(*log_num); -? ? *lfd = open_log(*log_num, O_RDWR|O_CREAT|O_TRUNC|O_SYNC); +? ? *lfd = open_log(*log_num, flags); ? } ? /* Write to log file */ @@ -872,7 +885,8 @@ static void write_to_log(int* lfd, int* log_num, char* buf, int len) ? } #if USE_FSYNC -? fsync(*lfd); +? if (log_sync_on_write) +? ? fsync(*lfd); #endif } -------------- next part -------------- An HTML attachment was scrubbed... URL: From zabrane3@REDACTED Fri Jul 27 17:23:50 2012 From: zabrane3@REDACTED (Zabrane Mickael) Date: Fri, 27 Jul 2012 17:23:50 +0200 Subject: [erlang-questions] Unix "look" command in Erlang Message-ID: <73DD658A-3C4E-497A-A733-826C199AB7EC@gmail.com> Hi, Is there any implementation of the Unix "look" command in Erlang? http://linux.die.net/man/1/look I need to perfom binary search queries on very big (+100GB) sorted files on disk. Regards Zab -------------- next part -------------- An HTML attachment was scrubbed... URL: From zabrane3@REDACTED Fri Jul 27 18:57:00 2012 From: zabrane3@REDACTED (Zabrane Mickael) Date: Fri, 27 Jul 2012 18:57:00 +0200 Subject: [erlang-questions] prim_inet socket incompatible with file:sendfile/2/5 Message-ID: <3152847A-9DA4-45FC-A16A-4D016A9D059E@gmail.com> Hi, Is there any reason why a client socket created from prim_inet do not work with the new file:sendfile fonction? I got {error, badarg}. Any undocumented function to solves this? Regards, Zabrane -------------- next part -------------- An HTML attachment was scrubbed... URL: From adam@REDACTED Fri Jul 27 20:14:39 2012 From: adam@REDACTED (Adam Warski) Date: Fri, 27 Jul 2012 20:14:39 +0200 Subject: [erlang-questions] Replicated messaging in Erlang? In-Reply-To: <6D0A817F-D49B-46A6-BC24-D8988333101C@jtendo.com> References: <6D0A817F-D49B-46A6-BC24-D8988333101C@jtendo.com> Message-ID: Thanks, not sure how I could have missed that :) So, you would use mirrored queues in that use-case? Regards, Adam Warski On 27 lip 2012, at 13:01, Adam Rutkowski wrote: > > On Jul 27, 2012, at 12:47 PM, Adam Warski wrote: > >> However RabbitMQ doesn't have replicated queues, only the meta-data is clustered (as far as I know). > > How about mirrored queues? > > http://www.rabbitmq.com/ha.html > > -- > AR > > > > > > > > > From adam@REDACTED Fri Jul 27 20:27:41 2012 From: adam@REDACTED (Adam Warski) Date: Fri, 27 Jul 2012 20:27:41 +0200 Subject: [erlang-questions] Replicated messaging in Erlang? In-Reply-To: References: Message-ID: Thank you for the answer, On 27 lip 2012, at 13:24, CGS wrote: > Hi Adam, > > If I understood correctly your question, there are two points there: > 1. how you can replicate a message; > 2. how you can be sure that the message is received. > > If that is the case, here are few simple answers (hopefully, more skilled Erlang users will come with better ideas/solutions): > > 1. Take a look at gproc or any equivalent implementation. Thanks, looks quite big, but I'll try to dig through it. Though it's not only replicating the message, it's also consuming the message across the cluster once it is consumed on the master node, etc. Regards, Adam Warski From brian@REDACTED Fri Jul 27 23:37:56 2012 From: brian@REDACTED (Brian L. Troutwine) Date: Fri, 27 Jul 2012 14:37:56 -0700 Subject: [erlang-questions] In what way am I misusing inet's tftpd? Message-ID: <0517E48C-7266-4E24-B6BA-F9533DA973E8@troutwine.us> I'm attempting to use inet's tftp, but I'm doing something incorrectly. I have a simple example application here: https://github.com/blt/tftp-problems While tftp-problemst does bind to the correct port, it never returns data to a client. To reproduce, open a terminal in the project root and: make && ./bin/console This should boot the tftp_hellp application, and drop you into an erlang shell. Confirm that inets is running: 1> application:which_applications(). [{inets,"INETS CXC 138 49","5.9"}, {sasl,"SASL CXC 138 11","2.2.1"}, {stdlib,"ERTS CXC 138 10","1.18.1"}, {kernel,"ERTS CXC 138 10","2.15.1"}] and that the tftp daemon is running: 2> inets:services(). [{tftpd,<0.56.0>},{httpc,<0.50.0>}] Great. Now, if you look in `etc/inets.config` you'll notice we're binding tftpd to 6969. Open another terminal and: > tftp localhost 6969 tftp> get hello.txt Transfer timed out. Bother. If my understanding of tftpd were correct--which clearly it's not--we'd get `<<"hello world">>` back. What am I doing wrong? - blt From max.lapshin@REDACTED Sat Jul 28 15:31:39 2012 From: max.lapshin@REDACTED (Max Lapshin) Date: Sat, 28 Jul 2012 17:31:39 +0400 Subject: [erlang-questions] [ANN] erlualib 0.1 - seamlessly implement *any* Erlang behaviour in Lua In-Reply-To: <20120727124048.GA20335@localhost> References: <20120727124048.GA20335@localhost> Message-ID: It is a really interesting example of gluing two languages together. However, I don't see error handling: lua exceptions should be converted to erlang errors. I'm missing it? From watson.timothy@REDACTED Sat Jul 28 16:03:26 2012 From: watson.timothy@REDACTED (Tim Watson) Date: Sat, 28 Jul 2012 15:03:26 +0100 Subject: [erlang-questions] Replicated messaging in Erlang? In-Reply-To: References: Message-ID: <46013ED3-9D5B-45F8-B9E3-7EA683E614F3@gmail.com> Adam, please take some time to walk through the rabbitmq documentation - there's quite a lot of it! Rabbit supports replication using HA (master/slave mirror) queues, or federation for replicating over the WAN/Internet. This does exactly what you've asked for. If you'd like to understand how it works, take a lot at gm.erl (guaranteed multicast) which is a fully asynchronous ring protocol developed independently at rabbit to support HA. Cheers. Tim On 27 Jul 2012, at 19:27, Adam Warski wrote: > Thank you for the answer, > > On 27 lip 2012, at 13:24, CGS wrote: > >> Hi Adam, >> >> If I understood correctly your question, there are two points there: >> 1. how you can replicate a message; >> 2. how you can be sure that the message is received. >> >> If that is the case, here are few simple answers (hopefully, more skilled Erlang users will come with better ideas/solutions): >> >> 1. Take a look at gproc or any equivalent implementation. > > Thanks, looks quite big, but I'll try to dig through it. Though it's not only replicating the message, it's also consuming the message across the cluster once it is consumed on the master node, etc. > > Regards, > Adam Warski > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From bryan@REDACTED Sat Jul 28 17:42:35 2012 From: bryan@REDACTED (Bryan Hughes) Date: Sat, 28 Jul 2012 08:42:35 -0700 Subject: [erlang-questions] Erlang plugin for IntelliJ IDEA In-Reply-To: References: Message-ID: <5014086B.90307@wobblesoft.com> An HTML attachment was scrubbed... URL: From desired.mta@REDACTED Sat Jul 28 23:08:37 2012 From: desired.mta@REDACTED (=?UTF-8?Q?Motiejus_Jak=C5=A1tys?=) Date: Sun, 29 Jul 2012 00:08:37 +0300 Subject: [erlang-questions] [ANN] erlualib 0.1 - seamlessly implement *any* Erlang behaviour in Lua In-Reply-To: References: <20120727124048.GA20335@localhost> Message-ID: On Saturday, July 28, 2012, Max Lapshin wrote: > It is a really interesting example of gluing two languages together. > > However, I don't see error handling: lua exceptions should be > converted to erlang errors. I'm missing it? > Hi, Indeed, you are right. Next version will throw an erlang exception shall Lua error occur. Motiejus -- Motiejus Jak?tys -------------- next part -------------- An HTML attachment was scrubbed... URL: From bourinov@REDACTED Sun Jul 29 09:39:32 2012 From: bourinov@REDACTED (Max Bourinov) Date: Sun, 29 Jul 2012 11:39:32 +0400 Subject: [erlang-questions] Erlang plugin for IntelliJ IDEA In-Reply-To: <5014086B.90307@wobblesoft.com> References: <5014086B.90307@wobblesoft.com> Message-ID: Hi Sergey, Great news! Will your plugin support rebar (project configuration, deps. etc)? For it is the must feature. Best regards, Max > On 7/25/12 8:22 AM, Sergey Ignatov wrote: > > Hi all, > > Today I'm happy to announce the pre-alpha version of Erlang support > plugin for IntelliJ Platform based products, e.g. IntelliJ IDEA, > RubyMine etc. > > At the moment there are such basic features: > - Syntax and errors highlighting > - References resolving > - Code completion for functions, records and variables > - Keyword code completion > - Rename refactoring for modules, functions, records and variables > - Safe delete refactoring > - Structure view > - Find usages > - Code commenting/uncommenting > - Brace matching > - Basic code formatter > > Plugin page in IntelliJ repository:http://plugins.jetbrains.com/plugin/?pluginId=7083 > Source code: https://github.com/ignatov/intellij-erlang > > If you are interesting in this pluging, feel free to use the issue > tracker: https://github.com/ignatov/intellij-erlang/issues > > Stay tuned. > > Cheers, > Sergey Ignatov > _______________________________________________ > erlang-questions mailing listerlang-questions@REDACTED://erlang.org/mailman/listinfo/erlang-questions > > > -- > > Bryan Hughes > Master Spoonbender / *Wobblesoft* > (415) 515-7916 > > http://www.wobblesoft.com > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Sun Jul 29 13:34:54 2012 From: max.lapshin@REDACTED (Max Lapshin) Date: Sun, 29 Jul 2012 15:34:54 +0400 Subject: [erlang-questions] Erlang SSL: how to connect to server, having password-protected key and cer? Message-ID: Hi. My service provider asked me to generate CSR for SSL connection. I've generated SSL key (RSA PRIVATE KEY) and CSR for it. Provider sent me cer file (Signature Algorithm: sha1WithRSAEncryption written inside it). It seems that I could convert this cer to pem with: openssl x508 -outform pem -in client.cer -out client.pem How can I connect for provider with these things? Host="..." Port=2000 PrivateKey="client.key" PrivatePassword=".." ClientPem="client.pem" What should I do? Options = [ {certfile, ClientPem} ,{keyfile, PrivateKey} ,{password, PrivatePassword} ,binary ,{active,false} ], {ok, SSL} = ssl:connect(Host, Port, Options), This code gives me SSL socket, which is closed right after first sent data packet. From wde@REDACTED Sun Jul 29 14:19:01 2012 From: wde@REDACTED (wde) Date: Sun, 29 Jul 2012 14:19:01 +0200 Subject: [erlang-questions] update erlang shell prompt dynamically Message-ID: <20120729121901.AB1907000084@msfrf2221.sfr.fr> Hello, I would like to use a custom prompt inside the erlang shell. To do it, I defined the following environment variable for stdlib : {shell_prompt_func,{my_module,my_prompt}}. It works fine ! In my prompt definition, I set the number of nodes inside the cluster. The prompt is updated only when I type something in the shell. ? TOTO[titi@REDACTED](cluster_wait)::> blabla(). ** exception error: undefined shell command blabla/0 ? TOTO[titi@REDACTED]:3/3::> How could I refresh the prompt when I push "enter" in the shell ? I didn't find the trick. Thank you for your help. wde From ignatovs@REDACTED Sun Jul 29 22:33:49 2012 From: ignatovs@REDACTED (Sergey Ignatov) Date: Mon, 30 Jul 2012 00:33:49 +0400 Subject: [erlang-questions] Erlang plugin for IntelliJ IDEA In-Reply-To: References: <5014086B.90307@wobblesoft.com> Message-ID: On 29 July 2012 11:39, Max Bourinov wrote: > Hi Sergey, > > Great news! Will your plugin support rebar (project configuration, deps. > etc)? For it is the must feature. I have a feature request in the issue tracker: https://github.com/ignatov/intellij-erlang/issues/9 Will be greate if someone add additional information about rebar support. After that I'll start to implement this issue. Sergey Ignatov From wojtek@REDACTED Sun Jul 29 23:55:30 2012 From: wojtek@REDACTED (=?UTF-8?B?V29qdGVrIE5hcmN6ecWEc2tp?=) Date: Sun, 29 Jul 2012 23:55:30 +0200 Subject: [erlang-questions] Unix "look" command in Erlang In-Reply-To: <73DD658A-3C4E-497A-A733-826C199AB7EC@gmail.com> References: <73DD658A-3C4E-497A-A733-826C199AB7EC@gmail.com> Message-ID: <5015B152.2090809@power.com.pl> On 07/27/2012 05:23 PM, Zabrane Mickael wrote: > > Is there any implementation of the Unix "*look*" command in Erlang? > http://linux.die.net/man/1/look > > I need to perfom binary search queries on very big (+100GB) sorted > files on disk. > > Ahem, let me put it indirectly. Do you plan to deploy on Windows? --Regards, Wojtek -------------- next part -------------- An HTML attachment was scrubbed... URL: From zabrane3@REDACTED Mon Jul 30 00:03:33 2012 From: zabrane3@REDACTED (Zabrane Mickael) Date: Mon, 30 Jul 2012 00:03:33 +0200 Subject: [erlang-questions] Unix "look" command in Erlang In-Reply-To: <5015B152.2090809@power.com.pl> References: <73DD658A-3C4E-497A-A733-826C199AB7EC@gmail.com> <5015B152.2090809@power.com.pl> Message-ID: <017A6818-E6E3-4E2F-BD37-0116253FEE2B@gmail.com> Hi Wojtek, On Jul 29, 2012, at 11:55 PM, Wojtek Narczy?ski wrote: > On 07/27/2012 05:23 PM, Zabrane Mickael wrote: >> >> Is there any implementation of the Unix "look" command in Erlang? >> http://linux.die.net/man/1/look >> >> I need to perfom binary search queries on very big (+100GB) sorted files on disk. >> >> > Ahem, let me put it indirectly. Do you plan to deploy on Windows? Yep. Regards, Zabrane -------------- next part -------------- An HTML attachment was scrubbed... URL: From emmiller@REDACTED Mon Jul 30 04:19:05 2012 From: emmiller@REDACTED (Evan Miller) Date: Sun, 29 Jul 2012 22:19:05 -0400 Subject: [erlang-questions] [ANN] Chicago Boss 0.8.0 Message-ID: Hi everyone, Chicago Boss 0.8.0 is out, featuring support for Cowboy, infrastructure for WebSockets, routing via regular expressions, and a new mechanism for streaming HTTP responses. In addition this version includes a number of bug fixes and some nifty new features from bleeding-edge ErlyDTL and BossDB. Full announcement: http://groups.google.com/group/chicagoboss/browse_thread/thread/b060d6eec7d764a3 Download: http://www.chicagoboss.org/ChicagoBoss-0.8.0.tar.gz Project home page: http://www.chicagoboss.org/ Chicago Boss is a complete Erlang web framework, modeled after Ruby on Rails, that has a vibrant user community, support for a half-dozen databases, and now with first-class support for real-time web page interactions via WebSockets. The goal of the project is to bring modern web programming techniques to Erlang developers, and to introduce the magic of Erlang/OTP to web developers from other language communities. Please direct any questions about this release to the CB mailing list. All the best, Evan From max.lapshin@REDACTED Mon Jul 30 06:05:10 2012 From: max.lapshin@REDACTED (Max Lapshin) Date: Mon, 30 Jul 2012 08:05:10 +0400 Subject: [erlang-questions] [ANN] Chicago Boss 0.8.0 In-Reply-To: References: Message-ID: It's very nice! I've looked through framework, found very nice ORM layer, cool realtime layer, but it is not clear, what about view? Ruby on Rails's power is 60% in its forms helpers. Am I missing it in documentation? From desired.mta@REDACTED Mon Jul 30 08:47:50 2012 From: desired.mta@REDACTED (=?UTF-8?Q?Motiejus_Jak=C5=A1tys?=) Date: Mon, 30 Jul 2012 09:47:50 +0300 Subject: [erlang-questions] [ANN] erlualib 0.1 - seamlessly implement *any* Erlang behaviour in Lua In-Reply-To: References: <20120727124048.GA20335@localhost> Message-ID: On Sat, Jul 28, 2012 at 4:31 PM, Max Lapshin wrote: > It is a really interesting example of gluing two languages together. > > However, I don't see error handling: lua exceptions should be > converted to erlang errors. I'm missing it? Done: https://github.com/Motiejus/erlualib#error-handling Thanks for your feedback. -- Motiejus Jak?tys From barys_ilyushonak@REDACTED Mon Jul 30 09:15:21 2012 From: barys_ilyushonak@REDACTED (Ilyushonak Barys) Date: Mon, 30 Jul 2012 07:15:21 +0000 Subject: [erlang-questions] Common test for Large Scale Testing - how to restart slave node in right way? Message-ID: Greetings, erlangers. I'm working on integration tests for failover requirements in my distributed system. I use Common Test for such purposes, and especially http://www.erlang.org/doc/apps/common_test/ct_master_chapter.html The test case runs several slave nodes, and kill them time to time. As the result I check system state after such experiment. I'm looking for possibility to restart slave node after my test kill it. The best way for me is to configure master node as supervisor which can restart crashed nodes. Is it possible? I have found a few variants: 1. Use ct_slave module, stop and start function from test case code. The question is where can I get test specification in this case? 2. Use event handler, or write custom code, but it still require to parse test spec manually. Both are required to write custom test spec parser. Please, advice. Regards, Boris _______________________________________________________ The information contained in this message may be privileged and conf idential and protected from disclosure. If you are not the original intended recipient, you are hereby notified that any review, retransmission, dissemination, or other use of, or taking of any action in reliance upon, this information is prohibited. If you have received this communication in error, please notify the sender immediately by replying to this message and delete it from your computer. Thank you for your cooperation. Troika Dialog, Russia. If you need assistance please contact our Contact Center (+7495) 258 0500 or go to www.troika.ru/eng/Contacts/system.wbp From ghenry@REDACTED Mon Jul 30 12:17:35 2012 From: ghenry@REDACTED (Gavin Henry) Date: Mon, 30 Jul 2012 11:17:35 +0100 Subject: [erlang-questions] Adding StartTLS support to the newly merged eldap? Message-ID: Hi all, I'd contacted Torbj?rn T?rnkvist about sponsoring StartTLS support in eldap and he informed me it's now part of core. I'm no at any type of level to do this myself right now (I'm very new to Erlang and am just reading Programming Erlang by Joe Armstrong), but wondered if we could sponsor the addition as we'd like it for ejabberd. I'm also ghenry@REDACTED so can help on that side of things for testing etc. Thanks, Gavin. -- Kind Regards, Gavin Henry. Managing Director. T +44 (0) 1224 279484 M +44 (0) 7930 323266 F +44 (0) 1224 824887 E ghenry@REDACTED Open Source. Open Solutions(tm). http://www.suretecsystems.com/ Suretec Systems is a limited company registered in Scotland. Registered number: SC258005. Registered office: 24 Cormack Park, Rothienorman, Inverurie, Aberdeenshire, AB51 8GL. Subject to disclaimer at http://www.suretecgroup.com/disclaimer.html Do you know we have our own VoIP provider called SureVoIP? See http://www.surevoip.co.uk Did you see our API? http://www.surevoip.co.uk/api From erlang@REDACTED Mon Jul 30 14:35:50 2012 From: erlang@REDACTED (Joe Armstrong) Date: Mon, 30 Jul 2012 14:35:50 +0200 Subject: [erlang-questions] unicode in string literals Message-ID: What is a literal string in Erlang? Originally it was a list of integers, each integer being a single character code - this made strings very easy to work with The code test() -> "a?b". Compiles to code which returns the list of integers [97,226,136,158,98]. This is very inconvenient. I had expected it to return [97, 8734, 98]. The length of the list should be 3 not 5 since it contains three unicode characters not five. Is this a bug or a horrible misfeature? So how can I make a string with the three characters 'a' 'infinity' 'b' test() -> "a\x{221e}b" is ugly test() -> <<"a?b"/utf8>> seems to be a bug it gives an error in the shell but is ok in compiled code and returns <<97,195,162,194,136,194,158,98>> which is very strange test() -> [$a,8734,$b] is ugly /Joe From carlsson.richard@REDACTED Mon Jul 30 15:02:13 2012 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Mon, 30 Jul 2012 15:02:13 +0200 Subject: [erlang-questions] unicode in string literals In-Reply-To: References: Message-ID: <501685D5.10904@gmail.com> On 07/30/2012 02:35 PM, Joe Armstrong wrote: > What is a literal string in Erlang? Originally it was a list of > integers, each integer > being a single character code - this made strings very easy to work with > > The code > > test() -> "a?b". > > Compiles to code which returns the list > of integers [97,226,136,158,98]. > > This is very inconvenient. I had expected it to return > [97, 8734, 98]. The length of the list should be 3 not 5 > since it contains three unicode characters not five. > > Is this a bug or a horrible misfeature? You saved your source file as UTF-8, so between the two double-quotes, the source file contains exactly those bytes. But the Erlang compiler assumes your source code is Latin-1, so it thinks that you wrote a Latin-1 string of 5 characters (some of which are non-printing). There's as yet no support for telling the compiler that the input is anything else than Latin-1, so you can't save your source files as UTF-8. (One thing you can do is put the UTF-8 strings in another file and read them at runtime.) > test() -> <<"a?b"/utf8>> seems to be a bug Try <<"???"/utf8>>. It works, but like your first example, the source string is limited to Latin-1. Strings entered in the shell may be interpreted differently though, depending on your locale settings. /Richard From cgsmcmlxxv@REDACTED Mon Jul 30 15:06:38 2012 From: cgsmcmlxxv@REDACTED (CGS) Date: Mon, 30 Jul 2012 15:06:38 +0200 Subject: [erlang-questions] unicode in string literals In-Reply-To: References: Message-ID: Hi Joe, You may try unicode module: test() -> unicode:characters_to_list("a?b",utf8). which will return the desired list [97,8734,98]. As Richard said, the default is Latin-1 (0-255 integers). As for binaries, the same problem (assuming Latin-1). CGS On Mon, Jul 30, 2012 at 2:35 PM, Joe Armstrong wrote: > What is a literal string in Erlang? Originally it was a list of > integers, each integer > being a single character code - this made strings very easy to work with > > The code > > test() -> "a?b". > > Compiles to code which returns the list > of integers [97,226,136,158,98]. > > This is very inconvenient. I had expected it to return > [97, 8734, 98]. The length of the list should be 3 not 5 > since it contains three unicode characters not five. > > Is this a bug or a horrible misfeature? > > So how can I make a string with the three characters 'a' 'infinity' 'b' > > test() -> "a\x{221e}b" is ugly > > test() -> <<"a?b"/utf8>> seems to be a bug > it gives an error in the > shell but is ok in compiled code and > returns > <<97,195,162,194,136,194,158,98>> which is > very strange > > test() -> [$a,8734,$b] is ugly > > /Joe > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From masklinn@REDACTED Mon Jul 30 15:13:49 2012 From: masklinn@REDACTED (Masklinn) Date: Mon, 30 Jul 2012 15:13:49 +0200 Subject: [erlang-questions] unicode in string literals In-Reply-To: <501685D5.10904@gmail.com> References: <501685D5.10904@gmail.com> Message-ID: <3B0B6501-9158-4E5E-B7ED-C76294A699E4@masklinn.net> On 2012-07-30, at 15:02 , Richard Carlsson wrote: > On 07/30/2012 02:35 PM, Joe Armstrong wrote: >> What is a literal string in Erlang? Originally it was a list of >> integers, each integer >> being a single character code - this made strings very easy to work with >> >> The code >> >> test() -> "a?b". >> >> Compiles to code which returns the list >> of integers [97,226,136,158,98]. >> >> This is very inconvenient. I had expected it to return >> [97, 8734, 98]. The length of the list should be 3 not 5 >> since it contains three unicode characters not five. >> >> Is this a bug or a horrible misfeature? > > You saved your source file as UTF-8, so between the two double-quotes, the source file contains exactly those bytes. But the Erlang compiler assumes your source code is Latin-1 I'd expect the string manipulation functions of Erlang assume that as well (that strings are lists of "bytes"), don't they? E.g. that `words` splits on 0x20 (and maybe 0xA0), not on the {{Zs}} general category? From carlsson.richard@REDACTED Mon Jul 30 15:23:09 2012 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Mon, 30 Jul 2012 15:23:09 +0200 Subject: [erlang-questions] unicode in string literals In-Reply-To: References: Message-ID: <50168ABD.50304@gmail.com> On 07/30/2012 03:06 PM, CGS wrote: > Hi Joe, > > You may try unicode module: > > test() -> unicode:characters_to_list("a?b",utf8). > > which will return the desired list [97,8734,98]. As Richard said, the > default is Latin-1 (0-255 integers). No! Don't save a source file as UTF8, at least without a way of marking up such files as being special. The problem is that if you do the trick above, you have to ensure that you convert _all_ string literals explicitly this way (at least if they may contain characters outside ASCII). But if you have a character such as ?, or ?, in a string and you forget to convert explicitly from UTF8 to single code points, then that "?" will in fact be 2 bytes, while in another module saved in Latin-1, the string "?" that looks the same in your editor will be a single byte, and they won't compare equal. Having modules saved with different encodings is a recipe for disaster (in particular when it comes to future maintenance). Erlang currently only supports Latin-1 in source files; until that is fixed, you should keep your UTF8-data in separate files. /Richard From carlsson.richard@REDACTED Mon Jul 30 15:28:12 2012 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Mon, 30 Jul 2012 15:28:12 +0200 Subject: [erlang-questions] unicode in string literals In-Reply-To: <3B0B6501-9158-4E5E-B7ED-C76294A699E4@masklinn.net> References: <501685D5.10904@gmail.com> <3B0B6501-9158-4E5E-B7ED-C76294A699E4@masklinn.net> Message-ID: <50168BEC.8020706@gmail.com> On 07/30/2012 03:13 PM, Masklinn wrote: > I'd expect the string manipulation functions of Erlang assume that as > well (that strings are lists of "bytes"), don't they? E.g. that `words` > splits on 0x20 (and maybe 0xA0), not on the {{Zs}} general category? Yes, the old "string" module in the Erlang stdlib is not much use for working with Unicode strings. You should use something like the "ux" library (https://github.com/freeakk/ux) or Erlang bindings to ICU (can't seem to find the link, but I think there are more than one implementation of such bindings) /Richard From carlsson.richard@REDACTED Mon Jul 30 15:39:16 2012 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Mon, 30 Jul 2012 15:39:16 +0200 Subject: [erlang-questions] unicode in string literals In-Reply-To: <501685D5.10904@gmail.com> References: <501685D5.10904@gmail.com> Message-ID: <50168E84.9010209@gmail.com> Since this encoding confusion seems to be regularly occurring on this list, I might as well post a link to a set of slides I originally made for our internal training: http://www.scribd.com/doc/86177907/Encodings-Unicode-and-Erlang-by-Richard-Carlsson I do apologize for the uglyness; I'm no powerpoint wizard to begin with, and they seem to have been a bit mangled by the upload to Scribd. /Richard From qrilka@REDACTED Mon Jul 30 15:43:37 2012 From: qrilka@REDACTED (Kirill Zaborsky) Date: Mon, 30 Jul 2012 17:43:37 +0400 Subject: [erlang-questions] unicode in string literals In-Reply-To: <50168E84.9010209@gmail.com> References: <501685D5.10904@gmail.com> <50168E84.9010209@gmail.com> Message-ID: Richard, Is it possible to get presentation pdf for free? Kind regards, Kirill Zaborsky 2012/7/30 Richard Carlsson > Since this encoding confusion seems to be regularly occurring on this > list, I might as well post a link to a set of slides I originally made for > our internal training: > > http://www.scribd.com/doc/**86177907/Encodings-Unicode-** > and-Erlang-by-Richard-Carlsson > > I do apologize for the uglyness; I'm no powerpoint wizard to begin with, > and they seem to have been a bit mangled by the upload to Scribd. > > > /Richard > > ______________________________**_________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/**listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlsson.richard@REDACTED Mon Jul 30 15:50:42 2012 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Mon, 30 Jul 2012 15:50:42 +0200 Subject: [erlang-questions] unicode in string literals In-Reply-To: References: <501685D5.10904@gmail.com> <50168E84.9010209@gmail.com> Message-ID: <50169132.1070304@gmail.com> On 07/30/2012 03:43 PM, Kirill Zaborsky wrote: > Richard, > Is it possible to get presentation pdf for free? Absolutely: https://dl.dropbox.com/u/985859/Encodings.pdf /Richard From cgsmcmlxxv@REDACTED Mon Jul 30 15:52:00 2012 From: cgsmcmlxxv@REDACTED (CGS) Date: Mon, 30 Jul 2012 15:52:00 +0200 Subject: [erlang-questions] unicode in string literals In-Reply-To: <50168ABD.50304@gmail.com> References: <50168ABD.50304@gmail.com> Message-ID: Valid point. I didn't say the solution should be necessary used, I just gave a solution which gives an answer for the raised problem. How it is used, I think Joe doesn't need any other instruction (especially from me). :) CGS On Mon, Jul 30, 2012 at 3:23 PM, Richard Carlsson < carlsson.richard@REDACTED> wrote: > On 07/30/2012 03:06 PM, CGS wrote: > >> Hi Joe, >> >> You may try unicode module: >> >> test() -> unicode:characters_to_list("a?**b",utf8). >> >> which will return the desired list [97,8734,98]. As Richard said, the >> default is Latin-1 (0-255 integers). >> > > No! Don't save a source file as UTF8, at least without a way of marking up > such files as being special. The problem is that if you do the trick above, > you have to ensure that you convert _all_ string literals explicitly this > way (at least if they may contain characters outside ASCII). But if you > have a character such as ?, or ?, in a string and you forget to convert > explicitly from UTF8 to single code points, then that "?" will in fact be 2 > bytes, while in another module saved in Latin-1, the string "?" that looks > the same in your editor will be a single byte, and they won't compare > equal. Having modules saved with different encodings is a recipe for > disaster (in particular when it comes to future maintenance). Erlang > currently only supports Latin-1 in source files; until that is fixed, you > should keep your UTF8-data in separate files. > > /Richard > > > ______________________________**_________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/**listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From masklinn@REDACTED Mon Jul 30 16:01:05 2012 From: masklinn@REDACTED (Masklinn) Date: Mon, 30 Jul 2012 16:01:05 +0200 Subject: [erlang-questions] unicode in string literals In-Reply-To: <50168E84.9010209@gmail.com> References: <501685D5.10904@gmail.com> <50168E84.9010209@gmail.com> Message-ID: On 2012-07-30, at 15:39 , Richard Carlsson wrote: > Since this encoding confusion seems to be regularly occurring on this list, I might as well post a link to a set of slides I originally made for our internal training: > > http://www.scribd.com/doc/86177907/Encodings-Unicode-and-Erlang-by-Richard-Carlsson > > I do apologize for the uglyness; I'm no powerpoint wizard to begin with, and they seem to have been a bit mangled by the upload to Scribd. Looks fine to me, although Speakerdeck would probably be lighter on the browser. From watson.timothy@REDACTED Mon Jul 30 16:11:07 2012 From: watson.timothy@REDACTED (Tim Watson) Date: Mon, 30 Jul 2012 15:11:07 +0100 Subject: [erlang-questions] Common test for Large Scale Testing - how to restart slave node in right way? In-Reply-To: References: Message-ID: <8BB668FE-1587-493A-8864-FF0E9035E87D@gmail.com> Https://GitHub.com/nebularis/systest Very new, undocumented. Ping me if you have any questions. :) On 30 Jul 2012, at 08:15, Ilyushonak Barys wrote: > Greetings, erlangers. > > I'm working on integration tests for failover requirements in my distributed system. > I use Common Test for such purposes, and especially http://www.erlang.org/doc/apps/common_test/ct_master_chapter.html > The test case runs several slave nodes, and kill them time to time. As the result I check system state after such experiment. > I'm looking for possibility to restart slave node after my test kill it. The best way for me is to configure master node as supervisor which can restart crashed nodes. Is it possible? > > I have found a few variants: > 1. Use ct_slave module, stop and start function from test case code. The question is where can I get test specification in this case? > 2. Use event handler, or write custom code, but it still require to parse test spec manually. > Both are required to write custom test spec parser. > > Please, advice. > > Regards, > Boris > > _______________________________________________________ > > The information contained in this message may be privileged and conf idential and protected from disclosure. If you are not the original intended recipient, you are hereby notified that any review, retransmission, dissemination, or other use of, or taking of any action in reliance upon, this information is prohibited. If you have received this communication in error, please notify the sender immediately by replying to this message and delete it from your computer. Thank you for your cooperation. Troika Dialog, Russia. > If you need assistance please contact our Contact Center (+7495) 258 0500 or go to www.troika.ru/eng/Contacts/system.wbp > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From erlang@REDACTED Mon Jul 30 16:25:54 2012 From: erlang@REDACTED (Joe Armstrong) Date: Mon, 30 Jul 2012 16:25:54 +0200 Subject: [erlang-questions] unicode in string literals In-Reply-To: References: Message-ID: On Mon, Jul 30, 2012 at 3:06 PM, CGS wrote: > Hi Joe, > > You may try unicode module: > > test() -> unicode:characters_to_list("a?b",utf8). > > which will return the desired list [97,8734,98]. As Richard said, the > default is Latin-1 (0-255 integers). Very strange I tried that earlier, this is what happens: $ Eshell V5.9 (abort with ^G) 1> unicode:characters_to_list([97,226,136,158,98], utf8). [97,226,136,158,98] The manual says the first argument is a utf8 string /Joe /Joe > > As for binaries, the same problem (assuming Latin-1). > > CGS > > > > > On Mon, Jul 30, 2012 at 2:35 PM, Joe Armstrong wrote: >> >> What is a literal string in Erlang? Originally it was a list of >> integers, each integer >> being a single character code - this made strings very easy to work with >> >> The code >> >> test() -> "a?b". >> >> Compiles to code which returns the list >> of integers [97,226,136,158,98]. >> >> This is very inconvenient. I had expected it to return >> [97, 8734, 98]. The length of the list should be 3 not 5 >> since it contains three unicode characters not five. >> >> Is this a bug or a horrible misfeature? >> >> So how can I make a string with the three characters 'a' 'infinity' 'b' >> >> test() -> "a\x{221e}b" is ugly >> >> test() -> <<"a?b"/utf8>> seems to be a bug >> it gives an error in the >> shell but is ok in compiled code and >> returns >> <<97,195,162,194,136,194,158,98>> which is >> very strange >> >> test() -> [$a,8734,$b] is ugly >> >> /Joe >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > From adam@REDACTED Mon Jul 30 16:38:46 2012 From: adam@REDACTED (Adam Warski) Date: Mon, 30 Jul 2012 16:38:46 +0200 Subject: [erlang-questions] Replicated messaging in Erlang? In-Reply-To: <46013ED3-9D5B-45F8-B9E3-7EA683E614F3@gmail.com> References: <46013ED3-9D5B-45F8-B9E3-7EA683E614F3@gmail.com> Message-ID: <4C62CC95-EBE0-4F79-8AD4-48F51C26904C@warski.org> > Adam, please take some time to walk through the rabbitmq documentation - there's quite a lot of it! Rabbit supports replication using HA (master/slave mirror) queues, or federation for replicating over the WAN/Internet. This does exactly what you've asked for. I did some time ago, but either the docs got improved, or I just skipped a whole paragraph after reading "An exception to this are message queues, which by default reside on the node that created them, though they are visible and reachable from all nodes.". Anyway, all is clear now :). > If you'd like to understand how it works, take a lot at gm.erl (guaranteed multicast) which is a fully asynchronous ring protocol developed independently at rabbit to support HA. Thanks for the pointer! Adam -- Adam Warski http://twitter.com/#!/adamwarski http://www.softwaremill.com http://www.warski.org From carlsson.richard@REDACTED Mon Jul 30 16:39:03 2012 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Mon, 30 Jul 2012 16:39:03 +0200 Subject: [erlang-questions] unicode in string literals In-Reply-To: References: <501685D5.10904@gmail.com> <50168E84.9010209@gmail.com> Message-ID: <50169C87.4050106@gmail.com> It only works for members (such as yourself, Dmitrii), so I guessed that's why he used the words "for free". /Richard On 07/30/2012 04:20 PM, Dmitrii Dimandt wrote: > There's a "Download or Print" button on Scribd as well ;) To the right > of the document > >> Richard, >> Is it possible to get presentation pdf for free? >> >> Kind regards, >> Kirill Zaborsky >> >> 2012/7/30 Richard Carlsson > > >> >> Since this encoding confusion seems to be regularly occurring on >> this list, I might as well post a link to a set of slides I >> originally made for our internal training: >> >> http://www.scribd.com/doc/__86177907/Encodings-Unicode-__and-Erlang-by-Richard-Carlsson >> >> >> I do apologize for the uglyness; I'm no powerpoint wizard to begin >> with, and they seem to have been a bit mangled by the upload to >> Scribd. >> >> >> /Richard >> >> _________________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/__listinfo/erlang-questions >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > From erlang@REDACTED Mon Jul 30 16:42:08 2012 From: erlang@REDACTED (Joe Armstrong) Date: Mon, 30 Jul 2012 16:42:08 +0200 Subject: [erlang-questions] unicode in string literals In-Reply-To: <501685D5.10904@gmail.com> References: <501685D5.10904@gmail.com> Message-ID: On Mon, Jul 30, 2012 at 3:02 PM, Richard Carlsson wrote: > On 07/30/2012 02:35 PM, Joe Armstrong wrote: >> >> What is a literal string in Erlang? Originally it was a list of >> integers, each integer >> being a single character code - this made strings very easy to work with >> >> The code >> >> test() -> "a?b". >> >> Compiles to code which returns the list >> of integers [97,226,136,158,98]. >> >> This is very inconvenient. I had expected it to return >> [97, 8734, 98]. The length of the list should be 3 not 5 >> since it contains three unicode characters not five. >> >> Is this a bug or a horrible misfeature? > > > You saved your source file as UTF-8, so between the two double-quotes, the > source file contains exactly those bytes. But the Erlang compiler assumes > your source code is Latin-1, so it thinks that you wrote a Latin-1 string of > 5 characters (some of which are non-printing). There's as yet no support for > telling the compiler that the input is anything else than Latin-1, so you > can't save your source files as UTF-8. (One thing you can do is put the > UTF-8 strings in another file and read them at runtime.) Oh dear - you're right of course. This means that the only portable and 100% correct way to get 'a' 'INFINITY' 'b' into a string literal to say "a,\x{221e},b" "a?b" in any form won't work if the compiler is not explicitly told "this file is utf8" Should the pre-processor make a rude noise and only accept latin1 printable characters? /joe > > >> test() -> <<"a?b"/utf8>> seems to be a bug > > > Try <<"???"/utf8>>. It works, but like your first example, the source string > is limited to Latin-1. Strings entered in the shell may be interpreted > differently though, depending on your locale settings. > > /Richard > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From robert.virding@REDACTED Mon Jul 30 17:02:01 2012 From: robert.virding@REDACTED (Robert Virding) Date: Mon, 30 Jul 2012 16:02:01 +0100 (BST) Subject: [erlang-questions] update erlang shell prompt dynamically In-Reply-To: <20120729121901.AB1907000084@msfrf2221.sfr.fr> Message-ID: There is no trick, it can't be done. The shell sends off a request to read an erlang expression, this requests contains the prompt. The io-system prints this prompt at the start of every line entered for that input. It is not possible to change the prompt in the middle of a read request. The reason it is done this is to be able to sensibly handle concurrent io-requests, each request behaves as an atomic transaction. Robert ----- Original Message ----- > Hello, > > > I would like to use a custom prompt inside the erlang shell. > > To do it, I defined the following environment variable for stdlib : > {shell_prompt_func,{my_module,my_prompt}}. It works fine ! > > In my prompt definition, I set the number of nodes inside the > cluster. The prompt is updated only when I type something in the > shell. > > ? TOTO[titi@REDACTED](cluster_wait)::> blabla(). > ** exception error: undefined shell command blabla/0 > ? TOTO[titi@REDACTED]:3/3::> > > > How could I refresh the prompt when I push "enter" in the shell ? I > didn't find the trick. > > > Thank you for your help. > > > wde > > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From carlsson.richard@REDACTED Mon Jul 30 17:07:16 2012 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Mon, 30 Jul 2012 17:07:16 +0200 Subject: [erlang-questions] unicode in string literals In-Reply-To: References: Message-ID: <5016A324.5060100@gmail.com> n 07/30/2012 04:25 PM, Joe Armstrong wrote: > Very strange I tried that earlier, this is what happens: > > $ Eshell V5.9 (abort with ^G) > 1> unicode:characters_to_list([97,226,136,158,98], utf8). > [97,226,136,158,98] > > The manual says the first argument is a utf8 string The unicode:characters_to_list() function has tripped me up more than once, and the documentation isn't very clear. The key to understanding it seems to be to look at the possible types for the input: Data = latin1_chardata() | chardata() | external_chardata() These are just versions of chardata(), i.e., possibly deep lists with mixed integers and binaries, and they only differ in how binary segments should be interpreted. If there are integers in the list, they will always be interpreted as full Unicode code points, not needing any conversion. So if your input is a list (or any Latin1-encoded IO-list), the following should work: unicode:characters_to_list(iolist_to_binary([97,226,136,158,98]), utf8). /Richard From carlsson.richard@REDACTED Mon Jul 30 17:17:51 2012 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Mon, 30 Jul 2012 17:17:51 +0200 Subject: [erlang-questions] unicode in string literals In-Reply-To: References: <501685D5.10904@gmail.com> Message-ID: <5016A59F.8090402@gmail.com> On 07/30/2012 04:42 PM, Joe Armstrong wrote: > Should the pre-processor make a rude noise and only accept latin1 > printable characters? Maybe. Personally, I've never liked non-printing characters or whitespace other than plain spaces in strings; if you write a program that should output e.g. a tab as part of a string, and you use ^I in the source code instead of \t, how can you stay sure that the editor hasn't at some point changed it to a space, in particular with many different people working on the code? Even leading or trailing spaces are prone to being edited out by mistake. (Erlang supports the \s escape as a synonym for space, for example $\s, but I don't know any other language that has this pretty obvious feature.) /Richard From freeakk@REDACTED Mon Jul 30 19:09:46 2012 From: freeakk@REDACTED (Michael Uvarov) Date: Mon, 30 Jul 2012 21:09:46 +0400 Subject: [erlang-questions] [ANN] Gin (guard in): lists:member using parse_transform Message-ID: Hi, It is a parse_transform, that converts this code `in(X, [1,2,3,Y,Z])' into `(X =:= 1 orelse X =:= 2 orelse X =:= 3 orelse X =:= Y orelse X =:= Z).' -- Best regards, Uvarov Michael From freeakk@REDACTED Mon Jul 30 19:33:20 2012 From: freeakk@REDACTED (Michael Uvarov) Date: Mon, 30 Jul 2012 21:33:20 +0400 Subject: [erlang-questions] unicode in string literals In-Reply-To: <5016A59F.8090402@gmail.com> References: <501685D5.10904@gmail.com> <5016A59F.8090402@gmail.com> Message-ID: Hi, I can write a parse transform for ux, that converts bytes into code points. Something like this: ustr("a?b") The same idea is used here: https://github.com/freeakk/i18n#using-unicode-strings-in-source-code But it converts strings into a ICU format: binary, utf-16. And it is a hack. From dmkolesnikov@REDACTED Mon Jul 30 20:03:30 2012 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Mon, 30 Jul 2012 21:03:30 +0300 Subject: [erlang-questions] [ANN] svg rendering Message-ID: Hello, I've started to work on SVG rendering. The library provides interface to render SVG graphics from Erlang application. The current version targets server-side rendering use-cases with help of ImageMagic command line tool 'convert'. The library internally uses xmerl for XML output. The initial library is available here https://github.com/fogfish/svg See src/svg.erl for api specification See test/svg_test.erl for api example See README for list of feature and backlog. Any comments on API and other subjects are appreciated... Regards, Dmitry From gumm@REDACTED Mon Jul 30 20:42:20 2012 From: gumm@REDACTED (Jesse Gumm) Date: Mon, 30 Jul 2012 13:42:20 -0500 Subject: [erlang-questions] [ANN] Gin (guard in): lists:member using parse_transform In-Reply-To: References: Message-ID: Neat! However, you forgot to include a link to the repo :) -- Jesse Gumm Owner, Sigma Star Systems 414.940.4866 || sigma-star.com || @jessegumm On Jul 30, 2012 12:10 PM, "Michael Uvarov" wrote: > Hi, > > It is a parse_transform, that converts this code `in(X, [1,2,3,Y,Z])' > into `(X =:= 1 orelse X =:= 2 orelse X =:= 3 orelse X =:= Y orelse X > =:= Z).' > > -- > Best regards, > Uvarov Michael > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From freeakk@REDACTED Mon Jul 30 21:21:29 2012 From: freeakk@REDACTED (Michael Uvarov) Date: Mon, 30 Jul 2012 23:21:29 +0400 Subject: [erlang-questions] [ANN] Gin (guard in): lists:member using parse_transform In-Reply-To: References: Message-ID: Here it is: https://github.com/freeakk/gin From ok@REDACTED Tue Jul 31 00:44:46 2012 From: ok@REDACTED (Richard O'Keefe) Date: Tue, 31 Jul 2012 10:44:46 +1200 Subject: [erlang-questions] unicode in string literals In-Reply-To: <50168ABD.50304@gmail.com> References: <50168ABD.50304@gmail.com> Message-ID: The thing that puzzles me about Erlang assuming that source files are in Latin 1 is that I have a tokenizer for Erlang that assumes Latin 1 and in every Erlang/OTP release I've checked there has been at least one file it tripped up on because of UTF-8 characters. When can we expect -encoding('whatever'). to be supported? From ok@REDACTED Tue Jul 31 00:57:33 2012 From: ok@REDACTED (Richard O'Keefe) Date: Tue, 31 Jul 2012 10:57:33 +1200 Subject: [erlang-questions] unicode in string literals In-Reply-To: <5016A59F.8090402@gmail.com> References: <501685D5.10904@gmail.com> <5016A59F.8090402@gmail.com> Message-ID: <3380EBF8-EBD6-4AA0-918E-52AA05C6E40B@cs.otago.ac.nz> On 31/07/2012, at 3:17 AM, Richard Carlsson wrote: > Even leading or trailing spaces are prone to being edited out by mistake. (Erlang supports the \s escape as a synonym for space, for example $\s, but I don't know any other language that has this pretty obvious feature.) Some Prolog systems do (the tokeniser in my book offered it, IIRC). One of the people on the Prolog standard mailing list has been jumping up and down in anger because SWI Prolog has it. But when you have characters like 0'x, 0'y, 0' , it's really nice to have 0\s. (Don't laugh. Erlang's $x, $y, $ , is just as bad.) I actually have a keystroke in my emacs-ish editor to remove trailing spaces because their presence breaks things more often than it helps. From mjtruog@REDACTED Tue Jul 31 01:41:24 2012 From: mjtruog@REDACTED (Michael Truog) Date: Mon, 30 Jul 2012 16:41:24 -0700 Subject: [erlang-questions] unicode in string literals In-Reply-To: References: <50168ABD.50304@gmail.com> Message-ID: <50171BA4.3000007@gmail.com> On 07/30/2012 03:44 PM, Richard O'Keefe wrote: > The thing that puzzles me about Erlang assuming that source files are in > Latin 1 is that I have a tokenizer for Erlang that assumes Latin 1 and > in every Erlang/OTP release I've checked there has been at least one > file it tripped up on because of UTF-8 characters. > > When can we expect -encoding('whatever'). to be supported? The solution with the way things are currently, is just to use modelines (within the first 3 lines of the file) which are supported in your favorite editor, vi or emacs: % -*- coding: utf-8; Mode: erlang; tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- % ex: set softtabstop=4 tabstop=4 shiftwidth=4 expandtab fileencoding=utf-8: You just need to make sure modeline support is turned on (vim seems to default modeline support to off). From barys_ilyushonak@REDACTED Tue Jul 31 08:29:43 2012 From: barys_ilyushonak@REDACTED (Ilyushonak Barys) Date: Tue, 31 Jul 2012 06:29:43 +0000 Subject: [erlang-questions] Common test for Large Scale Testing - how to restart slave node in right way? In-Reply-To: <8BB668FE-1587-493A-8864-FF0E9035E87D@gmail.com> References: <8BB668FE-1587-493A-8864-FF0E9035E87D@gmail.com> Message-ID: Thank you very much. -----Original Message----- From: Tim Watson [mailto:watson.timothy@REDACTED] Sent: Monday, July 30, 2012 6:12 PM To: Ilyushonak Barys Cc: Erlang Subject: Re: [erlang-questions] Common test for Large Scale Testing - how to restart slave node in right way? Https://GitHub.com/nebularis/systest Very new, undocumented. Ping me if you have any questions. :) On 30 Jul 2012, at 08:15, Ilyushonak Barys wrote: > Greetings, erlangers. > > I'm working on integration tests for failover requirements in my distributed system. > I use Common Test for such purposes, and especially http://www.erlang.org/doc/apps/common_test/ct_master_chapter.html > The test case runs several slave nodes, and kill them time to time. As the result I check system state after such experiment. > I'm looking for possibility to restart slave node after my test kill it. The best way for me is to configure master node as supervisor which can restart crashed nodes. Is it possible? > > I have found a few variants: > 1. Use ct_slave module, stop and start function from test case code. The question is where can I get test specification in this case? > 2. Use event handler, or write custom code, but it still require to parse test spec manually. > Both are required to write custom test spec parser. > > Please, advice. > > Regards, > Boris > > _______________________________________________________ > > The information contained in this message may be privileged and conf idential and protected from disclosure. If you are not the original intended recipient, you are hereby notified that any review, retransmission, dissemination, or other use of, or taking of any action in reliance upon, this information is prohibited. If you have received this communication in error, please notify the sender immediately by replying to this message and delete it from your computer. Thank you for your cooperation. Troika Dialog, Russia. > If you need assistance please contact our Contact Center (+7495) 258 0500 or go to www.troika.ru/eng/Contacts/system.wbp > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From g.a.c.rijnders@REDACTED Tue Jul 31 09:00:16 2012 From: g.a.c.rijnders@REDACTED (Michel Rijnders) Date: Tue, 31 Jul 2012 09:00:16 +0200 Subject: [erlang-questions] unicode in string literals In-Reply-To: <50171BA4.3000007@gmail.com> References: <50168ABD.50304@gmail.com> <50171BA4.3000007@gmail.com> Message-ID: On Tue, Jul 31, 2012 at 1:41 AM, Michael Truog wrote: > On 07/30/2012 03:44 PM, Richard O'Keefe wrote: >> The thing that puzzles me about Erlang assuming that source files are in >> Latin 1 is that I have a tokenizer for Erlang that assumes Latin 1 and >> in every Erlang/OTP release I've checked there has been at least one >> file it tripped up on because of UTF-8 characters. >> >> When can we expect -encoding('whatever'). to be supported? > > The solution with the way things are currently, is just to use modelines (within the first 3 lines of the file) which are supported in your favorite editor, vi or emacs: > % -*- coding: utf-8; Mode: erlang; tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > % ex: set softtabstop=4 tabstop=4 shiftwidth=4 expandtab fileencoding=utf-8: > Shouldn't that modeline read: % -*- coding: latin-1; mode: erlang; tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- Since the compiler assumes source files are in Latin 1? > You just need to make sure modeline support is turned on (vim seems to default modeline support to off). > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- My other car is a cdr. From erlang@REDACTED Tue Jul 31 09:05:54 2012 From: erlang@REDACTED (Joe Armstrong) Date: Tue, 31 Jul 2012 09:05:54 +0200 Subject: [erlang-questions] unicode in string literals In-Reply-To: References: <50168ABD.50304@gmail.com> Message-ID: Is "encoding(...)" a good idea? There are four reasonable alternatives a) - all files are Latin1 b) - all files are UTF8 c) - all files are Latin1 or UTF8 and you guess d) - all files are Latin1 or UTF8 or anything else and you tell Today we do a). What would be the consequences of changing to b) in (say) the next major release? This would break some code - but how much? - how much code is there with non Latin1 printable characters in string literals? - it should be easy to write a program to test for this and flag sting literals that might causes problems if the default convention was changed. /Joe On Tue, Jul 31, 2012 at 12:44 AM, Richard O'Keefe wrote: > The thing that puzzles me about Erlang assuming that source files are in > Latin 1 is that I have a tokenizer for Erlang that assumes Latin 1 and > in every Erlang/OTP release I've checked there has been at least one > file it tripped up on because of UTF-8 characters. > > When can we expect -encoding('whatever'). to be supported? > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From mjtruog@REDACTED Tue Jul 31 09:09:46 2012 From: mjtruog@REDACTED (Michael Truog) Date: Tue, 31 Jul 2012 00:09:46 -0700 Subject: [erlang-questions] unicode in string literals In-Reply-To: References: <50168ABD.50304@gmail.com> <50171BA4.3000007@gmail.com> Message-ID: <501784BA.6080707@gmail.com> On 07/31/2012 12:00 AM, Michel Rijnders wrote: > On Tue, Jul 31, 2012 at 1:41 AM, Michael Truog wrote: >> On 07/30/2012 03:44 PM, Richard O'Keefe wrote: >>> The thing that puzzles me about Erlang assuming that source files are in >>> Latin 1 is that I have a tokenizer for Erlang that assumes Latin 1 and >>> in every Erlang/OTP release I've checked there has been at least one >>> file it tripped up on because of UTF-8 characters. >>> >>> When can we expect -encoding('whatever'). to be supported? >> The solution with the way things are currently, is just to use modelines (within the first 3 lines of the file) which are supported in your favorite editor, vi or emacs: >> % -*- coding: utf-8; Mode: erlang; tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- >> % ex: set softtabstop=4 tabstop=4 shiftwidth=4 expandtab fileencoding=utf-8: >> > Shouldn't that modeline read: > % -*- coding: latin-1; mode: erlang; tab-width: 4; c-basic-offset: 4; > indent-tabs-mode: nil -*- > > Since the compiler assumes source files are in Latin 1 I think the point was to use utf8 in the source file, thus the utf8 in the modeline. The encoding() would be necessary for various erlang names (like functions, variables, etc.) to be in utf8, but the modeline could help keep list data as utf8. From ok@REDACTED Tue Jul 31 09:19:47 2012 From: ok@REDACTED (Richard O'Keefe) Date: Tue, 31 Jul 2012 19:19:47 +1200 Subject: [erlang-questions] unicode in string literals In-Reply-To: <501784BA.6080707@gmail.com> References: <50168ABD.50304@gmail.com> <50171BA4.3000007@gmail.com> <501784BA.6080707@gmail.com> Message-ID: <88B18180-8E8F-4692-B99A-A3A16EA94C8A@cs.otago.ac.nz> The snag with mode lines is that they tell the *editor* what to do, but as they are comments they do not tell the *compiler* one blessed thing, unless you wire the peculiarities of two editors (Emacs, VIle) into your compiler (and then what of NetBeans, Eclipse, Visual Studio, and a horde of other editors). From g.a.c.rijnders@REDACTED Tue Jul 31 09:32:51 2012 From: g.a.c.rijnders@REDACTED (Michel Rijnders) Date: Tue, 31 Jul 2012 09:32:51 +0200 Subject: [erlang-questions] unicode in string literals In-Reply-To: <501784BA.6080707@gmail.com> References: <50168ABD.50304@gmail.com> <50171BA4.3000007@gmail.com> <501784BA.6080707@gmail.com> Message-ID: On Tue, Jul 31, 2012 at 9:09 AM, Michael Truog wrote: >-----8<---------- >>> The solution with the way things are currently, is just to use modelines (within the first 3 lines of the file) which are supported in your favorite editor, vi or emacs: >>> % -*- coding: utf-8; Mode: erlang; tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- >>> % ex: set softtabstop=4 tabstop=4 shiftwidth=4 expandtab fileencoding=utf-8: >>> >> Shouldn't that modeline read: >> % -*- coding: latin-1; mode: erlang; tab-width: 4; c-basic-offset: 4; >> indent-tabs-mode: nil -*- >> >> Since the compiler assumes source files are in Latin 1 > > I think the point was to use utf8 in the source file, thus the utf8 in the modeline. The encoding() would be necessary for various erlang names (like functions, variables, etc.) to be in utf8, but the modeline could help keep list data as utf8. IMO this doesn't solve the problem, and only confuses the issue; consider the following: test() -> io:format("~w~n", ["Just my ?0.02"]), io:format("~w~n", [lists:reverse("Just my ?0.02")]). > test(). [74,117,115,116,32,109,121,32,226,130,172,48,46,48,50] [50,48,46,48,172,130,226,32,121,109,32,116,115,117,74] If the list data was kept as UTF-8 then the output of the second statement should be: [50,48,46,48,226,130,172,32,121,109,32,116,115,117,74] The above of course depends on whether you view strings as lists of bytes vs lists of characters. -- My other car is a cdr. From ok@REDACTED Tue Jul 31 09:33:26 2012 From: ok@REDACTED (Richard O'Keefe) Date: Tue, 31 Jul 2012 19:33:26 +1200 Subject: [erlang-questions] unicode in string literals In-Reply-To: References: <50168ABD.50304@gmail.com> Message-ID: <4549881E-9A0B-404B-9DF3-25FF3DE27B35@cs.otago.ac.nz> On 31/07/2012, at 7:05 PM, Joe Armstrong wrote: > Is "encoding(...)" a good idea? > > There are four reasonable alternatives > > a) - all files are Latin1 No good for people who need to write (comments, strings, quoted atoms) in a language not limited to a Western European script. > b) - all files are UTF8 No good for people who are perfectly happy with Latin 1 (me!) and who need the occasional character outside ASCII (like, oh, some people in Sweden maybe?) But could be tolerable. > c) - all files are Latin1 or UTF8 and you guess Guessing is always a bad idea. > d) - all files are Latin1 or UTF8 or anything else and you tell It works for XML. :- encoding(...) works for SWI Prolog: :- encoding(+Encoding) This directive can appear anywhere in a source file to define how characters are encoded in the remainder of the file. It can be used in files that are encoded with a superset of ASCII, currently UTF-8 and Latin-1. See also section 2.18.1. A smart editor like Emacs can be taught to recognise [:]- ?encoding([']Encoding[']). at the top of a file just as easily as it can recognise its own mode-lines. From vladdu55@REDACTED Tue Jul 31 09:36:40 2012 From: vladdu55@REDACTED (Vlad Dumitrescu) Date: Tue, 31 Jul 2012 09:36:40 +0200 Subject: [erlang-questions] unicode in string literals In-Reply-To: References: <50168ABD.50304@gmail.com> Message-ID: Hi, On Tue, Jul 31, 2012 at 9:05 AM, Joe Armstrong wrote: > Is "encoding(...)" a good idea? > > There are four reasonable alternatives > a) - all files are Latin1 > b) - all files are UTF8 > c) - all files are Latin1 or UTF8 and you guess > d) - all files are Latin1 or UTF8 or anything else and you tell By the question above, do you mean to imply that '-encoding(...)' will allow mixed encodings in a project, which is not a reasonable alternative? > Today we do a). > What would be the consequences of changing to b) in (say) the next > major release? > > This would break some code - but how much? - how much code is there > with non Latin1 printable characters > in string literals? I don't think that would be the single problem, but also all the code that assumes that source code is latin-1. Also, tools that handle source code will need to be able to recognize both the old and new encodings, as they might need to have to work with an older version of a file, before the conversion. Another question that needs to be answered is also what encoding will the source code use outside strings and quoted atoms and comments: do we want atoms and variable names to be utf8 too? Because I've seen at least an example of code that uses extended latin-1 characters in those places. Also, what should string manipulation functions do by default, should they assume an encoding? I think the only way to remain sane would be to have a special string type, tagged with the encoding -- as it is now, one can use string manipulation functions on lists of arbitrary integers and list manipulation functions on strings. Would a syntactic construct like u"some string" that returns a tagged utf8 string help? best regards, Vlad From carlsson.richard@REDACTED Tue Jul 31 09:39:55 2012 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Tue, 31 Jul 2012 09:39:55 +0200 Subject: [erlang-questions] unicode in string literals In-Reply-To: <3380EBF8-EBD6-4AA0-918E-52AA05C6E40B@cs.otago.ac.nz> References: <501685D5.10904@gmail.com> <5016A59F.8090402@gmail.com> <3380EBF8-EBD6-4AA0-918E-52AA05C6E40B@cs.otago.ac.nz> Message-ID: <50178BCB.90509@gmail.com> On 07/31/2012 12:57 AM, Richard O'Keefe wrote: > > On 31/07/2012, at 3:17 AM, Richard Carlsson wrote: >> Even leading or trailing spaces are prone to being edited out by >> mistake. (Erlang supports the \s escape as a synonym for space, for >> example $\s, but I don't know any other language that has this >> pretty obvious feature.) > > Some Prolog systems do (the tokeniser in my book offered it, IIRC). > One of the people on the Prolog standard mailing list has been > jumping up and down in anger because SWI Prolog has it. But when you > have characters like 0'x, 0'y, 0' , it's really nice to have 0\s. > (Don't laugh. Erlang's $x, $y, $ , is just as bad.) Yes, I think it was some horrible occurrences of $ , in erl_scan that originally got me to suggest that \s should be added to Erlang, around the time when Barklund was working on the draft Standard. > I actually have a keystroke in my emacs-ish editor to remove > trailing spaces because their presence breaks things more often than > it helps. I wasn't terribly clear - what I meant was leading or trailing spaces within strings, as in "Hello ". It's always slightly worrying that someone might not see that they just removed something important. /Richard From g.a.c.rijnders@REDACTED Tue Jul 31 09:39:58 2012 From: g.a.c.rijnders@REDACTED (Michel Rijnders) Date: Tue, 31 Jul 2012 09:39:58 +0200 Subject: [erlang-questions] unicode in string literals In-Reply-To: References: <50168ABD.50304@gmail.com> Message-ID: On Tue, Jul 31, 2012 at 9:05 AM, Joe Armstrong wrote: > Is "encoding(...)" a good idea? > > There are four reasonable alternatives > > a) - all files are Latin1 > b) - all files are UTF8 > c) - all files are Latin1 or UTF8 and you guess > d) - all files are Latin1 or UTF8 or anything else and you tell I understand it is quite drastic but I would prefer a separate data type for (unicode) strings. -- My other car is a cdr. From ulf@REDACTED Tue Jul 31 09:40:41 2012 From: ulf@REDACTED (Ulf Wiger) Date: Tue, 31 Jul 2012 09:40:41 +0200 Subject: [erlang-questions] unicode in string literals In-Reply-To: References: <50168ABD.50304@gmail.com> Message-ID: <9ED33EAC-AF4B-4476-B0E1-C71FCC9D6816@feuerlabs.com> The problem is that this is editor-dependent. The one time I ran into problems with encoding was when editing a perfectly normal file with Notepad+, which saves in utf8 unless you dive into the settings and manage to tell it not to. For people who need to support multiple OTP versions, including those who maintain Open Source components, it would be a major headache to have to maintain different file formats for different OTP releases. I vote for a compiler option, keeping Latin-1 as the default. BR, Ulf W On 31 Jul 2012, at 09:05, Joe Armstrong wrote: > Is "encoding(...)" a good idea? > > There are four reasonable alternatives > > a) - all files are Latin1 > b) - all files are UTF8 > c) - all files are Latin1 or UTF8 and you guess > d) - all files are Latin1 or UTF8 or anything else and you tell > > Today we do a). > > What would be the consequences of changing to b) in (say) the next > major release? > > This would break some code - but how much? - how much code is there > with non Latin1 printable characters > in string literals? - it should be easy to write a program to test for > this and flag sting literals that > might causes problems if the default convention was changed. > > /Joe Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. http://feuerlabs.com From masklinn@REDACTED Tue Jul 31 09:41:44 2012 From: masklinn@REDACTED (Masklinn) Date: Tue, 31 Jul 2012 09:41:44 +0200 Subject: [erlang-questions] unicode in string literals In-Reply-To: <501784BA.6080707@gmail.com> References: <50168ABD.50304@gmail.com> <50171BA4.3000007@gmail.com> <501784BA.6080707@gmail.com> Message-ID: <457758D6-0F4B-4E82-8C0B-CD56FDC09CED@masklinn.net> On 2012-07-31, at 09:09 , Michael Truog wrote: > > I think the point was to use utf8 in the source file, thus the utf8 in the modeline. The encoding() would be necessary for various erlang names (like functions, variables, etc.) to be in utf8, but the modeline could help keep list data as utf8. For what it's worth, Python solved that particular issue (and redundancy) by adding limited modeline support in the language's parser[0]. Basically, when parsing a file it looks for the pattern `coding=` in a comment in the first and second (to account for shebangs) lines of the file, and if that line is present it uses the specified encoding for the file. That avoids the requirement of specifying the encoding for the editor *and* for the compiler. [0] http://www.python.org/dev/peps/pep-0263/ From masklinn@REDACTED Tue Jul 31 09:53:52 2012 From: masklinn@REDACTED (Masklinn) Date: Tue, 31 Jul 2012 09:53:52 +0200 Subject: [erlang-questions] unicode in string literals In-Reply-To: References: <50168ABD.50304@gmail.com> Message-ID: On 2012-07-31, at 09:39 , Michel Rijnders wrote: > On Tue, Jul 31, 2012 at 9:05 AM, Joe Armstrong wrote: >> Is "encoding(...)" a good idea? >> >> There are four reasonable alternatives >> >> a) - all files are Latin1 >> b) - all files are UTF8 >> c) - all files are Latin1 or UTF8 and you guess >> d) - all files are Latin1 or UTF8 or anything else and you tell > > I understand it is quite drastic but I would prefer a separate data > type for (unicode) strings. For historical reasons? Because on technical grounds, the existing scheme would work nicely by declaring that the integers are code points. And because Unicode is identical to latin-1 in the first 256 codepoints, latin1 strings would be identical. The `string` module would probably need to be fixed to be unicode-aware (or deprecated and removed altogether in favor of the unicode one), but I'm not sure there are good reasons to change the datatype.[-1] On the other hand, a dedicated datatype could allow things like Python's new Flexible String Representation[0] where an explicit "list of code points" would not allow such flexibility. The only thing I'd rather avoid is moving from "list of latin-1 bytes" to "list of utf-8 bytes", that's just crap. [-1] Actually there's one now that I re-think about it thanks to your previous mail about lists:reverse: naive list methods will completely break combining characters or decomposed (NFD and NFKD) strings, even if strings are encoded as lists of codepoints. [0] http://www.python.org/dev/peps/pep-0393/ where strings are opaque and can dynamically changed their internal representation between latin-1, UCS2 and UCS4 to best fit their content, one could even add rope-like structures so that strings are internally mixed between the representations if there is cause to) From carlsson.richard@REDACTED Tue Jul 31 10:02:58 2012 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Tue, 31 Jul 2012 10:02:58 +0200 Subject: [erlang-questions] unicode in string literals In-Reply-To: References: <50168ABD.50304@gmail.com> <50171BA4.3000007@gmail.com> <501784BA.6080707@gmail.com> Message-ID: <50179132.8070007@gmail.com> On 07/31/2012 09:32 AM, Michel Rijnders wrote: > IMO this doesn't solve the problem, and only confuses the issue; > consider the following: > > test() -> > io:format("~w~n", ["Just my ?0.02"]), > io:format("~w~n", [lists:reverse("Just my ?0.02")]). > >> test(). > [74,117,115,116,32,109,121,32,226,130,172,48,46,48,50] > [50,48,46,48,172,130,226,32,121,109,32,116,115,117,74] Yes, this is what happens today, because all involved parts (including the call to io:format with ~w) assumes Latin-1 and just passes all the bytes straight through. Basically, it's your editor and terminal that are lying by displaying a particular sequence of 3 bytes as ? although the program is really using Latin-1. They conspire against you to make you think that things are working correctly. > If the list data was kept as UTF-8 then the output of the second > statement should be: > [50,48,46,48,226,130,172,32,121,109,32,116,115,117,74] That would only be the result if you used a single code point representation for the input to reverse, and then converted the result back to a byte encoding (e.g. by printing with ~ts). > The above of course depends on whether you view strings as lists of > bytes vs lists of characters. Strings are lists of characters (code points), so when your example gets through tokenization, the encoding from the file would already be forgotten, and you'd have a single integer for the ?. (The same goes for atoms and variable names, by the way, the answer to so Vlad's question is that these will also get a greater range of available characters.) String manipulation functions should assume they are working on single code points, not on a byte encoding. /Richard From essen@REDACTED Tue Jul 31 10:10:14 2012 From: essen@REDACTED (=?ISO-8859-1?Q?Lo=EFc_Hoguin?=) Date: Tue, 31 Jul 2012 10:10:14 +0200 Subject: [erlang-questions] unicode in string literals In-Reply-To: References: <50168ABD.50304@gmail.com> Message-ID: <501792E6.1020805@ninenines.eu> On 07/31/2012 09:53 AM, Masklinn wrote: > On 2012-07-31, at 09:39 , Michel Rijnders wrote: > >> On Tue, Jul 31, 2012 at 9:05 AM, Joe Armstrong wrote: >>> Is "encoding(...)" a good idea? >>> >>> There are four reasonable alternatives >>> >>> a) - all files are Latin1 >>> b) - all files are UTF8 >>> c) - all files are Latin1 or UTF8 and you guess >>> d) - all files are Latin1 or UTF8 or anything else and you tell >> >> I understand it is quite drastic but I would prefer a separate data >> type for (unicode) strings. > > For historical reasons? Because on technical grounds, the existing > scheme would work nicely by declaring that the integers are code points. > And because Unicode is identical to latin-1 in the first 256 codepoints, > latin1 strings would be identical. > > The `string` module would probably need to be fixed to be unicode-aware > (or deprecated and removed altogether in favor of the unicode one), but > I'm not sure there are good reasons to change the datatype.[-1] > > On the other hand, a dedicated datatype could allow things like Python's > new Flexible String Representation[0] where an explicit "list of code > points" would not allow such flexibility. > > The only thing I'd rather avoid is moving from "list of latin-1 bytes" to > "list of utf-8 bytes", that's just crap. If strings are kept as lists: - there is no way to identify a variable as being a list or latin1 string or utf8 string - you would have to keep track of what encoding your list is in - you would have to do some type conversion when you use them with functions like gen_tcp:send, which don't accept lists of integers > 255 If strings are a new type: - you don't care about the encoding most of the time, Erlang is the one who should; if you want to know the encoding you could use a new BIF encoding(String) - you don't need to do type conversion when using it, Erlang can use the string type directly - you can convert encoding without caring about what the previous encoding was, for example str:convert(Str, utf8); if it was utf8 it doesn't change a thing, if it wasn't it's converted - you can export it as a list or binary in the encoding you want, for example str:to_binary(Str, utf8) - you still need to specify the encoding when converting a list or binary to string, but maybe we could have niceties like << Str/string-utf8 >>? -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From masklinn@REDACTED Tue Jul 31 10:40:59 2012 From: masklinn@REDACTED (Masklinn) Date: Tue, 31 Jul 2012 10:40:59 +0200 Subject: [erlang-questions] unicode in string literals In-Reply-To: <501792E6.1020805@ninenines.eu> References: <50168ABD.50304@gmail.com> <501792E6.1020805@ninenines.eu> Message-ID: <5D1B5392-1CF1-493F-A29D-6B23A2AB4A2F@masklinn.net> On 2012-07-31, at 10:10 , Lo?c Hoguin wrote: > On 07/31/2012 09:53 AM, Masklinn wrote: >> On 2012-07-31, at 09:39 , Michel Rijnders wrote: >> >>> On Tue, Jul 31, 2012 at 9:05 AM, Joe Armstrong wrote: >>>> Is "encoding(...)" a good idea? >>>> >>>> There are four reasonable alternatives >>>> >>>> a) - all files are Latin1 >>>> b) - all files are UTF8 >>>> c) - all files are Latin1 or UTF8 and you guess >>>> d) - all files are Latin1 or UTF8 or anything else and you tell >>> >>> I understand it is quite drastic but I would prefer a separate data >>> type for (unicode) strings. >> >> For historical reasons? Because on technical grounds, the existing >> scheme would work nicely by declaring that the integers are code points. >> And because Unicode is identical to latin-1 in the first 256 codepoints, >> latin1 strings would be identical. >> >> The `string` module would probably need to be fixed to be unicode-aware >> (or deprecated and removed altogether in favor of the unicode one), but >> I'm not sure there are good reasons to change the datatype.[-1] >> >> On the other hand, a dedicated datatype could allow things like Python's >> new Flexible String Representation[0] where an explicit "list of code >> points" would not allow such flexibility. >> >> The only thing I'd rather avoid is moving from "list of latin-1 bytes" to >> "list of utf-8 bytes", that's just crap. > > If strings are kept as lists: > > - there is no way to identify a variable as being a list or latin1 string or utf8 string > - you would have to keep track of what encoding your list is in None applies, strings would be lists of codepoints, the original encoding has been long forgotten at that point and is utterly irrelevant. > - you would have to do some type conversion when you use them with functions like gen_tcp:send, which don't accept lists of integers > 255 You would have to encode the list to whatever is expected by the other side, on input strings. > If strings are a new type: > > - you don't care about the encoding most of the time, Erlang is the one who should; if you want to know the encoding you could use a new BIF encoding(String) > - you can convert encoding without caring about what the previous encoding was, for example str:convert(Str, utf8); if it was utf8 it doesn't change a thing, if it wasn't it's converted > - you can export it as a list or binary in the encoding you want, for example str:to_binary(Str, utf8) See above, none of these makes sense unless you assume that the string-list is a list of bytes in a specific encoding which does not make sense either, in the first place. > - you don't need to do type conversion when using it, Erlang can use the string type directly How can't Erlang use the string-list type directly? That's what it currently does. There's no conversion. From erlang@REDACTED Tue Jul 31 11:24:05 2012 From: erlang@REDACTED (Joe Armstrong) Date: Tue, 31 Jul 2012 11:24:05 +0200 Subject: [erlang-questions] correct terminology for referring to strings Message-ID: I'm working on a 2'nd edition of my book, and have got to strings :-) Strings confuse everybody, including me so I have a few questions: To start with Erlang doesn't have strings - it has lists (not strings) and it has string literals. I want to define a string - is this correct: << A "string" is a list of integers where the integers represent Unicode codepoints. >> Questions: Is the sentence inside << .. >> using the correct terminology? If not what should it say? Is the sentence inside << ... >> widely understood, do you think this would confuse a lot of people? Is the phrase "string literal" widely understood? Cheers /Joe From cgsmcmlxxv@REDACTED Tue Jul 31 11:36:03 2012 From: cgsmcmlxxv@REDACTED (CGS) Date: Tue, 31 Jul 2012 11:36:03 +0200 Subject: [erlang-questions] unicode in string literals In-Reply-To: <501792E6.1020805@ninenines.eu> References: <50168ABD.50304@gmail.com> <501792E6.1020805@ninenines.eu> Message-ID: There are many pros and cons for switching from Latin-1 to UTF-8 (or whatever else which will nullify pretty much the understanding of byte character). On one hand, lists:reverse/1 really messes up the characters in the list (to follow the first example, the output of "a?b" in Latin-1 is totally different from the output of lists:reverse("b?a") in Latin-1 - the default now). On the other hand, having, for example, Polish characters like "? ? ?" or French "? ?" or German "? ?" or Turkish "?" and so on (things become more complicated if we add languages based on different alphabet/symbols) in the code would require your editor to have support for those languages or else you will see really strange characters there. I do not deny some specific projects would benefit from such a character encoding, but think of maintaining such a code in an international environment. "-encoding()" can make quite a mess in a file. Think of an open source project in which devs from different countries append their own code. You will see a lot of "-encoding()" directives in a single file. I might be wrong, but, switching to default UTF-8, wouldn't that force the compiler to use 2-byte (at least) per character? If so, for example, what about the databases based on Erlang for projects using strict Latin-1? My point here is that the string manipulation should be kept apart from the code itself and to have two modules for manipulating normal lists and IO-lists (e.g., by extending unicode module). But that would be my own preference. CGS On Tue, Jul 31, 2012 at 10:10 AM, Lo?c Hoguin wrote: > On 07/31/2012 09:53 AM, Masklinn wrote: > >> On 2012-07-31, at 09:39 , Michel Rijnders wrote: >> >> On Tue, Jul 31, 2012 at 9:05 AM, Joe Armstrong wrote: >>> >>>> Is "encoding(...)" a good idea? >>>> >>>> There are four reasonable alternatives >>>> >>>> a) - all files are Latin1 >>>> b) - all files are UTF8 >>>> c) - all files are Latin1 or UTF8 and you guess >>>> d) - all files are Latin1 or UTF8 or anything else and you tell >>>> >>> >>> I understand it is quite drastic but I would prefer a separate data >>> type for (unicode) strings. >>> >> >> For historical reasons? Because on technical grounds, the existing >> scheme would work nicely by declaring that the integers are code points. >> And because Unicode is identical to latin-1 in the first 256 codepoints, >> latin1 strings would be identical. >> >> The `string` module would probably need to be fixed to be unicode-aware >> (or deprecated and removed altogether in favor of the unicode one), but >> I'm not sure there are good reasons to change the datatype.[-1] >> >> On the other hand, a dedicated datatype could allow things like Python's >> new Flexible String Representation[0] where an explicit "list of code >> points" would not allow such flexibility. >> >> The only thing I'd rather avoid is moving from "list of latin-1 bytes" to >> "list of utf-8 bytes", that's just crap. >> > > If strings are kept as lists: > > - there is no way to identify a variable as being a list or latin1 string > or utf8 string > - you would have to keep track of what encoding your list is in > - you would have to do some type conversion when you use them with > functions like gen_tcp:send, which don't accept lists of integers > 255 > > If strings are a new type: > > - you don't care about the encoding most of the time, Erlang is the one > who should; if you want to know the encoding you could use a new BIF > encoding(String) > - you don't need to do type conversion when using it, Erlang can use the > string type directly > - you can convert encoding without caring about what the previous encoding > was, for example str:convert(Str, utf8); if it was utf8 it doesn't change a > thing, if it wasn't it's converted > - you can export it as a list or binary in the encoding you want, for > example str:to_binary(Str, utf8) > - you still need to specify the encoding when converting a list or binary > to string, but maybe we could have niceties like << Str/string-utf8 >>? > > -- > Lo?c Hoguin > Erlang Cowboy > Nine Nines > http://ninenines.eu > > > > ______________________________**_________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/**listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.james.barry@REDACTED Tue Jul 31 11:41:48 2012 From: paul.james.barry@REDACTED (Paul Barry) Date: Tue, 31 Jul 2012 10:41:48 +0100 Subject: [erlang-questions] correct terminology for referring to strings In-Reply-To: References: Message-ID: Hi Joe. I think "string literal" is pretty widely understood (it even has a WikiPedia entry, here: http://en.wikipedia.org/wiki/String_literal). What threw me about your sentence was the use of the word 'codepoint', which will be OK for those already familiar with Unicode, but might confuse those who are not. My feeling (and this might be a gross over-simplification) is that most North-American programmers know about Unicode but don't let it worry them too much, resulting in less of a familiarity with it than might be necessary (and I apologize to any North-American programmers that this comment rubs the wrong way). Perhaps "unicode characters" might be easier to read/understand? Although not probably totally technically correct... Another thing that you might wish to consider is breaking the sentence in two, as follows: << An Erlang "string" is simply a list of integers. Each integer can represent any Unicode codepoint/character. >> Just my 2 cent. Paul. On 31 July 2012 10:24, Joe Armstrong wrote: > I'm working on a 2'nd edition of my book, and have got to strings :-) > Strings confuse everybody, including me so I have a few questions: > > To start with Erlang doesn't have strings - it has lists (not strings) > and it has string literals. > > I want to define a string - is this correct: > > << A "string" is a list of integers where the integers > represent Unicode codepoints. >> > > Questions: > Is the sentence inside << .. >> using the correct terminology? > If not what should it say? > > Is the sentence inside << ... >> widely understood, do you think this > would confuse a lot of people? > > Is the phrase "string literal" widely understood? > > > Cheers > > /Joe > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- Paul Barry, w: http://paulbarry.itcarlow.ie - e: paul.barry@REDACTED Lecturer, Computer Networking: Institute of Technology, Carlow, Ireland. From masklinn@REDACTED Tue Jul 31 11:48:06 2012 From: masklinn@REDACTED (Masklinn) Date: Tue, 31 Jul 2012 11:48:06 +0200 Subject: [erlang-questions] unicode in string literals In-Reply-To: References: <50168ABD.50304@gmail.com> <501792E6.1020805@ninenines.eu> Message-ID: <34C3C026-12D4-479A-AAE5-FB2CB2CED969@masklinn.net> On 2012-07-31, at 11:36 , CGS wrote: > > I might be wrong, but, switching to default UTF-8, wouldn't that force the > compiler to use 2-byte (at least) per character? No? The first 128 code points (ASCII) fit in a single byte. > If so, for example, what > about the databases based on Erlang for projects using strict Latin-1? The ASCII (7-bit) characters would be stored on 1 byte, those beyond that (until the codepoint 2048) would be on 2 bytes. From g.a.c.rijnders@REDACTED Tue Jul 31 11:51:50 2012 From: g.a.c.rijnders@REDACTED (Michel Rijnders) Date: Tue, 31 Jul 2012 11:51:50 +0200 Subject: [erlang-questions] correct terminology for referring to strings In-Reply-To: References: Message-ID: On Tue, Jul 31, 2012 at 11:24 AM, Joe Armstrong wrote: > I'm working on a 2'nd edition of my book, and have got to strings :-) > Strings confuse everybody, including me so I have a few questions: > > To start with Erlang doesn't have strings - it has lists (not strings) > and it has string literals. > > I want to define a string - is this correct: > > << A "string" is a list of integers where the integers > represent Unicode codepoints. >> > > Questions: > Is the sentence inside << .. >> using the correct terminology? Is the sentence refering to "strings" in Erlang or to strings in general? For the first I prefer: << A "string" is represented by a list of integers, where the integers are Unicode codepoints.>> For the latter: << A "string" is a sequence of characters. >> > If not what should it say? > > Is the sentence inside << ... >> widely understood, do you think this > would confuse a lot of people? > > Is the phrase "string literal" widely understood? > > > Cheers > > /Joe > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- My other car is a cdr. From michael.eugene.turner@REDACTED Tue Jul 31 11:53:08 2012 From: michael.eugene.turner@REDACTED (Michael Turner) Date: Tue, 31 Jul 2012 18:53:08 +0900 Subject: [erlang-questions] correct terminology for referring to strings In-Reply-To: References: Message-ID: > << An Erlang "string" is simply a list of integers. Each integer can > represent any Unicode codepoint/character. >> Except that Unicode codepoints represents characters, right? You can't have a representation of a representation.[*] I suggest: << In Erlang, strings are represented as lists of integers. These integers are Unicode codepoints, each representing a character. >> That way, anybody who's unclear on what "codepoint" means gets a freebie definition of it. In the Unicode context, it's probably wrong, technically, but perhaps good enough for this purpose. -michael turner [*] Douglas Hofstadter might beg to differ, but he's not on this list. On Tue, Jul 31, 2012 at 6:41 PM, Paul Barry wrote: > Hi Joe. > > I think "string literal" is pretty widely understood (it even has a > WikiPedia entry, here: http://en.wikipedia.org/wiki/String_literal). > > What threw me about your sentence was the use of the word 'codepoint', > which will be OK for those already familiar with Unicode, but might > confuse those who are not. My feeling (and this might be a gross > over-simplification) is that most North-American programmers know > about Unicode but don't let it worry them too much, resulting in less > of a familiarity with it than might be necessary (and I apologize to > any North-American programmers that this comment rubs the wrong way). > Perhaps "unicode characters" might be easier to read/understand? > Although not probably totally technically correct... > > Another thing that you might wish to consider is breaking the sentence > in two, as follows: > > << An Erlang "string" is simply a list of integers. Each integer can > represent any Unicode codepoint/character. >> > > Just my 2 cent. > > Paul. > > On 31 July 2012 10:24, Joe Armstrong wrote: >> I'm working on a 2'nd edition of my book, and have got to strings :-) >> Strings confuse everybody, including me so I have a few questions: >> >> To start with Erlang doesn't have strings - it has lists (not strings) >> and it has string literals. >> >> I want to define a string - is this correct: >> >> << A "string" is a list of integers where the integers >> represent Unicode codepoints. >> >> >> Questions: >> Is the sentence inside << .. >> using the correct terminology? >> If not what should it say? >> >> Is the sentence inside << ... >> widely understood, do you think this >> would confuse a lot of people? >> >> Is the phrase "string literal" widely understood? >> >> >> Cheers >> >> /Joe >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > > > -- > Paul Barry, w: http://paulbarry.itcarlow.ie - e: paul.barry@REDACTED > Lecturer, Computer Networking: Institute of Technology, Carlow, Ireland. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From hobson42@REDACTED Tue Jul 31 13:48:31 2012 From: hobson42@REDACTED (Ian) Date: Tue, 31 Jul 2012 12:48:31 +0100 Subject: [erlang-questions] correct terminology for referring to strings In-Reply-To: References: Message-ID: <5017C60F.1050807@gmail.com> On 31/07/2012 10:24, Joe Armstrong wrote: > I'm working on a 2'nd edition of my book, and have got to strings :-) > Strings confuse everybody, including me so I have a few questions: > > To start with Erlang doesn't have strings - it has lists (not strings) > and it has string literals. > > I want to define a string - is this correct: > > << A "string" is a list of integers where the integers > represent Unicode codepoints. >> I think this is technically correct, but it is very confusing because it implies that the source file may be encoded as unicode. As I understand it, source files are always treated as being in Latin-1. This means that string literals are lists of Latin-1 values, and not lists of unicode codepoints. (The values from 128 to 255 have different/no meanings, and values > 255 will not happen). If you encode your source as something other than Latin-1, the result is a miss-coding of your literal string, with all the problems that presents (vanishing and reappearing characters, wrong lengths etc.). The REPL does take notice of the locale and so can produce different results from the same source strings! I don't envy you the task of writing something that is clear, correct, concise and comprehensible. That will be a challenge! Regards Ian > Questions: > Is the sentence inside << .. >> using the correct terminology? > If not what should it say? > > Is the sentence inside << ... >> widely understood, do you think this > would confuse a lot of people? > > Is the phrase "string literal" widely understood? > > > Cheers > > /Joe > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From cgsmcmlxxv@REDACTED Tue Jul 31 14:14:18 2012 From: cgsmcmlxxv@REDACTED (CGS) Date: Tue, 31 Jul 2012 14:14:18 +0200 Subject: [erlang-questions] unicode in string literals In-Reply-To: References: <50168ABD.50304@gmail.com> <501792E6.1020805@ninenines.eu> Message-ID: On Tue, Jul 31, 2012 at 11:45 AM, Vlad Dumitrescu wrote: > On Tue, Jul 31, 2012 at 11:36 AM, CGS wrote: > > There are many pros and cons for switching from Latin-1 to UTF-8 (or > > whatever else which will nullify pretty much the understanding of byte > > character). ...snip... I do > > not deny some specific projects would benefit from such a character > > encoding, but think of maintaining such a code in an international > > environment. > > Also, think about having to debug a system from a remote console that > doesn't support the right encoding (that's probably long-fetched in > this day and age, but possible). > > > "-encoding()" can make quite a mess in a file. Think of an open source > > project in which devs from different countries append their own code. You > > will see a lot of "-encoding()" directives in a single file. > > My understanding was that there should be one and only one such > directive, at the beginning of the file. I'm not even sure if there > are any editors that can handle files with mixed encodings... > > > My point here is that the string manipulation should be kept apart from > the > > code itself and to have two modules for manipulating normal lists and > > IO-lists (e.g., by extending unicode module). But that would be my own > > preference. > > Yes, but what do you do about string literals? They are in the code... > I would prefer, for example, string literals to be used in debugging using i18n library for translations from English (in the source code) instead of getting strange characters in the log and not understanding the messages in the source code. So, Latin-1 should be enough for that. > > regards, > Vlad > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cgsmcmlxxv@REDACTED Tue Jul 31 14:25:35 2012 From: cgsmcmlxxv@REDACTED (CGS) Date: Tue, 31 Jul 2012 14:25:35 +0200 Subject: [erlang-questions] unicode in string literals In-Reply-To: <34C3C026-12D4-479A-AAE5-FB2CB2CED969@masklinn.net> References: <50168ABD.50304@gmail.com> <501792E6.1020805@ninenines.eu> <34C3C026-12D4-479A-AAE5-FB2CB2CED969@masklinn.net> Message-ID: Correct. My bad. Still, a question remains: how does the compiler make any difference in between a list of integers and a string coded in UTF-8? For example, consider the following case: a list of indexes vs. a string containing special characters in UTF-8. If you apply lists:reverse/1 in UTF-8, you get undesired list for the reversed list of indexes and, vice-versa, if you apply lists:reverse/1 in Latin-1 you get an undesired reversed list for your string. And I don't suppose "-encoding()" would solve this problem either. By dividing the problem in two types of list manipulation, one can easily decide where to apply what. CGS On Tue, Jul 31, 2012 at 11:48 AM, Masklinn wrote: > On 2012-07-31, at 11:36 , CGS wrote: > > > > I might be wrong, but, switching to default UTF-8, wouldn't that force > the > > compiler to use 2-byte (at least) per character? > > No? The first 128 code points (ASCII) fit in a single byte. > > > If so, for example, what > > about the databases based on Erlang for projects using strict Latin-1? > > The ASCII (7-bit) characters would be stored on 1 byte, those beyond > that (until the codepoint 2048) would be on 2 bytes. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tyron.zerafa@REDACTED Tue Jul 31 14:29:45 2012 From: tyron.zerafa@REDACTED (Tyron Zerafa) Date: Tue, 31 Jul 2012 14:29:45 +0200 Subject: [erlang-questions] Process migration/ Process state extraction/ injection Message-ID: Hello everyone, Does Erlang support process migration? Meaning, is there a way in which I can stop a running process (A) on one node and resume it on another completely different node? If this is not supported, one way to go around it is to extract the process state before suspending it and somehow inject it into the new process on the remote node. Does Erlang provide such functionality? Any tips and suggestions on how one can extract/ inject the process state from/ into processes? - Thanks in advance -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.james.barry@REDACTED Tue Jul 31 14:41:08 2012 From: paul.james.barry@REDACTED (Paul Barry) Date: Tue, 31 Jul 2012 13:41:08 +0100 Subject: [erlang-questions] Process migration/ Process state extraction/ injection In-Reply-To: References: Message-ID: Hi Tyron. For my Masters thesis (way back in 2003), I looked at 'mobile agents' as a technology area, and implemented a 'strategy' I came up with in Perl. What I did was look at how Perl's debugger handled a Perl "process", then I used some of the debugging techniques to query the process state, extract & bundle it all together, then ship the code & the state to the new host... this worked reasonably well, and I was able to prove my point. It was/is a bit clunky, but it works! :-) I've an old(ish) website describing all of this here: http://glasnost.itcarlow.ie/~scooby/ So... my suggestion would be to look at how some of Erlang's debugging tools to see if you can attain inspiration... Good luck. Paul. On 31 July 2012 13:29, Tyron Zerafa wrote: > Hello everyone, > Does Erlang support process migration? Meaning, is there a way in which I > can stop a running process (A) on one node and resume it on another > completely different node? > If this is not supported, one way to go around it is to extract the process > state before suspending it and somehow inject it into the new process on the > remote node. Does Erlang provide such functionality? > > Any tips and suggestions on how one can extract/ inject the process state > from/ into processes? > > - Thanks in advance > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Paul Barry, w: http://paulbarry.itcarlow.ie - e: paul.barry@REDACTED Lecturer, Computer Networking: Institute of Technology, Carlow, Ireland. From watson.timothy@REDACTED Tue Jul 31 14:43:13 2012 From: watson.timothy@REDACTED (Tim Watson) Date: Tue, 31 Jul 2012 13:43:13 +0100 Subject: [erlang-questions] reliably figure out hostname Message-ID: <09A191ED-C75E-487C-AE5C-5F1AA6436685@gmail.com> Is there a way to calculate the hostname reliably across platforms in Erlang? I have a non-distributed node that I wish to become a distributed node. Normally I call net_kernel:start([Name, shortnames]) and this is just fine. It also works with [Name, longnames] *sometimes and on some systems* - but other times it pukes. I've tried looking in the 'domain' or 'search' entries from inet:get_rc/0 but these are not always populated, even when dns config is clearly in place. I've also tried using 'inet_db:get_searchlist/0' but again, sometimes this returns [[]] but net_kernel:start([foobar, longnames]) doesn't work, whereas doing net_kernel:start([foobar@REDACTED, longnames]) does. Am I missing something incredibly obvious here? *is* there actually a simple way of determining what the proper fqdn for the machine should be, without breaking out to the os? I had even considered doing inet:gethostbyname/1 but again, the search domains entry seems to be empty, so I'd assume that -name foobar will work whereas in fact, -name foobar@REDACTED is required otherwise net_kernel won't start. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 235 bytes Desc: Message signed with OpenPGP using GPGMail URL: From masklinn@REDACTED Tue Jul 31 14:44:34 2012 From: masklinn@REDACTED (Masklinn) Date: Tue, 31 Jul 2012 14:44:34 +0200 Subject: [erlang-questions] correct terminology for referring to strings In-Reply-To: References: Message-ID: <528339BD-A726-491C-AD3F-FB86E48F718A@masklinn.net> On 2012-07-31, at 11:53 , Michael Turner wrote: >> << An Erlang "string" is simply a list of integers. Each integer can >> represent any Unicode codepoint/character. >> > > Except that Unicode codepoints represents characters, right? Depends, the definition of "character" is quite ambiguous. By "character", many people mean what Unicode calls "grapheme" (a concrete shape or shape-group displayed on a medium[-1]). The meaning of the word may also change across cultures, for instance concerning diacritics some cultures consider the base+diacritic(s) as a single character, others as multiples. And it becomes very tought to define for e.g. hangul, is the character a hangul block or the jamo composing it?[0] The Unicode Standard itself lists 4 different and potentially incompatible meanings for the word: (1) The smallest component of written language that has semantic value; refers to the abstract meaning and/or shape, rather than a specific shape (see also glyph), though in code tables some form of visual representation is essential for the reader?s understanding. (2) Synonym for abstract character. (3) The basic unit of encoding for the Unicode character encoding. (4) The English name for the ideographic written elements of Chinese origin where "abstract character" is defined as: A unit of information used for the organization, control, or representation of textual data. * When representing data, the nature of that data is generally symbolic as opposed to some other kind of data (for example, aural or visual). Examples of such symbolic data include letters, ideographs, digits, punctuation, technical symbols, and dingbats. * An abstract character has no concrete form and should not be confused with a glyph. * An abstract character does not necessarily correspond to what a user thinks of as a ?character? and should not be confused with a grapheme. * The abstract characters encoded by the Unicode Standard are known as Unicode abstract characters. * Abstract characters not directly encoded by the Unicode Standard can often be represented by the use of combining character sequences. In most meanings of the word "character", a character maps to a (potentially unary) *sequence* of unicode code-points, there isn't a 1:1 mapping. [-1] don't take my word for it, I might have fucked up my recollection, I regularly get confused between the precise meanings of "glyph" and "grapheme" [0] Modern hangul is written as syllabic blocks, each block is composed of three jamo (letters, technically 2 to 5 for ancient/historical texts). For instance ? (han) is a block composed of three jamo ?, ?, and ?, and unicode allows encoding it as either HANGUL SYLLABLE HAN or a sequence of HANGUL CHOSEONG HIEUH, HANGUL JUNGSEONG A and HANGUL JONGSEONG NIEUN. But is it a character or not? From masklinn@REDACTED Tue Jul 31 14:48:56 2012 From: masklinn@REDACTED (Masklinn) Date: Tue, 31 Jul 2012 14:48:56 +0200 Subject: [erlang-questions] unicode in string literals In-Reply-To: References: <50168ABD.50304@gmail.com> <501792E6.1020805@ninenines.eu> <34C3C026-12D4-479A-AAE5-FB2CB2CED969@masklinn.net> Message-ID: On 2012-07-31, at 14:25 , CGS wrote: > Still, a question remains: how does the compiler make any difference in > between a list of integers and a string coded in UTF-8? It does not, just as it currently does not make a difference. The distinction is currently informal and based on the usage context of the list. > For example, > consider the following case: a list of indexes vs. a string containing > special characters in UTF-8. If you apply lists:reverse/1 in UTF-8, you get > undesired list for the reversed list of indexes and, vice-versa I touched upon this issue previously. See the first footnote to the message of id DD8AE349-CF34-42DE-A942-376A9A5F3573@REDACTED > if you > apply lists:reverse/1 in Latin-1 you get an undesired reversed list for > your string. Did you mis-write this? If you wanted to reverse your latin-1 string, this does reverse the string correctly. It works in neither UTF-8 nor Unicode contexts though. > And I don't suppose "-encoding()" would solve this problem > either. Correct. From carlsson.richard@REDACTED Tue Jul 31 16:04:05 2012 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Tue, 31 Jul 2012 16:04:05 +0200 Subject: [erlang-questions] correct terminology for referring to strings In-Reply-To: <5017C60F.1050807@gmail.com> References: <5017C60F.1050807@gmail.com> Message-ID: <5017E5D5.2030508@gmail.com> On 07/31/2012 01:48 PM, Ian wrote: >> << A "string" is a list of integers where the integers >> represent Unicode codepoints. >> > > I think this is technically correct, but it is very confusing because it > implies that the source file may be encoded as unicode. > > As I understand it, source files are always treated as being in Latin-1. > This means that string literals are lists of Latin-1 values, and not > lists of unicode codepoints. (The values from 128 to 255 have > different/no meanings, and values > 255 will not happen). No, you're confusing Unicode (a sequence of code points) with specific encodings such as UTF-8 and UTF-16. The first is downwards compatible with Latin-1: the values from 128 to 255 are the same. In UTF-8 they're not. At runtime, Erlang's strings are just plain sequences of Unicode code points (you can think of it as UTF-32 if you like). Whether the source code is encoded in UTF-8 or Latin-1 or any other encoding is irrelevant as long as the compiler knows how to transform the input to the single-codepoint representation. For example, reversing a Unicode string is a bad idea anyway because it could contain combining characters, and reversing the order of the codepoints in that case will create an illegal string. But an expression like lists:reverse("a?b") will be working on the list [97, 8734, 98] (once the compiler has been extended to accept other encodings than Latin-1), not the list [97,226,136,158,98], so it will produce the intended "b?a". This string might then become encoded as UTF-8 on its way to your terminal, but that's another story. /Richard From michael.eugene.turner@REDACTED Tue Jul 31 16:16:00 2012 From: michael.eugene.turner@REDACTED (Michael Turner) Date: Tue, 31 Jul 2012 23:16:00 +0900 Subject: [erlang-questions] correct terminology for referring to strings In-Reply-To: <528339BD-A726-491C-AD3F-FB86E48F718A@masklinn.net> References: <528339BD-A726-491C-AD3F-FB86E48F718A@masklinn.net> Message-ID: On Tue, Jul 31, 2012 at 9:44 PM, Masklinn wrote: > On 2012-07-31, at 11:53 , Michael Turner wrote: > >>> << An Erlang "string" is simply a list of integers. Each integer can >>> represent any Unicode codepoint/character. >> >> >> Except that Unicode codepoints represents characters, right? > > Depends, the definition of "character" is quite ambiguous. I think you're right, and if Joe goes with something like my wording (which is not included in what you quote above), what you say below should really be condensed into a footnote that refers to a more complete and accurate discussion. E.g., "[*] In its fullest generality, it's not quite that simple, since codepoints in some writing systems can actually refer to *parts* of what most Westerners might think of as a single 'character'; see XYZ for a more detailed discussion." [snip] -michael turner From michael.eugene.turner@REDACTED Tue Jul 31 16:19:49 2012 From: michael.eugene.turner@REDACTED (Michael Turner) Date: Tue, 31 Jul 2012 23:19:49 +0900 Subject: [erlang-questions] correct terminology for referring to strings In-Reply-To: <5017E5D5.2030508@gmail.com> References: <5017C60F.1050807@gmail.com> <5017E5D5.2030508@gmail.com> Message-ID: > At runtime, Erlang's strings are just plain sequences of Unicode code points > (you can think of it as UTF-32 if you like). Can you go further and say that it actually *is* UTF-32? A footnote like "[*] Basically, UTF-32; see ref XYZ for details" might be helpful. -michael turner On Tue, Jul 31, 2012 at 11:04 PM, Richard Carlsson wrote: > On 07/31/2012 01:48 PM, Ian wrote: >>> >>> << A "string" is a list of integers where the integers >>> represent Unicode codepoints. >> >> >> >> I think this is technically correct, but it is very confusing because it >> implies that the source file may be encoded as unicode. >> >> As I understand it, source files are always treated as being in Latin-1. >> This means that string literals are lists of Latin-1 values, and not >> lists of unicode codepoints. (The values from 128 to 255 have >> different/no meanings, and values > 255 will not happen). > > > No, you're confusing Unicode (a sequence of code points) with specific > encodings such as UTF-8 and UTF-16. The first is downwards compatible with > Latin-1: the values from 128 to 255 are the same. In UTF-8 they're not. At > runtime, Erlang's strings are just plain sequences of Unicode code points > (you can think of it as UTF-32 if you like). Whether the source code is > encoded in UTF-8 or Latin-1 or any other encoding is irrelevant as long as > the compiler knows how to transform the input to the single-codepoint > representation. > > For example, reversing a Unicode string is a bad idea anyway because it > could contain combining characters, and reversing the order of the > codepoints in that case will create an illegal string. But an expression > like lists:reverse("a?b") will be working on the list [97, 8734, 98] (once > the compiler has been extended to accept other encodings than Latin-1), not > the list [97,226,136,158,98], so it will produce the intended "b?a". This > string might then become encoded as UTF-8 on its way to your terminal, but > that's another story. > > /Richard > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From anders.nygren@REDACTED Tue Jul 31 16:31:53 2012 From: anders.nygren@REDACTED (Anders Nygren) Date: Tue, 31 Jul 2012 09:31:53 -0500 Subject: [erlang-questions] Process migration/ Process state extraction/ injection In-Reply-To: References: Message-ID: This is not exactly what You are asking about but take a look at distributed applications in the documentation, http://www.erlang.org/doc/design_principles/distributed_applications.html Also check the archives for "takeover", there has been some discussions in the past about this that I think can be helpful for You. /Anders On Tue, Jul 31, 2012 at 7:29 AM, Tyron Zerafa wrote: > Hello everyone, > Does Erlang support process migration? Meaning, is there a way in which I > can stop a running process (A) on one node and resume it on another > completely different node? > If this is not supported, one way to go around it is to extract the process > state before suspending it and somehow inject it into the new process on the > remote node. Does Erlang provide such functionality? > > Any tips and suggestions on how one can extract/ inject the process state > from/ into processes? > > - Thanks in advance > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From carlsson.richard@REDACTED Tue Jul 31 16:37:10 2012 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Tue, 31 Jul 2012 16:37:10 +0200 Subject: [erlang-questions] correct terminology for referring to strings In-Reply-To: References: <5017C60F.1050807@gmail.com> <5017E5D5.2030508@gmail.com> Message-ID: <5017ED96.30606@gmail.com> On 07/31/2012 04:19 PM, Michael Turner wrote: >> At runtime, Erlang's strings are just plain sequences of Unicode code points >> (you can think of it as UTF-32 if you like). > > Can you go further and say that it actually *is* UTF-32? A footnote > like "[*] Basically, UTF-32; see ref XYZ for details" might be > helpful. I'm loath to say that it *is* UTF-32, because with that term follows a bunch of connotations such as word width and endianism, which don't apply to the representation as Erlang integers. I'd like to just refer to it as Unicode, but apparently that makes most people think it's either UTF-8 or UTF-16. /Richard From masklinn@REDACTED Tue Jul 31 17:13:01 2012 From: masklinn@REDACTED (Masklinn) Date: Tue, 31 Jul 2012 17:13:01 +0200 Subject: [erlang-questions] correct terminology for referring to strings In-Reply-To: <5017ED96.30606@gmail.com> References: <5017C60F.1050807@gmail.com> <5017E5D5.2030508@gmail.com> <5017ED96.30606@gmail.com> Message-ID: <031E469A-2B7A-46E3-93A8-4A20A16E6DF2@masklinn.net> On 2012-07-31, at 16:37 , Richard Carlsson wrote: > On 07/31/2012 04:19 PM, Michael Turner wrote: >>> At runtime, Erlang's strings are just plain sequences of Unicode code points >>> (you can think of it as UTF-32 if you like). >> >> Can you go further and say that it actually *is* UTF-32? A footnote >> like "[*] Basically, UTF-32; see ref XYZ for details" might be >> helpful. > > I'm loath to say that it *is* UTF-32, because with that term follows a bunch of connotations such as word width and endianism, which don't apply to the representation as Erlang integers. I'd like to just refer to it as Unicode, but apparently that makes most people think it's either UTF-8 or UTF-16. Say it's a sequence of code points (reified as integers)? That's exactly what it is. If people don't know what a code point is, they can look it up. In any case, this shouldn't bring along any undue semantic baggage and misconception. From mononcqc@REDACTED Tue Jul 31 17:19:23 2012 From: mononcqc@REDACTED (Fred Hebert) Date: Tue, 31 Jul 2012 11:19:23 -0400 Subject: [erlang-questions] correct terminology for referring to strings In-Reply-To: <5017E5D5.2030508@gmail.com> References: <5017C60F.1050807@gmail.com> <5017E5D5.2030508@gmail.com> Message-ID: <5017F77B.502@ferd.ca> Even then the reversal is not guaranteed. The character '?' can be represented, for example, in two ways: ? = U+00E9 e+ ? = U+0065 + U+0301 The first one allows a representation as a single codepoint, but the second one is a 'grapheme cluster', a sequence of codepoints representing a single grapheme, a single unit of text. Grapheme clusters can be larger than two elements, and as far as I know, you cannot reverse them. The cluster should ideally remain in the same order in the reversed string: 2> io:format("~ts~n",[[16#0065,16#0301]]). e? ok 3> io:format("~ts~n",[[16#0301,16#0065]]). ?e ok This is fine with your plan -- if I force a single code point representation, this is a non-issue. The tricky thing is that if I enter a string containing " ?e" in my module and later reverse it, I will get "?" and not "e ?" as a final result. What was initially [16#0301,16#0065] gets reversed into [16#0065,16#0301], which is not the same as the correct visual representation " ?e" (represented as ([16#0065, $ , 16#0301]), with an implicit space in there) It works one way (starting the right direction then reversing), but without being very careful, it can break when going the other way (starting with two non-combined code points that get assembled in the same cluster when reversed). Just changing to single code point representations isn't enough to make sure nothing is broken. On 12-07-31 10:04 AM, Richard Carlsson wrote: > No, you're confusing Unicode (a sequence of code points) with specific > encodings such as UTF-8 and UTF-16. The first is downwards compatible > with Latin-1: the values from 128 to 255 are the same. In UTF-8 > they're not. At runtime, Erlang's strings are just plain sequences of > Unicode code points (you can think of it as UTF-32 if you like). > Whether the source code is encoded in UTF-8 or Latin-1 or any other > encoding is irrelevant as long as the compiler knows how to transform > the input to the single-codepoint representation. > > For example, reversing a Unicode string is a bad idea anyway because > it could contain combining characters, and reversing the order of the > codepoints in that case will create an illegal string. But an > expression like lists:reverse("a?b") will be working on the list [97, > 8734, 98] (once the compiler has been extended to accept other > encodings than Latin-1), not the list [97,226,136,158,98], so it will > produce the intended "b?a". This string might then become encoded as > UTF-8 on its way to your terminal, but that's another story. > > /Richard > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From mononcqc@REDACTED Tue Jul 31 17:24:34 2012 From: mononcqc@REDACTED (Fred Hebert) Date: Tue, 31 Jul 2012 11:24:34 -0400 Subject: [erlang-questions] correct terminology for referring to strings In-Reply-To: <5017F77B.502@ferd.ca> References: <5017C60F.1050807@gmail.com> <5017E5D5.2030508@gmail.com> <5017F77B.502@ferd.ca> Message-ID: <5017F8B2.1030404@ferd.ca> On 12-07-31 11:19 AM, Fred Hebert wrote: > The tricky thing is that if I enter a string containing " ?e" in my > module and later reverse it, I will get "?" and not "e ?" as a final > result. What was initially [16#0301,16#0065] gets reversed into > [16#0065,16#0301], which is not the same as the correct visual > representation "?e" (represented as ([16#0065, $ , 16#0301]), with an > implicit space in there) > Quick note that the last " ?e" in there (possibly in red, depending of your mail client) should have been "e ?", which is represented as [16#0065, $ , 16#0301]. Sorry for the confusion. -------------- next part -------------- An HTML attachment was scrubbed... URL: From janburse@REDACTED Tue Jul 31 17:40:46 2012 From: janburse@REDACTED (Jan Burse) Date: Tue, 31 Jul 2012 17:40:46 +0200 Subject: [erlang-questions] correct terminology for referring to strings In-Reply-To: <031E469A-2B7A-46E3-93A8-4A20A16E6DF2@masklinn.net> References: <5017C60F.1050807@gmail.com> <5017E5D5.2030508@gmail.com> <5017ED96.30606@gmail.com> <031E469A-2B7A-46E3-93A8-4A20A16E6DF2@masklinn.net> Message-ID: <5017FC7E.3050905@fastmail.fm> Masklinn schrieb: > Say it's a sequence of code points (reified as integers)? That's exactly > what it is. If people don't know what a code point is, they can look it > up. In any case, this shouldn't bring along any undue semantic baggage > and misconception. > If they are code points, there needs to be a reference to the Unicode version (4.0 or 6.0 etc..), a clarification whether on a specific platforms private codes are supported (i.e. apple sign on mac), a clarfication which planes are supported (basic plane only or supplementary planes also or UCS etc..). http://en.wikipedia.org/wiki/Universal_Character_Set In the ISO Core Standard for Prolog (ISO/IEC 13211-1) the problem is simply solved as follows: The processor character set PCS is an implementation defined character set. The members of PCS shall include each character defined by char (6.5). PCS may include additional members, known as extended characters. It shall be implementation defined for each extended character whether it is a graphic char, or an alphanumeric char, or a solo char, or a layout char, or a meta char. char (* 6.5 *) = graphic char (* 6.5.1 *) alphanumeric char (* 6.5.2 *) solo char (* 6.5.3 l ) layout char (* 65.4 *) meta char (* 6.5.5 *) ; Means the standard does not know about a Unicode extension. But it requires that in a Unicode extensions at least one can deal with the same minimal subset unchanged, and all else is implementation specific, i.e. Prolog system specific. Whereby even the subset is not specified exactly what coding it is, we only have: NOTE - These requirements on the collating sequence are satisfied by both ASCII and EBCDIC. What the standard not did forsee was that there could be different stream encodings on the same processor. So although we have already in the standard: NOTE - A character code may correspond to more than one byte in a stream. Thus, inputting a single character may consume several bytes from an input stream, and writing a single character may output several bytes to an output stream. The current practice is that many Prolog systems offer an encoding/1 option in the stream handling, although no corrigenda has yet picked that up. See for example SWI Prolog: http://www.swi-prolog.org/pldoc/doc_for?object=section%282,%272.18%27,swi%28%27/doc/Manual/widechars.html%27%29%29 Bye From carlsson.richard@REDACTED Tue Jul 31 17:50:11 2012 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Tue, 31 Jul 2012 17:50:11 +0200 Subject: [erlang-questions] correct terminology for referring to strings In-Reply-To: <5017F77B.502@ferd.ca> References: <5017C60F.1050807@gmail.com> <5017E5D5.2030508@gmail.com> <5017F77B.502@ferd.ca> Message-ID: <5017FEB3.3090001@gmail.com> If you take another look at what I wrote, this is precisely what I was talking about. But you are confusing grapheme clusters with combining characters; they are not the same thing. A grapheme cluster is the next higher conceptual level, and a cluster could consist of multiple characters, each of which could be individually made up of a base character (such as "e") plus one or more combining characters (like U+0301 COMBINING ACUTE ACCENT). /Richard On 2012-07-31 17:19, Fred Hebert wrote: > Even then the reversal is not guaranteed. > > The character '?' can be represented, for example, in two ways: > > ? = U+00E9 > e+ ? = U+0065 + U+0301 > > The first one allows a representation as a single codepoint, but the > second one is a 'grapheme cluster', a sequence of codepoints > representing a single grapheme, a single unit of text. Grapheme clusters > can be larger than two elements, and as far as I know, you cannot > reverse them. The cluster should ideally remain in the same order in the > reversed string: > > 2> io:format("~ts~n",[[16#0065,16#0301]]). > e? > ok > 3> io:format("~ts~n",[[16#0301,16#0065]]). > ?e > ok > > This is fine with your plan -- if I force a single code point > representation, this is a non-issue. > > The tricky thing is that if I enter a string containing " ?e" in my > module and later reverse it, I will get "?" and not "e ?" as a final > result. What was initially [16#0301,16#0065] gets reversed into > [16#0065,16#0301], which is not the same as the correct visual > representation " ?e" (represented as ([16#0065, $ , 16#0301]), with an > implicit space in there) > > It works one way (starting the right direction then reversing), but > without being very careful, it can break when going the other way > (starting with two non-combined code points that get assembled in the > same cluster when reversed). > > Just changing to single code point representations isn't enough to make > sure nothing is broken. > > On 12-07-31 10:04 AM, Richard Carlsson wrote: >> No, you're confusing Unicode (a sequence of code points) with specific >> encodings such as UTF-8 and UTF-16. The first is downwards compatible >> with Latin-1: the values from 128 to 255 are the same. In UTF-8 >> they're not. At runtime, Erlang's strings are just plain sequences of >> Unicode code points (you can think of it as UTF-32 if you like). >> Whether the source code is encoded in UTF-8 or Latin-1 or any other >> encoding is irrelevant as long as the compiler knows how to transform >> the input to the single-codepoint representation. >> >> For example, reversing a Unicode string is a bad idea anyway because >> it could contain combining characters, and reversing the order of the >> codepoints in that case will create an illegal string. But an >> expression like lists:reverse("a?b") will be working on the list [97, >> 8734, 98] (once the compiler has been extended to accept other >> encodings than Latin-1), not the list [97,226,136,158,98], so it will >> produce the intended "b?a". This string might then become encoded as >> UTF-8 on its way to your terminal, but that's another story. >> >> /Richard >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > From cgsmcmlxxv@REDACTED Tue Jul 31 18:00:56 2012 From: cgsmcmlxxv@REDACTED (CGS) Date: Tue, 31 Jul 2012 18:00:56 +0200 Subject: [erlang-questions] correct terminology for referring to strings In-Reply-To: <5017F8B2.1030404@ferd.ca> References: <5017C60F.1050807@gmail.com> <5017E5D5.2030508@gmail.com> <5017F77B.502@ferd.ca> <5017F8B2.1030404@ferd.ca> Message-ID: Actually, that depends on the environment you are working in, I suppose. Testing in my local environment, I got this: 1> io:format("~ts~n",[[16#00E9]]). ? ok 2> io:format("~ts~n",[[16#0065,16#0301]]). e ? ok 3> io:format("~ts~n",[[16#0301,16#0065]]). ?e ok CGS On Tue, Jul 31, 2012 at 5:24 PM, Fred Hebert wrote: > > On 12-07-31 11:19 AM, Fred Hebert wrote: > > The tricky thing is that if I enter a string containing " ?e" in my module > and later reverse it, I will get "?" and not "e ?" as a final result. What > was initially [16#0301,16#0065] gets reversed into [16#0065,16#0301], which > is not the same as the correct visual representation " ?e" (represented > as ([16#0065, $ , 16#0301]), with an implicit space in there) > > Quick note that the last " ?e" in there (possibly in red, depending of > your mail client) should have been "e ?", which is represented as > [16#0065, $ , 16#0301]. Sorry for the confusion. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mononcqc@REDACTED Tue Jul 31 18:03:01 2012 From: mononcqc@REDACTED (Fred Hebert) Date: Tue, 31 Jul 2012 12:03:01 -0400 Subject: [erlang-questions] correct terminology for referring to strings In-Reply-To: <5017FEB3.3090001@gmail.com> References: <5017C60F.1050807@gmail.com> <5017E5D5.2030508@gmail.com> <5017F77B.502@ferd.ca> <5017FEB3.3090001@gmail.com> Message-ID: <501801B5.3080208@ferd.ca> Your post seemed to imply that converting to single code point representation is good enough. I do not understand how that distinction solves the problem of string reversal as I wrote it here, though. I would expect, as a user of some string data type or bytestring that claims to support unicode, that reversing a string with the characters " ?e" would give me "e ?". Single code point representation or not. The concept of cluster has to be understood for it to make sense. Regarding your latest post (I received it while writing this one). Cursed be the problem of multiple environments. This is never going to be easy to figure out! On 12-07-31 11:50 AM, Richard Carlsson wrote: > If you take another look at what I wrote, this is precisely what I was > talking about. But you are confusing grapheme clusters with combining > characters; they are not the same thing. A grapheme cluster is the > next higher conceptual level, and a cluster could consist of multiple > characters, each of which could be individually made up of a base > character (such as "e") plus one or more combining characters (like > U+0301 COMBINING ACUTE ACCENT). > > /Richard > > On 2012-07-31 17:19, Fred Hebert wrote: >> Even then the reversal is not guaranteed. >> >> The character '?' can be represented, for example, in two ways: >> >> ? = U+00E9 >> e+ ? = U+0065 + U+0301 >> >> The first one allows a representation as a single codepoint, but the >> second one is a 'grapheme cluster', a sequence of codepoints >> representing a single grapheme, a single unit of text. Grapheme clusters >> can be larger than two elements, and as far as I know, you cannot >> reverse them. The cluster should ideally remain in the same order in the >> reversed string: >> >> 2> io:format("~ts~n",[[16#0065,16#0301]]). >> e? >> ok >> 3> io:format("~ts~n",[[16#0301,16#0065]]). >> ?e >> ok >> >> This is fine with your plan -- if I force a single code point >> representation, this is a non-issue. >> >> The tricky thing is that if I enter a string containing " ?e" in my >> module and later reverse it, I will get "?" and not "e ?" as a final >> result. What was initially [16#0301,16#0065] gets reversed into >> [16#0065,16#0301], which is not the same as the correct visual >> representation " ?e" (represented as ([16#0065, $ , 16#0301]), with an >> implicit space in there) >> >> It works one way (starting the right direction then reversing), but >> without being very careful, it can break when going the other way >> (starting with two non-combined code points that get assembled in the >> same cluster when reversed). >> >> Just changing to single code point representations isn't enough to make >> sure nothing is broken. >> >> On 12-07-31 10:04 AM, Richard Carlsson wrote: >>> No, you're confusing Unicode (a sequence of code points) with specific >>> encodings such as UTF-8 and UTF-16. The first is downwards compatible >>> with Latin-1: the values from 128 to 255 are the same. In UTF-8 >>> they're not. At runtime, Erlang's strings are just plain sequences of >>> Unicode code points (you can think of it as UTF-32 if you like). >>> Whether the source code is encoded in UTF-8 or Latin-1 or any other >>> encoding is irrelevant as long as the compiler knows how to transform >>> the input to the single-codepoint representation. >>> >>> For example, reversing a Unicode string is a bad idea anyway because >>> it could contain combining characters, and reversing the order of the >>> codepoints in that case will create an illegal string. But an >>> expression like lists:reverse("a?b") will be working on the list [97, >>> 8734, 98] (once the compiler has been extended to accept other >>> encodings than Latin-1), not the list [97,226,136,158,98], so it will >>> produce the intended "b?a". This string might then become encoded as >>> UTF-8 on its way to your terminal, but that's another story. >>> >>> /Richard >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >> > From pablo.platt@REDACTED Tue Jul 31 18:41:33 2012 From: pablo.platt@REDACTED (pablo platt) Date: Tue, 31 Jul 2012 19:41:33 +0300 Subject: [erlang-questions] High cpu usage with erlycairo Message-ID: Hi, Cairo is a 2D graphics library. http://www.cairographics.org/ erlycairo uses a c-node to talk to cairo. http://code.google.com/p/erlycairo/ I'm using erlycairo to draw an image on the server. I'm sending a path with ~40 points every 1/2 second and draw it on a 800x1200 px canvas. I'm getting 60% cpu load. When removing the last part which writes the canvas context to a png stream, I'm getting 20% cpu load. Is there a reason why it produces such a big load on the cpu? Is there something I can do to decrease it? Are there other options to draw on a canvas which might be better? This is the code I'm using: {ok, Ctx} = erlycairo_server:new_image_blank(800, 1200), % load previous stream (not shown) erlycairo_server:set_source_rgba(Ctx, R, G, B, 1.0), erlycairo_server:move_to(Ctx, X0, Y0), lists:foreach(fun({X, Y}) -> erlycairo_server:line_to(Ctx, X, Y) end, Path), erlycairo_server:set_line_width(Ctx, 1), erlycairo_server:stroke(Ctx), {ok, Stream} = erlycairo_server:write_to_png_stream(Ctx), % save the stream (not shown) erlycairo_server:close_image(Ctx). Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlsson.richard@REDACTED Tue Jul 31 18:49:07 2012 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Tue, 31 Jul 2012 18:49:07 +0200 Subject: [erlang-questions] correct terminology for referring to strings In-Reply-To: <501801B5.3080208@ferd.ca> References: <5017C60F.1050807@gmail.com> <5017E5D5.2030508@gmail.com> <5017F77B.502@ferd.ca> <5017FEB3.3090001@gmail.com> <501801B5.3080208@ferd.ca> Message-ID: <50180C83.5020504@gmail.com> On 07/31/2012 06:03 PM, Fred Hebert wrote: > Your post seemed to imply that converting to single code point > representation is good enough. I do not understand how that distinction > solves the problem of string reversal as I wrote it here, though. It doesn't. That's what I said: "reversing a Unicode string is a bad idea anyway because it could contain combining characters". But I also clarified that on the Erlang level, at runtime, strings will contain single code points rather than a UTF-8 encoded byte sequence, so for the particular example of "a?b" it happens to work. Nothing more, nothing less. > I would expect, as a user of some string data type or bytestring that > claims to support unicode, that reversing a string with the characters " > ?e" would give me "e ?". Single code point representation or not. Yes. That's why there needs to be a new Unicode-aware string library. Operating directly on lists (e.g. using lists:reverse/1, or even length/1) is always going to have surprising effects, and the old 'string' module in stdlib probably can't be modernized while maintaining backwards compatibility. > The concept of cluster has to be understood for it to make sense. Grapheme clusters are actually one of the things you don't need to think too much about unless you're writing an editor or similar where you need to figure out between which code points to move the cursor or select a sequence of code points based on what the user points to on the screen. Combining characters are a much more basic thing and need to be understood by pretty much anyone working with Unicode. /Richard From wsanchez@REDACTED Tue Jul 31 19:15:44 2012 From: wsanchez@REDACTED (=?ISO-8859-1?Q?Winston_S=E1nchez?=) Date: Tue, 31 Jul 2012 12:15:44 -0500 Subject: [erlang-questions] Chicagoboss doesn't work?? Message-ID: Hi everybody. I have problems with chicagoboss when i compile them, the application does not load routes and controllers don't work. Can anybody help me ?? -- -------------- next part -------------- An HTML attachment was scrubbed... URL: From dva.traktorista@REDACTED Tue Jul 31 19:26:35 2012 From: dva.traktorista@REDACTED (=?utf-8?B?ZHZhLnRyYWt0b3Jpc3RhQGdtYWlsLmNvbQ==?=) Date: Tue, 31 Jul 2012 20:26:35 +0300 Subject: [erlang-questions] =?utf-8?q?Chicagoboss_doesn=27t_work=3F=3F?= Message-ID: <5018157a.8790980a.6c9c.0d78@mx.google.com> Please be more verbose when you ask for help in the mailing list. Also, try to search for similar questions. For instance there was a good sir having troubles with launching ChicagoBoss on his Windows box recently. Answering to the two questions you asked in your mail: 1. ChicagoBoss does work. 2. Yes, there are people here (may be even myself) who can help you. Jonn Mostovoy (on the road), DA234FE7 ----- Reply message ----- From: "Winston S?nchez" Date: Tue, Jul 31, 2012 20:15 Subject: [erlang-questions] Chicagoboss doesn't work?? To: "erlang-questions" Hi everybody. I have problems with chicagoboss when i compile them, the application does not load routes and controllers don't work. Can anybody help me ?? -- -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.james.barry@REDACTED Tue Jul 31 19:31:58 2012 From: paul.james.barry@REDACTED (Paul Barry) Date: Tue, 31 Jul 2012 18:31:58 +0100 Subject: [erlang-questions] Chicagoboss doesn't work?? In-Reply-To: <5018157a.8790980a.6c9c.0d78@mx.google.com> References: <5018157a.8790980a.6c9c.0d78@mx.google.com> Message-ID: That latest CB (0.8.0) runs fine on Mac OS X and MongoDB (for me), and the latest Erlang (of course). This is also the *wrong list* to talk about this type of thing... as CB has its own mailing list. Paul. On 31 July 2012 18:26, dva.traktorista@REDACTED wrote: > Please be more verbose when you ask for help in the mailing list. > Also, try to search for similar questions. > For instance there was a good sir having troubles with launching ChicagoBoss > on his Windows box recently. > Answering to the two questions you asked in your mail: > 1. ChicagoBoss does work. > 2. Yes, there are people here (may be even myself) who can help you. > > Jonn Mostovoy (on the road), > DA234FE7 > > > ----- Reply message ----- > From: "Winston S?nchez" > Date: Tue, Jul 31, 2012 20:15 > Subject: [erlang-questions] Chicagoboss doesn't work?? > To: "erlang-questions" > > Hi everybody. > > I have problems with chicagoboss when i compile them, the application does > not load routes and controllers don't work. > > Can anybody help me ?? > -- > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Paul Barry, w: http://paulbarry.itcarlow.ie - e: paul.barry@REDACTED Lecturer, Computer Networking: Institute of Technology, Carlow, Ireland. From sam@REDACTED Tue Jul 31 19:35:24 2012 From: sam@REDACTED (Sam Elliott) Date: Tue, 31 Jul 2012 18:35:24 +0100 Subject: [erlang-questions] Erlang Message Guarantees Message-ID: <9422D177-3F3A-4BC4-85F0-54FE4E4A88A5@lenary.co.uk> Hi, Where I can find information on the inherent guarantees inbuilt into Erlang? I'm looking for information on whether order is preserved and associated issues. Sam From gleber.p@REDACTED Tue Jul 31 19:37:26 2012 From: gleber.p@REDACTED (Gleb Peregud) Date: Tue, 31 Jul 2012 19:37:26 +0200 Subject: [erlang-questions] Erlang Message Guarantees In-Reply-To: <9422D177-3F3A-4BC4-85F0-54FE4E4A88A5@lenary.co.uk> References: <9422D177-3F3A-4BC4-85F0-54FE4E4A88A5@lenary.co.uk> Message-ID: Take a look here: http://www.erlang.org/faq/academic.html at 10.10 On Tue, Jul 31, 2012 at 7:35 PM, Sam Elliott wrote: > Hi, > > Where I can find information on the inherent guarantees inbuilt into Erlang? > > I'm looking for information on whether order is preserved and associated issues. > > Sam > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From dan@REDACTED Tue Jul 31 19:49:34 2012 From: dan@REDACTED (Daniel Dormont) Date: Tue, 31 Jul 2012 13:49:34 -0400 Subject: [erlang-questions] Erlang Message Guarantees In-Reply-To: References: <9422D177-3F3A-4BC4-85F0-54FE4E4A88A5@lenary.co.uk> Message-ID: 10.9 above it as well - that has the answer to the question about delivery order. On Tue, Jul 31, 2012 at 1:37 PM, Gleb Peregud wrote: > Take a look here: http://www.erlang.org/faq/academic.html at 10.10 > > On Tue, Jul 31, 2012 at 7:35 PM, Sam Elliott wrote: > > Hi, > > > > Where I can find information on the inherent guarantees inbuilt into > Erlang? > > > > I'm looking for information on whether order is preserved and associated > issues. > > > > Sam > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From desired.mta@REDACTED Tue Jul 31 19:55:13 2012 From: desired.mta@REDACTED (Motiejus =?utf-8?Q?Jak=C5=A1tys?=) Date: Tue, 31 Jul 2012 20:55:13 +0300 Subject: [erlang-questions] Process migration/ Process state extraction/ injection In-Reply-To: References: Message-ID: <20120731175513.GA21145@localhost> On Tue, Jul 31, 2012 at 02:29:45PM +0200, Tyron Zerafa wrote: > Hello everyone, > Does Erlang support process migration? Meaning, is there a way in which > I can stop a running process (A) on one node and resume it on another > completely different node? > If this is not supported, one way to go around it is to extract the process > state before suspending it and somehow inject it into the new process on > the remote node. Does Erlang provide such functionality? > > Any tips and suggestions on how one can extract/ inject the process state > from/ into processes? Hi, there is no easy general way to do it. But, if you said what exactly you are trying to achieve, there might be an other, unexpected, easy way to do what you want. If you gave us a higher-level description, we might help you better. Motiejus -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: Digital signature URL: From tyron.zerafa@REDACTED Tue Jul 31 20:11:37 2012 From: tyron.zerafa@REDACTED (Tyron Zerafa) Date: Tue, 31 Jul 2012 20:11:37 +0200 Subject: [erlang-questions] Process migration/ Process state extraction/ injection In-Reply-To: <20120731175513.GA21145@localhost> References: <20120731175513.GA21145@localhost> Message-ID: Hey, My primary intention is to implement strong mobility of processes. Let's say that process A is running on Node 1, I want to be able to suspend this process, transfer it to Node 2 and resume such process from where it halted. I believe that in order to achieve such I need to somehow preserve the stack trace, memory and other info. On Tue, Jul 31, 2012 at 7:55 PM, Motiejus Jak?tys wrote: > On Tue, Jul 31, 2012 at 02:29:45PM +0200, Tyron Zerafa wrote: > > Hello everyone, > > Does Erlang support process migration? Meaning, is there a way in > which > > I can stop a running process (A) on one node and resume it on another > > completely different node? > > If this is not supported, one way to go around it is to extract the > process > > state before suspending it and somehow inject it into the new process on > > the remote node. Does Erlang provide such functionality? > > > > Any tips and suggestions on how one can extract/ inject the process state > > from/ into processes? > > Hi, > > there is no easy general way to do it. But, if you said what exactly you > are trying to achieve, there might be an other, unexpected, easy way to > do what you want. > > If you gave us a higher-level description, we might help you better. > > Motiejus > -- Best Regards, Tyron Zerafa -------------- next part -------------- An HTML attachment was scrubbed... URL: From 7stud@REDACTED Tue Jul 31 20:13:36 2012 From: 7stud@REDACTED (7stud) Date: Tue, 31 Jul 2012 14:13:36 -0400 Subject: [erlang-questions] installing erlang documentation Message-ID: <20120731141336.10964@web002.roc2.bluetie.com> Hi, I successfully installed Erlang on Mac OSX 10.6, after untaring and doing: ./configure make sudo make install and I've been practicing Erlang for a couple of weeks as I read "Erlang Programming". But I'm having no luck installing the erlang documentation. I downloaded the binary version of fop and set my JAVA_HOME environment variable to /Library/Java/Home, and I can successfully echo $JAVA_HOME, but the fop command is not recognized on the command line: -bash: fop: command not found so I assume that means that the elrang installation can't use the fop command to install the documentation either. At this point, I don't know what to do. $ java -version java version "1.6.0_15" Java(TM) SE Runtime Environment (build 1.6.0_15-b03-219) Java HotSpot(TM) 64-Bit Server VM (build 14.1-b02-90, mixed mode) $ From gleber.p@REDACTED Tue Jul 31 20:15:03 2012 From: gleber.p@REDACTED (Gleb Peregud) Date: Tue, 31 Jul 2012 20:15:03 +0200 Subject: [erlang-questions] Process migration/ Process state extraction/ injection In-Reply-To: References: <20120731175513.GA21145@localhost> Message-ID: On Tue, Jul 31, 2012 at 8:11 PM, Tyron Zerafa wrote: > Hey, > My primary intention is to implement strong mobility of processes. Let's > say that process A is running on Node 1, I want to be able to suspend this > process, transfer it to Node 2 and resume such process from where it halted. > I believe that in order to achieve such I need to somehow preserve the stack > trace, memory and other info. This task is non-trivial and will require a lot of work with ERTS and whole Erlang VM. Things you have to handle are: - stack - heap - binary refs - monitors - links - ets ownership - port ownership - messages sent to old process pid? - replacing old pid with new in other processes? - replacing old process with "replay" process? And I'm sure that those are not all details which will have to be handled. Cheers, Gleb From tyron.zerafa@REDACTED Tue Jul 31 20:21:40 2012 From: tyron.zerafa@REDACTED (Tyron Zerafa) Date: Tue, 31 Jul 2012 20:21:40 +0200 Subject: [erlang-questions] Process migration/ Process state extraction/ injection In-Reply-To: References: <20120731175513.GA21145@localhost> Message-ID: Hey again, I am thinking that a weaker form of such mobility might suffice for my project, but I am not sure. In this weaker form, it is enough to transfer the code and execution can restart again. I need this primary to implement the code-on-demand and remote-evaluation architectures. I do not believe that this is so complex in Erlang, in fact if I remember correctly some functionality already exists for such. On Tue, Jul 31, 2012 at 8:15 PM, Gleb Peregud wrote: > On Tue, Jul 31, 2012 at 8:11 PM, Tyron Zerafa > wrote: > > Hey, > > My primary intention is to implement strong mobility of processes. > Let's > > say that process A is running on Node 1, I want to be able to suspend > this > > process, transfer it to Node 2 and resume such process from where it > halted. > > I believe that in order to achieve such I need to somehow preserve the > stack > > trace, memory and other info. > > This task is non-trivial and will require a lot of work with ERTS and > whole Erlang VM. Things you have to handle are: > - stack > - heap > - binary refs > - monitors > - links > - ets ownership > - port ownership > - messages sent to old process pid? > - replacing old pid with new in other processes? > - replacing old process with "replay" process? > > And I'm sure that those are not all details which will have to be handled. > > Cheers, > Gleb > -- Best Regards, Tyron Zerafa -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.a.c.rijnders@REDACTED Tue Jul 31 21:01:58 2012 From: g.a.c.rijnders@REDACTED (Michel Rijnders) Date: Tue, 31 Jul 2012 21:01:58 +0200 Subject: [erlang-questions] installing erlang documentation In-Reply-To: <20120731141336.10964@web002.roc2.bluetie.com> References: <20120731141336.10964@web002.roc2.bluetie.com> Message-ID: On Tue, Jul 31, 2012 at 8:13 PM, 7stud <7stud@REDACTED> wrote: > Hi, > > I successfully installed Erlang on Mac OSX 10.6, after untaring and doing: > > ./configure > make > sudo make install > > and I've been practicing Erlang for a couple of weeks as I read "Erlang Programming". > > But I'm having no luck installing the erlang documentation. I downloaded the binary version of fop and set my JAVA_HOME environment variable to /Library/Java/Home, and I can successfully echo $JAVA_HOME, but the fop command is not recognized on the command line: > > -bash: fop: command not found > > so I assume that means that the elrang installation can't use the fop command to install the documentation either. At this point, I don't know what to do. You probably know this, but just to be sure, both HTML documentation and man pages are available as separate downloads from: http://www.erlang.org/download.html > > $ java -version > java version "1.6.0_15" > Java(TM) SE Runtime Environment (build 1.6.0_15-b03-219) > Java HotSpot(TM) 64-Bit Server VM (build 14.1-b02-90, mixed mode) > $ > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- My other car is a cdr. From eric@REDACTED Tue Jul 31 21:03:08 2012 From: eric@REDACTED (Eric Moritz) Date: Tue, 31 Jul 2012 15:03:08 -0400 Subject: [erlang-questions] unicode in string literals In-Reply-To: References: <50168ABD.50304@gmail.com> Message-ID: Source code in Latin-1 that used non-ASCII (>=127) bytes would be invalid UTF-8. On Tue, Jul 31, 2012 at 3:05 AM, Joe Armstrong wrote: > Is "encoding(...)" a good idea? > > There are four reasonable alternatives > > a) - all files are Latin1 > b) - all files are UTF8 > c) - all files are Latin1 or UTF8 and you guess > d) - all files are Latin1 or UTF8 or anything else and you tell > > Today we do a). > > What would be the consequences of changing to b) in (say) the next > major release? > > This would break some code - but how much? - how much code is there > with non Latin1 printable characters > in string literals? - it should be easy to write a program to test for > this and flag sting literals that > might causes problems if the default convention was changed. > > /Joe > > > > On Tue, Jul 31, 2012 at 12:44 AM, Richard O'Keefe wrote: >> The thing that puzzles me about Erlang assuming that source files are in >> Latin 1 is that I have a tokenizer for Erlang that assumes Latin 1 and >> in every Erlang/OTP release I've checked there has been at least one >> file it tripped up on because of UTF-8 characters. >> >> When can we expect -encoding('whatever'). to be supported? >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From ali.sabil@REDACTED Tue Jul 31 22:18:43 2012 From: ali.sabil@REDACTED (Ali Sabil) Date: Tue, 31 Jul 2012 22:18:43 +0200 Subject: [erlang-questions] Process migration/ Process state extraction/ injection In-Reply-To: References: <20120731175513.GA21145@localhost> Message-ID: Hi Tyron, You could implement process migration yourself, I remember seeing something like that implemented in ejabberd: https://github.com/esl/ejabberd/blob/master/apps/ejabberd/src/p1_fsm.erl#L557 The idea is to spawn another process in the remote node, and use a handover protocol to coordinate the migration. Best, Ali On Tue, Jul 31, 2012 at 8:21 PM, Tyron Zerafa wrote: > Hey again, > I am thinking that a weaker form of such mobility might suffice for my > project, but I am not sure. In this weaker form, it is enough to transfer > the code and execution can restart again. I need this primary to implement > the code-on-demand and remote-evaluation architectures. I do not believe > that this is so complex in Erlang, in fact if I remember correctly some > functionality already exists for such. > > > On Tue, Jul 31, 2012 at 8:15 PM, Gleb Peregud wrote: >> >> On Tue, Jul 31, 2012 at 8:11 PM, Tyron Zerafa >> wrote: >> > Hey, >> > My primary intention is to implement strong mobility of processes. >> > Let's >> > say that process A is running on Node 1, I want to be able to suspend >> > this >> > process, transfer it to Node 2 and resume such process from where it >> > halted. >> > I believe that in order to achieve such I need to somehow preserve the >> > stack >> > trace, memory and other info. >> >> This task is non-trivial and will require a lot of work with ERTS and >> whole Erlang VM. Things you have to handle are: >> - stack >> - heap >> - binary refs >> - monitors >> - links >> - ets ownership >> - port ownership >> - messages sent to old process pid? >> - replacing old pid with new in other processes? >> - replacing old process with "replay" process? >> >> And I'm sure that those are not all details which will have to be handled. >> >> Cheers, >> Gleb > > > > > -- > Best Regards, > Tyron Zerafa > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From brum76@REDACTED Tue Jul 31 22:33:01 2012 From: brum76@REDACTED (Radu Brumariu) Date: Tue, 31 Jul 2012 16:33:01 -0400 Subject: [erlang-questions] Python pickle protocol 0 to erlang Message-ID: Hi, is there a module that converts a python pickle into erlang term(s) ? Thanks, Radu From philippe.mclean@REDACTED Tue Jul 31 22:34:54 2012 From: philippe.mclean@REDACTED (Philippe McLean) Date: Tue, 31 Jul 2012 13:34:54 -0700 Subject: [erlang-questions] Process migration/ Process state extraction/ injection In-Reply-To: References: <20120731175513.GA21145@localhost> Message-ID: What goals are you trying to achieve? many problems can be solved with another level of indirection, but a robust approach to error handling and process creation might give the same or better results. On Tue, Jul 31, 2012 at 11:21 AM, Tyron Zerafa wrote: > Hey again, > I am thinking that a weaker form of such mobility might suffice for my > project, but I am not sure. In this weaker form, it is enough to transfer > the code and execution can restart again. I need this primary to implement > the code-on-demand and remote-evaluation architectures. I do not believe > that this is so complex in Erlang, in fact if I remember correctly some > functionality already exists for such. > > > On Tue, Jul 31, 2012 at 8:15 PM, Gleb Peregud wrote: > >> On Tue, Jul 31, 2012 at 8:11 PM, Tyron Zerafa >> wrote: >> > Hey, >> > My primary intention is to implement strong mobility of processes. >> Let's >> > say that process A is running on Node 1, I want to be able to suspend >> this >> > process, transfer it to Node 2 and resume such process from where it >> halted. >> > I believe that in order to achieve such I need to somehow preserve the >> stack >> > trace, memory and other info. >> >> This task is non-trivial and will require a lot of work with ERTS and >> whole Erlang VM. Things you have to handle are: >> - stack >> - heap >> - binary refs >> - monitors >> - links >> - ets ownership >> - port ownership >> - messages sent to old process pid? >> - replacing old pid with new in other processes? >> - replacing old process with "replay" process? >> >> And I'm sure that those are not all details which will have to be handled. >> >> Cheers, >> Gleb >> > > > > -- > Best Regards, > Tyron Zerafa > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From puzza007@REDACTED Tue Jul 31 22:46:12 2012 From: puzza007@REDACTED (Paul Oliver) Date: Tue, 31 Jul 2012 16:46:12 -0400 Subject: [erlang-questions] Python pickle protocol 0 to erlang In-Reply-To: References: Message-ID: I've seen https://github.com/jdavisp3/epryl used On Tue, Jul 31, 2012 at 4:33 PM, Radu Brumariu wrote: > Hi, > is there a module that converts a python pickle into erlang term(s) ? > > Thanks, > Radu > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.james.barry@REDACTED Tue Jul 31 22:56:20 2012 From: paul.james.barry@REDACTED (Paul Barry) Date: Tue, 31 Jul 2012 21:56:20 +0100 Subject: [erlang-questions] Python pickle protocol 0 to erlang In-Reply-To: References: Message-ID: Another approach might be to save your data to a file as JSON (as opposed to Pickle), then read the JSON into your Erlang process. Assuming, of course, you control the code that creates the Pickle in the first place... On 31 July 2012 21:46, Paul Oliver wrote: > I've seen https://github.com/jdavisp3/epryl used > > > On Tue, Jul 31, 2012 at 4:33 PM, Radu Brumariu wrote: >> >> Hi, >> is there a module that converts a python pickle into erlang term(s) ? >> >> Thanks, >> Radu >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Paul Barry, w: http://paulbarry.itcarlow.ie - e: paul.barry@REDACTED Lecturer, Computer Networking: Institute of Technology, Carlow, Ireland. From eric@REDACTED Tue Jul 31 22:58:25 2012 From: eric@REDACTED (Eric Moritz) Date: Tue, 31 Jul 2012 16:58:25 -0400 Subject: [erlang-questions] Python pickle protocol 0 to erlang In-Reply-To: References: Message-ID: You could also use the bert python module, but JSON is more portable. On Tue, Jul 31, 2012 at 4:56 PM, Paul Barry wrote: > Another approach might be to save your data to a file as JSON (as > opposed to Pickle), then read the JSON into your Erlang process. > Assuming, of course, you control the code that creates the Pickle in > the first place... > > On 31 July 2012 21:46, Paul Oliver wrote: >> I've seen https://github.com/jdavisp3/epryl used >> >> >> On Tue, Jul 31, 2012 at 4:33 PM, Radu Brumariu wrote: >>> >>> Hi, >>> is there a module that converts a python pickle into erlang term(s) ? >>> >>> Thanks, >>> Radu >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > > > -- > Paul Barry, w: http://paulbarry.itcarlow.ie - e: paul.barry@REDACTED > Lecturer, Computer Networking: Institute of Technology, Carlow, Ireland. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From brum76@REDACTED Tue Jul 31 23:25:30 2012 From: brum76@REDACTED (Radu Brumariu) Date: Tue, 31 Jul 2012 17:25:30 -0400 Subject: [erlang-questions] Python pickle protocol 0 to erlang In-Reply-To: References: Message-ID: Thanks all who responded. I cannot really change the protocol, otherwise I'd use Thrift. I am trying to inject a erlang piece in a python project, so it has to support pickle. I saw epryl, but that uses only pickle protocol 2. Still looking, if anyone is aware of anything. Thanks, Radu On Tue, Jul 31, 2012 at 4:58 PM, Eric Moritz wrote: > You could also use the bert python module, but JSON is more portable. > > On Tue, Jul 31, 2012 at 4:56 PM, Paul Barry wrote: >> Another approach might be to save your data to a file as JSON (as >> opposed to Pickle), then read the JSON into your Erlang process. >> Assuming, of course, you control the code that creates the Pickle in >> the first place... >> >> On 31 July 2012 21:46, Paul Oliver wrote: >>> I've seen https://github.com/jdavisp3/epryl used >>> >>> >>> On Tue, Jul 31, 2012 at 4:33 PM, Radu Brumariu wrote: >>>> >>>> Hi, >>>> is there a module that converts a python pickle into erlang term(s) ? >>>> >>>> Thanks, >>>> Radu >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >> >> >> >> -- >> Paul Barry, w: http://paulbarry.itcarlow.ie - e: paul.barry@REDACTED >> Lecturer, Computer Networking: Institute of Technology, Carlow, Ireland. >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From paperless@REDACTED Tue Jul 31 23:30:53 2012 From: paperless@REDACTED (Tim McNamara) Date: Wed, 1 Aug 2012 09:30:53 +1200 Subject: [erlang-questions] Python pickle protocol 0 to erlang In-Reply-To: References: Message-ID: Yes/No. Pickle is an object serialisation format. Translating arbitrary objects wont work. You can easily pickle Python functions, which are represented as references to the __main__ namespace. >>> import pickle >>> def sumr(*args): ... try: ... return args[0] + summa(*args[1:]) ... except IndexError: ... return 0 >>> print pickle.dumps(sumr, 0) c__main__ summa p0 . If you are transferring data, there are probably better alternatievs. People have promoted JSON in this thread. My personal preference lies with YAML when human readability counts, or Google Protocol Buffers for guarantees, speed & reliability. On 1 August 2012 08:33, Radu Brumariu wrote: > Hi, > is there a module that converts a python pickle into erlang term(s) ? > > Thanks, > Radu > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From paperless@REDACTED Tue Jul 31 23:32:54 2012 From: paperless@REDACTED (Tim McNamara) Date: Wed, 1 Aug 2012 09:32:54 +1200 Subject: [erlang-questions] Python pickle protocol 0 to erlang In-Reply-To: References: Message-ID: Apologies. Just noticed an error translating from shell to email and also wanting to rename the function. On 1 August 2012 09:30, Tim McNamara wrote: >>>> print pickle.dumps(sumr, 0) > c__main__ > summa > p0 > . This should be; >>> print pickle.dumps(sumr, 0) c__main__ sumr p0 . From zabrane3@REDACTED Tue Jul 31 23:32:39 2012 From: zabrane3@REDACTED (Zabrane Mickael) Date: Tue, 31 Jul 2012 23:32:39 +0200 Subject: [erlang-questions] External sorting for large files in Erlang Message-ID: Hi, I'm looking for something similar to this, but in Erlang: http://code.google.com/p/externalsortinginjava/ I found an old post suggesting file_sorter: http://www.erlang.org/doc/man/file_sorter.html But file_sorter seems to only work on binary files. In my cas, I need something more flexible. What about controlling the Unix sort command from Erlang? Any hints, ideas, suggestions, code? Regards, Zabrane -------------- next part -------------- An HTML attachment was scrubbed... URL: From brum76@REDACTED Tue Jul 31 23:57:16 2012 From: brum76@REDACTED (Radu Brumariu) Date: Tue, 31 Jul 2012 17:57:16 -0400 Subject: [erlang-questions] Python pickle protocol 0 to erlang In-Reply-To: References: Message-ID: Right, the upstream process is pushing pickled data, not objects. Again, cannot change the serialization protocol , at this point. Radu On Tue, Jul 31, 2012 at 5:32 PM, Tim McNamara wrote: > Apologies. Just noticed an error translating from shell to email and > also wanting to rename the function. > > On 1 August 2012 09:30, Tim McNamara wrote: >>>>> print pickle.dumps(sumr, 0) >> c__main__ >> summa >> p0 >> . > > This should be; > >>>> print pickle.dumps(sumr, 0) > c__main__ > sumr > p0 > .