From richard.youngkin@REDACTED Wed Apr 1 04:16:44 2015 From: richard.youngkin@REDACTED (Youngkin, Rich) Date: Tue, 31 Mar 2015 20:16:44 -0600 Subject: [erlang-questions] Erlang and Akka, The Sequel In-Reply-To: <20150330133920.GA75677@ferdair.local> References: <20150330133920.GA75677@ferdair.local> Message-ID: Fred & ok@REDACTED (not sure what else to call you :>), thanks for your thoughtful answers. They're helping to provide the insight I'm looking for. More below... On Mon, Mar 30, 2015 at 7:39 AM, Fred Hebert wrote: > On 03/29, Youngkin, Rich wrote: > >> Fred's response in [2] is interesting and informative, but it misses the >> mark in that it's up to the developer to know/understand how and when to >> implement the appropriate pattern. That's what capabilities such as >> gen_server are intended to eliminate. >> > > I'm not sure I understand the criticism there. [2] was about using > RPC/futures, and the answer shows how to use RPC/futures. > Maybe I'm splitting hairs here, but my point is that an RPC isn't a Future (even though the Erlang documentation on rpc:async_call/4 states that it implements "call streams with promises"). Futures can be implemented in Erlang using RPCs, but, to me, they're not equivalent semantic concepts. My intent wasn't to criticise your answer. Your answer does clearly show how to implement futures using RPCs and I found it extremely helpful. Garrett Smith gave a great presentation at last week's Erlang conference about "The Timeless Way of Building Erlang Apps" (see [3] and erlangpatterns.org). What I think could be useful is a "pattern" description for futures that incorporates the implementation using RPCs. I may just do this if someone doesn't beat me to it. > >> The example above is interesting from a semantic perspective, but it's >> mixing the happy path with failure handling. This is where the next >> concept/capability of Akka is interesting, namely the ability to compose >> operations while separating the failure handling path from the "happy >> path". Here's the follow-on example from the same course: >> >> > This is where a difference is made with these types/monads, yes! > > 1 val treasure: Try[Treasure] = >> 2 adventure.collectCoins().flatMap(coins => { >> 3 adventure.buyTreasure(coins) >> 4 }) >> 5 >> 6 treasure match { >> 7 case Success(successValue) => >> 9 do something like continue to the next challenge... >> 10 case Failure(errorValue) => >> 11 do something like make the character repeat the previous >> challenge... >> >> So the "happy path" of collectCoins() and buyTreasure() isn't intermingled >> with what to do if one or both of these operations fail. Specifically, >> buyTreasure() won't throw an exception if collectCoins() fails. I don't >> know of any way to express this in Erlang. >> > > Yes. So the regular `try .. catch` attempt would be: > > try > {ok, Cs} = adventure:collect_coins(), > Res = lists:flatmap(fun(Coins) -> > {ok, Val} = adventure:buy_treasure(Coins), > Val > end, Cs), > of > SuccessValue -> > %% Do something like continue to next challenge > catch > Type:Reason -> > %% Do something like maybe repeat the previous challenge > end. > > Given what we care about here is whether we actually failed *anywhere* or > *nowhere* (at least based on your failure value match), this is equivalent > to what you have. Interestingly, because you could be expected to see each > operation fail, you might also have a callee go for harder failures: > > try > lists:flatmap(fun(Coins) -> adventure:buy_treasure(Coins) end, > adventure:collect_coins()) > of > SuccessValue -> % Keep going > catch > _:_ -> % alt path > end > > The distinction is there, and really, the challenge is picking which one > to implement when you design the 'adventure' module. I'd argue for the > former if you expect multiple operations to 'fail' (it is likely that a > character cannot connect coins without it being a programmer error), so > that the caller can choose how to handle alternative branches at every > level. > I don't see a difference between the previous 2 Erlang implementations. Can you elaborate? > >> 1. Are these concepts generally useful or just interesting from an >> academic >> perspective? >> > > They are truly useful in my opinion, but how needed they are may depend on > what exception handling mechanism you have. Erlang does tend to have that > pattern made explicit with sequences of case expression (as in my last > example). Whether this is boilerplate that ought to be eliminated is likely > a question of personal preferences. > > 2. Would it be useful to support these capabilities as first-class >> concepts > > in Erlang (similar to gen_servers)? Or is this so trivial in Erlang that >> it's not worth making these first class capabilities? >> > > This is a more interesting question, because Erlang does have a lot of > ways to handle exceptions. I mean you can have actual exceptions, option > types, tagged values, multiple return values, signals, continuations, mixes > of them, and so on. > > ... Yes, I do see why you wouldn't want to limit Erlang's capabilities in this area, especially if it results in losing required contextual data regarding the failure. I also had a 3rd question regarding futures. Here it is from ok@ cs.otago.ac.nz's response: >> 3. Is there any way to express these capabilities in Erlang (in addition >> to >> the rpc:async_call as described by Fred in [2], which only covers Futures, >> and doesn't support composition)? > You've now changed the subject from Try to Futures. > I think you may have misunderstood > [2] http://erlang.org/pipermail/erlang-questions/2012-November/070679.html Yes, I did change the subject to Futures rather abruptly. Sorry about that. It's possible that I misunderstood [2] above, but I don't think so. So to press on this a little more, is it possible to compose futures in Erlang in a manner similar to *Try* as shown above? I think the answer is yes, I think it's just a matter of chaining together rpc:async calls passing along the returned "Key" to the remaining calls to rpc:async_call, e.g., 27> Fn = fun(Key) -> rpc:nb_yield(Key, 45000), timer:sleep(10), 42 end. #Fun 28> Key1 = rpc:async_call(node(), timer, sleep, [30000]). <0.74.0> 29> Key2 = rpc:async_call(node(), erlang, apply, [Fn,Key1]). <0.76.0> 30> rpc:nb_yield(Key2, 60000). Note, this example fails with a badrpc on the final line (30), but I'm guessing this can be worked out. To wrap up, Try, Future, Observable are useful concepts, perhaps worth expressing in a pattern someplace such as erlangpatterns.org. I don't think this has to take away from the ability to use "actual exceptions, option types, tagged values...." if they are more appropriate to the problem at hand and are needed retain needed contextual data about a failure. *Try* is a semantic that may be best expressed in Erlang as something like *-type try() :: any() | error*, and perhaps only in the context of a module. This example is a bit rough around the edges, but it's hopefully good enough to get the point across for now. Composing *Try* may be as simple as a try/catch block depending on the level of granularity that is needed and assuming that "let it crash" isn't appropriate. *Future* is a pattern that can be implemented using rpc:async. I think this is worth documenting as a pattern. *Observable" is something I didn't cover very well except to say that it's is future applied to a stream. This is a bit simplistic. An *Observable* is something that may produce events. To receive events, a subscription must be created by an *Observer*. After a subscription is created the *Observer* just listens for events. I can see how this can be easily accomplished in Erlang by registering an *Observer* (e.g., register self()) with an *Observable*) and then processing events as they arrive in the Observer's mailbox. In fact, this sounds a lot like gen_event. Thanks again for the comments. Additional comments are welcome. Cheers, Rich [3] https://www.youtube.com/watch?v=UUvU8cjCIcs&list=PLWbHc_FXPo2h0sJW6X2RZDtT1ndw6KKpQ&index=22 -------------- next part -------------- An HTML attachment was scrubbed... URL: From kenneth.lundin@REDACTED Wed Apr 1 11:02:22 2015 From: kenneth.lundin@REDACTED (Kenneth Lundin) Date: Wed, 1 Apr 2015 11:02:22 +0200 Subject: [erlang-questions] [ANN] Erlang/OTP 17.5 has been released Message-ID: Erlang/OTP 17.5 has been released. Erlang/OTP 17.5 is a service release on the 17 track with mostly bug fixes, but is does contain a number of new features and characteristics improvements as well. Some highlights of the release are: - ERTS: Added command line argument option for setting the initial size of process dictionaries. - Diameter: configurable incoming_max len and string_decode for diameter messages - Bugfixes and minor small features in applications such as compiler, common_test, crypto, debugger, eldap, erts, hipe, inets, ssh, ssl, ... - 43 contributions from 32 different contributors For more details see the README file at http://www.erlang.org/download/otp_src_17.5.readme You can download the full source distribution from http://www.erlang.org/download/otp_src_17.5.tar.gz Note: To unpack the TAR archive you need a GNU TAR compatible program. For installation instructions please read the README that is part of the distribution. You can also find this release at the official Erlang/OTP Git-repository at Github here: https://github.com/erlang/otp tagged "OTP-17.5" The Windows binary distribution can be downloaded from http://www.erlang.org/download/otp_win32_17.5.exe http://www.erlang.org/download/otp_win64_17.5.exe You can also download the complete HTML documentation or the Unix manual files http://www.erlang.org/download/otp_doc_html_17.5.tar.gz http://www.erlang.org/download/otp_doc_man_17.5.tar.gz We also want to thank those that sent us patches, suggestions and bug reports. The Erlang/OTP Team at Ericsson -------------- next part -------------- An HTML attachment was scrubbed... URL: From mononcqc@REDACTED Wed Apr 1 20:23:57 2015 From: mononcqc@REDACTED (Fred Hebert) Date: Wed, 1 Apr 2015 14:23:57 -0400 Subject: [erlang-questions] Erlang and Akka, The Sequel In-Reply-To: References: <20150330133920.GA75677@ferdair.local> Message-ID: <20150401182356.GE75677@ferdair.local> On 03/31, Youngkin, Rich wrote: >> try >> {ok, Cs} = adventure:collect_coins(), >> Res = lists:flatmap(fun(Coins) -> >> {ok, Val} = adventure:buy_treasure(Coins), >> Val >> end, Cs), >> of >> SuccessValue -> >> %% Do something like continue to next challenge >> catch >> Type:Reason -> >> %% Do something like maybe repeat the previous challenge >> end. >> ... >> try >> lists:flatmap(fun(Coins) -> adventure:buy_treasure(Coins) end, >> adventure:collect_coins()) >> of >> SuccessValue -> % Keep going >> catch >> _:_ -> % alt path >> end >> > >I don't see a difference between the previous 2 Erlang implementations. Can >you elaborate? > One of them matches on `{ok, Value}' to cause a failure, which makes the assumption 'bad' cases are returned as `undefined' or `{ok, Error}'. In the latter case it is expected that the called code raises exceptions when something goes wrong. They're two fundamentally different approaches to designing your interface. They can, of course, be mixed together depending on the importance (or frequency) of the error. Regards, Fred. From pierrefenoll@REDACTED Wed Apr 1 20:42:50 2015 From: pierrefenoll@REDACTED (Pierre Fenoll) Date: Wed, 1 Apr 2015 11:42:50 -0700 Subject: [erlang-questions] Warning not emitted on string matching clauses Message-ID: -module(wat_clauses). -export([authenticate_nouns/1]). authenticate_nouns([{<<"user_auth">>, _}]) -> 'true'; authenticate_nouns([{<<"user_auth">>, [<<"recovery">>]}]) -> hi; authenticate_nouns(_Nouns) -> 'false'. In this code (or above), the second clause will never match (because of source-order). But erlc does not complain about it. Ferd on IRC mentioned that the compiler might feel free to reorder clauses. But the call wat_clauses:authenticate_nouns([{<<"user_auth">>, [<<"recovery">>]}]) returns true instead of hi, so no reordering is done. Dialyzer does not complain either. Am I right in thinking we should have a warning about a never matching clause? -- Pierre Fenoll -------------- next part -------------- An HTML attachment was scrubbed... URL: From mononcqc@REDACTED Wed Apr 1 20:57:47 2015 From: mononcqc@REDACTED (Fred Hebert) Date: Wed, 1 Apr 2015 14:57:47 -0400 Subject: [erlang-questions] Warning not emitted on string matching clauses In-Reply-To: References: Message-ID: <20150401185745.GF75677@ferdair.local> On 04/01, Pierre Fenoll wrote: >Ferd on IRC mentioned that the compiler might feel free to reorder >clauses. The compiler would only reorder clauses that are free to do so. For example: f(a) -> 1; f(b) -> 2; f(X) when is_atom(X) -> 3; f([_|_]) -> 4; f({_}) -> 5. Would let the compiler possibly reorder 1 & 2, and maybe 4 & 5. But the presence of a guard clause or ambiguous matches would prevent it from doing more than that, if my understanding is accurate. From pierrefenoll@REDACTED Wed Apr 1 21:19:39 2015 From: pierrefenoll@REDACTED (Pierre Fenoll) Date: Wed, 1 Apr 2015 12:19:39 -0700 Subject: [erlang-questions] =?utf-8?q?=5BANN=5D_Other_Erldocs_=CE=B1?= Message-ID: Like erldocs.com? Now you can access a regularly-updated repository of your favorite Erlang projects' documentation over at http://other.erldocs.com/ You can search for a repository on the home page or just paste a repo's URL in the OpenSearch bar. Example: https://github.com/comtihon/mongodb-erlang ? A redirection to the other.erldocs.com page: http://other.erldocs.com/opensearch.html?q=https://github.com/comtihon/mongodb-erlang ? The other.erldocs.com page: http://other.erldocs.com/github.com/comtihon/mongodb-erlang/ ? Project's documentation for tag "v0.3.1": http://other.erldocs.com/github.com/comtihon/mongodb-erlang/v0.3.1/ ? Project's documentation for branch "devbranch": http://other.erldocs.com/github.com/comtihon/mongodb-erlang/devbranch/ ? Erlang repository information: http://other.erldocs.com/github.com/comtihon/mongodb-erlang/meta.txt ? Log of repo's build http://other.erldocs.com/github.com/comtihon/mongodb-erlang/_.txt Code for generating repos' erldocs: https://github.com/erldocs/erldocs_other Repo hosting the full webiste: https://github.com/erldocs/other.erldocs.com Notes: ? If an Erlang repo is not listed please submit an issue. ? meta.txt does not hold too much information for now, but should allow for statistical analysis later on. ? The website should be updated often enough (more than once a week) ? You'll notice some 404s: keep in mind this is a work in progress! Cheers -- Pierre Fenoll -------------- next part -------------- An HTML attachment was scrubbed... URL: From wallentin.dahlberg@REDACTED Wed Apr 1 21:28:52 2015 From: wallentin.dahlberg@REDACTED (=?UTF-8?Q?Bj=C3=B6rn=2DEgil_Dahlberg?=) Date: Wed, 1 Apr 2015 21:28:52 +0200 Subject: [erlang-questions] Warning not emitted on string matching clauses In-Reply-To: <20150401185745.GF75677@ferdair.local> References: <20150401185745.GF75677@ferdair.local> Message-ID: The compiler doesn't really "reorder" the clauses. It builds a match tree depending on types, values and variables. In the current implementation it does so by inspecting all clauses left to right. You can inspect the matching tree by compiling to kernel, i.e. erlc +to_kernel t.erl // Bj?rn-Egil 2015-04-01 20:57 GMT+02:00 Fred Hebert : > On 04/01, Pierre Fenoll wrote: > >> Ferd on IRC mentioned that the compiler might feel free to reorder >> clauses. >> > > The compiler would only reorder clauses that are free to do so. > > For example: > > f(a) -> 1; > f(b) -> 2; > f(X) when is_atom(X) -> 3; > f([_|_]) -> 4; > f({_}) -> 5. > > Would let the compiler possibly reorder 1 & 2, and maybe 4 & 5. But the > presence of a guard clause or ambiguous matches would prevent it from doing > more than that, if my understanding is accurate. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tuncer.ayaz@REDACTED Wed Apr 1 22:25:39 2015 From: tuncer.ayaz@REDACTED (Tuncer Ayaz) Date: Wed, 1 Apr 2015 22:25:39 +0200 Subject: [erlang-questions] [ANN] Erlang/OTP 18.0-rc1 is available for testing. In-Reply-To: References: Message-ID: On Thu, Mar 26, 2015 at 5:32 PM, Kenneth Lundin wrote: > Erlang/OTP 18.0-rc1 is available for testing. > > This is an alpha release, which will be followed by a planned beta > release in May and a final OTP 18.0 product release in June 2015. [...] > You can find the Release Notes with more detailed info at > > http://www.erlang.org/download/otp18rc1_relnotes.pdf Kenneth, any chance of reviving the .readme text file for the next archives? It has always been very convenient to fetch together with the source archive. From ok@REDACTED Thu Apr 2 01:56:24 2015 From: ok@REDACTED (ok@REDACTED) Date: Thu, 2 Apr 2015 12:56:24 +1300 Subject: [erlang-questions] Erlang and Akka, The Sequel In-Reply-To: References: <20150330133920.GA75677@ferdair.local> Message-ID: <0b564da6b36a82f44f4ddfaee798bbc8.squirrel@chasm.otago.ac.nz> > Maybe I'm splitting hairs here, but my point is that an RPC isn't a Future > (even though the Erlang documentation on rpc:async_call/4 states that it > implements "call streams with promises"). I don't think anyone is claiming that an RPC *is* a future. The claim is that a Future is just a process with a peculiarly restricted communication pattern. I mean by this that a future in *any* programming language is just a process (abstractly conceived) with a peculiarly restricted communication pattern. More precisely, as originally defined in https://www.dreamsongs.com/Files/hicss.pdf, a future is a shared synchronisation data structure through which that restricted communication is performed. That was why I pointed out that in my Smalltalk library there is no Future class; they are just joinable threads. The data structure that represents any kind of thread can perfectly well represent and support communication with a joinable thread. In Java you'd use a java.util.concurrent.FutureTask or one of its many relatives (java.util.concurrent is over the complexity event horizon and accelerating). I find the Java example compelling: if I want some threads communicating in a particular way in Java, as of 1.8 it is easier to go back to basics than to read all the library documentation and figure out what's going on. What saves the OTP behaviours from the same fate? Mainly the fact that there aren't very many of them, and the number isn't growing. Erlang/OTP seems to strike a very nice balance between a core of simple Erlang primitives and a limited palette of "big" behaviours that take a lot of the work off your hands. Futures fall into the "much needed gap". When you might reach for a FutureTask in Java, in Erlang you would think about what the processes are doing and how they communicate and you would look at the constellation of communicating processes and you might very well end up thinking that the Future-style communication pattern was *not* after all the best. For example, the QLISP paper says "we generalize the notion of a future to allow several processes to be associated with it, along with a combining function. As each process nishes, it calls the combining function with the value of the form it has just nished computing. When all of the processes have completed, the future will be realized." They also allow an end-test predicate so that you can do OR-parallel searches, killing off any remaining processes as soon as the final result is known. This is an example of taking a step back and realising that the restricted communication of a Future is *TOO* restricted. > Futures can be implemented in > Erlang using RPCs, but, to me, they're not equivalent semantic concepts. No. RPCs are strictly more powerful than futures. > Garrett Smith gave a great presentation at last week's Erlang conference > about "The Timeless Way of Building Erlang Apps" (see [3] and > erlangpatterns.org). What I think could be useful is a "pattern" > description for futures that incorporates the implementation using RPCs. I > may just do this if someone doesn't beat me to it. For me, the single most important piece of advice is DON'T THINK IN TERMS OF FUTURES AND THEN ENCODE THAT IN ERLANG. Think in terms of processes and the communication between them. Consider alternative communication architectures. > It's possible that I misunderstood [2] above, but I don't think so. So to > press on this a little more, is it possible to compose futures in Erlang > in > a manner similar to *Try* as shown above? Be more specific. Give a real example. Are you talking about making a Future> instance? Why do you want to do this? If you started by thinking about concurrent activities (without reference to any programming language) and the communication patterns between them (ditto), for what kind of problem would this be a live issue? > To wrap up, Try, Future, Observable are useful concepts, in some languages. ("Try" is a really *horrible* name for ML's "option"/Haskell's "Maybe". It just doesn't work grammatically.) "Try" is somewhere between "trivial" and "obfuscatory". "Future" is just a strangely restricted process, putting it also somewhere between "trivial" and "obfuscatory" in Erlang. > *Observable" is something I didn't cover very well except to say that it's > is future applied to a stream. This is a bit simplistic. An *Observable* > is something that may produce events. In other words, it's a process that may send messages. Sounds like any other Erlang process, really. > To receive events, a subscription > must be created by an *Observer*. This whole Observer/Observable thing goes back to the Smalltalk-80 "dependency" machinery, which is actually a trivial chunk of Smalltalk code. It's 78 lines in my Smalltalk library, about 1/3 of that is convenience interface and another 1/3 is locking. The Gang Of Four book succeeded in making it look non-trivial, indeed complex, but trivial is what it is in any good OO language. Indeed, I'd have to say somewhere between trivial and obfuscatory, as it can make it quite hard to see the connections between things. In Erlang, one approach would be to make a simple process that receives three basic kinds of messages: - I want to subscribe - I want to stop subscribing - Here is a message to forward to the subscribers observable:spawn(Fun) would create two processes. The first one manages the subscriptions. Its identity would be passed to the Fun in the other process. It would be hard to spend a whole afternoon implementing this. Smalltalk teaches us a lot here. Because the original simple machinery described in the Blue Book, in Inside Smalltalk, and elsewhere proved to be a bit *too* simple. Object Arts decided that you needed to be able to *test* this stuff in SUnit, which required planting hooks for SUnit to use. Another complete observer implementation called TriggerEvent was implemented. This is sometimes called the Self-Addressed Stamped Envelope pattern. And then Vassili Bykov implemented a *third* scheme, called Announcements, originally for VisualWorks, but used by a couple of other Smalltalks as well. So the Smalltalk world has (at least) *three* observer interfaces. Having implemented all three in my system, it's painfully obvious that they *all* have problems. The saving thing about Smalltalk is that none of them needs compiler support. On the evidence, I don't trust *anyone* to design an Observable kit well, myself included. So you should think of "future" as a *Design Pattern*, not a built-in type, which a programmer should be *able to implement appropriately for their needs*, not a ready-made strait-jacket. And you should think of "Observable" in the same way as a *Design Pattern*, which people can *easily implement*. Of course, in Erlang, the existing tracing machinery is probably a good place to start for Observable. But you should *always* start by thinking about what the processes want to be and how they want to communicate *before* trying to squeeze them into no-thinking-required moulds. package was created and adopted by some but not all Smalltalks. From henrik@REDACTED Thu Apr 2 08:06:05 2015 From: henrik@REDACTED (Henrik Nord) Date: Thu, 2 Apr 2015 08:06:05 +0200 Subject: [erlang-questions] [ANN] Erlang/OTP 18.0-rc1 is available for testing. In-Reply-To: References: Message-ID: <551CDC4D.8090300@erlang.org> On 2015-04-01 22:25, Tuncer Ayaz wrote: > On Thu, Mar 26, 2015 at 5:32 PM, Kenneth Lundin wrote: >> Erlang/OTP 18.0-rc1 is available for testing. >> >> This is an alpha release, which will be followed by a planned beta >> release in May and a final OTP 18.0 product release in June 2015. > [...] > >> You can find the Release Notes with more detailed info at >> >> http://www.erlang.org/download/otp18rc1_relnotes.pdf > Kenneth, any chance of reviving the .readme text file for the > next archives? It has always been very convenient to fetch > together with the source archive. We do intend to deliver that readme. As the release notes normally are one of the last things to get into place before a new release, it was simply not even close to complete at that time. It made more sense to give you the highlights. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- /Henrik Nord Erlang/OTP From richard.youngkin@REDACTED Thu Apr 2 21:42:22 2015 From: richard.youngkin@REDACTED (Youngkin, Rich) Date: Thu, 2 Apr 2015 13:42:22 -0600 Subject: [erlang-questions] Erlang and Akka, The Sequel In-Reply-To: <0b564da6b36a82f44f4ddfaee798bbc8.squirrel@chasm.otago.ac.nz> References: <20150330133920.GA75677@ferdair.local> <0b564da6b36a82f44f4ddfaee798bbc8.squirrel@chasm.otago.ac.nz> Message-ID: Thanks Fred for clarifying the example in your original response. Thanks too to OK for the additional information, more below. Cheers, Rich On Mon, Mar 30, 2015 at 7:39 AM, Fred Hebert wrote: > On 03/31, Youngkin, Rich wrote: > try >> {ok, Cs} = adventure:collect_coins(), >> Res = lists:flatmap(fun(Coins) -> >> {ok, Val} = adventure:buy_treasure(Coins), >> Val >> end, Cs), >> of >> SuccessValue -> >> %% Do something like continue to next challenge >> catch >> Type:Reason -> >> %% Do something like maybe repeat the previous challenge >> end. >> ... >> try >> lists:flatmap(fun(Coins) -> adventure:buy_treasure(Coins) end, >> adventure:collect_coins()) >> of >> SuccessValue -> % Keep going >> catch >> _:_ -> % alt path >> end >> >> > >> I don't see a difference between the previous 2 Erlang > implementations. Can > >> you elaborate? > > > One of them matches on `{ok, Value}' to cause a failure, which makes the assumption 'bad' cases are returned > as `undefined' or `{ok, Error}'. In the latter case it is expected that the called code raises exceptions when something > goes wrong. Got it, thanks! On Wed, Apr 1, 2015 at 5:56 PM, wrote: > > I find the Java example compelling: if I want some threads > communicating in a particular way in Java, as of 1.8 it is > easier to go back to basics than to read all the library > documentation and figure out what's going on. > > What saves the OTP behaviours from the same fate? > Mainly the fact that there aren't very many of them, > and the number isn't growing. Erlang/OTP seems to strike > a very nice balance between a core of simple Erlang > primitives and a limited palette of "big" behaviours that > take a lot of the work off your hands. Futures fall into > the "much needed gap". > Interesting point. One of the things I like about Erlang is it's simplicity. I don't quite follow what you mean by "Futures fall into the "much needed gap"". Are you saying that futures should, or should not, be added to Erlang/OTP? I'm thinking not. > > When you might reach for a FutureTask in Java, in > Erlang you would think about what the processes are > doing and how they communicate and you would look at the > constellation of communicating processes and you might > very well end up thinking that the Future-style > communication pattern was *not* after all the best. > > Agreed. Understanding the problem and picking an appropriate solution is always the best approach. Blindly implementing a solution, either because you don't really understand the problem or because something is cool (e.g., futures), is never good. > > Garrett Smith gave a great presentation at last week's Erlang conference > > about "The Timeless Way of Building Erlang Apps" (see [3] and > > erlangpatterns.org). What I think could be useful is a "pattern" > > description for futures that incorporates the implementation using RPCs. > I > > may just do this if someone doesn't beat me to it. > > For me, the single most important piece of advice is > DON'T THINK IN TERMS OF FUTURES AND THEN ENCODE THAT IN ERLANG. > Think in terms of processes and the communication between them. > Consider alternative communication architectures. > One of my motivations in starting this email thread is due to my relative newness to Erlang. As might be apparent from some of my questions, I don't necessarily have a good understanding of the best practices for solving a given class of problems. For someone like me it would be good to at least have code examples of how to implement certain best practices. These examples could also include information on what problems they might not be appropriate to for. Regarding futures, I'm assuming, perhaps mistakenly, that they could be useful in some Erlang applications. Regarding best practices, this is the area that I believe erlangpatterns.org is intended to address. To reiterate what I said above, one shouldn't blindly implement a solution without considering when it is appropriate and when it isn't appropriate. This is what I think you meant when you stated above "DON'T THINK IN TERMS OF FUTURES AND THEN ENCODE THAT IN ERLANG". -------------- next part -------------- An HTML attachment was scrubbed... URL: From lloyd@REDACTED Fri Apr 3 05:29:33 2015 From: lloyd@REDACTED (Lloyd R. Prentice) Date: Thu, 2 Apr 2015 23:29:33 -0400 Subject: [erlang-questions] Nitrogen-based XMPP client? Message-ID: Hello, I'm considering integrating multi-user chat into a Nitrogen application I'm building--- assuming that I'll support it with Mongoose IM. As usual, I'm in over my head, but persistent. I'm imagining a small number ( four? six?) thematic chat rooms serving a relatively small number of visitors at a time. As to scale, think of a book-signing event at a bookstore. I see that there are several JavaScript XMPP client packages that I could integrate into my Nitrogen web pages but, if that's the way, which one? Or, is there a native Erlang XMPP client app that I should look at? Indeed, is Mongoose IM overkill? After all, the Nitrogen demos provide a very simple tinker-toy chat example, and numerous others can be found on the web though, most, rather long in the tooth. And more, how much harder would it be to integrate video chat using, say, web sockets? In other words, how can I best keep the gotchas, demons, and goblins at bay sufficiently to master the arcane arts of multi-user chat sufficiently to build my application before I die of old age? I'd much appreciate pointers or, better yet, pointers to actual code. All the best, LRP Sent from my iPad From marc@REDACTED Fri Apr 3 09:30:02 2015 From: marc@REDACTED (Marc Worrell) Date: Fri, 3 Apr 2015 09:30:02 +0200 Subject: [erlang-questions] Nitrogen-based XMPP client? In-Reply-To: References: Message-ID: Hi Lloyd, Did you have a look at the exmpp client library? There are some tutorials which might help in getting you started. http://processone.github.io/exmpp/ I don?t know of any other xmpp client libraries. Cheers, Marc > On 3 apr. 2015, at 05:29, Lloyd R. Prentice wrote: > > Hello, > > I'm considering integrating multi-user chat into a Nitrogen application I'm building--- assuming that I'll support it with Mongoose IM. As usual, I'm in over my head, but persistent. > > I'm imagining a small number ( four? six?) thematic chat rooms serving a relatively small number of visitors at a time. As to scale, think of a book-signing event at a bookstore. > > I see that there are several JavaScript XMPP client packages that I could integrate into my Nitrogen web pages but, if that's the way, which one? Or, is there a native Erlang XMPP client app that I should look at? Indeed, is Mongoose IM overkill? After all, the Nitrogen demos provide a very simple tinker-toy chat example, and numerous others can be found on the web though, most, rather long in the tooth. > > And more, how much harder would it be to integrate video chat using, say, web sockets? > > In other words, how can I best keep the gotchas, demons, and goblins at bay sufficiently to master the arcane arts of multi-user chat sufficiently to build my application before I die of old age? > > I'd much appreciate pointers or, better yet, pointers to actual code. > > All the best, > > LRP > > Sent from my iPad > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang@REDACTED Fri Apr 3 11:13:33 2015 From: erlang@REDACTED (Joe Armstrong) Date: Fri, 3 Apr 2015 11:13:33 +0200 Subject: [erlang-questions] Erlang and Akka, The Sequel In-Reply-To: References: <20150330133920.GA75677@ferdair.local> <0b564da6b36a82f44f4ddfaee798bbc8.squirrel@chasm.otago.ac.nz> Message-ID: On Thu, Apr 2, 2015 at 9:42 PM, Youngkin, Rich wrote: > Thanks Fred for clarifying the example in your original response. Thanks > too to OK for the additional information, more below. > > Cheers, > Rich > > On Mon, Mar 30, 2015 at 7:39 AM, Fred Hebert wrote: >> On 03/31, Youngkin, Rich wrote: >>> >>> try >>> {ok, Cs} = adventure:collect_coins(), >>> Res = lists:flatmap(fun(Coins) -> >>> {ok, Val} = adventure:buy_treasure(Coins), >>> Val >>> end, Cs), >>> of >>> SuccessValue -> >>> %% Do something like continue to next challenge >>> catch >>> Type:Reason -> >>> %% Do something like maybe repeat the previous challenge >>> end. >>> ... >>> try >>> lists:flatmap(fun(Coins) -> adventure:buy_treasure(Coins) end, >>> adventure:collect_coins()) >>> of >>> SuccessValue -> % Keep going >>> catch >>> _:_ -> % alt path >>> end >>> >> >> >> I don't see a difference between the previous 2 Erlang implementations. >> >> Can >> >> you elaborate? >> > >> One of them matches on `{ok, Value}' to cause a failure, which makes the >> assumption 'bad' cases are returned >> as `undefined' or `{ok, Error}'. In the latter case it is expected that >> the called code raises exceptions when something >> goes wrong. > > Got it, thanks! > > On Wed, Apr 1, 2015 at 5:56 PM, wrote: >> >> >> I find the Java example compelling: if I want some threads >> communicating in a particular way in Java, as of 1.8 it is >> easier to go back to basics than to read all the library >> documentation and figure out what's going on. >> >> What saves the OTP behaviours from the same fate? >> Mainly the fact that there aren't very many of them, >> and the number isn't growing. Erlang/OTP seems to strike >> a very nice balance between a core of simple Erlang >> primitives and a limited palette of "big" behaviours that >> take a lot of the work off your hands. Futures fall into >> the "much needed gap". > > > Interesting point. One of the things I like about Erlang is it's simplicity. > I don't quite follow what you mean by "Futures fall into the "much needed > gap"". Are you saying that futures should, or should not, be added to > Erlang/OTP? I'm thinking not. > >> >> >> When you might reach for a FutureTask in Java, in >> Erlang you would think about what the processes are >> doing and how they communicate and you would look at the >> constellation of communicating processes and you might >> very well end up thinking that the Future-style >> communication pattern was *not* after all the best. >> > > Agreed. Understanding the problem and picking an appropriate solution is > always the best approach. Blindly implementing a solution, either because > you don't really understand the problem or because something is cool (e.g., > futures), is never good. > >> >> > Garrett Smith gave a great presentation at last week's Erlang conference >> > about "The Timeless Way of Building Erlang Apps" (see [3] and >> > erlangpatterns.org). What I think could be useful is a "pattern" >> > description for futures that incorporates the implementation using RPCs. >> > I >> > may just do this if someone doesn't beat me to it. >> >> For me, the single most important piece of advice is >> DON'T THINK IN TERMS OF FUTURES AND THEN ENCODE THAT IN ERLANG. >> Think in terms of processes and the communication between them. >> Consider alternative communication architectures. > > > One of my motivations in starting this email thread is due to my relative > newness to Erlang. As might be apparent from some of my questions, I don't > necessarily have a good understanding of the best practices for solving a > given class of problems. For someone like me it would be good to at least > have code examples of how to implement certain best practices. These > examples could also include information on what problems they might not be > appropriate to for. Regarding futures, I'm assuming, perhaps mistakenly, > that they could be useful in some Erlang applications. Regarding best > practices, this is the area that I believe erlangpatterns.org is intended to > address. > > To reiterate what I said above, one shouldn't blindly implement a solution > without considering when it is appropriate and when it isn't appropriate. > This is what I think you meant when you stated above "DON'T THINK IN TERMS > OF FUTURES AND THEN ENCODE THAT IN ERLANG". Promises, futures and so on are extremely easy to implement in Erlang. This is how I explain things when I teach Erlang: We'll start with an RPC - we can write this in many ways. One way might be: rpc(Pid, Query) -> Ref = make_ref(), Pid ! {self(), Ref, Query}, receive {Ref, Response} -> Response end. The server that receives this message does something like: receive ... {From, Ref, Query} -> Response = ... From ! {Ref, Response} ... end, This basic pattern is repeated all over the place is with many small variations. For example, all the gen_server does is wrap this pattern with a few convenience functions. Now keeping the server code unchanged we can modify the RPC Start with the original (and stare at the added comment): rpc(Pic, Query) -> Ref = make_ref(), Pid ! {self(), Ref, Query}, %% watch this space ************** receive {Ref, Response} -> Response end. Now I'll rename rpc as rpc1 and split it into two functions at the comment: rpc1(Pid, Query) -> Ref = make_ref(), Pid ! {self(), Ref, Query}, Ref. wait(Ref) -> receive {Ref, Response} -> Response end. So obviously rpc(Pid, Query) -> Ref = rpc1(Pid, Query), wait(Ref). How about some renaming? I'll call rpc1 "promise" and wait "yield" So promise(Pid, Query) -> Ref = make_ref(), Pid ! {self(), Ref, Query}, Ref. yield(Ref) -> receive {Ref, Response} -> Response end. Now we can do something in the gap between the promise and the yield: compute_something(...) -> P1 = promise(...) Val1 = ... some local computation ... Val2 = yield(P1), ... So now Val1 and Val2 are computed in parallel. (We've now invented one of the basic mechanisms for parallel programming this might appear as parbegin ... parend in some programming language :-) The *reason* why Erlang does not have futures/promises/ .. or whatever else you might like to call them is that they are trivially implemented in a few lines of code using just spawn/send/receive. In languages that are basically sequential this is not possible - that's why it's a big deal (TM) to have libraries or language features to support this. And now for the tricky part .... *Broken Promises* - Remember that scene in Casablanca when Iisa confronts Rick, this is all about broken promises. Fulfilled promises are easy to deal with, but we must ask what happens if the server crashes and never sends back a message? Now life gets difficult, and as Rick found out the consequences of a broken promise lead to all sorts of problems... In Erlang, dealing with broken promises is possible (though not easy) using links, monitors and by trapping exits. The spawn/send/receive group of primitives are used to program the non error cases where things don't go wrong. trap_exits/links/monitors deal with error cases where things go wrong. The gen_servers, supervision trees and so on just wrap these primitives in combinations that we have found useful for solving "common" problems. In cases where the library behaviors don't do *exactly* what you want it's often easier to "roll you own" rather than shoehorning your problem into the wrong solution. The reason there are not a load of design patterns is that we don't need them. We do need to teach the basics though. spawn/send/receive are as basic to Erlang as for/if/case are to sequential programming This is where we have a problem - in sequential languages nobody bothers to teach what for/if/case/switch/while etc do - it is "implicit knowledge that all programmers have" (I'm excluding total beginners here) - Experienced Erlang programers know the power of spawn/send/receive so rarely bother to explain how to build things with these primitives. My advice would be to stare hard at spawn/send/receive and *memorise* the RPC pattern and the basic cleint/server setup. Then understand links. Write things without using the libraries - then learn the libraries. Note I said you can write RPC in "many" ways - what did I mean by this? To illustrate, here's are some variations: rpc(Pid, Query) -> Pid ! {self(), Query}, receive {Pid, Response} -> Response end. or rpc(Pid, Query) -> Pid ! {self(), Query}, receive {Pid, Response} -> Response after 10000 -> exit(timout) end. or rpc(Pid, Query) -> Pid ! {self(), Query}, receive {Pid, Response} -> {ok, Response} after 10000 -> {error, timeout} end. You can see how the basic pattern remains - we can add timeouts etc. but then we have to decide what to do with the variants. Only the most common patterns are found in the gen_server so it's a good idea to understand the basic pattern and modify it to your specific needs. /Joe > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From essen@REDACTED Fri Apr 3 11:18:10 2015 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Fri, 03 Apr 2015 12:18:10 +0300 Subject: [erlang-questions] crypto EVP transition In-Reply-To: References: Message-ID: <551E5AD2.5040308@ninenines.eu> On 03/28/2015 02:48 PM, Maas-Maarten Zeeman wrote: > This is the current proof-of-concept api, minus meta-data functions for retrieving block size, iv,- and key length, and a bytes to key function: > > -spec cipher_init(binary(), binary(), binary(), encrypt | decrypt, boolean()) -> {ok, cipher_ctx()}. > cipher_init(_Alg, _Key, _Iv, _Mode, _Padding) -> > exit(nif_library_not_loaded). > > -spec cipher_update(cipher_ctx(), iolist()) -> binary(). > cipher_update(_Ctx, _Data) -> > exit(nif_library_not_loaded). > > -spec cipher_final(cipher_ctx()) -> binary(). > cipher_final(_Ctx) -> > exit(nif_library_not_loaded). > > Last word from the otp-team was that they think this api is bit too un-erlang because of the rather ugly mutating cipher context. There is not much I can do about that. Openssl does not have a context copy function in its api. Copying the context from call to call could also be impossible when the cipher is implemented in hardware. Perhaps it can be made similar to zlib, where the context is a port()? Or would that kill the performance benefits? > I?m not sure if all this is on track for 18.0 though. I hope something can be done in the near future. HTTP is increasingly depending on TLS and good performance will soon matter to a lot more people than before. It sounds like the ball is currently in OTP Team's camp but if I can help with it I will. Cheers, -- Lo?c Hoguin http://ninenines.eu From gfborn@REDACTED Fri Apr 3 12:10:46 2015 From: gfborn@REDACTED (Grigory Fateyev) Date: Fri, 3 Apr 2015 13:10:46 +0300 Subject: [erlang-questions] Nitrogen-based XMPP client? In-Reply-To: References: Message-ID: <20150403131046.46149dd5@gmail.com> Hello Marc Worrell! On Fri, 3 Apr 2015 09:30:02 +0200 you wrote: > Hi Lloyd, > > Did you have a look at the exmpp client library? > There are some tutorials which might help in getting you started. > > http://processone.github.io/exmpp/ > > > I don?t know of any other xmpp client libraries. I have a very simple (hope in future it'll be more useful) a pet project xmpp client that connects to rooms and does simple things. Like a bot. I found exmpp more ready to use it like an example, but the author of exmpp said: "Just as a head up, note that post is > 5 years old already. The main goal of exmpp itself was to be used as parser/xml core in ejabberd". So I started my project as simple request-response flow with server. The main problem is to read XMPP XEPs. :) > > On 3 apr. 2015, at 05:29, Lloyd R. Prentice > > wrote: > > > > Hello, > > > > I'm considering integrating multi-user chat into a Nitrogen > > application I'm building--- assuming that I'll support it with > > Mongoose IM. As usual, I'm in over my head, but persistent. > > > > I'm imagining a small number ( four? six?) thematic chat rooms > > serving a relatively small number of visitors at a time. As to > > scale, think of a book-signing event at a bookstore. > > > > I see that there are several JavaScript XMPP client packages that I > > could integrate into my Nitrogen web pages but, if that's the way, > > which one? Or, is there a native Erlang XMPP client app that I > > should look at? Indeed, is Mongoose IM overkill? After all, the > > Nitrogen demos provide a very simple tinker-toy chat example, and > > numerous others can be found on the web though, most, rather long > > in the tooth. > > > > And more, how much harder would it be to integrate video chat > > using, say, web sockets? > > > > In other words, how can I best keep the gotchas, demons, and > > goblins at bay sufficiently to master the arcane arts of multi-user > > chat sufficiently to build my application before I die of old age? > > > > I'd much appreciate pointers or, better yet, pointers to actual > > code. > > > > All the best, > > > > LRP > > > > Sent from my iPad > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > -- Best regards! gfborn [at] gmail [dot] com From gomoripeti@REDACTED Fri Apr 3 15:55:18 2015 From: gomoripeti@REDACTED (=?ISO-8859-1?Q?Peti_G=F6m=F6ri?=) Date: Fri, 3 Apr 2015 15:55:18 +0200 Subject: [erlang-questions] Inet config - don't monitor config files Message-ID: Hi all, We are not using native DNS lookup but Erlang's own implementation. We would like to read config files at startup but we don't want inet to monitor them for changes. I was happy to find in the documentation an example that does just that for the hosts file ( http://www.erlang.org/doc/apps/erts/inet_cfg.html#id86806 ) %% -- ERLANG INET CONFIGURATION FILE -- %% read the hosts file {file, hosts, "/etc/hosts"}. %% do not monitor the hosts file {hosts_file, ""}. Great, this works fine so I tried the same for resolv.conf: %% read nameserver config from the resolv.conf file {file, resolv, "/etc/resolv.conf"}. %% do not monitor the resolv.conf file {resolv_conf, ""}. Unfortunately this does not work for resolv.conf because inet_config will remove all previous nameserver entries upon processing the {resolv_conf, _} parameter and it always processes the {file, _, _} parameters first (so reordering the config file does not help either). Actually there is also problem if I don't specify any resolv_conf option. Although the default resolv.conf won't be read at startup (because of this commit https://github.com/erlang/otp/commit/0377ecdfb001c4dc1b507ebc06211f29b19048e3) but it will be at the first name lookup and it will remove previous entries too. See attached logs with empty and default resolv_conf settings respectively. (Using R16B03-1 but there were hardly any changes on the relevant files since.) So I don't see any way how {file, resolv, _} option could have any effect. Is this intentional or a bug? How should the desired goal be achieved? If this is a bug I would be happy to create a patch with some guidance - should the empty string be special cased to not clear the previous config? thanks Peter -------------- next part -------------- A non-text attachment was scrubbed... Name: empty_resolv_conf.log Type: application/octet-stream Size: 774 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: default_resolv_conf.log Type: application/octet-stream Size: 899 bytes Desc: not available URL: From roberto@REDACTED Fri Apr 3 17:59:57 2015 From: roberto@REDACTED (Roberto Ostinelli) Date: Fri, 3 Apr 2015 08:59:57 -0700 Subject: [erlang-questions] Date parsing (RFC 7231) Message-ID: All, The Date field of HTTP headers uses date format as defined in RFC 7231, for example: Date: Tue, 15 Nov 1994 08:12:31 GMT What is the fastest way to parse this header and convert it to an epoch time? Best, r. -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Fri Apr 3 18:07:14 2015 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Fri, 03 Apr 2015 19:07:14 +0300 Subject: [erlang-questions] Date parsing (RFC 7231) In-Reply-To: References: Message-ID: <551EBAB2.6040901@ninenines.eu> cow_date:parse_date(Bin). Returns {{Y,Mo,D},{H,Mi,S}}. On 04/03/2015 06:59 PM, Roberto Ostinelli wrote: > All, > The Date field of HTTP headers uses date format as defined in RFC 7231, > for example: > > Date: Tue, 15 Nov 1994 08:12:31 GMT > > What is the fastest way to parse this header and convert it to an epoch > time? > > Best, > r. > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin http://ninenines.eu From gumm@REDACTED Fri Apr 3 18:09:58 2015 From: gumm@REDACTED (Jesse Gumm) Date: Fri, 3 Apr 2015 11:09:58 -0500 Subject: [erlang-questions] Date parsing (RFC 7231) In-Reply-To: References: Message-ID: qdate:to_unixtime(DateStringOrBin) should do the trick for you, with the timezone awareness. On Apr 3, 2015 11:00 AM, "Roberto Ostinelli" wrote: > All, > The Date field of HTTP headers uses date format as defined in RFC 7231, > for example: > > Date: Tue, 15 Nov 1994 08:12:31 GMT > > What is the fastest way to parse this header and convert it to an epoch > time? > > Best, > r. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto@REDACTED Fri Apr 3 18:09:57 2015 From: roberto@REDACTED (Roberto Ostinelli) Date: Fri, 3 Apr 2015 09:09:57 -0700 Subject: [erlang-questions] Date parsing (RFC 7231) In-Reply-To: <551EBAB2.6040901@ninenines.eu> References: <551EBAB2.6040901@ninenines.eu> Message-ID: I was playing with: io_lib:fread("~3s, ~2d ~3s ~4d ~2d:~2d:~2d GMT", Date7231) Looking at cowlib you're binary parsing instead. Thank you Lo?c. Best, r. On Fri, Apr 3, 2015 at 9:07 AM, Lo?c Hoguin wrote: > cow_date:parse_date(Bin). > > Returns {{Y,Mo,D},{H,Mi,S}}. > > > On 04/03/2015 06:59 PM, Roberto Ostinelli wrote: > >> All, >> The Date field of HTTP headers uses date format as defined in RFC 7231, >> for example: >> >> Date: Tue, 15 Nov 1994 08:12:31 GMT >> >> What is the fastest way to parse this header and convert it to an epoch >> time? >> >> Best, >> r. >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -- > Lo?c Hoguin > http://ninenines.eu > -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Fri Apr 3 18:10:45 2015 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Fri, 03 Apr 2015 19:10:45 +0300 Subject: [erlang-questions] Date parsing (RFC 7231) In-Reply-To: References: Message-ID: <551EBB85.4030405@ninenines.eu> The date header is always in GMT though. :-) On 04/03/2015 07:09 PM, Jesse Gumm wrote: > qdate:to_unixtime(DateStringOrBin) should do the trick for you, with the > timezone awareness. > > On Apr 3, 2015 11:00 AM, "Roberto Ostinelli" > wrote: > > All, > The Date field of HTTP headers uses date format as defined in RFC > 7231, for example: > > Date: Tue, 15 Nov 1994 08:12:31 GMT > > What is the fastest way to parse this header and convert it to an > epoch time? > > Best, > r. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin http://ninenines.eu From roberto@REDACTED Fri Apr 3 18:11:19 2015 From: roberto@REDACTED (Roberto Ostinelli) Date: Fri, 3 Apr 2015 09:11:19 -0700 Subject: [erlang-questions] Date parsing (RFC 7231) In-Reply-To: References: Message-ID: Thank you Jesse, I don't need timezone awareness since RFC 7231 specifies that dates must be specified in GMT. On Fri, Apr 3, 2015 at 9:09 AM, Jesse Gumm wrote: > qdate:to_unixtime(DateStringOrBin) should do the trick for you, with the > timezone awareness. > On Apr 3, 2015 11:00 AM, "Roberto Ostinelli" wrote: > >> All, >> The Date field of HTTP headers uses date format as defined in RFC 7231, >> for example: >> >> Date: Tue, 15 Nov 1994 08:12:31 GMT >> >> What is the fastest way to parse this header and convert it to an epoch >> time? >> >> Best, >> r. >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From lloyd@REDACTED Fri Apr 3 19:03:01 2015 From: lloyd@REDACTED (lloyd@REDACTED) Date: Fri, 3 Apr 2015 13:03:01 -0400 (EDT) Subject: [erlang-questions] =?utf-8?q?Nitrogen-based_XMPP_client=3F?= In-Reply-To: <20150403131046.46149dd5@gmail.com> References: <20150403131046.46149dd5@gmail.com> Message-ID: <1428080581.02616486@apps.rackspace.com> Hi Grigory, Have you published code anywhere? Thanks, Lloyd -----Original Message----- From: "Grigory Fateyev" Sent: Friday, April 3, 2015 6:10am To: erlang-questions@REDACTED Subject: Re: [erlang-questions] Nitrogen-based XMPP client? Hello Marc Worrell! On Fri, 3 Apr 2015 09:30:02 +0200 you wrote: > Hi Lloyd, > > Did you have a look at the exmpp client library? > There are some tutorials which might help in getting you started. > > http://processone.github.io/exmpp/ > > > I don?t know of any other xmpp client libraries. I have a very simple (hope in future it'll be more useful) a pet project xmpp client that connects to rooms and does simple things. Like a bot. I found exmpp more ready to use it like an example, but the author of exmpp said: "Just as a head up, note that post is > 5 years old already. The main goal of exmpp itself was to be used as parser/xml core in ejabberd". So I started my project as simple request-response flow with server. The main problem is to read XMPP XEPs. :) > > On 3 apr. 2015, at 05:29, Lloyd R. Prentice > > wrote: > > > > Hello, > > > > I'm considering integrating multi-user chat into a Nitrogen > > application I'm building--- assuming that I'll support it with > > Mongoose IM. As usual, I'm in over my head, but persistent. > > > > I'm imagining a small number ( four? six?) thematic chat rooms > > serving a relatively small number of visitors at a time. As to > > scale, think of a book-signing event at a bookstore. > > > > I see that there are several JavaScript XMPP client packages that I > > could integrate into my Nitrogen web pages but, if that's the way, > > which one? Or, is there a native Erlang XMPP client app that I > > should look at? Indeed, is Mongoose IM overkill? After all, the > > Nitrogen demos provide a very simple tinker-toy chat example, and > > numerous others can be found on the web though, most, rather long > > in the tooth. > > > > And more, how much harder would it be to integrate video chat > > using, say, web sockets? > > > > In other words, how can I best keep the gotchas, demons, and > > goblins at bay sufficiently to master the arcane arts of multi-user > > chat sufficiently to build my application before I die of old age? > > > > I'd much appreciate pointers or, better yet, pointers to actual > > code. > > > > All the best, > > > > LRP > > > > Sent from my iPad > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > -- Best regards! gfborn [at] gmail [dot] com _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions From abejideayodele@REDACTED Fri Apr 3 15:37:10 2015 From: abejideayodele@REDACTED (ayodele abejide) Date: Fri, 3 Apr 2015 08:37:10 -0500 Subject: [erlang-questions] Erlang and Akka, The Sequel In-Reply-To: References: <20150330133920.GA75677@ferdair.local> <0b564da6b36a82f44f4ddfaee798bbc8.squirrel@chasm.otago.ac.nz> Message-ID: There is an implementation of Future/Promise in Elixir its called Task[1] Example drawn from the documentation: task = Task.async(fn -> do_some_work() end) res = do_some_other_work() res + Task.await(task) Reading the source[2] Task is a wrapper around proc_lib that monitors the calling process for error handling and uses patterns described by Joe in its implementation. Understanding the primitives would make implementing something similar in Erlang trivial 1. http://elixir-lang.org/docs/stable/elixir/Task.html 2. https://github.com/elixir-lang/elixir/blob/v1.0.3/lib/elixir/lib/task.ex#L146-L209 -------------- next part -------------- An HTML attachment was scrubbed... URL: From mononcqc@REDACTED Fri Apr 3 20:30:43 2015 From: mononcqc@REDACTED (Fred Hebert) Date: Fri, 3 Apr 2015 14:30:43 -0400 Subject: [erlang-questions] Erlang and Akka, The Sequel In-Reply-To: References: <20150330133920.GA75677@ferdair.local> <0b564da6b36a82f44f4ddfaee798bbc8.squirrel@chasm.otago.ac.nz> Message-ID: <20150403183041.GG75677@ferdair.local> On 04/03, ayodele abejide wrote: >There is an implementation of Future/Promise in Elixir its called Task[1] >Reading the source[2] Task is a wrapper around proc_lib that monitors >the >calling process for error handling and uses patterns described by Joe in >its implementation. Understanding the primitives would make implementing >something similar in Erlang trivial > >https://github.com/elixir-lang/elixir/blob/v1.0.3/lib/elixir/lib/task.ex#L146-L209 Interestingly enough, this shows these patterns are harder to implement correctly than it seems. Looking at the code, if the caller process is trapping exits (process_flag(trap_exit, true)), then the caller won't only receive the 'DOWN' message sent by the monitor, but also an 'EXIT' tuple. If the `await` function is wrapped in a `catch`, then the message won't be handled properly. The Elixir solution conveniently ignores that possibility in await() and therefore risks leaking messages (that will remain stuck in the process mailbox). The Erlang RPC mechanism handles that one properly by spawning [1] a middle-man process to handle these failures [2]. [1]: https://github.com/erlang/otp/blob/maint/lib/kernel/src/rpc.erl#L629-L649 [2]: https://github.com/erlang/otp/blob/maint/lib/kernel/src/rpc.erl#L328-L361 The basic pattern is simple enough, but fault-tolerance is often trickier than one expects it to be. Regards, Fred. From abejideayodele@REDACTED Fri Apr 3 23:05:03 2015 From: abejideayodele@REDACTED (ayodele abejide) Date: Fri, 3 Apr 2015 16:05:03 -0500 Subject: [erlang-questions] Erlang and Akka, The Sequel In-Reply-To: <20150403183041.GG75677@ferdair.local> References: <20150330133920.GA75677@ferdair.local> <0b564da6b36a82f44f4ddfaee798bbc8.squirrel@chasm.otago.ac.nz> <20150403183041.GG75677@ferdair.local> Message-ID: Fred, The Elixir solution conveniently ignores that possibility in await() and therefore risks leaking messages (that will remain stuck in the process mailbox). You are right, the Elixir implementation actually breaks when any receive block that consumes all messages off the queue is placed in-between the async and await calls. I submitted an issue. Abejide Ayodele It always seems impossible until it's done. --Nelson Mandela On Fri, Apr 3, 2015 at 1:30 PM, Fred Hebert wrote: > On 04/03, ayodele abejide wrote: > >> There is an implementation of Future/Promise in Elixir its called Task[1] >> > > Reading the source[2] Task is a wrapper around proc_lib that monitors the >> calling process for error handling and uses patterns described by Joe in >> its implementation. Understanding the primitives would make implementing >> something similar in Erlang trivial >> >> https://github.com/elixir-lang/elixir/blob/v1.0.3/lib/ >> elixir/lib/task.ex#L146-L209 >> > > Interestingly enough, this shows these patterns are harder to implement > correctly than it seems. Looking at the code, if the caller process is > trapping exits (process_flag(trap_exit, true)), then the caller won't only > receive the 'DOWN' message sent by the monitor, but also an 'EXIT' tuple. > If the `await` function is wrapped in a `catch`, then the message won't be > handled properly. > > The Elixir solution conveniently ignores that possibility in await() and > therefore risks leaking messages (that will remain stuck in the process > mailbox). > > The Erlang RPC mechanism handles that one properly by spawning [1] a > middle-man process to handle these failures [2]. > > [1]: https://github.com/erlang/otp/blob/maint/lib/kernel/src/rpc. > erl#L629-L649 > [2]: https://github.com/erlang/otp/blob/maint/lib/kernel/src/rpc. > erl#L328-L361 > > The basic pattern is simple enough, but fault-tolerance is often trickier > than one expects it to be. > > Regards, > Fred. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jose.valim@REDACTED Sat Apr 4 00:18:44 2015 From: jose.valim@REDACTED (=?UTF-8?Q?Jos=C3=A9_Valim?=) Date: Sat, 4 Apr 2015 00:18:44 +0200 Subject: [erlang-questions] Erlang and Akka, The Sequel In-Reply-To: <20150403183041.GG75677@ferdair.local> References: <20150330133920.GA75677@ferdair.local> <0b564da6b36a82f44f4ddfaee798bbc8.squirrel@chasm.otago.ac.nz> <20150403183041.GG75677@ferdair.local> Message-ID: > If the `await` function is wrapped in a `catch`, then the message won't be handled properly. We have actually discussed this extensively. We have decided to not handle those cases because similar errors happen if you wrap a gen_server:call/3 in a catch. If you trigger the call timeout and catch it, the gen server is going to eventually reply which will sit in your inbox. We believe there is one httpc bug caused due to this (anti-)pattern. (insert huge misplaced rant about catch) So we could either use middleman processes for everything or document a task is another process and catching can lead to idle messages regardless of trap exits. We went with the latter. I do agree with Fred this is trickier than it looks though, even more if you are using lager and want nicely formatted error messages (then you need to use proc_lib and what not). -- *Jos? Valim* www.plataformatec.com.br Skype: jv.ptec Founder and Lead Developer -------------- next part -------------- An HTML attachment was scrubbed... URL: From raould@REDACTED Sat Apr 4 01:04:17 2015 From: raould@REDACTED (Raoul Duke) Date: Fri, 3 Apr 2015 16:04:17 -0700 Subject: [erlang-questions] Erlang and Akka, The Sequel In-Reply-To: References: <20150330133920.GA75677@ferdair.local> <0b564da6b36a82f44f4ddfaee798bbc8.squirrel@chasm.otago.ac.nz> <20150403183041.GG75677@ferdair.local> Message-ID: > (insert huge misplaced rant about catch) do tell, please (perhaps under a new subject, tho). From rvirding@REDACTED Sat Apr 4 02:05:40 2015 From: rvirding@REDACTED (Robert Virding) Date: Sat, 4 Apr 2015 02:05:40 +0200 Subject: [erlang-questions] Erlang and Akka, The Sequel In-Reply-To: References: <20150330133920.GA75677@ferdair.local> <0b564da6b36a82f44f4ddfaee798bbc8.squirrel@chasm.otago.ac.nz> <20150403183041.GG75677@ferdair.local> Message-ID: I think one very interesting point from this discussion is what Richard, Fred and Joe have all mentioned is that it is not always meaningful or interesting to take primitives, which might actually be design patterns, from one language and transfer them directly to another language. This maybe because of the primitives in the second language, or maybe because of how applications are structured in the second language. Robert On 4 April 2015 at 01:04, Raoul Duke wrote: > > (insert huge misplaced rant about catch) > > > do tell, please (perhaps under a new subject, tho). > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rvirding@REDACTED Sat Apr 4 02:08:58 2015 From: rvirding@REDACTED (Robert Virding) Date: Sat, 4 Apr 2015 02:08:58 +0200 Subject: [erlang-questions] Warning not emitted on string matching clauses In-Reply-To: References: <20150401185745.GF75677@ferdair.local> Message-ID: Also any reordering the compiler does in the way it tests the patterns is guaranteed not to change the semantics of the original. The resultant code will always behave as if the clauses had been tested top-to-bottom as written. Robert On 1 April 2015 at 21:28, Bj?rn-Egil Dahlberg wrote: > The compiler doesn't really "reorder" the clauses. It builds a match tree > depending on types, values and variables. In the current implementation it > does so by inspecting all clauses left to right. > > You can inspect the matching tree by compiling to kernel, i.e. erlc > +to_kernel t.erl > > // Bj?rn-Egil > > 2015-04-01 20:57 GMT+02:00 Fred Hebert : > >> On 04/01, Pierre Fenoll wrote: >> >>> Ferd on IRC mentioned that the compiler might feel free to reorder >>> clauses. >>> >> >> The compiler would only reorder clauses that are free to do so. >> >> For example: >> >> f(a) -> 1; >> f(b) -> 2; >> f(X) when is_atom(X) -> 3; >> f([_|_]) -> 4; >> f({_}) -> 5. >> >> Would let the compiler possibly reorder 1 & 2, and maybe 4 & 5. But the >> presence of a guard clause or ambiguous matches would prevent it from doing >> more than that, if my understanding is accurate. >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rvirding@REDACTED Sat Apr 4 04:22:02 2015 From: rvirding@REDACTED (Robert Virding) Date: Sat, 4 Apr 2015 04:22:02 +0200 Subject: [erlang-questions] Dialyzer input file types Message-ID: Why does dialyzer only allow me to input either all source (.erl) files or all .beam files? Or does it allow mixing and I have missed the option? I can't see any logical reason why it should not allow mixing. Robert -------------- next part -------------- An HTML attachment was scrubbed... URL: From roger@REDACTED Sat Apr 4 15:43:03 2015 From: roger@REDACTED (Roger Lipscombe) Date: Sat, 4 Apr 2015 14:43:03 +0100 Subject: [erlang-questions] CT Code Coverage on remote nodes? Message-ID: Are there any good tutorials showing how to get Common Test to generate coverage reports for remote nodes? I'm using ct_run to start my tests; one of my CT hooks starts up an external Erlang node (packaged as a release using relx) from which I'd like to get coverage details. >From what I've found in the CT documentation, it is possible for CT to collect coverage information from remote nodes, so... 1. In my CTH, I call cover:start() in init/2. 2. Immediately before starting the remote node, I call cover:compile_beam_directory on the _rel/foo/lib/foo-Vsn/ebin directory, which returns [{ok,Mod}...]. 3. Immediately after starting the remote node (using erlexec -- https://github.com/saleyn/erlexec), I call ct_cover:add_nodes([Node]). All of this succeeds, but when CT runs the coverage analysis, it reports zero coverage. What am I missing? Thanks, Roger. From jesper.louis.andersen@REDACTED Sat Apr 4 16:56:21 2015 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Sat, 4 Apr 2015 16:56:21 +0200 Subject: [erlang-questions] Erlang and Akka, The Sequel In-Reply-To: References: <20150330133920.GA75677@ferdair.local> <0b564da6b36a82f44f4ddfaee798bbc8.squirrel@chasm.otago.ac.nz> <20150403183041.GG75677@ferdair.local> Message-ID: On Sat, Apr 4, 2015 at 12:18 AM, Jos? Valim wrote: > We have actually discussed this extensively. We have decided to not handle > those cases because similar errors happen if you wrap a gen_server:call/3 > in a catch. If you trigger the call timeout and catch it, the gen server is > going to eventually reply which will sit in your inbox. We believe there is > one httpc bug caused due to this (anti-)pattern. > I'm not sure it is possible to avoid "late-arriving" messages. By the two-generals problem, we can't make a system which can safely cancel the message, but we will have to rely on a timeout. Once we rely on the timeout, we must have a way to remove messages which arrive late because it breaks with the contract between the caller and callee. In other words, you will need a "collector" case in your gen_server which can evict those late-arriving messages out of the mailbox, whenever you start catching rather than dying. There is currently no way in Erlang to "mask" what references arriving the mailbox are ok, and which should be silently removed. But perhaps there should be. -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mononcqc@REDACTED Sat Apr 4 17:32:12 2015 From: mononcqc@REDACTED (Fred Hebert) Date: Sat, 4 Apr 2015 11:32:12 -0400 Subject: [erlang-questions] Erlang and Akka, The Sequel In-Reply-To: References: <20150330133920.GA75677@ferdair.local> <0b564da6b36a82f44f4ddfaee798bbc8.squirrel@chasm.otago.ac.nz> <20150403183041.GG75677@ferdair.local> Message-ID: <20150404153211.GC6812@ferdmbp.local> On 04/04, Jos? Valim wrote: >> If the `await` function is wrapped in a `catch`, then the message won't >be handled properly. > >We have actually discussed this extensively. We have decided to not handle >those cases because similar errors happen if you wrap a gen_server:call/3 >in a catch. If you trigger the call timeout and catch it, the gen server is >going to eventually reply which will sit in your inbox. We believe there is >one httpc bug caused due to this (anti-)pattern. > The difference is that gen_server's call has it only on timeouts and these responses. The Async task has it on timeouts *and* errors where you know you will receive the 'DOWN' message *and* the 'EXIT' signal if you are trapping exits because you use spawn_link. So any exit where it happens and is catched will leak these messages in a surefire way -- not just timeouts. OTP calls don't have this because they only set monitors, not monitors + links. From albertaixendri@REDACTED Sat Apr 4 19:48:18 2015 From: albertaixendri@REDACTED (Albert Aixendri) Date: Sat, 4 Apr 2015 19:48:18 +0200 Subject: [erlang-questions] Key/value embeddable database Message-ID: <52B3A8EF-31D7-44DA-A924-A4D5DC87D975@gmail.com> Hello, I am looking a simple embeddable key/value database. I have discarded ETS (limited by RAM memory), DETS (limited in 2GB) Mnesia (it?s no so simple). On the other hand I have found these: - elevelDB: very popular. Developed by Basho and used in Riak. - HanoiDB: Similar to LevelDB https://github.com/krestenkrab/hanoidb - LETS: Based in LevelDB too. https://github.com/norton/lets - Some that use Tokyo Cabinet: Toke https://bitbucket.org/lshift/toke/ and tcerl http://code.google.com/p/tcerl/ Do you know others with features like these? Thanks in advance. Albert. From sergej.jurecko@REDACTED Sat Apr 4 21:11:41 2015 From: sergej.jurecko@REDACTED (Sergej Jurecko) Date: Sat, 4 Apr 2015 21:11:41 +0200 Subject: [erlang-questions] Key/value embeddable database In-Reply-To: <52B3A8EF-31D7-44DA-A924-A4D5DC87D975@gmail.com> References: <52B3A8EF-31D7-44DA-A924-A4D5DC87D975@gmail.com> Message-ID: <3BD6F33C-8DF3-4A22-9818-72421E0ED57F@gmail.com> cowdb: https://github.com/refuge/cowdb The nice thing about it is that it is purely Erlang. Sergej On 04 Apr 2015, at 19:48, Albert Aixendri wrote: > Hello, > > I am looking a simple embeddable key/value database. I have discarded ETS (limited by RAM memory), DETS (limited in 2GB) Mnesia (it?s no so simple). > On the other hand I have found these: > - elevelDB: very popular. Developed by Basho and used in Riak. > - HanoiDB: Similar to LevelDB https://github.com/krestenkrab/hanoidb > - LETS: Based in LevelDB too. https://github.com/norton/lets > - Some that use Tokyo Cabinet: Toke https://bitbucket.org/lshift/toke/ and tcerl http://code.google.com/p/tcerl/ > > Do you know others with features like these? > > Thanks in advance. > > Albert. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From mfidelman@REDACTED Sat Apr 4 21:37:36 2015 From: mfidelman@REDACTED (Miles Fidelman) Date: Sat, 04 Apr 2015 15:37:36 -0400 Subject: [erlang-questions] Key/value embeddable database In-Reply-To: <52B3A8EF-31D7-44DA-A924-A4D5DC87D975@gmail.com> References: <52B3A8EF-31D7-44DA-A924-A4D5DC87D975@gmail.com> Message-ID: <55203D80.20003@meetinghouse.net> Albert Aixendri wrote: > Hello, > > I am looking a simple embeddable key/value database. I have discarded ETS (limited by RAM memory), DETS (limited in 2GB) Mnesia (it?s no so simple). > On the other hand I have found these: > - elevelDB: very popular. Developed by Basho and used in Riak. > - HanoiDB: Similar to LevelDB https://github.com/krestenkrab/hanoidb > - LETS: Based in LevelDB too. https://github.com/norton/lets > - Some that use Tokyo Cabinet: Toke https://bitbucket.org/lshift/toke/ and tcerl http://code.google.com/p/tcerl/ > > Simple, embeddable, key/value - sounds like Berkeley DB to me. Do some googling and you'll find erlang bindings. Miles Fidelman -- In theory, there is no difference between theory and practice. In practice, there is. .... Yogi Berra From antoine.koener@REDACTED Sun Apr 5 09:14:31 2015 From: antoine.koener@REDACTED (Antoine Koener) Date: Sun, 5 Apr 2015 09:14:31 +0200 Subject: [erlang-questions] Key/value embeddable database In-Reply-To: <55203D80.20003@meetinghouse.net> References: <52B3A8EF-31D7-44DA-A924-A4D5DC87D975@gmail.com> <55203D80.20003@meetinghouse.net> Message-ID: <5B9E5EA0-D3BB-4585-8F96-851C57E3A682@gmail.com> Hello ! > On 04 Apr 2015, at 21:37, Miles Fidelman wrote: > > Albert Aixendri wrote: >> Hello, >> >> I am looking a simple embeddable key/value database. I have discarded ETS (limited by RAM memory), DETS (limited in 2GB) Mnesia (it?s no so simple). >> On the other hand I have found these: >> - elevelDB: very popular. Developed by Basho and used in Riak. >> - HanoiDB: Similar to LevelDB https://github.com/krestenkrab/hanoidb >> - LETS: Based in LevelDB too. https://github.com/norton/lets >> - Some that use Tokyo Cabinet: Toke https://bitbucket.org/lshift/toke/ and tcerl http://code.google.com/p/tcerl/ > Simple, embeddable, key/value - sounds like Berkeley DB to me. Do some googling and you'll find erlang bindings. > Egtm : https://github.com/ztmr/egtm/wiki/EGTM,-M%5BUMPS%5D-and-GT.M-technology-FAQ > Miles Fidelman > > -- > In theory, there is no difference between theory and practice. > In practice, there is. .... Yogi Berra > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From rvirding@REDACTED Sun Apr 5 18:26:18 2015 From: rvirding@REDACTED (Robert Virding) Date: Sun, 5 Apr 2015 18:26:18 +0200 Subject: [erlang-questions] Dialyzer input file types In-Reply-To: References: Message-ID: A funny thing with dialyzer is that it works on Core erlang but only gets the core erlang from either the AST in 'debug_info' compiled .beam files or from .erl. There is no pluggable method to get the core erlang. Very extensible unfriendly. I have done some special hacks in the dialyzer input files which allow it to get the core data from .lfe files. It was quite simple to do and works like a charm. It would be quite easy to extend this so it could take files of any kind in a pluggable way. For every file type there would be a module which extracts the core forms for that type, so for example for .foo files there should be a module 'dialyzer_foo_utils'. By setting the code path you could make sure it finds your specific files. I *know* this should be fixed by adding an option to the erlang compiler to include the Core forms in the .beam files and getting dialyzer to get its Core data from there as well, but there seems to be little interest in either camp to fix this problem. Robert P.S. You are aware that compiling with debug_info does add the erlang AST to the .beam file so the source code is actually included? On 4 April 2015 at 04:22, Robert Virding wrote: > Why does dialyzer only allow me to input either all source (.erl) files or > all .beam files? Or does it allow mixing and I have missed the option? I > can't see any logical reason why it should not allow mixing. > > Robert > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From albertaixendri@REDACTED Sun Apr 5 20:07:01 2015 From: albertaixendri@REDACTED (Albert Aixendri) Date: Sun, 5 Apr 2015 20:07:01 +0200 Subject: [erlang-questions] Key/value embeddable database In-Reply-To: <5B9E5EA0-D3BB-4585-8F96-851C57E3A682@gmail.com> References: <52B3A8EF-31D7-44DA-A924-A4D5DC87D975@gmail.com> <55203D80.20003@meetinghouse.net> <5B9E5EA0-D3BB-4585-8F96-851C57E3A682@gmail.com> Message-ID: Thanks to all!! I wil try them. Albert. > On 05 Apr 2015, at 09:14, Antoine Koener wrote: > > > Hello ! > >> On 04 Apr 2015, at 21:37, Miles Fidelman wrote: >> >> Albert Aixendri wrote: >>> Hello, >>> >>> I am looking a simple embeddable key/value database. I have discarded ETS (limited by RAM memory), DETS (limited in 2GB) Mnesia (it?s no so simple). >>> On the other hand I have found these: >>> - elevelDB: very popular. Developed by Basho and used in Riak. >>> - HanoiDB: Similar to LevelDB https://github.com/krestenkrab/hanoidb >>> - LETS: Based in LevelDB too. https://github.com/norton/lets >>> - Some that use Tokyo Cabinet: Toke https://bitbucket.org/lshift/toke/ and tcerl http://code.google.com/p/tcerl/ >> Simple, embeddable, key/value - sounds like Berkeley DB to me. Do some googling and you'll find erlang bindings. >> > > Egtm : https://github.com/ztmr/egtm/wiki/EGTM,-M%5BUMPS%5D-and-GT.M-technology-FAQ > >> Miles Fidelman >> >> -- >> In theory, there is no difference between theory and practice. >> In practice, there is. .... Yogi Berra >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From k.petrauskas@REDACTED Mon Apr 6 11:02:02 2015 From: k.petrauskas@REDACTED (Karolis Petrauskas) Date: Mon, 6 Apr 2015 12:02:02 +0300 Subject: [erlang-questions] [ANN] Erlang/OTP 17.5 has been released In-Reply-To: References: Message-ID: Hi, My Common Test suites started to fail after upgrading Erlang to 17.5 (i am using ESL's Debian distribution). I am running CT from rebar. The error is the following: Test run crashed! This could be an internal error - please report! {{badmatch,["ct_run","test@REDACTED","5grupe","lt","2015-04-06_11","56", "45"]}, [{ct_logs,'-sort_ct_runs/1-fun-0-',2,[{file,"ct_logs.erl"},{line,1912}]}, {lists,sort,2,[{file,"lists.erl"},{line,967}]}, {ct_logs,make_all_suites_index,1,[{file,"ct_logs.erl"},{line,2232}]}, {ct_logs,close,2,[{file,"ct_logs.erl"},{line,178}]}, {ct_util,loop,3,[{file,"ct_util.erl"},{line,471}]}]} Karolis On Wed, Apr 1, 2015 at 12:02 PM, Kenneth Lundin wrote: > Erlang/OTP 17.5 has been released. > > Erlang/OTP 17.5 is a service release on the 17 track with mostly bug fixes, > but is does contain a number of new features and characteristics > improvements as well. > > Some highlights of the release are: > > ERTS: Added command line argument option for setting the initial size of > process dictionaries. > Diameter: configurable incoming_max len and string_decode for diameter > messages > Bugfixes and minor small features in applications such as compiler, > common_test, crypto, debugger, eldap, erts, hipe, inets, ssh, ssl, ... > 43 contributions from 32 different contributors > > For more details see the README file at > http://www.erlang.org/download/otp_src_17.5.readme > > You can download the full source distribution from > http://www.erlang.org/download/otp_src_17.5.tar.gz > > Note: To unpack the TAR archive you need a GNU TAR compatible program. For > installation instructions please read the README that is part of the > distribution. > > You can also find this release at the official Erlang/OTP Git-repository at > Github here: https://github.com/erlang/otp tagged "OTP-17.5" > > The Windows binary distribution can be downloaded from > > http://www.erlang.org/download/otp_win32_17.5.exe > > http://www.erlang.org/download/otp_win64_17.5.exe > > You can also download the complete HTML documentation or the Unix manual > files > > http://www.erlang.org/download/otp_doc_html_17.5.tar.gz > http://www.erlang.org/download/otp_doc_man_17.5.tar.gz > > We also want to thank those that sent us patches, suggestions and bug > reports. > > The Erlang/OTP Team at Ericsson > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From sean@REDACTED Mon Apr 6 18:00:01 2015 From: sean@REDACTED (Functional Jobs) Date: Mon, 6 Apr 2015 12:00:01 -0400 Subject: [erlang-questions] New Functional Programming Job Opportunities Message-ID: <5522ad828ea04@functionaljobs.com> Here are some functional programming job opportunities that were posted recently: Full-stack Software Engineer (updated) at Capital Match http://functionaljobs.com/jobs/8802-full-stack-software-engineer-updated-at-capital-match Cheers, Sean Murphy FunctionalJobs.com From jay@REDACTED Mon Apr 6 18:17:04 2015 From: jay@REDACTED (Jay Nelson) Date: Mon, 6 Apr 2015 12:17:04 -0400 Subject: [erlang-questions] Key/value embeddable database Message-ID: Hanoi DB https://github.com/krestenkrab/hanoidb is an erlang implementation of LevelDB. jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From garret.smith@REDACTED Mon Apr 6 19:23:16 2015 From: garret.smith@REDACTED (Garret Smith) Date: Mon, 6 Apr 2015 10:23:16 -0700 Subject: [erlang-questions] [ANN] Erlang/OTP 17.5 has been released In-Reply-To: References: Message-ID: I reported this a few days back: http://erlang.org/pipermail/erlang-bugs/2015-April/004881.html This patch has been working for me: https://github.com/garret-smith/otp/tree/gs-ct-longnames-dirparse On Mon, Apr 6, 2015 at 2:02 AM, Karolis Petrauskas wrote: > Hi, > > My Common Test suites started to fail after upgrading Erlang to 17.5 > (i am using ESL's Debian distribution). I am running CT from rebar. > The error is the following: > > Test run crashed! This could be an internal error - please report! > > {{badmatch,["ct_run","test@REDACTED","5grupe","lt","2015-04-06_11","56", > "45"]}, > [{ct_logs,'-sort_ct_runs/1-fun-0-',2,[{file,"ct_logs.erl"},{line,1912}]}, > {lists,sort,2,[{file,"lists.erl"},{line,967}]}, > {ct_logs,make_all_suites_index,1,[{file,"ct_logs.erl"},{line,2232}]}, > {ct_logs,close,2,[{file,"ct_logs.erl"},{line,178}]}, > {ct_util,loop,3,[{file,"ct_util.erl"},{line,471}]}]} > > Karolis > > On Wed, Apr 1, 2015 at 12:02 PM, Kenneth Lundin > wrote: >> Erlang/OTP 17.5 has been released. >> >> Erlang/OTP 17.5 is a service release on the 17 track with mostly bug fixes, >> but is does contain a number of new features and characteristics >> improvements as well. >> >> Some highlights of the release are: >> >> ERTS: Added command line argument option for setting the initial size of >> process dictionaries. >> Diameter: configurable incoming_max len and string_decode for diameter >> messages >> Bugfixes and minor small features in applications such as compiler, >> common_test, crypto, debugger, eldap, erts, hipe, inets, ssh, ssl, ... >> 43 contributions from 32 different contributors >> >> For more details see the README file at >> http://www.erlang.org/download/otp_src_17.5.readme >> >> You can download the full source distribution from >> http://www.erlang.org/download/otp_src_17.5.tar.gz >> >> Note: To unpack the TAR archive you need a GNU TAR compatible program. For >> installation instructions please read the README that is part of the >> distribution. >> >> You can also find this release at the official Erlang/OTP Git-repository at >> Github here: https://github.com/erlang/otp tagged "OTP-17.5" >> >> The Windows binary distribution can be downloaded from >> >> http://www.erlang.org/download/otp_win32_17.5.exe >> >> http://www.erlang.org/download/otp_win64_17.5.exe >> >> You can also download the complete HTML documentation or the Unix manual >> files >> >> http://www.erlang.org/download/otp_doc_html_17.5.tar.gz >> http://www.erlang.org/download/otp_doc_man_17.5.tar.gz >> >> We also want to thank those that sent us patches, suggestions and bug >> reports. >> >> The Erlang/OTP Team at Ericsson >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From roberto@REDACTED Mon Apr 6 22:03:59 2015 From: roberto@REDACTED (Roberto Ostinelli) Date: Mon, 6 Apr 2015 13:03:59 -0700 Subject: [erlang-questions] exometer core only Message-ID: All, Reading the HEAD documentation of exometer: NOTE: Exometer has been split into exometer_core, and exometer (as well as separate reporter applications). The latest monolithic version of Exometer is 1.1. I'm currently using 1.1, however there are a bunch of things that I do not use (I'm only using custom static metrics with hostedgraphite, nothing from folsom, etc). I'm guessing that's the rationale into splitting exometer, so that one can choose which components to use. Though, I'm not understanding if I should start using exometer_core, or if the split is still a WIP. Anyone can shed some light? Thank you, r. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tino.breddin@REDACTED Tue Apr 7 07:40:34 2015 From: tino.breddin@REDACTED (Tino Breddin) Date: Tue, 7 Apr 2015 07:40:34 +0200 Subject: [erlang-questions] exometer core only In-Reply-To: References: Message-ID: Hi, The split was performed in order to simplify the use of core exometer functionality. As you can see the new `exometer` is mostly a container for reporters which have dependencies which you might not want to use in your system. The reporters which are kept in exometer_core are dependency-free. You can still use the "feature selection" during build to exclude certain reporters and their dependencies, but the main build has become simpler this way. As for using the split, it is stable so far and all tests are green, however not all reporters are tested. So feel free to take it for a test. LGT On Mon, Apr 6, 2015 at 10:03 PM, Roberto Ostinelli wrote: > All, > Reading the HEAD documentation of exometer: > > NOTE: Exometer has been split into exometer_core, and exometer (as well as > separate reporter applications). The latest monolithic version of Exometer > is 1.1. > > I'm currently using 1.1, however there are a bunch of things that I do not > use (I'm only using custom static metrics with hostedgraphite, nothing from > folsom, etc). I'm guessing that's the rationale into splitting exometer, so > that one can choose which components to use. > > Though, I'm not understanding if I should start using exometer_core, or if > the split is still a WIP. > > Anyone can shed some light? > > Thank you, > r. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.abrahamsson@REDACTED Tue Apr 7 09:31:26 2015 From: daniel.abrahamsson@REDACTED (Daniel Abrahamsson) Date: Tue, 7 Apr 2015 09:31:26 +0200 Subject: [erlang-questions] Key/value embeddable database In-Reply-To: References: Message-ID: Bitcask (https://github.com/basho/bitcask) is also a nice key-value store from Basho. //Daniel On Mon, Apr 6, 2015 at 6:17 PM, Jay Nelson wrote: > Hanoi DB https://github.com/krestenkrab/hanoidb is an > erlang implementation of LevelDB. > > jay > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From martink@REDACTED Tue Apr 7 10:48:10 2015 From: martink@REDACTED (Martin Karlsson) Date: Tue, 7 Apr 2015 20:48:10 +1200 Subject: [erlang-questions] Looking for an erlang function benchmarking library/framework Message-ID: Hello, I want to benchmark individual functions to see how they perform. I've looked around for some library or framework which provides a repeatable and structured way of doing benchmarking (much like eunit and common tests do for unit and integration tests) but haven't found anything on a micro-level. Currently I have an ad-hoc collection of test functions wrapping the function under test in timer:tc but as mentioned looking for something more structured with proper reporting. Before writing it myself I wonder if there is anything out there already doing this? Ideally doing: $ make benchmark or $ rebar benchmark would run the benchmark tests and output a nice report. I've played with golangs benchmark utilities in the testing package ( http://golang.org/pkg/testing/) and found it quite useful and wonder if there is an erlang equivalent? Perhaps this is trivial to write in a normal eunit/ct test together with timer:tc or fprof or something and if so I'd be happy to get any pointers on where to start. Cheers, Martin -------------- next part -------------- An HTML attachment was scrubbed... URL: From martink@REDACTED Tue Apr 7 10:54:42 2015 From: martink@REDACTED (Martin Karlsson) Date: Tue, 7 Apr 2015 20:54:42 +1200 Subject: [erlang-questions] Looking for an erlang function benchmarking library/framework In-Reply-To: References: Message-ID: Obviously a second after this email Lo?c, on IRC, told me about "horse"(https://github.com/extend/horse) which seems to be a good fit and I'll start looking into that. Any other suggestions welcome. Cheers, Martin On 7 April 2015 at 20:48, Martin Karlsson wrote: > Hello, > > I want to benchmark individual functions to see how they perform. I've > looked around for some library or framework which provides a repeatable and > structured way of doing benchmarking (much like eunit and common tests do > for unit and integration tests) but haven't found anything on a micro-level. > > Currently I have an ad-hoc collection of test functions wrapping the > function under test in timer:tc but as mentioned looking for something more > structured with proper reporting. Before writing it myself I wonder if there > is anything out there already doing this? > > Ideally doing: > > $ make benchmark > or > $ rebar benchmark > > would run the benchmark tests and output a nice report. > > I've played with golangs benchmark utilities in the testing package > (http://golang.org/pkg/testing/) and found it quite useful and wonder if > there is an erlang equivalent? > > Perhaps this is trivial to write in a normal eunit/ct test together with > timer:tc or fprof or something and if so I'd be happy to get any pointers on > where to start. > > > Cheers, > Martin From darach@REDACTED Tue Apr 7 11:00:04 2015 From: darach@REDACTED (Darach Ennis) Date: Tue, 7 Apr 2015 10:00:04 +0100 Subject: [erlang-questions] Looking for an erlang function benchmarking library/framework In-Reply-To: References: Message-ID: Hi Martin, You won't find a general purpose benchmarking framework suitable for all tasks but there are plenty of examples: * http://release.softlab.ntua.gr/bencherl/ - targets scalability * https://github.com/gar1t/erlang-bench - Garrett uses erlang scripts for his * https://github.com/extend/horse - Lo?c's horse [ glad i waited a second before responding... ] For small or microbenchmarks there's also HDR histogram which will give fairly extensive statistics and allows compensating for coordinated omission for latency sensitive tests if that is relevant: https://github.com/hdrhistogram/hdr_histogram_erl Cheers, Darach. On Tue, Apr 7, 2015 at 9:48 AM, Martin Karlsson wrote: > Hello, > > I want to benchmark individual functions to see how they perform. I've > looked around for some library or framework which provides a repeatable > and structured way of doing benchmarking (much like eunit and common tests > do for unit and integration tests) but haven't found anything on a > micro-level. > > Currently I have an ad-hoc collection of test functions wrapping the > function under test in timer:tc but as mentioned looking for something more > structured with proper reporting. Before writing it myself I wonder if > there is anything out there already doing this? > > Ideally doing: > > $ make benchmark > or > $ rebar benchmark > > would run the benchmark tests and output a nice report. > > I've played with golangs benchmark utilities in the testing package ( > http://golang.org/pkg/testing/) and found it quite useful and wonder if > there is an erlang equivalent? > > Perhaps this is trivial to write in a normal eunit/ct test together with > timer:tc or fprof or something and if so I'd be happy to get any pointers > on where to start. > > > Cheers, > Martin > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From baliulia@REDACTED Tue Apr 7 11:03:09 2015 From: baliulia@REDACTED (=?UTF-8?Q?Ignas_Vy=C5=A1niauskas?=) Date: Tue, 7 Apr 2015 12:03:09 +0300 Subject: [erlang-questions] Looking for an erlang function benchmarking library/framework In-Reply-To: References: Message-ID: During Spawnfest 2012 we developed a unit-testing style library for this purpose, though it was never completed: https://github.com/yfyf/perforator It is probably broken under newer versions of rebar / R17 and not really documented much, but I think the main idea is exactly what you're looking for and if you are interesting I could walk you through the thing / help you fix it. -- Ignas From martink@REDACTED Tue Apr 7 11:09:36 2015 From: martink@REDACTED (Martin Karlsson) Date: Tue, 7 Apr 2015 21:09:36 +1200 Subject: [erlang-questions] Looking for an erlang function benchmarking library/framework In-Reply-To: References: Message-ID: > * http://release.softlab.ntua.gr/bencherl/ - targets scalability > * https://github.com/gar1t/erlang-bench - Garrett uses erlang scripts for > his > * https://github.com/extend/horse - Lo?c's horse [ glad i waited a second > before responding... ] > *https://github.com/hdrhistogram/hdr_histogram_erl > > Cheers, > > Darach. > Thanks! Lots of useful information From martink@REDACTED Tue Apr 7 11:10:48 2015 From: martink@REDACTED (Martin Karlsson) Date: Tue, 7 Apr 2015 21:10:48 +1200 Subject: [erlang-questions] Looking for an erlang function benchmarking library/framework In-Reply-To: References: Message-ID: > https://github.com/yfyf/perforator > > but I think the main idea is exactly what > you're looking for Thanks Ignas, Indeed, this looks pretty good. I will look into this as well as the other suggestions and will let you know if I get stuck. Cheers, Martin From essen@REDACTED Tue Apr 7 11:14:37 2015 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Tue, 07 Apr 2015 12:14:37 +0300 Subject: [erlang-questions] Looking for an erlang function benchmarking library/framework In-Reply-To: References: Message-ID: <55239FFD.9020506@ninenines.eu> On 04/07/2015 12:03 PM, Ignas Vy?niauskas wrote:> During Spawnfest 2012 we developed a unit-testing style library for > this purpose, though it was never completed: > > https://github.com/yfyf/perforator Actually wrote horse inspired by your project. Just I wanted the tests in the same module alongside eunit tests (putting function then its eunit/triq tests then its micro benchmarks in that order in the file, repeat for each function). On 04/07/2015 12:00 PM, Darach Ennis wrote: > For small or microbenchmarks there's also HDR histogram which will give > fairly extensive > statistics and allows compensating for coordinated omission for latency > sensitive tests if > that is relevant: > > https://github.com/hdrhistogram/hdr_histogram_erl Looks interesting. Might be good to use inside horse. I will have to try it. Cheers, -- Lo?c Hoguin http://ninenines.eu From gordeev.vladimir.v@REDACTED Tue Apr 7 12:43:07 2015 From: gordeev.vladimir.v@REDACTED (Vladimir Gordeev) Date: Tue, 7 Apr 2015 12:43:07 +0200 Subject: [erlang-questions] Binary prettyprinting Message-ID: Is there is a hack that globally makes format binaries differently? Sometimes while debugging I do print some data. Sometimes this data is compound and two-three levels down there is a long binary value. I want to see <<39,122,6,247,77,47,80,222,192,57,233,137,230, ... 1500 bytes total>> instead of actual 1,5 Kb data in text. Of course I can write such formatting function, but I have to use it explicitly each time (god, my three-level compound data). I want this only on my local machine, so any hacks welcome. Any ideas? -------------- next part -------------- An HTML attachment was scrubbed... URL: From radek@REDACTED Tue Apr 7 13:43:23 2015 From: radek@REDACTED (Rad Gruchalski) Date: Tue, 7 Apr 2015 13:43:23 +0200 Subject: [erlang-questions] Binary prettyprinting In-Reply-To: References: Message-ID: And what do you use to print them at the moment? Kind regards, Radek Gruchalski radek@REDACTED (mailto:radek@REDACTED) (mailto:radek@REDACTED) de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/) Confidentiality: This communication is intended for the above-named person and may be confidential and/or legally privileged. If it has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender immediately. On Tuesday, 7 April 2015 at 12:43, Vladimir Gordeev wrote: > Is there is a hack that globally makes format binaries differently? > > Sometimes while debugging I do print some data. Sometimes this data is compound and two-three levels down there is a long binary value. > > I want to see <<39,122,6,247,77,47,80,222,192,57,233,137,230, ... 1500 bytes total>> instead of actual 1,5 Kb data in text. > > Of course I can write such formatting function, but I have to use it explicitly each time (god, my three-level compound data). > > I want this only on my local machine, so any hacks welcome. Any ideas? > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED (mailto:erlang-questions@REDACTED) > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ali.sabil@REDACTED Tue Apr 7 14:05:59 2015 From: ali.sabil@REDACTED (Ali Sabil) Date: Tue, 07 Apr 2015 12:05:59 +0000 Subject: [erlang-questions] Why does Dialyzer crash on this map? In-Reply-To: <55095D8E.2050103@erlang.org> References: <54749C90.6000602@llaisdy.com> <5474BA4C.4030706@llaisdy.com> <547716F8.7010208@llaisdy.com> <4B9AEB77-A709-4D97-8EF2-8FE1AF5A4071@llaisdy.com> <55085881.3060508@erlang.org> <55095D8E.2050103@erlang.org> Message-ID: Hi again, Running dialyzer shipped with 17.5 on the same code base leads now to the following error (17.4 works without any errors): ===> Error in dialyzing apps: Analysis failed with error: {function_clause,[{erl_types,t_form_to_string, [{type,36,map_field_assoc, {type,36,atom,[]}, {type,36,term,[]}}], [{file,"erl_types.erl"},{line,4546}]}, {erl_types,t_form_to_string_list,2, [{file,"erl_types.erl"},{line,4637}]}, {erl_types,t_form_to_string,1, [{file,"erl_types.erl"},{line,4634}]}, {erl_types,t_form_to_string_list,2, [{file,"erl_types.erl"},{line,4637}]}, {erl_types,t_form_to_string,1, [{file,"erl_types.erl"},{line,4634}]}, {dialyzer_contracts,contract_to_string_1,1, [{file,"dialyzer_contracts.erl"}, {line,107}]}, {dialyzer_contracts,extra_contract_warning,6, [{file,"dialyzer_contracts.erl"}, {line,712}]}, {dialyzer_contracts,picky_contract_check,7, [{file,"dialyzer_contracts.erl"}, {line,686}]}]} Last messages in the log cache: Reading files and computing callgraph... done in 1.21 secs Removing edges... done in 0.04 secs On Wed, Mar 18, 2015 at 12:12 PM Bj?rn-Egil Dahlberg wrote: > On 2015-03-18 12:01, Ali Sabil wrote: > > I tried to create a minimal testcase but I unfortunately haven't been > > able to. I was running dialyzer on a quite large code base and now > > even the unpatched dialyzer works without any issue after I fixed all > > the issues reported by dialyzer. > > Ah, well .. I suspect it was the missing clause in find_terminals and I > had it on a TODO somewhere. Should be included to 17.5. > > // Bj?rn-Egil > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wallentin.dahlberg@REDACTED Tue Apr 7 14:31:43 2015 From: wallentin.dahlberg@REDACTED (=?UTF-8?Q?Bj=C3=B6rn=2DEgil_Dahlberg?=) Date: Tue, 7 Apr 2015 14:31:43 +0200 Subject: [erlang-questions] Why does Dialyzer crash on this map? In-Reply-To: References: <54749C90.6000602@llaisdy.com> <5474BA4C.4030706@llaisdy.com> <547716F8.7010208@llaisdy.com> <4B9AEB77-A709-4D97-8EF2-8FE1AF5A4071@llaisdy.com> <55085881.3060508@erlang.org> <55095D8E.2050103@erlang.org> Message-ID: Again - could you provide me with a sample code .. or at least some sort of a clue to what you are dialyzing? 2015-04-07 14:05 GMT+02:00 Ali Sabil : > Hi again, > > Running dialyzer shipped with 17.5 on the same code base leads now to the > following error (17.4 works without any errors): > > ===> Error in dialyzing apps: Analysis failed with error: > {function_clause,[{erl_types,t_form_to_string, > [{type,36,map_field_assoc, > {type,36,atom,[]}, > {type,36,term,[]}}], > [{file,"erl_types.erl"},{line,4546}]}, > {erl_types,t_form_to_string_list,2, > [{file,"erl_types.erl"},{line,4637}]}, > {erl_types,t_form_to_string,1, > [{file,"erl_types.erl"},{line,4634}]}, > {erl_types,t_form_to_string_list,2, > [{file,"erl_types.erl"},{line,4637}]}, > {erl_types,t_form_to_string,1, > [{file,"erl_types.erl"},{line,4634}]}, > {dialyzer_contracts,contract_to_string_1,1, > [{file,"dialyzer_contracts.erl"}, > {line,107}]}, > {dialyzer_contracts,extra_contract_warning,6, > [{file,"dialyzer_contracts.erl"}, > {line,712}]}, > {dialyzer_contracts,picky_contract_check,7, > [{file,"dialyzer_contracts.erl"}, > {line,686}]}]} > Last messages in the log cache: > Reading files and computing callgraph... done in 1.21 secs > Removing edges... done in 0.04 secs > > > On Wed, Mar 18, 2015 at 12:12 PM Bj?rn-Egil Dahlberg > wrote: > >> On 2015-03-18 12:01, Ali Sabil wrote: >> > I tried to create a minimal testcase but I unfortunately haven't been >> > able to. I was running dialyzer on a quite large code base and now >> > even the unpatched dialyzer works without any issue after I fixed all >> > the issues reported by dialyzer. >> >> Ah, well .. I suspect it was the missing clause in find_terminals and I >> had it on a TODO somewhere. Should be included to 17.5. >> >> // Bj?rn-Egil >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ulf@REDACTED Tue Apr 7 18:45:21 2015 From: ulf@REDACTED (Ulf Wiger) Date: Tue, 7 Apr 2015 18:45:21 +0200 Subject: [erlang-questions] exometer core only In-Reply-To: References: Message-ID: <65FC5D9B-BEA3-4DB2-BF49-376117489742@feuerlabs.com> > On 07 Apr 2015, at 07:40, Tino Breddin wrote: > > As for using the split, it is stable so far and all tests are green, however not all reporters are tested. So feel free to take it for a test. Put slightly differently, exometer_core is as well tested as ever exometer was. :) Several projects already use exometer_core, and I?ve seen nothing to indicate that it would be worse than exometer 1.1. On the contrary, several fixes and improvements have been made in exometer_core, and more is coming, as it?s actively maintained. BR, Ulf W Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. http://feuerlabs.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto@REDACTED Wed Apr 8 01:41:07 2015 From: roberto@REDACTED (Roberto Ostinelli) Date: Tue, 7 Apr 2015 16:41:07 -0700 Subject: [erlang-questions] exometer core only In-Reply-To: <65FC5D9B-BEA3-4DB2-BF49-376117489742@feuerlabs.com> References: <65FC5D9B-BEA3-4DB2-BF49-376117489742@feuerlabs.com> Message-ID: Thank you Tino and Ulf. What I'm seeing is that exometer_core still includes folsom, which I'm not using but I guess it's ok. Only one question: reading the docs it's not clear to me how I can add the hostedgraphite reporter to exometer_core. Can you please point me in the right direction? FYI I'm using rebar (if that does even matter). Thank you, r. On Tue, Apr 7, 2015 at 9:45 AM, Ulf Wiger wrote: > > On 07 Apr 2015, at 07:40, Tino Breddin wrote: > > As for using the split, it is stable so far and all tests are green, > however not all reporters are tested. So feel free to take it for a test. > > > Put slightly differently, exometer_core is as well tested as ever exometer > was. :) > > Several projects already use exometer_core, and I?ve seen nothing to > indicate that it would be worse than exometer 1.1. On the contrary, several > fixes and improvements have been made in exometer_core, and more is coming, > as it?s actively maintained. > > BR, > Ulf W > > Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. > http://feuerlabs.com > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tino.breddin@REDACTED Wed Apr 8 07:35:41 2015 From: tino.breddin@REDACTED (Tino Breddin) Date: Wed, 8 Apr 2015 07:35:41 +0200 Subject: [erlang-questions] exometer core only In-Reply-To: References: <65FC5D9B-BEA3-4DB2-BF49-376117489742@feuerlabs.com> Message-ID: Hi, The exometer_report_graphite is now part of exometer [1], thus you need to add that to your dependencies. Then simply create and subscribe the reporter as you've done before. Some hints on configuring [2] and subscribing [3] reporters can be found in the README. LGT [1] https://github.com/Feuerlabs/exometer [2] https://github.com/Feuerlabs/exometer#configuring-graphite-reporter [3] https://github.com/Feuerlabs/exometer#configuring-static-subscriptions On Wed, Apr 8, 2015 at 1:41 AM, Roberto Ostinelli wrote: > Thank you Tino and Ulf. > What I'm seeing is that exometer_core still includes folsom, which I'm not > using but I guess it's ok. > > Only one question: reading the docs it's not clear to me how I can add the > hostedgraphite reporter to exometer_core. > Can you please point me in the right direction? FYI I'm using rebar (if > that does even matter). > > Thank you, > r. > > > On Tue, Apr 7, 2015 at 9:45 AM, Ulf Wiger wrote: > >> >> On 07 Apr 2015, at 07:40, Tino Breddin wrote: >> >> As for using the split, it is stable so far and all tests are green, >> however not all reporters are tested. So feel free to take it for a test. >> >> >> Put slightly differently, exometer_core is as well tested as ever >> exometer was. :) >> >> Several projects already use exometer_core, and I?ve seen nothing to >> indicate that it would be worse than exometer 1.1. On the contrary, several >> fixes and improvements have been made in exometer_core, and more is coming, >> as it?s actively maintained. >> >> BR, >> Ulf W >> >> Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. >> http://feuerlabs.com >> >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eriksoe@REDACTED Wed Apr 8 08:46:16 2015 From: eriksoe@REDACTED (=?UTF-8?Q?Erik_S=C3=B8e_S=C3=B8rensen?=) Date: Wed, 8 Apr 2015 08:46:16 +0200 Subject: [erlang-questions] Warning not emitted on string matching clauses In-Reply-To: References: Message-ID: It would of course make sense to have the compiler warn in this case. However - if my understanding of the compiler implementation is correct - matching of binaries is handled specially, and apparently in such a manner that it doesn't realise that the two binary patterns are identical. /Erik Den 01/04/2015 20.43 skrev "Pierre Fenoll" : > -module(wat_clauses). > -export([authenticate_nouns/1]). > authenticate_nouns([{<<"user_auth">>, _}]) -> 'true'; > authenticate_nouns([{<<"user_auth">>, [<<"recovery">>]}]) -> hi; > authenticate_nouns(_Nouns) -> 'false'. > > > In this code (or > above), the second clause will never match (because of source-order). > But erlc does not complain about it. > > Ferd on IRC mentioned that the compiler might feel free to reorder clauses. > But the call wat_clauses:authenticate_nouns([{<<"user_auth">>, > [<<"recovery">>]}]) > returns true instead of hi, so no reordering is done. > > Dialyzer does not complain either. > > Am I right in thinking we should have a warning about a never matching > clause? > > -- > Pierre Fenoll > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto.ostinelli@REDACTED Wed Apr 8 09:17:32 2015 From: roberto.ostinelli@REDACTED (Roberto Ostinelli) Date: Wed, 8 Apr 2015 00:17:32 -0700 Subject: [erlang-questions] exometer core only In-Reply-To: References: <65FC5D9B-BEA3-4DB2-BF49-376117489742@feuerlabs.com> Message-ID: Thank you Tino, I'm already using exometer with the hostedgraphite reporter, with static subscriptions. My question was if there's any way to use this reporter with exometer_core, I'm probably unable to find how to do it in the docs. Thank you, r. > On 07/apr/2015, at 22:35, Tino Breddin wrote: > > Hi, > > The exometer_report_graphite is now part of exometer [1], thus you need to add that to your dependencies. Then simply create and subscribe the reporter as you've done before. Some hints on configuring [2] and subscribing [3] reporters can be found in the README. > > LGT > > [1] https://github.com/Feuerlabs/exometer > [2] https://github.com/Feuerlabs/exometer#configuring-graphite-reporter > [3] https://github.com/Feuerlabs/exometer#configuring-static-subscriptions > >> On Wed, Apr 8, 2015 at 1:41 AM, Roberto Ostinelli wrote: >> Thank you Tino and Ulf. >> What I'm seeing is that exometer_core still includes folsom, which I'm not using but I guess it's ok. >> >> Only one question: reading the docs it's not clear to me how I can add the hostedgraphite reporter to exometer_core. >> Can you please point me in the right direction? FYI I'm using rebar (if that does even matter). >> >> Thank you, >> r. >> >> >>> On Tue, Apr 7, 2015 at 9:45 AM, Ulf Wiger wrote: >>> >>>> On 07 Apr 2015, at 07:40, Tino Breddin wrote: >>>> >>>> As for using the split, it is stable so far and all tests are green, however not all reporters are tested. So feel free to take it for a test. >>> >>> Put slightly differently, exometer_core is as well tested as ever exometer was. :) >>> >>> Several projects already use exometer_core, and I?ve seen nothing to indicate that it would be worse than exometer 1.1. On the contrary, several fixes and improvements have been made in exometer_core, and more is coming, as it?s actively maintained. >>> >>> BR, >>> Ulf W >>> >>> Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. >>> http://feuerlabs.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From n.oxyde@REDACTED Wed Apr 8 09:52:29 2015 From: n.oxyde@REDACTED (Anthony Ramine) Date: Wed, 8 Apr 2015 09:52:29 +0200 Subject: [erlang-questions] Warning not emitted on string matching clauses In-Reply-To: References: Message-ID: Le 8 avr. 2015 ? 08:46, Erik S?e S?rensen a ?crit : > It would of course make sense to have the compiler warn in this case. > However - if my understanding of the compiler implementation is correct - matching of binaries is handled specially, and apparently in such a manner that it doesn't realise that the two binary patterns are identical. > /Erik That would be a reasonable explanation, but unfortunately it's not the case. If you compile that module to BEAM assembly, you can see very that the two patterns are indeed merged together, there is only one occurrence of "user_auth" in it. {function, authenticate_nouns, 1, 2}. {label,1}. {line,[{location,"wat_clauses.erl",3}]}. {func_info,{atom,wat_clauses},{atom,authenticate_nouns},1}. {label,2}. {test,is_nonempty_list,{f,4},[{x,0}]}. {get_list,{x,0},{x,1},{x,2}}. {test,is_tuple,{f,4},[{x,1}]}. {test,test_arity,{f,4},[{x,1},2]}. {get_tuple_element,{x,1},0,{x,3}}. {get_tuple_element,{x,1},1,{x,4}}. {test,bs_start_match2,{f,4},5,[{x,3},0],{x,5}}. {test,bs_match_string,{f,4},[{x,5},72,{string,"user_auth"}]}. {test,bs_test_tail2,{f,4},[{x,5},0]}. {test,is_nil,{f,3},[{x,2}]}. {move,{atom,true},{x,0}}. return. {label,3}. {test,is_nonempty_list,{f,4},[{x,4}]}. {get_list,{x,4},{x,6},{x,7}}. {test,bs_start_match2,{f,4},8,[{x,6},0],{x,8}}. {test,bs_match_string,{f,4},[{x,8},64,{string,"recovery"}]}. {test,bs_test_tail2,{f,4},[{x,8},0]}. {test,is_nil,{f,4},[{x,7}]}. {test,is_nil,{f,4},[{x,2}]}. {move,{atom,hi},{x,0}}. return. {label,4}. {move,{atom,false},{x,0}}. return. From tino.breddin@REDACTED Wed Apr 8 10:02:01 2015 From: tino.breddin@REDACTED (Tino Breddin) Date: Wed, 08 Apr 2015 10:02:01 +0200 Subject: [erlang-questions] exometer core only In-Reply-To: References: <65FC5D9B-BEA3-4DB2-BF49-376117489742@feuerlabs.com> Message-ID: <5524E079.2050609@gmail.com> exometer itself (versions later than 1.1) depends directly on exometer_core. Thus, if you use that, you use exometer_core implicitely as well. See https://github.com/Feuerlabs/exometer/blob/master/rebar.config#L15 LGT > Roberto Ostinelli > 8 Apr 2015 09:17 > Thank you Tino, > I'm already using exometer with the hostedgraphite reporter, with > static subscriptions. My question was if there's any way to use this > reporter with exometer_core, I'm probably unable to find how to do it > in the docs. > > Thank you, > r. > > > > On 07/apr/2015, at 22:35, Tino Breddin > wrote: > > Ulf Wiger > 7 Apr 2015 18:45 > > > Put slightly differently, exometer_core is as well tested as ever > exometer was. :) > > Several projects already use exometer_core, and I?ve seen nothing to > indicate that it would be worse than exometer 1.1. On the contrary, > several fixes and improvements have been made in exometer_core, and > more is coming, as it?s actively maintained. > > BR, > Ulf W > > Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. > http://feuerlabs.com > > > > Tino Breddin > 7 Apr 2015 07:40 > Hi, > > The split was performed in order to simplify the use of core exometer > functionality. As you can see the new `exometer` is mostly a container > for reporters which have dependencies which you might not want to use > in your system. The reporters which are kept in exometer_core are > dependency-free. You can still use the "feature selection" during > build to exclude certain reporters and their dependencies, but the > main build has become simpler this way. > > As for using the split, it is stable so far and all tests are green, > however not all reporters are tested. So feel free to take it for a test. > > LGT > > > Roberto Ostinelli > 6 Apr 2015 22:03 > All, > Reading the HEAD documentation of exometer: > > NOTE: Exometer has been split into exometer_core, and exometer (as > well as separate reporter applications). The latest monolithic version > of Exometer is 1.1. > > I'm currently using 1.1, however there are a bunch of things that I do > not use (I'm only using custom static metrics with hostedgraphite, > nothing from folsom, etc). I'm guessing that's the rationale into > splitting exometer, so that one can choose which components to use. > > Though, I'm not understanding if I should start using exometer_core, > or if the split is still a WIP. > > Anyone can shed some light? > > Thank you, > r. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: postbox-contact.jpg Type: image/jpeg Size: 1218 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: postbox-contact.jpg Type: image/jpeg Size: 1152 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: postbox-contact.jpg Type: image/jpeg Size: 1382 bytes Desc: not available URL: From wallentin.dahlberg@REDACTED Wed Apr 8 10:27:43 2015 From: wallentin.dahlberg@REDACTED (=?UTF-8?Q?Bj=C3=B6rn=2DEgil_Dahlberg?=) Date: Wed, 8 Apr 2015 10:27:43 +0200 Subject: [erlang-questions] Warning not emitted on string matching clauses In-Reply-To: References: Message-ID: This is a part of the compiler I'm not really an expert on, I would defer to Robert or Bj?rn in this case. However, I believe it is the "variable split" that happens in match compilation in kernel that is the cause of this. First I believed it was just the order in which the compiler chooses the values to match (left to right) but i don't think it's the case here (though i've seen that too). Ideally, we shouldn't traverse the clauses left to right but instead weight and sniff out the optimal matching path in the tree but that's a debate for another time perhaps (and a fun one). One could also question why the kernel should emit this warning and not the linter. I guess pragmatism. 2015-04-08 9:52 GMT+02:00 Anthony Ramine : > Le 8 avr. 2015 ? 08:46, Erik S?e S?rensen a ?crit : > > > It would of course make sense to have the compiler warn in this case. > > However - if my understanding of the compiler implementation is correct > - matching of binaries is handled specially, and apparently in such a > manner that it doesn't realise that the two binary patterns are identical. > > /Erik > > That would be a reasonable explanation, but unfortunately it's not the > case. If you compile that module to BEAM assembly, you can see very that > the two patterns are indeed merged together, there is only one occurrence > of "user_auth" in it. > > {function, authenticate_nouns, 1, 2}. > {label,1}. > {line,[{location,"wat_clauses.erl",3}]}. > {func_info,{atom,wat_clauses},{atom,authenticate_nouns},1}. > {label,2}. > {test,is_nonempty_list,{f,4},[{x,0}]}. > {get_list,{x,0},{x,1},{x,2}}. > {test,is_tuple,{f,4},[{x,1}]}. > {test,test_arity,{f,4},[{x,1},2]}. > {get_tuple_element,{x,1},0,{x,3}}. > {get_tuple_element,{x,1},1,{x,4}}. > {test,bs_start_match2,{f,4},5,[{x,3},0],{x,5}}. > {test,bs_match_string,{f,4},[{x,5},72,{string,"user_auth"}]}. > {test,bs_test_tail2,{f,4},[{x,5},0]}. > {test,is_nil,{f,3},[{x,2}]}. > {move,{atom,true},{x,0}}. > return. > {label,3}. > {test,is_nonempty_list,{f,4},[{x,4}]}. > {get_list,{x,4},{x,6},{x,7}}. > {test,bs_start_match2,{f,4},8,[{x,6},0],{x,8}}. > {test,bs_match_string,{f,4},[{x,8},64,{string,"recovery"}]}. > {test,bs_test_tail2,{f,4},[{x,8},0]}. > {test,is_nil,{f,4},[{x,7}]}. > {test,is_nil,{f,4},[{x,2}]}. > {move,{atom,hi},{x,0}}. > return. > {label,4}. > {move,{atom,false},{x,0}}. > return. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vladdu55@REDACTED Wed Apr 8 11:36:22 2015 From: vladdu55@REDACTED (Vlad Dumitrescu) Date: Wed, 8 Apr 2015 11:36:22 +0200 Subject: [erlang-questions] erl_scan: CR after comment Message-ID: Hi! I was about to create a PR for this, but maybe I should ask what people think. erl_scan creates comment tokens up to a LF character. On Windows, this means that the comment includes a CR character at the end. This requires special handling when the tokens are manipulated outside erlang (by an editor, for example) because Erlang and the editor consider the end of the token to lie on different lines. Is it meaningful to let erl_scan end comments before both CR and LF? The downside is that there will be an extra whitespace token for the CR between consecutive comments. best regards, Vlad -------------- next part -------------- An HTML attachment was scrubbed... URL: From khaelin@REDACTED Wed Apr 8 12:49:16 2015 From: khaelin@REDACTED (Nicolas Martyanoff) Date: Wed, 8 Apr 2015 12:49:16 +0200 Subject: [erlang-questions] starting otp application dependencies Message-ID: <20150408104916.GA5025@valhala.home> Hi, Given an OTP application with dependencies, is there a way to automatically start these dependencies before starting the application itself, during development ? In production, I can create a release (using relx for example). But during development, I compile the application with rebar, then run erl (erl -pa ebin deps/*/ebin), then my application (application:start(App)). Even though app(4) indicates that the 'applications' list in the app file contains "All applications which must be started before this application is allowed to be started", application:start(App) does not automatically start all dependencies of App. This also means that indirect dependencies must be found and started manually. Therefore if my application depends on webmachine, I have to find all the dependencies of webmachine (and there are more than a few). There obviously is a simpler way. A trick would be to add a function available in erl which parse an app file and load all dependencies, but then I still have to find all indirect dependencies. How do you avoid these issues yourself ? Thank you in advance. -- Nicolas Martyanoff http://wandrian.net khaelin@REDACTED From sergej.jurecko@REDACTED Wed Apr 8 13:43:31 2015 From: sergej.jurecko@REDACTED (=?UTF-8?Q?Sergej_Jure=C4=8Dko?=) Date: Wed, 8 Apr 2015 13:43:31 +0200 Subject: [erlang-questions] starting otp application dependencies In-Reply-To: <20150408104916.GA5025@valhala.home> References: <20150408104916.GA5025@valhala.home> Message-ID: application:ensure_all_started(app). Sergej On Apr 8, 2015 1:41 PM, "Nicolas Martyanoff" wrote: > Hi, > > Given an OTP application with dependencies, is there a way to automatically > start these dependencies before starting the application itself, during > development ? > > In production, I can create a release (using relx for example). But during > development, I compile the application with rebar, then run erl (erl -pa > ebin > deps/*/ebin), then my application (application:start(App)). > > Even though app(4) indicates that the 'applications' list in the app file > contains "All applications which must be started before this application > is > allowed to be started", application:start(App) does not automatically > start all > dependencies of App. > > This also means that indirect dependencies must be found and started > manually. > Therefore if my application depends on webmachine, I have to find all the > dependencies of webmachine (and there are more than a few). > > There obviously is a simpler way. A trick would be to add a function > available > in erl which parse an app file and load all dependencies, but then I still > have to find all indirect dependencies. > > How do you avoid these issues yourself ? > > Thank you in advance. > > -- > Nicolas Martyanoff > http://wandrian.net > khaelin@REDACTED > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From khaelin@REDACTED Wed Apr 8 13:48:01 2015 From: khaelin@REDACTED (Nicolas Martyanoff) Date: Wed, 8 Apr 2015 13:48:01 +0200 Subject: [erlang-questions] starting otp application dependencies In-Reply-To: References: <20150408104916.GA5025@valhala.home> Message-ID: <20150408114801.GB5025@valhala.home> On 2015-04-08 13:43, Sergej Jure?ko wrote: > application:ensure_all_started(app). Exactly what I needed. Thank you ! Regards, -- Nicolas Martyanoff http://wandrian.net khaelin@REDACTED From mononcqc@REDACTED Wed Apr 8 14:25:57 2015 From: mononcqc@REDACTED (Fred Hebert) Date: Wed, 8 Apr 2015 08:25:57 -0400 Subject: [erlang-questions] =?utf-8?q?=5BANN=5D_Other_Erldocs_=CE=B1?= In-Reply-To: References: Message-ID: <20150408122556.GA4884@ferdair.local> On 04/01, Pierre Fenoll wrote: >Like erldocs.com? > >Now you can access a regularly-updated repository of your favorite Erlang >projects' >documentation over at http://other.erldocs.com/ > >You can search for a repository on the home page >or just paste a repo's URL in the OpenSearch bar. > Just wanted to say that this is pretty damn cool! Thanks for the work on this! Regards, Fred. From eric.pailleau@REDACTED Wed Apr 8 21:51:49 2015 From: eric.pailleau@REDACTED (=?ISO-8859-1?Q?=C9ric_Pailleau?=) Date: Wed, 08 Apr 2015 21:51:49 +0200 Subject: [erlang-questions] Binary prettyprinting In-Reply-To: Message-ID: hello, This may help to do this : https://github.com/crownedgrouse/swab (new native features are welcome ;-)...) Regards Le?7 avr. 2015 12:43, Vladimir Gordeev a ?crit?: > > Is there is a hack that?globally?makes format binaries differently? > > Sometimes while debugging I do print some data. Sometimes this data is compound and two-three levels down there is a long binary value. > > I want to see <<39,122,6,247,77,47,80,222,192,57,233,137,230, ... 1500 bytes total>> instead of actual 1,5 Kb data in text. > > Of course I can write such formatting function, but I have to use it explicitly each time (god, my three-level compound data). > > ?I want this only on my local machine, so any hacks welcome. Any ideas? From lloyd@REDACTED Wed Apr 8 22:47:02 2015 From: lloyd@REDACTED (lloyd@REDACTED) Date: Wed, 8 Apr 2015 16:47:02 -0400 (EDT) Subject: [erlang-questions] Amazon API -- Lookup by ISBN Message-ID: <1428526022.03741188@apps.rackspace.com> Hello, I'm striving to look up books in Amazon's db by ISBN. At first blush it looks easy enough: http://docs.aws.amazon.com/AWSECommerceService/latest/DG/EX_LookupbyISBN.html But the last item, Signature, baffles me. Procedure here: http://docs.aws.amazon.com/AWSECommerceService/latest/DG/rest-signature.html I'm fine with this until I hit step 4: -- Sort parameter/value pairs by byte value --- I can see how to do this manually, but don't know how put Erlang to the task And I'm really stumped when I hit step 8: -- Calculate an RFC 2104-compliant HMAC with the SHA256 hash algorithm Any help? Better yet, does anyone have actual code to make such requests they're willing to share? NOTE: Dave Thomas solved this problem way back in 2007. But looks like Amazon has changed their request format: http://pragdave.me/blog/2007/04/15/a-first-erlang-program/ Many thanks, LRP From lloyd@REDACTED Wed Apr 8 23:50:53 2015 From: lloyd@REDACTED (lloyd@REDACTED) Date: Wed, 8 Apr 2015 17:50:53 -0400 (EDT) Subject: [erlang-questions] Amazon API -- Lookup by ISBN -- amended In-Reply-To: <1428526022.03741188@apps.rackspace.com> References: <1428526022.03741188@apps.rackspace.com> Message-ID: <1428529853.686926152@apps.rackspace.com> Hello again, OK, I've found erlcloud. Looks like it has what I need. But user documentation is far to skimpy for my feeble brain: https://github.com/gleber/erlcloud Can some kind soul help me translate this example... http://docs.aws.amazon.com/AWSECommerceService/latest/DG/EX_LookupbyISBN.html ...into the appropriate erlcloud request function? Thanks, LRP -----Original Message----- From: lloyd@REDACTED Sent: Wednesday, April 8, 2015 4:47pm To: "Erlang Questions" Subject: [erlang-questions] Amazon API -- Lookup by ISBN Hello, I'm striving to look up books in Amazon's db by ISBN. At first blush it looks easy enough: http://docs.aws.amazon.com/AWSECommerceService/latest/DG/EX_LookupbyISBN.html But the last item, Signature, baffles me. Procedure here: http://docs.aws.amazon.com/AWSECommerceService/latest/DG/rest-signature.html I'm fine with this until I hit step 4: -- Sort parameter/value pairs by byte value --- I can see how to do this manually, but don't know how put Erlang to the task And I'm really stumped when I hit step 8: -- Calculate an RFC 2104-compliant HMAC with the SHA256 hash algorithm Any help? Better yet, does anyone have actual code to make such requests they're willing to share? NOTE: Dave Thomas solved this problem way back in 2007. But looks like Amazon has changed their request format: http://pragdave.me/blog/2007/04/15/a-first-erlang-program/ Many thanks, LRP _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions From eric.pailleau@REDACTED Wed Apr 8 23:01:48 2015 From: eric.pailleau@REDACTED (=?ISO-8859-1?Q?=C9ric_Pailleau?=) Date: Wed, 08 Apr 2015 23:01:48 +0200 Subject: [erlang-questions] Binary prettyprinting In-Reply-To: Message-ID: <7b78b2ab-bd6f-4d28-9816-11bc7de6f275@email.android.com> Hi, I forgot the more important in your question... You may use dbg:tracer with a fun checking if the binary is big, and then apply a swab macro or you own formatting on it. Le?7 avr. 2015 12:43, Vladimir Gordeev a ?crit?: > > Is there is a hack that?globally?makes format binaries differently? > > Sometimes while debugging I do print some data. Sometimes this data is compound and two-three levels down there is a long binary value. > > I want to see <<39,122,6,247,77,47,80,222,192,57,233,137,230, ... 1500 bytes total>> instead of actual 1,5 Kb data in text. > > Of course I can write such formatting function, but I have to use it explicitly each time (god, my three-level compound data). > > ?I want this only on my local machine, so any hacks welcome. Any ideas? From g@REDACTED Thu Apr 9 01:45:35 2015 From: g@REDACTED (Garrett Smith) Date: Wed, 8 Apr 2015 18:45:35 -0500 Subject: [erlang-questions] Amazon API -- Lookup by ISBN -- amended In-Reply-To: <1428529853.686926152@apps.rackspace.com> References: <1428526022.03741188@apps.rackspace.com> <1428529853.686926152@apps.rackspace.com> Message-ID: I'm completely punting on this problem, but can you take some code examples provided in other languages and retrofit them for Erlang? If I'm hitting a wall, I'll get something working in, say, Python - and then work toward forming an identical request in Erlang. Sorry for the lack of specificity here - but this is the approach I'd take in your case. OMG, I just realized that I top posted here! On Wed, Apr 8, 2015 at 4:50 PM, wrote: > Hello again, > > OK, I've found erlcloud. Looks like it has what I need. But user > documentation is far to skimpy for my feeble brain: > > https://github.com/gleber/erlcloud > > Can some kind soul help me translate this example... > > > http://docs.aws.amazon.com/AWSECommerceService/latest/DG/EX_LookupbyISBN.html > > ...into the appropriate erlcloud request function? > > Thanks, > > LRP > > -----Original Message----- > From: lloyd@REDACTED > Sent: Wednesday, April 8, 2015 4:47pm > To: "Erlang Questions" > Subject: [erlang-questions] Amazon API -- Lookup by ISBN > > Hello, > > I'm striving to look up books in Amazon's db by ISBN. At first blush it > looks easy enough: > > > http://docs.aws.amazon.com/AWSECommerceService/latest/DG/EX_LookupbyISBN.html > > But the last item, Signature, baffles me. Procedure here: > > > http://docs.aws.amazon.com/AWSECommerceService/latest/DG/rest-signature.html > > I'm fine with this until I hit step 4: > > -- Sort parameter/value pairs by byte value --- I can see how to do this > manually, but don't know how put Erlang to the task > > And I'm really stumped when I hit step 8: > > -- Calculate an RFC 2104-compliant HMAC with the SHA256 hash algorithm > > Any help? Better yet, does anyone have actual code to make such requests > they're willing to share? > > NOTE: Dave Thomas solved this problem way back in 2007. But looks like > Amazon has changed their request format: > > http://pragdave.me/blog/2007/04/15/a-first-erlang-program/ > > Many thanks, > > LRP > > > > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From darach@REDACTED Thu Apr 9 02:07:57 2015 From: darach@REDACTED (Darach Ennis) Date: Thu, 9 Apr 2015 01:07:57 +0100 Subject: [erlang-questions] Amazon API -- Lookup by ISBN In-Reply-To: <1428526022.03741188@apps.rackspace.com> References: <1428526022.03741188@apps.rackspace.com> Message-ID: If you don't need the rank information you can get the metadata through composing a URL against the library of congress database: http://lx2.loc.gov:210/lcdb?version=1.1&operation=searchRetrieve&query=bath.isbn=0448421658&maximumRecords=1&recordSchema=mods This will deliver an XML document. You can get similar information from Google's API: http://www.google.com/books/feeds/volumes/?q=ISBN%3C0448421658%3E Both are simple HTTP GET requests so unless you absolutely need data unique to Amazon's DB perhaps that would suffice. Cheers, Darach. On Wed, Apr 8, 2015 at 9:47 PM, wrote: > Hello, > > I'm striving to look up books in Amazon's db by ISBN. At first blush it > looks easy enough: > > > http://docs.aws.amazon.com/AWSECommerceService/latest/DG/EX_LookupbyISBN.html > > But the last item, Signature, baffles me. Procedure here: > > > http://docs.aws.amazon.com/AWSECommerceService/latest/DG/rest-signature.html > > I'm fine with this until I hit step 4: > > -- Sort parameter/value pairs by byte value --- I can see how to do this > manually, but don't know how put Erlang to the task > > And I'm really stumped when I hit step 8: > > -- Calculate an RFC 2104-compliant HMAC with the SHA256 hash algorithm > > Any help? Better yet, does anyone have actual code to make such requests > they're willing to share? > > NOTE: Dave Thomas solved this problem way back in 2007. But looks like > Amazon has changed their request format: > > http://pragdave.me/blog/2007/04/15/a-first-erlang-program/ > > Many thanks, > > LRP > > > > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jpsiyu@REDACTED Thu Apr 9 09:28:24 2015 From: jpsiyu@REDACTED (Fisher) Date: Thu, 9 Apr 2015 15:28:24 +0800 (CST) Subject: [erlang-questions] try horse, got undefined parse transform error Message-ID: <4fe1b6c2.d8f6.14c9d146ec9.Coremail.jpsiyu@163.com> Hi, all! I am trying horse:https://github.com/extend/horse I got it in this folder: MyProject -> deps -> horse I added this in my makefile like the website says: deps/horse: gitclone-n--https://github.com/extend/horse $(DEPS_DIR)/horse cd $(DEPS_DIR)/horse ; git checkout -q master $(MAKE) -C $(DEPS_DIR)/horse perfs: ERLC_OPTS += -DPERF=1 +'{parse_transform, horse_autoexport}' perfs: clean deps deps/horse app $(gen_verbose) erl -noshell -pa ebin deps/horse/ebin \ -eval'horse:app_perf($(PROJECT)), init:stop().' then i compile: make perfs and i got this error: src/test01.erl: undefined parse transform 'horse_autoexport' i tried add {erl_opts, [{parse_transform, horse_autoexport}]} to my app file or i add -compile([{parse_transform, horse_autoexport}]) to test01.erl module but the error continue. anyone can help? thanks! _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Thu Apr 9 11:24:02 2015 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Thu, 09 Apr 2015 12:24:02 +0300 Subject: [erlang-questions] try horse, got undefined parse transform error In-Reply-To: <4fe1b6c2.d8f6.14c9d146ec9.Coremail.jpsiyu@163.com> References: <4fe1b6c2.d8f6.14c9d146ec9.Coremail.jpsiyu@163.com> Message-ID: <55264532.1090603@ninenines.eu> Hey, These instructions apply only if you use erlang.mk. For rebar I think you will need to add the project as a dependency and then the option to use the parse_transform, though I am not sure on the details. Cheers, On 04/09/2015 10:28 AM, Fisher wrote: > Hi, all! > I am trying horse:https://github.com/extend/horse > I got it in this folder: MyProject -> deps -> horse > I added this in my makefile like the website says: > > deps/horse: > git clone -n -- https://github.com/extend/horse$(DEPS_DIR)/horse > cd $(DEPS_DIR)/horse ; git checkout -q master > $(MAKE) -C $(DEPS_DIR)/horse > > perfs: ERLC_OPTS += -DPERF=1 +'{parse_transform, horse_autoexport}' > perfs: clean deps deps/horse app > $(gen_verbose) erl -noshell -pa ebin deps/horse/ebin \ > -eval 'horse:app_perf($(PROJECT)), init:stop().' > > then i compile: make perfs > and i got this error: > > src/test01.erl: undefined parse transform 'horse_autoexport' > > i tried add {erl_opts, [{parse_transform, horse_autoexport}]} to my app file > or i add -compile([{parse_transform, horse_autoexport}]) to test01.erl > module > but the error continue. > > anyone can help? thanks! > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin http://ninenines.eu From ryan.bamford@REDACTED Thu Apr 9 11:33:28 2015 From: ryan.bamford@REDACTED (Ryan Bamford) Date: Thu, 9 Apr 2015 09:33:28 +0000 Subject: [erlang-questions] Erlang tls dist issues Message-ID: Hi am having an issue with the use of inet_tls_dist.while using rabbitmq Setup - I have a 3 node rabbit cluster running on 3 separate centos machines In the erlang otp we have fixed the issue of SSL not binding to the correct ports (respecting the inet_dist_listen_min and inet_dist_listen_max values) porting the code from the tcp_dist the final issue we are having is in the restarting of the nodes on running a rabbitmqctl stop MOST of the time in the other 2 nodes we get the error in the log =ERROR REPORT==== 8-Apr-2015::16:36:45 === SSL: connection: ssl_alert.erl:92:Fatal error: internal error the stack trace from the error area the error is created is [{ssl_alert,decode,3,[{file,"ssl_alert.erl"},{line,92}]}, {tls_connection,next_state,4,[{file,"tls_connection.erl"},{line,400}]}, {gen_fsm,handle_msg,7,[{file,"gen_fsm.erl"},{line,503}]}, {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,237}]}] this does not occur when doing rabbitmqctl stop_app Due to this we can see that the issue happens in the halt stop_and_halt() -> try stop() after rabbit_log:info("Halting Erlang VM~n", []), init:stop() end, ok. when it does the init:stop() in the init.erl what it seems to be is a corrupted message being sent at the point when the ssl distriibution is being closed in the instances when is operates correctly the SSL close message is received in loop_con (in the tls_proxy) at which point it calls the tcp close then the message =INFO REPORT==== 8-Apr-2015::16:10:34 === node 'rabbit@REDACTED' down: connection_closed is shown when it errors we never reach the receipt of the ssl close Dont know if anyone has seen this before looking for any help if they have sorry if ive missed anything I am quite an erlang noob Regards Ryan -- Ryan Bamford Senior Research & Development Engineer Eckoh UK Limited Telford House, Corner Hall Hemel Hempstead, Hertfordshire HP3 9HN T 01442 458316 M n/a W www.eckoh.com [cid:imageddfd0a.PNG@REDACTED] [cid:image24bae3.PNG@REDACTED] US Headquarters: Eckoh, Inc. 11811 N. Tatum Blvd., Suite 3031 Phoenix, AZ 85028 This communication contains information, which is confidential and may also be privileged. It is for the exclusive use of the intended recipient(s) only. If you are not the intended recipient(s) please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. If you have received this communication in error please return it to the sender and then delete it. Opinions expressed in this message are those of the author, and are not binding on the company. Registered in England and Wales, No. 2796531 Registered office: Telford House, Corner Hall, Hemel Hempstead, Hertfordshire HP3 9HN A member of the Eckoh Plc group of companies. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: imageddfd0a.PNG Type: image/png Size: 1931 bytes Desc: imageddfd0a.PNG URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image24bae3.PNG Type: image/png Size: 1275 bytes Desc: image24bae3.PNG URL: From essen@REDACTED Thu Apr 9 11:54:26 2015 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Thu, 09 Apr 2015 12:54:26 +0300 Subject: [erlang-questions] try horse, got undefined parse transform error In-Reply-To: <2eead9fc.1273c.14c9d92d81b.Coremail.jpsiyu@163.com> References: <4fe1b6c2.d8f6.14c9d146ec9.Coremail.jpsiyu@163.com> <55264532.1090603@ninenines.eu> <427dcb49.11f82.14c9d82583e.Coremail.jpsiyu@163.com> <55264769.4080003@ninenines.eu> <2eead9fc.1273c.14c9d92d81b.Coremail.jpsiyu@163.com> Message-ID: <55264C52.4010507@ninenines.eu> Not too sure what you are doing there. This should work all in the Makefile file, if using horse's erlang.mk: PROJECT = test include deps/horse/erlang.mk deps/horse: git clone -n -- https://github.com/extend/horse $(DEPS_DIR)/horse cd $(DEPS_DIR)/horse ; git checkout -q master $(MAKE) -C $(DEPS_DIR)/horse perfs: ERLC_OPTS += -DPERF=1 +'{parse_transform, horse_autoexport}' perfs: clean deps deps/horse app $(gen_verbose) erl -noshell -pa ebin deps/horse/ebin \ -eval 'horse:app_perf($(PROJECT)), init:stop().' If you use a more recent erlang.mk it's much simpler, see https://github.com/ninenines/cowlib/blob/master/Makefile for example. On 04/09/2015 12:46 PM, Fisher wrote: > > Here is my test: > > makefile: > PROJECT = test > include erlang.mk > > erlang.mk > include deps/horse/erlang.mk > > .PHONY: all start clean > > INCLUDE = "include" > > EBIN = "ebin" > > HORSE = "deps/horse/ebin" > > DEPS_DIR = "deps" > > MODULE = "test02.erl" > > all: > erlc -o ${EBIN} -I ${INCLUDE} \ > src/*.erl > > start: > erl -pa ${EBIN} ${HORSE} \ > -eval 'horse:app_perf(horse), init:stop().' > > deps/horse: > git clone -n -- https://github.com/extend/horse $(DEPS_DIR)/horse > cd $(DEPS_DIR)/horse; git checkout -q master > $(MAKE) -C $(DEPS_DIR)/horse > > perfs: ERLC_OPTS += -DPERF=1 +'{parse_transform, horse_autoexport}' > %perfs: clean deps deps/horse app > perfs: deps deps/horse app > $(gen_verbose) erl -noshell -pa ebin deps/horse/ebin \ > -eval 'horse:app_perf($(PROJECT)), init:stop().' > > > #clean: > # rm -rf ebin/*.beam > > and my test: > -module(test01). > -compile({parse_transform, horse_autoexport}). > -export([test_fun/0, > horse_do_nothing/0, > horse_rfc2019/0 > ]). > > -include("test.hrl"). > > test_fun() -> > io:format("this is Test01/test01~n"), ok. > > > horse_do_nothing() -> > io:format("test01 running horse test~n"), > ok. > > horse_rfc2019() -> > horse:repeat(100000, test_fun()). > > > > > > At 2015-04-09 17:33:29, "Lo?c Hoguin" > wrote: >>Show me the Makefile. >> >>On 04/09/2015 12:28 PM, Fisher wrote: >>> Thanks for your answer! >>> and i already added an include to my makefile like this: >>> include deps/horse/erlang.mk !!! >>> and the error still here -.- >>> >>> >>> >>> >>> >>> >>> At 2015-04-09 17:24:02, "Lo?c Hoguin" > wrote: >>>>Hey, >>>> >>>>These instructions apply only if you use erlang.mk. For rebar I think >>>>you will need to add the project as a dependency and then the option to >>>>use the parse_transform, though I am not sure on the details. >>>> >>>>Cheers, >>>> >>>>On 04/09/2015 10:28 AM, Fisher wrote: >>>>> Hi, all! >>>>> I am trying horse:https://github.com/extend/horse >>>>> I got it in this folder: MyProject -> deps -> horse >>>>> I added this in my makefile like the website says: >>>>> >>>>> deps/horse: >>>>> git clone -n -- https://github.com/extend/horse$(DEPS_DIR)/horse >>>>> cd $(DEPS_DIR)/horse ; git checkout -q master >>>>> $(MAKE) -C $(DEPS_DIR)/horse >>>>> >>>>> perfs: ERLC_OPTS += -DPERF=1 +'{parse_transform, horse_autoexport}' >>>>> perfs: clean deps deps/horse app >>>>> $(gen_verbose) erl -noshell -pa ebin deps/horse/ebin \ >>>>> -eval 'horse:app_perf($(PROJECT)), init:stop().' >>>>> >>>>> then i compile: make perfs >>>>> and i got this error: >>>>> >>>>> src/test01.erl: undefined parse transform 'horse_autoexport' >>>>> >>>>> i tried add {erl_opts, [{parse_transform, horse_autoexport}]} to my app file >>>>> or i add -compile([{parse_transform, horse_autoexport}]) to test01.erl >>>>> module >>>>> but the error continue. >>>>> >>>>> anyone can help? thanks! >>>>> >>>>> _______________________________________________ >>>>> erlang-questions mailing list >>>>>erlang-questions@REDACTED >>>>> http://erlang.org/mailman/listinfo/erlang-questions >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> erlang-questions mailing list >>>>>erlang-questions@REDACTED >>>>> http://erlang.org/mailman/listinfo/erlang-questions >>>>> >>>> >>>>-- >>>>Lo?c Hoguin >>>>http://ninenines.eu >>> >> >>-- >>Lo?c Hoguin >>http://ninenines.eu > -- Lo?c Hoguin http://ninenines.eu From jpsiyu@REDACTED Thu Apr 9 11:46:28 2015 From: jpsiyu@REDACTED (Fisher) Date: Thu, 9 Apr 2015 17:46:28 +0800 (CST) Subject: [erlang-questions] try horse, got undefined parse transform error In-Reply-To: <55264769.4080003@ninenines.eu> References: <4fe1b6c2.d8f6.14c9d146ec9.Coremail.jpsiyu@163.com> <55264532.1090603@ninenines.eu> <427dcb49.11f82.14c9d82583e.Coremail.jpsiyu@163.com> <55264769.4080003@ninenines.eu> Message-ID: <2eead9fc.1273c.14c9d92d81b.Coremail.jpsiyu@163.com> Here is my test: makefile: PROJECT = test include erlang.mk erlang.mk include deps/horse/erlang.mk .PHONY: all start clean INCLUDE = "include" EBIN = "ebin" HORSE = "deps/horse/ebin" DEPS_DIR = "deps" MODULE = "test02.erl" all: erlc -o ${EBIN} -I ${INCLUDE} \ src/*.erl start: erl -pa ${EBIN} ${HORSE} \ -eval 'horse:app_perf(horse), init:stop().' deps/horse: git clone -n -- https://github.com/extend/horse $(DEPS_DIR)/horse cd $(DEPS_DIR)/horse; git checkout -q master $(MAKE) -C $(DEPS_DIR)/horse perfs: ERLC_OPTS += -DPERF=1 +'{parse_transform, horse_autoexport}' %perfs: clean deps deps/horse app perfs: deps deps/horse app $(gen_verbose) erl -noshell -pa ebin deps/horse/ebin \ -eval 'horse:app_perf($(PROJECT)), init:stop().' #clean: # rm -rf ebin/*.beam and my test: -module(test01). -compile({parse_transform, horse_autoexport}). -export([test_fun/0, horse_do_nothing/0, horse_rfc2019/0 ]). -include("test.hrl"). test_fun() -> io:format("this is Test01/test01~n"), ok. horse_do_nothing() -> io:format("test01 running horse test~n"), ok. horse_rfc2019() -> horse:repeat(100000, test_fun()). At 2015-04-09 17:33:29, "Lo?c Hoguin" wrote: >Show me the Makefile. > >On 04/09/2015 12:28 PM, Fisher wrote: >> Thanks for your answer! >> and i already added an include to my makefile like this: >> include deps/horse/erlang.mk !!! >> and the error still here -.- >> >> >> >> >> >> >> At 2015-04-09 17:24:02, "Lo?c Hoguin" wrote: >>>Hey, >>> >>>These instructions apply only if you use erlang.mk. For rebar I think >>>you will need to add the project as a dependency and then the option to >>>use the parse_transform, though I am not sure on the details. >>> >>>Cheers, >>> >>>On 04/09/2015 10:28 AM, Fisher wrote: >>>> Hi, all! >>>> I am trying horse:https://github.com/extend/horse >>>> I got it in this folder: MyProject -> deps -> horse >>>> I added this in my makefile like the website says: >>>> >>>> deps/horse: >>>> git clone -n -- https://github.com/extend/horse$(DEPS_DIR)/horse >>>> cd $(DEPS_DIR)/horse ; git checkout -q master >>>> $(MAKE) -C $(DEPS_DIR)/horse >>>> >>>> perfs: ERLC_OPTS += -DPERF=1 +'{parse_transform, horse_autoexport}' >>>> perfs: clean deps deps/horse app >>>> $(gen_verbose) erl -noshell -pa ebin deps/horse/ebin \ >>>> -eval 'horse:app_perf($(PROJECT)), init:stop().' >>>> >>>> then i compile: make perfs >>>> and i got this error: >>>> >>>> src/test01.erl: undefined parse transform 'horse_autoexport' >>>> >>>> i tried add {erl_opts, [{parse_transform, horse_autoexport}]} to my app file >>>> or i add -compile([{parse_transform, horse_autoexport}]) to test01.erl >>>> module >>>> but the error continue. >>>> >>>> anyone can help? thanks! >>>> >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> >>>> >>>> >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> >>> >>>-- >>>Lo?c Hoguin >>>http://ninenines.eu >> > >-- >Lo?c Hoguin >http://ninenines.eu -------------- next part -------------- An HTML attachment was scrubbed... URL: From jpsiyu@REDACTED Thu Apr 9 12:08:32 2015 From: jpsiyu@REDACTED (Fisher) Date: Thu, 9 Apr 2015 18:08:32 +0800 (CST) Subject: [erlang-questions] try horse, got undefined parse transform error In-Reply-To: <55264C52.4010507@ninenines.eu> References: <4fe1b6c2.d8f6.14c9d146ec9.Coremail.jpsiyu@163.com> <55264532.1090603@ninenines.eu> <427dcb49.11f82.14c9d82583e.Coremail.jpsiyu@163.com> <55264769.4080003@ninenines.eu> <2eead9fc.1273c.14c9d92d81b.Coremail.jpsiyu@163.com> <55264C52.4010507@ninenines.eu> Message-ID: <7714f24e.12e0e.14c9da709cb.Coremail.jpsiyu@163.com> Your help is useful ! I solved it ! Thanks a lot !! At 2015-04-09 17:54:26, "Lo?c Hoguin" wrote: >Not too sure what you are doing there. > >This should work all in the Makefile file, if using horse's erlang.mk: > >PROJECT = test >include deps/horse/erlang.mk > >deps/horse: > git clone -n -- https://github.com/extend/horse $(DEPS_DIR)/horse > cd $(DEPS_DIR)/horse ; git checkout -q master > $(MAKE) -C $(DEPS_DIR)/horse > >perfs: ERLC_OPTS += -DPERF=1 +'{parse_transform, horse_autoexport}' >perfs: clean deps deps/horse app > $(gen_verbose) erl -noshell -pa ebin deps/horse/ebin \ > -eval 'horse:app_perf($(PROJECT)), init:stop().' > >If you use a more recent erlang.mk it's much simpler, see >https://github.com/ninenines/cowlib/blob/master/Makefile for example. > >On 04/09/2015 12:46 PM, Fisher wrote: >> >> Here is my test: >> >> makefile: >> PROJECT = test >> include erlang.mk >> >> erlang.mk >> include deps/horse/erlang.mk >> >> .PHONY: all start clean >> >> INCLUDE = "include" >> >> EBIN = "ebin" >> >> HORSE = "deps/horse/ebin" >> >> DEPS_DIR = "deps" >> >> MODULE = "test02.erl" >> >> all: >> erlc -o ${EBIN} -I ${INCLUDE} \ >> src/*.erl >> >> start: >> erl -pa ${EBIN} ${HORSE} \ >> -eval 'horse:app_perf(horse), init:stop().' >> >> deps/horse: >> git clone -n -- https://github.com/extend/horse $(DEPS_DIR)/horse >> cd $(DEPS_DIR)/horse; git checkout -q master >> $(MAKE) -C $(DEPS_DIR)/horse >> >> perfs: ERLC_OPTS += -DPERF=1 +'{parse_transform, horse_autoexport}' >> %perfs: clean deps deps/horse app >> perfs: deps deps/horse app >> $(gen_verbose) erl -noshell -pa ebin deps/horse/ebin \ >> -eval 'horse:app_perf($(PROJECT)), init:stop().' >> >> >> #clean: >> # rm -rf ebin/*.beam >> >> and my test: >> -module(test01). >> -compile({parse_transform, horse_autoexport}). >> -export([test_fun/0, >> horse_do_nothing/0, >> horse_rfc2019/0 >> ]). >> >> -include("test.hrl"). >> >> test_fun() -> >> io:format("this is Test01/test01~n"), ok. >> >> >> horse_do_nothing() -> >> io:format("test01 running horse test~n"), >> ok. >> >> horse_rfc2019() -> >> horse:repeat(100000, test_fun()). >> >> >> >> >> >> At 2015-04-09 17:33:29, "Lo?c Hoguin" > wrote: >>>Show me the Makefile. >>> >>>On 04/09/2015 12:28 PM, Fisher wrote: >>>> Thanks for your answer! >>>> and i already added an include to my makefile like this: >>>> include deps/horse/erlang.mk !!! >>>> and the error still here -.- >>>> >>>> >>>> >>>> >>>> >>>> >>>> At 2015-04-09 17:24:02, "Lo?c Hoguin" > wrote: >>>>>Hey, >>>>> >>>>>These instructions apply only if you use erlang.mk. For rebar I think >>>>>you will need to add the project as a dependency and then the option to >>>>>use the parse_transform, though I am not sure on the details. >>>>> >>>>>Cheers, >>>>> >>>>>On 04/09/2015 10:28 AM, Fisher wrote: >>>>>> Hi, all! >>>>>> I am trying horse:https://github.com/extend/horse >>>>>> I got it in this folder: MyProject -> deps -> horse >>>>>> I added this in my makefile like the website says: >>>>>> >>>>>> deps/horse: >>>>>> git clone -n -- https://github.com/extend/horse$(DEPS_DIR)/horse >>>>>> cd $(DEPS_DIR)/horse ; git checkout -q master >>>>>> $(MAKE) -C $(DEPS_DIR)/horse >>>>>> >>>>>> perfs: ERLC_OPTS += -DPERF=1 +'{parse_transform, horse_autoexport}' >>>>>> perfs: clean deps deps/horse app >>>>>> $(gen_verbose) erl -noshell -pa ebin deps/horse/ebin \ >>>>>> -eval 'horse:app_perf($(PROJECT)), init:stop().' >>>>>> >>>>>> then i compile: make perfs >>>>>> and i got this error: >>>>>> >>>>>> src/test01.erl: undefined parse transform 'horse_autoexport' >>>>>> >>>>>> i tried add {erl_opts, [{parse_transform, horse_autoexport}]} to my app file >>>>>> or i add -compile([{parse_transform, horse_autoexport}]) to test01.erl >>>>>> module >>>>>> but the error continue. >>>>>> >>>>>> anyone can help? thanks! >>>>>> >>>>>> _______________________________________________ >>>>>> erlang-questions mailing list >>>>>>erlang-questions@REDACTED >>>>>> http://erlang.org/mailman/listinfo/erlang-questions >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> erlang-questions mailing list >>>>>>erlang-questions@REDACTED >>>>>> http://erlang.org/mailman/listinfo/erlang-questions >>>>>> >>>>> >>>>>-- >>>>>Lo?c Hoguin >>>>>http://ninenines.eu >>>> >>> >>>-- >>>Lo?c Hoguin >>>http://ninenines.eu >> > >-- >Lo?c Hoguin >http://ninenines.eu -------------- next part -------------- An HTML attachment was scrubbed... URL: From garry@REDACTED Thu Apr 9 16:22:57 2015 From: garry@REDACTED (Garry Hodgson) Date: Thu, 09 Apr 2015 10:22:57 -0400 Subject: [erlang-questions] segfaults in cowboy rest server Message-ID: <55268B41.2020207@research.att.com> i've got a problem i'm trying to solve wrt controlling memory consumption in a cloud environment. i've got a server that receives log data from security appliances and stores in a mariadb database. logs are sent to us via RESTful api calls, with batches of logs embedded in the json body of a POST call. they can get rather large, and we get a lot of them, at a high rate. when production load increased beyond what was anticipated (doesn't it always?) we began having failures, with the server disappearing without a trace. in some cases oom-killer killed it, in others it would fail trying to allocate memory. we only saw the latter by running in erlang shell and waiting until it died, then we saw a terse error message. to prevent this, i added a check in service_available() to see if erlang:memory( total ) + content-length > some threshold, and reject the request if so. also, having read the recent threads about garbage collecting binaries, i added a timer to check every 30 seconds that forces gc on all processes if memory usage is too high. this seems to work pretty well, except that after a few days of running, we get hard crashes, with segfaults showing up in /var/log/messages: kernel: beam[18098]: segfault at 7f09a004040c ip 000000000049e209 sp 00007fff860d32b0 error 4 in beam[400000+2ce000] kernel: beam[14177]: segfault at 7fce288829bc ip 000000000049e209 sp 00007fffa0d2d7a0 error 4 in beam[400000+2ce000] i've been using erlang for 15 years, and have never seen a segfault. we've recently updated from r15b02 to r17.4, and we've also switched from webmachine to cowboy. i don't know if either of those things are relevant. i'm kind of at a loss as to how to diagnose or deal with this. any advice would be greatly appreciated. -- Garry Hodgson Lead Member of Technical Staff AT&T Chief Security Office (CSO) "This e-mail and any files transmitted with it are AT&T property, are confidential, and are intended solely for the use of the individual or entity to whom this e-mail is addressed. If you are not one of the named recipient(s) or otherwise have reason to believe that you have received this message in error, please notify the sender and delete this message immediately from your computer. Any other use, retention, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited." From ali.sabil@REDACTED Thu Apr 9 17:21:19 2015 From: ali.sabil@REDACTED (Ali Sabil) Date: Thu, 09 Apr 2015 15:21:19 +0000 Subject: [erlang-questions] Why does Dialyzer crash on this map? In-Reply-To: References: <54749C90.6000602@llaisdy.com> <5474BA4C.4030706@llaisdy.com> <547716F8.7010208@llaisdy.com> <4B9AEB77-A709-4D97-8EF2-8FE1AF5A4071@llaisdy.com> <55085881.3060508@erlang.org> <55095D8E.2050103@erlang.org> Message-ID: Sorry for the late reply, I finally managed to get a minimal test case that reproduces the bug in 17.5: -module(sum). -export([ test/1 ]). -spec test(#{atom() => term()}) -> integer(). test(Data) -> maps:fold(fun (_Key, Value, Acc) when is_integer(Value) -> Acc + Value; (_Key, _Value, Acc) -> Acc end, 0, Data). I don't know if this is the correct fix, but this makes dialyzer work again: diff --git a/lib/hipe/cerl/erl_types.erl b/lib/hipe/cerl/erl_types.erl index 4215448..bb4c1c1 100644 --- a/lib/hipe/cerl/erl_types.erl +++ b/lib/hipe/cerl/erl_types.erl @@ -4594,6 +4594,8 @@ t_form_to_string({type, _L, list, [Type]}) -> "[" ++ t_form_to_string(Type) ++ "]"; t_form_to_string({type, _L, map, Args}) when not is_list(Args) -> "#{}"; +t_form_to_string({type, _L, map_field_assoc, Key, Value}) -> + "#{" ++ t_form_to_string(Key) ++ "=>" ++ t_form_to_string(Value) ++ "}"; t_form_to_string({type, _L, mfa, []}) -> "mfa()"; t_form_to_string({type, _L, module, []}) -> "module()"; t_form_to_string({type, _L, node, []}) -> "node()"; Thanks, Ali On Tue, Apr 7, 2015 at 2:31 PM Bj?rn-Egil Dahlberg < wallentin.dahlberg@REDACTED> wrote: > Again - could you provide me with a sample code .. or at least some sort > of a clue to what you are dialyzing? > > 2015-04-07 14:05 GMT+02:00 Ali Sabil : > >> Hi again, >> >> Running dialyzer shipped with 17.5 on the same code base leads now to the >> following error (17.4 works without any errors): >> >> ===> Error in dialyzing apps: Analysis failed with error: >> {function_clause,[{erl_types,t_form_to_string, >> [{type,36,map_field_assoc, >> {type,36,atom,[]}, >> {type,36,term,[]}}], >> [{file,"erl_types.erl"},{line,4546}]}, >> {erl_types,t_form_to_string_list,2, >> [{file,"erl_types.erl"},{line,4637}]}, >> {erl_types,t_form_to_string,1, >> [{file,"erl_types.erl"},{line,4634}]}, >> {erl_types,t_form_to_string_list,2, >> [{file,"erl_types.erl"},{line,4637}]}, >> {erl_types,t_form_to_string,1, >> [{file,"erl_types.erl"},{line,4634}]}, >> {dialyzer_contracts,contract_to_string_1,1, >> [{file,"dialyzer_contracts.erl"}, >> {line,107}]}, >> {dialyzer_contracts,extra_contract_warning,6, >> [{file,"dialyzer_contracts.erl"}, >> {line,712}]}, >> {dialyzer_contracts,picky_contract_check,7, >> [{file,"dialyzer_contracts.erl"}, >> {line,686}]}]} >> Last messages in the log cache: >> Reading files and computing callgraph... done in 1.21 secs >> Removing edges... done in 0.04 secs >> >> >> On Wed, Mar 18, 2015 at 12:12 PM Bj?rn-Egil Dahlberg >> wrote: >> >>> On 2015-03-18 12:01, Ali Sabil wrote: >>> > I tried to create a minimal testcase but I unfortunately haven't been >>> > able to. I was running dialyzer on a quite large code base and now >>> > even the unpatched dialyzer works without any issue after I fixed all >>> > the issues reported by dialyzer. >>> >>> Ah, well .. I suspect it was the missing clause in find_terminals and I >>> had it on a TODO somewhere. Should be included to 17.5. >>> >>> // Bj?rn-Egil >>> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From community-manager@REDACTED Thu Apr 9 17:41:04 2015 From: community-manager@REDACTED (Bruce Yinhe) Date: Thu, 9 Apr 2015 17:41:04 +0200 Subject: [erlang-questions] New Erlang job openings Message-ID: Hi, 21 US-based and 13 Europe-based Erlang job openings are available on Erlang Central. https://erlangcentral.org/jobs To publish new Erlang jobs, free of charge: https://erlangcentral.org/jobs/add To subscribe for weekly Erlang job updates https://erlangcentral.org/login/?action=register *Americas:* [Expiring soon] Erlang Developer at Inaka - Buenos Aires, Argentina https://erlangcentral.org/erlang-developer-inaka/ Erlang Developer at Sqor Sports - San Francisco, CA https://erlangcentral.org/erlang-developer-sqor/ Software Developer ? Home Based at Basho Technologies - Home Based https://erlangcentral.org/software-developer-home-based-basho/ [Expiring soon] Sr. Software Engineer at SynapSense - Folsom, CA https://erlangcentral.org/sr-software-engineer-synapsense/ [Expiring soon] IoT Infrastructure/DevOps Engineer at iDevices - Cupertino, CA https://erlangcentral.org/iot-infrastructuredevops-engineer-idevices/ [Expiring soon] Elixir / Erlang Developer at Vitamin Talent - Austin, TX (open to remote) https://erlangcentral.org/elixir-erlang-developer-vitamin-talent/ [Expiring soon] Senior Erlang Engineer at Machine Zone - Palo Alto, CA https://erlangcentral.org/senior-erlang-engineer-machine-zone/ [Expiring soon] Erlang Developer at INT Technologies - San Antonio, TX https://erlangcentral.org/erlang-int-technologies/ Lead Functional Programmer at Visa Inc. - Foster City, CA https://erlangcentral.org/visa-lead-functional-programmer/ Sr. Staff SW Engineer at Visa Inc. - Foster City, CA https://erlangcentral.org/sr-staff-sw-engineer-visa/ Senior Back End Engineer (Erlang) at Hyperledger - San Francisco, CA https://erlangcentral.org/senior-back-end-engineer-erlang-hyperledger/ Senior Software Engineer ? Distributed Systems at Couchbase - Mountain View, CA https://erlangcentral.org/senior-software-engineer-distributed-systems-couchbase/ Software Engineer ? Tools and Packaging Development at Couchbase - Mountain View, CA https://erlangcentral.org/software-engineer-tools-and-packaging-development-couchbase/ Erlang Developer at TigerText - Mountain View, CA *http://erlangcentral.org/erlang-developer-tigertext/ * Backend Engineer at issuu - Palo Alto, CA https://erlangcentral.org/backend-engineer-issuu/ Senior Staff & Lead Erlang Engineers at Raretec Consulting - California, CA https://erlangcentral.org/senior-staff-lead-erlang -engineers-raretec-consulting/ Senior Erlang Software Developer at IBM Cloudant - Remote https://erlangcentral.org/senior-erlang-software-developer-ibm-cloudant/ Software Engineer at Riot Games - Saint Louis, MO. Santa Monica, CA http://erlangcentral.org/software-engineer-riot-games/ Infrastructure Engineer at Getaround - San Francisco, CA http://erlangcentral.org/infrastructure-engineer-getaround/ *Europe:* [Expiring soon] Software Engineer at Ubiquiti Networks - Krakow, Poland https://erlangcentral.org/software-engineer-ubiquiti-networks/ Erlang Developer at Klarna - Stockholm, Sweden https://erlangcentral.org/erlang-developer-klarna/ Erlang Developer (Junior/Senior) at Erlang Solutions - London, Budapest, Krakow, Stockholm https://erlangcentral.org/erlang-developer-erlang-solutions/ RabbitMQ Implementation and Development Engineer at Erlang Solutions - London, Budapest, Krakow, Stockholm https://erlangcentral.org/rabbitmq-implementation-and-development-engineer/ [Expiring soon] Senior Engineer with DevOps Focus at Campanja - Stockholm, Sweden https://erlangcentral.org/senior-engineer-with-devops-focus-campanja/ [Expiring soon] Erlang developer at IoP - anywhere https://erlangcentral.org/erlang -developer-ambitious-startup-internet-of-people/ [Expiring soon] Experienced Erlang developer for an ambitious Startup - Remote work possible (Europe) https://erlangcentral.org/experienced-erlang-developer-for-an-ambitious-startup/ Senior Back-end Engineer at Spatch - London, UK https://erlangcentral.org/senior-back-end-engineer-spatch/ Senior DevOps Engineer at Spatch - London, UK https://erlangcentral.org/senior-devops-engineer-spatch/ Erlang Software Developers at AccionLabs - London, UK https://erlangcentral.org/erlang-software-developers-accionlabs/ Backend Engineer at Centralway - Zurich, Switzerland https://erlangcentral.org/backend-engineer-centralway/ Back-end Developer at NGTI - Rotterdam, The Netherlands https://erlangcentral.org/back-end-developer-ngti-nl/ *Asia & Oceania * System Analyst (Erlang software engineer) at Chaatz Limited - Wan Chai,Hong Kong https://erlangcentral.org/system-analyst-erlang-software-engineer-chaatz/ Best regards, Bruce Yinhe ErlangCentral.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From lloyd@REDACTED Thu Apr 9 18:55:55 2015 From: lloyd@REDACTED (lloyd@REDACTED) Date: Thu, 9 Apr 2015 12:55:55 -0400 (EDT) Subject: [erlang-questions] Amazon API -- Lookup by ISBN In-Reply-To: References: <1428526022.03741188@apps.rackspace.com> Message-ID: <1428598555.718513473@apps.rackspace.com> Thanks all. Darach--- looks like the Library of Congress API fits the bill. Garrett--- for sake of self-enlightenment I'll take a look at Python implementations. I can understand that Amazon needs to enforce security on their api, but one would think that the inventor of one-click ordering could come up with a simpler api request implementation. Maybe they're just trying to keep pesky users like me out of the goodies. Wizards only apply. Best wishes, Lloyd -----Original Message----- From: "Darach Ennis" Sent: Wednesday, April 8, 2015 8:07pm To: "Lloyd Prentice" Cc: "Erlang Questions" Subject: Re: [erlang-questions] Amazon API -- Lookup by ISBN If you don't need the rank information you can get the metadata through composing a URL against the library of congress database: http://lx2.loc.gov:210/lcdb?version=1.1&operation=searchRetrieve&query=bath.isbn=0448421658&maximumRecords=1&recordSchema=mods This will deliver an XML document. You can get similar information from Google's API: http://www.google.com/books/feeds/volumes/?q=ISBN%3C0448421658%3E Both are simple HTTP GET requests so unless you absolutely need data unique to Amazon's DB perhaps that would suffice. Cheers, Darach. On Wed, Apr 8, 2015 at 9:47 PM, wrote: > Hello, > > I'm striving to look up books in Amazon's db by ISBN. At first blush it > looks easy enough: > > > http://docs.aws.amazon.com/AWSECommerceService/latest/DG/EX_LookupbyISBN.html > > But the last item, Signature, baffles me. Procedure here: > > > http://docs.aws.amazon.com/AWSECommerceService/latest/DG/rest-signature.html > > I'm fine with this until I hit step 4: > > -- Sort parameter/value pairs by byte value --- I can see how to do this > manually, but don't know how put Erlang to the task > > And I'm really stumped when I hit step 8: > > -- Calculate an RFC 2104-compliant HMAC with the SHA256 hash algorithm > > Any help? Better yet, does anyone have actual code to make such requests > they're willing to share? > > NOTE: Dave Thomas solved this problem way back in 2007. But looks like > Amazon has changed their request format: > > http://pragdave.me/blog/2007/04/15/a-first-erlang-program/ > > Many thanks, > > LRP > > > > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From bob@REDACTED Thu Apr 9 19:44:02 2015 From: bob@REDACTED (Bob Ippolito) Date: Thu, 9 Apr 2015 10:44:02 -0700 Subject: [erlang-questions] Amazon API -- Lookup by ISBN In-Reply-To: <1428598555.718513473@apps.rackspace.com> References: <1428526022.03741188@apps.rackspace.com> <1428598555.718513473@apps.rackspace.com> Message-ID: I'm sure they're just trying to keep tabs on who uses the API so that they can implement rate limiting and such to prevent third parties from scraping the entire database or adversely affecting performance for everyone else. This actually looks like one of the simpler APIs to implement, since it's just in the query string, and the signature is using standard algorithms. I'm sure you could get it if you put some effort into just building it rather than trying to find an implementation that someone else has already built. On Thu, Apr 9, 2015 at 9:55 AM, wrote: > Thanks all. > > Darach--- looks like the Library of Congress API fits the bill. > > Garrett--- for sake of self-enlightenment I'll take a look at Python > implementations. > > I can understand that Amazon needs to enforce security on their api, but > one would think that the inventor of one-click ordering could come up with > a simpler api request implementation. Maybe they're just trying to keep > pesky users like me out of the goodies. Wizards only apply. > > Best wishes, > > Lloyd > > > > -----Original Message----- > From: "Darach Ennis" > Sent: Wednesday, April 8, 2015 8:07pm > To: "Lloyd Prentice" > Cc: "Erlang Questions" > Subject: Re: [erlang-questions] Amazon API -- Lookup by ISBN > > If you don't need the rank information you can get the metadata through > composing a URL > against the library of congress database: > > > http://lx2.loc.gov:210/lcdb?version=1.1&operation=searchRetrieve&query=bath.isbn=0448421658&maximumRecords=1&recordSchema=mods > > This will deliver an XML document. You can get similar information from > Google's API: > > http://www.google.com/books/feeds/volumes/?q=ISBN%3C0448421658%3E > > Both are simple HTTP GET requests so unless you absolutely need data unique > to Amazon's DB perhaps that would suffice. > > Cheers, > > Darach. > > On Wed, Apr 8, 2015 at 9:47 PM, wrote: > > > Hello, > > > > I'm striving to look up books in Amazon's db by ISBN. At first blush it > > looks easy enough: > > > > > > > http://docs.aws.amazon.com/AWSECommerceService/latest/DG/EX_LookupbyISBN.html > > > > But the last item, Signature, baffles me. Procedure here: > > > > > > > http://docs.aws.amazon.com/AWSECommerceService/latest/DG/rest-signature.html > > > > I'm fine with this until I hit step 4: > > > > -- Sort parameter/value pairs by byte value --- I can see how to do this > > manually, but don't know how put Erlang to the task > > > > And I'm really stumped when I hit step 8: > > > > -- Calculate an RFC 2104-compliant HMAC with the SHA256 hash algorithm > > > > Any help? Better yet, does anyone have actual code to make such requests > > they're willing to share? > > > > NOTE: Dave Thomas solved this problem way back in 2007. But looks like > > Amazon has changed their request format: > > > > http://pragdave.me/blog/2007/04/15/a-first-erlang-program/ > > > > Many thanks, > > > > LRP > > > > > > > > > > > > > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abejideayodele@REDACTED Thu Apr 9 20:20:31 2015 From: abejideayodele@REDACTED (ayodele abejide) Date: Thu, 9 Apr 2015 13:20:31 -0500 Subject: [erlang-questions] Amazon API -- Lookup by ISBN In-Reply-To: References: <1428526022.03741188@apps.rackspace.com> <1428598555.718513473@apps.rackspace.com> Message-ID: I played around with implementing this: https://gist.github.com/bjhaid/5d3c58aca0dbee0d96fd PS: It's not tested outside of erl, and was a lunch time hacking. BR, Ayo On Thu, Apr 9, 2015 at 12:44 PM, Bob Ippolito wrote: > I'm sure they're just trying to keep tabs on who uses the API so that they > can implement rate limiting and such to prevent third parties from scraping > the entire database or adversely affecting performance for everyone else. > This actually looks like one of the simpler APIs to implement, since it's > just in the query string, and the signature is using standard algorithms. > I'm sure you could get it if you put some effort into just building it > rather than trying to find an implementation that someone else has already > built. > > On Thu, Apr 9, 2015 at 9:55 AM, wrote: > >> Thanks all. >> >> Darach--- looks like the Library of Congress API fits the bill. >> >> Garrett--- for sake of self-enlightenment I'll take a look at Python >> implementations. >> >> I can understand that Amazon needs to enforce security on their api, but >> one would think that the inventor of one-click ordering could come up with >> a simpler api request implementation. Maybe they're just trying to keep >> pesky users like me out of the goodies. Wizards only apply. >> >> Best wishes, >> >> Lloyd >> >> >> >> -----Original Message----- >> From: "Darach Ennis" >> Sent: Wednesday, April 8, 2015 8:07pm >> To: "Lloyd Prentice" >> Cc: "Erlang Questions" >> Subject: Re: [erlang-questions] Amazon API -- Lookup by ISBN >> >> If you don't need the rank information you can get the metadata through >> composing a URL >> against the library of congress database: >> >> >> http://lx2.loc.gov:210/lcdb?version=1.1&operation=searchRetrieve&query=bath.isbn=0448421658&maximumRecords=1&recordSchema=mods >> >> This will deliver an XML document. You can get similar information from >> Google's API: >> >> http://www.google.com/books/feeds/volumes/?q=ISBN%3C0448421658%3E >> >> Both are simple HTTP GET requests so unless you absolutely need data >> unique >> to Amazon's DB perhaps that would suffice. >> >> Cheers, >> >> Darach. >> >> On Wed, Apr 8, 2015 at 9:47 PM, wrote: >> >> > Hello, >> > >> > I'm striving to look up books in Amazon's db by ISBN. At first blush it >> > looks easy enough: >> > >> > >> > >> http://docs.aws.amazon.com/AWSECommerceService/latest/DG/EX_LookupbyISBN.html >> > >> > But the last item, Signature, baffles me. Procedure here: >> > >> > >> > >> http://docs.aws.amazon.com/AWSECommerceService/latest/DG/rest-signature.html >> > >> > I'm fine with this until I hit step 4: >> > >> > -- Sort parameter/value pairs by byte value --- I can see how to do this >> > manually, but don't know how put Erlang to the task >> > >> > And I'm really stumped when I hit step 8: >> > >> > -- Calculate an RFC 2104-compliant HMAC with the SHA256 hash algorithm >> > >> > Any help? Better yet, does anyone have actual code to make such requests >> > they're willing to share? >> > >> > NOTE: Dave Thomas solved this problem way back in 2007. But looks like >> > Amazon has changed their request format: >> > >> > http://pragdave.me/blog/2007/04/15/a-first-erlang-program/ >> > >> > Many thanks, >> > >> > LRP >> > >> > >> > >> > >> > >> > >> > >> > >> > _______________________________________________ >> > erlang-questions mailing list >> > erlang-questions@REDACTED >> > http://erlang.org/mailman/listinfo/erlang-questions >> > >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lloyd@REDACTED Thu Apr 9 21:22:22 2015 From: lloyd@REDACTED (lloyd@REDACTED) Date: Thu, 9 Apr 2015 15:22:22 -0400 (EDT) Subject: [erlang-questions] Amazon API -- Lookup by ISBN In-Reply-To: References: <1428526022.03741188@apps.rackspace.com> <1428598555.718513473@apps.rackspace.com> Message-ID: <1428607342.62345821@apps.rackspace.com> Hi Ayo, Looks cool. I'll give it spin. Thanks. Meanwhile, taking Darach Ennis suggestion to use the Library of Commerce db, I came up with this: https://gist.github.com/anonymous/dd9846ef1cb2826f59da Bob Ippolito, I did put effort into trying to build it-- put half a day into it and got 98% of the way there before I got stuck. Why stuck? I simply could not understand the rather cryptic crypto documentation sufficiently to "Calculate an RFC 2104-compliant HMAC with the SHA256 hash algorithm". This could speak to my mental shortcomings, lack of experience, or heavens, could it be that the documentation could use elaboration and better examples? I appreciate your encouragement to work it out and I try. But code is just that, code, unless you have the keys to the kingdom. Nevertheless, I appreciate your response. Best to all, Lloyd -----Original Message----- From: "ayodele abejide" Sent: Thursday, April 9, 2015 2:20pm To: "Bob Ippolito" Cc: "Lloyd Prentice" , "Erlang Questions" Subject: Re: [erlang-questions] Amazon API -- Lookup by ISBN I played around with implementing this: https://gist.github.com/bjhaid/5d3c58aca0dbee0d96fd PS: It's not tested outside of erl, and was a lunch time hacking. BR, Ayo On Thu, Apr 9, 2015 at 12:44 PM, Bob Ippolito wrote: > I'm sure they're just trying to keep tabs on who uses the API so that they > can implement rate limiting and such to prevent third parties from scraping > the entire database or adversely affecting performance for everyone else. > This actually looks like one of the simpler APIs to implement, since it's > just in the query string, and the signature is using standard algorithms. > I'm sure you could get it if you put some effort into just building it > rather than trying to find an implementation that someone else has already > built. > > On Thu, Apr 9, 2015 at 9:55 AM, wrote: > >> Thanks all. >> >> Darach--- looks like the Library of Congress API fits the bill. >> >> Garrett--- for sake of self-enlightenment I'll take a look at Python >> implementations. >> >> I can understand that Amazon needs to enforce security on their api, but >> one would think that the inventor of one-click ordering could come up with >> a simpler api request implementation. Maybe they're just trying to keep >> pesky users like me out of the goodies. Wizards only apply. >> >> Best wishes, >> >> Lloyd >> >> >> >> -----Original Message----- >> From: "Darach Ennis" >> Sent: Wednesday, April 8, 2015 8:07pm >> To: "Lloyd Prentice" >> Cc: "Erlang Questions" >> Subject: Re: [erlang-questions] Amazon API -- Lookup by ISBN >> >> If you don't need the rank information you can get the metadata through >> composing a URL >> against the library of congress database: >> >> >> http://lx2.loc.gov:210/lcdb?version=1.1&operation=searchRetrieve&query=bath.isbn=0448421658&maximumRecords=1&recordSchema=mods >> >> This will deliver an XML document. You can get similar information from >> Google's API: >> >> http://www.google.com/books/feeds/volumes/?q=ISBN%3C0448421658%3E >> >> Both are simple HTTP GET requests so unless you absolutely need data >> unique >> to Amazon's DB perhaps that would suffice. >> >> Cheers, >> >> Darach. >> >> On Wed, Apr 8, 2015 at 9:47 PM, wrote: >> >> > Hello, >> > >> > I'm striving to look up books in Amazon's db by ISBN. At first blush it >> > looks easy enough: >> > >> > >> > >> http://docs.aws.amazon.com/AWSECommerceService/latest/DG/EX_LookupbyISBN.html >> > >> > But the last item, Signature, baffles me. Procedure here: >> > >> > >> > >> http://docs.aws.amazon.com/AWSECommerceService/latest/DG/rest-signature.html >> > >> > I'm fine with this until I hit step 4: >> > >> > -- Sort parameter/value pairs by byte value --- I can see how to do this >> > manually, but don't know how put Erlang to the task >> > >> > And I'm really stumped when I hit step 8: >> > >> > -- Calculate an RFC 2104-compliant HMAC with the SHA256 hash algorithm >> > >> > Any help? Better yet, does anyone have actual code to make such requests >> > they're willing to share? >> > >> > NOTE: Dave Thomas solved this problem way back in 2007. But looks like >> > Amazon has changed their request format: >> > >> > http://pragdave.me/blog/2007/04/15/a-first-erlang-program/ >> > >> > Many thanks, >> > >> > LRP >> > >> > >> > >> > >> > >> > >> > >> > >> > _______________________________________________ >> > erlang-questions mailing list >> > erlang-questions@REDACTED >> > http://erlang.org/mailman/listinfo/erlang-questions >> > >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > From bob@REDACTED Thu Apr 9 21:33:25 2015 From: bob@REDACTED (Bob Ippolito) Date: Thu, 9 Apr 2015 12:33:25 -0700 Subject: [erlang-questions] Amazon API -- Lookup by ISBN In-Reply-To: <1428607342.62345821@apps.rackspace.com> References: <1428526022.03741188@apps.rackspace.com> <1428598555.718513473@apps.rackspace.com> <1428607342.62345821@apps.rackspace.com> Message-ID: They give you examples in the documentation, rather than worrying about what that RFC means you can just try crypto:hmac/3 and see if it works (hint: it does). 1> base64:encode(crypto:hmac(sha256, <<"1234567890">>, <<"GET\ nwebservices.amazon.com \n/onca/xml\nAWSAccessKeyId=AKIAIOSFODNN7EXAMPLE&AssociateTag=mytag-20&ItemId=0679722769&Operation=ItemLookup&ResponseGroup=Images%2CItemAttributes%2COffers%2CReviews&Service=AWSECommerceService&Timestamp=2014-08-18T12%3A00%3A00Z&Version=2013-08-01">>)). <<"j7bZM0LXZ9eXeZruTqWm2DIvDYVUU3wxPPpp+iXxzQc=">> This matches their example: j7bZM0LXZ9eXeZruTqWm2DIvDYVUU3wxPPpp+iXxzQc= -bob On Thu, Apr 9, 2015 at 12:22 PM, wrote: > Hi Ayo, > > Looks cool. I'll give it spin. Thanks. > > Meanwhile, taking Darach Ennis suggestion to use the Library of Commerce > db, I came up with this: > > https://gist.github.com/anonymous/dd9846ef1cb2826f59da > > Bob Ippolito, I did put effort into trying to build it-- put half a day > into it and got 98% of the way there before I got stuck. Why stuck? I > simply could not understand the rather cryptic crypto documentation > sufficiently to "Calculate an RFC 2104-compliant HMAC with the SHA256 hash > algorithm". This could speak to my mental shortcomings, lack of experience, > or heavens, could it be that the documentation could use elaboration and > better examples? I appreciate your encouragement to work it out and I try. > But code is just that, code, unless you have the keys to the kingdom. > Nevertheless, I appreciate your response. > > Best to all, > > Lloyd > > -----Original Message----- > From: "ayodele abejide" > Sent: Thursday, April 9, 2015 2:20pm > To: "Bob Ippolito" > Cc: "Lloyd Prentice" , "Erlang Questions" < > erlang-questions@REDACTED> > Subject: Re: [erlang-questions] Amazon API -- Lookup by ISBN > > I played around with implementing this: > > https://gist.github.com/bjhaid/5d3c58aca0dbee0d96fd > > PS: It's not tested outside of erl, and was a lunch time hacking. > > BR, > > Ayo > > > On Thu, Apr 9, 2015 at 12:44 PM, Bob Ippolito wrote: > > > I'm sure they're just trying to keep tabs on who uses the API so that > they > > can implement rate limiting and such to prevent third parties from > scraping > > the entire database or adversely affecting performance for everyone else. > > This actually looks like one of the simpler APIs to implement, since it's > > just in the query string, and the signature is using standard algorithms. > > I'm sure you could get it if you put some effort into just building it > > rather than trying to find an implementation that someone else has > already > > built. > > > > On Thu, Apr 9, 2015 at 9:55 AM, wrote: > > > >> Thanks all. > >> > >> Darach--- looks like the Library of Congress API fits the bill. > >> > >> Garrett--- for sake of self-enlightenment I'll take a look at Python > >> implementations. > >> > >> I can understand that Amazon needs to enforce security on their api, but > >> one would think that the inventor of one-click ordering could come up > with > >> a simpler api request implementation. Maybe they're just trying to keep > >> pesky users like me out of the goodies. Wizards only apply. > >> > >> Best wishes, > >> > >> Lloyd > >> > >> > >> > >> -----Original Message----- > >> From: "Darach Ennis" > >> Sent: Wednesday, April 8, 2015 8:07pm > >> To: "Lloyd Prentice" > >> Cc: "Erlang Questions" > >> Subject: Re: [erlang-questions] Amazon API -- Lookup by ISBN > >> > >> If you don't need the rank information you can get the metadata through > >> composing a URL > >> against the library of congress database: > >> > >> > >> > http://lx2.loc.gov:210/lcdb?version=1.1&operation=searchRetrieve&query=bath.isbn=0448421658&maximumRecords=1&recordSchema=mods > >> > >> This will deliver an XML document. You can get similar information from > >> Google's API: > >> > >> http://www.google.com/books/feeds/volumes/?q=ISBN%3C0448421658%3E > >> > >> Both are simple HTTP GET requests so unless you absolutely need data > >> unique > >> to Amazon's DB perhaps that would suffice. > >> > >> Cheers, > >> > >> Darach. > >> > >> On Wed, Apr 8, 2015 at 9:47 PM, wrote: > >> > >> > Hello, > >> > > >> > I'm striving to look up books in Amazon's db by ISBN. At first blush > it > >> > looks easy enough: > >> > > >> > > >> > > >> > http://docs.aws.amazon.com/AWSECommerceService/latest/DG/EX_LookupbyISBN.html > >> > > >> > But the last item, Signature, baffles me. Procedure here: > >> > > >> > > >> > > >> > http://docs.aws.amazon.com/AWSECommerceService/latest/DG/rest-signature.html > >> > > >> > I'm fine with this until I hit step 4: > >> > > >> > -- Sort parameter/value pairs by byte value --- I can see how to do > this > >> > manually, but don't know how put Erlang to the task > >> > > >> > And I'm really stumped when I hit step 8: > >> > > >> > -- Calculate an RFC 2104-compliant HMAC with the SHA256 hash algorithm > >> > > >> > Any help? Better yet, does anyone have actual code to make such > requests > >> > they're willing to share? > >> > > >> > NOTE: Dave Thomas solved this problem way back in 2007. But looks like > >> > Amazon has changed their request format: > >> > > >> > http://pragdave.me/blog/2007/04/15/a-first-erlang-program/ > >> > > >> > Many thanks, > >> > > >> > LRP > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > _______________________________________________ > >> > erlang-questions mailing list > >> > erlang-questions@REDACTED > >> > http://erlang.org/mailman/listinfo/erlang-questions > >> > > >> > >> > >> _______________________________________________ > >> erlang-questions mailing list > >> erlang-questions@REDACTED > >> http://erlang.org/mailman/listinfo/erlang-questions > >> > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bob@REDACTED Thu Apr 9 23:34:51 2015 From: bob@REDACTED (Bob Ippolito) Date: Thu, 9 Apr 2015 14:34:51 -0700 Subject: [erlang-questions] Amazon API -- Lookup by ISBN In-Reply-To: <1428609814.721110899@apps.rackspace.com> References: <1428526022.03741188@apps.rackspace.com> <1428598555.718513473@apps.rackspace.com> <1428607342.62345821@apps.rackspace.com> <1428609814.721110899@apps.rackspace.com> Message-ID: This answers everything you need to know about how to get a secret key, one click away from the page about signatures that you linked to: http://docs.aws.amazon.com/AWSECommerceService/latest/DG/AWSCredentials.html Erlang type specs are described here: http://erlang.org/doc/reference_manual/typespec.html When you see `sometype() = foo | bar | baz` then you know that when you see `sometype()` then you know that the only valid values are those three atoms, `foo`, `bar`, or `baz`. The two iodata parameters to hmac/3 are named Key and Data. Key is the secret key, and Data is the string to sign (what else could it be?). The other hmac functions you found are just the lower-level bits that you could use to implement your own version of hmac/3, the most common use case would be to stream data into it rather than to provide all of Data up-front. hmac/3 is the right choice because the documentation says it does exactly what you're trying to do: "Computes a HMAC". On Thu, Apr 9, 2015 at 1:03 PM, wrote: > Hi Bob, > > This helps. > > I still find the crypo docs confusing, however. With help from the list I > see that I need hmac/3, but when I first looked at crypto I saw six > functions referencing hmac which led to the question, which one do I need? > When I looked at hmac/3 the first two parameters were quite mysterious. I > see that Type is hash_algorithms(), go to top of page to find > hash_algorithms() and see SHA256 but it's not clear to me--- is that what I > plug into the function? Then I see that Key is iodata(). OK, I know what > iodata() is but it could be anything between square brackets... So, is the > SecretKey something I just make up out of thin air? > > I hope you can see the difficulty and frustration that confronts one > encountering these functions for the first time. I did look for tutorials > and examples, but found nothing that cleared away the fog. > > Tell you what, if you're willing to take the time to mentor me, I'll write > up a tutorial that may help the next noobie down the line. > > Thanks again, > > Lloyd > > > > hmac(Type, Key, Data) > -----Original Message----- > From: "Bob Ippolito" > Sent: Thursday, April 9, 2015 3:33pm > To: "Lloyd Prentice" > Cc: "ayodele abejide" , "Erlang Questions" < > erlang-questions@REDACTED> > Subject: Re: [erlang-questions] Amazon API -- Lookup by ISBN > > They give you examples in the documentation, rather than worrying about > what that RFC means you can just try crypto:hmac/3 and see if it works > (hint: it does). > > 1> base64:encode(crypto:hmac(sha256, <<"1234567890">>, <<"GET\ > nwebservices.amazon.com > > \n/onca/xml\nAWSAccessKeyId=AKIAIOSFODNN7EXAMPLE&AssociateTag=mytag-20&ItemId=0679722769&Operation=ItemLookup&ResponseGroup=Images%2CItemAttributes%2COffers%2CReviews&Service=AWSECommerceService&Timestamp=2014-08-18T12%3A00%3A00Z&Version=2013-08-01">>)). > <<"j7bZM0LXZ9eXeZruTqWm2DIvDYVUU3wxPPpp+iXxzQc=">> > > This matches their example: > j7bZM0LXZ9eXeZruTqWm2DIvDYVUU3wxPPpp+iXxzQc= > > -bob > > On Thu, Apr 9, 2015 at 12:22 PM, wrote: > > > Hi Ayo, > > > > Looks cool. I'll give it spin. Thanks. > > > > Meanwhile, taking Darach Ennis suggestion to use the Library of Commerce > > db, I came up with this: > > > > https://gist.github.com/anonymous/dd9846ef1cb2826f59da > > > > Bob Ippolito, I did put effort into trying to build it-- put half a day > > into it and got 98% of the way there before I got stuck. Why stuck? I > > simply could not understand the rather cryptic crypto documentation > > sufficiently to "Calculate an RFC 2104-compliant HMAC with the SHA256 > hash > > algorithm". This could speak to my mental shortcomings, lack of > experience, > > or heavens, could it be that the documentation could use elaboration and > > better examples? I appreciate your encouragement to work it out and I > try. > > But code is just that, code, unless you have the keys to the kingdom. > > Nevertheless, I appreciate your response. > > > > Best to all, > > > > Lloyd > > > > -----Original Message----- > > From: "ayodele abejide" > > Sent: Thursday, April 9, 2015 2:20pm > > To: "Bob Ippolito" > > Cc: "Lloyd Prentice" , "Erlang Questions" < > > erlang-questions@REDACTED> > > Subject: Re: [erlang-questions] Amazon API -- Lookup by ISBN > > > > I played around with implementing this: > > > > https://gist.github.com/bjhaid/5d3c58aca0dbee0d96fd > > > > PS: It's not tested outside of erl, and was a lunch time hacking. > > > > BR, > > > > Ayo > > > > > > On Thu, Apr 9, 2015 at 12:44 PM, Bob Ippolito wrote: > > > > > I'm sure they're just trying to keep tabs on who uses the API so that > > they > > > can implement rate limiting and such to prevent third parties from > > scraping > > > the entire database or adversely affecting performance for everyone > else. > > > This actually looks like one of the simpler APIs to implement, since > it's > > > just in the query string, and the signature is using standard > algorithms. > > > I'm sure you could get it if you put some effort into just building it > > > rather than trying to find an implementation that someone else has > > already > > > built. > > > > > > On Thu, Apr 9, 2015 at 9:55 AM, wrote: > > > > > >> Thanks all. > > >> > > >> Darach--- looks like the Library of Congress API fits the bill. > > >> > > >> Garrett--- for sake of self-enlightenment I'll take a look at Python > > >> implementations. > > >> > > >> I can understand that Amazon needs to enforce security on their api, > but > > >> one would think that the inventor of one-click ordering could come up > > with > > >> a simpler api request implementation. Maybe they're just trying to > keep > > >> pesky users like me out of the goodies. Wizards only apply. > > >> > > >> Best wishes, > > >> > > >> Lloyd > > >> > > >> > > >> > > >> -----Original Message----- > > >> From: "Darach Ennis" > > >> Sent: Wednesday, April 8, 2015 8:07pm > > >> To: "Lloyd Prentice" > > >> Cc: "Erlang Questions" > > >> Subject: Re: [erlang-questions] Amazon API -- Lookup by ISBN > > >> > > >> If you don't need the rank information you can get the metadata > through > > >> composing a URL > > >> against the library of congress database: > > >> > > >> > > >> > > > http://lx2.loc.gov:210/lcdb?version=1.1&operation=searchRetrieve&query=bath.isbn=0448421658&maximumRecords=1&recordSchema=mods > > >> > > >> This will deliver an XML document. You can get similar information > from > > >> Google's API: > > >> > > >> http://www.google.com/books/feeds/volumes/?q=ISBN%3C0448421658%3E > > >> > > >> Both are simple HTTP GET requests so unless you absolutely need data > > >> unique > > >> to Amazon's DB perhaps that would suffice. > > >> > > >> Cheers, > > >> > > >> Darach. > > >> > > >> On Wed, Apr 8, 2015 at 9:47 PM, wrote: > > >> > > >> > Hello, > > >> > > > >> > I'm striving to look up books in Amazon's db by ISBN. At first blush > > it > > >> > looks easy enough: > > >> > > > >> > > > >> > > > >> > > > http://docs.aws.amazon.com/AWSECommerceService/latest/DG/EX_LookupbyISBN.html > > >> > > > >> > But the last item, Signature, baffles me. Procedure here: > > >> > > > >> > > > >> > > > >> > > > http://docs.aws.amazon.com/AWSECommerceService/latest/DG/rest-signature.html > > >> > > > >> > I'm fine with this until I hit step 4: > > >> > > > >> > -- Sort parameter/value pairs by byte value --- I can see how to do > > this > > >> > manually, but don't know how put Erlang to the task > > >> > > > >> > And I'm really stumped when I hit step 8: > > >> > > > >> > -- Calculate an RFC 2104-compliant HMAC with the SHA256 hash > algorithm > > >> > > > >> > Any help? Better yet, does anyone have actual code to make such > > requests > > >> > they're willing to share? > > >> > > > >> > NOTE: Dave Thomas solved this problem way back in 2007. But looks > like > > >> > Amazon has changed their request format: > > >> > > > >> > http://pragdave.me/blog/2007/04/15/a-first-erlang-program/ > > >> > > > >> > Many thanks, > > >> > > > >> > LRP > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > _______________________________________________ > > >> > erlang-questions mailing list > > >> > erlang-questions@REDACTED > > >> > http://erlang.org/mailman/listinfo/erlang-questions > > >> > > > >> > > >> > > >> _______________________________________________ > > >> erlang-questions mailing list > > >> erlang-questions@REDACTED > > >> http://erlang.org/mailman/listinfo/erlang-questions > > >> > > > > > > > > > _______________________________________________ > > > erlang-questions mailing list > > > erlang-questions@REDACTED > > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From 44290016@REDACTED Fri Apr 10 03:30:52 2015 From: 44290016@REDACTED (=?gb18030?B?vMW8xQ==?=) Date: Fri, 10 Apr 2015 09:30:52 +0800 Subject: [erlang-questions] [erlang-question][cowboy] Why got 415 unsupported media type when using POST method Message-ID: Dear All, If I use GET method the REST API work properly, but when I change to POST, got 415 error My code: -export([init/2]). -export([allowed_methods/2, content_types_provided/2]). -export([to_json/2]). init(Req, Opts) -> {cowboy_rest, Req, Opts}. allowed_methods(Req, State) -> {[<<"POST">>], Req, State}. content_types_provided(Req, State) -> {[ {<<"application/json">>, to_json} ], Req, State}. to_json(Req, State) -> ..... when I test the REST API I: curl http://192.168.1.103:8080/cloud_init -v -XPOST -H"Content-Type: application/json" -d '{"client_id": "4342jltj3i"}' got : * Hostname was NOT found in DNS cache * Trying 192.168.1.103... * Connected to 192.168.1.103 (192.168.1.103) port 8080 (#0) > POST /cloudbox_init HTTP/1.1 > User-Agent: curl/7.37.1 > Host: 192.168.1.103:8080 > Accept: */* > content-type: application/json > Content-Length: 28 > * upload completely sent off: 28 out of 28 bytes < HTTP/1.1 415 Unsupported Media Type < connection: keep-alive * Server Cowboy is not blacklisted < server: Cowboy < date: Fri, 10 Apr 2015 00:56:15 GMT < content-length: 0 < content-type: application/json < * Connection #0 to host 192.168.1.103 left intact -------------- next part -------------- An HTML attachment was scrubbed... URL: From jpsiyu@REDACTED Fri Apr 10 05:56:13 2015 From: jpsiyu@REDACTED (Fisher) Date: Fri, 10 Apr 2015 11:56:13 +0800 (CST) Subject: [erlang-questions] application:load(App, modules) Message-ID: <43ceb23a.1f75e.14ca178877f.Coremail.jpsiyu@163.com> Hi, all ! Here is my app file {application, test, [ {description, "an test."}, {vsn, "0.1.0"}, {modules, []}, {registered, []}, {applications, [kerne, stdlib]} ]} note that: {modules, []} i compile it use two different ways, and the result(application:get_key/2) comes: ------------------------------------------------------------------------------------------- compile method application:get_key(test, modules) ------------------------------------------------------------------------------------------- erlang:mk {ok, [test01, test02, test03, ops, test01]} ------------------------------------------------------------------------------------------- Emakefile {ok, []} ------------------------------------------------------------------------------------------- and Emakefile is like this: {'src/*', [debug_info, {i, "src"}, {i, "include"}, {outdir, "ebin"}]}. which command in erlang:mk makes the result different? any one can help? _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From martink@REDACTED Fri Apr 10 06:13:45 2015 From: martink@REDACTED (Martin Karlsson) Date: Fri, 10 Apr 2015 16:13:45 +1200 Subject: [erlang-questions] application:load(App, modules) In-Reply-To: <43ceb23a.1f75e.14ca178877f.Coremail.jpsiyu@163.com> References: <43ceb23a.1f75e.14ca178877f.Coremail.jpsiyu@163.com> Message-ID: Hi Fisher, > which command in erlang:mk makes the result different? any one can help? erlang.mk finds all your modules (looking in the ebin directory) and "compiles" the *.app.src file including these modules. Depending on version of erlang.mk it is done in different targets. In the latest version in git it is done in the app-build target on L311 Cheers, Martin From jpsiyu@REDACTED Fri Apr 10 06:17:49 2015 From: jpsiyu@REDACTED (Fisher) Date: Fri, 10 Apr 2015 12:17:49 +0800 (CST) Subject: [erlang-questions] application:load(App, modules) In-Reply-To: References: <43ceb23a.1f75e.14ca178877f.Coremail.jpsiyu@163.com> Message-ID: Your information is helpfull, thank you ! At 2015-04-10 12:13:45, "Martin Karlsson" wrote: >Hi Fisher, >> which command in erlang:mk makes the result different? any one can help? > >erlang.mk finds all your modules (looking in the ebin directory) and >"compiles" the *.app.src file including these modules. > >Depending on version of erlang.mk it is done in different targets. In >the latest version in git it is done in the app-build target on L311 > >Cheers, >Martin -------------- next part -------------- An HTML attachment was scrubbed... URL: From sdl.web@REDACTED Fri Apr 10 06:52:21 2015 From: sdl.web@REDACTED (Leo Liu) Date: Fri, 10 Apr 2015 12:52:21 +0800 Subject: [erlang-questions] how does a DistEntry get its `creation' value? Message-ID: Hi there, I am looking at function erts_find_or_insert_dist_entry in erl_node_tables.c and there is no `creation' at insertion time. Any idea how a DistEntry gets its `creation' value? I also checked the distribution protocol and it seems a node can get its own creation when publish to epmd. So when node A connects to node B, it doesn't have node B's `creation'. How does the DistEntry for node B have a correct creation value? Thanks. Leo From bengt.kleberg@REDACTED Fri Apr 10 08:23:25 2015 From: bengt.kleberg@REDACTED (Bengt Kleberg) Date: Fri, 10 Apr 2015 08:23:25 +0200 Subject: [erlang-questions] how: Getting -export_type information at run time? Message-ID: <55276C5D.8000301@ericsson.com> Greetings, How do I find type information at run time? A very nice Erlang module takes an options list. The options are -type defined and -export_type'ed. How do another module get that information during run time? I thought that it might be in very_nice:module_info(attributes), but no luck. bengt From mjtruog@REDACTED Fri Apr 10 08:38:11 2015 From: mjtruog@REDACTED (Michael Truog) Date: Thu, 09 Apr 2015 23:38:11 -0700 Subject: [erlang-questions] how: Getting -export_type information at run time? In-Reply-To: <55276C5D.8000301@ericsson.com> References: <55276C5D.8000301@ericsson.com> Message-ID: <55276FD3.8040707@gmail.com> Use the parse transform at https://gist.github.com/okeuday/76724b20dca62c27420f and use the very_nice:$abstract_code$() function it adds to access the abstract code as data. On 04/09/2015 11:23 PM, Bengt Kleberg wrote: > Greetings, > > How do I find type information at run time? > > A very nice Erlang module takes an options list. The options are -type defined and -export_type'ed. > How do another module get that information during run time? > > I thought that it might be in very_nice:module_info(attributes), but no luck. > > > bengt > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From vladdu55@REDACTED Fri Apr 10 08:41:33 2015 From: vladdu55@REDACTED (Vlad Dumitrescu) Date: Fri, 10 Apr 2015 08:41:33 +0200 Subject: [erlang-questions] how: Getting -export_type information at run time? In-Reply-To: <55276C5D.8000301@ericsson.com> References: <55276C5D.8000301@ericsson.com> Message-ID: Hi Bengt, You have to extract this information from beam_lib:chunks(BEAM, [abstract_code]) by means of erl_syntax_lib:fold/3 to keep only the typespecs. It would be nice to have a higher-level library that gives access to these internals. regards, Vlad On Fri, Apr 10, 2015 at 8:23 AM, Bengt Kleberg wrote: > Greetings, > > How do I find type information at run time? > > A very nice Erlang module takes an options list. The options are -type > defined and -export_type'ed. > How do another module get that information during run time? > > I thought that it might be in very_nice:module_info(attributes), but no > luck. > > > bengt > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mjtruog@REDACTED Fri Apr 10 08:44:35 2015 From: mjtruog@REDACTED (Michael Truog) Date: Thu, 09 Apr 2015 23:44:35 -0700 Subject: [erlang-questions] segfaults in cowboy rest server In-Reply-To: <55268B41.2020207@research.att.com> References: <55268B41.2020207@research.att.com> Message-ID: <55277153.4000905@gmail.com> On 04/09/2015 07:22 AM, Garry Hodgson wrote: > i've got a problem i'm trying to solve wrt controlling memory > consumption in a cloud environment. i've got a server that > receives log data from security appliances and stores in a > mariadb database. logs are sent to us via RESTful api calls, > with batches of logs embedded in the json body of a POST > call. they can get rather large, and we get a lot of them, > at a high rate. > > when production load increased beyond what was anticipated > (doesn't it always?) we began having failures, with the server > disappearing without a trace. in some cases oom-killer killed > it, in others it would fail trying to allocate memory. we only > saw the latter by running in erlang shell and waiting until > it died, then we saw a terse error message. > > to prevent this, i added a check in service_available() to > see if erlang:memory( total ) + content-length > some threshold, > and reject the request if so. also, having read the recent threads > about garbage collecting binaries, i added a timer to check every > 30 seconds that forces gc on all processes if memory usage > is too high. > > this seems to work pretty well, except that after a few days > of running, we get hard crashes, with segfaults showing up > in /var/log/messages: > > kernel: beam[18098]: segfault at 7f09a004040c ip 000000000049e209 sp 00007fff860d32b0 error 4 in beam[400000+2ce000] > > kernel: beam[14177]: segfault at 7fce288829bc ip 000000000049e209 sp 00007fffa0d2d7a0 error 4 in beam[400000+2ce000] > > i've been using erlang for 15 years, and have never seen a segfault. > we've recently updated from r15b02 to r17.4, and we've also > switched from webmachine to cowboy. i don't know if either of > those things are relevant. i'm kind of at a loss as to how to diagnose > or deal with this. > > any advice would be greatly appreciated. > If you have any port drivers or NIFs that are used in the system, it would be best to examine them closely for errors hiding within. For example, a port driver that is written with C++ that throws an exception into the Erlang VM (for example, a timeout exception that has never before occurred) will crash the Erlang VM in new and exciting ways that can include seg faults. If you are using maps, that could also be a cause, since there are still bugs being worked on. From essen@REDACTED Fri Apr 10 09:28:56 2015 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Fri, 10 Apr 2015 10:28:56 +0300 Subject: [erlang-questions] [erlang-question][cowboy] Why got 415 unsupported media type when using POST method In-Reply-To: References: Message-ID: <55277BB8.5000309@ninenines.eu> Because you didn't define a content_types_accepted function and at least one function to accept the representation in the request. On 04/10/2015 04:30 AM, ?? wrote: > Dear All, > > If I use GET method the REST API work properly, but when I change to > POST, got 415 error > > My code: > > -export([init/2]). > -export([allowed_methods/2, content_types_provided/2]). > -export([to_json/2]). > init(Req, Opts) -> > {cowboy_rest, Req, Opts}. > > allowed_methods(Req, State) -> > {[<<"POST">>], Req, State}. > > content_types_provided(Req, State) -> > {[ > {<<"application/json">>, to_json} > ], Req, State}. > > to_json(Req, State) -> > ..... > > when I test the REST API I: > curl http://192.168.1.103:8080/cloud_init -v -XPOST -H"Content-Type: > application/json" -d '{"client_id": "4342jltj3i"}' > > got : > * Hostname was NOT found in DNS cache > * Trying 192.168.1.103... > * Connected to 192.168.1.103 (192.168.1.103) port 8080 (#0) > > POST /cloudbox_init HTTP/1.1 > > User-Agent: curl/7.37.1 > > Host: 192.168.1.103:8080 > > Accept: */* > > content-type: application/json > > Content-Length: 28 > > > * upload completely sent off: 28 out of 28 bytes > < HTTP/1.1 415 Unsupported Media Type > < connection: keep-alive > * Server Cowboy is not blacklisted > < server: Cowboy > < date: Fri, 10 Apr 2015 00:56:15 GMT > < content-length: 0 > < content-type: application/json > < > * Connection #0 to host 192.168.1.103 left intact > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin http://ninenines.eu From essen@REDACTED Fri Apr 10 10:00:06 2015 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Fri, 10 Apr 2015 11:00:06 +0300 Subject: [erlang-questions] [erlang-question][cowboy] Why got 415 unsupportedmedia type when using POST method In-Reply-To: References: <55277BB8.5000309@ninenines.eu> Message-ID: <55278306.5080001@ninenines.eu> "accepted" is for processing the resource representation in the request body (POST/PATCH/PUT). "provided" is for sending the resource representation in the response body (GET/HEAD, though HEAD will ultimately not send it). On 04/10/2015 10:44 AM, ?? wrote: > Thats' to say I should use content_types_accepted rather than > content_types_provided? > > I"m still puzzled with the difference between these two functions. when > to use which? > > Why is it ok when I use GET method with content_types_provided? > > thanks! > > > ------------------ Original ------------------ > *From: * "Lo?c Hoguin";; > *Send time:* Friday, Apr 10, 2015 3:28 PM > *To:* "??"<44290016@REDACTED>; > "erlang-questions"; > *Subject: * Re: [erlang-questions] [erlang-question][cowboy] Why got 415 > unsupportedmedia type when using POST method > > Because you didn't define a content_types_accepted function and at least > one function to accept the representation in the request. > > On 04/10/2015 04:30 AM, ?? wrote: > > Dear All, > > > > If I use GET method the REST API work properly, but when I change to > > POST, got 415 error > > > > My code: > > > > -export([init/2]). > > -export([allowed_methods/2, content_types_provided/2]). > > -export([to_json/2]). > > init(Req, Opts) -> > > {cowboy_rest, Req, Opts}. > > > > allowed_methods(Req, State) -> > > {[<<"POST">>], Req, State}. > > > > content_types_provided(Req, State) -> > > {[ > > {<<"application/json">>, to_json} > > ], Req, State}. > > > > to_json(Req, State) -> > > ..... > > > > when I test the REST API I: > > curl http://192.168.1.103:8080/cloud_init -v -XPOST -H"Content-Type: > > application/json" -d '{"client_id": "4342jltj3i"}' > > > > got : > > * Hostname was NOT found in DNS cache > > * Trying 192.168.1.103... > > * Connected to 192.168.1.103 (192.168.1.103) port 8080 (#0) > > > POST /cloudbox_init HTTP/1.1 > > > User-Agent: curl/7.37.1 > > > Host: 192.168.1.103:8080 > > > Accept: */* > > > content-type: application/json > > > Content-Length: 28 > > > > > * upload completely sent off: 28 out of 28 bytes > > < HTTP/1.1 415 Unsupported Media Type > > < connection: keep-alive > > * Server Cowboy is not blacklisted > > < server: Cowboy > > < date: Fri, 10 Apr 2015 00:56:15 GMT > > < content-length: 0 > > < content-type: application/json > > < > > * Connection #0 to host 192.168.1.103 left intact > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > > -- > Lo?c Hoguin > http://ninenines.eu -- Lo?c Hoguin http://ninenines.eu From bengt.kleberg@REDACTED Fri Apr 10 10:25:17 2015 From: bengt.kleberg@REDACTED (Bengt Kleberg) Date: Fri, 10 Apr 2015 10:25:17 +0200 Subject: [erlang-questions] how: Getting -export_type information at run time? In-Reply-To: References: <55276C5D.8000301@ericsson.com> Message-ID: <552788ED.6070407@ericsson.com> When I use beam_lib:chunks/2 I get tuples that are easy to identity as type information(*). When I turn the chunks into an abstract tree I do not understand how to find the type info any more. How should a fun for erl_syntax_lib:fold/3 look, to identify the type information nodes? bengt (*) [{attribute,64,type, {option_name, {type,64,union, [{atom,64,use_stdio}, {atom,65,cd}, {atom,66,compressed}, {atom,67,packet}, {atom,68,env}, {atom,69,start_timeout}, {atom,70,call_timeout}, {atom,71,buffer_size}]}, []}}, ...] On 04/10/2015 08:41 AM, Vlad Dumitrescu wrote: > Hi Bengt, > > You have to extract this information from beam_lib:chunks(BEAM, > [abstract_code]) by means of erl_syntax_lib:fold/3 to keep only the > typespecs. It would be nice to have a higher-level library that gives > access to these internals. > > regards, > Vlad > > > On Fri, Apr 10, 2015 at 8:23 AM, Bengt Kleberg > > wrote: > > Greetings, > > How do I find type information at run time? > > A very nice Erlang module takes an options list. The options are > -type defined and -export_type'ed. > How do another module get that information during run time? > > I thought that it might be in very_nice:module_info(attributes), > but no luck. > > > bengt > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Fri Apr 10 10:25:57 2015 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Fri, 10 Apr 2015 11:25:57 +0300 Subject: [erlang-questions] [erlang-question][cowboy] Why got 415 unsupportedmediatype when using POST method In-Reply-To: References: <55277BB8.5000309@ninenines.eu> <55278306.5080001@ninenines.eu> Message-ID: <55278915.4080900@ninenines.eu> No. "provided" doesn't return a result back to the client, it returns the representation of the resource, which is only requested through the HEAD and GET methods. GET has no request body so there's nothing to process. Changing "accepted" into "provided" won't work because processing a request body and providing a response body are completely different operations. I suggest reading the REST part of the guide here: http://ninenines.eu/docs/en/cowboy/1.0/guide/ - though if it confuses you then you should probably read the HTTP RFC 7231 or Fielding's REST chapter of his dissertation: http://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arch_style.htm If this is too difficult for you at this time I suggest using a plain HTTP handler. On 04/10/2015 11:04 AM, ?? wrote: > So that's to say I have to use "accepted" to process the request body > and then use "provided" to return a result back to client? But If I use > GET method I can both process request body and send back result in body > in only "provided" function. > > > ------------------ Original ------------------ > *From: * "Lo?c Hoguin";; > *Send time:* Friday, Apr 10, 2015 4:00 PM > *To:* "??"<44290016@REDACTED>; > "erlang-questions"; > *Subject: * Re: [erlang-questions] [erlang-question][cowboy] Why got 415 > unsupportedmediatype when using POST method > > "accepted" is for processing the resource representation in the request > body (POST/PATCH/PUT). > > "provided" is for sending the resource representation in the response > body (GET/HEAD, though HEAD will ultimately not send it). > > On 04/10/2015 10:44 AM, ?? wrote: > > Thats' to say I should use content_types_accepted rather than > > content_types_provided? > > > > I"m still puzzled with the difference between these two functions. when > > to use which? > > > > Why is it ok when I use GET method with content_types_provided? > > > > thanks! > > > > > > ------------------ Original ------------------ > > *From: * "Lo?c Hoguin";; > > *Send time:* Friday, Apr 10, 2015 3:28 PM > > *To:* "??"<44290016@REDACTED>; > > "erlang-questions"; > > *Subject: * Re: [erlang-questions] [erlang-question][cowboy] Why got 415 > > unsupportedmedia type when using POST method > > > > Because you didn't define a content_types_accepted function and at least > > one function to accept the representation in the request. > > > > On 04/10/2015 04:30 AM, ?? wrote: > > > Dear All, > > > > > > If I use GET method the REST API work properly, but when I change to > > > POST, got 415 error > > > > > > My code: > > > > > > -export([init/2]). > > > -export([allowed_methods/2, content_types_provided/2]). > > > -export([to_json/2]). > > > init(Req, Opts) -> > > > {cowboy_rest, Req, Opts}. > > > > > > allowed_methods(Req, State) -> > > > {[<<"POST">>], Req, State}. > > > > > > content_types_provided(Req, State) -> > > > {[ > > > {<<"application/json">>, to_json} > > > ], Req, State}. > > > > > > to_json(Req, State) -> > > > ..... > > > > > > when I test the REST API I: > > > curl http://192.168.1.103:8080/cloud_init -v -XPOST -H"Content-Type: > > > application/json" -d '{"client_id": "4342jltj3i"}' > > > > > > got : > > > * Hostname was NOT found in DNS cache > > > * Trying 192.168.1.103... > > > * Connected to 192.168.1.103 (192.168.1.103) port 8080 (#0) > > > > POST /cloudbox_init HTTP/1.1 > > > > User-Agent: curl/7.37.1 > > > > Host: 192.168.1.103:8080 > > > > Accept: */* > > > > content-type: application/json > > > > Content-Length: 28 > > > > > > > * upload completely sent off: 28 out of 28 bytes > > > < HTTP/1.1 415 Unsupported Media Type > > > < connection: keep-alive > > > * Server Cowboy is not blacklisted > > > < server: Cowboy > > > < date: Fri, 10 Apr 2015 00:56:15 GMT > > > < content-length: 0 > > > < content-type: application/json > > > < > > > * Connection #0 to host 192.168.1.103 left intact > > > > > > > > > _______________________________________________ > > > erlang-questions mailing list > > > erlang-questions@REDACTED > > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > > -- > > Lo?c Hoguin > > http://ninenines.eu > > -- > Lo?c Hoguin > http://ninenines.eu -- Lo?c Hoguin http://ninenines.eu From vladdu55@REDACTED Fri Apr 10 11:36:40 2015 From: vladdu55@REDACTED (Vlad Dumitrescu) Date: Fri, 10 Apr 2015 11:36:40 +0200 Subject: [erlang-questions] how: Getting -export_type information at run time? In-Reply-To: <552788ED.6070407@ericsson.com> References: <55276C5D.8000301@ericsson.com> <552788ED.6070407@ericsson.com> Message-ID: On Fri, Apr 10, 2015 at 10:25 AM, Bengt Kleberg wrote: > When I use beam_lib:chunks/2 I get tuples that are easy to identity as > type information(*). > When I turn the chunks into an abstract tree I do not understand how to > find the type info any more. How should a fun for erl_syntax_lib:fold/3 > look, to identify the type information nodes? > > I haven't used that myself, so I don't really know, but if you get tuples that you can use directly, maybe you don't need to call fold. /Vlad > > bengt > (*) > [{attribute,64,type, > {option_name, > {type,64,union, > [{atom,64,use_stdio}, > {atom,65,cd}, > {atom,66,compressed}, > {atom,67,packet}, > {atom,68,env}, > {atom,69,start_timeout}, > {atom,70,call_timeout}, > {atom,71,buffer_size}]}, > []}}, > ...] > > > > On 04/10/2015 08:41 AM, Vlad Dumitrescu wrote: > > Hi Bengt, > > You have to extract this information from beam_lib:chunks(BEAM, > [abstract_code]) by means of erl_syntax_lib:fold/3 to keep only the > typespecs. It would be nice to have a higher-level library that gives > access to these internals. > > regards, > Vlad > > > On Fri, Apr 10, 2015 at 8:23 AM, Bengt Kleberg > wrote: > >> Greetings, >> >> How do I find type information at run time? >> >> A very nice Erlang module takes an options list. The options are -type >> defined and -export_type'ed. >> How do another module get that information during run time? >> >> I thought that it might be in very_nice:module_info(attributes), but no >> luck. >> >> >> bengt >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris@REDACTED Fri Apr 10 07:54:30 2015 From: chris@REDACTED (Christopher Adams) Date: Fri, 10 Apr 2015 13:54:30 +0800 Subject: [erlang-questions] [erlang-question][cowboy] Why got 415 unsupported media type when using POST method In-Reply-To: References: Message-ID: On Fri, Apr 10, 2015 at 9:30 AM, ?? <44290016@REDACTED> wrote: > Dear All, > > If I use GET method the REST API work properly, but when I change to POST, > got 415 error > > My code: > > -export([init/2]). > -export([allowed_methods/2, content_types_provided/2]). You have implemented content_types_provided, but you also need to implement content_types_accepted with an appropriate callback (like from_json) in order to accept application/json from the client. Christopher From 44290016@REDACTED Fri Apr 10 09:44:46 2015 From: 44290016@REDACTED (=?utf-8?B?5a+C5a+C?=) Date: Fri, 10 Apr 2015 15:44:46 +0800 Subject: [erlang-questions] [erlang-question][cowboy] Why got 415 unsupportedmedia type when using POST method In-Reply-To: <55277BB8.5000309@ninenines.eu> References: <55277BB8.5000309@ninenines.eu> Message-ID: Thats' to say I should use content_types_accepted rather than content_types_provided? I"m still puzzled with the difference between these two functions. when to use which? Why is it ok when I use GET method with content_types_provided? thanks! ------------------ Original ------------------ From: "Lo?c Hoguin";; Send time: Friday, Apr 10, 2015 3:28 PM To: "??"<44290016@REDACTED>; "erlang-questions"; Subject: Re: [erlang-questions] [erlang-question][cowboy] Why got 415 unsupportedmedia type when using POST method Because you didn't define a content_types_accepted function and at least one function to accept the representation in the request. On 04/10/2015 04:30 AM, ?? wrote: > Dear All, > > If I use GET method the REST API work properly, but when I change to > POST, got 415 error > > My code: > > -export([init/2]). > -export([allowed_methods/2, content_types_provided/2]). > -export([to_json/2]). > init(Req, Opts) -> > {cowboy_rest, Req, Opts}. > > allowed_methods(Req, State) -> > {[<<"POST">>], Req, State}. > > content_types_provided(Req, State) -> > {[ > {<<"application/json">>, to_json} > ], Req, State}. > > to_json(Req, State) -> > ..... > > when I test the REST API I: > curl http://192.168.1.103:8080/cloud_init -v -XPOST -H"Content-Type: > application/json" -d '{"client_id": "4342jltj3i"}' > > got : > * Hostname was NOT found in DNS cache > * Trying 192.168.1.103... > * Connected to 192.168.1.103 (192.168.1.103) port 8080 (#0) > > POST /cloudbox_init HTTP/1.1 > > User-Agent: curl/7.37.1 > > Host: 192.168.1.103:8080 > > Accept: */* > > content-type: application/json > > Content-Length: 28 > > > * upload completely sent off: 28 out of 28 bytes > < HTTP/1.1 415 Unsupported Media Type > < connection: keep-alive > * Server Cowboy is not blacklisted > < server: Cowboy > < date: Fri, 10 Apr 2015 00:56:15 GMT > < content-length: 0 > < content-type: application/json > < > * Connection #0 to host 192.168.1.103 left intact > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin http://ninenines.eu -------------- next part -------------- An HTML attachment was scrubbed... URL: From 44290016@REDACTED Fri Apr 10 10:04:11 2015 From: 44290016@REDACTED (=?utf-8?B?5a+C5a+C?=) Date: Fri, 10 Apr 2015 16:04:11 +0800 Subject: [erlang-questions] [erlang-question][cowboy] Why got 415 unsupportedmediatype when using POST method In-Reply-To: <55278306.5080001@ninenines.eu> References: <55277BB8.5000309@ninenines.eu> <55278306.5080001@ninenines.eu> Message-ID: So that's to say I have to use "accepted" to process the request body and then use "provided" to return a result back to client? But If I use GET method I can both process request body and send back result in body in only "provided" function. ------------------ Original ------------------ From: "Lo?c Hoguin";; Send time: Friday, Apr 10, 2015 4:00 PM To: "??"<44290016@REDACTED>; "erlang-questions"; Subject: Re: [erlang-questions] [erlang-question][cowboy] Why got 415 unsupportedmediatype when using POST method "accepted" is for processing the resource representation in the request body (POST/PATCH/PUT). "provided" is for sending the resource representation in the response body (GET/HEAD, though HEAD will ultimately not send it). On 04/10/2015 10:44 AM, ?? wrote: > Thats' to say I should use content_types_accepted rather than > content_types_provided? > > I"m still puzzled with the difference between these two functions. when > to use which? > > Why is it ok when I use GET method with content_types_provided? > > thanks! > > > ------------------ Original ------------------ > *From: * "Lo?c Hoguin";; > *Send time:* Friday, Apr 10, 2015 3:28 PM > *To:* "??"<44290016@REDACTED>; > "erlang-questions"; > *Subject: * Re: [erlang-questions] [erlang-question][cowboy] Why got 415 > unsupportedmedia type when using POST method > > Because you didn't define a content_types_accepted function and at least > one function to accept the representation in the request. > > On 04/10/2015 04:30 AM, ?? wrote: > > Dear All, > > > > If I use GET method the REST API work properly, but when I change to > > POST, got 415 error > > > > My code: > > > > -export([init/2]). > > -export([allowed_methods/2, content_types_provided/2]). > > -export([to_json/2]). > > init(Req, Opts) -> > > {cowboy_rest, Req, Opts}. > > > > allowed_methods(Req, State) -> > > {[<<"POST">>], Req, State}. > > > > content_types_provided(Req, State) -> > > {[ > > {<<"application/json">>, to_json} > > ], Req, State}. > > > > to_json(Req, State) -> > > ..... > > > > when I test the REST API I: > > curl http://192.168.1.103:8080/cloud_init -v -XPOST -H"Content-Type: > > application/json" -d '{"client_id": "4342jltj3i"}' > > > > got : > > * Hostname was NOT found in DNS cache > > * Trying 192.168.1.103... > > * Connected to 192.168.1.103 (192.168.1.103) port 8080 (#0) > > > POST /cloudbox_init HTTP/1.1 > > > User-Agent: curl/7.37.1 > > > Host: 192.168.1.103:8080 > > > Accept: */* > > > content-type: application/json > > > Content-Length: 28 > > > > > * upload completely sent off: 28 out of 28 bytes > > < HTTP/1.1 415 Unsupported Media Type > > < connection: keep-alive > > * Server Cowboy is not blacklisted > > < server: Cowboy > > < date: Fri, 10 Apr 2015 00:56:15 GMT > > < content-length: 0 > > < content-type: application/json > > < > > * Connection #0 to host 192.168.1.103 left intact > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > > -- > Lo?c Hoguin > http://ninenines.eu -- Lo?c Hoguin http://ninenines.eu -------------- next part -------------- An HTML attachment was scrubbed... URL: From 44290016@REDACTED Fri Apr 10 10:14:37 2015 From: 44290016@REDACTED (=?utf-8?B?5a+C5a+C?=) Date: Fri, 10 Apr 2015 16:14:37 +0800 Subject: [erlang-questions] [erlang-question][cowboy] Why got 415 unsupportedmediatype when using POST method In-Reply-To: References: <55277BB8.5000309@ninenines.eu> <55278306.5080001@ninenines.eu> Message-ID: if I change "provided" to "accepted" still get error! ------------------ Original ------------------ From: "?? ";<44290016@REDACTED>; Send time: Friday, Apr 10, 2015 4:04 PM To: "Lo?c Hoguin"; "erlang-questions"; Subject: Re: [erlang-questions] [erlang-question][cowboy] Why got 415 unsupportedmediatype when using POST method So that's to say I have to use "accepted" to process the request body and then use "provided" to return a result back to client? But If I use GET method I can both process request body and send back result in body in only "provided" function. ------------------ Original ------------------ From: "Lo?c Hoguin";; Send time: Friday, Apr 10, 2015 4:00 PM To: "??"<44290016@REDACTED>; "erlang-questions"; Subject: Re: [erlang-questions] [erlang-question][cowboy] Why got 415 unsupportedmediatype when using POST method "accepted" is for processing the resource representation in the request body (POST/PATCH/PUT). "provided" is for sending the resource representation in the response body (GET/HEAD, though HEAD will ultimately not send it). On 04/10/2015 10:44 AM, ?? wrote: > Thats' to say I should use content_types_accepted rather than > content_types_provided? > > I"m still puzzled with the difference between these two functions. when > to use which? > > Why is it ok when I use GET method with content_types_provided? > > thanks! > > > ------------------ Original ------------------ > *From: * "Lo?c Hoguin";; > *Send time:* Friday, Apr 10, 2015 3:28 PM > *To:* "??"<44290016@REDACTED>; > "erlang-questions"; > *Subject: * Re: [erlang-questions] [erlang-question][cowboy] Why got 415 > unsupportedmedia type when using POST method > > Because you didn't define a content_types_accepted function and at least > one function to accept the representation in the request. > > On 04/10/2015 04:30 AM, ?? wrote: > > Dear All, > > > > If I use GET method the REST API work properly, but when I change to > > POST, got 415 error > > > > My code: > > > > -export([init/2]). > > -export([allowed_methods/2, content_types_provided/2]). > > -export([to_json/2]). > > init(Req, Opts) -> > > {cowboy_rest, Req, Opts}. > > > > allowed_methods(Req, State) -> > > {[<<"POST">>], Req, State}. > > > > content_types_provided(Req, State) -> > > {[ > > {<<"application/json">>, to_json} > > ], Req, State}. > > > > to_json(Req, State) -> > > ..... > > > > when I test the REST API I: > > curl http://192.168.1.103:8080/cloud_init -v -XPOST -H"Content-Type: > > application/json" -d '{"client_id": "4342jltj3i"}' > > > > got : > > * Hostname was NOT found in DNS cache > > * Trying 192.168.1.103... > > * Connected to 192.168.1.103 (192.168.1.103) port 8080 (#0) > > > POST /cloudbox_init HTTP/1.1 > > > User-Agent: curl/7.37.1 > > > Host: 192.168.1.103:8080 > > > Accept: */* > > > content-type: application/json > > > Content-Length: 28 > > > > > * upload completely sent off: 28 out of 28 bytes > > < HTTP/1.1 415 Unsupported Media Type > > < connection: keep-alive > > * Server Cowboy is not blacklisted > > < server: Cowboy > > < date: Fri, 10 Apr 2015 00:56:15 GMT > > < content-length: 0 > > < content-type: application/json > > < > > * Connection #0 to host 192.168.1.103 left intact > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > > -- > Lo?c Hoguin > http://ninenines.eu -- Lo?c Hoguin http://ninenines.eu -------------- next part -------------- An HTML attachment was scrubbed... URL: From garry@REDACTED Fri Apr 10 15:01:46 2015 From: garry@REDACTED (Garry Hodgson) Date: Fri, 10 Apr 2015 09:01:46 -0400 Subject: [erlang-questions] segfaults in cowboy rest server In-Reply-To: <55277153.4000905@gmail.com> References: <55268B41.2020207@research.att.com> <55277153.4000905@gmail.com> Message-ID: <5527C9BA.8000509@research.att.com> On 4/10/15 2:44 AM, Michael Truog wrote: > If you have any port drivers or NIFs that are used in the system, it > would be best to examine them closely for errors hiding within. For > example, a port driver that is written with C++ that throws an > exception into the Erlang VM (for example, a timeout exception that > has never before occurred) will crash the Erlang VM in new and > exciting ways that can include seg faults. > > If you are using maps, that could also be a cause, since there are > still bugs being worked on. not using either port drivers or NIFs, nor maps. just plain vanilla erlang. the segfaults seemed to start when i added the forced gc, but that was shortly after migrating to cowboy and r17.4, so it's hard to isolate just what could have changed to cause this. From zandra@REDACTED Fri Apr 10 15:13:03 2015 From: zandra@REDACTED (Zandra Hird) Date: Fri, 10 Apr 2015 15:13:03 +0200 Subject: [erlang-questions] Patch package OTP 17.5.1 released Message-ID: <5527CC5F.10108@erlang.org> Patch Package: OTP 17.5.1 Git Tag: OTP-17.5.1 Check out the git tag OTP-17.5.1, and build a full OTP system including documentation. Apply one or more applications from this build as patches to your installation using the 'otp_patch_apply' tool. For information on install requirements, see descriptions for each application version below. --------------------------------------------------------------------- --- ssh-3.2.1 ------------------------------------------------------- --------------------------------------------------------------------- Note! The ssh-3.2.1 application can*not* be applied independently of other applications on an arbitrary OTP 17 installation. On a full OTP 17 installation, also the following runtime dependency has to be satisfied: -- stdlib-2.3 (first satisfied in OTP 17.4) --- Fixed Bugs and Malfunctions --- OTP-12645 Application(s): ssh Related Id(s): seq12816 Ssh crashed if a message was sent on a channel with packet_size = 0. A new option for ssh:daemon is also introduced: minimal_remote_max_packet_size. This option sets the least max packet size declaration that the daemon will accept from a client. The default value is 0 to maintain compatibility with OpenSSH and the rfc:s. Full runtime dependencies of ssh-3.2.1: crypto-3.3, erts-6.0, kernel-3.0, public_key-0.22, stdlib-2.3 --------------------------------------------------------------------- --------------------------------------------------------------------- --------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From ddosia@REDACTED Fri Apr 10 18:00:19 2015 From: ddosia@REDACTED (Daniil Churikov) Date: Fri, 10 Apr 2015 17:00:19 +0100 Subject: [erlang-questions] starting otp application dependencies Message-ID: Generally, if you use releases in production, to avoid some weird bugs I'd suggest using releases in development as well. This may be cumbersome to certain extent, but may be improved in a several ways: 1. relx have a `--dev-mode` flag, to create symlinks on libs you are using 2. hot code reload tool, like sync https://github.com/rustyio/sync 2015-04-08 12:48 GMT+01:00 Nicolas Martyanoff : > On 2015-04-08 13:43, Sergej Jure?ko wrote: > > application:ensure_all_started(app). > > Exactly what I needed. Thank you ! > > Regards, > > -- > Nicolas Martyanoff > http://wandrian.net > khaelin@REDACTED > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jlnavarro111@REDACTED Sat Apr 11 02:35:51 2015 From: jlnavarro111@REDACTED (Jose Luis Navarro) Date: Sat, 11 Apr 2015 02:35:51 +0200 Subject: [erlang-questions] Nitrogen-based XMPP client? In-Reply-To: <1428080581.02616486@apps.rackspace.com> References: <20150403131046.46149dd5@gmail.com> <1428080581.02616486@apps.rackspace.com> Message-ID: Maybe this can help you https :// github.com / artefactop / ebot is not updated but can be a good start. Regards On Apr 3, 2015 7:03 PM, wrote: > Hi Grigory, > > Have you published code anywhere? > > Thanks, > > Lloyd > > -----Original Message----- > From: "Grigory Fateyev" > Sent: Friday, April 3, 2015 6:10am > To: erlang-questions@REDACTED > Subject: Re: [erlang-questions] Nitrogen-based XMPP client? > > Hello Marc Worrell! > On Fri, 3 Apr 2015 09:30:02 +0200 you wrote: > > > Hi Lloyd, > > > > Did you have a look at the exmpp client library? > > There are some tutorials which might help in getting you started. > > > > http://processone.github.io/exmpp/ > > > > > > I don?t know of any other xmpp client libraries. > > I have a very simple (hope in future it'll be more useful) a pet project > xmpp client that connects to rooms and does simple things. Like a bot. > > I found exmpp more ready to use it like an example, but the author of > exmpp said: > > "Just as a head up, note that post is > 5 years old already. The > main goal of exmpp itself was to be used as parser/xml core in > ejabberd". > > So I started my project as simple request-response flow with server. > The main problem is to read XMPP XEPs. :) > > > > > On 3 apr. 2015, at 05:29, Lloyd R. Prentice > > > wrote: > > > > > > Hello, > > > > > > I'm considering integrating multi-user chat into a Nitrogen > > > application I'm building--- assuming that I'll support it with > > > Mongoose IM. As usual, I'm in over my head, but persistent. > > > > > > I'm imagining a small number ( four? six?) thematic chat rooms > > > serving a relatively small number of visitors at a time. As to > > > scale, think of a book-signing event at a bookstore. > > > > > > I see that there are several JavaScript XMPP client packages that I > > > could integrate into my Nitrogen web pages but, if that's the way, > > > which one? Or, is there a native Erlang XMPP client app that I > > > should look at? Indeed, is Mongoose IM overkill? After all, the > > > Nitrogen demos provide a very simple tinker-toy chat example, and > > > numerous others can be found on the web though, most, rather long > > > in the tooth. > > > > > > And more, how much harder would it be to integrate video chat > > > using, say, web sockets? > > > > > > In other words, how can I best keep the gotchas, demons, and > > > goblins at bay sufficiently to master the arcane arts of multi-user > > > chat sufficiently to build my application before I die of old age? > > > > > > I'd much appreciate pointers or, better yet, pointers to actual > > > code. > > > > > > All the best, > > > > > > LRP > > > > > > Sent from my iPad > > > _______________________________________________ > > > erlang-questions mailing list > > > erlang-questions@REDACTED > > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > -- > Best regards! > gfborn [at] gmail [dot] com > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jose.valim@REDACTED Sat Apr 11 17:22:20 2015 From: jose.valim@REDACTED (=?UTF-8?Q?Jos=C3=A9_Valim?=) Date: Sat, 11 Apr 2015 17:22:20 +0200 Subject: [erlang-questions] Proposal for not allowing tuple calls Message-ID: More than a year ago, the OTP team has decided to not remove tuple calls from the language i.e. this is valid Erlang code: ({erlang, 1, 2, 3}):is_tuple(). Most times though, such calls are written in the format: Var:some_fun(Arg1, Arg2). The reasoning is that we had to maintain backwards compatibility since such calls were used by many projects. This behaviour has many downsides though, many of them debated in this mailing list, for example the performance hit of such calls as well as the fact this behaviour is confusing, specially when you accidentally trigger it for the first time. Furthermore, erlang:apply/3 also allows tuple calls, not giving us any alternative to disallow such behaviour, even if it explicitly. For those reasons, I would like to propose a new function and a compiler option that disables the tuple calls. erlang:apply_no_tuple(Mod, Fun, Args). The function above works as apply, except it will raise badarg unless Mod is an atom. The idea is that the optimizations we had in mind for the regular dispatch could now be applied to this new function. We could also have a compiler option, no_tuple_calls, that will compile all underlying Var:some_fun(Arg1, Arg2) into erlang:apply_no_tuple/3 instead of erlang:apply/3 for the current module. This is a great opportunity because build tools like rebar3, mix and erlang.mk could ship with such option as a default, slowly phasing out the tuple behaviour. In the long term, we could even make the tuple call behaviour opt-in (rather than opt-out). PS: erlang:apply_no_tuple/3 and no_tuple_calls are horrible names but I believe we can find good names along the way if there is interest. Another alternative I had in mind was erlang:apply_strict/3 and only_strict_calls for the compiler option. *Jos? Valim* www.plataformatec.com.br Skype: jv.ptec Founder and Lead Developer -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Sat Apr 11 19:39:49 2015 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Sat, 11 Apr 2015 20:39:49 +0300 Subject: [erlang-questions] Proposal for not allowing tuple calls In-Reply-To: References: Message-ID: <55295C65.3090806@ninenines.eu> On 04/11/2015 06:22 PM, Jos? Valim wrote: > For those reasons, I would like to propose a new function and a compiler > option that disables the tuple calls. > > erlang:apply_no_tuple(Mod, Fun, Args). What about: Erlang 19: add command line option to disable tuple calls Erlang 20: make that option disable tuple calls by default Erlang 21: add an option when you compile the VM to disable tuple calls Erlang 22: make that option disable tuple calls by default and be happy Though I suspect any attempt to remove this functionality will be met with resistance from part of the Erlang community. -- Lo?c Hoguin http://ninenines.eu From askjuise@REDACTED Sun Apr 12 19:23:09 2015 From: askjuise@REDACTED (Alexander Petrovsky) Date: Sun, 12 Apr 2015 21:23:09 +0400 Subject: [erlang-questions] Efficient sum matrix by column Message-ID: Hello! I have about 100 nodes in cluster, each node in cluster contains about 100000 elements in proplist. All proplists have equal length, the same keys, but different values. I need to get the sum values for every key between all proplists. By example, it can be represented as matrix NxM (N~=100, M~=100000), and I need to get the sum by column. The problem is that it's too slow, even when I use the hackish way with ets:update_counter and N=2, it's take about 1.5 secs, If I make in parallel, it's take about 2-3 seconds. How can I make it fast and efficient? -- ?????????? ????????? / Alexander Petrovsky, Skype: askjuise Phone: +7 914 8 820 815 -------------- next part -------------- An HTML attachment was scrubbed... URL: From gfborn@REDACTED Sun Apr 12 19:53:11 2015 From: gfborn@REDACTED (Grigory Fateyev) Date: Sun, 12 Apr 2015 20:53:11 +0300 Subject: [erlang-questions] Efficient sum matrix by column In-Reply-To: References: Message-ID: <20150412205311.65c26989@gmail.com> Hello Alexander Petrovsky! On Sun, 12 Apr 2015 21:23:09 +0400 you wrote: > Hello! > > I have about 100 nodes in cluster, each node in cluster contains about > 100000 elements in proplist. All proplists have equal length, the same > keys, but different values. I need to get the sum values for every key > between all proplists. By example, it can be represented as matrix NxM > (N~=100, M~=100000), and I need to get the sum by column. > > The problem is that it's too slow, even when I use the hackish way > with ets:update_counter and N=2, it's take about 1.5 secs, If I make > in parallel, it's take about 2-3 seconds. > > How can I make it fast and efficient? > Look at lists:keyfind/4 It is much faster than proplists. -- Best regards! gfborn [at] gmail [dot] com From ilya.khlopotov@REDACTED Sun Apr 12 20:18:22 2015 From: ilya.khlopotov@REDACTED (ILYA Khlopotov) Date: Sun, 12 Apr 2015 11:18:22 -0700 Subject: [erlang-questions] Efficient sum matrix by column In-Reply-To: <20150412205311.65c26989@gmail.com> References: <20150412205311.65c26989@gmail.com> Message-ID: I would suggest to try couple of things. 1. If N is always the same and as you've mentioned contains same keys and the total number of elements in proplists is less than 67108863 you can replace proplist with tuple. In this case you can use either foldl with getelement/setelement. 2. review the requirements and consider incremental way of updating the counters on insertion. BR, ILYA On Sun, Apr 12, 2015 at 10:53 AM, Grigory Fateyev wrote: > Hello Alexander Petrovsky! > On Sun, 12 Apr 2015 21:23:09 +0400 you wrote: > > > Hello! > > > > I have about 100 nodes in cluster, each node in cluster contains about > > 100000 elements in proplist. All proplists have equal length, the same > > keys, but different values. I need to get the sum values for every key > > between all proplists. By example, it can be represented as matrix NxM > > (N~=100, M~=100000), and I need to get the sum by column. > > > > The problem is that it's too slow, even when I use the hackish way > > with ets:update_counter and N=2, it's take about 1.5 secs, If I make > > in parallel, it's take about 2-3 seconds. > > > > How can I make it fast and efficient? > > > > Look at lists:keyfind/4 It is much faster than proplists. > > -- > Best regards! > gfborn [at] gmail [dot] com > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vances@REDACTED Sun Apr 12 20:46:17 2015 From: vances@REDACTED (Vance Shipley) Date: Mon, 13 Apr 2015 00:16:17 +0530 Subject: [erlang-questions] Efficient sum matrix by column In-Reply-To: References: Message-ID: On Sun, Apr 12, 2015 at 10:53 PM, Alexander Petrovsky wrote: > I have about 100 nodes in cluster, each node in cluster contains about > 100000 elements in proplist. All proplists have equal length, the same keys, > but different values. I need to get the sum values for every key between all > proplists. By example, it can be represented as matrix NxM (N~=100, > M~=100000), and I need to get the sum by column. Firstly proplists is probably not the data structure you want to use, as has been pointed out. Secondly you may want to consider storing the column sums, updating each time a column row is updated. That could be quite cheap to do if the sums are always incremented for example. -- -Vance From askjuise@REDACTED Sun Apr 12 21:08:30 2015 From: askjuise@REDACTED (Alexander Petrovsky) Date: Sun, 12 Apr 2015 23:08:30 +0400 Subject: [erlang-questions] Efficient sum matrix by column In-Reply-To: References: <20150412205311.65c26989@gmail.com> Message-ID: Hi! 2015-04-12 22:18 GMT+04:00 ILYA Khlopotov : > I would suggest to try couple of things. > > 1. If N is always the same and as you've mentioned contains same keys and > the total number of elements in proplists is less than 67108863 you can > replace proplist with tuple. In this case you can use either foldl with > getelement/setelement. > Yes, it's less then 67198863. Do I understand correctly, you propose instead prolists like [{a,1}, {b,2} ...] use tuple like {a, 1, b, 2, ...} or just {1, 2, ...} ? Does it will be more efficient(getelement/setelement) then pattern matching in list comprehension over proplist? > 2. review the requirements and consider incremental way of updating the > counters on insertion. > Yep, as I say in my first letter, I'm using incremental way right now via ets:update_counter, it takes about 1.5 secs when N=2. But it's still too slow. > > BR, > ILYA > > > > > On Sun, Apr 12, 2015 at 10:53 AM, Grigory Fateyev > wrote: > >> Hello Alexander Petrovsky! >> On Sun, 12 Apr 2015 21:23:09 +0400 you wrote: >> >> > Hello! >> > >> > I have about 100 nodes in cluster, each node in cluster contains about >> > 100000 elements in proplist. All proplists have equal length, the same >> > keys, but different values. I need to get the sum values for every key >> > between all proplists. By example, it can be represented as matrix NxM >> > (N~=100, M~=100000), and I need to get the sum by column. >> > >> > The problem is that it's too slow, even when I use the hackish way >> > with ets:update_counter and N=2, it's take about 1.5 secs, If I make >> > in parallel, it's take about 2-3 seconds. >> > >> > How can I make it fast and efficient? >> > >> >> Look at lists:keyfind/4 It is much faster than proplists. >> >> -- >> Best regards! >> gfborn [at] gmail [dot] com >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -- ?????????? ????????? / Alexander Petrovsky, Skype: askjuise Phone: +7 914 8 820 815 -------------- next part -------------- An HTML attachment was scrubbed... URL: From askjuise@REDACTED Sun Apr 12 21:49:28 2015 From: askjuise@REDACTED (Alexander Petrovsky) Date: Sun, 12 Apr 2015 23:49:28 +0400 Subject: [erlang-questions] Efficient sum matrix by column In-Reply-To: References: Message-ID: The proplists already sorted. Thanks, I will look at N-ary tree. Looks like the NIF it's not a good idea due the sum of two lists with 100000 each will be more then 1ms (btw I know about dirty schedulers). 2015-04-12 22:42 GMT+04:00 Hynek Vychodil : > Sort by proper key and then merge join. Form N-nary tree between your > nodes for merging and experiment with N. If not fast enough, convert to > binaries and write NIFs, compress (Snappy, LZ4, ...) and so on. Unless you > write more how keys in your proplists look like, there is not so much to > advice. > > On Sun, Apr 12, 2015 at 7:23 PM, Alexander Petrovsky > wrote: > >> Hello! >> >> I have about 100 nodes in cluster, each node in cluster contains about >> 100000 elements in proplist. All proplists have equal length, the same >> keys, but different values. I need to get the sum values for every key >> between all proplists. By example, it can be represented as matrix NxM >> (N~=100, M~=100000), and I need to get the sum by column. >> >> The problem is that it's too slow, even when I use the hackish way with >> ets:update_counter and N=2, it's take about 1.5 secs, If I make in >> parallel, it's take about 2-3 seconds. >> >> How can I make it fast and efficient? >> >> -- >> ?????????? ????????? / Alexander Petrovsky, >> >> Skype: askjuise >> Phone: +7 914 8 820 815 >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -- ?????????? ????????? / Alexander Petrovsky, Skype: askjuise Phone: +7 914 8 820 815 -------------- next part -------------- An HTML attachment was scrubbed... URL: From askjuise@REDACTED Sun Apr 12 22:03:03 2015 From: askjuise@REDACTED (Alexander Petrovsky) Date: Mon, 13 Apr 2015 00:03:03 +0400 Subject: [erlang-questions] Efficient sum matrix by column In-Reply-To: References: Message-ID: I can update only full row every dt second only. Every element in proplist it's number of requests from client for 1 second smoothed by ewma (i.e. smoothed rate). Every second I want to recalculate summary rate per client per cluster (as I say sum by column; sum between all nodes per client). 2015-04-12 22:46 GMT+04:00 Vance Shipley : > On Sun, Apr 12, 2015 at 10:53 PM, Alexander Petrovsky > wrote: > > I have about 100 nodes in cluster, each node in cluster contains about > > 100000 elements in proplist. All proplists have equal length, the same > keys, > > but different values. I need to get the sum values for every key between > all > > proplists. By example, it can be represented as matrix NxM (N~=100, > > M~=100000), and I need to get the sum by column. > > Firstly proplists is probably not the data structure you want to use, > as has been pointed out. > > Secondly you may want to consider storing the column sums, updating > each time a column row is updated. > That could be quite cheap to do if the sums are always incremented for > example. > > -- > -Vance > -- ?????????? ????????? / Alexander Petrovsky, Skype: askjuise Phone: +7 914 8 820 815 -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinoski@REDACTED Sun Apr 12 22:48:15 2015 From: vinoski@REDACTED (Steve Vinoski) Date: Sun, 12 Apr 2015 16:48:15 -0400 Subject: [erlang-questions] Efficient sum matrix by column In-Reply-To: References: Message-ID: On Sun, Apr 12, 2015 at 3:49 PM, Alexander Petrovsky wrote: > The proplists already sorted. Thanks, I will look at N-ary tree. Looks > like the NIF it's not a good idea due the sum of two lists with 100000 each > will be more then 1ms (btw I know about dirty schedulers). > You don't need dirty schedulers in this case. Have a look at enif_schedule_nif, which would allow you to break the computation into chunks that don't violate scheduler time constraints: http://www.erlang.org/doc/man/erl_nif.html#enif_schedule_nif Here's an example from a presentation I've given covering schedulers and nifs: https://github.com/vinoski/bitwise/blob/master/c_src/bitwise_nif.c#L40-L110 --steve -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric.pailleau@REDACTED Sun Apr 12 22:58:56 2015 From: eric.pailleau@REDACTED (=?ISO-8859-1?Q?=C9ric_Pailleau?=) Date: Sun, 12 Apr 2015 22:58:56 +0200 Subject: [erlang-questions] Efficient sum matrix by column In-Reply-To: Message-ID: <6cca1a60-76bf-4887-b847-4805cab943ae@email.android.com> Hi, Did you try gb_trees instead ? Iterator/1 is very efficient. Le?12 avr. 2015 21:08, Alexander Petrovsky a ?crit?: > > Hi! > > 2015-04-12 22:18 GMT+04:00 ILYA Khlopotov : >> >> I would suggest to try couple of things. >> >> 1. If N is always the same and as you've mentioned contains same keys and the total number of elements in proplists is less than 67108863 you can replace proplist with tuple. In this case you can use either foldl with getelement/setelement. > > > Yes, it's less then 67198863. Do I understand correctly, you propose instead prolists like [{a,1}, {b,2} ...] use tuple like {a, 1, b, 2, ...} or just {1, 2, ...} ? Does it will be more efficient(getelement/setelement) then pattern matching in?list comprehension over proplist? > ? >> >> 2. review the requirements and consider incremental way of updating the counters on insertion. > > > Yep, as I say in my first letter, I'm using incremental way right now via ets:update_counter, it takes about 1.5 secs when N=2. But it's still too slow.? > ? >> >> >> BR, >> ILYA >> >> >> >> >> On Sun, Apr 12, 2015 at 10:53 AM, Grigory Fateyev wrote: >>> >>> Hello Alexander Petrovsky! >>> On Sun, 12 Apr 2015 21:23:09 +0400 you wrote: >>> >>> > Hello! >>> > >>> > I have about 100 nodes in cluster, each node in cluster contains about >>> > 100000 elements in proplist. All proplists have equal length, the same >>> > keys, but different values. I need to get the sum values for every key >>> > between all proplists. By example, it can be represented as matrix NxM >>> > (N~=100, M~=100000), and I need to get the sum by column. >>> > >>> > The problem is that it's too slow, even when I use the hackish way >>> > with ets:update_counter and N=2, it's take about 1.5 secs, If I make >>> > in parallel, it's take about 2-3 seconds. >>> > >>> > How can I make it fast and efficient? >>> > >>> >>> Look at lists:keyfind/4 It is much faster than proplists. >>> >>> -- >>> Best regards! >>> gfborn [at] gmail [dot] com >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > > > -- > ?????????? ????????? / Alexander Petrovsky, > > Skype: askjuise > Phone: +7 914 8 820 815 > From gordeev.vladimir.v@REDACTED Mon Apr 13 00:37:03 2015 From: gordeev.vladimir.v@REDACTED (Vladimir Gordeev) Date: Mon, 13 Apr 2015 00:37:03 +0200 Subject: [erlang-questions] custom lager levels or filtering Message-ID: Let me describe my problem. I have two different kinds of logging inside my code: regular console/file output and special log calls that should be processed by my log backend. By their meaning and usage they are completely separate. Semantically both logs actually are info level, but still they should be handled separately. I can add metadata to my specific logs, and select such marked log entries in my custom backed. But such entries will still get into default lager file/console backends. As far as I see there are several ways to solve this problem: 1) Add feature: custom log levels 2) Add feature to default backends: filtering by metadata Second looks easier. Any other ideas? -------------- next part -------------- An HTML attachment was scrubbed... URL: From liudanking@REDACTED Mon Apr 13 03:13:57 2015 From: liudanking@REDACTED (Daniel) Date: Mon, 13 Apr 2015 09:13:57 +0800 Subject: [erlang-questions] Erlang OTP 17.3 memory fragment problem (maybe) Message-ID: <941A7EF3-4F1A-4C55-ACDA-EF1AF22E63B6@gmail.com> I am running ejabbed 14.07 with some custom modules on a cluster consisting of 3 servers, each server is equiped with 15.75GB memory. Other server infomation is as follows: Erlang/OTP version: 17.3. OS: Ubuntu 12.04 x64 ejabberd start parameters: +K true -smp auto +P 250000 The cluster has about 2000 online users and 2000 MUC rooms everyday. The problem is that the memory usage increases with time. I have read this post: http://erlang.org/pipermail/erlang-questions/2014-April/078773.html. And I think my problem is similar to that one (BTW, Recon is really a nice tool): From *top*, it shows that beam.smp cost 3.8G memory, which is the same with recon_alloc:memory(allocated). But the recon_alloc:memory(allocated_types) shows that most of the memory is allocated to binary (about 82%): (ejabberd@REDACTED)27> recon_alloc:memory(allocated_types). [{binary_alloc,3504538080}, {driver_alloc,17498592}, {eheap_alloc,460747232}, {ets_alloc,112394720}, {fix_alloc,4391392}, {ll_alloc,36700592}, {sl_alloc,197088}, {std_alloc,1769952}, {temp_alloc,393528}] Memory usage of erlang VM is only 16.8%: (ejabberd@REDACTED)31> recon_alloc:memory(usage). 0.1680850734782954 Then I checked the VM memory fragment with recon_alloc:fragmentation(current)? (ejabberd@REDACTED)30> recon_alloc:fragmentation(current). [{{binary_alloc,2}, [{sbcs_usage,0.5666586947278912}, {mbcs_usage,0.03268338368702292}, {sbcs_block_size,341192}, {sbcs_carriers_size,602112}, {mbcs_block_size,66693536}, {mbcs_carriers_size,2040594592}]}, {{binary_alloc,1}, [{sbcs_usage,0.6408052884615385}, {mbcs_usage,0.03707295337512568}, {sbcs_block_size,341216}, {sbcs_carriers_size,532480}, {mbcs_block_size,52870816}, {mbcs_carriers_size,1426129056}]}, {{eheap_alloc,2}, [{sbcs_usage,0.966106053866951}, {mbcs_usage,0.6588223404170993}, {sbcs_block_size,247722824}, {sbcs_carriers_size,256413696}, {mbcs_block_size,66492040}, {mbcs_carriers_size,100925600}]}, {{eheap_alloc,1}, [{sbcs_usage,0.9537586503097174}, {mbcs_usage,0.8195803978109758}, {sbcs_block_size,211905456}, {sbcs_carriers_size,222179328}, {mbcs_block_size,53497304}, {mbcs_carriers_size,65274016}]}, {{ets_alloc,1}, [{sbcs_usage,1.0}, {mbcs_usage,0.9124718235517926}, {sbcs_block_size,0}, {sbcs_carriers_size,0}, {mbcs_block_size,74211640}, {mbcs_carriers_size,81330336}]}, {{ll_alloc,0}, [{sbcs_usage,1.0}, {mbcs_usage,0.7071616423963475}, {sbcs_block_size,0}, {sbcs_carriers_size,0}, {mbcs_block_size,10381280}, {mbcs_carriers_size,14680208}]}, {{ets_alloc,2}, [{sbcs_usage,1.0}, {mbcs_usage,0.879908981954333}, {sbcs_block_size,0}, {sbcs_carriers_size,0}, {mbcs_block_size,27276024}, {mbcs_carriers_size,30998688}]}, {{driver_alloc,2}, [{sbcs_usage,1.0}, {mbcs_usage,0.7534514469892656}, {sbcs_block_size,0}, {sbcs_carriers_size,0}, {mbcs_block_size,9530112}, {mbcs_carriers_size,12648608}]}, {{driver_alloc,1}, [{sbcs_usage,1.0}, {mbcs_usage,0.39700285601535695}, {sbcs_block_size,0}, {sbcs_carriers_size,0}, {mbcs_block_size,1899376}, {mbcs_carriers_size,4784288}]}, {{fix_alloc,1}, [{sbcs_usage,1.0}, {mbcs_usage,0.2179459675390966}, {sbcs_block_size,0}, {sbcs_carriers_size,0}, {mbcs_block_size,471384}, {mbcs_carriers_size,2162848}]}, {{ll_alloc,1}, [{sbcs_usage,1.0}, {mbcs_usage,0.8666539121534861}, {sbcs_block_size,0}, {sbcs_carriers_size,0}, {mbcs_block_size,5907016}, {mbcs_carriers_size,6815888}]}, {{fix_alloc,2}, [{sbcs_usage,1.0}, {mbcs_usage,0.7809480832679874}, {sbcs_block_size,0}, {sbcs_carriers_size,0}, {mbcs_block_size,1689072}, {mbcs_carriers_size,2162848}]}, {{ll_alloc,2}, [{sbcs_usage,1.0}, {mbcs_usage,0.9754967214960627}, {sbcs_block_size,0}, {sbcs_carriers_size,0}, {mbcs_block_size,14831936}, {mbcs_carriers_size,15204496}]}, {{std_alloc,1}, [{sbcs_usage,1.0}, {mbcs_usage,0.40772631122199926}, {sbcs_block_size,0}, {sbcs_carriers_size,0}, {mbcs_block_size,240552}, {mbcs_carriers_size,589984}]}, {{std_alloc,0}, [{sbcs_usage,1.0}, {mbcs_usage,0.43574063025437976}, {sbcs_block_size,0}, {sbcs_carriers_size,0}, {mbcs_block_size,257080}, {mbcs_carriers_size,589984}]}, {{std_alloc,2}, [{sbcs_usage,1.0}, {mbcs_usage,0.4444866301459023}, {sbcs_block_size,0}, {sbcs_carriers_size,0}, {mbcs_block_size,262240}, {mbcs_carriers_size,589984}]}, {{eheap_alloc,0}, [{sbcs_usage,1.0}, {mbcs_usage,0.12620470903989264}, {sbcs_block_size,0}, {sbcs_carriers_size,0}, {mbcs_block_size,33104}, {mbcs_carriers_size,262304}]}, {{temp_alloc,2}, [{sbcs_usage,1.0}, {mbcs_usage,0.0}, {sbcs_block_size,0}, {sbcs_carriers_size,0}, {mbcs_block_size,0}, {mbcs_carriers_size,131176}]}, {{temp_alloc,1}, [{sbcs_usage,1.0}, {mbcs_usage,0.0}, {sbcs_block_size,0}, {sbcs_carriers_size,0}, {mbcs_block_size,0}, {mbcs_carriers_size,131176}]}, {{temp_alloc,0}, [{sbcs_usage,1.0}, {mbcs_usage,0.0}, {sbcs_block_size,0}, {sbcs_carriers_size,0}, {mbcs_block_size,0}, {mbcs_carriers_size,...}]}, {{sl_alloc,0}, [{sbcs_usage,1.0}, {mbcs_usage,0.0}, {sbcs_block_size,0}, {sbcs_carriers_size,0}, {mbcs_block_size,...}, {...}]}, {{binary_alloc,0}, [{sbcs_usage,1.0}, {mbcs_usage,0.0017048222113979542}, {sbcs_block_size,0}, {sbcs_carriers_size,...}, {...}|...]}, {{sl_alloc,1}, [{sbcs_usage,1.0}, {mbcs_usage,0.006941061860691671}, {sbcs_block_size,...}, {...}|...]}, {{fix_alloc,0}, [{sbcs_usage,1.0},{mbcs_usage,...},{...}|...]}, {{driver_alloc,0},[{sbcs_usage,...},{...}|...]}, {{sl_alloc,2},[{...}|...]}, {{ets_alloc,...},[...]}] It shows that macs_usage of binary_alloc is very low. So my question is: Does it means that I meat a ?classic' memory fragmentation problem? If YES, how can I free the fragment memory to OS?Because my ejabberd have been killed by Linux OOM for several times. If NO, what cause the memory increase all the time? Daniel Liu From bjorn@REDACTED Mon Apr 13 08:08:38 2015 From: bjorn@REDACTED (=?UTF-8?Q?Bj=C3=B6rn_Gustavsson?=) Date: Mon, 13 Apr 2015 08:08:38 +0200 Subject: [erlang-questions] Proposal for not allowing tuple calls In-Reply-To: References: Message-ID: On Sat, Apr 11, 2015 at 5:22 PM, Jos? Valim wrote: [...] > Furthermore, erlang:apply/3 also allows tuple calls, not giving us any > alternative to disallow such behaviour, even if it explicitly. > > For those reasons, I would like to propose a new function and a compiler > option that disables the tuple calls. > > erlang:apply_no_tuple(Mod, Fun, Args). > My personal opinion is that in some future release of OTP, the plain apply/3 should *not* allow tuple modules, and that there should be some other apply-like function to allow tuple modules (e.g. apply_tuple_module/3). Similarly, there should be a compiler option to allow calls such as Mod:f() to support tuple modules. That is, in that future release, those who want to use tuple modules will have to to opt in. /Bj?rn -- Bj?rn Gustavsson, Erlang/OTP, Ericsson AB From vances@REDACTED Mon Apr 13 08:57:21 2015 From: vances@REDACTED (Vance Shipley) Date: Mon, 13 Apr 2015 12:27:21 +0530 Subject: [erlang-questions] Efficient sum matrix by column In-Reply-To: References: Message-ID: On Mon, Apr 13, 2015 at 1:33 AM, Alexander Petrovsky wrote: > Every element in proplist it's number of requests from client for 1 second So conceptually a table where each row is indexed by client with columns having per second totals for various data points. > Every second I want to recalculate summary rate per client per cluster Every second, at each node, for each row, hash the client to select a node (e.g. ClientNum rem NumNodes = NodeNum). Send each node just it's client data. At each node sum the data received from all nodes and send the result to the collecting node. At the collecting node receive the summary data from all nodes and reassemble into the summary table. On Sun, Apr 12, 2015 at 10:53 PM, Alexander Petrovsky wrote: > I have about 100 nodes in cluster, each node in cluster contains about > 100000 elements in proplist. All proplists have equal length, the same keys, > but different values. I need to get the sum values for every key between all > proplists. By example, it can be represented as matrix NxM (N~=100, > M~=100000), and I need to get the sum by column. -- -Vance From garazdawi@REDACTED Mon Apr 13 09:28:31 2015 From: garazdawi@REDACTED (Lukas Larsson) Date: Mon, 13 Apr 2015 09:28:31 +0200 Subject: [erlang-questions] Erlang OTP 17.3 memory fragment problem (maybe) In-Reply-To: <941A7EF3-4F1A-4C55-ACDA-EF1AF22E63B6@gmail.com> References: <941A7EF3-4F1A-4C55-ACDA-EF1AF22E63B6@gmail.com> Message-ID: Hello Daniel, On Mon, Apr 13, 2015 at 3:13 AM, Daniel > > > So my question is: > Does it means that I meat a ?classic' memory fragmentation problem? If > YES, how can I free the fragment memory to OS?Because my ejabberd have been > killed by Linux OOM for several times. If NO, what cause the memory > increase all the time? Would you mind checking how much of the binary_alloc memory is part of the mbc pool? ("erlang:system_info({allocator,binary_alloc}).", look for mbcs_pool). If it is very large you may have hit the bug described in OTP-12323, which has been fixed in Erlang/OTP 17.4. The scenario you are describing where the memory utilization constantly decreases sound very similar to what the bug fixed. If you don't want to upgrade to 17.4, you can also disable the pool in 17.3. Lukas -------------- next part -------------- An HTML attachment was scrubbed... URL: From mjtruog@REDACTED Mon Apr 13 09:55:10 2015 From: mjtruog@REDACTED (Michael Truog) Date: Mon, 13 Apr 2015 00:55:10 -0700 Subject: [erlang-questions] [ANN] CloudI 1.5.0 Released! Message-ID: <552B765E.8070003@gmail.com> Download 1.5.0 from http://sourceforge.net/projects/cloudi/files/latest/download (checksums at the bottom of this email) CloudI (http://cloudi.org/) is a "universal integrator" using an Erlang core to provide fault-tolerance with efficiency and scalability. The CloudI API provides a minimal interface to communicate among services so programming language agnostic, database agnostic, and messaging bus agnostic integration can occur. CloudI currently integrates with the programming languages C/C++, Elixir, Erlang, Java, JavaScript, PHP, Perl, Python, and Ruby, the databases PostgreSQL, elasticsearch, Cassandra, MySQL, couchdb, memcached, riak, and tokyotyrant, the messaging bus ZeroMQ, websockets, and the internal CloudI service bus. HTTP is supported with both cowboy and elli integration. The details for this release are below: * backwards compatibility difference: * cloudi module (Erlang/Elixir-only usage) now returns a 2 element tuple with the 1st element as the result (original return value) and the 2nd element as the modified Context data (or the Dispatcher pid if a Dispatcher pid was provided instead of a Context) * Was a necessary change to keep the transaction id time as strictly monotonically increasing with the changes in Erlang 18.x that deprecate erlang:now/0 * Allows a lazy destination refresh to update the service name lookup data so the developer doesn't need to handle the update manually * The change impacted the return value of internal service external interface functions * The service application was changed to depend only on the cloudi_service module instead of the cloudi module to make its usage clearer and avoid Dispatcher pid value handling * Despite the return value changes, only the minor version number of CloudI was incremented due to the interfaces being outside the main documentation * External services can now set OS process resource limits with the 'limit' service configuration option (http://cloudi.org/api.html#2_services_add_config_opts_limit) * A few new CloudI data structures: cloudi_queue, cloudi_future, cloudi_service_future (cloudi_queue can handle service request validation, failures, and retries with in-memory internal service data) (cloudi_future and cloudi_service_future provide Futures for Erlang processes (sending to CloudI services) and internal CloudI services, respectively) * cloudi_service_validate is a new CloudI service for validating service requests with the requests using the service as a proxy (similar functionality was added to cloudi_service_quorum and cloudi_service_router, though cloudi_service_router is forwarding the service request instead of sending it) * The Python CloudI API now handles unicode stdout/stderr data properly (immediate flushing of stdout/stderr is now fixed for Python 3 usage) * Bugs were fixed and more documentation was added (see the ChangeLog for more detail) Please mention any problems, issues, or ideas! Thanks, Michael SHA256 CHECKSUMS 0ba7174780b6a0b699d909bc41ddc0d5596cd027539e5d9a271ff3250b60dd9c cloudi-1.5.0.tar.bz2 (10960 bytes) 94fdebeb7406bc360f6858e08c846ab6c8bd8707a8597da00b61f513d9ea99df cloudi-1.5.0.tar.gz (13616 bytes) From garazdawi@REDACTED Mon Apr 13 10:59:18 2015 From: garazdawi@REDACTED (Lukas Larsson) Date: Mon, 13 Apr 2015 10:59:18 +0200 Subject: [erlang-questions] Erlang OTP 17.3 memory fragment problem (maybe) In-Reply-To: <2C0946BB-CB18-4CCD-97F8-FCD0348E3764@gmail.com> References: <941A7EF3-4F1A-4C55-ACDA-EF1AF22E63B6@gmail.com> <2C0946BB-CB18-4CCD-97F8-FCD0348E3764@gmail.com> Message-ID: On Mon, Apr 13, 2015 at 10:43 AM, Daniel wrote: > > > On Apr 13, 2015, at 3:28 PM, Lukas Larsson wrote: > > > > Hello Daniel, > > > > On Mon, Apr 13, 2015 at 3:13 AM, Daniel > > > So my question is: > > Does it means that I meat a ?classic' memory fragmentation problem? If > YES, how can I free the fragment memory to OS?Because my ejabberd have been > killed by Linux OOM for several times. If NO, what cause the memory > increase all the time? > > > > Would you mind checking how much of the binary_alloc memory is part of > the mbc pool? ("erlang:system_info({allocator,binary_alloc}).", look for > mbcs_pool). If it is very large you may have hit the bug described in > OTP-12323, which has been fixed in Erlang/OTP 17.4. The scenario you are > describing where the memory utilization constantly decreases sound very > similar to what the bug fixed. If you don't want to upgrade to 17.4, you > can also disable the pool in 17.3. > > > > Lukas > > Hi, Dear Lukas and list, > > Currently, linux *top* shows that beam.smp reserves 6.5GB memory , > recon_alloc:memory(allocated) outputs 6.5GB and recon_alloc:memory(usage) > outputs 0.0969. > > Bellow is my binary_alloc memory of mbcs_pool: > > <<< -- cut -- >>> > {mbcs_pool,[{blocks,52044}, > {blocks_size,22538272}, > {carriers,115}, > {carriers_size,792723456}]}, > <<< -- cut -- >>> > {mbcs_pool,[{blocks,139776}, > {blocks_size,35429768}, > {carriers,752}, > {carriers_size,5443158016}]}, <<< -- cut -- >>> > mbcs_pool cost (5443158016 + 792723456) bytes (5.8GB) memory. >From this data I'm pretty sure that you are hitting the bug I mentioned. So either upgrade to 17.4 or disable the pool (+Muacul 0) and your issue should go away. For a detailed description of the problem see this: https://github.com/erlang/otp/blob/master/erts/emulator/internal_doc/CarrierMigration.md#searching-the-pool -------------- next part -------------- An HTML attachment was scrubbed... URL: From liudanking@REDACTED Mon Apr 13 10:43:41 2015 From: liudanking@REDACTED (Daniel) Date: Mon, 13 Apr 2015 16:43:41 +0800 Subject: [erlang-questions] Erlang OTP 17.3 memory fragment problem (maybe) In-Reply-To: References: <941A7EF3-4F1A-4C55-ACDA-EF1AF22E63B6@gmail.com> Message-ID: <2C0946BB-CB18-4CCD-97F8-FCD0348E3764@gmail.com> > On Apr 13, 2015, at 3:28 PM, Lukas Larsson wrote: > > Hello Daniel, > > On Mon, Apr 13, 2015 at 3:13 AM, Daniel > So my question is: > Does it means that I meat a ?classic' memory fragmentation problem? If YES, how can I free the fragment memory to OS?Because my ejabberd have been killed by Linux OOM for several times. If NO, what cause the memory increase all the time? > > Would you mind checking how much of the binary_alloc memory is part of the mbc pool? ("erlang:system_info({allocator,binary_alloc}).", look for mbcs_pool). If it is very large you may have hit the bug described in OTP-12323, which has been fixed in Erlang/OTP 17.4. The scenario you are describing where the memory utilization constantly decreases sound very similar to what the bug fixed. If you don't want to upgrade to 17.4, you can also disable the pool in 17.3. > > Lukas Hi, Dear Lukas and list, Currently, linux *top* shows that beam.smp reserves 6.5GB memory , recon_alloc:memory(allocated) outputs 6.5GB and recon_alloc:memory(usage) outputs 0.0969. Bellow is my binary_alloc memory of mbcs_pool: (ejabberd@REDACTED)8> erlang:system_info({allocator,binary_alloc}). [{instance,0, [{versions,"0.9","3.0"}, {options,[{e,true}, {t,true}, {ramv,false}, {sbct,524288}, {asbcst,4145152}, {rsbcst,20}, {rsbcmt,80}, {rmbcmt,50}, {mmbcs,65536}, {mmmbc,18446744073709551615}, {mmsbc,256}, {lmbcs,5242880}, {smbcs,524288}, {mbcgs,10}, {acul,0}, {as,aoffcbf}]}, {mbcs,[{blocks,4,4,95}, {blocks_size,84208,84208,607944}, {carriers,2,2,3}, {mseg_alloc_carriers,1}, {sys_alloc_carriers,1}, {carriers_size,589984,589984,1638560}, {mseg_alloc_carriers_size,524288}, {sys_alloc_carriers_size,65696}]}, {sbcs,[{blocks,0,0,0}, {blocks_size,0,0,0}, {carriers,0,0,0}, {mseg_alloc_carriers,0}, {sys_alloc_carriers,0}, {carriers_size,0,0,0}, {mseg_alloc_carriers_size,0}, {sys_alloc_carriers_size,0}]}, {calls,[{binary_alloc,0,11582}, {binary_free,0,11578}, {binary_realloc,0,0}, {mseg_alloc,0,23}, {mseg_dealloc,0,22}, {mseg_realloc,0,0}, {sys_alloc,0,1}, {sys_free,0,0}, {sys_realloc,0,0}]}]}, {instance,1, [{versions,"0.9","3.0"}, {options,[{e,true}, {t,true}, {ramv,false}, {sbct,524288}, {asbcst,4145152}, {rsbcst,20}, {rsbcmt,80}, {rmbcmt,50}, {mmbcs,65536}, {mmmbc,18446744073709551615}, {mmsbc,256}, {lmbcs,5242880}, {smbcs,524288}, {mbcgs,10}, {acul,60}, {as,aoffcbf}]}, {mbcs,[{blocks,91398,118957,1072611}, {blocks_size,19822160,25965328,154167640}, {carriers,7,10,29}, {mseg_alloc_carriers,6}, {sys_alloc_carriers,1}, {carriers_size,29425824,46203040,205586592}, {mseg_alloc_carriers_size,29360128}, {sys_alloc_carriers_size,65696}]}, {mbcs_pool,[{blocks,52044}, {blocks_size,22538272}, {carriers,115}, {carriers_size,792723456}]}, {sbcs,[{blocks,0,0,13}, {blocks_size,0,0,4312368}, {carriers,0,0,13}, {mseg_alloc_carriers,0}, {sys_alloc_carriers,0}, {carriers_size,0,0,7483392}, {mseg_alloc_carriers_size,0}, {sys_alloc_carriers_size,0}]}, {calls,[{binary_alloc,1,808658319}, {binary_free,1,808237034}, {binary_realloc,0,660820827}, {mseg_alloc,0,25140}, {mseg_dealloc,0,24698}, {mseg_realloc,0,22983}, {sys_alloc,0,1}, {sys_free,0,0}, {sys_realloc,0,0}]}]}, {instance,2, [{versions,"0.9","3.0"}, {options,[{e,true}, {t,true}, {ramv,false}, {sbct,524288}, {asbcst,4145152}, {rsbcst,20}, {rsbcmt,80}, {rmbcmt,50}, {mmbcs,65536}, {mmmbc,18446744073709551615}, {mmsbc,256}, {lmbcs,5242880}, {smbcs,524288}, {mbcgs,10}, {acul,60}, {as,aoffcbf}]}, {mbcs,[{blocks,16137,16819,1068983}, {blocks_size,5291776,5983792,151274536}, {carriers,4,4,28}, {mseg_alloc_carriers,3}, {sys_alloc_carriers,1}, {carriers_size,13697184,13697184,198246560}, {mseg_alloc_carriers_size,13631488}, {sys_alloc_carriers_size,65696}]}, {mbcs_pool,[{blocks,139776}, {blocks_size,35429768}, {carriers,752}, {carriers_size,5443158016}]}, {sbcs,[{blocks,0,0,13}, {blocks_size,0,0,4526120}, {carriers,0,0,13}, {mseg_alloc_carriers,0}, {sys_alloc_carriers,0}, {carriers_size,0,0,7524352}, {mseg_alloc_carriers_size,0}, {sys_alloc_carriers_size,0}]}, {calls,[{binary_alloc,1,757135985}, {binary_free,1,757257915}, {binary_realloc,0,646773973}, {mseg_alloc,0,24913}, {mseg_dealloc,0,24479}, {mseg_realloc,0,22617}, {sys_alloc,0,1}, {sys_free,0,0}, {sys_realloc,0,0}]}]}] mbcs_pool cost (5443158016 + 792723456) bytes (5.8GB) memory. From liudanking@REDACTED Mon Apr 13 11:18:40 2015 From: liudanking@REDACTED (Daniel) Date: Mon, 13 Apr 2015 17:18:40 +0800 Subject: [erlang-questions] Erlang OTP 17.3 memory fragment problem (maybe) In-Reply-To: References: <941A7EF3-4F1A-4C55-ACDA-EF1AF22E63B6@gmail.com> <2C0946BB-CB18-4CCD-97F8-FCD0348E3764@gmail.com> Message-ID: <3875FB52-1AD5-4794-AB43-03E6FD06BDAC@gmail.com> > On Apr 13, 2015, at 4:59 PM, Lukas Larsson wrote: > > > > On Mon, Apr 13, 2015 at 10:43 AM, Daniel wrote: > > > On Apr 13, 2015, at 3:28 PM, Lukas Larsson wrote: > > > > Hello Daniel, > > > > On Mon, Apr 13, 2015 at 3:13 AM, Daniel > > > So my question is: > > Does it means that I meat a ?classic' memory fragmentation problem? If YES, how can I free the fragment memory to OS?Because my ejabberd have been killed by Linux OOM for several times. If NO, what cause the memory increase all the time? > > > > Would you mind checking how much of the binary_alloc memory is part of the mbc pool? ("erlang:system_info({allocator,binary_alloc}).", look for mbcs_pool). If it is very large you may have hit the bug described in OTP-12323, which has been fixed in Erlang/OTP 17.4. The scenario you are describing where the memory utilization constantly decreases sound very similar to what the bug fixed. If you don't want to upgrade to 17.4, you can also disable the pool in 17.3. > > > > Lukas > > Hi, Dear Lukas and list, > > Currently, linux *top* shows that beam.smp reserves 6.5GB memory , recon_alloc:memory(allocated) outputs 6.5GB and recon_alloc:memory(usage) outputs 0.0969. > > Bellow is my binary_alloc memory of mbcs_pool: > > <<< -- cut -- >>> > {mbcs_pool,[{blocks,52044}, > {blocks_size,22538272}, > {carriers,115}, > {carriers_size,792723456}]}, > <<< -- cut -- >>> > {mbcs_pool,[{blocks,139776}, > {blocks_size,35429768}, > {carriers,752}, > {carriers_size,5443158016}]}, > <<< -- cut -- >>> > mbcs_pool cost (5443158016 + 792723456) bytes (5.8GB) memory. > > From this data I'm pretty sure that you are hitting the bug I mentioned. So either upgrade to 17.4 or disable the pool (+Muacul 0) and your issue should go away. For a detailed description of the problem see this: https://github.com/erlang/otp/blob/master/erts/emulator/internal_doc/CarrierMigration.md#searching-the-pool > I will try +Muacul 0 first, and give my feedback in this week after several days obsersation. From roger@REDACTED Mon Apr 13 17:27:50 2015 From: roger@REDACTED (Roger Lipscombe) Date: Mon, 13 Apr 2015 16:27:50 +0100 Subject: [erlang-questions] custom lager levels or filtering In-Reply-To: References: Message-ID: On 12 April 2015 at 23:37, Vladimir Gordeev wrote: > Let me describe my problem. I have two different kinds of logging inside my > code: regular console/file output and special log calls that should be > processed by my log backend. By their meaning and usage they are completely > separate. If you've got completely separate logging streams, why not use completely separate logging systems? That is: use lager for general stuff, and something else (possibly custom) for the specific stuff. From z@REDACTED Mon Apr 13 18:11:12 2015 From: z@REDACTED (Danil Zagoskin) Date: Mon, 13 Apr 2015 19:11:12 +0300 Subject: [erlang-questions] custom lager levels or filtering In-Reply-To: References: Message-ID: Lager has so-called tracing. With tracing you can route messages by their metadata. This feature use is counter-intuitive, you should find some examples of its use (my example: https://github.com/stolen/erdico/blob/e71870c6bf045364cf3250772271f49a13352b5e/erdico.config ) I don't know if it is possible to completely separate traced events from ordinary handlers, i just use lower severity for traced messages than for configured global logs. On Mon, Apr 13, 2015 at 1:37 AM, Vladimir Gordeev < gordeev.vladimir.v@REDACTED> wrote: > Let me describe my problem. I have two different kinds of logging inside > my code: regular console/file output and special log calls that should be > processed by my log backend. By their meaning and usage they are completely > separate. > > Semantically both logs actually are info level, but still they should be > handled separately. I can add metadata to my specific logs, and select such > marked log entries in my custom backed. > > But such entries will still get into default lager file/console backends. > > As far as I see there are several ways to solve this problem: > > 1) Add feature: custom log levels > 2) Add feature to default backends: filtering by metadata > > Second looks easier. Any other ideas? > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -- Danil Zagoskin | z@REDACTED -------------- next part -------------- An HTML attachment was scrubbed... URL: From t@REDACTED Mon Apr 13 19:38:21 2015 From: t@REDACTED (Tristan Sloughter) Date: Mon, 13 Apr 2015 12:38:21 -0500 Subject: [erlang-questions] Escript using old i/o server still? Message-ID: <1428946701.3514444.253161877.2800254B@webmail.messagingengine.com> Attempting to accept a password input from an escript led me to discover this issue from 2009 is still the case: http://erlang.org/pipermail/erlang-questions/2009-April/043311.html Is there any chance escript will get some love in the near future? -- Tristan Sloughter t@REDACTED From tandy.nilsson@REDACTED Mon Apr 13 21:06:05 2015 From: tandy.nilsson@REDACTED (Marcus Nilsson) Date: Mon, 13 Apr 2015 21:06:05 +0200 Subject: [erlang-questions] Private key encryption Message-ID: I'am trying to encrypt a bigger file with the public_key module. Everything works fine as long as the content of the file is small. But when the size of the binary exceed's a certain size I get a **error:encrypt_failed. I guess this has to do with the padding parameter. But I have not been able to find any documentation how to compute the padding to get this working any help would be very welcomed! I use this code to perform the encryption public_key:encrypt_private(Input, PrivKey); /Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From martink@REDACTED Tue Apr 14 00:54:38 2015 From: martink@REDACTED (Martin Karlsson) Date: Tue, 14 Apr 2015 10:54:38 +1200 Subject: [erlang-questions] Private key encryption In-Reply-To: References: Message-ID: Hi Marcus, The encrypt_private is doing an RSA encryption using PKCS1 padding by default. RSA can't encrypt large payloads (i.e. 256 bytes - 11 for padding for 2048 bit RSA keys) so this is the likely reason you can only encrypt small portions of the file. Normally you use public key crypto to encrypt a symmetrical key and then encrypt the large payload with the symmetric key using AES or something. In addition you don't want to encrypt using the private key but rather the public key, otherwise anyone with access to your public key can decrypt the cipher. Private key encryption is usually only used for signatures. Crypto is hard to get right, especially if you are only working with RSA primitives (you need to think about padding, hashing, MDCs, signatures). You might want to have a look a NaCl (https://github.com/jloius/enacl for a binding to erlang) which is much friendlier to use. Cheers, Martin On Tuesday, 14 April 2015, Marcus Nilsson wrote: > I'am trying to encrypt a bigger file with the public_key module. > Everything works fine as long as the content of the file is small. > > But when the size of the binary exceed's a certain size I get a > **error:encrypt_failed. I guess this has to do with the padding parameter. > > But I have not been able to find any documentation how to compute the > padding to get this working any help would be very welcomed! > > I use this code to perform the encryption > > public_key:encrypt_private(Input, PrivKey); > > /Marcus > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex@REDACTED Tue Apr 14 04:35:57 2015 From: alex@REDACTED (Alex Wilson) Date: Tue, 14 Apr 2015 12:35:57 +1000 Subject: [erlang-questions] Escript using old i/o server still? In-Reply-To: <1428946701.3514444.253161877.2800254B@webmail.messagingengine.com> References: <1428946701.3514444.253161877.2800254B@webmail.messagingengine.com> Message-ID: <1428978957.16798.4.camel@cooperi.net> On Mon, 2015-04-13 at 12:38 -0500, Tristan Sloughter wrote: > Attempting to accept a password input from an escript led me to discover > this issue from 2009 is still the case: Changing this behaviour is actually fairly complicated, if you want to do it without compromising the ability of escripts to accept input from non-terminal sources (eg from a shell pipeline). So I'm not sure it's going to happen any time soon... If you're desperate you can use a really horrible workaround, something like this: -define(OP_PUTC,0). getpw() -> case io:setopts([binary, {echo, false}]) of ok -> PwLine = io:get_line(<<"Password:">>), ok = io:setopts([binary, {echo, true}]), io:format("\n"), [Pw | _] = binary:split(PwLine, <<"\n">>), Pw; _ -> Port = open_port({spawn, 'tty_sl -e'}, [binary, eof]), port_command(Port, <>), receive {Port, {data, PwLine}} -> [Pw | _] = binary:split(PwLine, <<"\n">>), port_command(Port, <>), port_close(Port), Pw end end. This uses the tty_sl port driver that's only for internal use inside the new I/O server, which is private API and could disappear at any time. However, in practice it's been very stable since it was introduced, so... It should be safe for tty_sl to just temporarily hijack the terminal like this as long as nothing else in your program causes the old I/O server to talk to it at the same time. If there are any other processes running that could cause io: functions to be called during this, things could possibly get hairy. I'd only recommend it from an escript main/1 or similar, before starting any other processes, and probably after turning off the error_logger or putting it in silent mode. From t@REDACTED Tue Apr 14 05:40:46 2015 From: t@REDACTED (Tristan Sloughter) Date: Mon, 13 Apr 2015 22:40:46 -0500 Subject: [erlang-questions] Escript using old i/o server still? In-Reply-To: <1428978957.16798.4.camel@cooperi.net> References: <1428946701.3514444.253161877.2800254B@webmail.messagingengine.com> <1428978957.16798.4.camel@cooperi.net> Message-ID: <1428982846.55839.253375053.688AC633@webmail.messagingengine.com> Ah, damn. But this hack works. I take it there isn't a Windows equivalent for this hack? Otherwise I'll use the hack they used in the Hex client: https://github.com/hexpm/hex/blob/3e3abc78f333162fb968b744d583d04b2eb3aeae/lib/mix/tasks/hex/util.ex#L34 Thanks, Tristan Sloughter From chandrashekhar.mullaparthi@REDACTED Tue Apr 14 09:29:04 2015 From: chandrashekhar.mullaparthi@REDACTED (Chandru) Date: Tue, 14 Apr 2015 08:29:04 +0100 Subject: [erlang-questions] Amazon API -- Lookup by ISBN In-Reply-To: <1428526022.03741188@apps.rackspace.com> References: <1428526022.03741188@apps.rackspace.com> Message-ID: On 8 April 2015 at 21:47, wrote: > Hello, > > I'm striving to look up books in Amazon's db by ISBN. At first blush it > looks easy enough: > > > http://docs.aws.amazon.com/AWSECommerceService/latest/DG/EX_LookupbyISBN.html > > But the last item, Signature, baffles me. Procedure here: > > > http://docs.aws.amazon.com/AWSECommerceService/latest/DG/rest-signature.html > > I'm fine with this until I hit step 4: > > -- Sort parameter/value pairs by byte value --- I can see how to do this > manually, but don't know how put Erlang to the task > > And I'm really stumped when I hit step 8: > > -- Calculate an RFC 2104-compliant HMAC with the SHA256 hash algorithm > > Any help? Better yet, does anyone have actual code to make such requests > they're willing to share? > > Sorry, quite a late reply here, but better late than never I guess. Here is some sample code I dug out of one of my pet projects. sign_amazon_aws_req(Method, Params) -> {ok, {AWS_Access_id, Secret_key}} = application:get_env(my_app, aws_access_credentials), Boiler_plate_params = ["Service=AWSECommerceService", "AWSAccessKeyId=" ++ AWS_Access_id, "AssociateTag=my_associate_tag", "Timestamp=" ++ http_uri:encode(aws_timestamp()) ], Params_1 = Params ++ Boiler_plate_params, Req_params = lists:flatten(string:join(lists:sort(Params_1), "&")), Rest_endpoint = amazon_rest_endpoint(), Rest_path = "/onca/xml", String_to_sign = [Method, "\n", Rest_endpoint, "\n", Rest_path, "\n", Req_params], Signature = http_uri:encode( base64:encode_to_string( crypto:hmac(sha256, Secret_key, String_to_sign))), lists:flatten(["https://" ++ Rest_endpoint ++ Rest_path ++ "?", Req_params, "&Signature=", Signature]). amazon_rest_endpoint() -> "webservices.amazon.co.uk". aws_timestamp() -> {{Y, M, D}, {H, Mi, S}} = calendar:universal_time(), lists:flatten(io_lib:format("~p-~s-~sT~s:~s:~sZ", [Y, two_digits(M), two_digits(D), two_digits(H), two_digits(Mi), two_digits(S)])). two_digits(X) when X < 10 -> [$0 | integer_to_list(X)]; two_digits(X) when is_integer(X) -> integer_to_list(X); two_digits([X]) -> [$0, X]; two_digits(X) -> X. -------------- next part -------------- An HTML attachment was scrubbed... URL: From gguthrie@REDACTED Tue Apr 14 09:53:27 2015 From: gguthrie@REDACTED (Gordon Guthrie) Date: Tue, 14 Apr 2015 08:53:27 +0100 Subject: [erlang-questions] Amazon API -- Lookup by ISBN In-Reply-To: References: <1428526022.03741188@apps.rackspace.com> Message-ID: <31338D70-45BB-4D61-87DF-BAEE88D1AE52@basho.com> There?s a pretty full clone of the Erlang AWS library in mochi that has a set of unit tests taken from the AWS documentation that you might want to use https://github.com/mochi/mochiweb/tree/master/examples/hmac_api Beware though - the AWS canonically rewrites the URL?s a little bit before signing them so this won?t work out of the box. G > Le 14 avr. 2015 ? 08:29, Chandru a ?crit : > > On 8 April 2015 at 21:47, > wrote: > Hello, > > I'm striving to look up books in Amazon's db by ISBN. At first blush it looks easy enough: > > http://docs.aws.amazon.com/AWSECommerceService/latest/DG/EX_LookupbyISBN.html > > But the last item, Signature, baffles me. Procedure here: > > http://docs.aws.amazon.com/AWSECommerceService/latest/DG/rest-signature.html > > I'm fine with this until I hit step 4: > > -- Sort parameter/value pairs by byte value --- I can see how to do this manually, but don't know how put Erlang to the task > > And I'm really stumped when I hit step 8: > > -- Calculate an RFC 2104-compliant HMAC with the SHA256 hash algorithm > > Any help? Better yet, does anyone have actual code to make such requests they're willing to share? > > > Sorry, quite a late reply here, but better late than never I guess. Here is some sample code I dug out of one of my pet projects. > > sign_amazon_aws_req(Method, Params) -> > {ok, {AWS_Access_id, Secret_key}} = application:get_env(my_app, aws_access_credentials), > > Boiler_plate_params = > ["Service=AWSECommerceService", > "AWSAccessKeyId=" ++ AWS_Access_id, > "AssociateTag=my_associate_tag", > "Timestamp=" ++ http_uri:encode(aws_timestamp()) > ], > Params_1 = Params ++ Boiler_plate_params, > Req_params = lists:flatten(string:join(lists:sort(Params_1), "&")), > Rest_endpoint = amazon_rest_endpoint(), > Rest_path = "/onca/xml", > String_to_sign = [Method, "\n", > Rest_endpoint, "\n", > Rest_path, "\n", > Req_params], > Signature = http_uri:encode( > base64:encode_to_string( > crypto:hmac(sha256, Secret_key, String_to_sign))), > lists:flatten(["https://" ++ Rest_endpoint ++ Rest_path ++ "?", > Req_params, > "&Signature=", Signature]). > > amazon_rest_endpoint() -> "webservices.amazon.co.uk ". > > aws_timestamp() -> > {{Y, M, D}, {H, Mi, S}} = calendar:universal_time(), > lists:flatten(io_lib:format("~p-~s-~sT~s:~s:~sZ", > [Y, two_digits(M), two_digits(D), > two_digits(H), two_digits(Mi), two_digits(S)])). > > two_digits(X) when X < 10 -> [$0 | integer_to_list(X)]; > two_digits(X) when is_integer(X) -> integer_to_list(X); > two_digits([X]) -> [$0, X]; > two_digits(X) -> X. > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From llbgurs@REDACTED Tue Apr 14 10:26:06 2015 From: llbgurs@REDACTED (linbo liao) Date: Tue, 14 Apr 2015 16:26:06 +0800 Subject: [erlang-questions] poolboy new_worker failed cause poolboy crashed Message-ID: Hi, I open an issue for poolboy (https://github.com/devinus/poolboy/issues/72), but looks author doesn't think this is an issue. Looks like to solve this issue have two ways: 1. Modify eredis client, start_link still return ok when connect failed, and when use it should check eredis socket alive or not 2. Modify poolboy library, protected all logic when call new_worker failed. But it require change ereids or poolboy code, I was wondering to know anyone hit this issue, how to resolve it? Thanks, Linbo -------------- next part -------------- An HTML attachment was scrubbed... URL: From be.dmitry@REDACTED Tue Apr 14 10:40:26 2015 From: be.dmitry@REDACTED (Dmitry Belyaev) Date: Tue, 14 Apr 2015 18:40:26 +1000 Subject: [erlang-questions] Proposal for not allowing tuple calls In-Reply-To: References: Message-ID: Are there any plans to introduce a 'module' type in erlang? That would allow to get rid of that abomination called tuple calls and introduce a correct implementation of parameterized modules. Exactly the same way as lambdas capturing context values were introduced. -- Best wishes, Dmitry Belyaev On 13 April 2015 4:08:38 PM AEST, "Bj?rn Gustavsson" wrote: >On Sat, Apr 11, 2015 at 5:22 PM, Jos? Valim > wrote: >[...] >> Furthermore, erlang:apply/3 also allows tuple calls, not giving us >any >> alternative to disallow such behaviour, even if it explicitly. >> >> For those reasons, I would like to propose a new function and a >compiler >> option that disables the tuple calls. >> >> erlang:apply_no_tuple(Mod, Fun, Args). >> > >My personal opinion is that in some future release >of OTP, the plain apply/3 should *not* allow tuple modules, >and that there should be some other apply-like function >to allow tuple modules (e.g. apply_tuple_module/3). >Similarly, there should be a compiler option to allow >calls such as Mod:f() to support tuple modules. > >That is, in that future release, those who want to use tuple >modules will have to to opt in. > >/Bj?rn > >-- >Bj?rn Gustavsson, Erlang/OTP, Ericsson AB >_______________________________________________ >erlang-questions mailing list >erlang-questions@REDACTED >http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From jose.valim@REDACTED Tue Apr 14 10:49:12 2015 From: jose.valim@REDACTED (=?UTF-8?Q?Jos=C3=A9_Valim?=) Date: Tue, 14 Apr 2015 10:49:12 +0200 Subject: [erlang-questions] poolboy new_worker failed cause poolboy crashed In-Reply-To: References: Message-ID: You don't need to modify eredis per se. You can create a worker that is between poolboy and eredis that tracks the connected/disconnected semantics as Devin described. Unfortunately I don't know of any sample Erlang code but, if you don't mind, here is a similar solution using redo written in Elixir: https://github.com/phoenixframework/phoenix/blob/master/lib/phoenix/pubsub/redis_conn.ex To use it, you checkout the worker above from the pool, ask the redis connection to the worker and do the operations you want to in the connection as usual. As a note, I did find it a bit weird that eredis handles disconnects automatically but it does not handle the case where it can't connect to Redis on start_link. It would be handful if they supported lazy connections (i.e. don't connect on start_link but on first command). They already have all the code to handle this, it is just a matter of supporting it on start_link too. *Jos? Valim* www.plataformatec.com.br Skype: jv.ptec Founder and Lead Developer On Tue, Apr 14, 2015 at 10:26 AM, linbo liao wrote: > Hi, > > I open an issue for poolboy (https://github.com/devinus/poolboy/issues/72), > but looks author doesn't think this is an issue. > > Looks like to solve this issue have two ways: > > 1. Modify eredis client, start_link still return ok when connect failed, > and when use it should check eredis socket alive or not > > 2. Modify poolboy library, protected all logic when call new_worker failed. > > But it require change ereids or poolboy code, I was wondering to know > anyone hit this issue, how to resolve it? > > Thanks, > Linbo > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jon@REDACTED Tue Apr 14 11:25:59 2015 From: jon@REDACTED (Jon Schneider) Date: Tue, 14 Apr 2015 10:25:59 +0100 Subject: [erlang-questions] Efficient sum matrix by column In-Reply-To: <20150412205311.65c26989@gmail.com> References: <20150412205311.65c26989@gmail.com> Message-ID: <44fc4c24853624eba502444b7bf9feb5.squirrel@mail.jschneider.net> Surely if we're talking efficiency then use a library written for the purpose. This is standard matrix manipulation surely ? Assuming we're talking PC type hardware there are instructions that do this kind of thing with a speed-up of some factor of 2. Jon > Hello Alexander Petrovsky! > On Sun, 12 Apr 2015 21:23:09 +0400 you wrote: > >> Hello! >> >> I have about 100 nodes in cluster, each node in cluster contains about >> 100000 elements in proplist. All proplists have equal length, the same >> keys, but different values. I need to get the sum values for every key >> between all proplists. By example, it can be represented as matrix NxM >> (N~=100, M~=100000), and I need to get the sum by column. >> >> The problem is that it's too slow, even when I use the hackish way >> with ets:update_counter and N=2, it's take about 1.5 secs, If I make >> in parallel, it's take about 2-3 seconds. >> >> How can I make it fast and efficient? >> > > Look at lists:keyfind/4 It is much faster than proplists. > > -- > Best regards! > gfborn [at] gmail [dot] com > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > From askjuise@REDACTED Tue Apr 14 15:38:33 2015 From: askjuise@REDACTED (Alexander Petrovsky) Date: Tue, 14 Apr 2015 14:38:33 +0100 Subject: [erlang-questions] Efficient sum matrix by column In-Reply-To: <44fc4c24853624eba502444b7bf9feb5.squirrel@mail.jschneider.net> References: <20150412205311.65c26989@gmail.com> <44fc4c24853624eba502444b7bf9feb5.squirrel@mail.jschneider.net> Message-ID: I found a way that give enough performance for me. I'm just get rid of proplists, and start using tuples as pointed ILYA. The simple sum by column of two tuples with 100000 in it, take about 5ms. Thaks a lot for help, folks! 2015-04-14 10:25 GMT+01:00 Jon Schneider : > Surely if we're talking efficiency then use a library written for the > purpose. This is standard matrix manipulation surely ? > > Assuming we're talking PC type hardware there are instructions that do > this kind of thing with a speed-up of some factor of 2. > > Jon > > > > Hello Alexander Petrovsky! > > On Sun, 12 Apr 2015 21:23:09 +0400 you wrote: > > > >> Hello! > >> > >> I have about 100 nodes in cluster, each node in cluster contains about > >> 100000 elements in proplist. All proplists have equal length, the same > >> keys, but different values. I need to get the sum values for every key > >> between all proplists. By example, it can be represented as matrix NxM > >> (N~=100, M~=100000), and I need to get the sum by column. > >> > >> The problem is that it's too slow, even when I use the hackish way > >> with ets:update_counter and N=2, it's take about 1.5 secs, If I make > >> in parallel, it's take about 2-3 seconds. > >> > >> How can I make it fast and efficient? > >> > > > > Look at lists:keyfind/4 It is much faster than proplists. > > > > -- > > Best regards! > > gfborn [at] gmail [dot] com > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- ?????????? ????????? / Alexander Petrovsky, Skype: askjuise Phone: +7 914 8 820 815 -------------- next part -------------- An HTML attachment was scrubbed... URL: From tandy.nilsson@REDACTED Tue Apr 14 15:54:27 2015 From: tandy.nilsson@REDACTED (Marcus Nilsson) Date: Tue, 14 Apr 2015 15:54:27 +0200 Subject: [erlang-questions] Private key encryption In-Reply-To: References: Message-ID: Thanks for the reply Martin and helping me with this. The reason for doing this is that I want to share the files with a few people. After getting a better understanding of how this works I changed the my approach to this. Encryption 1. A shared secret is generated in the following form [Key, IV] 2. This secret is then used to encrypt the binary with aes_cfb128 a symmetric block encryption algo 3. The secret is then encrypted with the private key with RSA a asymmetric encryption algo 4. Both the encrypted shared secret and content is concatenated in the output file like this <> Decryption 1. Extract the encrypted secret from the file 2. Decrypt the shared secret with the public key with RSA 3. Decrypt the encrypted content with the now decrypted secret with aes_cfb128 algo The following snippet seems to do the job Key = crypto:rand_bytes(16), IV = crypto:rand_bytes(16), EncryptedContent = crypto:block_encrypt(aes_cfb128, Key, IV, Input), EncryptedKey = public_key:encrypt_private(list_to_binary([Key, IV]), PrivKey), [integer_to_binary(byte_size(EncryptedKey)), EncryptedKey, EncryptedContent]. Thanks /Marcus 2015-04-14 0:54 GMT+02:00 Martin Karlsson : > Hi Marcus, > > The encrypt_private is doing an RSA encryption using PKCS1 padding by > default. > > RSA can't encrypt large payloads (i.e. 256 bytes - 11 for padding for 2048 > bit RSA keys) so this is the likely reason you can only encrypt small > portions of the file. > > Normally you use public key crypto to encrypt a symmetrical key and then > encrypt the large payload with the symmetric key using AES or something. > > In addition you don't want to encrypt using the private key but rather the > public key, otherwise anyone with access to your public key can decrypt the > cipher. Private key encryption is usually only used for signatures. > > Crypto is hard to get right, especially if you are only working with RSA > primitives (you need to think about padding, hashing, MDCs, signatures). You > might want to have a look a NaCl (https://github.com/jloius/enacl for a > binding to erlang) which is much friendlier to use. > > Cheers, > Martin > > > On Tuesday, 14 April 2015, Marcus Nilsson wrote: >> >> I'am trying to encrypt a bigger file with the public_key module. >> Everything works fine as long as the content of the file is small. >> >> But when the size of the binary exceed's a certain size I get a >> **error:encrypt_failed. I guess this has to do with the padding parameter. >> >> But I have not been able to find any documentation how to compute the >> padding to get this working any help would be very welcomed! >> >> I use this code to perform the encryption >> >> public_key:encrypt_private(Input, PrivKey); >> >> /Marcus From carlsson.richard@REDACTED Tue Apr 14 22:04:54 2015 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Tue, 14 Apr 2015 22:04:54 +0200 Subject: [erlang-questions] Proposal for not allowing tuple calls In-Reply-To: References: Message-ID: On Tue, Apr 14, 2015 at 10:40 AM, Dmitry Belyaev wrote: > Are there any plans to introduce a 'module' type in erlang? That would > allow to get rid of that abomination called tuple calls and introduce a > correct implementation of parameterized modules. Exactly the same way as > lambdas capturing context values were introduced. > -- > Best wishes, > Dmitry Belyaev > That was of course the original plan, but nobody had time to work on it, and eventually the tentative implementation of parameterized modules was removed for lack of maintenance. /Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From lloyd@REDACTED Tue Apr 14 22:10:06 2015 From: lloyd@REDACTED (lloyd@REDACTED) Date: Tue, 14 Apr 2015 16:10:06 -0400 (EDT) Subject: [erlang-questions] Amazon API -- Lookup by ISBN In-Reply-To: <31338D70-45BB-4D61-87DF-BAEE88D1AE52@basho.com> References: <1428526022.03741188@apps.rackspace.com> <31338D70-45BB-4D61-87DF-BAEE88D1AE52@basho.com> Message-ID: <1429042206.830324726@apps.rackspace.com> Yeah! With the vast and generous help of the incredible Erlang posse, I can now retrieve book author, title, and publisher by isbn from Amazon services db. See: https://gist.github.com/anonymous/466ee04cbb57fd75d606 Given the gist I published a few days ago, this gives us two ways to retrieve book information. https://gist.github.com/anonymous/dd9846ef1cb2826f59da Many thanks to all. Lloyd -----Original Message----- From: "Gordon Guthrie" Sent: Tuesday, April 14, 2015 3:53am To: "Chandru" Cc: lloyd@REDACTED, "Erlang Questions" Subject: Re: [erlang-questions] Amazon API -- Lookup by ISBN There?s a pretty full clone of the Erlang AWS library in mochi that has a set of unit tests taken from the AWS documentation that you might want to use https://github.com/mochi/mochiweb/tree/master/examples/hmac_api Beware though - the AWS canonically rewrites the URL?s a little bit before signing them so this won?t work out of the box. G > Le 14 avr. 2015 ? 08:29, Chandru a ?crit : > > On 8 April 2015 at 21:47, > wrote: > Hello, > > I'm striving to look up books in Amazon's db by ISBN. At first blush it looks easy enough: > > http://docs.aws.amazon.com/AWSECommerceService/latest/DG/EX_LookupbyISBN.html > > But the last item, Signature, baffles me. Procedure here: > > http://docs.aws.amazon.com/AWSECommerceService/latest/DG/rest-signature.html > > I'm fine with this until I hit step 4: > > -- Sort parameter/value pairs by byte value --- I can see how to do this manually, but don't know how put Erlang to the task > > And I'm really stumped when I hit step 8: > > -- Calculate an RFC 2104-compliant HMAC with the SHA256 hash algorithm > > Any help? Better yet, does anyone have actual code to make such requests they're willing to share? > > > Sorry, quite a late reply here, but better late than never I guess. Here is some sample code I dug out of one of my pet projects. > > sign_amazon_aws_req(Method, Params) -> > {ok, {AWS_Access_id, Secret_key}} = application:get_env(my_app, aws_access_credentials), > > Boiler_plate_params = > ["Service=AWSECommerceService", > "AWSAccessKeyId=" ++ AWS_Access_id, > "AssociateTag=my_associate_tag", > "Timestamp=" ++ http_uri:encode(aws_timestamp()) > ], > Params_1 = Params ++ Boiler_plate_params, > Req_params = lists:flatten(string:join(lists:sort(Params_1), "&")), > Rest_endpoint = amazon_rest_endpoint(), > Rest_path = "/onca/xml", > String_to_sign = [Method, "\n", > Rest_endpoint, "\n", > Rest_path, "\n", > Req_params], > Signature = http_uri:encode( > base64:encode_to_string( > crypto:hmac(sha256, Secret_key, String_to_sign))), > lists:flatten(["https://" ++ Rest_endpoint ++ Rest_path ++ "?", > Req_params, > "&Signature=", Signature]). > > amazon_rest_endpoint() -> "webservices.amazon.co.uk ". > > aws_timestamp() -> > {{Y, M, D}, {H, Mi, S}} = calendar:universal_time(), > lists:flatten(io_lib:format("~p-~s-~sT~s:~s:~sZ", > [Y, two_digits(M), two_digits(D), > two_digits(H), two_digits(Mi), two_digits(S)])). > > two_digits(X) when X < 10 -> [$0 | integer_to_list(X)]; > two_digits(X) when is_integer(X) -> integer_to_list(X); > two_digits([X]) -> [$0, X]; > two_digits(X) -> X. > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From martink@REDACTED Tue Apr 14 23:28:50 2015 From: martink@REDACTED (Martin Karlsson) Date: Wed, 15 Apr 2015 09:28:50 +1200 Subject: [erlang-questions] Private key encryption In-Reply-To: References: Message-ID: Hi Marcus, It is getting better:) My only problem with this is that you are still using private_encrypt (end hence public decrypt). How are you going the distribute the RSA public key? Normally the public key is meant to be public but if something is encrypted using the RSA private key *any* party holding the RSA public key can decrypt the cipher. This means that you need to securely deliver the RSA public key. And if you had a way to securely distribute a key you wouldn't need RSA crypto in the first place:) (you would of course then securely distribute the shared secret) Public Key Crypto is mainly there to be able to share a secret key and this can be done in one of two ways: 1) Your friends send their public RSA key to you. You do encryption as above but replace step 3 with a public encrypt using your friends RSA key. Then they will decrypt the shared secret using their private key. In this case it doesn't matter who gets the public key because it is only the one holding the private key that can decrypt. 2) You send your friends your public RSA key. They generate a shared secret which they encrypt using your public key and then sends to you. You decrypt the shared secret with your private key and then use that shared secret to encrypt the file and send to you friend. Hopefully I've got this right. Cheers, Martin From nayibor@REDACTED Wed Apr 15 03:25:49 2015 From: nayibor@REDACTED (Nuku Ameyibor) Date: Wed, 15 Apr 2015 01:25:49 +0000 Subject: [erlang-questions] Private key encryption In-Reply-To: References: Message-ID: hi Martin, just to add a little to step 1 ) there is also the option of encrypting one file but using multiple recipients for that single encryption step instead of looping and encrypting it differently for each recipients public key so marcus could obtain a list of all the recipients keys and encrypt the single file he wants to send around and get a single output encrypted file which is sent to all the recipients . each recipient can decrypt the output file using his own private key . this can prove convenient especially when u have multiple recipients for the same file . i use bash/gpg for this on a daily basis to send sensitive files to multiple recipients but the idea can still be used in your scenario above . On Tue, Apr 14, 2015 at 9:28 PM, Martin Karlsson wrote: > Hi Marcus, > > It is getting better:) > > My only problem with this is that you are still using private_encrypt > (end hence public decrypt). > > How are you going the distribute the RSA public key? Normally the > public key is meant to be public but if something is encrypted using > the RSA private key *any* party holding the RSA public key can decrypt > the cipher. > > This means that you need to securely deliver the RSA public key. And > if you had a way to securely distribute a key you wouldn't need RSA > crypto in the first place:) (you would of course then securely > distribute the shared secret) > > Public Key Crypto is mainly there to be able to share a secret key and > this can be done in one of two ways: > > 1) Your friends send their public RSA key to you. You do encryption as > above but replace step 3 with a public encrypt using your friends RSA > key. Then they will decrypt the shared secret using their private key. > > In this case it doesn't matter who gets the public key because it is > only the one holding the private key that can decrypt. > > 2) You send your friends your public RSA key. They generate a shared > secret which they encrypt using your public key and then sends to you. > You decrypt the shared secret with your private key and then use that > shared secret to encrypt the file and send to you friend. > > Hopefully I've got this right. > > Cheers, > Martin > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sdl.web@REDACTED Wed Apr 15 03:58:57 2015 From: sdl.web@REDACTED (Leo Liu) Date: Wed, 15 Apr 2015 09:58:57 +0800 Subject: [erlang-questions] how does a DistEntry get its `creation' value? In-Reply-To: (Leo Liu's message of "Fri, 10 Apr 2015 12:52:21 +0800") References: Message-ID: On 2015-04-10 12:52 +0800, Leo Liu wrote: > I am looking at function erts_find_or_insert_dist_entry in > erl_node_tables.c and there is no `creation' at insertion time. Any idea > how a DistEntry gets its `creation' value? > > I also checked the distribution protocol and it seems a node can get its > own creation when publish to epmd. So when node A connects to node B, it > doesn't have node B's `creation'. How does the DistEntry for node B have > a correct creation value? Thanks. Any pointers? Thanks. Leo From juan@REDACTED Tue Apr 14 22:28:48 2015 From: juan@REDACTED (=?utf-8?Q?Juan_Puig_Mart=C3=ADnez?=) Date: Tue, 14 Apr 2015 13:28:48 -0700 Subject: [erlang-questions] Mismatch between dbg output and source code Message-ID: <2456B383-FB83-4991-8D95-AC06C83DF2DC@layer.com> I?m debugging a cause clause exception thrown inside the following function: create_nonce(DeviceId, Reason, Details, Message, Req) -> case tmc_nonce:get_nonce(DeviceId) of {error, memcached_unavailable = Reason} -> (?); {error, Reason} -> (?); {ok, Nonce} -> (?) end. As you see in the output from dbg below, the function claims a case clause {error, memcached_unavailable} not being handled. Apr 14 20:08:06 juan-standalone docker[5157]: (<0.3308.0>) call tmc_auth_middleman:create_nonce(...) Apr 14 20:08:06 juan-standalone docker[5157]: (<0.3308.0>) call tmc_nonce:get_nonce(<<246,135,71,188,226,225,17,228,179,150,14,124,0,0,28,12>>) Apr 14 20:08:06 juan-standalone docker[5157]: (<0.3308.0>) returned from tmc_nonce:get_nonce/1 -> {error, Apr 14 20:08:06 juan-standalone docker[5157]: memcached_unavailable} Apr 14 20:08:06 juan-standalone docker[5157]: (<0.3308.0>) exception_from {tmc_auth_middleman,create_nonce,5} {error,{case_clause,{error,memcached_unavailable}}} I?m double checked the beam loaded by the VM is correct (given the compilation time) and I?ve used `recon:source/1` to obtain source code based on the loaded beam (thanks to the debug_info flag). Below is the trimmed output (I?ve removed noise for simplicity): create_nonce(DeviceId, Reason, Details, Message, Req) ->\n case tmc_nonce:get_nonce(DeviceId) of\n {error, memcached_unavailable = Reason} ->\n\t (?);\n {error, Reason} ->\n\t (...);\n {ok, Nonce} ->\n\t (...)\n end. Knowing that this is the real code being executed, how could it possibly throw that exception? Is there other debugging levels I could use to understand why the mismatch happens? Thanks, /Juan From bjorn@REDACTED Wed Apr 15 06:31:47 2015 From: bjorn@REDACTED (=?UTF-8?Q?Bj=C3=B6rn_Gustavsson?=) Date: Wed, 15 Apr 2015 06:31:47 +0200 Subject: [erlang-questions] Mismatch between dbg output and source code In-Reply-To: <2456B383-FB83-4991-8D95-AC06C83DF2DC@layer.com> References: <2456B383-FB83-4991-8D95-AC06C83DF2DC@layer.com> Message-ID: On Tue, Apr 14, 2015 at 10:28 PM, Juan Puig Mart?nez wrote: > I?m debugging a cause clause exception thrown inside the following function: > > create_nonce(DeviceId, Reason, Details, Message, Req) -> What is the value of Reason passed into the function? > case tmc_nonce:get_nonce(DeviceId) of > {error, memcached_unavailable = Reason} -> This clause will only be executed if the value of Reason given as a function argument is memcached_unavailable. You probably want to change the line to something like: {error, memcached_unavailable = EReason} -> > (?); > {error, Reason} -> Same problem here. > (?); > {ok, Nonce} -> > (?) > end. /Bjorn -- Bj?rn Gustavsson, Erlang/OTP, Ericsson AB From erlang@REDACTED Wed Apr 15 11:49:18 2015 From: erlang@REDACTED (Joe Armstrong) Date: Wed, 15 Apr 2015 11:49:18 +0200 Subject: [erlang-questions] Private key encryption In-Reply-To: References: Message-ID: On Tue, Apr 14, 2015 at 3:54 PM, Marcus Nilsson wrote: > Thanks for the reply Martin and helping me with this. > > The reason for doing this is that I want to share the files with a few people. > After getting a better understanding of how this works I changed the > my approach to this. > > Encryption > 1. A shared secret is generated in the following form [Key, IV] > 2. This secret is then used to encrypt the binary with aes_cfb128 a > symmetric block encryption algo > 3. The secret is then encrypted with the private key with RSA a > asymmetric encryption algo > 4. Both the encrypted shared secret and content is concatenated in the > output file like this < EncryptedContent>> > > Decryption One alternative you might like to consider is making the envelope with term_to_binary To encode say Bin = term_to_binary({rsa_aes, EncryptedSecret, EncryptedContent}) and to decode {Tag, X, Y} = binary_to_term(Bin) The tag tells how the arguments were encrypted - this can be extended to make multiple nested containers (rather like the ASN.1 containers used in RSA etc. but far easier :-) /Joe > > 1. Extract the encrypted secret from the file > 2. Decrypt the shared secret with the public key with RSA > 3. Decrypt the encrypted content with the now decrypted secret with > aes_cfb128 algo > > The following snippet seems to do the job > > Key = crypto:rand_bytes(16), > IV = crypto:rand_bytes(16), > EncryptedContent = crypto:block_encrypt(aes_cfb128, Key, IV, Input), > EncryptedKey = public_key:encrypt_private(list_to_binary([Key, > IV]), PrivKey), > [integer_to_binary(byte_size(EncryptedKey)), EncryptedKey, > EncryptedContent]. > > Thanks > /Marcus > > 2015-04-14 0:54 GMT+02:00 Martin Karlsson : >> Hi Marcus, >> >> The encrypt_private is doing an RSA encryption using PKCS1 padding by >> default. >> >> RSA can't encrypt large payloads (i.e. 256 bytes - 11 for padding for 2048 >> bit RSA keys) so this is the likely reason you can only encrypt small >> portions of the file. >> >> Normally you use public key crypto to encrypt a symmetrical key and then >> encrypt the large payload with the symmetric key using AES or something. >> >> In addition you don't want to encrypt using the private key but rather the >> public key, otherwise anyone with access to your public key can decrypt the >> cipher. Private key encryption is usually only used for signatures. >> >> Crypto is hard to get right, especially if you are only working with RSA >> primitives (you need to think about padding, hashing, MDCs, signatures). You >> might want to have a look a NaCl (https://github.com/jloius/enacl for a >> binding to erlang) which is much friendlier to use. >> >> Cheers, >> Martin >> >> >> On Tuesday, 14 April 2015, Marcus Nilsson wrote: >>> >>> I'am trying to encrypt a bigger file with the public_key module. >>> Everything works fine as long as the content of the file is small. >>> >>> But when the size of the binary exceed's a certain size I get a >>> **error:encrypt_failed. I guess this has to do with the padding parameter. >>> >>> But I have not been able to find any documentation how to compute the >>> padding to get this working any help would be very welcomed! >>> >>> I use this code to perform the encryption >>> >>> public_key:encrypt_private(Input, PrivKey); >>> >>> /Marcus > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From erlang@REDACTED Wed Apr 15 11:57:34 2015 From: erlang@REDACTED (Joe Armstrong) Date: Wed, 15 Apr 2015 11:57:34 +0200 Subject: [erlang-questions] Private key encryption In-Reply-To: References: Message-ID: On Tue, Apr 14, 2015 at 11:28 PM, Martin Karlsson wrote: > Hi Marcus, > > It is getting better:) > > My only problem with this is that you are still using private_encrypt > (end hence public decrypt). > > How are you going the distribute the RSA public key? Normally the > public key is meant to be public but if something is encrypted using > the RSA private key *any* party holding the RSA public key can decrypt > the cipher. > > This means that you need to securely deliver the RSA public key. And > if you had a way to securely distribute a key you wouldn't need RSA > crypto in the first place:) (you would of course then securely > distribute the shared secret) There is a slight variation on this: Distribute the SHA1 checksum of the RSA public key (this also needs a secure channel) - but the SHA1 checksum is far shorter so can be scribbled on the back of an envelope. If anybody claims to have the public key you can ask them for it and validate with the SHA1 checksum - this can take place over an insecure channel, since a man-in-the-middle will cause the SHA1 checksum to be invalid. This is one of the tricks used in the self certifying file system (see) http://web.archive.org/web/20080725193436/http://www.fs.net/sfswww/sfsfaq.html There's a very nice paper that has this idea http://www.scs.stanford.edu/~dm/home/papers/mazieres:escape.ps.gz Self certifying systems need no centralized key authority :-) /Joe > > Public Key Crypto is mainly there to be able to share a secret key and > this can be done in one of two ways: > > 1) Your friends send their public RSA key to you. You do encryption as > above but replace step 3 with a public encrypt using your friends RSA > key. Then they will decrypt the shared secret using their private key. > > In this case it doesn't matter who gets the public key because it is > only the one holding the private key that can decrypt. > > 2) You send your friends your public RSA key. They generate a shared > secret which they encrypt using your public key and then sends to you. > You decrypt the shared secret with your private key and then use that > shared secret to encrypt the file and send to you friend. > > Hopefully I've got this right. > > Cheers, > Martin > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From sverker.eriksson@REDACTED Wed Apr 15 12:03:06 2015 From: sverker.eriksson@REDACTED (Sverker Eriksson) Date: Wed, 15 Apr 2015 12:03:06 +0200 Subject: [erlang-questions] how does a DistEntry get its `creation' value? In-Reply-To: References: Message-ID: <552E375A.5040204@ericsson.com> On 04/15/2015 03:58 AM, Leo Liu wrote: > On 2015-04-10 12:52 +0800, Leo Liu wrote: >> I am looking at function erts_find_or_insert_dist_entry in >> erl_node_tables.c and there is no `creation' at insertion time. Any idea >> how a DistEntry gets its `creation' value? >> >> I also checked the distribution protocol and it seems a node can get its >> own creation when publish to epmd. So when node A connects to node B, it >> doesn't have node B's `creation'. How does the DistEntry for node B have >> a correct creation value? Thanks. > Any pointers? Thanks. > > I did a quick search in the code before, and I could not find any place where DistrEntry.creation for other nodes was actually used. /Sverker From martink@REDACTED Wed Apr 15 12:25:28 2015 From: martink@REDACTED (Martin Karlsson) Date: Wed, 15 Apr 2015 22:25:28 +1200 Subject: [erlang-questions] Private key encryption In-Reply-To: References: Message-ID: Thanks Joe, > This is one of the tricks used in the self certifying file system > (see) http://web.archive.org/web/20080725193436/http://www.fs.net/sfswww/sfsfaq.html > There's a very nice paper that has this idea > http://www.scs.stanford.edu/~dm/home/papers/mazieres:escape.ps.gz > > Self certifying systems need no centralized key authority :-) Very useful. Cheers, Martin From andra.dinu@REDACTED Wed Apr 15 14:59:17 2015 From: andra.dinu@REDACTED (Andra Dinu) Date: Wed, 15 Apr 2015 13:59:17 +0100 Subject: [erlang-questions] Erlang User of the Year 2015 - voting is open Message-ID: Hi all, We are looking for the 18th Erlang User of The Year. Following the tradition, the Erlang user who has made outstanding contributions widely recognised by the community in 2014 will receive a trophy at the Erlang User Conference Stockholm 11-12 June. Please name your favourite and give a short explanation on why your nominee deserves to take the trophy home this year in this Google doc: https://docs.google.com/a/erlang-solutions.com/forms/d/1xIVkeDHx95FR-6YW3dfs-hBcLCJqh1EPyrxdYftfxfo/viewform We?re accepting nominations until 8 May. The selection will be made by a panel chaired by Bjarne D?cker and consisting of Joe Armstrong (co-inventor of Erlang), Kenneth Lundin (manager of the Erlang/OTP team at Ericsson), Robert Virding (co-inventor of Erlang, technical expert at Erlang Solutions Ltd) and the last three years' recipients of the award: Fred Hebert, Lo?c Hoguin and Anthony Ramine. If you're curious to see who were all the users of the year starting with 1995, we posted a list on the website: http://www.erlang-factory.com/euc2015#erlang-user-of-the-year Best, Andra *Andra Dinu* Community & Social Check out the Erlang User Conference in Stockholm on 11-12 June! -------------- next part -------------- An HTML attachment was scrubbed... URL: From sdl.web@REDACTED Wed Apr 15 15:48:16 2015 From: sdl.web@REDACTED (Leo Liu) Date: Wed, 15 Apr 2015 21:48:16 +0800 Subject: [erlang-questions] how does a DistEntry get its `creation' value? In-Reply-To: <552E375A.5040204@ericsson.com> (Sverker Eriksson's message of "Wed, 15 Apr 2015 12:03:06 +0200") References: <552E375A.5040204@ericsson.com> Message-ID: Hi Sverker, On 2015-04-15 18:03 +0800, Sverker Eriksson wrote: > I did a quick search in the code before, and I could not find any place > where DistrEntry.creation for other nodes was actually used. Thanks for the answer. Does this mean list_to_pid("") can not recover a remote pid correctly i.e. the creation info is lost? Thanks, Leo From fw339tgy@REDACTED Wed Apr 15 15:28:25 2015 From: fw339tgy@REDACTED (=?GBK?B?zLi549TG?=) Date: Wed, 15 Apr 2015 21:28:25 +0800 (CST) Subject: [erlang-questions] the supervisor 's error when the child process occur 'function_clause' error!! Message-ID: <59949129.110b6.14cbd443327.Coremail.fw339tgy@126.com> hi joe, when i use erlang laguage , i meet a problem. i use supervisor to manage the child process. test module is supervisor , the child is my_bank moudle is gen_server's callback . the code is : %%%%test module -module(test). -behaviour(supervisor). %% ==================================================================== %% API functions %% ==================================================================== %% Supervisor callbacks -export([init/1]). -export([start_link/0,start_child/0]). %% ==================================================================== %% Internal functions %% ==================================================================== init([]) -> process_flag(trap_exit, true), Poker = {my_bank, {my_bank, start, []}, permanent, brutal_kill, worker, [my_bank]}, {ok, {{one_for_one, 5, 10}, [Poker]}}. start_link() -> supervisor:start_link({local, ?MODULE}, ?MODULE, []). %%%%% ============= %%%% my_bank module -module(my_bank). -behaviour(gen_server). -export([start/0]). %% gen_server callbacks -export([init/1, handle_call/3, handle_cast/2, handle_info/2, terminate/2, code_change/3]). -compile(export_all). -define(SERVER, ?MODULE). start() -> gen_server:start_link({local, ?MODULE}, ?MODULE, [], []). stop() -> gen_server:call(?MODULE, stop). new_account(Who) -> gen_server:call(?MODULE, {new, Who}). deposit(Who, Amount) -> gen_server:call(?MODULE, {add, Who, Amount}). withdraw(Who, Amount) -> gen_server:call(?MODULE, {remove, Who, Amount}). init([]) -> %process_flag(trap_exit, true), %gen_server:start({local, my_bank2}, my_bank2, [], []), io:format("=====start===== my_bank ~n"), {ok, ets:new(?MODULE,[])}. %%%%%% i call gen_server:call(my_bank,{new2,inzaghi}). gen_server:call(my_bank,{new,inzaghi}) is right . the my_bank will occur a error "function_clause". when the my_bank process occur ' function_clause' error, the supervisor can not restat the my_bank process , and quit with no error. This problem has troubled me for a long time, very much look forward to your reply. thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From lavrin@REDACTED Wed Apr 15 17:08:05 2015 From: lavrin@REDACTED (=?UTF-8?Q?Rados=C5=82aw_Szymczyszyn?=) Date: Wed, 15 Apr 2015 17:08:05 +0200 Subject: [erlang-questions] Nitrogen-based XMPP client? In-Reply-To: References: Message-ID: Hi Lloyd, It's nice to see interest in MongooseIM. One of the support libraries we use when developing the server is Escalus - an XMPP client brought to life as a lib facilitating functional testing, but usable outside a testing environment. After struggling with Tsung for stressing XMPP servers we also started using Escalus as core of a load testing tool and never looked back. The thing to keep in mind if you intend to use it is: use escalus_connection directly, not escalus_client. The latter might be a bottleneck if you intend to drive a lot of connections with it (it's a single process) - it's not a problem for a functional test suite, but might be for some real world applications. The project is available on GitHub: https://github.com/esl/escalus/ At this point it's probably obvious, but to make it explicit: I work at Erlang Solutions on MongooseIM, Escalus and XMPP. I'm also reachable as radoslaw.szymczyszyn @ the company's domain. Best regards, Radek Szymczyszyn 2015-04-03 5:29 GMT+02:00 Lloyd R. Prentice : > Hello, > > I'm considering integrating multi-user chat into a Nitrogen application I'm building--- assuming that I'll support it with Mongoose IM. As usual, I'm in over my head, but persistent. > > I'm imagining a small number ( four? six?) thematic chat rooms serving a relatively small number of visitors at a time. As to scale, think of a book-signing event at a bookstore. > > I see that there are several JavaScript XMPP client packages that I could integrate into my Nitrogen web pages but, if that's the way, which one? Or, is there a native Erlang XMPP client app that I should look at? Indeed, is Mongoose IM overkill? After all, the Nitrogen demos provide a very simple tinker-toy chat example, and numerous others can be found on the web though, most, rather long in the tooth. > > And more, how much harder would it be to integrate video chat using, say, web sockets? > > In other words, how can I best keep the gotchas, demons, and goblins at bay sufficiently to master the arcane arts of multi-user chat sufficiently to build my application before I die of old age? > > I'd much appreciate pointers or, better yet, pointers to actual code. > > All the best, > > LRP > > Sent from my iPad > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From rtrlists@REDACTED Wed Apr 15 18:44:49 2015 From: rtrlists@REDACTED (Robert Raschke) Date: Wed, 15 Apr 2015 17:44:49 +0100 Subject: [erlang-questions] the supervisor 's error when the child process occur 'function_clause' error!! In-Reply-To: <59949129.110b6.14cbd443327.Coremail.fw339tgy@126.com> References: <59949129.110b6.14cbd443327.Coremail.fw339tgy@126.com> Message-ID: Try removing the process_flag(trap_exit, true) from the supervisor. I don't know if that is the problem. I have no idea what trapping exits in a supervisor does, but it looks wrong :-) Regards Robby On Apr 15, 2015 3:17 PM, "???" wrote: > hi joe, > > when i use erlang laguage , i meet a problem. > > i use supervisor to manage the child process. > > test module is supervisor , the child is my_bank moudle is gen_server's > callback . > > the code is : > > %%%%test module > -module(test). > -behaviour(supervisor). > > %% ==================================================================== > %% API functions > %% ==================================================================== > %% Supervisor callbacks > -export([init/1]). > > -export([start_link/0,start_child/0]). > > %% ==================================================================== > %% Internal functions > %% ==================================================================== > > init([]) -> > process_flag(trap_exit, true), > Poker = {my_bank, {my_bank, start, []}, permanent, brutal_kill, worker, > [my_bank]}, > {ok, {{one_for_one, 5, 10}, [Poker]}}. > > start_link() -> > supervisor:start_link({local, ?MODULE}, ?MODULE, []). > > %%%%% ============= > > %%%% my_bank module > -module(my_bank). > > -behaviour(gen_server). > -export([start/0]). > %% gen_server callbacks > -export([init/1, handle_call/3, handle_cast/2, handle_info/2, > terminate/2, code_change/3]). > -compile(export_all). > -define(SERVER, ?MODULE). > > start() -> gen_server:start_link({local, ?MODULE}, ?MODULE, [], []). > stop() -> gen_server:call(?MODULE, stop). > > new_account(Who) -> gen_server:call(?MODULE, {new, Who}). > deposit(Who, Amount) -> gen_server:call(?MODULE, {add, Who, Amount}). > withdraw(Who, Amount) -> gen_server:call(?MODULE, {remove, Who, Amount}). > > init([]) -> > %process_flag(trap_exit, true), > %gen_server:start({local, my_bank2}, my_bank2, [], []), > io:format("=====start===== my_bank ~n"), > {ok, ets:new(?MODULE,[])}. > > > %%%%%% > i call gen_server:call(my_bank,{new2,inzaghi}). > > gen_server:call(my_bank,{new,inzaghi}) is right . > > the my_bank will occur a error "function_clause". > > > when the my_bank process occur ' function_clause' error, the supervisor > can not restat the my_bank process , and quit with no error. > > > This problem has troubled me for a long time, very much look forward to > your reply. > > thanks. > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From juan@REDACTED Wed Apr 15 19:01:20 2015 From: juan@REDACTED (=?utf-8?Q?Juan_Puig_Mart=C3=ADnez?=) Date: Wed, 15 Apr 2015 10:01:20 -0700 Subject: [erlang-questions] Mismatch between dbg output and source code In-Reply-To: References: <2456B383-FB83-4991-8D95-AC06C83DF2DC@layer.com> Message-ID: <765FA875-9F01-4F71-9B71-DB36C3684F32@layer.com> Good catch! I totally ignored that ?Reason? was passed from another context with different semantics. > On Apr 14, 2015, at 9:31 PM, Bj?rn Gustavsson wrote: > > On Tue, Apr 14, 2015 at 10:28 PM, Juan Puig Mart?nez wrote: >> I?m debugging a cause clause exception thrown inside the following function: >> >> create_nonce(DeviceId, Reason, Details, Message, Req) -> > > What is the value of Reason passed into the function? > >> case tmc_nonce:get_nonce(DeviceId) of >> {error, memcached_unavailable = Reason} -> > > This clause will only be executed if the value of Reason > given as a function argument is memcached_unavailable. > > You probably want to change the line to something like: > > {error, memcached_unavailable = EReason} -> > >> (?); >> {error, Reason} -> > > Same problem here. > >> (?); >> {ok, Nonce} -> >> (?) >> end. > > /Bjorn > -- > Bj?rn Gustavsson, Erlang/OTP, Ericsson AB -------------- next part -------------- An HTML attachment was scrubbed... URL: From lloyd@REDACTED Wed Apr 15 22:15:20 2015 From: lloyd@REDACTED (lloyd@REDACTED) Date: Wed, 15 Apr 2015 16:15:20 -0400 (EDT) Subject: [erlang-questions] =?utf-8?q?Parsing_XML_with_xmerl=5Fxpath/strin?= =?utf-8?q?g/2?= Message-ID: <1429128920.670127795@apps.rackspace.com> Well, pick myself up from one stumble only to trip over the next. I've successfully downloaded an XML book file from Amazon and extracted title and author code using this function, much of which I borrowed from Dave Thomas: get_book_data(ISBN) -> {ok, {_Status, _Headers, Body}} = httpc:request(get_book_request(ISBN)), {Xml, _Rest} = xmerl_scan:string(Body), [ #xmlText{value=Author}] = xmerl_xpath:string("//Author/text()", Xml), [ #xmlText{value=Title}] = xmerl_xpath:string("//Title/text()", Xml), [ #xmlText{value=Publisher}] = xmerl_xpath:string("//Manufacturer/text()", Xml), {Author, Title, Publisher}. That same file has URLs for downloading book cover images. The XML looks like this: ... http://ecx.images-amazon.com/images/I/51shI6vQ-SL.jpg 500 333 ... Modifying get_book_data/1 to include a xmerl_xpath:string/2 call that looks like this: [ #xmlText{value=LargeImage}] = xmerl_xpath:string("//LargeImage/text()", Xml), fails: ** exception error: no match of right hand side value [] in function amz_lookup:get_book_cover_images/1 (/home/lloyd/wga/site/src/LitUtils/amz_lookup.erl, line 110) My hunch is that the string "//LargeImage/text())" is the problem. xmerl_xpath:string/2 docs give me no comfort. Clearly I don't understand them sufficiently. I've tried xmerl_xpath:string("//URL/text()"), xmerl_xpath:string("//LargeImage/URL/text()"), etc. etc. etc.--- all to no avail. Google search also comes up empty. So once more to the well of wisdom. Can some kind soul point out the error of my ways? All the best, LRP From michael.santos@REDACTED Wed Apr 15 22:42:41 2015 From: michael.santos@REDACTED (Michael Santos) Date: Wed, 15 Apr 2015 16:42:41 -0400 Subject: [erlang-questions] Parsing XML with xmerl_xpath/string/2 In-Reply-To: <1429128920.670127795@apps.rackspace.com> References: <1429128920.670127795@apps.rackspace.com> Message-ID: <20150415204241.GA29149@brk> On Wed, Apr 15, 2015 at 04:15:20PM -0400, lloyd@REDACTED wrote: > Well, pick myself up from one stumble only to trip over the next. > > I've successfully downloaded an XML book file from Amazon and extracted title and author code using this function, much of which I borrowed from Dave Thomas: > > get_book_data(ISBN) -> > {ok, {_Status, _Headers, Body}} = httpc:request(get_book_request(ISBN)), > {Xml, _Rest} = xmerl_scan:string(Body), > [ #xmlText{value=Author}] = xmerl_xpath:string("//Author/text()", Xml), > [ #xmlText{value=Title}] = xmerl_xpath:string("//Title/text()", Xml), > [ #xmlText{value=Publisher}] = xmerl_xpath:string("//Manufacturer/text()", Xml), > {Author, Title, Publisher}. > > That same file has URLs for downloading book cover images. The XML looks like this: > > ... > > > http://ecx.images-amazon.com/images/I/51shI6vQ-SL.jpg > > 500 > 333 > > ... > > Modifying get_book_data/1 to include a xmerl_xpath:string/2 call that looks like this: > > [ #xmlText{value=LargeImage}] = xmerl_xpath:string("//LargeImage/text()", Xml), > > fails: > > ** exception error: no match of right hand side value [] > in function amz_lookup:get_book_cover_images/1 (/home/lloyd/wga/site/src/LitUtils/amz_lookup.erl, line 110) > > My hunch is that the string "//LargeImage/text())" is the problem. xmerl_xpath:string/2 docs give me no comfort. Clearly I don't understand them sufficiently. > > I've tried xmerl_xpath:string("//URL/text()"), xmerl_xpath:string("//LargeImage/URL/text()"), etc. etc. etc.--- all to no avail. Google search also comes up empty. > > So once more to the well of wisdom. Can some kind soul point out the error of my ways? Using "//LargeImage/URL/text()" works for me: 1> [ #xmlText{value=LargeImage}] = xmerl_xpath:string("//LargeImage/URL/text()", Xml). [#xmlText{parents = [{'URL',1},{'LargeImage',1}], pos = 1,language = [], value = "http://ecx.images-amazon.com/images/I/51shI6vQ-SL.jpg", type = text}] You can figure out what is going on by building up the xpath statement in the shell: 1> Str = "http://ecx.images-amazon.com/images/I/51shI6vQ-SL.jpg500333". 2> {Xml,_} = xmerl_scan:string(Str). 3> rr(xmerl). 4> xmerl_xpath:string("/", Xml). 5> xmerl_xpath:string("/LargeImage", Xml). 6> xmerl_xpath:string("/LargeImage/URL", Xml). From lloyd@REDACTED Thu Apr 16 01:48:22 2015 From: lloyd@REDACTED (lloyd@REDACTED) Date: Wed, 15 Apr 2015 19:48:22 -0400 (EDT) Subject: [erlang-questions] =?utf-8?q?Parsing_XML_with_xmerl=5Fxpath/strin?= =?utf-8?q?g/2?= In-Reply-To: <20150415204241.GA29149@brk> References: <1429128920.670127795@apps.rackspace.com> <20150415204241.GA29149@brk> Message-ID: <1429141702.331216955@apps.rackspace.com> Thanks Michael, You inspired confidence that I'm on the right track. With that I discovered the problem: My httpc query returned an XML structure with several LargeImage tags. xmerl_scan:string/1 returned them all in a list. When I stripped off the head of the list, ala: get_book_cover_images(ISBN) -> {ok, {_Status, _Headers, Body}} = httpc:request(get_book_request(ISBN)), {Xml, _Rest} = xmerl_scan:string(Body), [H|_] = xmerl_xpath:string("//LargeImage/URL/text()", Xml), #xmlText{value=LargeImage} = H, LargeImage. ...it works. Thanks again, Lloyd -----Original Message----- From: "Michael Santos" Sent: Wednesday, April 15, 2015 4:42pm To: lloyd@REDACTED Cc: "Erlang Questions" Subject: Re: [erlang-questions] Parsing XML with xmerl_xpath/string/2 On Wed, Apr 15, 2015 at 04:15:20PM -0400, lloyd@REDACTED wrote: > Well, pick myself up from one stumble only to trip over the next. > > I've successfully downloaded an XML book file from Amazon and extracted title and author code using this function, much of which I borrowed from Dave Thomas: > > get_book_data(ISBN) -> > {ok, {_Status, _Headers, Body}} = httpc:request(get_book_request(ISBN)), > {Xml, _Rest} = xmerl_scan:string(Body), > [ #xmlText{value=Author}] = xmerl_xpath:string("//Author/text()", Xml), > [ #xmlText{value=Title}] = xmerl_xpath:string("//Title/text()", Xml), > [ #xmlText{value=Publisher}] = xmerl_xpath:string("//Manufacturer/text()", Xml), > {Author, Title, Publisher}. > > That same file has URLs for downloading book cover images. The XML looks like this: > > ... > > > http://ecx.images-amazon.com/images/I/51shI6vQ-SL.jpg > > 500 > 333 > > ... > > Modifying get_book_data/1 to include a xmerl_xpath:string/2 call that looks like this: > > [ #xmlText{value=LargeImage}] = xmerl_xpath:string("//LargeImage/text()", Xml), > > fails: > > ** exception error: no match of right hand side value [] > in function amz_lookup:get_book_cover_images/1 (/home/lloyd/wga/site/src/LitUtils/amz_lookup.erl, line 110) > > My hunch is that the string "//LargeImage/text())" is the problem. xmerl_xpath:string/2 docs give me no comfort. Clearly I don't understand them sufficiently. > > I've tried xmerl_xpath:string("//URL/text()"), xmerl_xpath:string("//LargeImage/URL/text()"), etc. etc. etc.--- all to no avail. Google search also comes up empty. > > So once more to the well of wisdom. Can some kind soul point out the error of my ways? Using "//LargeImage/URL/text()" works for me: 1> [ #xmlText{value=LargeImage}] = xmerl_xpath:string("//LargeImage/URL/text()", Xml). [#xmlText{parents = [{'URL',1},{'LargeImage',1}], pos = 1,language = [], value = "http://ecx.images-amazon.com/images/I/51shI6vQ-SL.jpg", type = text}] You can figure out what is going on by building up the xpath statement in the shell: 1> Str = "http://ecx.images-amazon.com/images/I/51shI6vQ-SL.jpg500333". 2> {Xml,_} = xmerl_scan:string(Str). 3> rr(xmerl). 4> xmerl_xpath:string("/", Xml). 5> xmerl_xpath:string("/LargeImage", Xml). 6> xmerl_xpath:string("/LargeImage/URL", Xml). From llbgurs@REDACTED Thu Apr 16 03:19:44 2015 From: llbgurs@REDACTED (linbo liao) Date: Thu, 16 Apr 2015 09:19:44 +0800 Subject: [erlang-questions] Fwd: poolboy new_worker failed cause poolboy crashed In-Reply-To: References: Message-ID: Hi Jos?, I add a proxy process to handle the issue ( https://github.com/yunba/eredis_pool/blob/master/src/eredis_proxy.erl), but wired if connect redis timeout, proxy process sometimes will be shutdown although I set trap_exit = true. I test it , find all connections timeout when multi client call connnect simulataneous, after eredis_client exit info received, eredis_proxy will be shutdown. I have totally no idea about it. Did I miss something? Thanks, Linbo 2015-04-14 19:01 GMT+08:00 linbo liao : > Thanks, looks a nice way to solve the issue, I will create a proxy worker > to have a try. > > 2015-04-14 16:49 GMT+08:00 Jos? Valim : > >> You don't need to modify eredis per se. You can create a worker that is >> between poolboy and eredis that tracks the connected/disconnected semantics >> as Devin described. Unfortunately I don't know of any sample Erlang code >> but, if you don't mind, here is a similar solution using redo written in >> Elixir: >> >> >> https://github.com/phoenixframework/phoenix/blob/master/lib/phoenix/pubsub/redis_conn.ex >> >> To use it, you checkout the worker above from the pool, ask the redis >> connection to the worker and do the operations you want to in the >> connection as usual. >> >> As a note, I did find it a bit weird that eredis handles disconnects >> automatically but it does not handle the case where it can't connect to >> Redis on start_link. It would be handful if they supported lazy connections >> (i.e. don't connect on start_link but on first command). They already have >> all the code to handle this, it is just a matter of supporting it on >> start_link too. >> >> >> >> *Jos? Valim* >> www.plataformatec.com.br >> Skype: jv.ptec >> Founder and Lead Developer >> >> On Tue, Apr 14, 2015 at 10:26 AM, linbo liao wrote: >> >>> Hi, >>> >>> I open an issue for poolboy ( >>> https://github.com/devinus/poolboy/issues/72), but looks author doesn't >>> think this is an issue. >>> >>> Looks like to solve this issue have two ways: >>> >>> 1. Modify eredis client, start_link still return ok when connect >>> failed, and when use it should check eredis socket alive or not >>> >>> 2. Modify poolboy library, protected all logic when call new_worker >>> failed. >>> >>> But it require change ereids or poolboy code, I was wondering to know >>> anyone hit this issue, how to resolve it? >>> >>> Thanks, >>> Linbo >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From llbgurs@REDACTED Thu Apr 16 05:14:17 2015 From: llbgurs@REDACTED (linbo liao) Date: Thu, 16 Apr 2015 11:14:17 +0800 Subject: [erlang-questions] poolboy new_worker failed cause poolboy crashed In-Reply-To: References: Message-ID: I find the issue, refre to poolboy checkin ( https://github.com/devinus/poolboy/blob/master/src/poolboy.erl#L309), when checkin process and overflow >0, it will close process to make sure pool size return back, won't overflow. It will cause process be closed (maybe old process, maybe new process). Thanks, Linbo 2015-04-16 9:19 GMT+08:00 linbo liao : > Hi Jos?, > > I add a proxy process to handle the issue ( > https://github.com/yunba/eredis_pool/blob/master/src/eredis_proxy.erl), > but wired if connect redis timeout, proxy process sometimes will be > shutdown although I set trap_exit = true. > > I test it , find all connections timeout when multi client call connnect > simulataneous, after eredis_client exit info received, eredis_proxy will be > shutdown. I have totally no idea about it. > > Did I miss something? > > Thanks, > Linbo > > 2015-04-14 19:01 GMT+08:00 linbo liao : > >> Thanks, looks a nice way to solve the issue, I will create a proxy worker >> to have a try. >> >> 2015-04-14 16:49 GMT+08:00 Jos? Valim : >> >>> You don't need to modify eredis per se. You can create a worker that is >>> between poolboy and eredis that tracks the connected/disconnected semantics >>> as Devin described. Unfortunately I don't know of any sample Erlang code >>> but, if you don't mind, here is a similar solution using redo written in >>> Elixir: >>> >>> >>> https://github.com/phoenixframework/phoenix/blob/master/lib/phoenix/pubsub/redis_conn.ex >>> >>> To use it, you checkout the worker above from the pool, ask the redis >>> connection to the worker and do the operations you want to in the >>> connection as usual. >>> >>> As a note, I did find it a bit weird that eredis handles disconnects >>> automatically but it does not handle the case where it can't connect to >>> Redis on start_link. It would be handful if they supported lazy connections >>> (i.e. don't connect on start_link but on first command). They already have >>> all the code to handle this, it is just a matter of supporting it on >>> start_link too. >>> >>> >>> >>> *Jos? Valim* >>> www.plataformatec.com.br >>> Skype: jv.ptec >>> Founder and Lead Developer >>> >>> On Tue, Apr 14, 2015 at 10:26 AM, linbo liao wrote: >>> >>>> Hi, >>>> >>>> I open an issue for poolboy ( >>>> https://github.com/devinus/poolboy/issues/72), but looks author >>>> doesn't think this is an issue. >>>> >>>> Looks like to solve this issue have two ways: >>>> >>>> 1. Modify eredis client, start_link still return ok when connect >>>> failed, and when use it should check eredis socket alive or not >>>> >>>> 2. Modify poolboy library, protected all logic when call new_worker >>>> failed. >>>> >>>> But it require change ereids or poolboy code, I was wondering to know >>>> anyone hit this issue, how to resolve it? >>>> >>>> Thanks, >>>> Linbo >>>> >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> >>>> >>> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From freecnpro@REDACTED Thu Apr 16 01:07:58 2015 From: freecnpro@REDACTED (Bill Wang) Date: Thu, 16 Apr 2015 07:07:58 +0800 Subject: [erlang-questions] the supervisor 's error when the child process occur 'function_clause' error!! In-Reply-To: <59949129.110b6.14cbd443327.Coremail.fw339tgy@126.com> References: <59949129.110b6.14cbd443327.Coremail.fw339tgy@126.com> Message-ID: Hi, Application processes should normally not trap exits. http://www.erlang.org/doc/man/erlang.html#process_flag-2 Cheers, Bill 2015?4?15? ??10:17? "???" ??? > hi joe, > > when i use erlang laguage , i meet a problem. > > i use supervisor to manage the child process. > > test module is supervisor , the child is my_bank moudle is gen_server's > callback . > > the code is : > > %%%%test module > -module(test). > -behaviour(supervisor). > > %% ==================================================================== > %% API functions > %% ==================================================================== > %% Supervisor callbacks > -export([init/1]). > > -export([start_link/0,start_child/0]). > > %% ==================================================================== > %% Internal functions > %% ==================================================================== > > init([]) -> > process_flag(trap_exit, true), > Poker = {my_bank, {my_bank, start, []}, permanent, brutal_kill, worker, > [my_bank]}, > {ok, {{one_for_one, 5, 10}, [Poker]}}. > > start_link() -> > supervisor:start_link({local, ?MODULE}, ?MODULE, []). > > %%%%% ============= > > %%%% my_bank module > -module(my_bank). > > -behaviour(gen_server). > -export([start/0]). > %% gen_server callbacks > -export([init/1, handle_call/3, handle_cast/2, handle_info/2, > terminate/2, code_change/3]). > -compile(export_all). > -define(SERVER, ?MODULE). > > start() -> gen_server:start_link({local, ?MODULE}, ?MODULE, [], []). > stop() -> gen_server:call(?MODULE, stop). > > new_account(Who) -> gen_server:call(?MODULE, {new, Who}). > deposit(Who, Amount) -> gen_server:call(?MODULE, {add, Who, Amount}). > withdraw(Who, Amount) -> gen_server:call(?MODULE, {remove, Who, Amount}). > > init([]) -> > %process_flag(trap_exit, true), > %gen_server:start({local, my_bank2}, my_bank2, [], []), > io:format("=====start===== my_bank ~n"), > {ok, ets:new(?MODULE,[])}. > > > %%%%%% > i call gen_server:call(my_bank,{new2,inzaghi}). > > gen_server:call(my_bank,{new,inzaghi}) is right . > > the my_bank will occur a error "function_clause". > > > when the my_bank process occur ' function_clause' error, the supervisor > can not restat the my_bank process , and quit with no error. > > > This problem has troubled me for a long time, very much look forward to > your reply. > > thanks. > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From davidnwelton@REDACTED Thu Apr 16 11:33:15 2015 From: davidnwelton@REDACTED (David Welton) Date: Thu, 16 Apr 2015 11:33:15 +0200 Subject: [erlang-questions] Reinventing the database wheel Message-ID: It'd be cool if there were some kind of generic pool + circuit breaker kind of thing. Everyone ends up reinventing and rewriting this wheel, which is is a big waste of programmer time. I did some code for Postgres here: https://github.com/epgsql/pgapp but the problem is essentially the same across other databases. -- David N. Welton http://www.welton.it/davidw/ http://www.dedasys.com/ From carlsson.richard@REDACTED Thu Apr 16 13:43:13 2015 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Thu, 16 Apr 2015 13:43:13 +0200 Subject: [erlang-questions] Time to change the default for +W? Message-ID: Isn't it about time that we got three levels of error_logger messages by default? Quoting the erl documentation: *+W w | i* Sets the mapping of warning messages for error_logger. Messages sent to the error logger using one of the warning routines can be mapped either to errors (default), warnings (+W w), or info reports (+W i). The current mapping can be retrieved using error_logger:warning_map/0. See error_logger(3) for further information. I feel that in this day and age, this ought to be changed so that +W w is the default, and a new flag "+W e" should be added for those who still want the old behaviour. /Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From fw339tgy@REDACTED Thu Apr 16 08:02:57 2015 From: fw339tgy@REDACTED (=?GBK?B?zLi549TG?=) Date: Thu, 16 Apr 2015 14:02:57 +0800 (CST) Subject: [erlang-questions] [help] The supervisor can not restart the child process , when the child process occur 'function_clause' error!! Message-ID: <218790b9.2415a.14cc0d2ba28.Coremail.fw339tgy@126.com> The supervisor can not restart the child process , when the child process occur 'function_clause' error!! the test moudle is supervisor 's callback, the my_bank is child process 's callback. The code in the appendix. This problem has troubled me for a long time, very much look forward to your reply. thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: my_bank.erl Type: application/octet-stream Size: 2394 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: test.erl Type: application/octet-stream Size: 953 bytes Desc: not available URL: From sverker.eriksson@REDACTED Thu Apr 16 15:16:21 2015 From: sverker.eriksson@REDACTED (Sverker Eriksson) Date: Thu, 16 Apr 2015 15:16:21 +0200 Subject: [erlang-questions] how does a DistEntry get its `creation' value? In-Reply-To: References: <552E375A.5040204@ericsson.com> Message-ID: <552FB625.1080108@ericsson.com> On 04/15/2015 03:48 PM, Leo Liu wrote: > Hi Sverker, > > On 2015-04-15 18:03 +0800, Sverker Eriksson wrote: >> I did a quick search in the code before, and I could not find any place >> where DistrEntry.creation for other nodes was actually used. > Thanks for the answer. > > Does this mean list_to_pid("") can not recover a remote pid > correctly i.e. the creation info is lost? > > Yes, list_to_pid can not restore the correct 'creation' value. But the main reason for this is that the string representation "" does not contain any information about 'creation'. The DistEntry.creation field is always set to zero for remote nodes and is only read by list_to_pid. Creation value zero has a special meaning of "this node instance" if the node name matches the local node. list_to_pid is documented as a debugging feature, which means the priority to "fix" this has not been very high. /Sverker, Erlang/OTP From paulperegud@REDACTED Thu Apr 16 17:17:20 2015 From: paulperegud@REDACTED (Paul Peregud) Date: Thu, 16 Apr 2015 17:17:20 +0200 Subject: [erlang-questions] [help] The supervisor can not restart the child process , when the child process occur 'function_clause' error!! In-Reply-To: <218790b9.2415a.14cc0d2ba28.Coremail.fw339tgy@126.com> References: <218790b9.2415a.14cc0d2ba28.Coremail.fw339tgy@126.com> Message-ID: Please provide a sequence of commands that leads to crash. You can check if my_bank gen_server has restarted by running whereis(my_bank) in console. On Thu, Apr 16, 2015 at 8:02 AM, ??? wrote: > The supervisor can not restart the child process , when the child process > occur 'function_clause' error!! > > the test moudle is supervisor 's callback, > > the my_bank is child process 's callback. > > The code in the appendix. > > This problem has troubled me for a long time, very much look forward to your > reply. > > thanks. > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Best regards, Paul Peregud +48602112091 From paulperegud@REDACTED Thu Apr 16 17:36:37 2015 From: paulperegud@REDACTED (Paul Peregud) Date: Thu, 16 Apr 2015 17:36:37 +0200 Subject: [erlang-questions] Fwd: [help] The supervisor can not restart the child process , when the child process occur 'function_clause' error!! In-Reply-To: References: <218790b9.2415a.14cc0d2ba28.Coremail.fw339tgy@126.com> <70F132F1-DF79-42A9-81F9-324AC808FF55@126.com> Message-ID: ---------- Forwarded message ---------- From: Paul Peregud Date: Thu, Apr 16, 2015 at 5:35 PM Subject: Re: [erlang-questions] [help] The supervisor can not restart the child process , when the child process occur 'function_clause' error!! To: Fw339tgy Your code is fine. Problem is in the fact that you are starting root supervisor of your tree from console using start_link. This means that your shell process is linked to root supervisor. When shell process crashes, it takes down your supervisor. You can test this behavior by using following commands: test:start_link(). whereis(test). whereis(my_bank). true = false. %% this will crash your shell. whereis(test). whereis(bank). Additional information is in process_info(self()) in section "links". On Thu, Apr 16, 2015 at 5:25 PM, Fw339tgy wrote: > test:start_link(). Tha app start. > > gen_sever:call(my_bank,{new2,inzaghi}. The command will courr the error. > > the right command is gen_sever:call(my_bank,{new, > inzaghi}. > > ???? iPhone > >> ? 2015?4?16????11:17?Paul Peregud ??? >> >> Please provide a sequence of commands that leads to crash. >> >> You can check if my_bank gen_server has restarted by running >> whereis(my_bank) in console. >> >>> On Thu, Apr 16, 2015 at 8:02 AM, ??? wrote: >>> The supervisor can not restart the child process , when the child process >>> occur 'function_clause' error!! >>> >>> the test moudle is supervisor 's callback, >>> >>> the my_bank is child process 's callback. >>> >>> The code in the appendix. >>> >>> This problem has troubled me for a long time, very much look forward to your >>> reply. >>> >>> thanks. >>> >>> >>> >>> >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> >> -- >> Best regards, >> Paul Peregud >> +48602112091 > -- Best regards, Paul Peregud +48602112091 -- Best regards, Paul Peregud +48602112091 From sdl.web@REDACTED Fri Apr 17 03:37:36 2015 From: sdl.web@REDACTED (Leo Liu) Date: Fri, 17 Apr 2015 09:37:36 +0800 Subject: [erlang-questions] how does a DistEntry get its `creation' value? References: <552E375A.5040204@ericsson.com> <552FB625.1080108@ericsson.com> Message-ID: On 2015-04-16 21:16 +0800, Sverker Eriksson wrote: > Yes, list_to_pid can not restore the correct 'creation' value. > But the main reason for this is that the string representation "" > does not contain any information about 'creation'. > > The DistEntry.creation field is always set to zero for remote nodes > and is only read by list_to_pid. Creation value zero has a special > meaning of "this node instance" if the node name matches the local > node. Many thanks for the information. That clears my confusion. > list_to_pid is documented as a debugging feature, which means the > priority to "fix" this has not been very high. Noted. Thanks again, Leo From llbgurs@REDACTED Fri Apr 17 04:16:34 2015 From: llbgurs@REDACTED (linbo liao) Date: Fri, 17 Apr 2015 10:16:34 +0800 Subject: [erlang-questions] Reinventing the database wheel In-Reply-To: References: Message-ID: Hi David? Could you help to point the problem? Here I hit this issue, when client exceed the pool size and start new overflow client, now due to network or other issue, new connection isn't succeed. At same time, old good client will checkin, but overflow >0, those good clients will be shutdown by poolby. Thanks, Linbo 2015-04-16 17:33 GMT+08:00 David Welton : > It'd be cool if there were some kind of generic pool + circuit breaker > kind of thing. Everyone ends up reinventing and rewriting this wheel, > which is is a big waste of programmer time. > > I did some code for Postgres here: https://github.com/epgsql/pgapp but > the problem is essentially the same across other databases. > > -- > David N. Welton > > http://www.welton.it/davidw/ > > http://www.dedasys.com/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vladdu55@REDACTED Fri Apr 17 21:59:55 2015 From: vladdu55@REDACTED (Vlad Dumitrescu) Date: Fri, 17 Apr 2015 21:59:55 +0200 Subject: [erlang-questions] abstract format issues Message-ID: Hi! The documentation of the abstract format specifies that the second element in the node tuples is LINE, an integer denoting the line number where the node was built (the line of the "defining" token). Issue 1: This is no longer entirely correct, since the lexical scanner can return more complex information (line, column and text value). I will try to update the docs for that. Issue 2: For applications where the abstract syntax is used to refer to source code, it is just as important to know the position of the first and last tokens. I would like to know if there are more people interested in a solution integrated in the regular parser (provided that backwards compatibility can be preserved). If not, it is simpler to me to just hack my own version of erl_parse. best regards, Vlad -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlsson.richard@REDACTED Sat Apr 18 00:15:02 2015 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Sat, 18 Apr 2015 00:15:02 +0200 Subject: [erlang-questions] Time to change the default for +W? In-Reply-To: References: Message-ID: Since nobody seemed to have an opinion about this, I went ahead and created a pull request: https://github.com/erlang/otp/pull/685 It turned out that there was already some code implementing the +We option, but it was broken, and the option was never documented. /Richard On Thu, Apr 16, 2015 at 1:43 PM, Richard Carlsson < carlsson.richard@REDACTED> wrote: > Isn't it about time that we got three levels of error_logger messages by > default? Quoting the erl documentation: > *+W w | i* > > Sets the mapping of warning messages for error_logger. Messages sent to > the error logger using one of the warning routines can be mapped either to > errors (default), warnings (+W w), or info reports (+W i). The current > mapping can be retrieved using error_logger:warning_map/0. See > error_logger(3) > for > further information. > > I feel that in this day and age, this ought to be changed so that +W w is > the default, and a new flag "+W e" should be added for those who still want > the old behaviour. > /Richard > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abstractwater@REDACTED Sun Apr 19 03:01:57 2015 From: abstractwater@REDACTED (abstractwater) Date: Sat, 18 Apr 2015 18:01:57 -0700 Subject: [erlang-questions] pre-installation smoke test failure in cpu_sup_SUITE Message-ID: <5080C552-A58F-4BF3-885C-778BE9EAA99C@gmail.com> I compiled OTP 17.5 on a Mac OS X 10.9.5 but i am getting an error in the pre-installation test suite, specifically in the cpu_sup_SUITE. Here?s how i compiled it: $ export ERL_TOP=`pwd` $ ./configure --with-ssl=/usr/lib --enable-builtin-zlib --enable-darwin-64bit $ make $ make release_tests $ cd release/tests/test_server $ $ERL_TOP/bin/erl -s ts install -s ts smoke_test batch -s init stop I looked at the actual test source file but i couldn?t really understand the problem: 130 %% util() 131 ?line Util1 = cpu_sup:util(), 132 ?line true = is_number(Util1), 133 ?line true = Util1>0, %% FAIL I am new to Erlang to I may be missing something obvious. Any insight on what can i do to fix it? Here?s the test log output: =emulator_vsn 6.4 =emulator beam =otp_release 17 =started 2015-04-18 17:25:57 =case cpu_sup_SUITE:init_per_suite =logfile cpu_sup_suite.init_per_suite.html =started 2015-04-18 17:25:57 =ended 2015-04-18 17:25:57 =result ok =elapsed 0.002177 =case cpu_sup_SUITE:load_api =logfile cpu_sup_suite.load_api.html =started 2015-04-18 17:25:57 =ended 2015-04-18 17:25:57 =result ok =elapsed 0.066456 =case cpu_sup_SUITE:util_api =logfile cpu_sup_suite.util_api.html =started 2015-04-18 17:25:57 =ended 2015-04-18 17:25:57 =result failed: {{badmatch,false}, [{cpu_sup_SUITE,util_api,1, [{file,"cpu_sup_SUITE.erl"},{line,133}]}, {test_server,ts_tc,3, [{file,"test_server.erl"},{line,1429}]}, {test_server,run_test_case_eval1,6, [{file,"test_server.erl"},{line,1026}]}, {test_server,run_test_case_eval,9, [{file,"test_server.erl"},{line,968}]}]}, [{cpu_sup_SUITE, util_api, 133}, {test_server, ts_tc, 1429}, {test_server, run_test_case_eval1, 1026}, {test_server, run_test_case_eval, 968}] Thanks in advance! From jb_duffy@REDACTED Sun Apr 19 17:30:40 2015 From: jb_duffy@REDACTED (John Duffy) Date: Sun, 19 Apr 2015 16:30:40 +0100 (BST) Subject: [erlang-questions] Streaming Data using httpc Message-ID: <21202172.29185.1429457440273.JavaMail.defaultUser@defaultHost> Hi I'm new to Erlang so please forgive my ignorance. I'm trying to stream data from a REST API using httpc, and although I have scoured the internet and the documentation I can't find a good example of how to do this, in particular how the "sync" and "receiver" options interoperate. My unsuccessful module looks like this... -module(streaming). -export([data/0]). data() -> {ok, RequestId} = httpc:request(get, {"http://my_streaming_data_source.com", []}, [], [{sync, false}, {receiver, self()]), receive_data(RequestId). receive_data(RequestId) -> receive {http, {RequestId, stream_start, Headers}} -> do something...; {http, {RequestId, stream, Data}} - > do something...; {http, {RequestId, stream_end, Headers}} -> do something...; end, receive_data(RequestId). Is the above how I should be structuring my module? Kind regards John Duffy -------------- next part -------------- An HTML attachment was scrubbed... URL: From paulperegud@REDACTED Sun Apr 19 20:06:37 2015 From: paulperegud@REDACTED (Paul Peregud) Date: Sun, 19 Apr 2015 20:06:37 +0200 Subject: [erlang-questions] Streaming Data using httpc In-Reply-To: <21202172.29185.1429457440273.JavaMail.defaultUser@defaultHost> References: <21202172.29185.1429457440273.JavaMail.defaultUser@defaultHost> Message-ID: I don't have experience using httpc, so one remark only: receive_data(RequestId, State) -> receive {http, {RequestId, stream_start, Headers}} -> do something..., receive_data(RequestId, State); {http, {RequestId, stream, Data}} - > do something..., receive_data(RequestId, State); {http, {RequestId, stream_end, Headers}} -> do something... end. Because you want to exit receive_data when no more data is going your way. On Sun, Apr 19, 2015 at 5:30 PM, John Duffy wrote: > Hi > > I'm new to Erlang so please forgive my ignorance. I'm trying to stream data > from a REST API using httpc, and although I have scoured the internet and > the documentation I can't find a good example of how to do this, in > particular how the "sync" and "receiver" options interoperate. My > unsuccessful module looks like this... > > -module(streaming). > > -export([data/0]). > > data() -> > {ok, RequestId} = httpc:request(get, > {"http://my_streaming_data_source.com", []}, [], [{sync, false}, {receiver, > self()]), > receive_data(RequestId). > > receive_data(RequestId) -> > receive > {http, {RequestId, stream_start, Headers}} -> do something...; > {http, {RequestId, stream, Data}} - > do something...; > {http, {RequestId, stream_end, Headers}} -> do something...; > end, > receive_data(RequestId). > > > Is the above how I should be structuring my module? > > Kind regards > > John Duffy > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Best regards, Paul Peregud +48602112091 From jesper.louis.andersen@REDACTED Sun Apr 19 23:00:05 2015 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Sun, 19 Apr 2015 21:00:05 +0000 Subject: [erlang-questions] Streaming Data using httpc In-Reply-To: <21202172.29185.1429457440273.JavaMail.defaultUser@defaultHost> References: <21202172.29185.1429457440273.JavaMail.defaultUser@defaultHost> Message-ID: On Sun, Apr 19, 2015 at 6:14 PM John Duffy wrote: > > > -module(streaming). > > [...] You are pretty close to the goal, but you are confusing the stream/receiver options I think. In streaming, you will receive the data as a series of chunks, which is what your code expect, but you don't supply an option requesting streaming operation. So you don't retrieve an expected tuple. You can add a catchall tuple to your receieve clause in receive_data/1 to make sure you have the right format in your match. Also, you can add a 'after' clause to time out after a while. This can make debugging easier since you "get back" to the the REPL. The following works on a quick test in my end. Note how the receive clause is different from yours, and that you get everything in one fell swoop, rather than having to match on a multitude of clauses. For more serious work, you might want to check out some of the numerous other projects for HTTP client requests. I'm partial to Gun and Hackney myself, but there are also ibrowse, lhttpc and fusco. They have slightly different semantics and areas at which they excel, so choose wisely :) For a prototype however, I think httpc is fine. -module(streaming). -export([data/0]). data() -> {ok, RequestId} = httpc:request(get, {"example.com", []}, [], [{sync, false}, {receiver, self()}]), receive_data(RequestId). receive_data(RequestId) -> receive {http, {RequestId, {StatusLine, Headers, Body}}} -> error_logger:info_report( #{ status => StatusLine, headers => length(Headers), body_size => byte_size(Body) }), ok after 5000 -> error_logger:info_report(timeout) end. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nx@REDACTED Sun Apr 19 23:20:48 2015 From: nx@REDACTED (nx) Date: Sun, 19 Apr 2015 17:20:48 -0400 Subject: [erlang-questions] Streaming Data using httpc In-Reply-To: References: <21202172.29185.1429457440273.JavaMail.defaultUser@defaultHost> Message-ID: Not sure what you're trying to do, but if you're actually processing the body of the message in a stream you may want to use the stream option and make your module a process. Otherwise you can stream the data to a file (i.e. downloading files) using the {stream, filename()} option. If you're wanting the former, and you want to do something with each part of the stream and ask for the next block in the stream, you can do something like this. This assumes you're module is a gen_server and you're using the {stream, once} option to step through the stream response: handle_call({async_request, Url}, _From, State) -> httpc:request(get, {Url, _Headers=[]}, _HttpOptions=[], [{stream, {self, once}}, {sync, false}]), {reply, ok, State}. %% Start stream, store httpc Pid, continue handle_info({http, {_RequestId, stream_start, _Headers, Pid}}, State}) -> ok = httpc:stream_next(Pid), {noreply, State#state{pid=Pid}}; %% Handle a chunk, continue handle_info({http, {_RequestId, stream, Part}}, #state{pid=Pid}=State}) -> handle_part(Part), ok = httpc:stream_next(Pid), {noreply, State}; %% Handle a 404 error, cancel the request handle_info({http, {RequestId, {{_HTTPVersion, 404, "Not Found"}, _Headers, _Body}}}, State) -> ok = httpc:cancel_request(RequestId), {stop, normal, State}; %% Handle stream error, cancel the request handle_info({http, {RequestId, {error, Reason}}}, State) -> ok = httpc:cancel_request(RequestId), {stop, normal, State}; %% Stream end handle_info({http, {_RequestId, stream_end, _Headers}}, State) -> do_something(), {stop, normal, State}. The standard httpc library is flexible enough if you want a precise level of control over the lifetime of the HTTP stream. On Sun, Apr 19, 2015 at 2:06 PM, Paul Peregud wrote: > I don't have experience using httpc, so one remark only: > > receive_data(RequestId, State) -> > receive > {http, {RequestId, stream_start, Headers}} -> do something..., > receive_data(RequestId, State); > {http, {RequestId, stream, Data}} - > do > something..., receive_data(RequestId, State); > {http, {RequestId, stream_end, Headers}} -> do something... > end. > > Because you want to exit receive_data when no more data is going your way. > > > On Sun, Apr 19, 2015 at 5:30 PM, John Duffy wrote: >> Hi >> >> I'm new to Erlang so please forgive my ignorance. I'm trying to stream data >> from a REST API using httpc, and although I have scoured the internet and >> the documentation I can't find a good example of how to do this, in >> particular how the "sync" and "receiver" options interoperate. My >> unsuccessful module looks like this... >> >> -module(streaming). >> >> -export([data/0]). >> >> data() -> >> {ok, RequestId} = httpc:request(get, >> {"http://my_streaming_data_source.com", []}, [], [{sync, false}, {receiver, >> self()]), >> receive_data(RequestId). >> >> receive_data(RequestId) -> >> receive >> {http, {RequestId, stream_start, Headers}} -> do something...; >> {http, {RequestId, stream, Data}} - > do something...; >> {http, {RequestId, stream_end, Headers}} -> do something...; >> end, >> receive_data(RequestId). >> >> >> Is the above how I should be structuring my module? >> >> Kind regards >> >> John Duffy >> >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > > > -- > Best regards, > Paul Peregud > +48602112091 > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From carlosj.gf@REDACTED Sun Apr 19 23:27:38 2015 From: carlosj.gf@REDACTED (=?ISO-8859-1?Q?Carlos_Gonz=E1lez_Florido?=) Date: Sun, 19 Apr 2015 23:27:38 +0200 Subject: [erlang-questions] [ANN] NkPACKET Message-ID: Hello, just a short notice to present NkPACKET, a generic transport layer for Erlang, that can be used to develop high perfomance, low latency network servers, clients and proxies. Features: - Support for UDP, TCP/TLS, SCTP and WS/WSS. - STUN server. - Connection-oriented (even for UDP). - DNS engine with full support for NAPTR and SRV location, including priority and weights. - URL-mapping of servers and connections. - Wrap over Cowboy to write domain-specific, high perfomance http servers. NkPACKET uses Ranch, Cowboy and Cowlib. Grab it at https://github.com/Nekso/nkpacket. Regards, Carlos -------------- next part -------------- An HTML attachment was scrubbed... URL: From liudanking@REDACTED Mon Apr 20 08:41:55 2015 From: liudanking@REDACTED (Daniel) Date: Mon, 20 Apr 2015 14:41:55 +0800 Subject: [erlang-questions] Does erlang:now() guarantee that subsequent calls to this BIF returns continuously increasing values even in erlang cluster? Message-ID: <3F7FBF7F-3FE4-432E-B2F6-BB4898CE03A7@gmail.com> Hi, dear list, I am looking for a global sequence generator in erlang cluster. I think erlang:now() is promising, because it is fast enough and ?also guarantees that subsequent calls to this BIF returns continuously increasing values?. But I am not quite sure whether this guarantee also works for erlang cluster which have several nodes running on different servers? I have read Time and time correction in Erlang (http://www.erlang.org/doc/apps/erts/time_correction.html), but still have no conclusion. Daniel Liu From sergej.jurecko@REDACTED Mon Apr 20 08:44:33 2015 From: sergej.jurecko@REDACTED (=?UTF-8?Q?Sergej_Jure=C4=8Dko?=) Date: Mon, 20 Apr 2015 08:44:33 +0200 Subject: [erlang-questions] Does erlang:now() guarantee that subsequent calls to this BIF returns continuously increasing values even in erlang cluster? In-Reply-To: <3F7FBF7F-3FE4-432E-B2F6-BB4898CE03A7@gmail.com> References: <3F7FBF7F-3FE4-432E-B2F6-BB4898CE03A7@gmail.com> Message-ID: It does not guarantee unique values in cluster. Sergej On Apr 20, 2015 8:42 AM, "Daniel" wrote: > Hi, dear list, > > I am looking for a global sequence generator in erlang cluster. > > I think erlang:now() is promising, because it is fast enough and ?also > guarantees that subsequent calls to this BIF returns continuously > increasing values?. But I am not quite sure whether this guarantee also > works for erlang cluster which have several nodes running on different > servers? > > I have read Time and time correction in Erlang ( > http://www.erlang.org/doc/apps/erts/time_correction.html), but still have > no conclusion. > > Daniel Liu > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mjtruog@REDACTED Mon Apr 20 09:33:24 2015 From: mjtruog@REDACTED (Michael Truog) Date: Mon, 20 Apr 2015 00:33:24 -0700 Subject: [erlang-questions] Does erlang:now() guarantee that subsequent calls to this BIF returns continuously increasing values even in erlang cluster? In-Reply-To: References: <3F7FBF7F-3FE4-432E-B2F6-BB4898CE03A7@gmail.com> Message-ID: <5534ABC4.9030207@gmail.com> You might want to look at the v1 UUID as created by https://github.com/okeuday/uuid since the node id part of the v1 UUID is specific to both the Erlang node (including the MAC address) and the Erlang pid that created the UUID state. For time synchronization among nodes ntpd is normally used to minimize time differences, which then influences the time reported by erlang:now/0 and erlang:system_time/1. On 04/19/2015 11:44 PM, Sergej Jure?ko wrote: > > It does not guarantee unique values in cluster. > > Sergej > > On Apr 20, 2015 8:42 AM, "Daniel" > wrote: > > Hi, dear list, > > I am looking for a global sequence generator in erlang cluster. > > I think erlang:now() is promising, because it is fast enough and ?also guarantees that subsequent calls to this BIF returns continuously increasing values?. But I am not quite sure whether this guarantee also works for erlang cluster which have several nodes running on different servers? > > I have read Time and time correction in Erlang (http://www.erlang.org/doc/apps/erts/time_correction.html), but still have no conclusion. > > Daniel Liu > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From sergej.jurecko@REDACTED Mon Apr 20 09:16:54 2015 From: sergej.jurecko@REDACTED (=?UTF-8?Q?Sergej_Jure=C4=8Dko?=) Date: Mon, 20 Apr 2015 09:16:54 +0200 Subject: [erlang-questions] Does erlang:now() guarantee that subsequent calls to this BIF returns continuously increasing values even in erlang cluster? In-Reply-To: <23CF4BC4-C430-429E-8ED4-A98502F5C918@gmail.com> References: <3F7FBF7F-3FE4-432E-B2F6-BB4898CE03A7@gmail.com> <23CF4BC4-C430-429E-8ED4-A98502F5C918@gmail.com> Message-ID: You could use something like this: <> and assign an index to every server. Sergej On Mon, Apr 20, 2015 at 9:06 AM, Daniel wrote: > If so, is there any method that can get unique values in cluster? I prefer > time based unique values if there is any. > > > On Apr 20, 2015, at 2:44 PM, Sergej Jure?ko > wrote: > > > > It does not guarantee unique values in cluster. > > > > Sergej > > > > On Apr 20, 2015 8:42 AM, "Daniel" wrote: > > Hi, dear list, > > > > I am looking for a global sequence generator in erlang cluster. > > > > I think erlang:now() is promising, because it is fast enough and ?also > guarantees that subsequent calls to this BIF returns continuously > increasing values?. But I am not quite sure whether this guarantee also > works for erlang cluster which have several nodes running on different > servers? > > > > I have read Time and time correction in Erlang ( > http://www.erlang.org/doc/apps/erts/time_correction.html), but still have > no conclusion. > > > > Daniel Liu > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlsson.richard@REDACTED Mon Apr 20 10:51:32 2015 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Mon, 20 Apr 2015 10:51:32 +0200 Subject: [erlang-questions] Uptime shell function Message-ID: Inspired by this discussion: http://erlang.org/pipermail/erlang-questions/2010-March/050078.html https://github.com/erlang/otp/pull/688 /Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From jb_duffy@REDACTED Mon Apr 20 13:05:24 2015 From: jb_duffy@REDACTED (John Duffy) Date: Mon, 20 Apr 2015 12:05:24 +0100 (BST) Subject: [erlang-questions] Streaming Data using httpc Message-ID: <30987936.26037.1429527924762.JavaMail.defaultUser@defaultHost> Hi Jesper Thank you for your reply, very helpful. I'm still a bit puzzled as to why my test doesn't work, I'm being to wonder if I need to be sending additional headers, or something, to the server. If I use 'curl' then everything works, I get a steady stream of data... curl "http://stream-sandbox.oanda.com/v1/prices?accountId=99999&instruments=EUR_USD" However, putting the same URL into your example results in a single time-out and the Erlang emulator stalling. Kind regards John ----Original message---- >From : jesper.louis.andersen@REDACTED Date : 19/04/2015 - 22:00 (GMTDT) To : jb_duffy@REDACTED, erlang-questions@REDACTED Subject : Re: [erlang-questions] Streaming Data using httpc On Sun, Apr 19, 2015 at 6:14 PM John Duffy wrote: -module(streaming). [...] You are pretty close to the goal, but you are confusing the stream/receiver options I think. In streaming, you will receive the data as a series of chunks, which is what your code expect, but you don't supply an option requesting streaming operation. So you don't retrieve an expected tuple. You can add a catchall tuple to your receieve clause in receive_data/1 to make sure you have the right format in your match. Also, you can add a 'after' clause to time out after a while. This can make debugging easier since you "get back" to the the REPL. The following works on a quick test in my end. Note how the receive clause is different from yours, and that you get everything in one fell swoop, rather than having to match on a multitude of clauses. For more serious work, you might want to check out some of the numerous other projects for HTTP client requests. I'm partial to Gun and Hackney myself, but there are also ibrowse, lhttpc and fusco. They have slightly different semantics and areas at which they excel, so choose wisely :) For a prototype however, I think httpc is fine. -module(streaming). -export([data/0]). data() -> {ok, RequestId} = httpc:request(get, {"example.com", []}, [], [{sync, false}, {receiver, self()}]), receive_data(RequestId). receive_data(RequestId) -> receive {http, {RequestId, {StatusLine, Headers, Body}}} -> error_logger:info_report( #{ status => StatusLine, headers => length(Headers), body_size => byte_size(Body) }), ok after 5000 -> error_logger:info_report(timeout) end. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mononcqc@REDACTED Mon Apr 20 14:54:20 2015 From: mononcqc@REDACTED (Fred Hebert) Date: Mon, 20 Apr 2015 08:54:20 -0400 Subject: [erlang-questions] Does erlang:now() guarantee that subsequent calls to this BIF returns continuously increasing values even in erlang cluster? In-Reply-To: <3F7FBF7F-3FE4-432E-B2F6-BB4898CE03A7@gmail.com> References: <3F7FBF7F-3FE4-432E-B2F6-BB4898CE03A7@gmail.com> Message-ID: <20150420125419.GA16522@ferdair.local> On 04/20, Daniel wrote: >I am looking for a global sequence generator in erlang cluster. > >I think erlang:now() is promising, because it is fast enough and ?also guarantees that subsequent calls to this BIF returns continuously increasing values?. But I am not quite sure whether this guarantee also works for erlang cluster which have several nodes running on different servers? > >I have read Time and time correction in Erlang >(http://www.erlang.org/doc/apps/erts/time_correction.html), but still >have no conclusion. > There is a very important thing. You did mention *in a cluster*. What you are asking for, a "global sequence" requires putting every call you have into a well-known order where any value can be compared to any value and sorted properly. This is known as a "total order", and requires synchronization. This means that doing this requires your nodes to discuss together, and in concert either: a) elect a leader to create the sequence values b) come to an agreement for the sequence of values These are very broad lines (the theoretical aspect of it gets mixed in with a lot of fancy consistency model in distributed systems talk). You will want this [generally costly] mechanism when: - You need an absolute order in the sequence of your events - You are ready to see potential unavailability when all the nodes in the system cannot agree on how to allocate the numbers. There is no way around it. On the other hand, there are some alternative options: you could decide to have only *some* events ordered. This means that for some values you could definitely say which comes before or after the other, but not for all of them. This is called a 'partial order'. It's what would happen if I generated logs on two different nodes and kept the node identifier in the log. When that happens, I know that logs from A are in the right order (I trust their timestamp), but I cannot know if node A's 2015/04/20T12:59:23+00:00 comes before node B's 2015/04/20T12:59:23+00:00: their respective clocks might be off, drifting, or their resolution too low for me to decide. So therefore, when sorting logs for these two systems, I can decide whether one came before or after the others from the same node (as long as I used a monotonic clock like erlang:now() to generate the values), but cannot reliably sort logs across nodes. The best answer I can personally give you here is not to tell you how to properly generate these numbers, but rather to ask right back about what it is you're trying to accomplish, and why do you believe you need these numbers for? If there's a way to get away without the continuously increasing values at a global level, this will be much simpler operationally than trying to maintain healthy clusters that synchronize and block on every ID you want to generate. From michael.eugene.turner@REDACTED Mon Apr 20 16:06:32 2015 From: michael.eugene.turner@REDACTED (Michael Turner) Date: Mon, 20 Apr 2015 23:06:32 +0900 Subject: [erlang-questions] Does erlang:now() guarantee that subsequent calls to this BIF returns continuously increasing values even in erlang cluster? In-Reply-To: <20150420125419.GA16522@ferdair.local> References: <3F7FBF7F-3FE4-432E-B2F6-BB4898CE03A7@gmail.com> <20150420125419.GA16522@ferdair.local> Message-ID: If you can make do with a partial order rather than a total order, somehow: http://www.erlang.org/doc/man/seq_trace.html Basically, Lamport clocks. http://research.microsoft.com/en-us/um/people/lamport/pubs/time-clocks.pdf Regards, Michael Turner Executive Director Project Persephone K-1 bldg 3F 7-2-6 Nishishinjuku Shinjuku-ku Tokyo 160-0023 Tel: +81 (3) 6890-1140 Fax: +81 (3) 6890-1158 Mobile: +81 (90) 5203-8682 turner@REDACTED http://www.projectpersephone.org/ "Love does not consist in gazing at each other, but in looking outward together in the same direction." -- Antoine de Saint-Exup?ry On Mon, Apr 20, 2015 at 9:54 PM, Fred Hebert wrote: > On 04/20, Daniel wrote: > >> I am looking for a global sequence generator in erlang cluster. >> >> I think erlang:now() is promising, because it is fast enough and ?also >> guarantees that subsequent calls to this BIF returns continuously >> increasing values?. But I am not quite sure whether this guarantee also >> works for erlang cluster which have several nodes running on different >> servers? >> >> I have read Time and time correction in Erlang ( >> http://www.erlang.org/doc/apps/erts/time_correction.html), but still >> have no conclusion. >> >> > There is a very important thing. You did mention *in a cluster*. What you > are asking for, a "global sequence" requires putting every call you have > into a well-known order where any value can be compared to any value and > sorted properly. This is known as a "total order", and requires > synchronization. > > This means that doing this requires your nodes to discuss together, and in > concert either: > > a) elect a leader to create the sequence values > b) come to an agreement for the sequence of values > > These are very broad lines (the theoretical aspect of it gets mixed in > with a lot of fancy consistency model in distributed systems talk). > > You will want this [generally costly] mechanism when: > > - You need an absolute order in the sequence of your events > - You are ready to see potential unavailability when all the nodes in the > system cannot agree on how to allocate the numbers. > > There is no way around it. On the other hand, there are some alternative > options: you could decide to have only *some* events ordered. This means > that for some values you could definitely say which comes before or after > the other, but not for all of them. This is called a 'partial order'. > > It's what would happen if I generated logs on two different nodes and kept > the node identifier in the log. When that happens, I know that logs from A > are in the right order (I trust their timestamp), but I cannot know if node > A's 2015/04/20T12:59:23+00:00 comes before node B's > 2015/04/20T12:59:23+00:00: their respective clocks might be off, drifting, > or their resolution too low for me to decide. > > So therefore, when sorting logs for these two systems, I can decide > whether one came before or after the others from the same node (as long as > I used a monotonic clock like erlang:now() to generate the values), but > cannot reliably sort logs across nodes. > > The best answer I can personally give you here is not to tell you how to > properly generate these numbers, but rather to ask right back about what it > is you're trying to accomplish, and why do you believe you need these > numbers for? > > If there's a way to get away without the continuously increasing values at > a global level, this will be much simpler operationally than trying to > maintain healthy clusters that synchronize and block on every ID you want > to generate. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mononcqc@REDACTED Mon Apr 20 16:51:13 2015 From: mononcqc@REDACTED (Fred Hebert) Date: Mon, 20 Apr 2015 10:51:13 -0400 Subject: [erlang-questions] Does erlang:now() guarantee that subsequent calls to this BIF returns continuously increasing values even in erlang cluster? In-Reply-To: References: <3F7FBF7F-3FE4-432E-B2F6-BB4898CE03A7@gmail.com> <20150420125419.GA16522@ferdair.local> Message-ID: <20150420145111.GB16522@ferdair.local> On 04/20, Michael Turner wrote: >If you can make do with a partial order rather than a total order, somehow: > > http://www.erlang.org/doc/man/seq_trace.html > >Basically, Lamport clocks. > Lamport clocks are one form of it. Sorting by {Node, Timestamp} also gives you a partial order. So do vector clocks and version vectors, interval tree clocks, and so on. Lamport/vector clocks and other similar ones operate on *causality*, but this partial ordering is not the only one available or workable. You could have them operating on users, nodes, shards, clusters, geographical regions, customers, or any other partitioning mechanism desired, for example. It's important to look at the system and its desired properties to be able to figure out which constraints allows us to best fit it to find the most appropriate solution possible. Regards, Fred. From elbrujohalcon@REDACTED Mon Apr 20 21:30:39 2015 From: elbrujohalcon@REDACTED (Fernando 'Brujo' Benavides) Date: Mon, 20 Apr 2015 16:30:39 -0300 Subject: [erlang-questions] [ANN] Gadget - The Ultimate Code-Checking Machine Message-ID: <288F7404-FD24-437E-B00F-4229352525AB@inaka.net> Hi all, I?ve already introduced Gadget on the Erlang Factory (you can see me here ), but since then I?ve been adding some things and improving others and the current version is actually ready for all of you to hit it with your best open-source repos! So, everyone who has an open-source project on github is invited to let Gadget run compiler, xref, dialyzer and elvis on its pull requests and keep constantly improving the quality of its code! You can find Gadget at http://gadget.inakalabs.com and there is a blog post about it here: http://inaka.net/blog/2015/04/20/gadget/ Cheers!! Fernando "Brujo" Benavides about.me/elbrujohalcon -------------- next part -------------- An HTML attachment was scrubbed... URL: From martink@REDACTED Tue Apr 21 05:54:33 2015 From: martink@REDACTED (Martin Karlsson) Date: Tue, 21 Apr 2015 15:54:33 +1200 Subject: [erlang-questions] exometer core only In-Reply-To: <5524E079.2050609@gmail.com> References: <65FC5D9B-BEA3-4DB2-BF49-376117489742@feuerlabs.com> <5524E079.2050609@gmail.com> Message-ID: Hi Tino, > exometer itself (versions later than 1.1) depends directly on exometer_core Following up on this. I need some of the reporters in exometer. However the latest tagged release is 1.1 which is not using exometer_core. Further to this exometer uses {branch, "master"} in rebar.config to pull a dependency. It is very hard to use this in production as we are developing against a moving target. Is there a better way to fix this? Ideally tag exometer with a fixed version of exometer_core? Cheers, Martin -------------- next part -------------- An HTML attachment was scrubbed... URL: From tino.breddin@REDACTED Tue Apr 21 07:09:34 2015 From: tino.breddin@REDACTED (Tino Breddin) Date: Tue, 21 Apr 2015 07:09:34 +0200 Subject: [erlang-questions] exometer core only In-Reply-To: References: <65FC5D9B-BEA3-4DB2-BF49-376117489742@feuerlabs.com> <5524E079.2050609@gmail.com> Message-ID: Hi. Agreed. I think we are ready for a new version since most work recently has been to harden exometer. We will look at it shortly. Tino On Apr 21, 2015 5:54 AM, "Martin Karlsson" wrote: > Hi Tino, > > > exometer itself (versions later than 1.1) depends directly on > exometer_core > > Following up on this. I need some of the reporters in exometer. However > the latest tagged release is 1.1 which is not using exometer_core. Further > to this exometer uses {branch, "master"} in rebar.config to pull a > dependency. > > It is very hard to use this in production as we are developing against a > moving target. > > Is there a better way to fix this? Ideally tag exometer with a fixed > version of exometer_core? > > Cheers, > Martin > -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.eugene.turner@REDACTED Tue Apr 21 07:11:40 2015 From: michael.eugene.turner@REDACTED (Michael Turner) Date: Tue, 21 Apr 2015 14:11:40 +0900 Subject: [erlang-questions] Does erlang:now() guarantee that subsequent calls to this BIF returns continuously increasing values even in erlang cluster? In-Reply-To: <20150420145111.GB16522@ferdair.local> References: <3F7FBF7F-3FE4-432E-B2F6-BB4898CE03A7@gmail.com> <20150420125419.GA16522@ferdair.local> <20150420145111.GB16522@ferdair.local> Message-ID: "Lamport/vector clocks and other similar ones operate on *causality*, but this partial ordering is not the only one available or workable." Whether it's "workable" depends on what's desired. Sorting by {Node, Timestamp} is not accurate if causality matters and clocks have drifted out of synch. As they will. Hence Lamport's work, and the work of others. And if causality doesn't matter, well, I wonder: why bother? Unless you just want a rough idea of when certain things happened, in which case {Node, Timestamp} can give you a /total/ order that's, if anything, more accurate than what you need. Regards, Michael Turner Executive Director Project Persephone K-1 bldg 3F 7-2-6 Nishishinjuku Shinjuku-ku Tokyo 160-0023 Tel: +81 (3) 6890-1140 Fax: +81 (3) 6890-1158 Mobile: +81 (90) 5203-8682 turner@REDACTED http://www.projectpersephone.org/ "Love does not consist in gazing at each other, but in looking outward together in the same direction." -- Antoine de Saint-Exup?ry On Mon, Apr 20, 2015 at 11:51 PM, Fred Hebert wrote: > On 04/20, Michael Turner wrote: > >> If you can make do with a partial order rather than a total order, >> somehow: >> >> http://www.erlang.org/doc/man/seq_trace.html >> >> Basically, Lamport clocks. >> >> > Lamport clocks are one form of it. Sorting by {Node, Timestamp} also gives > you a partial order. So do vector clocks and version vectors, interval tree > clocks, and so on. > > Lamport/vector clocks and other similar ones operate on *causality*, but > this partial ordering is not the only one available or workable. > > You could have them operating on users, nodes, shards, clusters, > geographical regions, customers, or any other partitioning mechanism > desired, for example. > > It's important to look at the system and its desired properties to be able > to figure out which constraints allows us to best fit it to find the most > appropriate solution possible. > > Regards, > Fred. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vladdu55@REDACTED Tue Apr 21 13:53:49 2015 From: vladdu55@REDACTED (Vlad Dumitrescu) Date: Tue, 21 Apr 2015 13:53:49 +0200 Subject: [erlang-questions] erl_scan issues Message-ID: Hi! I found some unexpected behaviour for erl_scan and I hope someone can shed a light as to if it's supposed to be like that or it's a bug. IMHO the latter applies. > erl_scan:string("a. b",{1,1},[return,text]). {ok,[{atom,[{line,1},{column,1},{text,"a"}],a}, {dot,[{line,1},{column,2},{text,". "}]}, {atom,[{line,1},{column,4},{text,"b"}],b}], {1,5}} In short, the first newline or whitespace after a dot is included in the textual representation of the token. Why would anyone have that? regards, Vlad -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric.pailleau@REDACTED Tue Apr 21 14:04:25 2015 From: eric.pailleau@REDACTED (=?ISO-8859-1?Q?=C9ric_Pailleau?=) Date: Tue, 21 Apr 2015 14:04:25 +0200 Subject: [erlang-questions] erl_scan issues In-Reply-To: Message-ID: <7d5a4148-353d-4eae-b506-0c39b11d4f02@email.android.com> Hi. I suppose because a dot, as well any further blanks and a newline is the normal end of an Erlang term. Using scan for another purpose may result in this unespected behaviour. Regards Le?21 avr. 2015 13:53, Vlad Dumitrescu a ?crit?: > > Hi! > > I found some unexpected behaviour for erl_scan and I hope someone can shed a light as to if it's supposed to be like that or it's a bug. IMHO the latter applies. > > > erl_scan:string("a. b",{1,1},[return,text]). > {ok,[{atom,[{line,1},{column,1},{text,"a"}],a}, > ? ? ?{dot,[{line,1},{column,2},{text,". "}]}, > ? ? ?{atom,[{line,1},{column,4},{text,"b"}],b}], > ? ? {1,5}} > > In short, the first newline or whitespace after a dot is included in the textual representation of the token. Why would anyone have that? > > regards, > Vlad > From vladdu55@REDACTED Tue Apr 21 14:15:54 2015 From: vladdu55@REDACTED (Vlad Dumitrescu) Date: Tue, 21 Apr 2015 14:15:54 +0200 Subject: [erlang-questions] erl_scan issues In-Reply-To: <7d5a4148-353d-4eae-b506-0c39b11d4f02@email.android.com> References: <7d5a4148-353d-4eae-b506-0c39b11d4f02@email.android.com> Message-ID: Hi! Why should the first whitespace be part of the dot token and not a separate white_space token? The dot is recognized anyway (and it can also be followed by a comment, in which case the % character is correctly kept together with the rest of the comment). Of course I can handle this case specially, but it feels that it is unnecessary to have special cases without a meaningful reason. If there is a reason, I suppose it should be documented in the file, like other conventions about how the syntax is handled are. regards, Vlad On Tue, Apr 21, 2015 at 2:04 PM, ?ric Pailleau wrote: > Hi. > I suppose because a dot, as well any further blanks and a newline is the > normal end of an Erlang term. > Using scan for another purpose may result in this unespected behaviour. > Regards > > > > Le 21 avr. 2015 13:53, Vlad Dumitrescu a ?crit : > > > > Hi! > > > > I found some unexpected behaviour for erl_scan and I hope someone can > shed a light as to if it's supposed to be like that or it's a bug. IMHO the > latter applies. > > > > > erl_scan:string("a. b",{1,1},[return,text]). > > {ok,[{atom,[{line,1},{column,1},{text,"a"}],a}, > > {dot,[{line,1},{column,2},{text,". "}]}, > > {atom,[{line,1},{column,4},{text,"b"}],b}], > > {1,5}} > > > > In short, the first newline or whitespace after a dot is included in the > textual representation of the token. Why would anyone have that? > > > > regards, > > Vlad > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mononcqc@REDACTED Tue Apr 21 14:20:18 2015 From: mononcqc@REDACTED (Fred Hebert) Date: Tue, 21 Apr 2015 08:20:18 -0400 Subject: [erlang-questions] Does erlang:now() guarantee that subsequent calls to this BIF returns continuously increasing values even in erlang cluster? In-Reply-To: References: <3F7FBF7F-3FE4-432E-B2F6-BB4898CE03A7@gmail.com> <20150420125419.GA16522@ferdair.local> <20150420145111.GB16522@ferdair.local> Message-ID: <20150421122017.GG16522@ferdair.local> On 04/21, Michael Turner wrote: >"Lamport/vector clocks and other similar ones operate on *causality*, but >this partial ordering is not the only one available or workable." > >Whether it's "workable" depends on what's desired. Sorting by {Node, >Timestamp} is not accurate if causality matters and clocks have drifted out >of synch. As they will. Hence Lamport's work, and the work of others. And >if causality doesn't matter, well, I wonder: why bother? Unless you just >want a rough idea of when certain things happened, in which case {Node, >Timestamp} can give you a /total/ order that's, if anything, more accurate >than what you need. > That's not necessarily true. Let's see for different options and when they can be useful. - Lamport/vector clocks: causality. I wan to track the logical dependencies of changes. - `{Node, Timestamp}`: I have lots of local events (say HTTP requests and responses in logs) and want to see *when* they happen and how far apart. The timestmap might need to be monotonic, but the per-node value lets me impose a logical order, track some density over time (assuming I at least have NTP working), and so on. - {Shard, Timestamp}: I require a total order, but for events within a sharded data set. - {Cluster, Timestamp}: Each cluster I run might belong to specific customers or whatever, or run a specific set of hardware, or be a logical division. In any case, it's possible they have their own time or id service and I may want a partial or total order based on the events within that cluster, without worrying I might want to compare cross-cluster activity. - {Region, Timestamp}: Similar to the above, but by geographical area. I might decide that I need a total order on some form of transactions and will run a service, but for latency (and if real world allows it), I won't try to synchronize my time across data-centers or large geographical areas. All of these 'labelled timestamps' *are* a partial order. They only define it on some label. I.e. you can sort all timestamps within a node/shard/cluster/region, but can't do it across boundaries. There are other avenues that even combine some of them; One interesting case is inspired by Google's Chubby and CRDTs: You use a timestamp synchronized by NTP, guaranteeing you a maximal drift interval. You then add in a lamport clock whenever two events happen within too close of an interval that we cannot guarantee from the system clocks they truly happened apart. The lamport clock is mergeable in a deterministic way that is also commutative and idempotent (that's a CRDT!), and acts as a tie-breaker between events that happen at too close together. This way you get reliable timestamps when you can, and when you suddenly can't, you get a form of causality (or global monotonicity) to break things up. slapping "lamport clock" on it is reductive. It's a good way to track some levels of causality, but has its limitations. If you only *need* node-local accuracy and you have access to a monotonic clock, it might be far less work to just slap the monotonic clock into things than weave the logical clock through everything, and obtain the same logical result in the end (plus more information). Maybe it's not the best solution either. But really, if we want to make good recommendations, we have to ask what the user needs. Not come with a pet solution to push through. From john.foldager@REDACTED Tue Apr 21 15:03:40 2015 From: john.foldager@REDACTED (John Foldager) Date: Tue, 21 Apr 2015 15:03:40 +0200 Subject: [erlang-questions] Possibly to change TLS record size? Message-ID: We're running RabbitMQ (which is using Erlang underneath). I tried to post a question about TLS record size to the RabbitMQ mailinglist, but was directed to Erlang instead, so here goes... Using RabbitMQ I would like to know if it is possible somehow to configure/set the running Erlang process to change the size of the TLS records? a) What is the default TLS record size in Erlang (RabbitMQ)? b) Can this be changed with a configuration setting? c) If changed to something below 16K (which is the max as per specification) will/can Erlang (RabbitMQ) change it on the fly? Refs: Question on RabbitMQ mailinglist: https://groups.google.com/forum/#!topic/rabbitmq-users/tPHUMprMioM TLS record size: https://www.igvita.com/2013/10/24/optimizing-tls-record-size-and-buffering-latency/ From 44290016@REDACTED Tue Apr 21 15:15:01 2015 From: 44290016@REDACTED (=?gb18030?B?vMW8xQ==?=) Date: Tue, 21 Apr 2015 21:15:01 +0800 Subject: [erlang-questions] [Riak] Why Riak search only return 1 result every query Message-ID: Dear All, I have many objects with the same "tag:hot" stored in Riak cluster with two nodes, all with "leveldb" backend. When I try to search with the same query condition, I just got one result returned, and everytime with different result: curl -v "http://192.168.1.102:10028/search/query/videos?wt=json&q=tag:hot" {"responseHeader":{"status":0,"QTime":18,"params":{"shards":"127.0.0.1:10014/internal_solr/videos,127.0.0.1:10024/internal_solr/videos","q":"tag:hot","127.0.0.1:10014":"_yz_pn:64 OR (_yz_pn:61 AND (_yz_fpn:61)) OR _yz_pn:60 OR _yz_pn:57 OR _yz_pn:48 OR _yz_pn:45 OR _yz_pn:36 OR _yz_pn:33 OR _yz_pn:24 OR _yz_pn:21 OR _yz_pn:12 OR _yz_pn:9","wt":"json","127.0.0.1:10024":"_yz_pn:54 OR _yz_pn:51 OR _yz_pn:42 OR _yz_pn:39 OR _yz_pn:30 OR _yz_pn:27 OR _yz_pn:18 OR _yz_pn:15 OR _yz_pn:6 OR _yz_pn:3"}},"response":{"numFound":1,"start":0,"maxScore":0.71231794,"docs":[{"title":"Love Story","code":"8vBcw6rJCS6D6til2u","_yz_id":"1*video*movie*8vBcw6rJCS6D6til2u*48","_yz_rk":"8vBcw6rJCS6D6til2u","_yz_rt":"video","_yz_rb":"movie"}]}} curl -v "http://192.168.1.102:10028/search/query/videos?wt=json&q=tag:hot" {"responseHeader":{"status":0,"QTime":14,"params":{"shards":"127.0.0.1:10014/internal_solr/videos,127.0.0.1:10024/internal_solr/videos","q":"tag:hot","127.0.0.1:10014":"(_yz_pn:60 AND (_yz_fpn:60)) OR _yz_pn:56 OR _yz_pn:53 OR _yz_pn:44 OR _yz_pn:41 OR _yz_pn:32 OR _yz_pn:29 OR _yz_pn:20 OR _yz_pn:17 OR _yz_pn:8 OR _yz_pn:5","wt":"json","127.0.0.1:10024":"_yz_pn:63 OR _yz_pn:59 OR _yz_pn:50 OR _yz_pn:47 OR _yz_pn:38 OR _yz_pn:35 OR _yz_pn:26 OR _yz_pn:23 OR _yz_pn:14 OR _yz_pn:11 OR _yz_pn:2"}},"response":{"numFound":1,"start":0,"maxScore":0.71231794,"docs":[{"title":"Forest Gan","code":"8vBhSeP2Q2hCEEGmsC","_yz_id":"1*video*movie*8vBhSeP2Q2hCEEGmsC*20","_yz_rk":"8vBhSeP2Q2hCEEGmsC","_yz_rt":"video","_yz_rb":"movie"}]}} How can I get all at once query? -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric.pailleau@REDACTED Tue Apr 21 15:33:01 2015 From: eric.pailleau@REDACTED (=?ISO-8859-1?Q?=C9ric_Pailleau?=) Date: Tue, 21 Apr 2015 15:33:01 +0200 Subject: [erlang-questions] erl_scan issues In-Reply-To: Message-ID: An HTML attachment was scrubbed... URL: From ulf@REDACTED Tue Apr 21 16:12:24 2015 From: ulf@REDACTED (Ulf Wiger) Date: Tue, 21 Apr 2015 16:12:24 +0200 Subject: [erlang-questions] exometer core only In-Reply-To: References: <65FC5D9B-BEA3-4DB2-BF49-376117489742@feuerlabs.com> <5524E079.2050609@gmail.com> Message-ID: <8A520EA2-4F39-4B79-A065-0ED6C52003D7@feuerlabs.com> Exometer-1.2 (perhaps a higher version had been preferable) now uses only tagged versions (exometer_core-1.2 and exometer_collectd-1.0) BR, Ulf W > On 21 Apr 2015, at 07:09, Tino Breddin wrote: > > Hi. > > Agreed. I think we are ready for a new version since most work recently has been to harden exometer. We will look at it shortly. > > Tino > > On Apr 21, 2015 5:54 AM, "Martin Karlsson" > wrote: > Hi Tino, > > > exometer itself (versions later than 1.1) depends directly on exometer_core > > Following up on this. I need some of the reporters in exometer. However the latest tagged release is 1.1 which is not using exometer_core. Further to this exometer uses {branch, "master"} in rebar.config to pull a dependency. > > It is very hard to use this in production as we are developing against a moving target. > > Is there a better way to fix this? Ideally tag exometer with a fixed version of exometer_core? > > Cheers, > Martin > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. http://feuerlabs.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonetsu@REDACTED Tue Apr 21 15:48:37 2015 From: jonetsu@REDACTED (jonetsu) Date: Tue, 21 Apr 2015 09:48:37 -0400 Subject: [erlang-questions] OTP in FIPS mode ? Message-ID: <276c51c38412b4daff5fc26285f48f2c@teksavvy.com> Hello, We are using an Erlang-based middleware using OTP, ConfD, which must now support FIPS mode.? Briefly, FIPS is a U.S. standard that imposes a set of crypto parameters (ciphers, algorithms, etc...).? FIPS-applications must use high-level OpenSSL methods (The EVP set of methods) since the low-level functions will make OpenSSL abort.? The application must also call FIPS_mode_set(1) to enable this mode for a suitable OpenSSL build that supports FIPS. OTP uses low-level OpenSSL functions. Initially I considered replacing, for instance, the AES_* uses in crypto.c by their EVP equivalent, while keeping the interface to Erlang intact. Now, looking at the extent of the FIPS modifications to the OTP code done last year by D?niel Szoboszlay, who worked at Ericsson and Erlang Solutions, I wonder about my na?ve approach. Are anyone here familiar with this FIPS OTP port ?? Any comments ? To anyone also familiar with ConfD: do you know of any effort done in using this FIPS-enabled OTP code ? Thanks for any comments and suggestions ! Regards. From roger@REDACTED Tue Apr 21 16:33:27 2015 From: roger@REDACTED (Roger Lipscombe) Date: Tue, 21 Apr 2015 15:33:27 +0100 Subject: [erlang-questions] Possibly to change TLS record size? In-Reply-To: References: Message-ID: On 21 April 2015 at 14:03, John Foldager wrote: > Using RabbitMQ I would like to know if it is possible somehow to > configure/set the running Erlang process to change the size of the TLS > records? Do you mean an equivalent to OpenSSL's SSL_CTRL_SET_MAX_SEND_FRAGMENT option? As far as I know, it's not possible to set it in Erlang. Way back in Jan 2014, I had a patch (see https://github.com/rlipscombe/otp/commit/71c53d20191d3ddf43fc0aa87fabf5ac84ef70f3), but it didn't make it into OTP -- I had some feedback from Ingela on the OTP team, but didn't get around to dealing with it -- and I've not updated it since. From n.oxyde@REDACTED Tue Apr 21 16:53:34 2015 From: n.oxyde@REDACTED (Anthony Ramine) Date: Tue, 21 Apr 2015 16:53:34 +0200 Subject: [erlang-questions] erl_scan issues In-Reply-To: References: <7d5a4148-353d-4eae-b506-0c39b11d4f02@email.android.com> Message-ID: <4EC6D6FE-C445-4CEF-BB99-E08C5D731CAB@gmail.com> Le 21 avr. 2015 ? 14:15, Vlad Dumitrescu a ?crit : > The dot is recognized anyway Not always. f() -> #foo.bar g() -> 3.14 From vladdu55@REDACTED Tue Apr 21 17:08:11 2015 From: vladdu55@REDACTED (Vlad Dumitrescu) Date: Tue, 21 Apr 2015 17:08:11 +0200 Subject: [erlang-questions] erl_scan issues In-Reply-To: <4EC6D6FE-C445-4CEF-BB99-E08C5D731CAB@gmail.com> References: <7d5a4148-353d-4eae-b506-0c39b11d4f02@email.android.com> <4EC6D6FE-C445-4CEF-BB99-E08C5D731CAB@gmail.com> Message-ID: On Tue, Apr 21, 2015 at 4:53 PM, Anthony Ramine wrote: > Le 21 avr. 2015 ? 14:15, Vlad Dumitrescu a ?crit : > > The dot is recognized anyway > > Not always. > > f() -> #foo.bar > g() -> 3.14 What I mean is that the dot token doesn't have to contain the whitespace in it in order to get recognized. Like for the % character, the whitespace doesn't need to get consumed and will be part of the next token. This will give more consistency too, as the newlines are otherwise always first in a white_space token. After digging in the code, I believe that I have an explanation, but am not sure if it's correct. It seems that this is an artifact of how the reentrant lexer (erl_scan:tokens) is implemented. Usually the end of the input is 'eof', but if the input is a string we don't know if the input is finished and we should return 'ok', or if there may be more input and return 'more'. Having the string end with ". " (dot space) will have the implementation return 'ok'. This is even mentioned in the docs for erl_scan:tokens. I would prefer for such cases to end the input with 'eof' (String ++ eof) which is already done in several places in erl_scan, in the compiler, in dialyzer. regards, Vlad -------------- next part -------------- An HTML attachment was scrubbed... URL: From antoine.koener@REDACTED Tue Apr 21 17:20:03 2015 From: antoine.koener@REDACTED (Antoine Koener) Date: Tue, 21 Apr 2015 17:20:03 +0200 Subject: [erlang-questions] Streaming Data using httpc In-Reply-To: <30987936.26037.1429527924762.JavaMail.defaultUser@defaultHost> References: <30987936.26037.1429527924762.JavaMail.defaultUser@defaultHost> Message-ID: > Le 20 avr. 2015 ? 13:05, John Duffy a ?crit : > > Hi Jesper > > Thank you for your reply, very helpful. > > I'm still a bit puzzled as to why my test doesn't work, I'm being to wonder if I need to be sending additional headers, or something, to the server. > > If I use 'curl' then everything works, I get a steady stream of data... > > curl "http://stream-sandbox.oanda.com/v1/prices?accountId=99999&instruments=EUR_USD" The problem seems to be the server because if you use -v for curl you will observe something like this: < HTTP/1.1 200 Ok * Server openresty/1.7.0.1 is not blacklisted < Server: openresty/1.7.0.1 < Date: Tue, 21 Apr 2015 15:15:16 GMT < Content-Type: application/json < Transfer-Encoding: chunked < Connection: close < Access-Control-Allow-Origin: * < {"tick":{"instrument":"EUR_CHF","time":"2015-04-21T15:13:48.585046Z","bid":1.2041,"ask":1.20435}} {"heartbeat":{"time":"2015-04-21T15:15:16.632703Z"}} {"heartbeat":{"time":"2015-04-21T15:15:21.632772Z"}} {"heartbeat":{"time":"2015-04-21T15:15:24.598956Z"}} As you can see there's some strange headers: Connection: close The connection is not closed because it's a stream. Transfer-Encoding: chunked What I see is absolutely not chunked transfer, it's a bunch of json lines... Chunks should be preceded by the size (hex encoded) and \r\n So I think that the erlang code is trying to respect headers, close the connection and search for chunk encoding, but there's none... It might be interesting to report those problems to the developers of this service :) > > However, putting the same URL into your example results in a single time-out and the Erlang emulator stalling. > > Kind regards > > John > > ----Original message---- > From : jesper.louis.andersen@REDACTED > Date : 19/04/2015 - 22:00 (GMTDT) > To : jb_duffy@REDACTED, erlang-questions@REDACTED > Subject : Re: [erlang-questions] Streaming Data using httpc > > > > On Sun, Apr 19, 2015 at 6:14 PM John Duffy > wrote: > > > -module(streaming). > > > [...] > > You are pretty close to the goal, but you are confusing the stream/receiver options I think. In streaming, you will receive the data as a series of chunks, which is what your code expect, but you don't supply an option requesting streaming operation. So you don't retrieve an expected tuple. You can add a catchall tuple to your receieve clause in receive_data/1 to make sure you have the right format in your match. Also, you can add a 'after' clause to time out after a while. This can make debugging easier since you "get back" to the the REPL. > > The following works on a quick test in my end. Note how the receive clause is different from yours, and that you get everything in one fell swoop, rather than having to match on a multitude of clauses. > > For more serious work, you might want to check out some of the numerous other projects for HTTP client requests. I'm partial to Gun and Hackney myself, but there are also ibrowse, lhttpc and fusco. They have slightly different semantics and areas at which they excel, so choose wisely :) > > For a prototype however, I think httpc is fine. > > -module(streaming). > > -export([data/0]). > > data() -> > {ok, RequestId} = httpc:request(get, {"example.com ", []}, [], [{sync, false}, {receiver, self()}]), > receive_data(RequestId). > > receive_data(RequestId) -> > receive > {http, {RequestId, {StatusLine, Headers, Body}}} -> > error_logger:info_report( > #{ > status => StatusLine, > headers => length(Headers), > body_size => byte_size(Body) > }), > ok > after 5000 -> > error_logger:info_report(timeout) > end. > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From mononcqc@REDACTED Tue Apr 21 17:41:05 2015 From: mononcqc@REDACTED (Fred Hebert) Date: Tue, 21 Apr 2015 11:41:05 -0400 Subject: [erlang-questions] Streaming Data using httpc In-Reply-To: References: <30987936.26037.1429527924762.JavaMail.defaultUser@defaultHost> Message-ID: <20150421154103.GJ16522@ferdair.local> On 04/21, Antoine Koener wrote: > >The problem seems to be the server because if you use -v for curl you will observe something like this: > > [...] > >As you can see there's some strange headers: >Connection: close >The connection is not closed because it's a stream. The connection:close header is used to say that once the request is done, and to avoid attempting to reuse it as keep-alive for a follow-up request. There's nothing bad about it, and it is not relevant to the fact you are streaming. The streaming is part of chunked encoding (which has no content length known ahead of time, and is self-delimiting), related to the body, not the connection. > >Transfer-Encoding: chunked >What I see is absolutely not chunked transfer, it's a bunch of json lines... >Chunks should be preceded by the size (hex encoded) and \r\n > That's because you need to use `--raw` in curl to avoid seeing the content body already decoded. `-v` will decode chunked content and display it as it comes. You can also try a tcpdump to see the raw data. >It might be interesting to report those problems to the developers of >this service :) > Don't. Nothing you specified there is actually a problem unless they advertise keep-alive connections forever. From nick@REDACTED Tue Apr 21 18:32:56 2015 From: nick@REDACTED (Niclas Eklund) Date: Tue, 21 Apr 2015 18:32:56 +0200 Subject: [erlang-questions] OTP in FIPS mode ? In-Reply-To: <276c51c38412b4daff5fc26285f48f2c@teksavvy.com> References: <276c51c38412b4daff5fc26285f48f2c@teksavvy.com> Message-ID: <55367BB8.2040800@tail-f.com> Hi! IMHO I think that it would be good if FIPS could supported by OTP, especially since the purpose of the FIPS standards are issued to ensure computer security and interoperability. I've seen a question about this at least once before on this list before - http://erlang.org/pipermail/erlang-questions/2012-April/065902.html But I don't know what became of it. Best regards, Nick On 04/21/2015 03:48 PM, jonetsu wrote: > Hello, > > We are using an Erlang-based middleware using OTP, ConfD, which > must now support FIPS mode. Briefly, FIPS is a U.S. standard > that imposes a set of crypto parameters (ciphers, algorithms, > etc...). FIPS-applications must use high-level OpenSSL > methods (The EVP set of methods) since the low-level functions > will make OpenSSL abort. The application must also call > FIPS_mode_set(1) to enable this mode for a suitable OpenSSL build > that supports FIPS. > > OTP uses low-level OpenSSL functions. > > Initially I considered replacing, for instance, the AES_* uses in > crypto.c by their EVP equivalent, while keeping the interface to > Erlang intact. > > Now, looking at the extent of the FIPS modifications to the OTP > code done last year by D?niel Szoboszlay, who worked at Ericsson > and Erlang Solutions, I wonder about my na?ve approach. > > Are anyone here familiar with this FIPS OTP port ? Any comments > ? To anyone also familiar with ConfD: do you know of any effort > done in using this FIPS-enabled OTP code ? > > Thanks for any comments and suggestions ! > > Regards. > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From drew.varner@REDACTED Tue Apr 21 20:22:05 2015 From: drew.varner@REDACTED (Drew Varner) Date: Tue, 21 Apr 2015 14:22:05 -0400 Subject: [erlang-questions] OTP in FIPS mode ? In-Reply-To: <55367BB8.2040800@tail-f.com> References: <276c51c38412b4daff5fc26285f48f2c@teksavvy.com> <55367BB8.2040800@tail-f.com> Message-ID: Here? s the discussion on a FIPS pull request that?s now closed: https://github.com/erlang/otp/pull/377 - Drew > On Apr 21, 2015, at 12:32 PM, Niclas Eklund wrote: > > Hi! > > IMHO I think that it would be good if FIPS could supported by OTP, especially since the purpose of the FIPS standards are issued to ensure computer security and interoperability. I've seen a question about this at least once before on this list before - http://erlang.org/pipermail/erlang-questions/2012-April/065902.html But I don't know what became of it. > > Best regards, > > Nick > > > On 04/21/2015 03:48 PM, jonetsu wrote: >> Hello, >> >> We are using an Erlang-based middleware using OTP, ConfD, which >> must now support FIPS mode. Briefly, FIPS is a U.S. standard >> that imposes a set of crypto parameters (ciphers, algorithms, >> etc...). FIPS-applications must use high-level OpenSSL >> methods (The EVP set of methods) since the low-level functions >> will make OpenSSL abort. The application must also call >> FIPS_mode_set(1) to enable this mode for a suitable OpenSSL build >> that supports FIPS. >> >> OTP uses low-level OpenSSL functions. >> >> Initially I considered replacing, for instance, the AES_* uses in >> crypto.c by their EVP equivalent, while keeping the interface to >> Erlang intact. >> >> Now, looking at the extent of the FIPS modifications to the OTP >> code done last year by D?niel Szoboszlay, who worked at Ericsson >> and Erlang Solutions, I wonder about my na?ve approach. >> >> Are anyone here familiar with this FIPS OTP port ? Any comments >> ? To anyone also familiar with ConfD: do you know of any effort >> done in using this FIPS-enabled OTP code ? >> >> Thanks for any comments and suggestions ! >> >> Regards. >> >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From jb_duffy@REDACTED Tue Apr 21 23:26:35 2015 From: jb_duffy@REDACTED (John Duffy) Date: Tue, 21 Apr 2015 22:26:35 +0100 (BST) Subject: [erlang-questions] Streaming Data using httpc In-Reply-To: <20150421154103.GJ16522@ferdair.local> References: <30987936.26037.1429527924762.JavaMail.defaultUser@defaultHost> <20150421154103.GJ16522@ferdair.local> Message-ID: <18144405.85218.1429651595326.JavaMail.defaultUser@defaultHost> Fred, Antoine Thank you for your replies. I will investigate further... Kind regards John ----Original message---- >From : mononcqc@REDACTED Date : 21/04/2015 - 16:41 (GMTDT) To : antoine.koener@REDACTED Cc : jb_duffy@REDACTED, erlang-questions@REDACTED Subject : Re: [erlang-questions] Streaming Data using httpc On 04/21, Antoine Koener wrote: > >The problem seems to be the server because if you use -v for curl you will observe something like this: > > [...] > >As you can see there's some strange headers: >Connection: close >The connection is not closed because it's a stream. The connection:close header is used to say that once the request is done, and to avoid attempting to reuse it as keep-alive for a follow-up request. There's nothing bad about it, and it is not relevant to the fact you are streaming. The streaming is part of chunked encoding (which has no content length known ahead of time, and is self-delimiting), related to the body, not the connection. > >Transfer-Encoding: chunked >What I see is absolutely not chunked transfer, it's a bunch of json lines... >Chunks should be preceded by the size (hex encoded) and \r\n > That's because you need to use `--raw` in curl to avoid seeing the content body already decoded. `-v` will decode chunked content and display it as it comes. You can also try a tcpdump to see the raw data. >It might be interesting to report those problems to the developers of >this service :) > Don't. Nothing you specified there is actually a problem unless they advertise keep-alive connections forever. From jb_duffy@REDACTED Tue Apr 21 23:34:26 2015 From: jb_duffy@REDACTED (John Duffy) Date: Tue, 21 Apr 2015 22:34:26 +0100 (BST) Subject: [erlang-questions] Streaming Data using httpc In-Reply-To: <20150421154103.GJ16522@ferdair.local> References: <30987936.26037.1429527924762.JavaMail.defaultUser@defaultHost> <20150421154103.GJ16522@ferdair.local> Message-ID: <3632090.85594.1429652066973.JavaMail.defaultUser@defaultHost> Fred By the way... a great book! It is constantly open on my desk at the moment as I try to get to grips with Erlang. Kind regards John Duffy ----Original message---- >From : mononcqc@REDACTED Date : 21/04/2015 - 16:41 (GMTDT) To : antoine.koener@REDACTED Cc : jb_duffy@REDACTED, erlang-questions@REDACTED Subject : Re: [erlang-questions] Streaming Data using httpc On 04/21, Antoine Koener wrote: > >The problem seems to be the server because if you use -v for curl you will observe something like this: > > [...] > >As you can see there's some strange headers: >Connection: close >The connection is not closed because it's a stream. The connection:close header is used to say that once the request is done, and to avoid attempting to reuse it as keep-alive for a follow-up request. There's nothing bad about it, and it is not relevant to the fact you are streaming. The streaming is part of chunked encoding (which has no content length known ahead of time, and is self-delimiting), related to the body, not the connection. > >Transfer-Encoding: chunked >What I see is absolutely not chunked transfer, it's a bunch of json lines... >Chunks should be preceded by the size (hex encoded) and \r\n > That's because you need to use `--raw` in curl to avoid seeing the content body already decoded. `-v` will decode chunked content and display it as it comes. You can also try a tcpdump to see the raw data. >It might be interesting to report those problems to the developers of >this service :) > Don't. Nothing you specified there is actually a problem unless they advertise keep-alive connections forever. From ok@REDACTED Wed Apr 22 02:35:47 2015 From: ok@REDACTED (Richard A. O'Keefe) Date: Wed, 22 Apr 2015 12:35:47 +1200 Subject: [erlang-questions] erl_scan issues In-Reply-To: <7d5a4148-353d-4eae-b506-0c39b11d4f02@email.android.com> References: <7d5a4148-353d-4eae-b506-0c39b11d4f02@email.android.com> Message-ID: Erlang syntax is adapted from Prolog syntax. It is traditional in Prolog parsers to distinguish between a "." token such as you might find in a.b.[] (the really old-fashioned way to write a list) and a ". " token which ends a clause. And it turns out that erl_scan:string/3 makes exactly the same distinction: "a. b" contains a ". " (dot) token while "a.b" contains a "." ('.') token. Now a full stop at the end of a string is also a dot token, but it has text ".". White space as such is never a token. This does not look like a bug at all to me. I will say that it would be nice if the http://www.erlang.org/doc/man/erl_scan.html page contained or linked to an explicit statement of what the tokens ARE. At a minimum, the type category() should be a bit more explicit than "atom()". For that matter, http://www.erlang.org/doc/man/erl_parse.html should contain or link to an explicit statement of what the grammar is. From michael.eugene.turner@REDACTED Wed Apr 22 02:59:57 2015 From: michael.eugene.turner@REDACTED (Michael Turner) Date: Wed, 22 Apr 2015 09:59:57 +0900 Subject: [erlang-questions] Does erlang:now() guarantee that subsequent calls to this BIF returns continuously increasing values even in erlang cluster? In-Reply-To: <20150421122017.GG16522@ferdair.local> References: <3F7FBF7F-3FE4-432E-B2F6-BB4898CE03A7@gmail.com> <20150420125419.GA16522@ferdair.local> <20150420145111.GB16522@ferdair.local> <20150421122017.GG16522@ferdair.local> Message-ID: "slapping "lamport clock" on it is reductive." -- it was just a suggestion, in case something like that might work. My suggestion was not intended to be all-inclusive or a panacea. Slapping "reductive" on my suggestion is ... well, reductive? "Not come with a pet solution to push through." It's not my "pet solution". I don't have a "pet solution." I don't even know what this guy's problem is, and you don't either. So how can I have a pet solution if there's no way to know what the solution is in the first place? I just happen to know that Erlang/OTP has this feature that's been in "beta" for what seems to be a decade or more, one that exposes a kind of Lamport clock functionality (even though the documentation fails to call it that, which might be why it's still waiting for enough user input to refine the interface -- people who go looking for something like that in Erlang/OTP are not finding it.) seq_trace might be part of /a/ solution to his problem. But since we don't know what his problem really is, I'm just making suggestions. Understand? Regards, Michael Turner Executive Director Project Persephone K-1 bldg 3F 7-2-6 Nishishinjuku Shinjuku-ku Tokyo 160-0023 Tel: +81 (3) 6890-1140 Fax: +81 (3) 6890-1158 Mobile: +81 (90) 5203-8682 turner@REDACTED http://www.projectpersephone.org/ "Love does not consist in gazing at each other, but in looking outward together in the same direction." -- Antoine de Saint-Exup?ry On Tue, Apr 21, 2015 at 9:20 PM, Fred Hebert wrote: > On 04/21, Michael Turner wrote: > >> "Lamport/vector clocks and other similar ones operate on *causality*, but >> this partial ordering is not the only one available or workable." >> >> Whether it's "workable" depends on what's desired. Sorting by {Node, >> Timestamp} is not accurate if causality matters and clocks have drifted >> out >> of synch. As they will. Hence Lamport's work, and the work of others. And >> if causality doesn't matter, well, I wonder: why bother? Unless you just >> want a rough idea of when certain things happened, in which case {Node, >> Timestamp} can give you a /total/ order that's, if anything, more accurate >> than what you need. >> >> > That's not necessarily true. Let's see for different options and when they > can be useful. > > - Lamport/vector clocks: causality. I wan to track the logical > dependencies of changes. > - `{Node, Timestamp}`: I have lots of local events (say HTTP requests and > responses in logs) and want to see *when* they happen and how far apart. > The timestmap might need to be monotonic, but the per-node value lets me > impose a logical order, track some density over time (assuming I at least > have NTP working), and so on. > - {Shard, Timestamp}: I require a total order, but for events within a > sharded data set. > - {Cluster, Timestamp}: Each cluster I run might belong to specific > customers or whatever, or run a specific set of hardware, or be a logical > division. In any case, it's possible they have their own time or id > service and I may want a partial or total order based on the events within > that cluster, without worrying I might want to compare cross-cluster > activity. > - {Region, Timestamp}: Similar to the above, but by geographical area. I > might decide that I need a total order on some form of transactions and > will run a service, but for latency (and if real world allows it), I won't > try to synchronize my time across data-centers or large geographical areas. > > All of these 'labelled timestamps' *are* a partial order. They only define > it on some label. I.e. you can sort all timestamps within a > node/shard/cluster/region, but can't do it across boundaries. > > There are other avenues that even combine some of them; One interesting > case is inspired by Google's Chubby and CRDTs: You use a timestamp > synchronized by NTP, guaranteeing you a maximal drift interval. You then > add in a lamport clock whenever two events happen within too close of an > interval that we cannot guarantee from the system clocks they truly > happened apart. > > The lamport clock is mergeable in a deterministic way that is also > commutative and idempotent (that's a CRDT!), and acts as a tie-breaker > between events that happen at too close together. > > This way you get reliable timestamps when you can, and when you suddenly > can't, you get a form of causality (or global monotonicity) to break things > up. > > slapping "lamport clock" on it is reductive. It's a good way to track some > levels of causality, but has its limitations. If you only *need* node-local > accuracy and you have access to a monotonic clock, it might be far less > work to just slap the monotonic clock into things than weave the logical > clock through everything, and obtain the same logical result in the end > (plus more information). Maybe it's not the best solution either. > > But really, if we want to make good recommendations, we have to ask what > the user needs. Not come with a pet solution to push through. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From liudanking@REDACTED Wed Apr 22 03:34:12 2015 From: liudanking@REDACTED (Daniel) Date: Wed, 22 Apr 2015 09:34:12 +0800 Subject: [erlang-questions] Does erlang:now() guarantee that subsequent calls to this BIF returns continuously increasing values even in erlang cluster? In-Reply-To: References: <3F7FBF7F-3FE4-432E-B2F6-BB4898CE03A7@gmail.com> <20150420125419.GA16522@ferdair.local> <20150420145111.GB16522@ferdair.local> <20150421122017.GG16522@ferdair.local> Message-ID: <63E1059C-981E-4954-92D6-51C226592A80@gmail.com> The background of my problem is as follows: I use AWS dynamoDB (http://aws.amazon.com/cn/dynamodb/) to store ejabbed chat messages. DynamoDB is a NoSQL storage and needs a *key* to look up a record. In my application, I construct the *key* as where jid is the hash key and erlang:now() is the range key (http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/WorkingWithTables.html). The reason that I prefer to use erlang:now() as the range key is that it will be convenient to query history messages according to timestamp. Unfortunately, may be not unique in erlang cluster, so I am looking for a global *erlang:now()* function that generate global unique timestamp in erlang cluster. BTW, small probability of collision is not a big problem in my application, so I am still using as the *key*. > On Apr 22, 2015, at 8:59 AM, Michael Turner wrote: > > "slapping "lamport clock" on it is reductive." > > -- it was just a suggestion, in case something like that might work. My suggestion was not intended to be all-inclusive or a panacea. Slapping "reductive" on my suggestion is ... well, reductive? > > "Not come with a pet solution to push through." > > It's not my "pet solution". I don't have a "pet solution." I don't even know what this guy's problem is, and you don't either. So how can I have a pet solution if there's no way to know what the solution is in the first place? > > I just happen to know that Erlang/OTP has this feature that's been in "beta" for what seems to be a decade or more, one that exposes a kind of Lamport clock functionality (even though the documentation fails to call it that, which might be why it's still waiting for enough user input to refine the interface -- people who go looking for something like that in Erlang/OTP are not finding it.) seq_trace might be part of /a/ solution to his problem. But since we don't know what his problem really is, I'm just making suggestions. > > Understand? > > > > Regards, > Michael Turner > Executive Director > Project Persephone > K-1 bldg 3F > 7-2-6 Nishishinjuku > Shinjuku-ku Tokyo 160-0023 > Tel: +81 (3) 6890-1140 > Fax: +81 (3) 6890-1158 > Mobile: +81 (90) 5203-8682 > turner@REDACTED > http://www.projectpersephone.org/ > > "Love does not consist in gazing at each other, but in looking outward together in the same direction." -- Antoine de Saint-Exup?ry > > On Tue, Apr 21, 2015 at 9:20 PM, Fred Hebert wrote: > On 04/21, Michael Turner wrote: > "Lamport/vector clocks and other similar ones operate on *causality*, but > this partial ordering is not the only one available or workable." > > Whether it's "workable" depends on what's desired. Sorting by {Node, > Timestamp} is not accurate if causality matters and clocks have drifted out > of synch. As they will. Hence Lamport's work, and the work of others. And > if causality doesn't matter, well, I wonder: why bother? Unless you just > want a rough idea of when certain things happened, in which case {Node, > Timestamp} can give you a /total/ order that's, if anything, more accurate > than what you need. > > > That's not necessarily true. Let's see for different options and when they can be useful. > > - Lamport/vector clocks: causality. I wan to track the logical dependencies of changes. > - `{Node, Timestamp}`: I have lots of local events (say HTTP requests and responses in logs) and want to see *when* they happen and how far apart. The timestmap might need to be monotonic, but the per-node value lets me impose a logical order, track some density over time (assuming I at least have NTP working), and so on. > - {Shard, Timestamp}: I require a total order, but for events within a sharded data set. > - {Cluster, Timestamp}: Each cluster I run might belong to specific customers or whatever, or run a specific set of hardware, or be a logical division. In any case, it's possible they have their own time or id service and I may want a partial or total order based on the events within that cluster, without worrying I might want to compare cross-cluster activity. > - {Region, Timestamp}: Similar to the above, but by geographical area. I might decide that I need a total order on some form of transactions and will run a service, but for latency (and if real world allows it), I won't try to synchronize my time across data-centers or large geographical areas. > > All of these 'labelled timestamps' *are* a partial order. They only define it on some label. I.e. you can sort all timestamps within a node/shard/cluster/region, but can't do it across boundaries. > > There are other avenues that even combine some of them; One interesting case is inspired by Google's Chubby and CRDTs: You use a timestamp synchronized by NTP, guaranteeing you a maximal drift interval. You then add in a lamport clock whenever two events happen within too close of an interval that we cannot guarantee from the system clocks they truly happened apart. > > The lamport clock is mergeable in a deterministic way that is also commutative and idempotent (that's a CRDT!), and acts as a tie-breaker between events that happen at too close together. > > This way you get reliable timestamps when you can, and when you suddenly can't, you get a form of causality (or global monotonicity) to break things up. > > slapping "lamport clock" on it is reductive. It's a good way to track some levels of causality, but has its limitations. If you only *need* node-local accuracy and you have access to a monotonic clock, it might be far less work to just slap the monotonic clock into things than weave the logical clock through everything, and obtain the same logical result in the end (plus more information). Maybe it's not the best solution either. > > But really, if we want to make good recommendations, we have to ask what the user needs. Not come with a pet solution to push through. > From mononcqc@REDACTED Wed Apr 22 05:35:30 2015 From: mononcqc@REDACTED (Fred Hebert) Date: Tue, 21 Apr 2015 23:35:30 -0400 Subject: [erlang-questions] Does erlang:now() guarantee that subsequent calls to this BIF returns continuously increasing values even in erlang cluster? In-Reply-To: References: <3F7FBF7F-3FE4-432E-B2F6-BB4898CE03A7@gmail.com> <20150420125419.GA16522@ferdair.local> <20150420145111.GB16522@ferdair.local> <20150421122017.GG16522@ferdair.local> Message-ID: <20150422033529.GB10369@ferdmbp.local> On 04/22, Michael Turner wrote: >Understand? Understood, yes :) From michael.eugene.turner@REDACTED Wed Apr 22 05:58:08 2015 From: michael.eugene.turner@REDACTED (Michael Turner) Date: Wed, 22 Apr 2015 12:58:08 +0900 Subject: [erlang-questions] Does erlang:now() guarantee that subsequent calls to this BIF returns continuously increasing values even in erlang cluster? In-Reply-To: <63E1059C-981E-4954-92D6-51C226592A80@gmail.com> References: <3F7FBF7F-3FE4-432E-B2F6-BB4898CE03A7@gmail.com> <20150420125419.GA16522@ferdair.local> <20150420145111.GB16522@ferdair.local> <20150421122017.GG16522@ferdair.local> <63E1059C-981E-4954-92D6-51C226592A80@gmail.com> Message-ID: If what you need is not exactly a down-to-the-microsecond time stamp but a temporally ordered globally unique ID, this https://github.com/boundary/flake could be part of the solution, I suppose. Regards, Michael Turner Executive Director Project Persephone K-1 bldg 3F 7-2-6 Nishishinjuku Shinjuku-ku Tokyo 160-0023 Tel: +81 (3) 6890-1140 Fax: +81 (3) 6890-1158 Mobile: +81 (90) 5203-8682 turner@REDACTED http://www.projectpersephone.org/ "Love does not consist in gazing at each other, but in looking outward together in the same direction." -- Antoine de Saint-Exup?ry On Wed, Apr 22, 2015 at 10:34 AM, Daniel wrote: > The background of my problem is as follows: > > I use AWS dynamoDB (http://aws.amazon.com/cn/dynamodb/) to store ejabbed > chat messages. DynamoDB is a NoSQL storage and needs a *key* to look up a > record. In my application, I construct the *key* as > where jid is the hash key and erlang:now() is the range key ( > http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/WorkingWithTables.html). > The reason that I prefer to use erlang:now() as the range key is that it > will be convenient to query history messages according to timestamp. > Unfortunately, may be not unique in erlang cluster, so > I am looking for a global *erlang:now()* function that generate global > unique timestamp in erlang cluster. > > BTW, small probability of collision is not a big > problem in my application, so I am still using as the > *key*. > > > > On Apr 22, 2015, at 8:59 AM, Michael Turner < > michael.eugene.turner@REDACTED> wrote: > > > > "slapping "lamport clock" on it is reductive." > > > > -- it was just a suggestion, in case something like that might work. My > suggestion was not intended to be all-inclusive or a panacea. Slapping > "reductive" on my suggestion is ... well, reductive? > > > > "Not come with a pet solution to push through." > > > > It's not my "pet solution". I don't have a "pet solution." I don't even > know what this guy's problem is, and you don't either. So how can I have a > pet solution if there's no way to know what the solution is in the first > place? > > > > I just happen to know that Erlang/OTP has this feature that's been in > "beta" for what seems to be a decade or more, one that exposes a kind of > Lamport clock functionality (even though the documentation fails to call it > that, which might be why it's still waiting for enough user input to refine > the interface -- people who go looking for something like that in > Erlang/OTP are not finding it.) seq_trace might be part of /a/ solution to > his problem. But since we don't know what his problem really is, I'm just > making suggestions. > > > > Understand? > > > > > > > > Regards, > > Michael Turner > > Executive Director > > Project Persephone > > K-1 bldg 3F > > 7-2-6 Nishishinjuku > > Shinjuku-ku Tokyo 160-0023 > > Tel: +81 (3) 6890-1140 > > Fax: +81 (3) 6890-1158 > > Mobile: +81 (90) 5203-8682 > > turner@REDACTED > > http://www.projectpersephone.org/ > > > > "Love does not consist in gazing at each other, but in looking outward > together in the same direction." -- Antoine de Saint-Exup?ry > > > > On Tue, Apr 21, 2015 at 9:20 PM, Fred Hebert wrote: > > On 04/21, Michael Turner wrote: > > "Lamport/vector clocks and other similar ones operate on *causality*, but > > this partial ordering is not the only one available or workable." > > > > Whether it's "workable" depends on what's desired. Sorting by {Node, > > Timestamp} is not accurate if causality matters and clocks have drifted > out > > of synch. As they will. Hence Lamport's work, and the work of others. And > > if causality doesn't matter, well, I wonder: why bother? Unless you just > > want a rough idea of when certain things happened, in which case {Node, > > Timestamp} can give you a /total/ order that's, if anything, more > accurate > > than what you need. > > > > > > That's not necessarily true. Let's see for different options and when > they can be useful. > > > > - Lamport/vector clocks: causality. I wan to track the logical > dependencies of changes. > > - `{Node, Timestamp}`: I have lots of local events (say HTTP requests > and responses in logs) and want to see *when* they happen and how far > apart. The timestmap might need to be monotonic, but the per-node value > lets me impose a logical order, track some density over time (assuming I > at least have NTP working), and so on. > > - {Shard, Timestamp}: I require a total order, but for events within a > sharded data set. > > - {Cluster, Timestamp}: Each cluster I run might belong to specific > customers or whatever, or run a specific set of hardware, or be a logical > division. In any case, it's possible they have their own time or id > service and I may want a partial or total order based on the events within > that cluster, without worrying I might want to compare cross-cluster > activity. > > - {Region, Timestamp}: Similar to the above, but by geographical area. > I might decide that I need a total order on some form of transactions and > will run a service, but for latency (and if real world allows it), I won't > try to synchronize my time across data-centers or large geographical areas. > > > > All of these 'labelled timestamps' *are* a partial order. They only > define it on some label. I.e. you can sort all timestamps within a > node/shard/cluster/region, but can't do it across boundaries. > > > > There are other avenues that even combine some of them; One interesting > case is inspired by Google's Chubby and CRDTs: You use a timestamp > synchronized by NTP, guaranteeing you a maximal drift interval. You then > add in a lamport clock whenever two events happen within too close of an > interval that we cannot guarantee from the system clocks they truly > happened apart. > > > > The lamport clock is mergeable in a deterministic way that is also > commutative and idempotent (that's a CRDT!), and acts as a tie-breaker > between events that happen at too close together. > > > > This way you get reliable timestamps when you can, and when you suddenly > can't, you get a form of causality (or global monotonicity) to break things > up. > > > > slapping "lamport clock" on it is reductive. It's a good way to track > some levels of causality, but has its limitations. If you only *need* > node-local accuracy and you have access to a monotonic clock, it might be > far less work to just slap the monotonic clock into things than weave the > logical clock through everything, and obtain the same logical result in the > end (plus more information). Maybe it's not the best solution either. > > > > But really, if we want to make good recommendations, we have to ask what > the user needs. Not come with a pet solution to push through. > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vladdu55@REDACTED Wed Apr 22 09:13:56 2015 From: vladdu55@REDACTED (Vlad Dumitrescu) Date: Wed, 22 Apr 2015 09:13:56 +0200 Subject: [erlang-questions] erl_scan issues In-Reply-To: References: <7d5a4148-353d-4eae-b506-0c39b11d4f02@email.android.com> Message-ID: Thank you Richard for the historical perspective. On Wed, Apr 22, 2015 at 2:35 AM, Richard A. O'Keefe wrote: > Erlang syntax is adapted from Prolog syntax. > It is traditional in Prolog parsers to distinguish > between a "." token such as you might find in > a.b.[] (the really old-fashioned way to write a list) > and a ". " token which ends a clause. > And it turns out that erl_scan:string/3 makes exactly > the same distinction: "a. b" contains a ". " (dot) token > while "a.b" contains a "." ('.') token. Now a full stop > at the end of a string is also a dot token, but it has > text ".". And a ".%" input also creates a dot token with a "." text. > White space as such is never a token. > It is if the scanner receives a 'return_whitespace" option. This is needed (together with the 'text' option) if one must be able to recreate the original source exactly as it was. > This does not look like a bug at all to me. > I can agree that it is a stretch to call it a bug, but it should be better specified. It is unexpected that some whitespace, especially newline, is part of the 'dot' token. I would have left it separate and had a special case for the opposite operation, i.e. add a whitespace after a 'dot' when reconstructing the source (if the token list doesn't already include whitespace tokens). best regards, Vlad > I will say that it would be nice if the > http://www.erlang.org/doc/man/erl_scan.html > page contained or linked to an explicit statement > of what the tokens ARE. > > At a minimum, the type category() should be a bit > more explicit than "atom()". > > For that matter, > http://www.erlang.org/doc/man/erl_parse.html > should contain or link to an explicit statement > of what the grammar is. > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eshikafe@REDACTED Wed Apr 22 10:36:43 2015 From: eshikafe@REDACTED (austin aigbe) Date: Wed, 22 Apr 2015 09:36:43 +0100 Subject: [erlang-questions] Erlang/OTP Complete SS7 Stack Message-ID: Hi, Any plans to include a complete ss7 stack as part of the Erlang/OTP library? Regards, Austin -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto@REDACTED Wed Apr 22 15:06:35 2015 From: roberto@REDACTED (Roberto Ostinelli) Date: Wed, 22 Apr 2015 15:06:35 +0200 Subject: [erlang-questions] Common Tests Message-ID: Dear list, I have a project with a standard OTP structure. Main app is in `apps/myapp`. My CT are in `apps/myapp/test`. When I run a simple CT file: ==================================== $ rebar ct skip_deps=true ==> myapp (ct) DONE. Testing apps.myapp: TEST COMPLETE, 1 ok, 0 failed of 1 test cases ==================================== Then a log directory gets created in `apps/myapp/logs` which contain all CT results. However, when I run the tests again I get: ==================================== $ rebar ct skip_deps=true ==> myapp (ct) ERROR: ct_run -noshell -pa "/Users/roberto/workspace/myapp/apps/myapp/ebin" "/Users/roberto/workspace/myapp/deps/edown/ebin" "/Users/roberto/workspace/myapp/deps/rabbit_common/ebin" "/Users/roberto/workspace/myapp/deps/meck/ebin" "/Users/roberto/workspace/myapp/deps/bear/ebin" "/Users/roberto/workspace/myapp/deps/lager/ebin" "/Users/roberto/workspace/myapp/deps/setup/ebin" "/Users/roberto/workspace/myapp/deps/jiffy/ebin" "/Users/roberto/workspace/myapp/deps/amqp_client/ebin" "/Users/roberto/workspace/myapp/deps/folsom/ebin" "/Users/roberto/workspace/myapp/deps/netlink/ebin" "/Users/roberto/workspace/myapp/deps/afunix/ebin" "/Users/roberto/workspace/myapp/deps/parse_trans/ebin" "/Users/roberto/workspace/myapp/deps/ranch/ebin" "/Users/roberto/workspace/myapp/deps/cowlib/ebin" "/Users/roberto/workspace/myapp/deps/goldrush/ebin" "/Users/roberto/workspace/myapp/deps/exometer/ebin" "/Users/roberto/workspace/myapp/deps/recon/ebin" "/Users/roberto/workspace/myapp/deps/cowboy/ebin" "/usr/local/bin/rebar/rebar/ebin" "/usr/local/bin/rebar" "/Users/roberto/workspace/myapp/." -name test@REDACTED -logdir "/Users/roberto/workspace/myapp/apps/myapp/logs" -env TEST_DIR "/Users/roberto/workspace/myapp/apps/myapp/test" -dir test >> /Users/roberto/workspace/myapp/apps/myapp/logs/raw.log 2>&1 failed with error: 2 and output: ERROR: ct failed while processing /Users/roberto/workspace/myapp/apps/myapp: rebar_abort ==================================== If I delete the `apps/myapp/logs` directory, I can re-run my tests with no problem. logs/raw.log` contains: ==================================== --- Test run on 2015/04/22 15:03:18 --- Converting "/Users/roberto/workspace/myapp/." to "/Users/roberto/workspace/myapp" and re-inserting with add_patha/1 Common Test v1.10 starting (cwd is /Users/roberto/workspace/myapp/apps/myapp) Common Test: Running make in test directories... CWD set to: "/Users/roberto/workspace/myapp/apps/myapp/logs/ct_run.test@REDACTED " TEST INFO: 1 test(s), 1 case(s) in 1 suite(s) Testing apps.myapp: Starting test, 1 test cases Testing apps.myapp: TEST COMPLETE, 1 ok, 0 failed of 1 test cases =ERROR REPORT==== 22-Apr-2015::15:03:19 === Error in process <0.37.0> on node 'test@REDACTED' with exit value: {{badmatch,["ct_run","test@REDACTED ","local","2015-04-22_14","59","52"]},[{ct_logs,'-sort_ct_runs/1-fun-0-',2,[{file,"ct_logs.erl"},{line,1912}]},{lists,sort,2,[{file,"lists.erl"},{line,967}]},{ct_logs,make_all_suites_index,1,[{file... Test run crashed! This could be an internal error - please report! {{badmatch,["ct_run","test@REDACTED","local","2015-04-22_14","59","52"]}, [{ct_logs,'-sort_ct_runs/1-fun-0-',2,[{file,"ct_logs.erl"},{line,1912}]}, {lists,sort,2,[{file,"lists.erl"},{line,967}]}, {ct_logs,make_all_suites_index,1,[{file,"ct_logs.erl"},{line,2232}]}, {ct_logs,close,2,[{file,"ct_logs.erl"},{line,178}]}, {ct_util,loop,3,[{file,"ct_util.erl"},{line,471}]}]} ==================================== ...Any ideas? Best, r. -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.goertzen@REDACTED Wed Apr 22 15:52:26 2015 From: daniel.goertzen@REDACTED (Daniel Goertzen) Date: Wed, 22 Apr 2015 08:52:26 -0500 Subject: [erlang-questions] wrapperless NIF module? Message-ID: I was thinking about the little Erlang wrapper that you need around a NIF module and then it hit me: Could the Erlang wrapper be thrown out and the NIF module plopped into ebin/ and directly loaded like a .beam file? I'm curious if this has been attempted or is even possible. Thanks, Dan. -------------- next part -------------- An HTML attachment was scrubbed... URL: From lloyd@REDACTED Wed Apr 22 17:19:44 2015 From: lloyd@REDACTED (lloyd@REDACTED) Date: Wed, 22 Apr 2015 11:19:44 -0400 (EDT) Subject: [erlang-questions] YARQ -- Yet another rebar question Message-ID: <1429715984.96717945@apps.rackspace.com> Hello, This may seem like a brain-dead question to some, but it's a mystery to me: I would like to integrate erlguten (https://github.com/richcarl/erlguten) into my Nitrogen application. The richcarl version of erlguten is built with make. My Nitrogen app is built with rebar. How can I configure rebar to build richcarl/erluten as a dependency in my Nitrogen app? Many thanks, LRP From eric.pailleau@REDACTED Wed Apr 22 18:31:23 2015 From: eric.pailleau@REDACTED (=?ISO-8859-1?Q?=C9ric_Pailleau?=) Date: Wed, 22 Apr 2015 18:31:23 +0200 Subject: [erlang-questions] YARQ -- Yet another rebar question In-Reply-To: <1429715984.96717945@apps.rackspace.com> Message-ID: Hi, https://github.com/crownedgrouse/rebar.fake Regards Le?22 avr. 2015 17:19, lloyd@REDACTED a ?crit?: > > Hello, > > This may seem like a brain-dead question to some, but it's a mystery to me: > > I would like to integrate erlguten (https://github.com/richcarl/erlguten) into my Nitrogen application. The richcarl version of erlguten is built with make. My Nitrogen app is built with rebar. > > How can I configure rebar to build richcarl/erluten as a dependency in my Nitrogen app? > > Many thanks, > > LRP > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From tomorrow_is_waiting_for_your_smile_@REDACTED Thu Apr 23 01:27:21 2015 From: tomorrow_is_waiting_for_your_smile_@REDACTED (nat n) Date: Wed, 22 Apr 2015 23:27:21 +0000 Subject: [erlang-questions] Messenger example Message-ID: I am having trouble running the example from this tutorial:http://www.erlang.org/doc/getting_started/conc_prog.html#id67907In specifics, the "A Larger Example" and the one before that. I have two computers on the same network that are running Mininet/Ubuntu (using Virtual Box).What confuses me is this: "(Note: erl -sname assumes that all nodes are in the same IP domain and we can use only the first component of the IP address, if we want to use nodes in different domains we use -name instead, but then all IP address must be given in full.)"How should I then call "erl", what options need to be placed when starting erlang? -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric.pailleau@REDACTED Thu Apr 23 07:09:28 2015 From: eric.pailleau@REDACTED (=?ISO-8859-1?Q?=C9ric_Pailleau?=) Date: Thu, 23 Apr 2015 07:09:28 +0200 Subject: [erlang-questions] Messenger example In-Reply-To: Message-ID: <248058bc-1c9a-4464-8f87-572f918aa0cc@email.android.com> Hi, You just need to give a different name for both nodes, let's say n1 and n2. 'erl -sname n1' for the first 'erl -sname n2' for the second Verify both nodes has the same cookie. Regards Le?23 avr. 2015 01:27, nat n a ?crit?: > > I am having trouble running the example from this tutorial: > http://www.erlang.org/doc/getting_started/conc_prog.html#id67907 > In specifics, the "A Larger Example" and the one before that. > > I have two computers on the same network that are running?Mininet/Ubuntu (using Virtual Box). > What confuses me is this: "(Note:?erl -sname?assumes that all nodes are in the same IP domain and we can use only the first component of the IP address, if we want to use nodes in different domains we use?-name?instead, but then all IP address must be given in full.)" > How should I then call "erl", what options need to be placed when starting erlang? ? From tuncer.ayaz@REDACTED Thu Apr 23 09:33:09 2015 From: tuncer.ayaz@REDACTED (Tuncer Ayaz) Date: Thu, 23 Apr 2015 09:33:09 +0200 Subject: [erlang-questions] YARQ -- Yet another rebar question In-Reply-To: <1429715984.96717945@apps.rackspace.com> References: <1429715984.96717945@apps.rackspace.com> Message-ID: On Wed, Apr 22, 2015 at 5:19 PM, wrote: > Hello, > > This may seem like a brain-dead question to some, but it's a mystery > to me: > > I would like to integrate erlguten > (https://github.com/richcarl/erlguten) into my Nitrogen application. > The richcarl version of erlguten is built with make. My Nitrogen app > is built with rebar. > > How can I configure rebar to build richcarl/erluten as a dependency > in my Nitrogen app? I've submitted a trivial patch[1] changing a placeholder in erlguten.app.src to be valid Erlang syntax. When that's merged, you can add this to rebar.config: %% Fetch erlguten as a dependency. {deps, [ {erlguten, ".*", {git, "https://github.com/richcarl/erlguten"}} ]}. %% Right after erlguten is fetched, build it correctly. {post_hooks, [ {'get-deps', 'make -C deps/erlguten'} ]}. [1] https://github.com/richcarl/erlguten/pull/4 From russelldb@REDACTED Thu Apr 23 11:20:07 2015 From: russelldb@REDACTED (Russell Brown) Date: Thu, 23 Apr 2015 10:20:07 +0100 Subject: [erlang-questions] file:consult {error, {1, file_io_server, invalid_unicode}} with pre-r17 files in r17 Message-ID: Hi, With the release of r17, certain files that were written in earlier versions of erlang are not `file:consult/1`-able in r17. Here is a simple module that shows the issue: https://gist.github.com/russelldb/4f8fe205bd5a5a1cd136 If I change line 13 to io_lib:format(?~w.", [Term]), Everything works just fine. However, legacy files were written in r16 using io_lib:format("~p.", [Term]), And I?d like to be able to read them into erlang terms in r17, has someone already tackled this? Many thanks in advance for your suggestions Russell From zandra@REDACTED Thu Apr 23 11:47:04 2015 From: zandra@REDACTED (Zandra Hird) Date: Thu, 23 Apr 2015 11:47:04 +0200 Subject: [erlang-questions] Patch Package OTP 17.5.2 Released Message-ID: <5538BF98.7050804@erlang.org> Patch Package: OTP 17.5.2 Git Tag: OTP-17.5.2 Check out the git tag OTP-17.5.2, and build a full OTP system including documentation. Apply one or more applications from this build as patches to your installation using the 'otp_patch_apply' tool. For information on install requirements, see descriptions for each application version below. --------------------------------------------------------------------- --- inets-5.10.7 ---------------------------------------------------- --------------------------------------------------------------------- The inets-5.10.7 application can be applied independently of other applications on a full OTP 17 installation. --- Improvements and New Features --- OTP-12661 Application(s): inets Related Id(s): seq12840 New value in server_tokens config for limiting banner grabbing attempts. By setting {server_tokens, none} in ServiceConfig for inets:start(httpd, ServiceConfig), the "Server:" header will not be set in messages from the server. Full runtime dependencies of inets-5.10.7: erts-6.0, kernel-3.0, mnesia-4.12, runtime_tools-1.8.14, ssl-5.3.4, stdlib-2.0 --------------------------------------------------------------------- --- ssh-3.2.2 ------------------------------------------------------- --------------------------------------------------------------------- Note! The ssh-3.2.2 application can *not* be applied independently of other applications on an arbitrary OTP 17 installation. On a full OTP 17 installation, also the following runtime dependency has to be satisfied: -- stdlib-2.3 (first satisfied in OTP 17.4) --- Improvements and New Features --- OTP-12659 Application(s): ssh New option id_string for ssh:daemon and ssh:connect for limiting banner grabbing attempts. The possible values are: {id_string,string()} and {id_string,random}. The latter will make ssh generate a random nonsence id-string for each new connection. Full runtime dependencies of ssh-3.2.2: crypto-3.3, erts-6.0, kernel-3.0, public_key-0.22, stdlib-2.3 --------------------------------------------------------------------- --------------------------------------------------------------------- --------------------------------------------------------------------- From roberto@REDACTED Thu Apr 23 12:26:50 2015 From: roberto@REDACTED (Roberto Ostinelli) Date: Thu, 23 Apr 2015 12:26:50 +0200 Subject: [erlang-questions] Common Tests In-Reply-To: References: Message-ID: This seems to be an erlang bug. Reported it here: http://erlang.org/pipermail/erlang-bugs/2015-April/004911.html Best, r. On Wed, Apr 22, 2015 at 3:06 PM, Roberto Ostinelli wrote: > Dear list, > I have a project with a standard OTP structure. Main app is in > `apps/myapp`. > My CT are in `apps/myapp/test`. > > When I run a simple CT file: > > ==================================== > $ rebar ct skip_deps=true > ==> myapp (ct) > DONE. > Testing apps.myapp: TEST COMPLETE, 1 ok, 0 failed of 1 test cases > ==================================== > > > Then a log directory gets created in `apps/myapp/logs` which contain all > CT results. > However, when I run the tests again I get: > > > ==================================== > $ rebar ct skip_deps=true > ==> myapp (ct) > ERROR: ct_run -noshell -pa > "/Users/roberto/workspace/myapp/apps/myapp/ebin" > "/Users/roberto/workspace/myapp/deps/edown/ebin" > "/Users/roberto/workspace/myapp/deps/rabbit_common/ebin" > "/Users/roberto/workspace/myapp/deps/meck/ebin" > "/Users/roberto/workspace/myapp/deps/bear/ebin" > "/Users/roberto/workspace/myapp/deps/lager/ebin" > "/Users/roberto/workspace/myapp/deps/setup/ebin" > "/Users/roberto/workspace/myapp/deps/jiffy/ebin" > "/Users/roberto/workspace/myapp/deps/amqp_client/ebin" > "/Users/roberto/workspace/myapp/deps/folsom/ebin" > "/Users/roberto/workspace/myapp/deps/netlink/ebin" > "/Users/roberto/workspace/myapp/deps/afunix/ebin" > "/Users/roberto/workspace/myapp/deps/parse_trans/ebin" > "/Users/roberto/workspace/myapp/deps/ranch/ebin" > "/Users/roberto/workspace/myapp/deps/cowlib/ebin" > "/Users/roberto/workspace/myapp/deps/goldrush/ebin" > "/Users/roberto/workspace/myapp/deps/exometer/ebin" > "/Users/roberto/workspace/myapp/deps/recon/ebin" > "/Users/roberto/workspace/myapp/deps/cowboy/ebin" > "/usr/local/bin/rebar/rebar/ebin" "/usr/local/bin/rebar" > "/Users/roberto/workspace/myapp/." -name test@REDACTED > -logdir "/Users/roberto/workspace/myapp/apps/myapp/logs" -env TEST_DIR > "/Users/roberto/workspace/myapp/apps/myapp/test" -dir test >> > /Users/roberto/workspace/myapp/apps/myapp/logs/raw.log 2>&1 failed with > error: 2 and output: > > ERROR: ct failed while processing > /Users/roberto/workspace/myapp/apps/myapp: rebar_abort > ==================================== > > > If I delete the `apps/myapp/logs` directory, I can re-run my tests with no > problem. > logs/raw.log` contains: > > > ==================================== > --- Test run on 2015/04/22 15:03:18 --- > Converting "/Users/roberto/workspace/myapp/." to > "/Users/roberto/workspace/myapp" and re-inserting with add_patha/1 > > > Common Test v1.10 starting (cwd is > /Users/roberto/workspace/myapp/apps/myapp) > > > Common Test: Running make in test directories... > > CWD set to: > "/Users/roberto/workspace/myapp/apps/myapp/logs/ct_run.test@REDACTED > " > > TEST INFO: 1 test(s), 1 case(s) in 1 suite(s) > > Testing apps.myapp: Starting test, 1 test cases > Testing apps.myapp: TEST COMPLETE, 1 ok, 0 failed of 1 test cases > > > =ERROR REPORT==== 22-Apr-2015::15:03:19 === > Error in process <0.37.0> on node 'test@REDACTED' with exit value: > {{badmatch,["ct_run","test@REDACTED > ","local","2015-04-22_14","59","52"]},[{ct_logs,'-sort_ct_runs/1-fun-0-',2,[{file,"ct_logs.erl"},{line,1912}]},{lists,sort,2,[{file,"lists.erl"},{line,967}]},{ct_logs,make_all_suites_index,1,[{file... > > Test run crashed! This could be an internal error - please report! > > {{badmatch,["ct_run","test@REDACTED","local","2015-04-22_14","59","52"]}, > [{ct_logs,'-sort_ct_runs/1-fun-0-',2,[{file,"ct_logs.erl"},{line,1912}]}, > {lists,sort,2,[{file,"lists.erl"},{line,967}]}, > {ct_logs,make_all_suites_index,1,[{file,"ct_logs.erl"},{line,2232}]}, > {ct_logs,close,2,[{file,"ct_logs.erl"},{line,178}]}, > {ct_util,loop,3,[{file,"ct_util.erl"},{line,471}]}]} > ==================================== > > > ...Any ideas? > > Best, > r. > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ali.sabil@REDACTED Thu Apr 23 12:46:56 2015 From: ali.sabil@REDACTED (Ali Sabil) Date: Thu, 23 Apr 2015 10:46:56 +0000 Subject: [erlang-questions] Why does Dialyzer crash on this map? In-Reply-To: References: <54749C90.6000602@llaisdy.com> <5474BA4C.4030706@llaisdy.com> <547716F8.7010208@llaisdy.com> <4B9AEB77-A709-4D97-8EF2-8FE1AF5A4071@llaisdy.com> <55085881.3060508@erlang.org> <55095D8E.2050103@erlang.org> Message-ID: Any update regarding this? On Thu, Apr 9, 2015 at 5:21 PM Ali Sabil wrote: > Sorry for the late reply, I finally managed to get a minimal test case > that reproduces the bug in 17.5: > > -module(sum). > -export([ > test/1 > ]). > > -spec test(#{atom() => term()}) -> integer(). > test(Data) -> > maps:fold(fun > (_Key, Value, Acc) when is_integer(Value) -> > Acc + Value; > (_Key, _Value, Acc) -> > Acc > end, 0, Data). > > > I don't know if this is the correct fix, but this makes dialyzer work > again: > > diff --git a/lib/hipe/cerl/erl_types.erl b/lib/hipe/cerl/erl_types.erl > index 4215448..bb4c1c1 100644 > --- a/lib/hipe/cerl/erl_types.erl > +++ b/lib/hipe/cerl/erl_types.erl > @@ -4594,6 +4594,8 @@ t_form_to_string({type, _L, list, [Type]}) -> > "[" ++ t_form_to_string(Type) ++ "]"; > t_form_to_string({type, _L, map, Args}) when not is_list(Args) -> > "#{}"; > +t_form_to_string({type, _L, map_field_assoc, Key, Value}) -> > + "#{" ++ t_form_to_string(Key) ++ "=>" ++ t_form_to_string(Value) ++ "}"; > t_form_to_string({type, _L, mfa, []}) -> "mfa()"; > t_form_to_string({type, _L, module, []}) -> "module()"; > t_form_to_string({type, _L, node, []}) -> "node()"; > > Thanks, > Are > > On Tue, Apr 7, 2015 at 2:31 PM Bj?rn-Egil Dahlberg < > wallentin.dahlberg@REDACTED> wrote: > Again - could you provide me with a sample code .. or at least some sort of >> a clue to what you are dialyzing? >> >> 2015-04-07 14:05 GMT+02:00 Ali Sabil : >> > Hi again,Running dialyzer shipped with 17.5 on the same code base leads >>> now to the following error (17.4 works without any errors): >>> >> >>> >>> >>> ===> Error in dialyzing apps: Analysis failed with error: >>> {function_clause,[{erl_types,t_form_to_string, >>> [{type,36,map_field_assoc, >>> {type,36,atom,[]}, >>> {type,36,term,[]}}], >>> [{file,"erl_types.erl"},{line,4546}]}, >>> {erl_types,t_form_to_string_list,2, >>> [{file,"erl_types.erl"},{line,4637}]}, >>> {erl_types,t_form_to_string,1, >>> [{file,"erl_types.erl"},{line,4634}]}, >>> {erl_types,t_form_to_string_list,2, >>> [{file,"erl_types.erl"},{line,4637}]}, >>> {erl_types,t_form_to_string,1, >>> [{file,"erl_types.erl"},{line,4634}]}, >>> {dialyzer_contracts,contract_to_string_1,1, >>> [{file,"dialyzer_contracts.erl"}, >>> {line,107}]}, >>> {dialyzer_contracts,extra_contract_warning,6, >>> [{file,"dialyzer_contracts.erl"}, >>> {line,712}]}, >>> {dialyzer_contracts,picky_contract_check,7, >>> [{file,"dialyzer_contracts.erl"}, >>> {line,686}]}]} >>> Last messages in the log cache: >>> Reading files and computing callgraph... done in 1.21 secs >>> Removing edges... done in 0.04 secs >>> >>> >>> On Wed, Mar 18, 2015 at 12:12 PM Bj?rn-Egil Dahlberg >>> wrote: >>> >>>> On 2015-03-18 12:01, Ali Sabil wrote: >>>> > I tried to create a minimal testcase but I unfortunately haven't been >>>> > able to. I was running dialyzer on a quite large code base and now >>>> > even the unpatched dialyzer works without any issue after I fixed all >>>> > the issues reported by dialyzer. Ah, well .. I suspect it was the >>>> missing clause in find_terminals and I had it on a TODO somewhere. >>>> Should be included to 17.5. // Bj?rn-Egil >>>> >>>> >>>> >>>> >>>> >>>> >>> _______________________________________________ >>> erlang-questions mailing listerlang-questions@REDACTED >>> http://erlang.org/mailman/list >>> >>> info/erlang-questions >>> >>> >>> >>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto@REDACTED Thu Apr 23 13:41:44 2015 From: roberto@REDACTED (Roberto Ostinelli) Date: Thu, 23 Apr 2015 13:41:44 +0200 Subject: [erlang-questions] CT and test.config Message-ID: All, I've got an app `myapp` which uses a standard `sys.config` file. I'd like to test it using Common Tests, with a test configuration file. I'm passing the variable `-config test.config` to ct_run. However, when I start my app from the Common Test suite `myapp_SUITE.erl` then the app fails to start because inside the app `application:get_env(myapp, mykey)` returns `undefined`. I can, however, read this variable with `ct:get_config({myapp, mykey})`. How am I supposed to use a custom `test.config` file that is loaded in ct_run *and* read by the app itself? I've spent almost a day on this and I'm honestly 5 minutes away from dropping all this in the dustbin and using an external test tool instead. Please help me stick to Erlang. :) r. -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Thu Apr 23 13:53:01 2015 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Thu, 23 Apr 2015 14:53:01 +0300 Subject: [erlang-questions] CT and test.config In-Reply-To: References: Message-ID: <5538DD1D.6010703@ninenines.eu> ct_run -erl_args -config test.config On 04/23/2015 02:41 PM, Roberto Ostinelli wrote: > All, > I've got an app `myapp` which uses a standard `sys.config` file. > I'd like to test it using Common Tests, with a test configuration file. > > I'm passing the variable `-config test.config` to ct_run. > > However, when I start my app from the Common Test suite > `myapp_SUITE.erl` then the app fails to start because inside the app > `application:get_env(myapp, mykey)` returns `undefined`. I can, however, > read this variable with `ct:get_config({myapp, mykey})`. > > How am I supposed to use a custom `test.config` file that is loaded in > ct_run *and* read by the app itself? > > I've spent almost a day on this and I'm honestly 5 minutes away from > dropping all this in the dustbin and using an external test tool > instead. Please help me stick to Erlang. :) > > r. > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin http://ninenines.eu From peppe@REDACTED Thu Apr 23 14:03:16 2015 From: peppe@REDACTED (Peter Andersson) Date: Thu, 23 Apr 2015 14:03:16 +0200 Subject: [erlang-questions] CT and test.config In-Reply-To: <5538DD1D.6010703@ninenines.eu> References: <5538DD1D.6010703@ninenines.eu> Message-ID: <5538DF84.2020002@erlang.org> Roberto, you can read about -erl_args here: http://www.erlang.org/doc/man/ct_run.html The CT flag config is unfortunately named, but we decided to keep the name and rely on the -erl_args feature. With -erl_args we don't have to prefix all CT flags in order to avoid possible clashes with erl flags (now and in the future). Best, Peter On 2015-04-23 13:53, Lo?c Hoguin wrote: > ct_run -erl_args -config test.config > > On 04/23/2015 02:41 PM, Roberto Ostinelli wrote: > > All, > > I've got an app `myapp` which uses a standard `sys.config` file. > > I'd like to test it using Common Tests, with a test configuration file. > > > > I'm passing the variable `-config test.config` to ct_run. > > > > However, when I start my app from the Common Test suite > > `myapp_SUITE.erl` then the app fails to start because inside the app > > `application:get_env(myapp, mykey)` returns `undefined`. I can, however, > > read this variable with `ct:get_config({myapp, mykey})`. > > > > How am I supposed to use a custom `test.config` file that is loaded in > > ct_run *and* read by the app itself? > > > > I've spent almost a day on this and I'm honestly 5 minutes away from > > dropping all this in the dustbin and using an external test tool > > instead. Please help me stick to Erlang. :) > > > > r. > > > > > > > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > From roberto@REDACTED Thu Apr 23 15:24:50 2015 From: roberto@REDACTED (Roberto Ostinelli) Date: Thu, 23 Apr 2015 15:24:50 +0200 Subject: [erlang-questions] CT and test.config In-Reply-To: <5538DD1D.6010703@ninenines.eu> References: <5538DD1D.6010703@ninenines.eu> Message-ID: Thank you Lo?c, this works (i.e. `myapp` now starts) but now I'm unable to read this config file in `myapp_SUITE.erl`. `application:get_env(myapp, mykey)` and `ct:get_config({myapp, mykey})` both return `undefined`. On Thu, Apr 23, 2015 at 1:53 PM, Lo?c Hoguin wrote: > ct_run -erl_args -config test.config > > > On 04/23/2015 02:41 PM, Roberto Ostinelli wrote: > >> All, >> I've got an app `myapp` which uses a standard `sys.config` file. >> I'd like to test it using Common Tests, with a test configuration file. >> >> I'm passing the variable `-config test.config` to ct_run. >> >> However, when I start my app from the Common Test suite >> `myapp_SUITE.erl` then the app fails to start because inside the app >> `application:get_env(myapp, mykey)` returns `undefined`. I can, however, >> read this variable with `ct:get_config({myapp, mykey})`. >> >> How am I supposed to use a custom `test.config` file that is loaded in >> ct_run *and* read by the app itself? >> >> I've spent almost a day on this and I'm honestly 5 minutes away from >> dropping all this in the dustbin and using an external test tool >> instead. Please help me stick to Erlang. :) >> >> r. >> >> >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -- > Lo?c Hoguin > http://ninenines.eu > -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto@REDACTED Thu Apr 23 15:25:13 2015 From: roberto@REDACTED (Roberto Ostinelli) Date: Thu, 23 Apr 2015 15:25:13 +0200 Subject: [erlang-questions] CT and test.config In-Reply-To: <5538DF84.2020002@erlang.org> References: <5538DD1D.6010703@ninenines.eu> <5538DF84.2020002@erlang.org> Message-ID: Thank you for clarifying Peter. That was confusing. On Thu, Apr 23, 2015 at 2:03 PM, Peter Andersson wrote: > > Roberto, you can read about -erl_args here: > > http://www.erlang.org/doc/man/ct_run.html > > The CT flag config is unfortunately named, but we decided to keep the > name and rely on the -erl_args feature. With -erl_args we don't have to > prefix all CT flags in order to avoid possible clashes with erl flags > (now and in the future). > > Best, > Peter > > On 2015-04-23 13:53, Lo?c Hoguin wrote: > > ct_run -erl_args -config test.config > > > > On 04/23/2015 02:41 PM, Roberto Ostinelli wrote: > > > All, > > > I've got an app `myapp` which uses a standard `sys.config` file. > > > I'd like to test it using Common Tests, with a test configuration file. > > > > > > I'm passing the variable `-config test.config` to ct_run. > > > > > > However, when I start my app from the Common Test suite > > > `myapp_SUITE.erl` then the app fails to start because inside the app > > > `application:get_env(myapp, mykey)` returns `undefined`. I can, > however, > > > read this variable with `ct:get_config({myapp, mykey})`. > > > > > > How am I supposed to use a custom `test.config` file that is loaded in > > > ct_run *and* read by the app itself? > > > > > > I've spent almost a day on this and I'm honestly 5 minutes away from > > > dropping all this in the dustbin and using an external test tool > > > instead. Please help me stick to Erlang. :) > > > > > > r. > > > > > > > > > > > > > > > > > > _______________________________________________ > > > erlang-questions mailing list > > > erlang-questions@REDACTED > > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto@REDACTED Thu Apr 23 15:41:22 2015 From: roberto@REDACTED (Roberto Ostinelli) Date: Thu, 23 Apr 2015 15:41:22 +0200 Subject: [erlang-questions] CT and test.config In-Reply-To: References: <5538DD1D.6010703@ninenines.eu> Message-ID: My main need is to be able to modify app variables on the fly, to test different behaviors from CT. I'm fed up with this so I'm no instead initializing in my CT `init_per_suite/1` function all the application variables like this: set_environment_variables() -> % read config file ConfigFilePath = filename:join([filename:dirname(code:which(?MODULE)), "test.config"]), {ok, [AppsConfig]} = file:consult(ConfigFilePath), % loop to set variables F = fun({AppName, AppConfig}) -> set_environment_for_app(AppName, AppConfig) end, lists:foreach(F, AppsConfig). set_environment_for_app(AppName, AppConfig) -> F = fun({Key, Val}) -> application:set_env(AppName, Key, Val) end, lists:foreach(F, AppConfig). If anyone knows a better way, please let me know. Best, r. On Thu, Apr 23, 2015 at 3:24 PM, Roberto Ostinelli wrote: > Thank you Lo?c, this works (i.e. `myapp` now starts) but now I'm unable to > read this config file in `myapp_SUITE.erl`. > > `application:get_env(myapp, mykey)` and `ct:get_config({myapp, mykey})` > both return `undefined`. > > On Thu, Apr 23, 2015 at 1:53 PM, Lo?c Hoguin wrote: > >> ct_run -erl_args -config test.config >> >> >> On 04/23/2015 02:41 PM, Roberto Ostinelli wrote: >> >>> All, >>> I've got an app `myapp` which uses a standard `sys.config` file. >>> I'd like to test it using Common Tests, with a test configuration file. >>> >>> I'm passing the variable `-config test.config` to ct_run. >>> >>> However, when I start my app from the Common Test suite >>> `myapp_SUITE.erl` then the app fails to start because inside the app >>> `application:get_env(myapp, mykey)` returns `undefined`. I can, however, >>> read this variable with `ct:get_config({myapp, mykey})`. >>> >>> How am I supposed to use a custom `test.config` file that is loaded in >>> ct_run *and* read by the app itself? >>> >>> I've spent almost a day on this and I'm honestly 5 minutes away from >>> dropping all this in the dustbin and using an external test tool >>> instead. Please help me stick to Erlang. :) >>> >>> r. >>> >>> >>> >>> >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >> -- >> Lo?c Hoguin >> http://ninenines.eu >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From peppe@REDACTED Thu Apr 23 15:50:09 2015 From: peppe@REDACTED (Peter Andersson) Date: Thu, 23 Apr 2015 15:50:09 +0200 Subject: [erlang-questions] CT and test.config In-Reply-To: References: <5538DD1D.6010703@ninenines.eu> Message-ID: <5538F891.1000504@erlang.org> Well, if you want the same config file as input to both your application and to CT: ct_run -config test.config -erl_args -config test.config /Peter On 2015-04-23 15:24, Roberto Ostinelli wrote: > Thank you Lo?c, this works (i.e. `myapp` now starts) but now I'm > unable to read this config file in `myapp_SUITE.erl`. > > `application:get_env(myapp, mykey)` and `ct:get_config({myapp, > mykey})` both return `undefined`. > > On Thu, Apr 23, 2015 at 1:53 PM, Lo?c Hoguin > wrote: > > ct_run -erl_args -config test.config > > > On 04/23/2015 02:41 PM, Roberto Ostinelli wrote: > > All, > I've got an app `myapp` which uses a standard `sys.config` file. > I'd like to test it using Common Tests, with a test > configuration file. > > I'm passing the variable `-config test.config` to ct_run. > > However, when I start my app from the Common Test suite > `myapp_SUITE.erl` then the app fails to start because inside > the app > `application:get_env(myapp, mykey)` returns `undefined`. I > can, however, > read this variable with `ct:get_config({myapp, mykey})`. > > How am I supposed to use a custom `test.config` file that is > loaded in > ct_run *and* read by the app itself? > > I've spent almost a day on this and I'm honestly 5 minutes > away from > dropping all this in the dustbin and using an external test tool > instead. Please help me stick to Erlang. :) > > r. > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > -- > Lo?c Hoguin > http://ninenines.eu > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From ingela.andin@REDACTED Thu Apr 23 15:52:01 2015 From: ingela.andin@REDACTED (Ingela Andin) Date: Thu, 23 Apr 2015 15:52:01 +0200 Subject: [erlang-questions] OTP in FIPS mode ? In-Reply-To: References: <276c51c38412b4daff5fc26285f48f2c@teksavvy.com> <55367BB8.2040800@tail-f.com> Message-ID: Hi! This question is in the OTP 19 scope. FIPS requires the use of the EVP-API which sounds interesting regardless of FIPS. We see this as a possible first step, then we can consider if and what else we might want to do, to support or facilitate for others to support FIPS. My current understanding is that using only the EVP-API functions we can not make the crypto API completely functional, which will require at least some additions to the crypto API and perhaps OTP-techincal board considerations. Regards Ingela Erlang/OTP - Ericsson AB 2015-04-21 20:22 GMT+02:00 Drew Varner : > Here? s the discussion on a FIPS pull request that?s now closed: > https://github.com/erlang/otp/pull/377 > > - Drew > > > On Apr 21, 2015, at 12:32 PM, Niclas Eklund wrote: > > Hi! > > IMHO I think that it would be good if FIPS could supported by OTP, > especially since the purpose of the FIPS standards are issued to ensure > computer security and interoperability. I've seen a question about this at > least once before on this list before - > http://erlang.org/pipermail/erlang-questions/2012-April/065902.html But I > don't know what became of it. > > Best regards, > > Nick > > > On 04/21/2015 03:48 PM, jonetsu wrote: > > Hello, > > We are using an Erlang-based middleware using OTP, ConfD, which > must now support FIPS mode. Briefly, FIPS is a U.S. standard > that imposes a set of crypto parameters (ciphers, algorithms, > etc...). FIPS-applications must use high-level OpenSSL > methods (The EVP set of methods) since the low-level functions > will make OpenSSL abort. The application must also call > FIPS_mode_set(1) to enable this mode for a suitable OpenSSL build > that supports FIPS. > > OTP uses low-level OpenSSL functions. > > Initially I considered replacing, for instance, the AES_* uses in > crypto.c by their EVP equivalent, while keeping the interface to > Erlang intact. > > Now, looking at the extent of the FIPS modifications to the OTP > code done last year by D?niel Szoboszlay, who worked at Ericsson > and Erlang Solutions, I wonder about my na?ve approach. > > Are anyone here familiar with this FIPS OTP port ? Any comments > ? To anyone also familiar with ConfD: do you know of any effort > done in using this FIPS-enabled OTP code ? > > Thanks for any comments and suggestions ! > > Regards. > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto@REDACTED Thu Apr 23 15:56:43 2015 From: roberto@REDACTED (Roberto Ostinelli) Date: Thu, 23 Apr 2015 15:56:43 +0200 Subject: [erlang-questions] CT and test.config In-Reply-To: <5538F891.1000504@erlang.org> References: <5538DD1D.6010703@ninenines.eu> <5538F891.1000504@erlang.org> Message-ID: Unfortunately that doesn't work. `test.config` for application needs to be a file which encloses a list of Applications, i.e. [{App1, [App1Config]}, {App2, [App2Config]}]. `test.config` for CT needs to be a file which has individual entries *not* enclosed in a list, i.e. {App1, [App1Config]}. {App2, [App2Config]}. On Thu, Apr 23, 2015 at 3:50 PM, Peter Andersson wrote: > > Well, if you want the same config file as input to both your application > and to CT: > > ct_run -config test.config -erl_args -config test.config > > /Peter > > > On 2015-04-23 15:24, Roberto Ostinelli wrote: > > Thank you Lo?c, this works (i.e. `myapp` now starts) but now I'm unable to > read this config file in `myapp_SUITE.erl`. > > `application:get_env(myapp, mykey)` and `ct:get_config({myapp, mykey})` > both return `undefined`. > > On Thu, Apr 23, 2015 at 1:53 PM, Lo?c Hoguin wrote: > >> ct_run -erl_args -config test.config >> >> >> On 04/23/2015 02:41 PM, Roberto Ostinelli wrote: >> >>> All, >>> I've got an app `myapp` which uses a standard `sys.config` file. >>> I'd like to test it using Common Tests, with a test configuration file. >>> >>> I'm passing the variable `-config test.config` to ct_run. >>> >>> However, when I start my app from the Common Test suite >>> `myapp_SUITE.erl` then the app fails to start because inside the app >>> `application:get_env(myapp, mykey)` returns `undefined`. I can, however, >>> read this variable with `ct:get_config({myapp, mykey})`. >>> >>> How am I supposed to use a custom `test.config` file that is loaded in >>> ct_run *and* read by the app itself? >>> >>> I've spent almost a day on this and I'm honestly 5 minutes away from >>> dropping all this in the dustbin and using an external test tool >>> instead. Please help me stick to Erlang. :) >>> >>> r. >>> >>> >>> >>> >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >> -- >> Lo?c Hoguin >> http://ninenines.eu >> > > > > _______________________________________________ > erlang-questions mailing listerlang-questions@REDACTED://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From peppe@REDACTED Thu Apr 23 16:05:10 2015 From: peppe@REDACTED (Peter Andersson) Date: Thu, 23 Apr 2015 16:05:10 +0200 Subject: [erlang-questions] CT and test.config In-Reply-To: References: <5538DD1D.6010703@ninenines.eu> <5538F891.1000504@erlang.org> Message-ID: <5538FC16.30203@erlang.org> True, if you want the same config file as input to both, you need the "userconfig" feature in Common Test: http://www.erlang.org/doc/apps/common_test/config_file_chapter.html#id79185 http://www.erlang.org/doc/man/ct_run.html http://www.erlang.org/doc/man/ct.html#run_test-1 /Peter On 2015-04-23 15:56, Roberto Ostinelli wrote: > Unfortunately that doesn't work. > > `test.config` for application needs to be a file which encloses a list > of Applications, i.e. [{App1, [App1Config]}, {App2, [App2Config]}]. > `test.config` for CT needs to be a file which has individual entries > *not* enclosed in a list, i.e. {App1, [App1Config]}. {App2, [App2Config]}. > > On Thu, Apr 23, 2015 at 3:50 PM, Peter Andersson > wrote: > > > Well, if you want the same config file as input to both your > application and to CT: > > ct_run -config test.config -erl_args -config test.config > > /Peter > > > On 2015-04-23 15:24, Roberto Ostinelli wrote: >> Thank you Lo?c, this works (i.e. `myapp` now starts) but now I'm >> unable to read this config file in `myapp_SUITE.erl`. >> >> `application:get_env(myapp, mykey)` and `ct:get_config({myapp, >> mykey})` both return `undefined`. >> >> On Thu, Apr 23, 2015 at 1:53 PM, Lo?c Hoguin > > wrote: >> >> ct_run -erl_args -config test.config >> >> >> On 04/23/2015 02:41 PM, Roberto Ostinelli wrote: >> >> All, >> I've got an app `myapp` which uses a standard >> `sys.config` file. >> I'd like to test it using Common Tests, with a test >> configuration file. >> >> I'm passing the variable `-config test.config` to ct_run. >> >> However, when I start my app from the Common Test suite >> `myapp_SUITE.erl` then the app fails to start because >> inside the app >> `application:get_env(myapp, mykey)` returns `undefined`. >> I can, however, >> read this variable with `ct:get_config({myapp, mykey})`. >> >> How am I supposed to use a custom `test.config` file that >> is loaded in >> ct_run *and* read by the app itself? >> >> I've spent almost a day on this and I'm honestly 5 >> minutes away from >> dropping all this in the dustbin and using an external >> test tool >> instead. Please help me stick to Erlang. :) >> >> r. >> >> >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> >> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> -- >> Lo?c Hoguin >> http://ninenines.eu >> >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From russelldb@REDACTED Thu Apr 23 16:14:42 2015 From: russelldb@REDACTED (Russell Brown) Date: Thu, 23 Apr 2015 15:14:42 +0100 Subject: [erlang-questions] file:consult {error, {1, file_io_server, invalid_unicode}} with pre-r17 files in r17 In-Reply-To: References: Message-ID: <4FEBE1E3-37C7-4C41-ACAB-5147971ECB90@basho.com> Hi, Top posting _and_ answering my own question, ban me now! In case anyone else runs into the problem, and doesn?t think of a better way (there must be one!) I accomplished reading files written from r16 using io_lib:format("~p.", [Term]), in r17 just by copying the code from `file:consult/1` but changing _ = epp:set_encoding(FD). to _ = epp:set_encoding(FD, latin1). Will just add it to some internal library module for robust consulting of files of unknown origin. Cheers Russell On 23 Apr 2015, at 10:20, Russell Brown wrote: > Hi, > With the release of r17, certain files that were written in earlier versions of erlang are not `file:consult/1`-able in r17. > > Here is a simple module that shows the issue: https://gist.github.com/russelldb/4f8fe205bd5a5a1cd136 > > If I change line 13 to > > io_lib:format(?~w.", [Term]), > > Everything works just fine. However, legacy files were written in r16 using > > io_lib:format("~p.", [Term]), > > And I?d like to be able to read them into erlang terms in r17, has someone already tackled this? > > Many thanks in advance for your suggestions > > Russell From roberto@REDACTED Thu Apr 23 16:31:53 2015 From: roberto@REDACTED (Roberto Ostinelli) Date: Thu, 23 Apr 2015 16:31:53 +0200 Subject: [erlang-questions] cth_log_redirect crashed Message-ID: All, I'm *still* trying to use Common Tests. My application uses lager. When I start my application in Common Tests, and the application uses lager to log, the application crashes with error: =ERROR REPORT==== 23-Apr-2015::16:28:53 === ** gen_event handler cth_log_redirect crashed. ** Was installed in error_logger ** Last event was: {info_report,<0.29.0>, {<0.7.0>,std_info, [{application,goldrush}, {exited,stopped}, {type,temporary}]}} ** When handler state == {eh_state,tc_log_async,undefined,undefined,undefined, false,false} ** Reason == {timeout,{gen_server,call, [application_controller, which_applications]}} Has anyone seen this? how can I fix it? Thank you, r. -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.goertzen@REDACTED Thu Apr 23 17:54:44 2015 From: daniel.goertzen@REDACTED (Daniel Goertzen) Date: Thu, 23 Apr 2015 10:54:44 -0500 Subject: [erlang-questions] NIF ErlNifEnv thread safety? Message-ID: Can multiple threads safely use the same process-independent ErlNifEnv? The documentation isn't clear to me. From http://www.erlang.org/doc/man/erl_nif.html ... "Threads and concurrency A NIF is thread-safe without any explicit synchronization as long as it acts as a pure function and only reads the supplied arguments. As soon as you write towards a shared state either through static variables or enif_priv_data you need to supply your own explicit synchronization. This includes terms in process independent environments that are shared between threads. Resource objects will also require synchronization if you treat them as mutable." -------------- next part -------------- An HTML attachment was scrubbed... URL: From sverker.eriksson@REDACTED Thu Apr 23 18:20:10 2015 From: sverker.eriksson@REDACTED (Sverker Eriksson) Date: Thu, 23 Apr 2015 18:20:10 +0200 Subject: [erlang-questions] NIF ErlNifEnv thread safety? In-Reply-To: References: Message-ID: <55391BBA.5010405@ericsson.com> On 04/23/2015 05:54 PM, Daniel Goertzen wrote: > Can multiple threads safely use the same process-independent > ErlNifEnv? The documentation isn't clear to me. From > http://www.erlang.org/doc/man/erl_nif.html ... > > "Threads and concurrency > > A NIF is thread-safe without any explicit synchronization as long as > it acts as a pure function and only reads the supplied arguments. As > soon as you write towards a shared state either through static > variables or enif_priv_data you need to supply your own explicit > synchronization. This includes terms in process independent > environments that are shared between threads. Resource objects will > also require synchronization if you treat them as mutable." > > Would it be clearer if "terms in" was removed from the third sentence: "This includes process independent environments that are shared between threads." /Sverker, Erlang/OTP -------------- next part -------------- An HTML attachment was scrubbed... URL: From gomoripeti@REDACTED Thu Apr 23 18:59:04 2015 From: gomoripeti@REDACTED (=?ISO-8859-1?Q?Peti_G=F6m=F6ri?=) Date: Thu, 23 Apr 2015 18:59:04 +0200 Subject: [erlang-questions] cth_log_redirect crashed In-Reply-To: References: Message-ID: Hi Roberto, Others have seen this as well on this thread http://erlang.org/pipermail/erlang-questions/2014-May/079010.html and they suggest a workaround (or two) On Thu, Apr 23, 2015 at 4:31 PM, Roberto Ostinelli wrote: > All, > I'm *still* trying to use Common Tests. My application uses lager. > > When I start my application in Common Tests, and the application uses lager > to log, the application crashes with error: > > =ERROR REPORT==== 23-Apr-2015::16:28:53 === > ** gen_event handler cth_log_redirect crashed. > ** Was installed in error_logger > ** Last event was: {info_report,<0.29.0>, > {<0.7.0>,std_info, > [{application,goldrush}, > {exited,stopped}, > {type,temporary}]}} > ** When handler state == > {eh_state,tc_log_async,undefined,undefined,undefined, > false,false} > ** Reason == {timeout,{gen_server,call, > [application_controller, > which_applications]}} > > Has anyone seen this? how can I fix it? > > Thank you, > r. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From roberto.ostinelli@REDACTED Thu Apr 23 19:27:36 2015 From: roberto.ostinelli@REDACTED (Roberto Ostinelli) Date: Thu, 23 Apr 2015 19:27:36 +0200 Subject: [erlang-questions] cth_log_redirect crashed In-Reply-To: References: Message-ID: <92F225EE-454D-476D-B6CF-31D940B110B5@widetag.com> Hello Peti, Yes thank you I saw that post. Infoortunately I haven't been able to solve it with the provided suggestions. I'll dig in further. Thank you, r. > On 23/apr/2015, at 18:59, Peti G?m?ri wrote: > > Hi Roberto, > > Others have seen this as well on this thread > http://erlang.org/pipermail/erlang-questions/2014-May/079010.html > and they suggest a workaround (or two) > >> On Thu, Apr 23, 2015 at 4:31 PM, Roberto Ostinelli wrote: >> All, >> I'm *still* trying to use Common Tests. My application uses lager. >> >> When I start my application in Common Tests, and the application uses lager >> to log, the application crashes with error: >> >> =ERROR REPORT==== 23-Apr-2015::16:28:53 === >> ** gen_event handler cth_log_redirect crashed. >> ** Was installed in error_logger >> ** Last event was: {info_report,<0.29.0>, >> {<0.7.0>,std_info, >> [{application,goldrush}, >> {exited,stopped}, >> {type,temporary}]}} >> ** When handler state == >> {eh_state,tc_log_async,undefined,undefined,undefined, >> false,false} >> ** Reason == {timeout,{gen_server,call, >> [application_controller, >> which_applications]}} >> >> Has anyone seen this? how can I fix it? >> >> Thank you, >> r. >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> From garry@REDACTED Thu Apr 23 19:38:40 2015 From: garry@REDACTED (Garry Hodgson) Date: Thu, 23 Apr 2015 13:38:40 -0400 Subject: [erlang-questions] Update: segfaults in cowboy rest server In-Reply-To: <55268B41.2020207@research.att.com> References: <55268B41.2020207@research.att.com> Message-ID: <55392E20.3070406@research.att.com> we never did find the exact cause, but it appears that it was unrelated to anything in cowboy or our code. after noticing the system was stable on one of the nodes, and failing on the other, we ended up re-creating the failing vm, and all has been well since. i suspect some kind of mismatch between erlang versions of code and external libraries, as we were in the midst of upgrading to 17.4 (from 15), but i don't know for sure. the system is working well now, with stable low memory usage. a few months ago, our webmachine/r15/other changes version took a few hours to post 500K logs, we are now handling 800K/minute using two smallish vm's. life is good. thanks to all who offered suggestions. we learned a lot chasing this down. On 4/9/15 10:22 AM, Garry Hodgson wrote: > i've got a problem i'm trying to solve wrt controlling memory > consumption in a cloud environment. i've got a server that > receives log data from security appliances and stores in a > mariadb database. logs are sent to us via RESTful api calls, > with batches of logs embedded in the json body of a POST > call. they can get rather large, and we get a lot of them, > at a high rate. > > when production load increased beyond what was anticipated > (doesn't it always?) we began having failures, with the server > disappearing without a trace. in some cases oom-killer killed > it, in others it would fail trying to allocate memory. we only > saw the latter by running in erlang shell and waiting until > it died, then we saw a terse error message. > > to prevent this, i added a check in service_available() to > see if erlang:memory( total ) + content-length > some threshold, > and reject the request if so. also, having read the recent threads > about garbage collecting binaries, i added a timer to check every > 30 seconds that forces gc on all processes if memory usage > is too high. > > this seems to work pretty well, except that after a few days > of running, we get hard crashes, with segfaults showing up > in /var/log/messages: > > kernel: beam[18098]: segfault at 7f09a004040c ip 000000000049e209 sp > 00007fff860d32b0 error 4 in beam[400000+2ce000] > > kernel: beam[14177]: segfault at 7fce288829bc ip 000000000049e209 sp > 00007fffa0d2d7a0 error 4 in beam[400000+2ce000] > > i've been using erlang for 15 years, and have never seen a segfault. > we've recently updated from r15b02 to r17.4, and we've also > switched from webmachine to cowboy. i don't know if either of > those things are relevant. i'm kind of at a loss as to how to diagnose > or deal with this. > > any advice would be greatly appreciated. > From per@REDACTED Thu Apr 23 20:01:27 2015 From: per@REDACTED (Per Hedeland) Date: Thu, 23 Apr 2015 20:01:27 +0200 (CEST) Subject: [erlang-questions] Messenger example In-Reply-To: Message-ID: <201504231801.t3NI1RUP056843@pluto.hedeland.org> You already got the answer to your question, but I would say that the author of that "Note" is the one that is seriously confused. There is no such thing as an "IP domain", and you most certainly can't give "the first component of the IP address" as argument to -sname. Suggested fix: Note: erl -sname assumes that all nodes are in the same DNS domain, and thus that we can use only the first component of the host name. If we want to use nodes in different DNS domains we must use -name instead, but then all host names must be fully qualified. Possibly it should mention the (AFAIK) otherwise undocumented fact that you can use an IP address as 'Host' in '-name Name@REDACTED'. --Per Hedeland nat n wrote: > >I am having trouble running the example from this tutorial:http://www.erlang.org/doc/getting_started/conc_prog.html#id67907In specifics, the "A Larger Example" and the one before that. >I have two computers on the same network that are running Mininet/Ubuntu (using Virtual Box).What confuses me is this: "(Note: erl -sname assumes that all nodes are in the same IP domain and we can use only the first component of the IP address, if we want to use nodes in different domains we use -name instead, but then all IP address must be given in full.)"How should I then call "erl", what options need to be placed when starting erlang? From daniel.goertzen@REDACTED Thu Apr 23 20:09:21 2015 From: daniel.goertzen@REDACTED (Daniel Goertzen) Date: Thu, 23 Apr 2015 13:09:21 -0500 Subject: [erlang-questions] NIF ErlNifEnv thread safety? In-Reply-To: <55391BBA.5010405@ericsson.com> References: <55391BBA.5010405@ericsson.com> Message-ID: Now that my coffee-induced ADHD has subsided I now understand the original text. The word "This" was the part that made things fuzzy for me. Maybe something like the following... "Process independent environments and associated terms that are shared between threads also require explicit synchronization." In any case, my question is answered. Cheers, Dan. On Thu, Apr 23, 2015 at 11:20 AM, Sverker Eriksson < sverker.eriksson@REDACTED> wrote: > > > On 04/23/2015 05:54 PM, Daniel Goertzen wrote: > > Can multiple threads safely use the same process-independent ErlNifEnv? > The documentation isn't clear to me. From > http://www.erlang.org/doc/man/erl_nif.html ... > > "Threads and concurrency > > A NIF is thread-safe without any explicit synchronization as long as it > acts as a pure function and only reads the supplied arguments. As soon as > you write towards a shared state either through static variables or > enif_priv_data you need to supply your own explicit synchronization. This > includes terms in process independent environments that are shared between > threads. Resource objects will also require synchronization if you treat > them as mutable." > > > Would it be clearer if "terms in" was removed from the third sentence: > > "This includes process independent environments that are shared between > threads." > > > /Sverker, Erlang/OTP > -------------- next part -------------- An HTML attachment was scrubbed... URL: From 44290016@REDACTED Fri Apr 24 05:58:27 2015 From: 44290016@REDACTED (=?gb18030?B?vMW8xQ==?=) Date: Fri, 24 Apr 2015 11:58:27 +0800 Subject: [erlang-questions] =?gb18030?q?=5Briak=5D_comile_riak_error_canno?= =?gb18030?b?dCBzdGF0IKGuc29sci00LjcuMC15ei0xLnRneqGvOiBObyBzdWNoIGZpbGUg?= =?gb18030?q?or_directory?= Message-ID: Hi, When I make all, got following error: Erlang R16B03, git clone https://github.com/basho/riak git checkout riak-2.0.4 ==> yokozuna (compile) Create dir ../build Pulling Solr from S3 cp: cannot stat ?solr-4.7.0-yz-1.tgz?: No such file or directory ERROR: Command [compile] failed! make: *** [compile] Error 1 Why? -------------- next part -------------- An HTML attachment was scrubbed... URL: From community-manager@REDACTED Fri Apr 24 09:39:15 2015 From: community-manager@REDACTED (Bruce Yinhe) Date: Fri, 24 Apr 2015 09:39:15 +0200 Subject: [erlang-questions] [ANN] CFP Lambdajam 2015 extended to May 8 Message-ID: Hello Erlangers, The submission deadline for Lambdajam conference, held in Chicago, IL, has been extended to May 8. It would be a great to place for some Erlang/Elixir talks. *Lambdajam, July 15-16, Chicago* http://www.lambdajam.com/ *Call for paper (Deadline May 8):* http://www.lambdajam.com/cfp/ The conference is looking for submissions in the following topic areas: - Core functional programming ? the bread and butter of the functional programmer, but pushing the envelope a bit. - Production functional programming ? what cool things are being done with functional programming in production systems which offer lessons to others. - Cutting edge/different functional programming ? The bleeding edge of the field of functional programming as well as un-thought of usages of functional programming languages and techniques. Best, Bruce Yinhe community-manager@REDACTED +46 72 311 43 89 Erlang Community Manager Industrial Erlang User Group ErlangCentral.org | @ErlangCentral | Your Erlang community site Sponsored by Industrial Erlang User Group -------------- next part -------------- An HTML attachment was scrubbed... URL: From peppe@REDACTED Fri Apr 24 09:58:11 2015 From: peppe@REDACTED (Peter Andersson) Date: Fri, 24 Apr 2015 09:58:11 +0200 Subject: [erlang-questions] cth_log_redirect crashed In-Reply-To: <92F225EE-454D-476D-B6CF-31D940B110B5@widetag.com> References: <92F225EE-454D-476D-B6CF-31D940B110B5@widetag.com> Message-ID: <5539F793.7050901@erlang.org> Hi Roberto, I've been reading through code but haven't been able to figure out how the heck the system can end up in a deadlock. It's application_controller that doesn't respond, but all application_controller operations are asynchronous so it doesn't make sense to me. I suspect it's rather an io-request/group-leader related problem. I went on to try to reproduce the error by running various stress tests, but that didn't help either. So, a few questions: 1. Do you get into this deadlock state each and every time, or does it happen occasionally? 2. What process/function starts and stops the "goldrush" application? 3. Will you help me do some debugging on your site? I think I need to make a dump! (what an inappropriate thing to say in any other context...) :-) Thanks, Peter On 2015-04-23 19:27, Roberto Ostinelli wrote: > Hello Peti, > Yes thank you I saw that post. Infoortunately I haven't been able to solve it with the provided suggestions. > > I'll dig in further. > > Thank you, > r. > > > > > On 23/apr/2015, at 18:59, Peti G?m?ri wrote: > > > > Hi Roberto, > > > > Others have seen this as well on this thread > > http://erlang.org/pipermail/erlang-questions/2014-May/079010.html > > and they suggest a workaround (or two) > > > >> On Thu, Apr 23, 2015 at 4:31 PM, Roberto Ostinelli wrote: > >> All, > >> I'm *still* trying to use Common Tests. My application uses lager. > >> > >> When I start my application in Common Tests, and the application uses lager > >> to log, the application crashes with error: > >> > >> =ERROR REPORT==== 23-Apr-2015::16:28:53 === > >> ** gen_event handler cth_log_redirect crashed. > >> ** Was installed in error_logger > >> ** Last event was: {info_report,<0.29.0>, > >> {<0.7.0>,std_info, > >> [{application,goldrush}, > >> {exited,stopped}, > >> {type,temporary}]}} > >> ** When handler state == > >> {eh_state,tc_log_async,undefined,undefined,undefined, > >> false,false} > >> ** Reason == {timeout,{gen_server,call, > >> [application_controller, > >> which_applications]}} > >> > >> Has anyone seen this? how can I fix it? > >> > >> Thank you, > >> r. > >> > >> _______________________________________________ > >> erlang-questions mailing list > >> erlang-questions@REDACTED > >> http://erlang.org/mailman/listinfo/erlang-questions > >> > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From tuncer.ayaz@REDACTED Fri Apr 24 12:18:36 2015 From: tuncer.ayaz@REDACTED (Tuncer Ayaz) Date: Fri, 24 Apr 2015 12:18:36 +0200 Subject: [erlang-questions] [riak] comile riak error cannot stat 'solr-4.7.0-yz-1.tgz': No such file or directory In-Reply-To: References: Message-ID: On Fri, Apr 24, 2015 at 5:58 AM, ?? <44290016@REDACTED> wrote: > Hi, > > When I make all, got following error: > Erlang R16B03, git clone https://github.com/basho/riak git checkout > riak-2.0.4 > > ==> yokozuna (compile) > Create dir ../build > Pulling Solr from S3 > cp: cannot stat ?solr-4.7.0-yz-1.tgz?: No such file or directory > ERROR: Command [compile] failed! > make: *** [compile] Error 1 > > Why? I don't know why, but I'd suggest to ask on the riak-users list as well (or instead). http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com From lloyd@REDACTED Fri Apr 24 14:47:06 2015 From: lloyd@REDACTED (Lloyd R. Prentice) Date: Fri, 24 Apr 2015 08:47:06 -0400 Subject: [erlang-questions] YARQ -- Yet another rebar question In-Reply-To: References: <1429715984.96717945@apps.rackspace.com> Message-ID: Thanks, folks! Lloyd Sent from my iPad > On Apr 23, 2015, at 3:33 AM, Tuncer Ayaz wrote: > >> On Wed, Apr 22, 2015 at 5:19 PM, wrote: >> Hello, >> >> This may seem like a brain-dead question to some, but it's a mystery >> to me: >> >> I would like to integrate erlguten >> (https://github.com/richcarl/erlguten) into my Nitrogen application. >> The richcarl version of erlguten is built with make. My Nitrogen app >> is built with rebar. >> >> How can I configure rebar to build richcarl/erluten as a dependency >> in my Nitrogen app? > > I've submitted a trivial patch[1] changing a placeholder in erlguten.app.src > to be valid Erlang syntax. When that's merged, you can add this to > rebar.config: > > %% Fetch erlguten as a dependency. > {deps, > [ > {erlguten, ".*", {git, "https://github.com/richcarl/erlguten"}} > ]}. > > %% Right after erlguten is fetched, build it correctly. > {post_hooks, > [ > {'get-deps', 'make -C deps/erlguten'} > ]}. > > [1] https://github.com/richcarl/erlguten/pull/4 From tuncer.ayaz@REDACTED Fri Apr 24 22:35:22 2015 From: tuncer.ayaz@REDACTED (Tuncer Ayaz) Date: Fri, 24 Apr 2015 22:35:22 +0200 Subject: [erlang-questions] Observer Crashdump Viewer bug? Message-ID: Can anyone else reproduce a responsive but blocked Observer gui following these steps? # install otp 18.0 from master # clone and build erlguten $ git clone https://github.com/richcarl/erlguten $ cd erlguten $ make $ ./erlguten test/test1.xml # start Observer use File->Examine Crash Dump to load the # erlguten crash dump $ erl 1> observer:start(). ok WARNING: Found unexpected tag:scheduler WARNING: Found unexpected tag:scheduler WARNING: Found unexpected line in general information: Calling Thread WARNING: Found unexpected line in ETS info: Chain Length Avg # now it stalls in the GUI at "Processing ets" This does not happen if you instead start crashdump_viewer:start/0. From tuncer.ayaz@REDACTED Sat Apr 25 15:10:16 2015 From: tuncer.ayaz@REDACTED (Tuncer Ayaz) Date: Sat, 25 Apr 2015 15:10:16 +0200 Subject: [erlang-questions] overhead of lists:dropwhile/2 Message-ID: While reviewing BjornG's compiler optimization in master, I've noticed one pattern of optimization which at least to me isn't as evident. I've reviewed the implementation of lists:dropwhile/2, and the only difference I can see is that a call to the predicate fun is avoided. Is that the - now removed - speed bump, or am I missing something? If so, does it have to be a speed bump? https://github.com/erlang/otp/commit/8470558 https://github.com/erlang/otp/commit/93ad33d https://github.com/erlang/otp/commit/08708c8 From lloyd@REDACTED Sat Apr 25 16:07:59 2015 From: lloyd@REDACTED (Lloyd R. Prentice) Date: Sat, 25 Apr 2015 10:07:59 -0400 Subject: [erlang-questions] Erlguten fonts Message-ID: <7455ADB4-F469-4889-BE34-F58D86EE14DC@writersglen.com> Hello, I'd love to add a few fonts to erlguten. Does anyone know the source of the character width data represented in the font modules, e.g. eg_font_1.erl through eg_font_17.erl? Thanks, LRP Sent from my iPad From bjorn@REDACTED Sun Apr 26 11:21:26 2015 From: bjorn@REDACTED (=?UTF-8?Q?Bj=C3=B6rn_Gustavsson?=) Date: Sun, 26 Apr 2015 11:21:26 +0200 Subject: [erlang-questions] overhead of lists:dropwhile/2 In-Reply-To: References: Message-ID: On Sat, Apr 25, 2015 at 3:10 PM, Tuncer Ayaz wrote: > While reviewing BjornG's compiler optimization in master, I've > noticed one pattern of optimization which at least to me isn't > as evident. I admit that the optimizations are tiny and that I have set a bad example by doing micro-optimizations. Don't try this at home! :-) lists:dropwhile/2 is a fine function and very rarely will you gain something by avoiding it. > I've reviewed the implementation of lists:dropwhile/2, and the > only difference I can see is that a call to the predicate fun is > avoided. Is that the - now removed - speed bump, or am I missing > something? If so, does it have to be a speed bump? You have missed the cost of building the fun that is passed to lists:dropwhile/2. If lists:dropwhile/2 drops many items from the list, that cost will not be noticeable. But it will be noticeable if no or few elements are dropped from the list. For example, in beam_utils, there is almost never any label to drop (i.e. the fun would only be called once for each call to lists:dropwhile/2). Similarly, in beam_jump it is also fairly common that nothing is dropped. /Bj?rn -- Bj?rn Gustavsson, Erlang/OTP, Ericsson AB From tuncer.ayaz@REDACTED Sun Apr 26 12:14:25 2015 From: tuncer.ayaz@REDACTED (Tuncer Ayaz) Date: Sun, 26 Apr 2015 12:14:25 +0200 Subject: [erlang-questions] overhead of lists:dropwhile/2 In-Reply-To: References: Message-ID: On Sun, Apr 26, 2015 at 11:21 AM, Bj?rn Gustavsson wrote: > On Sat, Apr 25, 2015 at 3:10 PM, Tuncer Ayaz wrote: > > While reviewing BjornG's compiler optimization in master, I've > > noticed one pattern of optimization which at least to me isn't as > > evident. > > I admit that the optimizations are tiny and that I have set a bad > example by doing micro-optimizations. > > Don't try this at home! :-) lists:dropwhile/2 is a fine function and > very rarely will you gain something by avoiding it. Only if measurements show it to be a hot spot :). > > I've reviewed the implementation of lists:dropwhile/2, and the > > only difference I can see is that a call to the predicate fun is > > avoided. Is that the - now removed - speed bump, or am I missing > > something? If so, does it have to be a speed bump? > > You have missed the cost of building the fun that is passed to > lists:dropwhile/2. If lists:dropwhile/2 I knew I missed something, and I might have probably noticed the difference if I had compared Core Erlang. I'll try to remember that the next time. > drops many items from the list, that cost will not be noticeable. > But it will be noticeable if no or few elements are dropped from the > list. For example, in beam_utils, there is almost never any label to > drop (i.e. the fun would only be called once for each call to > lists:dropwhile/2). Similarly, in beam_jump it is also fairly common > that nothing is dropped. Would it be as fast if you had extracted the predicate into a local fun and referenced it as "fun my_pred/1" in the dropwhile/2 call? Can't we enable inlining per function? From tuncer.ayaz@REDACTED Sun Apr 26 12:17:09 2015 From: tuncer.ayaz@REDACTED (Tuncer Ayaz) Date: Sun, 26 Apr 2015 12:17:09 +0200 Subject: [erlang-questions] overhead of lists:dropwhile/2 In-Reply-To: References: Message-ID: On Sun, Apr 26, 2015 at 12:14 PM, Tuncer Ayaz wrote: > Can't we enable inlining per function? No idea why I thought inlining would help. Please ignore that. From n.oxyde@REDACTED Sun Apr 26 13:08:43 2015 From: n.oxyde@REDACTED (Anthony Ramine) Date: Sun, 26 Apr 2015 13:08:43 +0200 Subject: [erlang-questions] overhead of lists:dropwhile/2 In-Reply-To: References: Message-ID: <61405750-2C65-4911-A668-4EC29C60E41B@gmail.com> Le 26 avr. 2015 ? 12:17, Tuncer Ayaz a ?crit : > No idea why I thought inlining would help. Please ignore that. It could if lists:dropwhile/2 was in the list of inlined lists functions. I could clean https://github.com/nox/otp/commits/inline_list_funcs-additions and submit it if there is interest for such a thing. From bjorn@REDACTED Sun Apr 26 14:34:42 2015 From: bjorn@REDACTED (=?UTF-8?Q?Bj=C3=B6rn_Gustavsson?=) Date: Sun, 26 Apr 2015 14:34:42 +0200 Subject: [erlang-questions] overhead of lists:dropwhile/2 In-Reply-To: References: Message-ID: On Sun, Apr 26, 2015 at 12:14 PM, Tuncer Ayaz wrote: > On Sun, Apr 26, 2015 at 11:21 AM, Bj?rn Gustavsson wrote: [...] >> drops many items from the list, that cost will not be noticeable. >> But it will be noticeable if no or few elements are dropped from the >> list. For example, in beam_utils, there is almost never any label to >> drop (i.e. the fun would only be called once for each call to >> lists:dropwhile/2). Similarly, in beam_jump it is also fairly common >> that nothing is dropped. > > Would it be as fast if you had extracted the predicate into a local > fun and referenced it as "fun my_pred/1" in the dropwhile/2 call? No. That would not make any difference. A fun object still has to be created on the heap in the current implementation of funs. It would be possible to treat funs with an empty environment (which includes all funs that are simple references to local functions) as constants, but it is not how they are currently implemented. /Bj?rn -- Bj?rn Gustavsson, Erlang/OTP, Ericsson AB From eric.pailleau@REDACTED Sun Apr 26 16:58:46 2015 From: eric.pailleau@REDACTED (PAILLEAU Eric) Date: Sun, 26 Apr 2015 16:58:46 +0200 Subject: [erlang-questions] LZMA/LZMA2 ? Message-ID: <553CFD26.403@wanadoo.fr> Hi, this question is addressed mainly to OTP team. Is there any plan to add lzma/lzma2 compression format library in Erlang, like current zlib ? This format becomes widely used in many operating systems, and Erlang primitives would be nice. Regards From tuncer.ayaz@REDACTED Sun Apr 26 18:06:52 2015 From: tuncer.ayaz@REDACTED (Tuncer Ayaz) Date: Sun, 26 Apr 2015 18:06:52 +0200 Subject: [erlang-questions] overhead of lists:dropwhile/2 In-Reply-To: <61405750-2C65-4911-A668-4EC29C60E41B@gmail.com> References: <61405750-2C65-4911-A668-4EC29C60E41B@gmail.com> Message-ID: On Sun, Apr 26, 2015 at 1:08 PM, Anthony Ramine wrote: > Le 26 avr. 2015 a 12:17, Tuncer Ayaz a ecrit : > >> No idea why I thought inlining would help. Please ignore that. > > It could if lists:dropwhile/2 was in the list of inlined lists functions. I was wondering if inlining could transform it into what Bjorn wrote in expanded form (inline the fun object and dropwhile/2 itself), but that seemed a little too much to expect from the inliner. > I could clean https://github.com/nox/otp/commits/inline_list_funcs-additions > and submit it if there is interest for such a thing. Is it always a win? From lloyd@REDACTED Sun Apr 26 18:51:58 2015 From: lloyd@REDACTED (lloyd@REDACTED) Date: Sun, 26 Apr 2015 12:51:58 -0400 (EDT) Subject: [erlang-questions] Erlguten fonts In-Reply-To: <7455ADB4-F469-4889-BE34-F58D86EE14DC@writersglen.com> References: <7455ADB4-F469-4889-BE34-F58D86EE14DC@writersglen.com> Message-ID: <1430067118.39921161@apps.rackspace.com> Hello, This is Take 2. Digging around erlguten source and Google I've discovered nuggets that may point toward an answer to my previous post: e.g. how to add fonts to erlguten. I don't have a complete solution yet, but consider this speculative information that points the way and a request for help to resolve outstanding mysteries. I'll write it all up as a blog or some such once I have answer at hand. 1) Font metrics, e.g. glyph widths, etc., are found in *.afm files 2) In Ubuntu 14.04 fonts are stored in: -- /usr/share/fonts/type1 -- /usr/share/fonts/truetype 3) List /usr/share/fonts/type1/gsfonts, for instance, and you see *.pfb, *.pfm, and *.afm files. -- *.pfb are printer binary files -- *.pfm are printer metric files -- *.afm are provide font metrics 4) When I open /usr/share/fonts/truetype I see dejavu, a font I happen to like. Listing dejavu, I see DejaVuSans.ttf and related *.ttf files. But no *.afm. 5) But I can convert from *.ttf files using a convenient utility:ttf2afm http://manpages.ubuntu.com/manpages/precise/man1/ttf2afm.1.html In my case $ sudo ttf2atm /usr/share/fonts/truetype/dejavu/DejaVuSans.ttf generates a *.afm file. I presume it should be called dejavusans.afm, but not 100% confident. 6) Now, in erlguten/priv/fonts/adobe we see font names. In erguten/priv/fonts/adobe/courier we see courier *.afm files. Open one and you can see what the font metrics look like. 7) There seems to be a necessary directory to these fonts in erlguten/priv/font_locations. 8) Now look in erlguten/src/eg_font_5.erl, say, and you'll see an erlang module with a bunch of font metrics. This raises the question, how do we convert from an *.afm file to a similar erlang font metric file? 9) Clue: (and here I'm very shaky ground) look in eg_afm and note a module called make/2. This appears to be a parser that will do the trick. I haven't tried it yet since I'm clear exactly what the parameters BaseDir and FurtherPath refer to. I'd much welcome any corrections or additional information that would span gaps in the road toward adding fonts to erlguten. In my view erlguten is a hidden jewel that deserves additional development and documentation. I'm looking toward a web-based GUI in Nitrogen. Would welcome help and will post what I find along the way. All the best, Lloyd -----Original Message----- From: "Lloyd R. Prentice" Sent: Saturday, April 25, 2015 10:07am To: "Erlang Questions" Subject: [erlang-questions] Erlguten fonts Hello, I'd love to add a few fonts to erlguten. Does anyone know the source of the character width data represented in the font modules, e.g. eg_font_1.erl through eg_font_17.erl? Thanks, LRP Sent from my iPad _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions From peter@REDACTED Mon Apr 27 11:45:50 2015 From: peter@REDACTED (Peter Membrey) Date: Mon, 27 Apr 2015 17:45:50 +0800 (HKT) Subject: [erlang-questions] Corrupted term on distribution channel from a C-Node Message-ID: <2080062346.185311.1430127950846.JavaMail.zimbra@membrey.hk> Hi guys, Bit of a weird mystery to share with you this time. We're using this particular C-Node in production and it handles many million messages a day with no problem. We've been running it without much in the way of modification for over a year and a half. There haven't been any recent changes made. A few weeks back, we had an application crash with a corrupted term coming from the C-Node: <<131,104,3,97,2,100,0,0,103,131,104,...>> Last week we had another crash with a similar corrupted term: <<131,104,3,97,2,100,0,0,0,131,104,...>> These two apps were running on different boxes, talking to different Erlang VMs, and everything was done over local loopback i.e. same app but everything else in the environment was independent. Having gone through the message with the EDP and external term man pages, it looks like we're getting a corrupt SEND command. In both cases the cookie is not being encoded correctly (the cookie is definitely not empty), and in the second instance, we have a 0 where we'd expect a 103. The messages following both corrupted SEND messages were decodable with binary_to_term/1 and the payload looked good. This looks similar to a problem that this guy had: http://erlang.org/pipermail/erlang-questions/2007-March/025393.html In our set up however: 1. All calls to ei_send are protected by the same mutex 2. Only one thread is sending this data (one thread sending data, the other listening for data from the Erlang VM) 3. Thread local variables for the ei_x_buff are being used 4. The ei_x_buff is not being reset after each send, but the index is set back to zero so encoding starts at the beginning Again, this app generally hums along beautifully and we've got no idea where this issue lies. I'm running the C-Node under load at the moment (roughly 15,000 messages per second) and it has done 161,200,000 messages so far this afternoon without any trouble. Any advice and guidance on where to look would be most gratefully appreciated! Kind Regards, Peter Membrey From jesper.louis.andersen@REDACTED Mon Apr 27 15:36:20 2015 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Mon, 27 Apr 2015 15:36:20 +0200 Subject: [erlang-questions] Corrupted term on distribution channel from a C-Node In-Reply-To: <2080062346.185311.1430127950846.JavaMail.zimbra@membrey.hk> References: <2080062346.185311.1430127950846.JavaMail.zimbra@membrey.hk> Message-ID: On Mon, Apr 27, 2015 at 11:45 AM, Peter Membrey wrote: > Having gone through the message with the EDP and external term man pages, > it looks like we're getting a corrupt SEND command. In both cases the > cookie is not being encoded correctly (the cookie is definitely not empty), > and in the second instance, we have a 0 where we'd expect a 103. The > messages following both corrupted SEND messages were decodable with > binary_to_term/1 and the payload looked good. > Is this nondeterministic? I wouldn't entirely rule out the possibility of the hardware messing up, or some error in the code elsewhere manipulating the wrong data. Not that I can pinpoint *that* is what is happening, but do not rule it out from the start that this could be a hardware thing. What version of Erlang is this? A recent one, or an earlier one? -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From sverker.eriksson@REDACTED Mon Apr 27 16:40:31 2015 From: sverker.eriksson@REDACTED (Sverker Eriksson) Date: Mon, 27 Apr 2015 16:40:31 +0200 Subject: [erlang-questions] wrapperless NIF module? In-Reply-To: References: Message-ID: <553E4A5F.6040104@ericsson.com> On 04/22/2015 03:52 PM, Daniel Goertzen wrote: > I was thinking about the little Erlang wrapper that you need around a > NIF module and then it hit me: Could the Erlang wrapper be thrown out > and the NIF module plopped into ebin/ and directly loaded like a .beam > file? I'm curious if this has been attempted or is even possible. > > Have not been attempted to my knowledge. Would be possible from a VM perspective by creating the "wrapper module" dynamically in memory. Problem areas that I can think of are * missing module_info * missing type info (dialyzer) * all the code that expect files to be named *.beam /Sverker, Erlang/OTP -------------- next part -------------- An HTML attachment was scrubbed... URL: From benmmurphy@REDACTED Mon Apr 27 17:21:58 2015 From: benmmurphy@REDACTED (Ben Murphy) Date: Mon, 27 Apr 2015 16:21:58 +0100 Subject: [erlang-questions] Corrupted term on distribution channel from a C-Node In-Reply-To: <2080062346.185311.1430127950846.JavaMail.zimbra@membrey.hk> References: <2080062346.185311.1430127950846.JavaMail.zimbra@membrey.hk> Message-ID: Hi, It might be worth running your load tests with address sanitizer (https://code.google.com/p/address-sanitizer/) assuming the erlang code and your code works under it. I've caught heap corruption in the erlang odbc module using address sanitizer. Maybe your code or the erlang code is corrupting the heap somewhere and this is introducing non-determinism into the system. Or if you are super desperate and can handle the slow down you could run it in production :) The problem might not show up in your load tests because it is a specific message that is triggering the heap corruption and this isn't part of your load tests. Though, address sanitizer will catch some issues that would never causes crashes (for example if you have an off by 1 error but it always overwrites slack space) so it can be dangerous running it in production because it can turn programs that will always run correctly into programs that crash. On Mon, Apr 27, 2015 at 10:45 AM, Peter Membrey wrote: > Hi guys, > > Bit of a weird mystery to share with you this time. We're using this particular C-Node in production and it handles many million messages a day with no problem. We've been running it without much in the way of modification for over a year and a half. There haven't been any recent changes made. > > A few weeks back, we had an application crash with a corrupted term coming from the C-Node: > > <<131,104,3,97,2,100,0,0,103,131,104,...>> > > Last week we had another crash with a similar corrupted term: > > <<131,104,3,97,2,100,0,0,0,131,104,...>> > > These two apps were running on different boxes, talking to different Erlang VMs, and everything was done over local loopback i.e. same app but everything else in the environment was independent. > > Having gone through the message with the EDP and external term man pages, it looks like we're getting a corrupt SEND command. In both cases the cookie is not being encoded correctly (the cookie is definitely not empty), and in the second instance, we have a 0 where we'd expect a 103. The messages following both corrupted SEND messages were decodable with binary_to_term/1 and the payload looked good. > > This looks similar to a problem that this guy had: > > http://erlang.org/pipermail/erlang-questions/2007-March/025393.html > > In our set up however: > > 1. All calls to ei_send are protected by the same mutex > 2. Only one thread is sending this data (one thread sending data, the other listening for data from the Erlang VM) > 3. Thread local variables for the ei_x_buff are being used > 4. The ei_x_buff is not being reset after each send, but the index is set back to zero so encoding starts at the beginning > > Again, this app generally hums along beautifully and we've got no idea where this issue lies. I'm running the C-Node under load at the moment (roughly 15,000 messages per second) and it has done 161,200,000 messages so far this afternoon without any trouble. > > Any advice and guidance on where to look would be most gratefully appreciated! > > Kind Regards, > > Peter Membrey > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From peter@REDACTED Mon Apr 27 15:53:43 2015 From: peter@REDACTED (Peter Membrey) Date: Mon, 27 Apr 2015 21:53:43 +0800 (HKT) Subject: [erlang-questions] Corrupted term on distribution channel from a C-Node In-Reply-To: References: <2080062346.185311.1430127950846.JavaMail.zimbra@membrey.hk> Message-ID: <580847300.185460.1430142823456.JavaMail.zimbra@mail.membrey.hk> An HTML attachment was scrubbed... URL: -------------- next part -------------- Hi, It's non deterministic for sure. We run both a primary and secondary app reading in the same data (multicast). One had this issue and the other did not. Although the hardware is slightly different, the versions and configurations are basically identical. I believe we are running R16B03. I'll be able to double check tomorrow (can't believe I forgot to include that in the first email). It's actually a slightly patched version to handle the issue of not being able to properly lock a tcp connection as the receive function would block but handle the heart beat response internally. That version can be found here: https://github.com/pmembrey/otp I'm sending from my phone so apologies for any typos or mistakes. Thanks again for your help! Kind regards, Peter Membrey On 27 Apr 2015 21:38, Jesper Louis Andersen wrote: On Mon, Apr 27, 2015 at 11:45 AM, Peter Membrey wrote: > Having gone through the message with the EDP and external term man pages, > it looks like we're getting a corrupt SEND command. In both cases the > cookie is not being encoded correctly (the cookie is definitely not empty), > and in the second instance, we have a 0 where we'd expect a 103. The > messages following both corrupted SEND messages were decodable with > binary_to_term/1 and the payload looked good. > Is this nondeterministic? I wouldn't entirely rule out the possibility of the hardware messing up, or some error in the code elsewhere manipulating the wrong data. Not that I can pinpoint *that* is what is happening, but do not rule it out from the start that this could be a hardware thing. What version of Erlang is this? A recent one, or an earlier one? -- J. From benmmurphy@REDACTED Mon Apr 27 18:54:34 2015 From: benmmurphy@REDACTED (Ben Murphy) Date: Mon, 27 Apr 2015 17:54:34 +0100 Subject: [erlang-questions] Corrupted term on distribution channel from a C-Node In-Reply-To: <580847300.185460.1430142823456.JavaMail.zimbra@mail.membrey.hk> References: <2080062346.185311.1430127950846.JavaMail.zimbra@membrey.hk> <580847300.185460.1430142823456.JavaMail.zimbra@mail.membrey.hk> Message-ID: Hi, There still might be a threading issue with writes to that fd. I would change the erlang code so that ei_write_t/ei_writev_t asserts that the lock is held when it is called. Maybe there is another place that is calling ei_write_t/ei_writev_t without the lock that you have not caught. Or alternatively move the lock into erlang and the lock presumably should be taken at ei_writev_fill_t and ei_write_fill_t. Because you are locking outside at a higher layer it is hard to know for sure if you are locking correctly at all the places. Also in connect this code looks suspect. Other stuff is calling get_ei_socket_info and then reading stuff from the structure. But it is possible that the memory pointed to by the pointer has been freed and is being used by some other allocation. stuff that reads from this structure needs to take out the mutex over the whole code path then copy it out before unlocking. It could be that this is the problem because it looks like one of your symptoms :) static ei_socket_info* get_ei_socket_info(int fd) { int i; #ifdef _REENTRANT ei_mutex_lock(ei_sockets_lock, 0); #endif /* _REENTRANT */ for (i = 0; i < ei_n_sockets; ++i) if (ei_sockets[i].socket == fd) { /*fprintf("get_ei_socket_info %d %d \"%s\"\n", fd, ei_sockets[i].dist_version, ei_sockets[i].cookie);*/ #ifdef _REENTRANT ei_mutex_unlock(ei_sockets_lock); #endif /* _REENTRANT */ return &ei_sockets[i]; } #ifdef _REENTRANT ei_mutex_unlock(ei_sockets_lock); #endif /* _REENTRANT */ return NULL; } On Mon, Apr 27, 2015 at 2:53 PM, Peter Membrey wrote: > Hi, > It's non deterministic for sure. We run both a primary and secondary app reading in the same data (multicast). One had this issue and the other did not. Although the hardware is slightly different, the versions and configurations are basically identical. > I believe we are running R16B03. I'll be able to double check tomorrow (can't believe I forgot to include that in the first email). > It's actually a slightly patched version to handle the issue of not being able to properly lock a tcp connection as the receive function would block but handle the heart beat response internally. That version can be found here: > https://github.com/pmembrey/otp > I'm sending from my phone so apologies for any typos or mistakes. > Thanks again for your help! > Kind regards, > Peter Membrey > On 27 Apr 2015 21:38, Jesper Louis Andersen wrote: > > > On Mon, Apr 27, 2015 at 11:45 AM, Peter Membrey wrote: > >> Having gone through the message with the EDP and external term man pages, >> it looks like we're getting a corrupt SEND command. In both cases the >> cookie is not being encoded correctly (the cookie is definitely not empty), >> and in the second instance, we have a 0 where we'd expect a 103. The >> messages following both corrupted SEND messages were decodable with >> binary_to_term/1 and the payload looked good. >> > > Is this nondeterministic? I wouldn't entirely rule out the possibility of > the hardware messing up, or some error in the code elsewhere manipulating > the wrong data. Not that I can pinpoint *that* is what is happening, but do > not rule it out from the start that this could be a hardware thing. What > version of Erlang is this? A recent one, or an earlier one? > > > -- > J. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From peter@REDACTED Tue Apr 28 02:34:52 2015 From: peter@REDACTED (Peter Membrey) Date: Tue, 28 Apr 2015 08:34:52 +0800 (HKT) Subject: [erlang-questions] Corrupted term on distribution channel from a C-Node In-Reply-To: References: <2080062346.185311.1430127950846.JavaMail.zimbra@membrey.hk> <580847300.185460.1430142823456.JavaMail.zimbra@mail.membrey.hk> Message-ID: <203854868.185615.1430181292465.JavaMail.zimbra@membrey.hk> Hi guys, Thanks for all the feedback! :) The modifications I made were at a higher level than ei_writev, so hopefully I haven't inadvertently caused this problem - though of course it's quite possible that I have :) The changes really only provided a way to tell the receive family of functions not to reply to a heart beat directly. Then when you get an ERL_TICK, you need to handle it yourself, which is easy enough to wrap in a mutex. That way you can still use blocking reads and don't have to worry about timeouts before whole messages could be read (and all sorts of funky other workarounds that were tried). The PR is here with the doco: https://github.com/erlang/otp/pulls/pmembrey I've been meaning to complete the test cases so this could be merged upstream... The send_tock() function sends a heartbeat reply which is {0x00,0x00,0x00,0x00} - I wonder if that is what is being interleaved by mistake? Still, if you look from line 93: https://github.com/pmembrey/otp/blob/maint/lib/erl_interface/src/connect/send.c#L93 v[0].iov_base = (char *)header; v[0].iov_len = index; v[1].iov_base = (char *)msg; v[1].iov_len = msglen; This gets sent to ei_send() - but from what I can tell, these are effectively separate write operations. Is it possible for the first vector to fail to write completely, and then for the second write to start and complete? Thanks again! Kind Regards, Peter Membrey ----- Original Message ----- From: "Ben Murphy" To: "Peter Membrey" Cc: "Jesper Louis Andersen" , "Erlang (E-mail)" Sent: Tuesday, 28 April, 2015 00:54:34 Subject: Re: [erlang-questions] Corrupted term on distribution channel from a C-Node Hi, There still might be a threading issue with writes to that fd. I would change the erlang code so that ei_write_t/ei_writev_t asserts that the lock is held when it is called. Maybe there is another place that is calling ei_write_t/ei_writev_t without the lock that you have not caught. Or alternatively move the lock into erlang and the lock presumably should be taken at ei_writev_fill_t and ei_write_fill_t. Because you are locking outside at a higher layer it is hard to know for sure if you are locking correctly at all the places. Also in connect this code looks suspect. Other stuff is calling get_ei_socket_info and then reading stuff from the structure. But it is possible that the memory pointed to by the pointer has been freed and is being used by some other allocation. stuff that reads from this structure needs to take out the mutex over the whole code path then copy it out before unlocking. It could be that this is the problem because it looks like one of your symptoms :) static ei_socket_info* get_ei_socket_info(int fd) { int i; #ifdef _REENTRANT ei_mutex_lock(ei_sockets_lock, 0); #endif /* _REENTRANT */ for (i = 0; i < ei_n_sockets; ++i) if (ei_sockets[i].socket == fd) { /*fprintf("get_ei_socket_info %d %d \"%s\"\n", fd, ei_sockets[i].dist_version, ei_sockets[i].cookie);*/ #ifdef _REENTRANT ei_mutex_unlock(ei_sockets_lock); #endif /* _REENTRANT */ return &ei_sockets[i]; } #ifdef _REENTRANT ei_mutex_unlock(ei_sockets_lock); #endif /* _REENTRANT */ return NULL; } On Mon, Apr 27, 2015 at 2:53 PM, Peter Membrey wrote: > Hi, > It's non deterministic for sure. We run both a primary and secondary app reading in the same data (multicast). One had this issue and the other did not. Although the hardware is slightly different, the versions and configurations are basically identical. > I believe we are running R16B03. I'll be able to double check tomorrow (can't believe I forgot to include that in the first email). > It's actually a slightly patched version to handle the issue of not being able to properly lock a tcp connection as the receive function would block but handle the heart beat response internally. That version can be found here: > https://github.com/pmembrey/otp > I'm sending from my phone so apologies for any typos or mistakes. > Thanks again for your help! > Kind regards, > Peter Membrey > On 27 Apr 2015 21:38, Jesper Louis Andersen wrote: > > > On Mon, Apr 27, 2015 at 11:45 AM, Peter Membrey wrote: > >> Having gone through the message with the EDP and external term man pages, >> it looks like we're getting a corrupt SEND command. In both cases the >> cookie is not being encoded correctly (the cookie is definitely not empty), >> and in the second instance, we have a 0 where we'd expect a 103. The >> messages following both corrupted SEND messages were decodable with >> binary_to_term/1 and the payload looked good. >> > > Is this nondeterministic? I wouldn't entirely rule out the possibility of > the hardware messing up, or some error in the code elsewhere manipulating > the wrong data. Not that I can pinpoint *that* is what is happening, but do > not rule it out from the start that this could be a hardware thing. What > version of Erlang is this? A recent one, or an earlier one? > > > -- > J. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From roberto@REDACTED Tue Apr 28 11:23:34 2015 From: roberto@REDACTED (Roberto Ostinelli) Date: Tue, 28 Apr 2015 11:23:34 +0200 Subject: [erlang-questions] [ANN] Lager-Airbrake Message-ID: I have a great pleasure to announce that today I?ve released Lager-Airbrake notifier. If you are already using the popular logging framework Lager[1] by Basho in your application, you might consider having an easy drop in component that will report the errors with the desired level of severity to AirBrake. Lager-Airbrake works as a Lager backend, it is therefore fully integrated with it, and easy to setup. In your project?s rebar.config file, add lager_airbrake as a dependency: {lager_airbrake, ?.*?, {git, ? https://github.com/ostinelli/lager_airbrake.git?, ?master?}} Then include lager_airbrake_backend in your lager configuration: %% Lager config {lager, [ {handlers, [ %% Add Airbrake monitoring {lager_airbrake_backend, [ {environment, "development"}, {api_key, "AIRBRAKE_API_KEY"}, {project_id, "AIRBRAKE_PROJECT_ID"}, {level, warning} ]}, [...] ]}, [...] } You?re done. Lager-Airbrake will try to use some heuristics to optimize the notification sent to Airbrake, so that errors can be easily differentiated. As you may know, in Erlang it is sometimes difficult to separate the errors based on their Type so some attempts have been done to address this reporting issue. You can check Lager-Aibrake here: https://github.com/ostinelli/lager_airbrake Best, r. [1] https://github.com/basho/lager -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto@REDACTED Tue Apr 28 11:26:48 2015 From: roberto@REDACTED (Roberto Ostinelli) Date: Tue, 28 Apr 2015 11:26:48 +0200 Subject: [erlang-questions] cth_log_redirect crashed In-Reply-To: <5539F793.7050901@erlang.org> References: <92F225EE-454D-476D-B6CF-31D940B110B5@widetag.com> <5539F793.7050901@erlang.org> Message-ID: > > 1. Do you get into this deadlock state each and every time, or does it > happen occasionally? > Every time. > 2. What process/function starts and stops the "goldrush" application? > Lager: https://github.com/basho/lager > 3. Will you help me do some debugging on your site? I think I need to > make a dump! Sure thing. I guess we can continue this conversation off-list. Best, r. -------------- next part -------------- An HTML attachment was scrubbed... URL: From n.oxyde@REDACTED Tue Apr 28 11:42:34 2015 From: n.oxyde@REDACTED (Anthony Ramine) Date: Tue, 28 Apr 2015 11:42:34 +0200 Subject: [erlang-questions] overhead of lists:dropwhile/2 In-Reply-To: References: <61405750-2C65-4911-A668-4EC29C60E41B@gmail.com> Message-ID: <861BF6C4-651A-4C3E-B9B7-CA23FDE7308A@gmail.com> Le 26 avr. 2015 ? 18:06, Tuncer Ayaz a ?crit : > I was wondering if inlining could transform it into what Bjorn wrote > in expanded form (inline the fun object and dropwhile/2 itself), but that > seemed a little too much to expect from the inliner. Not too much, IIRC it does. You can check yourself by inlining a lists:map/2 call and compiling to Core Erlang. From carlosj.gf@REDACTED Tue Apr 28 20:30:22 2015 From: carlosj.gf@REDACTED (=?ISO-8859-1?Q?Carlos_Gonz=E1lez_Florido?=) Date: Tue, 28 Apr 2015 20:30:22 +0200 Subject: [erlang-questions] [ANN] NkDOCKER released Message-ID: Hello, list. NkDOCKER is a new native, 100% Erlang Docker client, fully supporting the Docker Remote API v1.18. Every single command and option in the standard Docker client is available. - Full v1.18 (Docker 1.6) API supported. - It supports bidirectional attach. - It can control any number of local or remote Docker daemons. - Events, stdin/stdout/stderr attachs, logs, etc., are sent as Erlang messages. - It can reuse existing connections to speed up the message sending. - It supports TCP and TLS transports. Unix socket transport is not supported yet. NkDOCKER needs Erlang >= 17, and it is tested on Linux and OSX (using boot2docker). The minimum required Docker server version is 1.5 (Remote API v1.17). It is part of the Nekso software suite, but can be used stand-alone. Try it at https://github.com/Nekso/nkdocker. Comments are very welcomed! Carlos Gonzalez @carlosjgf https://www.linkedin.com/in/carlosjgf -------------- next part -------------- An HTML attachment was scrubbed... URL: From luis.rascao@REDACTED Wed Apr 29 00:19:31 2015 From: luis.rascao@REDACTED (=?UTF-8?B?THVpcyBSYXNjw6Nv?=) Date: Tue, 28 Apr 2015 23:19:31 +0100 Subject: [erlang-questions] Cowboy and fake supervisor Message-ID: Hi all, As many of you are familiar with Cowboy there is this practice of using a 'fake' supervisor and returning it's pid in the start/2 callback of your application behaviour, this can be seen in all cowboy examples at https://github.com/ninenines/cowboy/tree/master/examples, why is this supervisor really necessary? I've tried several things and on most of them the node just hangs when requesting a stop, however it exits successfully if i return {ok, self()} on start/2, i'm also unable to explain this behaviour. Can anyone shed some light in this matter? Thanks! Luis Rasc?o -------------- next part -------------- An HTML attachment was scrubbed... URL: From roger@REDACTED Wed Apr 29 11:12:11 2015 From: roger@REDACTED (Roger Lipscombe) Date: Wed, 29 Apr 2015 10:12:11 +0100 Subject: [erlang-questions] Measuring message queue delay Message-ID: For various reasons, I want a metric that measures how long messages spend in a process message queue. The process is a gen_server, if that has any bearing on the solution. Also, this is for production, so it will be always-on and must be low-impact. I could do this by timestamping _every_ message sent to the process and then reporting the deltas, but I think that's ugly. I thought of posting a message to self(), with a timestamp and then measuring the delta. I could then post another one as soon as that message is processed. Obviously, this leaves the process continually looping, which is not good. So, instead, I could use erlang:send_after, but that requires two messages: a delayed one triggered by erlang:send_after, and an immediate one for measuring the delay. That's a bit complicated. Would it be sensible to send a message, with erlang:send_after, with the _expected_ timestamp, and then compute the delta from that? Or, alternatively, what other ways could I measure how long a process is taking to handle its message queue? Thanks, Roger. From roger@REDACTED Wed Apr 29 11:35:47 2015 From: roger@REDACTED (Roger Lipscombe) Date: Wed, 29 Apr 2015 10:35:47 +0100 Subject: [erlang-questions] erlang:system_monitor in production? Message-ID: Is it OK to have erlang:system_monitor permanently enabled in production? The Erlang documentation doesn't say one way or the other; Ferd's "Erlang in Anger" kinda implies that you'd turn it on when debugging something specifically. I found https://github.com/basho/riak_sysmon which seems to imply that it is OK. On a related note, how do I go about picking good values for the thresholds? Are there any metrics available that relate to the system_monitor alarms? By that I mean that (e.g.) folsom can report a bunch of Erlang VM information. Are there comparable functions that return the current largest long_gc or long_schedule values? Thanks, Roger. From roberto@REDACTED Wed Apr 29 12:54:55 2015 From: roberto@REDACTED (Roberto Ostinelli) Date: Wed, 29 Apr 2015 12:54:55 +0200 Subject: [erlang-questions] Error in Common Tests Message-ID: Dear list, When running CT I have a test where I get: - - - - - - - - - - - - - - - - - - - - - - - - - - Error detected: {'EXIT',{badmatch,{...}}} - - - - - - - - - - - - - - - - - - - - - - - - - - No additional information. My application logs do not show any errors. Weirdly enough, if I put a timer:sleep(10) somewhere in my code, this error does not show up anymore. Thing is: how can I get CT to tell me the module / line where this error happens? Thank you, r. -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Wed Apr 29 12:58:15 2015 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Wed, 29 Apr 2015 13:58:15 +0300 Subject: [erlang-questions] Error in Common Tests In-Reply-To: References: Message-ID: <5540B947.1050705@ninenines.eu> Did you look inside the HTML logs that Common Test generates? They usually contain the complete error and stacktrace, while the console only gets partials. On 04/29/2015 01:54 PM, Roberto Ostinelli wrote: > Dear list, > When running CT I have a test where I get: > > - - - - - - - - - - - - - - - - - - - - - - - - - - > Error detected: {'EXIT',{badmatch,{...}}} > - - - - - - - - - - - - - - - - - - - - - - - - - - > > No additional information. My application logs do not show any errors. > > Weirdly enough, if I put a timer:sleep(10) somewhere in my code, this > error does not show up anymore. > > Thing is: how can I get CT to tell me the module / line where this error > happens? > > Thank you, > r. > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin http://ninenines.eu From roberto@REDACTED Wed Apr 29 13:15:05 2015 From: roberto@REDACTED (Roberto Ostinelli) Date: Wed, 29 Apr 2015 13:15:05 +0200 Subject: [erlang-questions] Error in Common Tests In-Reply-To: <5540B947.1050705@ninenines.eu> References: <5540B947.1050705@ninenines.eu> Message-ID: Lo?c, Yes that did help me. It just stated {error,eoc} (i.e. no line of where this happened) but at least I could track it down to the issue. Thank you. r. On Wed, Apr 29, 2015 at 12:58 PM, Lo?c Hoguin wrote: > Did you look inside the HTML logs that Common Test generates? They usually > contain the complete error and stacktrace, while the console only gets > partials. > > > On 04/29/2015 01:54 PM, Roberto Ostinelli wrote: > >> Dear list, >> When running CT I have a test where I get: >> >> - - - - - - - - - - - - - - - - - - - - - - - - - - >> Error detected: {'EXIT',{badmatch,{...}}} >> - - - - - - - - - - - - - - - - - - - - - - - - - - >> >> No additional information. My application logs do not show any errors. >> >> Weirdly enough, if I put a timer:sleep(10) somewhere in my code, this >> error does not show up anymore. >> >> Thing is: how can I get CT to tell me the module / line where this error >> happens? >> >> Thank you, >> r. >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -- > Lo?c Hoguin > http://ninenines.eu > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rtrlists@REDACTED Wed Apr 29 13:24:49 2015 From: rtrlists@REDACTED (Robert Raschke) Date: Wed, 29 Apr 2015 12:24:49 +0100 Subject: [erlang-questions] Measuring message queue delay In-Reply-To: References: Message-ID: Hi Roger, maybe you can use the tracing facilities of Erlang. At a low level it is reasonably easy to use erlang: trace/3 with options [send,'receive',timestamp] and get all messages from designated processes with timestamps sent to a "tracer" process. You can then do whatever analysis you need on that. Best regards, Robby On Apr 29, 2015 10:12 AM, "Roger Lipscombe" wrote: > For various reasons, I want a metric that measures how long messages > spend in a process message queue. The process is a gen_server, if that > has any bearing on the solution. Also, this is for production, so it > will be always-on and must be low-impact. > > I could do this by timestamping _every_ message sent to the process > and then reporting the deltas, but I think that's ugly. > > I thought of posting a message to self(), with a timestamp and then > measuring the delta. I could then post another one as soon as that > message is processed. Obviously, this leaves the process continually > looping, which is not good. > > So, instead, I could use erlang:send_after, but that requires two > messages: a delayed one triggered by erlang:send_after, and an > immediate one for measuring the delay. > > That's a bit complicated. > > Would it be sensible to send a message, with erlang:send_after, with > the _expected_ timestamp, and then compute the delta from that? > > Or, alternatively, what other ways could I measure how long a process > is taking to handle its message queue? > > Thanks, > Roger. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From llbgurs@REDACTED Wed Apr 29 13:38:12 2015 From: llbgurs@REDACTED (linbo liao) Date: Wed, 29 Apr 2015 19:38:12 +0800 Subject: [erlang-questions] Why not standard OTP behavior Message-ID: Hi All, When go through some open source code, I find sometimes processes are not standard OTP behavior like gen_server, gen_fsm, gen_event, but designed by internal OTP API. For example, ranch_conns_sup from ranch, riak_core_vnode_proxy from riak_core. May anyone help to point the reason, why they don't use OTP behavior but build by internal API, any drawback for OTP behavior in a certain situation? Thanks, Linbo -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Wed Apr 29 13:50:21 2015 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Wed, 29 Apr 2015 14:50:21 +0300 Subject: [erlang-questions] Why not standard OTP behavior In-Reply-To: References: Message-ID: <5540C57D.8070207@ninenines.eu> In the case of ranch_conns_sup, that's an optimization. Before it was a supervisor + gen_server, but the work was kind of duplicated (one was using links and the other monitors for the same processes). Now we have one process that do both tasks (supervise and limit the number of concurrent connections). On 04/29/2015 02:38 PM, linbo liao wrote: > Hi All, > > When go through some open source code, I find sometimes processes are > not standard OTP behavior like gen_server, gen_fsm, gen_event, but > designed by internal OTP API. > > For example, ranch_conns_sup > > from ranch, riak_core_vnode_proxy > > from riak_core. > > May anyone help to point the reason, why they don't use OTP behavior but > build by internal API, any drawback for OTP behavior in a certain situation? > > Thanks, > Linbo > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin http://ninenines.eu From roger@REDACTED Wed Apr 29 13:55:57 2015 From: roger@REDACTED (Roger Lipscombe) Date: Wed, 29 Apr 2015 12:55:57 +0100 Subject: [erlang-questions] Measuring message queue delay In-Reply-To: References: Message-ID: On 29 April 2015 at 12:24, Robert Raschke wrote: > maybe you can use the tracing facilities of Erlang. At a low level it is > reasonably easy to use erlang: trace/3 with options > [send,'receive',timestamp] and get all messages from designated processes > with timestamps sent to a "tracer" process. Ah, I *was* looking in the seq_trace stuff, but I missed the bit about timestamps. This is useful to know, thanks. > You can then do whatever analysis you need on that. Hmm, this is ultimately intended for publishing to graphite, rather than for live inspection. This means that designating specific processes is problematic. What impact would there be having this permanently enabled for, say, 30K processes in production? I assume that the "tracer" process would get hammered? From sergej.jurecko@REDACTED Wed Apr 29 13:56:10 2015 From: sergej.jurecko@REDACTED (=?UTF-8?Q?Sergej_Jure=C4=8Dko?=) Date: Wed, 29 Apr 2015 13:56:10 +0200 Subject: [erlang-questions] Measuring message queue delay In-Reply-To: References: Message-ID: Perhaps a crazy idea, but it should work and perform quite well.... Non 18+ compatible solution: - Keep track of refs. They are monotonically increasing across all schedulers. Create a timer somewhere that periodically saves to an ets table {make_ref(),os:timestamp()} (or some other data structure). When checking how old a message is, find which two refs in ETS your ref belongs to. That was the timeframe of that message. 18+ compatible solution: - Do this for every scheduler in the system, because make_ref will be per scheduler then. For every message, you must call erlang:system_info(scheduler_id) Sergej On Wed, Apr 29, 2015 at 11:12 AM, Roger Lipscombe wrote: > For various reasons, I want a metric that measures how long messages > spend in a process message queue. The process is a gen_server, if that > has any bearing on the solution. Also, this is for production, so it > will be always-on and must be low-impact. > > I could do this by timestamping _every_ message sent to the process > and then reporting the deltas, but I think that's ugly. > > I thought of posting a message to self(), with a timestamp and then > measuring the delta. I could then post another one as soon as that > message is processed. Obviously, this leaves the process continually > looping, which is not good. > > So, instead, I could use erlang:send_after, but that requires two > messages: a delayed one triggered by erlang:send_after, and an > immediate one for measuring the delay. > > That's a bit complicated. > > Would it be sensible to send a message, with erlang:send_after, with > the _expected_ timestamp, and then compute the delta from that? > > Or, alternatively, what other ways could I measure how long a process > is taking to handle its message queue? > > Thanks, > Roger. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rtrlists@REDACTED Wed Apr 29 15:06:05 2015 From: rtrlists@REDACTED (Robert Raschke) Date: Wed, 29 Apr 2015 14:06:05 +0100 Subject: [erlang-questions] Measuring message queue delay In-Reply-To: References: Message-ID: On Apr 29, 2015 12:55 PM, "Roger Lipscombe" wrote: > Hmm, this is ultimately intended for publishing to graphite, rather > than for live inspection. This means that designating specific > processes is problematic. What impact would there be having this > permanently enabled for, say, 30K processes in production? I assume > that the "tracer" process would get hammered? Err, yeah, you do not want to do this for all processes. >From your server, you could start a process that sends regular marker messages (using refs) to the server. And a tracer process to trace 'receive' against the server and send against the marker process. Your tracer process can correlate the marker messages and compute times between send and receive. That sounds reasonably scalable. Although you might need to make sure your marker process doesn't flood the server. Probably by synchronising, ie., have the server send a reply to the marker. Having written this down, I think you don't really need tracing with this at all. Just time stamp the marker messages. Regards, Robby -------------- next part -------------- An HTML attachment was scrubbed... URL: From mononcqc@REDACTED Wed Apr 29 15:19:35 2015 From: mononcqc@REDACTED (Fred Hebert) Date: Wed, 29 Apr 2015 09:19:35 -0400 Subject: [erlang-questions] Measuring message queue delay In-Reply-To: References: Message-ID: <20150429131933.GE32211@ferdair.local> On 04/29, Roger Lipscombe wrote: >For various reasons, I want a metric that measures how long messages >spend in a process message queue. The process is a gen_server, if that >has any bearing on the solution. Also, this is for production, so it >will be always-on and must be low-impact. > There's a possibility you can get the value in a roundabout way. Say for example you need an acknowledgement to know when a task it done. You could, then, set a local timer when you send the task and keep it in memory. When the server fetches the message, it takes a stamp, and when it's done with it, it takes another, tells you how long (in ms or ?s) it has worked on it, and returns it with the final ack. When the caller receives it, the delta between the reception and sending time is the time spent in transit. Sender Server | | Ts1 -----------------------> | | Tr1 | | | <----------------------- Tr2 Ts2 | | | Transit = Ts2 - Ts1 - (Tr2 - Tr2) Now this does mean you have two queues to worry about, but by adding this diagnostic message to a bunch of servers (and it can be as simple as a ping you assume takes 0ms to work on), you're able to report on the time overhead you're spending shuttling data around. If you maintain the sender's queue empty, you can likely handwave its overhead away, giving you a ping message a bit like this: Sender Server | | Ts1 -----------------------> | | | | | | <----------------------- Pong Ts2 | | | Transit = Ts2 - Ts1 - (0) The caveat is of course that if the Sender is scheduled out because the system is overloaded, it will appear that the message was in flight longer than it was truly the case. On the other hand, this reduces the load on your server process because it has to do a lot less work to handle the metrics calls. This does mean that if you do want to enable it permanently for 30k processes, you have to understand that no matter what you do, this means that you're gonna need two calls to a clock per message per process. If you poll every minute, you're gonna end up with 60k calls averaged over it all, or 1,000 per second. In practice it might be more intense than that. That might be tolerable, or that might not be. That really depends on your needs there. You can try it for a while, but what you may find out will depend on your load; if most of your messages are requiring a similar workload, you'll soon find out that the queue length is an adequate proxy for time spent waiting. If the message size or some complexity of it is equivalent to blocking for a long time not servicing the queue, then counting *these* messages in the system will likely be a lighter proxy that is almost as accurate. Eventually you may find bottlenecks on which multiple processes wait, and just looking at these bottleneck processes will work instead of monitoring everything in your system; better even, you'll be able to identify where to shed load or apply backpressure to keep the rest of the system healthy. Regards, Fred. From peppe@REDACTED Wed Apr 29 15:27:52 2015 From: peppe@REDACTED (Peter Andersson) Date: Wed, 29 Apr 2015 15:27:52 +0200 Subject: [erlang-questions] Error in Common Tests In-Reply-To: References: <5540B947.1050705@ninenines.eu> Message-ID: <5540DC58.6020307@erlang.org> Roberto, You won't get test case line info if it's the last line of execution that causes the timetrap timeout. Because of tail call elimination the {Suite,Testcase,Line} info is not saved on stack unless there's a trailing expression. I.e. tc(_) -> func_that_may_not_return(). will not generate test case line info if it fails, but tc(_) -> func_that_may_not_return(), ok. will. That said, Common Test removes internal common_test and test_server elements from the stack before printing the stack trace to the log. This operation is too agressive and removes all elements if no {Suite,Testcase,Line} tuple is found "to sync with". Knowing it must be the last call in the test case causing the timeout and knowing which internal function executed the test case, I should be able to print relevant stack info to the log even when a test case tuple is not present due to tail call elimination. Fix is coming asap... Best, Peter Ericsson AB, Erlang/OTP On 2015-04-29 13:15, Roberto Ostinelli wrote: > Lo?c, > Yes that did help me. It just stated {error,eoc} (i.e. no line of > where this happened) but at least I could track it down to the issue. > > Thank you. > r. > > On Wed, Apr 29, 2015 at 12:58 PM, Lo?c Hoguin > wrote: > > Did you look inside the HTML logs that Common Test generates? They > usually contain the complete error and stacktrace, while the > console only gets partials. > > > On 04/29/2015 01:54 PM, Roberto Ostinelli wrote: > > Dear list, > When running CT I have a test where I get: > > - - - - - - - - - - - - - - - - - - - - - - - - - - > Error detected: {'EXIT',{badmatch,{...}}} > - - - - - - - - - - - - - - - - - - - - - - - - - - > > No additional information. My application logs do not show any > errors. > > Weirdly enough, if I put a timer:sleep(10) somewhere in my > code, this > error does not show up anymore. > > Thing is: how can I get CT to tell me the module / line where > this error > happens? > > Thank you, > r. > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > -- > Lo?c Hoguin > http://ninenines.eu > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From prof3ta@REDACTED Wed Apr 29 16:50:19 2015 From: prof3ta@REDACTED (Roberto Aloi) Date: Wed, 29 Apr 2015 16:50:19 +0200 Subject: [erlang-questions] erlang:system_monitor in production? In-Reply-To: References: Message-ID: Hi Roger, Is it OK to have erlang:system_monitor permanently enabled in production? > One problem with `erlang:system_monitor/X` combined with an option such as `long_gc` is that, for each GC, the C equivalent of `erlang:now/0` - which has some very well known overhead - is called before and after each GC: https://github.com/erlang/otp/blob/feb45017da36be78d4c5784d758ede619fa7bfd3/erts/emulator/beam/erl_gc.c#L421 In other words, keeping long_gc monitoring enabled will add a couple of sync points for each GC. Disclaimer: It was not me discovering this, but a colleague, so not taking any merit or blame for the discovery. Roberto -------------- next part -------------- An HTML attachment was scrubbed... URL: From huanchuanjian@REDACTED Wed Apr 29 15:20:41 2015 From: huanchuanjian@REDACTED (=?UTF-8?B?5a6m5Lyg5bu6?=) Date: Wed, 29 Apr 2015 21:20:41 +0800 Subject: [erlang-questions] Erlang with lua Message-ID: Hi All, Our project want to use lua in erlang, through the comparison for some open source projects, we try to use moon [https://github.com/chaos-ad/moon], When go through some documents and source code, I find start lua vm need to spawn the os thread. May anyone help to point the the best practice of using moon, thread pool or single thread? Thanks, Chuanjian Huan -------------- next part -------------- An HTML attachment was scrubbed... URL: From desired.mta@REDACTED Wed Apr 29 17:33:49 2015 From: desired.mta@REDACTED (=?UTF-8?Q?Motiejus_Jak=C5=A1tys?=) Date: Wed, 29 Apr 2015 17:33:49 +0200 Subject: [erlang-questions] Erlang with lua In-Reply-To: References: Message-ID: On Wed, Apr 29, 2015 at 3:20 PM, ??? wrote: > Hi All, > > Our project want to use lua in erlang, through the comparison for some open > source projects, we try to use > > moon [https://github.com/chaos-ad/moon], When go through some documents and > source code, I find start lua vm need to spawn the os thread. > > May anyone help to point the the best practice of using moon, thread pool or > single thread? Moon was created by my ex-colleague who, to my knowledge, is no longer on this list. CCing him. To give some background, Moon is his child of frustration working with my much dumber erlualib: https://github.com/Motiejus/erlualib If I remember correctly, he used a thread pool. Anatoly, could you clarify? -- Motiejus Jak?tys From jesper.louis.andersen@REDACTED Wed Apr 29 17:50:20 2015 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Wed, 29 Apr 2015 17:50:20 +0200 Subject: [erlang-questions] Measuring message queue delay In-Reply-To: References: Message-ID: On Wed, Apr 29, 2015 at 11:12 AM, Roger Lipscombe wrote: > I thought of posting a message to self(), with a timestamp and then > measuring the delta. I could then post another one as soon as that > message is processed. Obviously, this leaves the process continually > looping, which is not good. > I don't think this should be thrown out. It is salvagable, I think, and here is how: We update our gen_server with a new element in the state record: -record(state, { ..., marker_active = false }). Whenever we get a normal message, we check this state: handle_call(Msg, From, #state { marker_active = false } = State) -> self() ! {marker, erlang:monotonic_time()}, handle_call(Msg, From, State#state { marker_active = true }); .... Of course handle_cast/2 and handle_info/2 needs the same treatment. This means whenever there is flow in the mailbox, we enable a marker with a timestamp. Now, we handle the marker: handle_info({marker, Old}, State) -> Diff = erlang:monotonic_time() - Old, report_to_hdrhistogram(erlang:convert_time_unit(Diff, native, milli_seconds)), {noreply, State#state { marker_active = false }} ... This means once we get a marker, we process its queue sojourn latency, but we avoid reinjecting a new marker. The next processed message will do that. In turn, if you have 1 message going in, there will be exactly one marker. But if other processes manages to enter 100 messages between a marker and the next one, then we have one marker per 100 messages. The process will never spin if there is nothing to process. An alternative is to add some sampling interval delay: handle_info({marker, Old}, State) -> Diff = erlang:monotonic_time() - Old, report_to_hdrhistogram(erlang:convert_time_unit(Diff, native, milli_seconds)), timer:send_after(50, self(), rearm_marker), {noreply, State}; handle_info(rearm_marker, State) -> {noreply, State#state { marker_active = false }}; ... which closes off the problem where you have markers all over the place all the time. The essential idea is to use the arrival of a "normal" message as a trigger for invoking a time sojourn marker. And never have the marker itself reinject a marker so it can't cycle. In turn, there has to be a flux through your mailbox queue for the markers to get added, and if there is no flux, no marker will ever get added. The really astute reader will prove my ramblings :) -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From roger@REDACTED Wed Apr 29 18:45:59 2015 From: roger@REDACTED (Roger Lipscombe) Date: Wed, 29 Apr 2015 17:45:59 +0100 Subject: [erlang-questions] erlang:system_monitor in production? In-Reply-To: References: Message-ID: Hmmm. Could that not be fixed with the new timing stuff in 18? It only needs this-scheduler elapsed-time, right? On 29 April 2015 at 15:50, Roberto Aloi wrote: > Hi Roger, > >> Is it OK to have erlang:system_monitor permanently enabled in production? > > > One problem with `erlang:system_monitor/X` combined with an option such as > `long_gc` > is that, for each GC, the C equivalent of `erlang:now/0` - which has some > very well known overhead - is called before and after each GC: > > https://github.com/erlang/otp/blob/feb45017da36be78d4c5784d758ede619fa7bfd3/erts/emulator/beam/erl_gc.c#L421 > > In other words, keeping long_gc monitoring enabled will add a couple of sync > points for each GC. > > Disclaimer: It was not me discovering this, but a colleague, so not taking > any merit or blame for the discovery. > > Roberto From gomoripeti@REDACTED Wed Apr 29 19:52:51 2015 From: gomoripeti@REDACTED (=?ISO-8859-1?Q?Peti_G=F6m=F6ri?=) Date: Wed, 29 Apr 2015 19:52:51 +0200 Subject: [erlang-questions] erlang:system_monitor in production? In-Reply-To: References: Message-ID: there is a related open pull request https://github.com/erlang/otp/pull/664 On Wed, Apr 29, 2015 at 6:45 PM, Roger Lipscombe wrote: > Hmmm. Could that not be fixed with the new timing stuff in 18? It only > needs this-scheduler elapsed-time, right? > > On 29 April 2015 at 15:50, Roberto Aloi wrote: >> Hi Roger, >> >>> Is it OK to have erlang:system_monitor permanently enabled in production? >> >> >> One problem with `erlang:system_monitor/X` combined with an option such as >> `long_gc` >> is that, for each GC, the C equivalent of `erlang:now/0` - which has some >> very well known overhead - is called before and after each GC: >> >> https://github.com/erlang/otp/blob/feb45017da36be78d4c5784d758ede619fa7bfd3/erts/emulator/beam/erl_gc.c#L421 >> >> In other words, keeping long_gc monitoring enabled will add a couple of sync >> points for each GC. >> >> Disclaimer: It was not me discovering this, but a colleague, so not taking >> any merit or blame for the discovery. >> >> Roberto > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From roger@REDACTED Thu Apr 30 00:08:39 2015 From: roger@REDACTED (Roger Lipscombe) Date: Wed, 29 Apr 2015 23:08:39 +0100 Subject: [erlang-questions] Why not standard OTP behavior In-Reply-To: <5540C57D.8070207@ninenines.eu> References: <5540C57D.8070207@ninenines.eu> Message-ID: I found this slide deck useful for explaining the pros and cons of using OTP behaviours vs. roll-your-own. Lo?c might find it familiar ;-) http://ninenines.eu/talks/beyond-otp/beyond-otp.html On 29 April 2015 at 12:50, Lo?c Hoguin wrote: > In the case of ranch_conns_sup, that's an optimization. > > Before it was a supervisor + gen_server, but the work was kind of duplicated > (one was using links and the other monitors for the same processes). Now we > have one process that do both tasks (supervise and limit the number of > concurrent connections). > > On 04/29/2015 02:38 PM, linbo liao wrote: >> >> Hi All, >> >> When go through some open source code, I find sometimes processes are >> not standard OTP behavior like gen_server, gen_fsm, gen_event, but >> designed by internal OTP API. >> >> For example, ranch_conns_sup >> >> from ranch, riak_core_vnode_proxy >> >> >> from riak_core. >> >> May anyone help to point the reason, why they don't use OTP behavior but >> build by internal API, any drawback for OTP behavior in a certain >> situation? >> >> Thanks, >> Linbo >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > -- > Lo?c Hoguin > http://ninenines.eu > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From jay@REDACTED Thu Apr 30 00:26:52 2015 From: jay@REDACTED (Jay Nelson) Date: Wed, 29 Apr 2015 15:26:52 -0700 Subject: [erlang-questions] Why not standard OTP behavior Message-ID: An older presentation that I did at Erlang Factory Lite in Vancouver gives more details about making a process ?OTP compliant?. https://erlangcentral.org/managing-processes-without-otp-and-how-to-make-them-otp-compliant/#.VUFaW61Vikp jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From peter@REDACTED Thu Apr 30 06:31:48 2015 From: peter@REDACTED (Peter Membrey) Date: Thu, 30 Apr 2015 12:31:48 +0800 (HKT) Subject: [erlang-questions] Corrupted term on distribution channel from a C-Node In-Reply-To: <203854868.185615.1430181292465.JavaMail.zimbra@membrey.hk> References: <2080062346.185311.1430127950846.JavaMail.zimbra@membrey.hk> <580847300.185460.1430142823456.JavaMail.zimbra@mail.membrey.hk> <203854868.185615.1430181292465.JavaMail.zimbra@membrey.hk> Message-ID: <1823916324.187657.1430368308754.JavaMail.zimbra@membrey.hk> Hi guys, Might be getting a little closer :) It looks like the cookie itself is not corrupted. I've gone over network captures and it seems none of the SEND messages actually include the cookie. In fact I wasn't able to find the cookie "in the clear" anywhere between the time the connection was made and when it was closed. However the next thing that should come after the cookie is the destination PID and this seems to be where we're seeing the problem. I've been able to sort of replicate it in development, in that I can now get it to crash within 30 minutes or so, rather than waiting 6 months and praying. If I run it under strace or valgrind, the problem appears to go away (good old Heisenbug). I've added code to print the node name part of the struct when it diverges from what I expect. So far this code has not been triggered, even though it has still crashed. This suggests that the PID struct is probably good. This PID struct is passed directly to ei_send. >From the network capture I can confirm that a corrupt PID was sent on the wire. Interestingly however, there were valid messages before it and valid messages after it which suggests it's not a global state issue. I'll continue to dig into this. Right now I suspect we're passing dodgy PIDs to ei_send, but is it possible that ei_send is doing something funky? As always and feedback or ideas gratefully received :) Kind Regards, Peter Membrey ----- Original Message ----- From: "Peter Membrey" To: "Ben Murphy" Cc: "Erlang (E-mail)" Sent: Tuesday, 28 April, 2015 08:34:52 Subject: Re: [erlang-questions] Corrupted term on distribution channel from a C-Node Hi guys, Thanks for all the feedback! :) The modifications I made were at a higher level than ei_writev, so hopefully I haven't inadvertently caused this problem - though of course it's quite possible that I have :) The changes really only provided a way to tell the receive family of functions not to reply to a heart beat directly. Then when you get an ERL_TICK, you need to handle it yourself, which is easy enough to wrap in a mutex. That way you can still use blocking reads and don't have to worry about timeouts before whole messages could be read (and all sorts of funky other workarounds that were tried). The PR is here with the doco: https://github.com/erlang/otp/pulls/pmembrey I've been meaning to complete the test cases so this could be merged upstream... The send_tock() function sends a heartbeat reply which is {0x00,0x00,0x00,0x00} - I wonder if that is what is being interleaved by mistake? Still, if you look from line 93: https://github.com/pmembrey/otp/blob/maint/lib/erl_interface/src/connect/send.c#L93 v[0].iov_base = (char *)header; v[0].iov_len = index; v[1].iov_base = (char *)msg; v[1].iov_len = msglen; This gets sent to ei_send() - but from what I can tell, these are effectively separate write operations. Is it possible for the first vector to fail to write completely, and then for the second write to start and complete? Thanks again! Kind Regards, Peter Membrey ----- Original Message ----- From: "Ben Murphy" To: "Peter Membrey" Cc: "Jesper Louis Andersen" , "Erlang (E-mail)" Sent: Tuesday, 28 April, 2015 00:54:34 Subject: Re: [erlang-questions] Corrupted term on distribution channel from a C-Node Hi, There still might be a threading issue with writes to that fd. I would change the erlang code so that ei_write_t/ei_writev_t asserts that the lock is held when it is called. Maybe there is another place that is calling ei_write_t/ei_writev_t without the lock that you have not caught. Or alternatively move the lock into erlang and the lock presumably should be taken at ei_writev_fill_t and ei_write_fill_t. Because you are locking outside at a higher layer it is hard to know for sure if you are locking correctly at all the places. Also in connect this code looks suspect. Other stuff is calling get_ei_socket_info and then reading stuff from the structure. But it is possible that the memory pointed to by the pointer has been freed and is being used by some other allocation. stuff that reads from this structure needs to take out the mutex over the whole code path then copy it out before unlocking. It could be that this is the problem because it looks like one of your symptoms :) static ei_socket_info* get_ei_socket_info(int fd) { int i; #ifdef _REENTRANT ei_mutex_lock(ei_sockets_lock, 0); #endif /* _REENTRANT */ for (i = 0; i < ei_n_sockets; ++i) if (ei_sockets[i].socket == fd) { /*fprintf("get_ei_socket_info %d %d \"%s\"\n", fd, ei_sockets[i].dist_version, ei_sockets[i].cookie);*/ #ifdef _REENTRANT ei_mutex_unlock(ei_sockets_lock); #endif /* _REENTRANT */ return &ei_sockets[i]; } #ifdef _REENTRANT ei_mutex_unlock(ei_sockets_lock); #endif /* _REENTRANT */ return NULL; } On Mon, Apr 27, 2015 at 2:53 PM, Peter Membrey wrote: > Hi, > It's non deterministic for sure. We run both a primary and secondary app reading in the same data (multicast). One had this issue and the other did not. Although the hardware is slightly different, the versions and configurations are basically identical. > I believe we are running R16B03. I'll be able to double check tomorrow (can't believe I forgot to include that in the first email). > It's actually a slightly patched version to handle the issue of not being able to properly lock a tcp connection as the receive function would block but handle the heart beat response internally. That version can be found here: > https://github.com/pmembrey/otp > I'm sending from my phone so apologies for any typos or mistakes. > Thanks again for your help! > Kind regards, > Peter Membrey > On 27 Apr 2015 21:38, Jesper Louis Andersen wrote: > > > On Mon, Apr 27, 2015 at 11:45 AM, Peter Membrey wrote: > >> Having gone through the message with the EDP and external term man pages, >> it looks like we're getting a corrupt SEND command. In both cases the >> cookie is not being encoded correctly (the cookie is definitely not empty), >> and in the second instance, we have a 0 where we'd expect a 103. The >> messages following both corrupted SEND messages were decodable with >> binary_to_term/1 and the payload looked good. >> > > Is this nondeterministic? I wouldn't entirely rule out the possibility of > the hardware messing up, or some error in the code elsewhere manipulating > the wrong data. Not that I can pinpoint *that* is what is happening, but do > not rule it out from the start that this could be a hardware thing. What > version of Erlang is this? A recent one, or an earlier one? > > > -- > J. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions From roberto@REDACTED Thu Apr 30 15:41:17 2015 From: roberto@REDACTED (Roberto Ostinelli) Date: Thu, 30 Apr 2015 15:41:17 +0200 Subject: [erlang-questions] Rebar to run CT Message-ID: Dear List, I'm trying to use CT on Travis. Test run fine when run locally. I start them from the project root, and use rebar ct command. However, when I run then in Travis I get: WARN: 'ct' command does not apply to directory /home/travis/build/myrepo/myapp Travis already uses the latest version of rebar, and just in case I even supplied a latest one. So, this command is definitely in the rebar being run, it's just not understanding that it should also look in subdirs. My project is strutured as follows: -- myproject rebar.config |-- apps |-- myapp |-- src |-- test rebar.config |-- deps |-- dep1 |-- dep2 |-- ... rebar.config at root contains: {require_otp_vsn, "17"}. {erl_opts, [ debug_info, fail_on_warning, {parse_transform, lager_transform} ]}. {sub_dirs, [ "apps/myapp" ]}. {deps, [ %% my deps here ]}. rebar.config in app/myapp contains: {ct_log_dir, "/tmp/logs"}. Any ideas on how I can make Travis understand to run the CT in app/myapp/test? Thanks, r. -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto@REDACTED Thu Apr 30 15:47:45 2015 From: roberto@REDACTED (Roberto Ostinelli) Date: Thu, 30 Apr 2015 15:47:45 +0200 Subject: [erlang-questions] Error in Common Tests In-Reply-To: <5540DC58.6020307@erlang.org> References: <5540B947.1050705@ninenines.eu> <5540DC58.6020307@erlang.org> Message-ID: Thank you Peter! :) On Wed, Apr 29, 2015 at 3:27 PM, Peter Andersson wrote: > > Roberto, > > You won't get test case line info if it's the last line of execution that > causes the timetrap timeout. Because of tail call elimination the > {Suite,Testcase,Line} info is not saved on stack unless there's a trailing > expression. I.e. > > tc(_) -> func_that_may_not_return(). > > will not generate test case line info if it fails, but > > tc(_) -> func_that_may_not_return(), ok. > > will. > > That said, Common Test removes internal common_test and test_server > elements from the stack before printing the stack trace to the log. This > operation is too agressive and removes all elements if no > {Suite,Testcase,Line} tuple is found "to sync with". Knowing it must be the > last call in the test case causing the timeout and knowing which internal > function executed the test case, I should be able to print relevant stack > info to the log even when a test case tuple is not present due to tail call > elimination. Fix is coming asap... > > Best, > Peter > > Ericsson AB, Erlang/OTP > > > > On 2015-04-29 13:15, Roberto Ostinelli wrote: > > Lo?c, > Yes that did help me. It just stated {error,eoc} (i.e. no line of where > this happened) but at least I could track it down to the issue. > > Thank you. > r. > > On Wed, Apr 29, 2015 at 12:58 PM, Lo?c Hoguin wrote: > >> Did you look inside the HTML logs that Common Test generates? They >> usually contain the complete error and stacktrace, while the console only >> gets partials. >> >> >> On 04/29/2015 01:54 PM, Roberto Ostinelli wrote: >> >>> Dear list, >>> When running CT I have a test where I get: >>> >>> - - - - - - - - - - - - - - - - - - - - - - - - - - >>> Error detected: {'EXIT',{badmatch,{...}}} >>> - - - - - - - - - - - - - - - - - - - - - - - - - - >>> >>> No additional information. My application logs do not show any errors. >>> >>> Weirdly enough, if I put a timer:sleep(10) somewhere in my code, this >>> error does not show up anymore. >>> >>> Thing is: how can I get CT to tell me the module / line where this error >>> happens? >>> >>> Thank you, >>> r. >>> >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >> -- >> Lo?c Hoguin >> http://ninenines.eu >> > > > > _______________________________________________ > erlang-questions mailing listerlang-questions@REDACTED://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From egil@REDACTED Thu Apr 30 15:50:26 2015 From: egil@REDACTED (=?UTF-8?B?QmrDtnJuLUVnaWwgRGFobGJlcmc=?=) Date: Thu, 30 Apr 2015 15:50:26 +0200 Subject: [erlang-questions] Why does Dialyzer crash on this map? In-Reply-To: References: <54749C90.6000602@llaisdy.com> <5474BA4C.4030706@llaisdy.com> <547716F8.7010208@llaisdy.com> <4B9AEB77-A709-4D97-8EF2-8FE1AF5A4071@llaisdy.com> <55085881.3060508@erlang.org> <55095D8E.2050103@erlang.org> Message-ID: <55423322.6010703@erlang.org> I've finally manage to pop my stack to this issue. I can confirm that your example does trigger an dialyzer error on 17.5 .. though that is not the case on current master. I believe other fixes in format has remedied this problem on master. If you have the possibility, recheck your codebase with dialyzer using current master or 18.0-rc1 and see if any other problems persist. // Bj?rn-Egil On 04/23/2015 12:46 PM, Ali Sabil wrote: > Any update regarding this? > > On Thu, Apr 9, 2015 at 5:21 PM Ali Sabil > wrote: > > Sorry for the late reply, I finally managed to get a minimal test > case that reproduces the bug in 17.5: > > -module(sum). > -export([ > test/1 > ]). > > -spec test(#{atom() => term()}) -> integer(). > test(Data) -> > maps:fold(fun > (_Key, Value, Acc) when is_integer(Value) -> > Acc + Value; > (_Key, _Value, Acc) -> > Acc > end, 0, Data). > > > I don't know if this is the correct fix, but this makes dialyzer > work again: > > diff --git a/lib/hipe/cerl/erl_types.erl b/lib/hipe/cerl/erl_types.erl > index 4215448..bb4c1c1 100644 > --- a/lib/hipe/cerl/erl_types.erl > +++ b/lib/hipe/cerl/erl_types.erl > @@ -4594,6 +4594,8 @@ t_form_to_string({type, _L, list, [Type]}) -> > "[" ++ t_form_to_string(Type) ++ "]"; > t_form_to_string({type, _L, map, Args}) when not is_list(Args) -> > "#{}"; > +t_form_to_string({type, _L, map_field_assoc, Key, Value}) -> > + "#{" ++ t_form_to_string(Key) ++ "=>" ++ > t_form_to_string(Value) ++ "}"; > t_form_to_string({type, _L, mfa, []}) -> "mfa()"; > t_form_to_string({type, _L, module, []}) -> "module()"; > t_form_to_string({type, _L, node, []}) -> "node()"; > > Thanks, > > Are > > On Tue, Apr 7, 2015 at 2:31 PM Bj?rn-Egil Dahlberg > > wrote: > > Again - could you provide me with a sample code .. or at least > some sort of a clue to what you are dialyzing? > > 2015-04-07 14:05 GMT+02:00 Ali Sabil >: > > Hi again,Running dialyzer shipped with 17.5 on the same > code base leads now to the following error (17.4 works > without any errors): > > > > > ===> Error in dialyzing apps: Analysis failed with error: > {function_clause,[{erl_types,t_form_to_string, > [{type,36,map_field_assoc, > {type,36,atom,[]}, > {type,36,term,[]}}], > [{file,"erl_types.erl"},{line,4546}]}, > {erl_types,t_form_to_string_list,2, > [{file,"erl_types.erl"},{line,4637}]}, > {erl_types,t_form_to_string,1, > [{file,"erl_types.erl"},{line,4634}]}, > {erl_types,t_form_to_string_list,2, > [{file,"erl_types.erl"},{line,4637}]}, > {erl_types,t_form_to_string,1, > [{file,"erl_types.erl"},{line,4634}]}, > {dialyzer_contracts,contract_to_string_1,1, > [{file,"dialyzer_contracts.erl"}, > {line,107}]}, > {dialyzer_contracts,extra_contract_warning,6, > [{file,"dialyzer_contracts.erl"}, > {line,712}]}, > {dialyzer_contracts,picky_contract_check,7, > [{file,"dialyzer_contracts.erl"}, > {line,686}]}]} > Last messages in the log cache: > Reading files and computing callgraph... done in 1.21 secs > Removing edges... done in 0.04 secs > > > On Wed, Mar 18, 2015 at 12:12 PM Bj?rn-Egil Dahlberg > > wrote: > > On 2015-03-18 12:01, Ali Sabil wrote: > > I tried to create a minimal testcase but I > unfortunately haven't been > > able to. I was running dialyzer on a quite large > code base and now > > even the unpatched dialyzer works without any issue > after I fixed all > > the issues reported by dialyzer.Ah, well .. I > suspect it was the missing clause in find_terminals > and Ihad it on a TODO somewhere. Should be included to > 17.5.// Bj?rn-Egil > > > > > > > _______________________________________________ > erlang-questions mailing listerlang-questions@REDACTED > http://erlang.org/mailman/list > info/erlang-questions > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mononcqc@REDACTED Thu Apr 30 15:53:30 2015 From: mononcqc@REDACTED (Fred Hebert) Date: Thu, 30 Apr 2015 09:53:30 -0400 Subject: [erlang-questions] Rebar to run CT In-Reply-To: References: Message-ID: <20150430135306.GA610@ferdair.local> On 04/30, Roberto Ostinelli wrote: >Dear List, >I'm trying to use CT on Travis. Test run fine when run locally. I start >them from the project root, and use rebar ct command. > >However, when I run then in Travis I get: > >WARN: 'ct' command does not apply to directory >/home/travis/build/myrepo/myapp > This is a problem with rebar 2.x's model versus the directory structure you have. The one you have described here: > >-- myproject > rebar.config > |-- apps > |-- myapp > |-- src > |-- test > rebar.config > |-- deps > |-- dep1 > |-- dep2 > |-- ... > Works for a release, but rebar 2.x more or less expects OTP applications everywhere. Anything else and you need special configurations or switches. In the case of ct for that one, you have to run `rebar ct -r' to run it recursively in all directories it finds, and then add `skip_deps=true' to avoid running it on all your dependencies. Final command is: rebar ct -r skip_deps=true You'll still see a warning printed (when rebar looks for a `test/' directory at the root) but then it'll run the tests fine. Rebar3 should take care of this and work fine on the first run, as we changed the model it uses to figure out structure of both OTP apps and releases and takes an app-based structure. Regards, Fred. From roberto@REDACTED Thu Apr 30 15:59:30 2015 From: roberto@REDACTED (Roberto Ostinelli) Date: Thu, 30 Apr 2015 15:59:30 +0200 Subject: [erlang-questions] Rebar to run CT In-Reply-To: <20150430135306.GA610@ferdair.local> References: <20150430135306.GA610@ferdair.local> Message-ID: Hi Fred, Thank you. Unfortunately I'm receiving a `ERROR: invalid_option "-r"`. Using the latest rebar (cd55176009df794f506771fd574de9303ff2a42e on master). I'm already using the `skip_deps=true` option. Any ideas? On Thu, Apr 30, 2015 at 3:53 PM, Fred Hebert wrote: > On 04/30, Roberto Ostinelli wrote: > >> Dear List, >> I'm trying to use CT on Travis. Test run fine when run locally. I start >> them from the project root, and use rebar ct command. >> >> However, when I run then in Travis I get: >> >> WARN: 'ct' command does not apply to directory >> /home/travis/build/myrepo/myapp >> >> > This is a problem with rebar 2.x's model versus the directory structure > you have. The one you have described here: > > >> -- myproject >> rebar.config >> |-- apps >> |-- myapp >> |-- src >> |-- test >> rebar.config >> |-- deps >> |-- dep1 >> |-- dep2 >> |-- ... >> >> > Works for a release, but rebar 2.x more or less expects OTP applications > everywhere. Anything else and you need special configurations or switches. > > In the case of ct for that one, you have to run `rebar ct -r' to run it > recursively in all directories it finds, and then add `skip_deps=true' to > avoid running it on all your dependencies. Final command is: > > rebar ct -r skip_deps=true > > You'll still see a warning printed (when rebar looks for a `test/' > directory at the root) but then it'll run the tests fine. > > Rebar3 should take care of this and work fine on the first run, as we > changed the model it uses to figure out structure of both OTP apps and > releases and takes an app-based structure. > > Regards, > Fred. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mononcqc@REDACTED Thu Apr 30 16:08:40 2015 From: mononcqc@REDACTED (Fred Hebert) Date: Thu, 30 Apr 2015 10:08:40 -0400 Subject: [erlang-questions] Rebar to run CT In-Reply-To: References: <20150430135306.GA610@ferdair.local> Message-ID: <20150430140839.GB610@ferdair.local> On 04/30, Roberto Ostinelli wrote: >Hi Fred, >Thank you. Unfortunately I'm receiving a `ERROR: invalid_option "-r"`. >Using the latest rebar (cd55176009df794f506771fd574de9303ff2a42e on master). > >I'm already using the `skip_deps=true` option. > >Any ideas? > That sounds like a bug. rebar ct -r skip_deps=true is the only way I ever got anything working with releases in rebar 2.x. From roberto@REDACTED Thu Apr 30 16:22:02 2015 From: roberto@REDACTED (Roberto Ostinelli) Date: Thu, 30 Apr 2015 16:22:02 +0200 Subject: [erlang-questions] Rebar to run CT In-Reply-To: <20150430140839.GB610@ferdair.local> References: <20150430135306.GA610@ferdair.local> <20150430140839.GB610@ferdair.local> Message-ID: ach. On Thu, Apr 30, 2015 at 4:08 PM, Fred Hebert wrote: > On 04/30, Roberto Ostinelli wrote: > >> Hi Fred, >> Thank you. Unfortunately I'm receiving a `ERROR: invalid_option "-r"`. >> Using the latest rebar (cd55176009df794f506771fd574de9303ff2a42e on >> master). >> >> I'm already using the `skip_deps=true` option. >> >> Any ideas? >> >> > That sounds like a bug. rebar ct -r skip_deps=true is the only way I ever > got anything working with releases in rebar 2.x. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From roger@REDACTED Thu Apr 30 16:39:22 2015 From: roger@REDACTED (Roger Lipscombe) Date: Thu, 30 Apr 2015 15:39:22 +0100 Subject: [erlang-questions] Rebar to run CT Message-ID: On 30 April 2015 at 14:53, Fred Hebert wrote: > Works for a release, but rebar 2.x more or less expects OTP applications > everywhere. Anything else and you need special configurations or switches. Ah. Might that be why I can't get "rebar eunit" to work on my (similar) directory structure on anything newer than rebar 2.2.0? From mononcqc@REDACTED Thu Apr 30 16:46:57 2015 From: mononcqc@REDACTED (Fred Hebert) Date: Thu, 30 Apr 2015 10:46:57 -0400 Subject: [erlang-questions] Rebar to run CT In-Reply-To: References: Message-ID: <20150430144656.GC610@ferdair.local> On 04/30, Roger Lipscombe wrote: >On 30 April 2015 at 14:53, Fred Hebert wrote: >> Works for a release, but rebar 2.x more or less expects OTP applications >> everywhere. Anything else and you need special configurations or switches. > >Ah. Might that be why I can't get "rebar eunit" to work on my >(similar) directory structure on anything newer than rebar 2.2.0? Yes. Somewhere in the midst of this entire thing, rebar, through a minor version bump, changed how it executed some commands -- some that were recursive stopped being so, and some that were not recursive started doing that. If it happened at around that time, it would explain it. From tuncer.ayaz@REDACTED Thu Apr 30 21:40:10 2015 From: tuncer.ayaz@REDACTED (Tuncer Ayaz) Date: Thu, 30 Apr 2015 21:40:10 +0200 Subject: [erlang-questions] Rebar to run CT In-Reply-To: References: Message-ID: {erl_opts, [ debug_info, fail_on_warning, {parse_transform, lager_transform} ]}. should be {erl_opts, [ warnings_as_errors, {parse_transform, lager_transform} ]}. to use the right compiler flag and omit what's already enabled by default. From siraaj@REDACTED Thu Apr 30 22:42:47 2015 From: siraaj@REDACTED (Siraaj Khandkar) Date: Thu, 30 Apr 2015 16:42:47 -0400 Subject: [erlang-questions] Automatic currying Message-ID: <554293C7.7000301@khandkar.net> One of the things I often wish for, when using Erlang, is staged application of any function without having to manually wrap it in stage funs (which you get in languages that feature curried-by-default functions). The other day it occurred to me that the ability to get fun arity info and to apply arguments as a list was all that was needed to get very, very close to the behavior I wanted. Voila: -module(function). -export([curry/1]). curry(F) -> Info = erlang:fun_info(F), {value, {arity, Arity}} = lists:keysearch(arity, 1, Info), curry(F, [], Arity). curry(F, Args, 0) -> apply(F, lists:reverse(Args)); curry(F, Args, Arity) -> fun (X) -> curry(F, [X | Args], Arity - 1) end. In action: $ erl 1> 1> Nums = lists:seq(1, 10). [1,2,3,4,5,6,7,8,9,10] 2> 2> Plus = function:curry(fun (X, Y) -> X + Y end). #Fun 3> 3> lists:map(Plus(1), Nums). [2,3,4,5,6,7,8,9,10,11] 4> 4> lists:map(Plus(3), Nums). [4,5,6,7,8,9,10,11,12,13] 5> 5> lists:map(Plus(5), Nums). [6,7,8,9,10,11,12,13,14,15] 6> 6> Fold = function:curry(fun lists:foldl/3). #Fun 7> 7> AddTo = Fold(fun (X, Y) -> X + Y end). #Fun 8> 8> AddTo0 = AddTo(0). #Fun 9> 9> AddTo100 = AddTo(100). #Fun 10> 10> AddTo0(Nums). 55 11> AddTo100(Nums). 155 12> 12> AddTo100(lists:seq(4, 78)). 3175 13> From dszoboszlay@REDACTED Thu Apr 30 23:27:17 2015 From: dszoboszlay@REDACTED (=?UTF-8?Q?D=C3=A1niel_Szoboszlay?=) Date: Thu, 30 Apr 2015 23:27:17 +0200 Subject: [erlang-questions] Measuring message queue delay In-Reply-To: References: Message-ID: Although I agree that the "normal message triggers a marker" algorithm solves the problem perfectly, I would also like to add a method to the table that could give a reasonably good approximation of the message queue delay without sending any marker messages. Just for the fun of it. Once again, our process has two states: doing a measurement and not doing a measurement. When it receives a message while not doing a measurement it shall: - Save the current length of its message queue (N) - Save the current time (T1) - Switch to doing a measurement state While doing a measurement simply count the number of messages received until we reach N messages. At this point save the current time (T2), and return to not doing a measurement mode. T2 - T1 is a *lower limit* on the message queue delay (for this N-th message), because the message we are processing at T2 was already in the queue at T1. The actual delay may be somewhat larger, since the message entered the queue at some T0 < T1 time. We can also get an *upper limit* on the message queue delay if we'd check the time (T3) at the processing of the N+1:th message: message N+1 was not yet in the message queue at T1 and got out of the queue at T3, so spent at most T3 - T1 time waiting. One more thing to note: T3 = T2 + the time spent processing the N-th message + the time spent between handling two messages (which is: gen_server overhead, waiting for the next message to arrive and waiting for the CPU to run). Since gen_server overhead is low and we can assume there was no time spent waiting for the next message (that would mean the queue is empty, and the delay is 0), we can approximate T3 as T2 + time spent processing the N-th message (the error is the time spent waiting for the CPU). So when we finish processing the N-th message we will have a definite lower limit and a quite accurate upper limit for the message queue delay. They are not measured for the same message though, but as long as there are no sudden jumps or drops in the message queue delay it doesn't really matter. It's up to you whether you want to report both the upper and lower limits as metrics or calculate a single value from them (e.g. take either of them, calculate the average or whatever). In case of gen_servers one possible caveat is that there may be system messages in the queue as well which are not delivered to your callback module. This means your message counting will be off by one. But system messages are extremely rare and unless the process is otherwise idle they won't cause a significant measurement error. Cheers, Daniel 2015-04-29 17:50 GMT+02:00 Jesper Louis Andersen < jesper.louis.andersen@REDACTED>: > > On Wed, Apr 29, 2015 at 11:12 AM, Roger Lipscombe > wrote: > >> I thought of posting a message to self(), with a timestamp and then >> measuring the delta. I could then post another one as soon as that >> message is processed. Obviously, this leaves the process continually >> looping, which is not good. >> > > I don't think this should be thrown out. It is salvagable, I think, and > here is how: > > We update our gen_server with a new element in the state record: > > -record(state, { ..., marker_active = false }). > > Whenever we get a normal message, we check this state: > > handle_call(Msg, From, #state { marker_active = false } = State) -> > self() ! {marker, erlang:monotonic_time()}, > handle_call(Msg, From, State#state { marker_active = true }); > .... > > Of course handle_cast/2 and handle_info/2 needs the same treatment. This > means whenever there is flow in the mailbox, we enable a marker with a > timestamp. > > Now, we handle the marker: > > handle_info({marker, Old}, State) -> > Diff = erlang:monotonic_time() - Old, > report_to_hdrhistogram(erlang:convert_time_unit(Diff, native, > milli_seconds)), > {noreply, State#state { marker_active = false }} > ... > > This means once we get a marker, we process its queue sojourn latency, but > we avoid reinjecting a new marker. The next processed message will do that. > > In turn, if you have 1 message going in, there will be exactly one marker. > But if other processes manages to enter 100 messages between a marker and > the next one, then we have one marker per 100 messages. The process will > never spin if there is nothing to process. An alternative is to add some > sampling interval delay: > > handle_info({marker, Old}, State) -> > Diff = erlang:monotonic_time() - Old, > report_to_hdrhistogram(erlang:convert_time_unit(Diff, native, > milli_seconds)), > timer:send_after(50, self(), rearm_marker), > {noreply, State}; > handle_info(rearm_marker, State) -> > {noreply, State#state { marker_active = false }}; > ... > > which closes off the problem where you have markers all over the place all > the time. > > The essential idea is to use the arrival of a "normal" message as a > trigger for invoking a time sojourn marker. And never have the marker > itself reinject a marker so it can't cycle. In turn, there has to be a flux > through your mailbox queue for the markers to get added, and if there is no > flux, no marker will ever get added. The really astute reader will prove my > ramblings :) > > -- > J. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: