From daniel.goertzen@REDACTED Sat Nov 1 02:57:01 2014 From: daniel.goertzen@REDACTED (Daniel Goertzen) Date: Fri, 31 Oct 2014 20:57:01 -0500 Subject: [erlang-questions] dirty scheduler segfault In-Reply-To: References: Message-ID: Thanks for trying it out. That gist was a bit of a hash; apologies. I made all the functions static and also put load and unload as NULL in ERL_NIF_INIT, but I get the same results. I ran it under valgrind and got... # ERL_LIBS=.. valgrind --trace-children=yes erl ... Eshell V6.2 (abort with ^G) 1> 1> dlibusb:mytest_io(). ==9029== Thread 18: ==9029== Invalid read of size 4 ==9029== at 0x8190B56: process_main (beam_hot.h:935) ==9029== by 0x80E565E: sched_thread_func (erl_process.c:7719) ==9029== by 0x820982B: thr_wrapper (ethread.c:106) ==9029== by 0x40FFF46: start_thread (in /lib/libpthread-2.20.so) ==9029== by 0x41FE97D: clone (in /lib/libc-2.20.so) ==9029== Address 0xfffffffe is not stack'd, malloc'd or (recently) free'd ==9029== ==9029== ==9029== Process terminating with default action of signal 11 (SIGSEGV) ==9029== Access not within mapped region at address 0xFFFFFFFE ==9029== at 0x8190B56: process_main (beam_hot.h:935) ==9029== by 0x80E565E: sched_thread_func (erl_process.c:7719) ==9029== by 0x820982B: thr_wrapper (ethread.c:106) ==9029== by 0x40FFF46: start_thread (in /lib/libpthread-2.20.so) ==9029== by 0x41FE97D: clone (in /lib/libc-2.20.so) ==9029== If you believe this happened as a result of a stack ==9029== overflow in your program's main thread (unlikely but ==9029== possible), you can try to increase the size of the ==9029== main thread stack using the --main-stacksize= flag. ==9029== The main thread stack size used in this run was 8388608. ==9029== ==9029== HEAP SUMMARY: ==9029== in use at exit: 9,020,474 bytes in 157 blocks ==9029== total heap usage: 211 allocs, 54 frees, 9,490,700 bytes allocated ==9029== ==9029== LEAK SUMMARY: ==9029== definitely lost: 0 bytes in 0 blocks ==9029== indirectly lost: 0 bytes in 0 blocks ==9029== possibly lost: 14,143 bytes in 41 blocks ==9029== still reachable: 9,006,331 bytes in 116 blocks ==9029== suppressed: 0 bytes in 0 blocks ==9029== Rerun with --leak-check=full to see details of leaked memory ==9029== ==9029== For counts of detected and suppressed errors, rerun with: -v ==9029== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0) Killed Line 935 of my beam_hot.h is... OpCase(is_integer_fx): { BeamInstr* next; PreFetch(2, next); IsInteger(xb(Arg(1)), ClauseFail()); // line 935 NextPF(2, next); } I know little about beam internals. I don't know if this is useful. On Fri, Oct 31, 2014 at 4:05 PM, Steve Vinoski wrote: > > > On Fri, Oct 31, 2014 at 4:33 PM, Daniel Goertzen < > daniel.goertzen@REDACTED> wrote: > >> I am seeing a segfault that seems to be related to dirty schedulers. >> I've reduced the fault to the erlang and C nif module below which executes >> the same nif with either the io dirty scheduler, the cpu dirty scheduler, >> or the normal erlang scheduler. >> >> >> When I start the emulator and run either dirty nif, I get a segfault. ( >> see https://gist.github.com/goertzenator/6237e0200a5f7bf22976) >> > > I found it hard to make sense of what's in that gist due to the > formatting, so I took your code and built it myself. When I ran it, it > failed in your NIF load function, but it failed in a way that didn't make > sense because all your function does is return 0. Then I realized none of > your C functions were declared static, which means they are global, and I > suspected your load() function was clashing with some other function of the > same name. I made all your C functions static, rebuilt, and then ran > everything and it seems like it worked: > > > c(dlibusb). > Reading symbols for shared libraries . done > {ok,dlibusb} > 2> dlibusb:mytest_cpu(). > [ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,ok] > 3> dlibusb:mytest_io(). > [ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,ok] > 4> dlibusb:mytest_none(). > [ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,ok] > > --steve > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mjtruog@REDACTED Sat Nov 1 03:27:11 2014 From: mjtruog@REDACTED (Michael Truog) Date: Fri, 31 Oct 2014 19:27:11 -0700 Subject: [erlang-questions] dirty scheduler segfault In-Reply-To: References: Message-ID: <545444FF.300@gmail.com> If "erl -smp disable" is being used I assume it is the problem at https://github.com/erlang/otp/pull/518 On 10/31/2014 06:57 PM, Daniel Goertzen wrote: > Thanks for trying it out. That gist was a bit of a hash; apologies. > > I made all the functions static and also put load and unload as NULL in ERL_NIF_INIT, but I get the same results. > > I ran it under valgrind and got... > > > # ERL_LIBS=.. valgrind --trace-children=yes erl > > ... > > Eshell V6.2 (abort with ^G) > > 1> > > 1> dlibusb:mytest_io(). > > ==9029== Thread 18: > > ==9029== Invalid read of size 4 > > ==9029== at 0x8190B56: process_main (beam_hot.h:935) > > ==9029== by 0x80E565E: sched_thread_func (erl_process.c:7719) > > ==9029== by 0x820982B: thr_wrapper (ethread.c:106) > > ==9029== by 0x40FFF46: start_thread (in /lib/libpthread-2.20.so ) > > ==9029== by 0x41FE97D: clone (in /lib/libc-2.20.so ) > > ==9029== Address 0xfffffffe is not stack'd, malloc'd or (recently) free'd > > ==9029== > > ==9029== > > ==9029== Process terminating with default action of signal 11 (SIGSEGV) > > ==9029== Access not within mapped region at address 0xFFFFFFFE > > ==9029== at 0x8190B56: process_main (beam_hot.h:935) > > ==9029== by 0x80E565E: sched_thread_func (erl_process.c:7719) > > ==9029== by 0x820982B: thr_wrapper (ethread.c:106) > > ==9029== by 0x40FFF46: start_thread (in /lib/libpthread-2.20.so ) > > ==9029== by 0x41FE97D: clone (in /lib/libc-2.20.so ) > > ==9029== If you believe this happened as a result of a stack > > ==9029== overflow in your program's main thread (unlikely but > > ==9029== possible), you can try to increase the size of the > > ==9029== main thread stack using the --main-stacksize= flag. > > ==9029== The main thread stack size used in this run was 8388608. > > ==9029== > > ==9029== HEAP SUMMARY: > > ==9029== in use at exit: 9,020,474 bytes in 157 blocks > > ==9029== total heap usage: 211 allocs, 54 frees, 9,490,700 bytes allocated > > ==9029== > > ==9029== LEAK SUMMARY: > > ==9029== definitely lost: 0 bytes in 0 blocks > > ==9029== indirectly lost: 0 bytes in 0 blocks > > ==9029== possibly lost: 14,143 bytes in 41 blocks > > ==9029== still reachable: 9,006,331 bytes in 116 blocks > > ==9029== suppressed: 0 bytes in 0 blocks > > ==9029== Rerun with --leak-check=full to see details of leaked memory > > ==9029== > > ==9029== For counts of detected and suppressed errors, rerun with: -v > > ==9029== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0) > > Killed > > > > Line 935 of my beam_hot.h is... > > OpCase(is_integer_fx): { BeamInstr* next; PreFetch(2, next); IsInteger(xb(Arg(1)), ClauseFail()); // line 935 NextPF(2, next); } > > > I know little about beam internals. I don't know if this is useful. > > > On Fri, Oct 31, 2014 at 4:05 PM, Steve Vinoski > wrote: > > > > On Fri, Oct 31, 2014 at 4:33 PM, Daniel Goertzen > wrote: > > I am seeing a segfault that seems to be related to dirty schedulers. I've reduced the fault to the erlang and C nif module below which executes the same nif with either the io dirty scheduler, the cpu dirty scheduler, or the normal erlang scheduler. > > > When I start the emulator and run either dirty nif, I get a segfault. ( see https://gist.github.com/goertzenator/6237e0200a5f7bf22976) > > > I found it hard to make sense of what's in that gist due to the formatting, so I took your code and built it myself. When I ran it, it failed in your NIF load function, but it failed in a way that didn't make sense because all your function does is return 0. Then I realized none of your C functions were declared static, which means they are global, and I suspected your load() function was clashing with some other function of the same name. I made all your C functions static, rebuilt, and then ran everything and it seems like it worked: > > > c(dlibusb). > Reading symbols for shared libraries . done > {ok,dlibusb} > 2> dlibusb:mytest_cpu(). > [ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,ok] > 3> dlibusb:mytest_io(). > [ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,ok] > 4> dlibusb:mytest_none(). > [ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,ok] > > --steve > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From mjtruog@REDACTED Sat Nov 1 03:31:30 2014 From: mjtruog@REDACTED (Michael Truog) Date: Fri, 31 Oct 2014 19:31:30 -0700 Subject: [erlang-questions] dirty scheduler segfault In-Reply-To: <545444FF.300@gmail.com> References: <545444FF.300@gmail.com> Message-ID: <54544602.1020906@gmail.com> Sorry, based on the Erlang shell it seems to be running with SMP enabled, so that shouldn't be it. On 10/31/2014 07:27 PM, Michael Truog wrote: > If "erl -smp disable" is being used I assume it is the problem at https://github.com/erlang/otp/pull/518 > > On 10/31/2014 06:57 PM, Daniel Goertzen wrote: >> Thanks for trying it out. That gist was a bit of a hash; apologies. >> >> I made all the functions static and also put load and unload as NULL in ERL_NIF_INIT, but I get the same results. >> >> I ran it under valgrind and got... >> >> >> # ERL_LIBS=.. valgrind --trace-children=yes erl >> >> ... >> >> Eshell V6.2 (abort with ^G) >> >> 1> >> >> 1> dlibusb:mytest_io(). >> >> ==9029== Thread 18: >> >> ==9029== Invalid read of size 4 >> >> ==9029== at 0x8190B56: process_main (beam_hot.h:935) >> >> ==9029== by 0x80E565E: sched_thread_func (erl_process.c:7719) >> >> ==9029== by 0x820982B: thr_wrapper (ethread.c:106) >> >> ==9029== by 0x40FFF46: start_thread (in /lib/libpthread-2.20.so ) >> >> ==9029== by 0x41FE97D: clone (in /lib/libc-2.20.so ) >> >> ==9029== Address 0xfffffffe is not stack'd, malloc'd or (recently) free'd >> >> ==9029== >> >> ==9029== >> >> ==9029== Process terminating with default action of signal 11 (SIGSEGV) >> >> ==9029== Access not within mapped region at address 0xFFFFFFFE >> >> ==9029== at 0x8190B56: process_main (beam_hot.h:935) >> >> ==9029== by 0x80E565E: sched_thread_func (erl_process.c:7719) >> >> ==9029== by 0x820982B: thr_wrapper (ethread.c:106) >> >> ==9029== by 0x40FFF46: start_thread (in /lib/libpthread-2.20.so ) >> >> ==9029== by 0x41FE97D: clone (in /lib/libc-2.20.so ) >> >> ==9029== If you believe this happened as a result of a stack >> >> ==9029== overflow in your program's main thread (unlikely but >> >> ==9029== possible), you can try to increase the size of the >> >> ==9029== main thread stack using the --main-stacksize= flag. >> >> ==9029== The main thread stack size used in this run was 8388608. >> >> ==9029== >> >> ==9029== HEAP SUMMARY: >> >> ==9029== in use at exit: 9,020,474 bytes in 157 blocks >> >> ==9029== total heap usage: 211 allocs, 54 frees, 9,490,700 bytes allocated >> >> ==9029== >> >> ==9029== LEAK SUMMARY: >> >> ==9029== definitely lost: 0 bytes in 0 blocks >> >> ==9029== indirectly lost: 0 bytes in 0 blocks >> >> ==9029== possibly lost: 14,143 bytes in 41 blocks >> >> ==9029== still reachable: 9,006,331 bytes in 116 blocks >> >> ==9029== suppressed: 0 bytes in 0 blocks >> >> ==9029== Rerun with --leak-check=full to see details of leaked memory >> >> ==9029== >> >> ==9029== For counts of detected and suppressed errors, rerun with: -v >> >> ==9029== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0) >> >> Killed >> >> >> >> Line 935 of my beam_hot.h is... >> >> OpCase(is_integer_fx): { BeamInstr* next; PreFetch(2, next); IsInteger(xb(Arg(1)), ClauseFail()); // line 935 NextPF(2, next); } >> >> >> I know little about beam internals. I don't know if this is useful. >> >> >> On Fri, Oct 31, 2014 at 4:05 PM, Steve Vinoski > wrote: >> >> >> >> On Fri, Oct 31, 2014 at 4:33 PM, Daniel Goertzen > wrote: >> >> I am seeing a segfault that seems to be related to dirty schedulers. I've reduced the fault to the erlang and C nif module below which executes the same nif with either the io dirty scheduler, the cpu dirty scheduler, or the normal erlang scheduler. >> >> >> When I start the emulator and run either dirty nif, I get a segfault. ( see https://gist.github.com/goertzenator/6237e0200a5f7bf22976) >> >> >> I found it hard to make sense of what's in that gist due to the formatting, so I took your code and built it myself. When I ran it, it failed in your NIF load function, but it failed in a way that didn't make sense because all your function does is return 0. Then I realized none of your C functions were declared static, which means they are global, and I suspected your load() function was clashing with some other function of the same name. I made all your C functions static, rebuilt, and then ran everything and it seems like it worked: >> >> > c(dlibusb). >> Reading symbols for shared libraries . done >> {ok,dlibusb} >> 2> dlibusb:mytest_cpu(). >> [ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,ok] >> 3> dlibusb:mytest_io(). >> [ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,ok] >> 4> dlibusb:mytest_none(). >> [ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,ok] >> >> --steve >> >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gadi@REDACTED Sat Nov 1 11:34:47 2014 From: gadi@REDACTED (Gadi Srebnik) Date: Sat, 1 Nov 2014 12:34:47 +0200 Subject: [erlang-questions] spawn and controlling_process Message-ID: Hi all, my first question here. How safe it is to use controlling_process after spawn? I am using additional gen_tcp:controlling_process inside the Mod:Fun itself after few additional commands. Is it possible that process will not be owner by then? Pid = erlang:spawn(Mod, Fun, Args), gen_tcp:controlling_process(Socket,Pid), Is one-liner safer? gen_tcp:controlling_process(Socket,erlang:spawn(Mod, Fun, Args)), Thanks, Gadi -------------- next part -------------- An HTML attachment was scrubbed... URL: From madabusi.govindarajan@REDACTED Sat Nov 1 12:34:57 2014 From: madabusi.govindarajan@REDACTED (Madabusi Govindarajan) Date: Sat, 1 Nov 2014 17:04:57 +0530 Subject: [erlang-questions] ANN: [1 week left] 'Intro to Erlang' at Bengaluru Message-ID: Hello all, Reminder: One week left to register for 'Intro to Erlang' event at Bengaluru, 15 November. 50% discount for students. http://www.keyfunda.com/events/ Thanks, Madabusi Govindarajan -------------- next part -------------- An HTML attachment was scrubbed... URL: From ingela.andin@REDACTED Sun Nov 2 20:03:28 2014 From: ingela.andin@REDACTED (Ingela Andin) Date: Sun, 2 Nov 2014 20:03:28 +0100 Subject: [erlang-questions] DTLS/SRTP for WebRTC In-Reply-To: References: Message-ID: Hi! 2014-10-30 22:33 GMT+01:00 pablo platt : > Hi, > > What is the state of DTLS in OTP? > Alas Ericsson has prioritised quite a few other things higher than DTLS just lately so the implementation has been delayed. I hope it will get back on track soon but I am afraid you can no count on it in a short term perspective. Regards Ingela Erlang/OTP team - Ericsson AB > > I'm interested in using DTLS/SRTP in Erlang for WebRTC. > > I've found two implementations: > > Clean code but I couldn't make it work because it's missing SRTP support. > https://github.com/RoadRunnr/otp/tree/new_crypto_dtls > > This works quite well. I'm able to connect clients to the server: > https://groups.google.com/d/msg/discuss-webrtc/MP-1sCrOljA/qAs4VK-18y4J > > The problem is that some clients can't connect and I'm getting HANDSHAKE > packets while in the CIPHER state in the following order: > HANDSHAKE, SERVER_HELLO > HANDSHAKE, CERTIFICATE > HANDSHAKE, SERVER_KEY_EXCHANGE > HANDSHAKE, CERTIFICATE_REQUEST > HANDSHAKE, SERVER_HELLO_DONE > > I'm assuming that is related to wrong state change or retransmission. > The code only implement retransmission for two packet types and have the > following comment: > %% problem remainning: different strategy should be adopted while > state==hello or cipher. > > Does anyone use DTLS/SRTP in Erlang and help solve this issue or recommend > another solution? > > Thanks > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vychodil.hynek@REDACTED Mon Nov 3 10:06:28 2014 From: vychodil.hynek@REDACTED (Hynek Vychodil) Date: Mon, 3 Nov 2014 10:06:28 +0100 Subject: [erlang-questions] spawn and controlling_process In-Reply-To: References: Message-ID: No, there is not any difference. Both versions of code is compiled exactly same: fun1(Socket,Mod, Fun, Args) -> Pid = erlang:spawn(Mod, Fun, Args), gen_tcp:controlling_process(Socket,Pid). fun2(Socket,Mod, Fun, Args) -> gen_tcp:controlling_process(Socket,erlang:spawn(Mod, Fun, Args)). Then you can compile it with 'S' and get "assembler": {function, fun1, 4, 2}. {label,1}. {line,[{location,"test2.erl",5}]}. {func_info,{atom,test2},{atom,fun1},4}. {label,2}. {allocate,1,4}. {move,{x,0},{y,0}}. {move,{x,1},{x,0}}. {move,{x,2},{x,1}}. {move,{x,3},{x,2}}. {line,[{location,"test2.erl",6}]}. {call_ext,3,{extfunc,erlang,spawn,3}}. {move,{x,0},{x,1}}. {move,{y,0},{x,0}}. {line,[{location,"test2.erl",7}]}. {call_ext_last,2,{extfunc,gen_tcp,controlling_process,2},1}. {function, fun2, 4, 4}. {label,3}. {line,[{location,"test2.erl",9}]}. {func_info,{atom,test2},{atom,fun2},4}. {label,4}. {allocate,1,4}. {move,{x,0},{y,0}}. {move,{x,1},{x,0}}. {move,{x,2},{x,1}}. {move,{x,3},{x,2}}. {line,[{location,"test2.erl",10}]}. {call_ext,3,{extfunc,erlang,spawn,3}}. {move,{x,0},{x,1}}. {move,{y,0},{x,0}}. {line,[{location,"test2.erl",10}]}. {call_ext_last,2,{extfunc,gen_tcp,controlling_process,2},1}. As you can see, both functions are exactly same. Why should be there any difference. Hynek Vychodil On Sat, Nov 1, 2014 at 11:34 AM, Gadi Srebnik wrote: > Hi all, my first question here. > > How safe it is to use controlling_process after spawn? I am using > additional gen_tcp:controlling_process inside the Mod:Fun itself after few > additional commands. Is it possible that process will not be owner by then? > > > Pid = erlang:spawn(Mod, Fun, Args), > > gen_tcp:controlling_process(Socket,Pid), > > Is one-liner safer? > > gen_tcp:controlling_process(Socket,erlang:spawn(Mod, Fun, Args)), > > > Thanks, > Gadi > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From freza@REDACTED Mon Nov 3 10:30:26 2014 From: freza@REDACTED (Jachym Holecek) Date: Mon, 3 Nov 2014 04:30:26 -0500 Subject: [erlang-questions] spawn and controlling_process In-Reply-To: References: Message-ID: <20141103093026.GA18056@circlewave.net> # Gadi Srebnik 2014-11-01: > How safe it is to use controlling_process after spawn? I am using > additional gen_tcp:controlling_process inside the Mod:Fun itself after few > additional commands. Is it possible that process will not be owner by then? Yes, there's a race condition there, you need explicit synchronization to be safe: start_worker(Sock) -> Pid = erlang:spawn(?MODULE, worker_init, [Sock]), gen_tcp:controlling_process(Sock, Pid), Pid ! proceed. worker_init(Sock) -> receive proceed -> %% Okay now we actually own it. worker_loop(Sock) after 1000 -> exit(things_seem_awfully_slow_today) end. I wish there were a variant of controlling_process that would advise the new owner automatically by sending {tcp_handover, Sock, Info} to it, the last item being arbitrary term passed by previous owner. No idea how hard would it be to implement. BR, -- Jachym From gadi@REDACTED Mon Nov 3 10:34:31 2014 From: gadi@REDACTED (Gadi Srebnik) Date: Mon, 3 Nov 2014 11:34:31 +0200 Subject: [erlang-questions] spawn and controlling_process In-Reply-To: <20141103093026.GA18056@circlewave.net> References: <20141103093026.GA18056@circlewave.net> Message-ID: Thank you, Jachym and Hynek. On Mon, Nov 3, 2014 at 11:30 AM, Jachym Holecek wrote: > # Gadi Srebnik 2014-11-01: > > How safe it is to use controlling_process after spawn? I am using > > additional gen_tcp:controlling_process inside the Mod:Fun itself after > few > > additional commands. Is it possible that process will not be owner by > then? > > Yes, there's a race condition there, you need explicit synchronization to > be safe: > > start_worker(Sock) -> > Pid = erlang:spawn(?MODULE, worker_init, [Sock]), > gen_tcp:controlling_process(Sock, Pid), > Pid ! proceed. > > worker_init(Sock) -> > receive > proceed -> > %% Okay now we actually own it. > worker_loop(Sock) > after 1000 -> > exit(things_seem_awfully_slow_today) > end. > > I wish there were a variant of controlling_process that would advise the > new owner automatically by sending {tcp_handover, Sock, Info} to it, the > last item being arbitrary term passed by previous owner. No idea how hard > would it be to implement. > > BR, > -- Jachym > -- Gadi Srebnik VP Communications m: +972 54 5929261 -------------- next part -------------- An HTML attachment was scrubbed... URL: From cbenac@REDACTED Mon Nov 3 14:03:25 2014 From: cbenac@REDACTED (Clara Benac Earle) Date: Mon, 03 Nov 2014 14:03:25 +0100 Subject: [erlang-questions] [CFP] womENcourage 2015 Message-ID: <54577D1D.7070507@fi.upm.es> Dear all, Please consider submitting a paper/poster/panel/workshop to the 2nd ACM-WE Celebration of Women in Computing womENcourage 2015. Calls: http://womencourage.acm.org/pdf/womencourage15-cfp.pdf http://womencourage.acm.org/pdf/womencourage15-cfpo.pdf http://womencourage.acm.org/pdf/womencourage15-cfpa.pdf http://womencourage.acm.org/pdf/womencourage15-cfwp.pdf More info: http://womencourage.acm.org/index.cfm Best regards Clara Benac Earle Technical University of Madrid, Spain From caisonglu@REDACTED Mon Nov 3 16:18:39 2014 From: caisonglu@REDACTED (songlu cai) Date: Mon, 3 Nov 2014 23:18:39 +0800 Subject: [erlang-questions] Unbalanced Scheduler Problem caused by Using out of date active_no_runq In-Reply-To: References: <28140.1414655602@snookles.snookles.com> Message-ID: Hi Scott, Last week I fix the bug in a simple way, then I run the fixed-version compared with the old unbalanced-version. The two nodes are with same pressure & timeline. The unbalanced(collapsed) state comes up several times on the old version, but never show up on the fixed version. The pressure spreads averagely among 24 schedulers on the fixed version (especially when with high pressure). In fact, the fixed version is with higher pressure when old version runs into the unbalanced state. Because the old version is only with 4 schedulers, and easily gets to the bottleneck, its cpu is 400%, at the same time, fixed version is with cpu 1200%. So I can be sure that the root cause of unbalanced scheduler (scheduler collapse) is "using out of date active_no_runq", just as analyzed before. I just modify the chk_wake_sched function, code diff as below: Index: emulator/beam/erl_process.c =================================================================== --- emulator/beam/erl_process.c (revision 298281) +++ emulator/beam/erl_process.c (working copy) @@ -2694,6 +2694,16 @@ return 0; wrq = ERTS_RUNQ_IX(ix); flags = ERTS_RUNQ_FLGS_GET(wrq); + + if ( activate && + (flags & ERTS_RUNQ_FLG_NONEMPTY) && + (flags & ERTS_RUNQ_FLG_INACTIVE)) { + if (try_inc_no_active_runqs(ix+1)) + (void) ERTS_RUNQ_FLGS_UNSET(wrq, ERTS_RUNQ_FLG_INACTIVE); + wake_scheduler(wrq, 0); + return 1; + } + if (!(flags & (ERTS_RUNQ_FLG_SUSPENDED|ERTS_RUNQ_FLG_NONEMPTY))) { if (activate) { if (try_inc_no_active_runqs(ix+1)) It saves the scheduler from the weird state. It is not a perfect fix, but an effective one. Scott, would you please apply this patch to R16B03 and run your test case again? Thank you very much and waiting for your reply. And I will run it for a week to ensure that we do fix the problem. Best Regards, Songlu Cai 2014-10-31 10:55 GMT+08:00 songlu cai : > Hi Scott, > > > > Thanks for your attention & quick reply. > > It seems that quite a few people suffer from this problem. > > > > Scott>The best workaround is to use "+scl false" and "+sfwi" with a value > of 500 or a bit smaller > > 1, we set +sfwi 500 > > 2,at first, we set +scl false, but it causes unbalanced runq length among > all runqs on R16B03, then we set +scl true (by default), so +scl false is > not a safe choice on R16B03 > > > > Our test cmdline: > > /home/Xxx/erts-5.10.4/bin/beam.smp -zdbbl 8192 -sbt db -sbwt very_short > -swt low -sfwi 500 -MBmmsbc 100 -MHmmsbc 100 -MBmmmbc 100 -MHmmmbc 100 > -MMscs 20480 -MBsmbcs 10240 -MHsbct 2048 -W w -e 50000 -Q 1000000 -hmbs > 46422 -hms 2586 -P 1000000 -A 16 -K true -d -Bi -sct > L23T0C0P0N0:L22T1C1P0N0:L21T2C2P0N0:L20T3C3P0N0:L19T4C4P0N0:L18T5C5P0N0:L17T6C0P1N1:L16T7C1P1N1:L15T8C2P1N1:L14T9C3P1N1:L13T10C4P1N1:L12T11C5P1N1:L11T12C0P0N0:L10T13C1P0N0:L9T14C2P0N0:L8T15C3P0N0:L7T16C4P0N0:L6T17C5P0N0:L5T18C0P1N1:L4T19C1P1N1:L3T20C2P1N1:L2T21C3P1N1:L1T22C4P1N1:L0T23C5P1N1 > -swct medium -- -root /home/Xxx/dir -progname Xxx -- -home /root -- -boot > /home/Xxx/dir -mode interactive -config /home/Xxx/sys.config -shutdown_time > 30000 -heart -setcookie Xxx -name proxy@REDACTED ? console > > > > And , apart from everything else, INACTIVE|NONEMPTY is not a normal state > of runq flag. > > Next few days, I will fix the not-yet-be-sure bug in my way based on > R16B03 and run the test cases again. > > > Best Regards, > > Songlu Cai > > 2014-10-30 15:53 GMT+08:00 Scott Lystig Fritchie : > >> songlu cai wrote: >> >> slc> How to fix: >> >> slc> [...] >> >> slc> 3, Or Another Way? >> >> Wow, that's quite a diagnosis. I'm not a good judge of the race >> condition that you've found or your fix. I can provide some context, >> however, in case that you weren't aware of it. It might help to create >> a Real, Final, 100% Correct Fix ... something which does not exist right >> now. >> >> The best workaround is to use "+scl false" and "+sfwi" with a value of >> 500 or a bit smaller. See the discussion last month about it, >> >> >> http://erlang.org/pipermail/erlang-questions/2014-September/081017.html >> >> My colleague Joe Blomstedt wrote a demo program that can cause scheduler >> collapse to happen pretty quickly. It might be useful for judging how >> well any fix works ... at Basho we had a terrible time trying to >> reproduce this bug before Joe found a semi-reliable trigger. >> >> https://github.com/basho/nifwait >> >> It is discussed in this email thread (which is broken across two URLs, >> sorry). >> >> http://erlang.org/pipermail/erlang-questions/2012-October/069503.html >> (first message only, I don't know why) >> http://erlang.org/pipermail/erlang-questions/2012-October/069585.html >> (the rest of the thread) >> >> If your analysis is correct ... then hopefully this can lead quickly to >> a Real, Final, 100% Correct Fix. I'm tired of diagnosing systems that >> suffer scheduler collapse and then discover that the customer forgot to >> add the magic +sfwi and +scl flags in their runtime configuration to >> work around that !@#$! bug. >> >> -Scott >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From elbrujohalcon@REDACTED Mon Nov 3 17:33:55 2014 From: elbrujohalcon@REDACTED (Brujo Benavides @ Inaka) Date: Mon, 3 Nov 2014 13:33:55 -0300 Subject: [erlang-questions] [ANN] Reminder: ErlangBA Meetup #2 In-Reply-To: <3DEA5FC0-4C5D-4309-9D2D-3AC31FC061BD@inaka.net> References: <3DEA5FC0-4C5D-4309-9D2D-3AC31FC061BD@inaka.net> Message-ID: <7DB0FC46-F96D-44BC-9D36-625B7EB24606@inaka.net> Hey there again! We would like to remind you all about the new ErlangBA Meetup that will take place next Monday, November 10th, 7PM at Inaka 's offices in Villa Crespo. These time our list of speakers will include ?lvaro Videla , Brujo Benavides and Federico Carrone . For more information and to register yourself: http://bit.ly/erlangba2 Remember that we have limited space and there are only a few open seats left ;) Fernando "Brujo" Benavides about.me/elbrujohalcon -------------- next part -------------- An HTML attachment was scrubbed... URL: From sean@REDACTED Mon Nov 3 18:00:01 2014 From: sean@REDACTED (Functional Jobs) Date: Mon, 3 Nov 2014 12:00:01 -0500 Subject: [erlang-questions] New Functional Programming Job Opportunities Message-ID: <5457b4943353a@functionaljobs.com> Here are some functional programming job opportunities that were posted recently: Senior Software Engineer at Soda Software Labs http://functionaljobs.com/jobs/8758-senior-software-engineer-at-soda-software-labs Cheers, Sean Murphy FunctionalJobs.com From daniel.goertzen@REDACTED Mon Nov 3 18:56:29 2014 From: daniel.goertzen@REDACTED (Daniel Goertzen) Date: Mon, 3 Nov 2014 11:56:29 -0600 Subject: [erlang-questions] dirty scheduler segfault Message-ID: This is how my erlang was built: git clone -b OTP-17.3.3 https://github.com/erlang/otp.git otp cd otp && ./otp_build autoconf && ./configure --prefix=/usr/local/ --without-javac --disable-hipe --enable-dynamic-ssl-lib --enable-dirty-schedulers && make && make install OS is a Gentoo chroot, gcc is 4.7.3. The nif build commands (via erlang.mk) are... cc -O3 -std=c99 -finline-functions -Wall -Wmissing-prototypes -I/usr/local/lib/erlang/erts-6.2/include/ -fPIC -c -o /dlibusb/c_src/dlibusb.o /dlibusb/c_src/dlibusb.c cc -shared -o /dlibusb/priv/dlibusb.so /dlibusb/c_src/dlibusb.o My minimal test app is here: https://github.com/goertzenator/dlibusb I use a Gentoo linux chroot to build an OS for a network appliance. The OS is built for a i486, although the variant that will have Erlang has a beefier i586 CPU. I fooled around changing things external to Erlang but could not get the segfault to go away. This morning I built a minimal vanilla Gentoo chroot and was able to repeat the segfault on my Kubuntu laptop and my Kubuntu desktop VM. The instruction to build the chroot with Erlang and to run the segfault test are here: https://gist.github.com/goertzenator/c0b19ee84d16f0e82681 I'm also providing my Gentoo chroot with prebuilt git and Erlang to save a bit of time and hassle, and to provide binaries of my failing system. (791 MB) https://drive.google.com/open?id=0B6luM5L22h1oT2QzMWR0Zk9yOWc&authuser=0 On Fri, Oct 31, 2014 at 10:38 PM, Steve Vinoski wrote: > Given the new calls you're using in your NIF, it looks like you're running > at least 17.3. I've tried this on Mavericks and Ubuntu 14.04, with and > without debug, with and without valgrind, and it always works fine for me. > Valgrind reports no problems. I've tried it with 17.3, and I've tried it > with a brand new build of maint, and they both work fine. > > More comments below. > > On Fri, Oct 31, 2014 at 9:57 PM, Daniel Goertzen < > daniel.goertzen@REDACTED> wrote: > >> Thanks for trying it out. That gist was a bit of a hash; apologies. >> >> I made all the functions static and also put load and unload as NULL in >> ERL_NIF_INIT, but I get the same results. >> >> I ran it under valgrind and got... >> >> >> # ERL_LIBS=.. valgrind --trace-children=yes erl >> >> ... >> >> Eshell V6.2 (abort with ^G) >> >> 1> >> >> 1> dlibusb:mytest_io(). >> >> ==9029== Thread 18: >> >> ==9029== Invalid read of size 4 >> >> ==9029== at 0x8190B56: process_main (beam_hot.h:935) >> >> ==9029== by 0x80E565E: sched_thread_func (erl_process.c:7719) >> >> ==9029== by 0x820982B: thr_wrapper (ethread.c:106) >> >> ==9029== by 0x40FFF46: start_thread (in /lib/libpthread-2.20.so) >> >> ==9029== by 0x41FE97D: clone (in /lib/libc-2.20.so) >> >> ==9029== Address 0xfffffffe is not stack'd, malloc'd or (recently) free'd >> > > There's no dirty scheduler involved in this traceback. The fact that > you're hitting an invalid read like this so early, and that it looks like > something is dereferencing something close to a null pointer, smells to me > like some sort of build problem somewhere. But it's hard to say where > exactly. > > I know little about beam internals. I don't know if this is useful. >> > > Unfortunately not. Perhaps you can provide some more details regarding > exact Erlang version, OS type and version, how your Erlang was built, and > what compiler & version and also the command line you're using to build > your nif? > > --steve > -------------- next part -------------- An HTML attachment was scrubbed... URL: From diptansu@REDACTED Mon Nov 3 19:55:23 2014 From: diptansu@REDACTED (Diptansu Das) Date: Mon, 3 Nov 2014 13:55:23 -0500 Subject: [erlang-questions] General interoperability questions Message-ID: Hi, I am an erlang newbie trying to figure out how best to communicate with an external program (which I need for my project). If I use C nodes according to the documentation I need to call erl_connect(...) How does erl_connect know which port to use to connect to the erlang node? Are there default ports that get used? On a related note, the ports documentation says that because stdin and stdout are buffered not to use those for ports communication -- but then promptly uses those in the examples. Isn't there a simple way to for ports to use (already opened) read and write ports to some existing process for the ports communication? It will be very helpful if somebody can clear up my confusion... Thanks, Diptansu -------------- next part -------------- An HTML attachment was scrubbed... URL: From kostis@REDACTED Mon Nov 3 22:32:09 2014 From: kostis@REDACTED (Kostis Sagonas) Date: Mon, 03 Nov 2014 22:32:09 +0100 Subject: [erlang-questions] code:get_object_code/1 Message-ID: <5457F459.7070008@cs.ntua.gr> A user of PropEr (http://proper.softlab.ntua.gr/) reported that he was not able to use the erlang:timestamp() type as generator. (I.e. be able to generate random time stamp values using the existing mechanisms that PropEr provides.) The culprit was traced to function code:get_object_code/1 which returns 'error' for the 'erlang' module, instead of returning a tuple with its binary data object and beam filename, as one would expect. 1> code:get_object_code(erlang). error While for pretty much all other modules, one gets: 2> code:get_object_code(kernel). {kernel,<<70,79,82,49,0,0,86,252,66,69,65,77,65,116,111, 109,0,0,2,228,0,0,0,65,6,107,101,...>>, "/home/kostis/HiPE/otp/lib/kernel/ebin/kernel.beam"} 3> code:get_object_code(beam_lib). {beam_lib,<<70,79,82,49,0,1,239,188,66,69,65,77,65,116, 111,109,0,0,10,19,0,0,0,225,8,98,101,...>>, "/home/kostis/HiPE/otp/lib/stdlib/ebin/beam_lib.beam"} Is there a reason for this discrepancy? There is a variety of tools that needs to be able to manipulate the object code or extract information from the abstract code of modules, such as the list of their exported types, and there needs to be some (preferably simple) way of getting access to this information. In investigating further the issue, I discovered that the problem can be fixed if one manually adds the erts/preloaded/erl dir in the code path. But in my opinion, having to do this is unnecessarily complicated, possibly error prone and breaks the principle of least astonishment. 4> code:add_path("/home/kostis/HiPE/otp/erts/preloaded/ebin"). true 5> code:get_object_code(erlang). {erlang,<<70,79,82,49,0,1,126,172,66,69,65,77,65,116,111, 109,0,0,15,110,0,0,1,92,6,101,114,...>>, "/home/kostis/HiPE/otp/erts/preloaded/ebin/erlang.beam"} So, I can ask my question differently: Can we please have the preloaded modules in the code path, since they arguably are there? Kostis From jinni.park@REDACTED Tue Nov 4 10:18:40 2014 From: jinni.park@REDACTED (Park, Sungjin) Date: Tue, 4 Nov 2014 18:18:40 +0900 Subject: [erlang-questions] How to support {cpu_time, true} in fprof? Message-ID: Hi, I get not_supported error when I do fprof:trace([start,{cpu_time,true}]). Doc says that it means my platform doesn't have high-resolution cpu clock feature but nothing about how to enable it. I couldn't find any help from google either. -- Park, Sungjin ------------------------------------------------------------------------------------------------------------------- Peculiar travel suggestions are dancing lessons from god. -- The Books of Bokonon ------------------------------------------------------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From taotao.li@REDACTED Tue Nov 4 11:32:17 2014 From: taotao.li@REDACTED (Taotao.Li) Date: Tue, 4 Nov 2014 18:32:17 +0800 (CST) Subject: [erlang-questions] how the lists:seq/3 implemented, confused when reading the source code of lists:seq/3 In-Reply-To: References: Message-ID: <1610674405.5824523.1415097137871.JavaMail.zimbra@datayes.com> ----Source Code-------- -spec seq(From, To, Incr) -> Seq when From :: integer(), To :: integer(), Incr :: integer(), Seq :: [integer()]. seq(First, Last, Inc) when is_integer(First), is_integer(Last), is_integer(Inc) -> if Inc > 0, First - Inc =< Last; Inc < 0, First - Inc >= Last -> N = (Last - First + Inc) div Inc, seq_loop(N, Inc*(N-1)+First, Inc, []); Inc =:= 0, First =:= Last -> seq_loop(1, First, Inc, []) end. seq_loop(N, X, D, L) when N >= 4 -> Y = X-D, Z = Y-D, W = Z-D, seq_loop(N-4, W-D, D, [W,Z,Y,X|L]); seq_loop(N, X, D, L) when N >= 2 -> Y = X-D, seq_loop(N-2, Y-D, D, [Y,X|L]); seq_loop(1, X, _, L) -> [X|L]; seq_loop(0, _, _, L) -> L. ----------------------- -----Problem----------- when I entered lists:seq(-10, 10, 2). what happened? I think it should execute 'Inc > 0, First-Inc =< Last;' and in this way it should just returning nothing. Feeling so confused, can anyone help me about this? thanks a lot! ----------------------- ----- ???? ----- ???: erlang-questions-request@REDACTED ???: erlang-questions@REDACTED ????: ???, 2014? 11 ? 03? ?? 7:00:00 ??: erlang-questions Digest, Vol 190, Issue 1 Send erlang-questions mailing list submissions to erlang-questions@REDACTED To subscribe or unsubscribe via the World Wide Web, visit http://erlang.org/mailman/listinfo/erlang-questions or, via email, send a message with subject or body 'help' to erlang-questions-request@REDACTED You can reach the person managing the list at erlang-questions-owner@REDACTED When replying, please edit your Subject line so it is more specific than "Re: Contents of erlang-questions digest..." Today's Topics: 1. Re: DTLS/SRTP for WebRTC (Ingela Andin) 2. Re: spawn and controlling_process (Hynek Vychodil) 3. Re: spawn and controlling_process (Jachym Holecek) 4. Re: spawn and controlling_process (Gadi Srebnik) ---------------------------------------------------------------------- Message: 1 Date: Sun, 2 Nov 2014 20:03:28 +0100 From: Ingela Andin To: pablo platt Cc: erlang questions Subject: Re: [erlang-questions] DTLS/SRTP for WebRTC Message-ID: Content-Type: text/plain; charset="utf-8" Hi! 2014-10-30 22:33 GMT+01:00 pablo platt : > Hi, > > What is the state of DTLS in OTP? > Alas Ericsson has prioritised quite a few other things higher than DTLS just lately so the implementation has been delayed. I hope it will get back on track soon but I am afraid you can no count on it in a short term perspective. Regards Ingela Erlang/OTP team - Ericsson AB > > I'm interested in using DTLS/SRTP in Erlang for WebRTC. > > I've found two implementations: > > Clean code but I couldn't make it work because it's missing SRTP support. > https://github.com/RoadRunnr/otp/tree/new_crypto_dtls > > This works quite well. I'm able to connect clients to the server: > https://groups.google.com/d/msg/discuss-webrtc/MP-1sCrOljA/qAs4VK-18y4J > > The problem is that some clients can't connect and I'm getting HANDSHAKE > packets while in the CIPHER state in the following order: > HANDSHAKE, SERVER_HELLO > HANDSHAKE, CERTIFICATE > HANDSHAKE, SERVER_KEY_EXCHANGE > HANDSHAKE, CERTIFICATE_REQUEST > HANDSHAKE, SERVER_HELLO_DONE > > I'm assuming that is related to wrong state change or retransmission. > The code only implement retransmission for two packet types and have the > following comment: > %% problem remainning: different strategy should be adopted while > state==hello or cipher. > > Does anyone use DTLS/SRTP in Erlang and help solve this issue or recommend > another solution? > > Thanks > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Message: 2 Date: Mon, 3 Nov 2014 10:06:28 +0100 From: Hynek Vychodil To: Gadi Srebnik Cc: erlang-questions Subject: Re: [erlang-questions] spawn and controlling_process Message-ID: Content-Type: text/plain; charset="utf-8" No, there is not any difference. Both versions of code is compiled exactly same: fun1(Socket,Mod, Fun, Args) -> Pid = erlang:spawn(Mod, Fun, Args), gen_tcp:controlling_process(Socket,Pid). fun2(Socket,Mod, Fun, Args) -> gen_tcp:controlling_process(Socket,erlang:spawn(Mod, Fun, Args)). Then you can compile it with 'S' and get "assembler": {function, fun1, 4, 2}. {label,1}. {line,[{location,"test2.erl",5}]}. {func_info,{atom,test2},{atom,fun1},4}. {label,2}. {allocate,1,4}. {move,{x,0},{y,0}}. {move,{x,1},{x,0}}. {move,{x,2},{x,1}}. {move,{x,3},{x,2}}. {line,[{location,"test2.erl",6}]}. {call_ext,3,{extfunc,erlang,spawn,3}}. {move,{x,0},{x,1}}. {move,{y,0},{x,0}}. {line,[{location,"test2.erl",7}]}. {call_ext_last,2,{extfunc,gen_tcp,controlling_process,2},1}. {function, fun2, 4, 4}. {label,3}. {line,[{location,"test2.erl",9}]}. {func_info,{atom,test2},{atom,fun2},4}. {label,4}. {allocate,1,4}. {move,{x,0},{y,0}}. {move,{x,1},{x,0}}. {move,{x,2},{x,1}}. {move,{x,3},{x,2}}. {line,[{location,"test2.erl",10}]}. {call_ext,3,{extfunc,erlang,spawn,3}}. {move,{x,0},{x,1}}. {move,{y,0},{x,0}}. {line,[{location,"test2.erl",10}]}. {call_ext_last,2,{extfunc,gen_tcp,controlling_process,2},1}. As you can see, both functions are exactly same. Why should be there any difference. Hynek Vychodil On Sat, Nov 1, 2014 at 11:34 AM, Gadi Srebnik wrote: > Hi all, my first question here. > > How safe it is to use controlling_process after spawn? I am using > additional gen_tcp:controlling_process inside the Mod:Fun itself after few > additional commands. Is it possible that process will not be owner by then? > > > Pid = erlang:spawn(Mod, Fun, Args), > > gen_tcp:controlling_process(Socket,Pid), > > Is one-liner safer? > > gen_tcp:controlling_process(Socket,erlang:spawn(Mod, Fun, Args)), > > > Thanks, > Gadi > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Message: 3 Date: Mon, 3 Nov 2014 04:30:26 -0500 From: Jachym Holecek To: Gadi Srebnik Cc: erlang-questions@REDACTED Subject: Re: [erlang-questions] spawn and controlling_process Message-ID: <20141103093026.GA18056@REDACTED> Content-Type: text/plain; charset=us-ascii # Gadi Srebnik 2014-11-01: > How safe it is to use controlling_process after spawn? I am using > additional gen_tcp:controlling_process inside the Mod:Fun itself after few > additional commands. Is it possible that process will not be owner by then? Yes, there's a race condition there, you need explicit synchronization to be safe: start_worker(Sock) -> Pid = erlang:spawn(?MODULE, worker_init, [Sock]), gen_tcp:controlling_process(Sock, Pid), Pid ! proceed. worker_init(Sock) -> receive proceed -> %% Okay now we actually own it. worker_loop(Sock) after 1000 -> exit(things_seem_awfully_slow_today) end. I wish there were a variant of controlling_process that would advise the new owner automatically by sending {tcp_handover, Sock, Info} to it, the last item being arbitrary term passed by previous owner. No idea how hard would it be to implement. BR, -- Jachym ------------------------------ Message: 4 Date: Mon, 3 Nov 2014 11:34:31 +0200 From: Gadi Srebnik To: erlang-questions@REDACTED Subject: Re: [erlang-questions] spawn and controlling_process Message-ID: Content-Type: text/plain; charset="utf-8" Thank you, Jachym and Hynek. On Mon, Nov 3, 2014 at 11:30 AM, Jachym Holecek wrote: > # Gadi Srebnik 2014-11-01: > > How safe it is to use controlling_process after spawn? I am using > > additional gen_tcp:controlling_process inside the Mod:Fun itself after > few > > additional commands. Is it possible that process will not be owner by > then? > > Yes, there's a race condition there, you need explicit synchronization to > be safe: > > start_worker(Sock) -> > Pid = erlang:spawn(?MODULE, worker_init, [Sock]), > gen_tcp:controlling_process(Sock, Pid), > Pid ! proceed. > > worker_init(Sock) -> > receive > proceed -> > %% Okay now we actually own it. > worker_loop(Sock) > after 1000 -> > exit(things_seem_awfully_slow_today) > end. > > I wish there were a variant of controlling_process that would advise the > new owner automatically by sending {tcp_handover, Sock, Info} to it, the > last item being arbitrary term passed by previous owner. No idea how hard > would it be to implement. > > BR, > -- Jachym > -- Gadi Srebnik VP Communications m: +972 54 5929261 -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions End of erlang-questions Digest, Vol 190, Issue 1 ************************************************ -- --------------------------------------------------------------------------- Thanks & Best regards ??? Taotao ? Li | Fixed Income@REDACTED | Software Engineer ??????????????? 99 ?????8 ?? 200120 Address :Wanxiang Towen 8 F, Lujiazui West Rd. No.99, Pudong New District, Shanghai, 200120 ?? |Phone ? 021-60216502 ?? |Mobile: +86-18202171279 From grahamrhay@REDACTED Tue Nov 4 12:01:54 2014 From: grahamrhay@REDACTED (Graham Hay) Date: Tue, 4 Nov 2014 11:01:54 +0000 Subject: [erlang-questions] how the lists:seq/3 implemented, confused when reading the source code of lists:seq/3 In-Reply-To: <1610674405.5824523.1415097137871.JavaMail.zimbra@datayes.com> References: <1610674405.5824523.1415097137871.JavaMail.zimbra@datayes.com> Message-ID: That's a guard sequence, not a block: http://erlang.org/doc/reference_manual/expressions.html#id81869 i.e. if (Inc > 0, First - Inc =< Last) OR (Inc < 0, First - Inc >= Last). On 4 November 2014 10:32, Taotao.Li wrote: > > ----Source Code-------- > > -spec seq(From, To, Incr) -> Seq when > From :: integer(), > To :: integer(), > Incr :: integer(), > Seq :: [integer()]. > > seq(First, Last, Inc) > when is_integer(First), is_integer(Last), is_integer(Inc) -> > if > Inc > 0, First - Inc =< Last; > Inc < 0, First - Inc >= Last -> > N = (Last - First + Inc) div Inc, > seq_loop(N, Inc*(N-1)+First, Inc, []); > Inc =:= 0, First =:= Last -> > seq_loop(1, First, Inc, []) > end. > > seq_loop(N, X, D, L) when N >= 4 -> > Y = X-D, Z = Y-D, W = Z-D, > seq_loop(N-4, W-D, D, [W,Z,Y,X|L]); > seq_loop(N, X, D, L) when N >= 2 -> > Y = X-D, > seq_loop(N-2, Y-D, D, [Y,X|L]); > seq_loop(1, X, _, L) -> > [X|L]; > seq_loop(0, _, _, L) -> > L. > > ----------------------- > > -----Problem----------- > when I entered lists:seq(-10, 10, 2). > what happened? > I think it should execute 'Inc > 0, First-Inc =< Last;' and in this way it > should just returning nothing. > > Feeling so confused, can anyone help me about this? thanks a lot! > > > ----------------------- > > > ----- ???? ----- > ???: erlang-questions-request@REDACTED > ???: erlang-questions@REDACTED > ????: ???, 2014? 11 ? 03? ?? 7:00:00 > ??: erlang-questions Digest, Vol 190, Issue 1 > > Send erlang-questions mailing list submissions to > erlang-questions@REDACTED > > To subscribe or unsubscribe via the World Wide Web, visit > http://erlang.org/mailman/listinfo/erlang-questions > or, via email, send a message with subject or body 'help' to > erlang-questions-request@REDACTED > > You can reach the person managing the list at > erlang-questions-owner@REDACTED > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of erlang-questions digest..." > > > Today's Topics: > > 1. Re: DTLS/SRTP for WebRTC (Ingela Andin) > 2. Re: spawn and controlling_process (Hynek Vychodil) > 3. Re: spawn and controlling_process (Jachym Holecek) > 4. Re: spawn and controlling_process (Gadi Srebnik) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sun, 2 Nov 2014 20:03:28 +0100 > From: Ingela Andin > To: pablo platt > Cc: erlang questions > Subject: Re: [erlang-questions] DTLS/SRTP for WebRTC > Message-ID: > Cx_v0LxL5EJLDDHFA@REDACTED> > Content-Type: text/plain; charset="utf-8" > > Hi! > > 2014-10-30 22:33 GMT+01:00 pablo platt : > > > Hi, > > > > What is the state of DTLS in OTP? > > > > Alas Ericsson has prioritised quite a few other things higher than DTLS > just lately so the implementation has been delayed. > I hope it will get back on track soon but I am afraid you can no count on > it in a short term perspective. > > Regards Ingela Erlang/OTP team - Ericsson AB > > > > > > > I'm interested in using DTLS/SRTP in Erlang for WebRTC. > > > > I've found two implementations: > > > > Clean code but I couldn't make it work because it's missing SRTP support. > > https://github.com/RoadRunnr/otp/tree/new_crypto_dtls > > > > This works quite well. I'm able to connect clients to the server: > > https://groups.google.com/d/msg/discuss-webrtc/MP-1sCrOljA/qAs4VK-18y4J > > > > The problem is that some clients can't connect and I'm getting HANDSHAKE > > packets while in the CIPHER state in the following order: > > HANDSHAKE, SERVER_HELLO > > HANDSHAKE, CERTIFICATE > > HANDSHAKE, SERVER_KEY_EXCHANGE > > HANDSHAKE, CERTIFICATE_REQUEST > > HANDSHAKE, SERVER_HELLO_DONE > > > > I'm assuming that is related to wrong state change or retransmission. > > The code only implement retransmission for two packet types and have the > > following comment: > > %% problem remainning: different strategy should be adopted while > > state==hello or cipher. > > > > Does anyone use DTLS/SRTP in Erlang and help solve this issue or > recommend > > another solution? > > > > Thanks > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://erlang.org/pipermail/erlang-questions/attachments/20141102/0530050f/attachment-0001.html > > > > ------------------------------ > > Message: 2 > Date: Mon, 3 Nov 2014 10:06:28 +0100 > From: Hynek Vychodil > To: Gadi Srebnik > Cc: erlang-questions > Subject: Re: [erlang-questions] spawn and controlling_process > Message-ID: > < > CAL_wnpfgA93QErFDh6UFQPVH+B837CiqEr7tBE0r3tuzvi9YFA@REDACTED> > Content-Type: text/plain; charset="utf-8" > > No, there is not any difference. Both versions of code is compiled exactly > same: > > fun1(Socket,Mod, Fun, Args) -> > Pid = erlang:spawn(Mod, Fun, Args), > gen_tcp:controlling_process(Socket,Pid). > > fun2(Socket,Mod, Fun, Args) -> > gen_tcp:controlling_process(Socket,erlang:spawn(Mod, Fun, Args)). > > Then you can compile it with 'S' and get "assembler": > > {function, fun1, 4, 2}. > {label,1}. > {line,[{location,"test2.erl",5}]}. > {func_info,{atom,test2},{atom,fun1},4}. > {label,2}. > {allocate,1,4}. > {move,{x,0},{y,0}}. > {move,{x,1},{x,0}}. > {move,{x,2},{x,1}}. > {move,{x,3},{x,2}}. > {line,[{location,"test2.erl",6}]}. > {call_ext,3,{extfunc,erlang,spawn,3}}. > {move,{x,0},{x,1}}. > {move,{y,0},{x,0}}. > {line,[{location,"test2.erl",7}]}. > {call_ext_last,2,{extfunc,gen_tcp,controlling_process,2},1}. > > > {function, fun2, 4, 4}. > {label,3}. > {line,[{location,"test2.erl",9}]}. > {func_info,{atom,test2},{atom,fun2},4}. > {label,4}. > {allocate,1,4}. > {move,{x,0},{y,0}}. > {move,{x,1},{x,0}}. > {move,{x,2},{x,1}}. > {move,{x,3},{x,2}}. > {line,[{location,"test2.erl",10}]}. > {call_ext,3,{extfunc,erlang,spawn,3}}. > {move,{x,0},{x,1}}. > {move,{y,0},{x,0}}. > {line,[{location,"test2.erl",10}]}. > {call_ext_last,2,{extfunc,gen_tcp,controlling_process,2},1}. > > As you can see, both functions are exactly same. Why should be there any > difference. > > Hynek Vychodil > > On Sat, Nov 1, 2014 at 11:34 AM, Gadi Srebnik wrote: > > > Hi all, my first question here. > > > > How safe it is to use controlling_process after spawn? I am using > > additional gen_tcp:controlling_process inside the Mod:Fun itself after > few > > additional commands. Is it possible that process will not be owner by > then? > > > > > > Pid = erlang:spawn(Mod, Fun, Args), > > > > gen_tcp:controlling_process(Socket,Pid), > > > > Is one-liner safer? > > > > gen_tcp:controlling_process(Socket,erlang:spawn(Mod, Fun, Args)), > > > > > > Thanks, > > Gadi > > > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://erlang.org/pipermail/erlang-questions/attachments/20141103/260ac0e5/attachment-0001.html > > > > ------------------------------ > > Message: 3 > Date: Mon, 3 Nov 2014 04:30:26 -0500 > From: Jachym Holecek > To: Gadi Srebnik > Cc: erlang-questions@REDACTED > Subject: Re: [erlang-questions] spawn and controlling_process > Message-ID: <20141103093026.GA18056@REDACTED> > Content-Type: text/plain; charset=us-ascii > > # Gadi Srebnik 2014-11-01: > > How safe it is to use controlling_process after spawn? I am using > > additional gen_tcp:controlling_process inside the Mod:Fun itself after > few > > additional commands. Is it possible that process will not be owner by > then? > > Yes, there's a race condition there, you need explicit synchronization to > be safe: > > start_worker(Sock) -> > Pid = erlang:spawn(?MODULE, worker_init, [Sock]), > gen_tcp:controlling_process(Sock, Pid), > Pid ! proceed. > > worker_init(Sock) -> > receive > proceed -> > %% Okay now we actually own it. > worker_loop(Sock) > after 1000 -> > exit(things_seem_awfully_slow_today) > end. > > I wish there were a variant of controlling_process that would advise the > new owner automatically by sending {tcp_handover, Sock, Info} to it, the > last item being arbitrary term passed by previous owner. No idea how hard > would it be to implement. > > BR, > -- Jachym > > > ------------------------------ > > Message: 4 > Date: Mon, 3 Nov 2014 11:34:31 +0200 > From: Gadi Srebnik > To: erlang-questions@REDACTED > Subject: Re: [erlang-questions] spawn and controlling_process > Message-ID: > Yg@REDACTED> > Content-Type: text/plain; charset="utf-8" > > Thank you, Jachym and Hynek. > > On Mon, Nov 3, 2014 at 11:30 AM, Jachym Holecek > wrote: > > > # Gadi Srebnik 2014-11-01: > > > How safe it is to use controlling_process after spawn? I am using > > > additional gen_tcp:controlling_process inside the Mod:Fun itself after > > few > > > additional commands. Is it possible that process will not be owner by > > then? > > > > Yes, there's a race condition there, you need explicit synchronization to > > be safe: > > > > start_worker(Sock) -> > > Pid = erlang:spawn(?MODULE, worker_init, [Sock]), > > gen_tcp:controlling_process(Sock, Pid), > > Pid ! proceed. > > > > worker_init(Sock) -> > > receive > > proceed -> > > %% Okay now we actually own it. > > worker_loop(Sock) > > after 1000 -> > > exit(things_seem_awfully_slow_today) > > end. > > > > I wish there were a variant of controlling_process that would advise the > > new owner automatically by sending {tcp_handover, Sock, Info} to it, the > > last item being arbitrary term passed by previous owner. No idea how hard > > would it be to implement. > > > > BR, > > -- Jachym > > > > > > -- > Gadi Srebnik > VP Communications > m: +972 54 5929261 > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://erlang.org/pipermail/erlang-questions/attachments/20141103/72d1c16d/attachment-0001.html > > > > ------------------------------ > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > End of erlang-questions Digest, Vol 190, Issue 1 > ************************************************ > > -- > > > --------------------------------------------------------------------------- > > Thanks & Best regards > > ??? Taotao ? Li | Fixed Income@REDACTED | Software Engineer > > ??????????????? 99 ?????8 ?? 200120 > Address :Wanxiang Towen 8 F, Lujiazui West Rd. No.99, Pudong New District, > Shanghai, 200120 > > ?? |Phone ? 021-60216502 ?? |Mobile: +86-18202171279 > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From maxim@REDACTED Tue Nov 4 12:10:16 2014 From: maxim@REDACTED (Maxim Sokhatsky) Date: Tue, 4 Nov 2014 13:10:16 +0200 Subject: [erlang-questions] edoc and nested src folder tree structure Message-ID: Hello! I just touched edoc and discovered it is unable to handle ?src? folder with subfolders inside in default rebar configuration. Could you point me how to setup edoc to handle this case? -- Maxim From carlsson.richard@REDACTED Tue Nov 4 12:52:04 2014 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Tue, 4 Nov 2014 12:52:04 +0100 Subject: [erlang-questions] edoc and nested src folder tree structure In-Reply-To: References: Message-ID: Yes, edoc still hasn't been updated since the experimental "packages" was removed from Erlang, so it assumes that subfolders represent subpackages. You can pass the option {packages, false} to prevent this behaviour. -------------- next part -------------- An HTML attachment was scrubbed... URL: From maxim@REDACTED Tue Nov 4 13:10:24 2014 From: maxim@REDACTED (Maxim Sokhatsky) Date: Tue, 4 Nov 2014 14:10:24 +0200 Subject: [erlang-questions] edoc and nested src folder tree structure In-Reply-To: References: Message-ID: Thanks, now it fails at -include("api.hrl"). -export(?API). whre in api.hrl I have? -define(API, [find_executable/0, start_port/2, known_events/0, line_to_event/1]). with: /Users/5HT/depot/.../deps/fs/src/sys/fanotify.erl: at line 3: bad function arity edoc: skipping source file '/Users/5HT/depot/.../deps/fs/src/sys/fanotify.erl': {'EXIT',error}. -- Maxim From askjuise@REDACTED Tue Nov 4 13:21:41 2014 From: askjuise@REDACTED (Alexander Petrovsky) Date: Tue, 4 Nov 2014 16:21:41 +0400 Subject: [erlang-questions] Cowboy back pressure Message-ID: Hi! Until recently, I thought that I can do "connection" back pressure from cowboy like: NbAcceptors = 256, > Backlog = 256, > MaxConnections = 256, > cowboy:start_http(http, NbAcceptors, [{port, ...}, {backlog, Backlog}, > {max_connections, MaxConnections}], [{env, ...}]), But some times ago I'm observed that I can't, the "ranch_server:count_connections/1" says that active ranch connection was gather than 256. While the "ss -lt" show me that ranch socket backlog is 256. And I can't understand why? - Does the "ranch_server:count_connections/1" show not only active connections in ESTABLISHED state, but and the TIME_WAIT state? - How can I properly do connection back pressure with ranch and cowboy? -- ?????????? ????????? / Alexander Petrovsky, Skype: askjuise Phone: +7 914 8 820 815 -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Tue Nov 4 13:26:42 2014 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Tue, 04 Nov 2014 14:26:42 +0200 Subject: [erlang-questions] Cowboy back pressure In-Reply-To: References: Message-ID: <5458C602.4030506@ninenines.eu> Hi, As explained in the documentation for max_connections, this is a soft limit. Messages being sent asynchronously, it can go higher than the limit, even more significantly if the number of acceptors is close to the maximum number of connections. That said, if you need to reduce the number of connections, why do you have so many acceptor processes? The documentation says that 100 is already a large enough number. If you want to accept connections slower, you should reduce it, not increase it. On 11/04/2014 02:21 PM, Alexander Petrovsky wrote: > Hi! > > Until recently, I thought that I can do "connection" back pressure from > cowboy like: > > NbAcceptors = 256, > Backlog = 256, > MaxConnections = 256, > > cowboy:start_http(http, NbAcceptors, [{port, ...}, {backlog, > Backlog}, {max_connections, MaxConnections}], [{env, ...}]), > > > But some times ago I'm observed that I can't, the > "ranch_server:count_connections/1" says that active ranch connection was > gather than 256. > While the "ss -lt" show me that ranch socket backlog is 256. And I can't > understand why? > > - Does the "ranch_server:count_connections/1" show not only active > connections in ESTABLISHED state, but and the TIME_WAIT state? > - How can I properly do connection back pressure with ranch and cowboy? > > -- > ?????????? ????????? / Alexander Petrovsky, > > Skype: askjuise > Phone: +7 914 8 820 815 > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin http://ninenines.eu From carlsson.richard@REDACTED Tue Nov 4 13:35:27 2014 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Tue, 4 Nov 2014 13:35:27 +0100 Subject: [erlang-questions] edoc and nested src folder tree structure In-Reply-To: References: Message-ID: Yeah, it's not possible to handle that sort of code without letting the preprocessor expand it first. If you don't want to rewrite it, you'll need to pass the 'preprocess' option to edoc. In that case you may also need to pass some additional include paths in order for it to find the header files. /Richard On Tue, Nov 4, 2014 at 1:10 PM, Maxim Sokhatsky wrote: > Thanks, now it fails at > > -include("api.hrl"). > -export(?API). > > whre in api.hrl I have > > -define(API, [find_executable/0, start_port/2, known_events/0, > line_to_event/1]). > > with: > > /Users/5HT/depot/.../deps/fs/src/sys/fanotify.erl: at line 3: bad function > arity > edoc: skipping source file > '/Users/5HT/depot/.../deps/fs/src/sys/fanotify.erl': {'EXIT',error}. > > -- > Maxim > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From askjuise@REDACTED Tue Nov 4 13:57:46 2014 From: askjuise@REDACTED (Alexander Petrovsky) Date: Tue, 4 Nov 2014 16:57:46 +0400 Subject: [erlang-questions] Cowboy back pressure In-Reply-To: <5458C602.4030506@ninenines.eu> References: <5458C602.4030506@ninenines.eu> Message-ID: > > As explained in the documentation for max_connections, this is a soft > limit. I can't find any thing about soft limit in http://ninenines.eu/docs/en/cowboy/HEAD/guide/architecture/ Do I understand correctly, that the {backlog, Backlog} option set the socket backlog size. The {max_connections, MaxConnections} option set the soft limit connections count (i.e. accepted connections?), and if the real connections count gather then MaxConnections, cowboy do https://github.com/ninenines/ranch/blob/master/src/ranch_conns_sup.erl#L127, but what happens with after that? Can you explain please? 2014-11-04 15:26 GMT+03:00 Lo?c Hoguin : > Hi, > > As explained in the documentation for max_connections, this is a soft > limit. Messages being sent asynchronously, it can go higher than the limit, > even more significantly if the number of acceptors is close to the maximum > number of connections. > > That said, if you need to reduce the number of connections, why do you > have so many acceptor processes? The documentation says that 100 is already > a large enough number. If you want to accept connections slower, you should > reduce it, not increase it. > > > On 11/04/2014 02:21 PM, Alexander Petrovsky wrote: > >> Hi! >> >> Until recently, I thought that I can do "connection" back pressure from >> cowboy like: >> >> NbAcceptors = 256, >> Backlog = 256, >> MaxConnections = 256, >> >> cowboy:start_http(http, NbAcceptors, [{port, ...}, {backlog, >> Backlog}, {max_connections, MaxConnections}], [{env, ...}]), >> >> >> But some times ago I'm observed that I can't, the >> "ranch_server:count_connections/1" says that active ranch connection was >> gather than 256. >> While the "ss -lt" show me that ranch socket backlog is 256. And I can't >> understand why? >> >> - Does the "ranch_server:count_connections/1" show not only active >> connections in ESTABLISHED state, but and the TIME_WAIT state? >> - How can I properly do connection back pressure with ranch and cowboy? >> >> -- >> ?????????? ????????? / Alexander Petrovsky, >> >> Skype: askjuise >> Phone: +7 914 8 820 815 >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -- > Lo?c Hoguin > http://ninenines.eu > -- ?????????? ????????? / Alexander Petrovsky, Skype: askjuise Phone: +7 914 8 820 815 -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Tue Nov 4 14:03:14 2014 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Tue, 04 Nov 2014 15:03:14 +0200 Subject: [erlang-questions] Cowboy back pressure In-Reply-To: References: <5458C602.4030506@ninenines.eu> Message-ID: <5458CE92.5030503@ninenines.eu> On 11/04/2014 02:57 PM, Alexander Petrovsky wrote: > As explained in the documentation for max_connections, this is a > soft limit. > > > I can't find any thing about soft limit in > http://ninenines.eu/docs/en/cowboy/HEAD/guide/architecture/ http://ninenines.eu/docs/en/ranch/1.0/manual/ranch/#types > Do I understand correctly, that the {backlog, Backlog} option set the > socket backlog size. Yes. > The {max_connections, MaxConnections} option set > the soft limit connections count (i.e. accepted connections?), and if > the real connections count gather then MaxConnections, cowboy do > https://github.com/ninenines/ranch/blob/master/src/ranch_conns_sup.erl#L127, > but what happens with after that? Can you explain please? Acceptors only accept connections when the number is below the limit you configured. So first the number gets above the limit, then Ranch acceptors suspend temporarily (they can only stop after accepting a connection, that's why it can go above), then Ranch resumes acceptors one at a time as soon as a spot opens. > 2014-11-04 15:26 GMT+03:00 Lo?c Hoguin >: > > Hi, > > As explained in the documentation for max_connections, this is a > soft limit. Messages being sent asynchronously, it can go higher > than the limit, even more significantly if the number of acceptors > is close to the maximum number of connections. > > That said, if you need to reduce the number of connections, why do > you have so many acceptor processes? The documentation says that 100 > is already a large enough number. If you want to accept connections > slower, you should reduce it, not increase it. > > > On 11/04/2014 02:21 PM, Alexander Petrovsky wrote: > > Hi! > > Until recently, I thought that I can do "connection" back > pressure from > cowboy like: > > NbAcceptors = 256, > Backlog = 256, > MaxConnections = 256, > > cowboy:start_http(http, NbAcceptors, [{port, ...}, {backlog, > Backlog}, {max_connections, MaxConnections}], [{env, ...}]), > > > But some times ago I'm observed that I can't, the > "ranch_server:count___connections/1" says that active ranch > connection was > gather than 256. > While the "ss -lt" show me that ranch socket backlog is 256. And > I can't > understand why? > > - Does the "ranch_server:count___connections/1" show not only active > connections in ESTABLISHED state, but and the TIME_WAIT state? > - How can I properly do connection back pressure with ranch and > cowboy? > > -- > ?????????? ????????? / Alexander Petrovsky, > > Skype: askjuise > Phone: +7 914 8 820 815 > > > > _________________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/__listinfo/erlang-questions > > > > -- > Lo?c Hoguin > http://ninenines.eu > > > > > -- > ?????????? ????????? / Alexander Petrovsky, > > Skype: askjuise > Phone: +7 914 8 820 815 > -- Lo?c Hoguin http://ninenines.eu From sverker.eriksson@REDACTED Tue Nov 4 15:46:34 2014 From: sverker.eriksson@REDACTED (Sverker Eriksson) Date: Tue, 4 Nov 2014 15:46:34 +0100 Subject: [erlang-questions] dirty scheduler segfault In-Reply-To: References: Message-ID: <5458E6CA.2040101@erix.ericsson.se> On 10/31/2014 10:05 PM, Steve Vinoski wrote: > On Fri, Oct 31, 2014 at 4:33 PM, Daniel Goertzen > wrote: > >> I am seeing a segfault that seems to be related to dirty schedulers. I've >> reduced the fault to the erlang and C nif module below which executes the >> same nif with either the io dirty scheduler, the cpu dirty scheduler, or >> the normal erlang scheduler. >> >> >> When I start the emulator and run either dirty nif, I get a segfault. ( >> see https://gist.github.com/goertzenator/6237e0200a5f7bf22976) >> > I found it hard to make sense of what's in that gist due to the formatting, > so I took your code and built it myself. When I ran it, it failed in your > NIF load function, but it failed in a way that didn't make sense because > all your function does is return 0. Then I realized none of your C > functions were declared static, which means they are global, and I > suspected your load() function was clashing with some other function of the > same name. I made all your C functions static, rebuilt, and then ran > everything and it seems like it worked: > >> c(dlibusb). > Reading symbols for shared libraries . done > {ok,dlibusb} > 2> dlibusb:mytest_cpu(). > [ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,ok] > 3> dlibusb:mytest_io(). > [ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,ok] > 4> dlibusb:mytest_none(). > [ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,ok] > > --steve > > Run on debug VM and increase 'cnt' in the NIF mytest to something bigger (like 1000) and this will segfault every time. The problem arise when a 0-arity dirty NIF like mytest triggers a GC. The return value from the NIF is not included in the rootset of the GC (as it should be) and the calling erlang code crashes when it later tries to read deallocated garbage. I did the following fix in init_nif_sched_data() which seems to work. ep->fp = indirect_fp; proc->freason = TRAP; + proc->arity = argc; return THE_NON_VALUE; } Not sure if that is always the right thing to do. What do you think, Steve? /Sverker, Erlang/OTP From maxim@REDACTED Tue Nov 4 15:52:21 2014 From: maxim@REDACTED (Maxim Sokhatsky) Date: Tue, 4 Nov 2014 16:52:21 +0200 Subject: [erlang-questions] edoc and nested src folder tree structure In-Reply-To: References: Message-ID: Thank you for your answers. The last question is how to specify a limited set of modules to proceed with edoc? Can I just exclude those modules which fails with broken AST edoc parser? -- Maxim From carlsson.richard@REDACTED Tue Nov 4 16:17:48 2014 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Tue, 4 Nov 2014 16:17:48 +0100 Subject: [erlang-questions] edoc and nested src folder tree structure In-Reply-To: References: Message-ID: There's no option for excluding specific modules, but you should be able to achieve what you want by launching edoc via the function edoc:files/2 instead of edoc:application/2, using an explicit list of modules and setting options to get things like application name right. Check out the source code for edoc:application/3 to see what it does to find modules and set up default options. /Richard On Tue, Nov 4, 2014 at 3:52 PM, Maxim Sokhatsky wrote: > Thank you for your answers. The last question is > how to specify a limited set of modules to proceed with edoc? > Can I just exclude those modules which fails with broken AST edoc parser? > > -- > Maxim > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinoski@REDACTED Tue Nov 4 17:01:56 2014 From: vinoski@REDACTED (Steve Vinoski) Date: Tue, 4 Nov 2014 11:01:56 -0500 Subject: [erlang-questions] dirty scheduler segfault In-Reply-To: <5458E6CA.2040101@erix.ericsson.se> References: <5458E6CA.2040101@erix.ericsson.se> Message-ID: On Tue, Nov 4, 2014 at 9:46 AM, Sverker Eriksson < sverker.eriksson@REDACTED> wrote: > > On 10/31/2014 10:05 PM, Steve Vinoski wrote: > >> On Fri, Oct 31, 2014 at 4:33 PM, Daniel Goertzen < >> daniel.goertzen@REDACTED> >> wrote: >> >> I am seeing a segfault that seems to be related to dirty schedulers. >>> I've >>> reduced the fault to the erlang and C nif module below which executes the >>> same nif with either the io dirty scheduler, the cpu dirty scheduler, or >>> the normal erlang scheduler. >>> >>> >>> When I start the emulator and run either dirty nif, I get a segfault. ( >>> see https://gist.github.com/goertzenator/6237e0200a5f7bf22976) >>> >>> I found it hard to make sense of what's in that gist due to the >> formatting, >> so I took your code and built it myself. When I ran it, it failed in your >> NIF load function, but it failed in a way that didn't make sense because >> all your function does is return 0. Then I realized none of your C >> functions were declared static, which means they are global, and I >> suspected your load() function was clashing with some other function of >> the >> same name. I made all your C functions static, rebuilt, and then ran >> everything and it seems like it worked: >> >> c(dlibusb). >>> >> Reading symbols for shared libraries . done >> {ok,dlibusb} >> 2> dlibusb:mytest_cpu(). >> [ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,ok] >> 3> dlibusb:mytest_io(). >> [ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,ok] >> 4> dlibusb:mytest_none(). >> [ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,ok] >> >> --steve >> >> >> > Run on debug VM and increase 'cnt' in the NIF mytest to something bigger > (like 1000) and this will segfault every time. > > The problem arise when a 0-arity dirty NIF like mytest triggers a GC. The > return value from the NIF > is not included in the rootset of the GC (as it should be) and the calling > erlang code crashes when it later tries to > read deallocated garbage. > > I did the following fix in init_nif_sched_data() which seems to work. > > ep->fp = indirect_fp; > proc->freason = TRAP; > + proc->arity = argc; > return THE_NON_VALUE; > } > > > Not sure if that is always the right thing to do. > What do you think, Steve? > Thanks Sverker, glad you were able to reproduce the problem -- I've tried and tried but have never gotten it to fail. Increasing the array size also makes it reliably crash for me. I'll investigate your proposed fix and will probably add a new test for this. --steve -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinoski@REDACTED Tue Nov 4 17:55:35 2014 From: vinoski@REDACTED (Steve Vinoski) Date: Tue, 4 Nov 2014 11:55:35 -0500 Subject: [erlang-questions] dirty scheduler segfault In-Reply-To: References: <5458E6CA.2040101@erix.ericsson.se> Message-ID: On Tue, Nov 4, 2014 at 11:01 AM, Steve Vinoski wrote: > > > On Tue, Nov 4, 2014 at 9:46 AM, Sverker Eriksson < > sverker.eriksson@REDACTED> wrote: > >> >> On 10/31/2014 10:05 PM, Steve Vinoski wrote: >> >>> On Fri, Oct 31, 2014 at 4:33 PM, Daniel Goertzen < >>> daniel.goertzen@REDACTED> >>> wrote: >>> >>> I am seeing a segfault that seems to be related to dirty schedulers. >>>> I've >>>> reduced the fault to the erlang and C nif module below which executes >>>> the >>>> same nif with either the io dirty scheduler, the cpu dirty scheduler, or >>>> the normal erlang scheduler. >>>> >>>> >>>> When I start the emulator and run either dirty nif, I get a segfault. ( >>>> see https://gist.github.com/goertzenator/6237e0200a5f7bf22976) >>>> >>>> I found it hard to make sense of what's in that gist due to the >>> formatting, >>> so I took your code and built it myself. When I ran it, it failed in your >>> NIF load function, but it failed in a way that didn't make sense because >>> all your function does is return 0. Then I realized none of your C >>> functions were declared static, which means they are global, and I >>> suspected your load() function was clashing with some other function of >>> the >>> same name. I made all your C functions static, rebuilt, and then ran >>> everything and it seems like it worked: >>> >>> c(dlibusb). >>>> >>> Reading symbols for shared libraries . done >>> {ok,dlibusb} >>> 2> dlibusb:mytest_cpu(). >>> [ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,ok] >>> 3> dlibusb:mytest_io(). >>> [ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,ok] >>> 4> dlibusb:mytest_none(). >>> [ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,ok] >>> >>> --steve >>> >>> >>> >> Run on debug VM and increase 'cnt' in the NIF mytest to something bigger >> (like 1000) and this will segfault every time. >> >> The problem arise when a 0-arity dirty NIF like mytest triggers a GC. The >> return value from the NIF >> is not included in the rootset of the GC (as it should be) and the >> calling erlang code crashes when it later tries to >> read deallocated garbage. >> >> I did the following fix in init_nif_sched_data() which seems to work. >> >> ep->fp = indirect_fp; >> proc->freason = TRAP; >> + proc->arity = argc; >> return THE_NON_VALUE; >> } >> >> >> Not sure if that is always the right thing to do. >> What do you think, Steve? >> > > Thanks Sverker, glad you were able to reproduce the problem -- I've tried > and tried but have never gotten it to fail. Increasing the array size also > makes it reliably crash for me. I'll investigate your proposed fix and will > probably add a new test for this. > Thanks again Sverker, this is definitely the right fix. I've submitted a PR for this: https://github.com/erlang/otp/pull/531 And Daniel, thanks for finding and reporting this. Sorry I couldn't reproduce it sooner. --steve -------------- next part -------------- An HTML attachment was scrubbed... URL: From pablo.platt@REDACTED Tue Nov 4 18:56:10 2014 From: pablo.platt@REDACTED (pablo platt) Date: Tue, 4 Nov 2014 19:56:10 +0200 Subject: [erlang-questions] DTLS/SRTP for WebRTC In-Reply-To: References: Message-ID: Thank you Ingela for your update. Does anybody have an interest in DTLS for WebRTC and can help improve this library until OTP will have official support? It is almost done. On Sun, Nov 2, 2014 at 9:03 PM, Ingela Andin wrote: > Hi! > > 2014-10-30 22:33 GMT+01:00 pablo platt : > >> Hi, >> >> What is the state of DTLS in OTP? >> > > Alas Ericsson has prioritised quite a few other things higher than DTLS > just lately so the implementation has been delayed. > I hope it will get back on track soon but I am afraid you can no count on > it in a short term perspective. > > Regards Ingela Erlang/OTP team - Ericsson AB > > > >> >> I'm interested in using DTLS/SRTP in Erlang for WebRTC. >> >> I've found two implementations: >> >> Clean code but I couldn't make it work because it's missing SRTP support. >> https://github.com/RoadRunnr/otp/tree/new_crypto_dtls >> >> This works quite well. I'm able to connect clients to the server: >> https://groups.google.com/d/msg/discuss-webrtc/MP-1sCrOljA/qAs4VK-18y4J >> >> The problem is that some clients can't connect and I'm getting HANDSHAKE >> packets while in the CIPHER state in the following order: >> HANDSHAKE, SERVER_HELLO >> HANDSHAKE, CERTIFICATE >> HANDSHAKE, SERVER_KEY_EXCHANGE >> HANDSHAKE, CERTIFICATE_REQUEST >> HANDSHAKE, SERVER_HELLO_DONE >> >> I'm assuming that is related to wrong state change or retransmission. >> The code only implement retransmission for two packet types and have the >> following comment: >> %% problem remainning: different strategy should be adopted while >> state==hello or cipher. >> >> Does anyone use DTLS/SRTP in Erlang and help solve this issue or >> recommend another solution? >> >> Thanks >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rickard@REDACTED Tue Nov 4 23:14:25 2014 From: rickard@REDACTED (Rickard Green) Date: Tue, 4 Nov 2014 23:14:25 +0100 Subject: [erlang-questions] Unbalanced Scheduler Problem caused by Using out of date active_no_runq In-Reply-To: References: <28140.1414655602@snookles.snookles.com> Message-ID: Hi Songlu Cai, Thanks for your work on this! Although it is not an error that NONEMPTY and INACTIVE flag is set at the same time it should occur rather infrequently. Assuming that this would be the case, your fix should have little effect on the behavior, so the root cause should be elsewhere. I've found a potential race when the NONEMPTY flag could end up on the run-queue of a waiting scheduler which could cause the problem you are seeing. When implementing support for balancing on scheduler utilization for OTP 17.0 I rewrote the code that set the NONEMPTY flag. This rewrite removes that potential race. I've back-ported this rewrite based on R16B03-1. It can be found in the rickard/R16B03-1/load_balance/OTP-11385 branch of my github repo . Please try it out and see if it solves your problem (use the same configuration as before). Something similar to your fix should perhaps in the end be introduced anyway (one also has to check the SUSPENDED flag though), since there are no good reason to prevent the activation just because the run-queue happen to be non-empty. It would, however, be good to see that the root issue has been fixed first before introducing this. Regards, Rickard Green, Erlang/OTP, Ericsson AB On Mon, Nov 3, 2014 at 4:18 PM, songlu cai wrote: > Hi Scott, > > Last week I fix the bug in a simple way, then I run the fixed-version > compared with the old unbalanced-version. > The two nodes are with same pressure & timeline. > The unbalanced(collapsed) state comes up several times on the old version, > but never show up on the fixed version. > The pressure spreads averagely among 24 schedulers on the fixed version > (especially when with high pressure). > In fact, the fixed version is with higher pressure when old version runs > into the unbalanced state. > Because the old version is only with 4 schedulers, and easily gets to the > bottleneck, its cpu is 400%, at the same time, fixed version is with cpu > 1200%. > So I can be sure that the root cause of unbalanced scheduler (scheduler > collapse) is "using out of date active_no_runq", just as analyzed before. > > I just modify the chk_wake_sched function, code diff as below: > > Index: emulator/beam/erl_process.c > =================================================================== > --- emulator/beam/erl_process.c (revision 298281) > +++ emulator/beam/erl_process.c (working copy) > @@ -2694,6 +2694,16 @@ > return 0; > wrq = ERTS_RUNQ_IX(ix); > flags = ERTS_RUNQ_FLGS_GET(wrq); > + > + if ( activate && > + (flags & ERTS_RUNQ_FLG_NONEMPTY) && > + (flags & ERTS_RUNQ_FLG_INACTIVE)) { > + if (try_inc_no_active_runqs(ix+1)) > + (void) ERTS_RUNQ_FLGS_UNSET(wrq, ERTS_RUNQ_FLG_INACTIVE); > + wake_scheduler(wrq, 0); > + return 1; > + } > + > if (!(flags & (ERTS_RUNQ_FLG_SUSPENDED|ERTS_RUNQ_FLG_NONEMPTY))) { > if (activate) { > if (try_inc_no_active_runqs(ix+1)) > > It saves the scheduler from the weird state. It is not a perfect fix, but an > effective one. > Scott, would you please apply this patch to R16B03 and run your test case > again? > Thank you very much and waiting for your reply. > And I will run it for a week to ensure that we do fix the problem. > > Best Regards, > Songlu Cai > > 2014-10-31 10:55 GMT+08:00 songlu cai : >> >> Hi Scott, >> >> >> >> Thanks for your attention & quick reply. >> >> It seems that quite a few people suffer from this problem. >> >> >> >> Scott>The best workaround is to use "+scl false" and "+sfwi" with a value >> of 500 or a bit smaller >> >> 1, we set +sfwi 500 >> >> 2,at first, we set +scl false, but it causes unbalanced runq length among >> all runqs on R16B03, then we set +scl true (by default), so +scl false is >> not a safe choice on R16B03 >> >> >> >> Our test cmdline: >> >> /home/Xxx/erts-5.10.4/bin/beam.smp -zdbbl 8192 -sbt db -sbwt very_short >> -swt low -sfwi 500 -MBmmsbc 100 -MHmmsbc 100 -MBmmmbc 100 -MHmmmbc 100 >> -MMscs 20480 -MBsmbcs 10240 -MHsbct 2048 -W w -e 50000 -Q 1000000 -hmbs >> 46422 -hms 2586 -P 1000000 -A 16 -K true -d -Bi -sct >> L23T0C0P0N0:L22T1C1P0N0:L21T2C2P0N0:L20T3C3P0N0:L19T4C4P0N0:L18T5C5P0N0:L17T6C0P1N1:L16T7C1P1N1:L15T8C2P1N1:L14T9C3P1N1:L13T10C4P1N1:L12T11C5P1N1:L11T12C0P0N0:L10T13C1P0N0:L9T14C2P0N0:L8T15C3P0N0:L7T16C4P0N0:L6T17C5P0N0:L5T18C0P1N1:L4T19C1P1N1:L3T20C2P1N1:L2T21C3P1N1:L1T22C4P1N1:L0T23C5P1N1 >> -swct medium -- -root /home/Xxx/dir -progname Xxx -- -home /root -- -boot >> /home/Xxx/dir -mode interactive -config /home/Xxx/sys.config -shutdown_time >> 30000 -heart -setcookie Xxx -name proxy@REDACTED ? console >> >> >> >> And , apart from everything else, INACTIVE|NONEMPTY is not a normal state >> of runq flag. >> >> Next few days, I will fix the not-yet-be-sure bug in my way based on >> R16B03 and run the test cases again. >> >> >> Best Regards, >> >> Songlu Cai >> >> >> 2014-10-30 15:53 GMT+08:00 Scott Lystig Fritchie : >>> >>> songlu cai wrote: >>> >>> slc> How to fix: >>> >>> slc> [...] >>> >>> slc> 3, Or Another Way? >>> >>> Wow, that's quite a diagnosis. I'm not a good judge of the race >>> condition that you've found or your fix. I can provide some context, >>> however, in case that you weren't aware of it. It might help to create >>> a Real, Final, 100% Correct Fix ... something which does not exist right >>> now. >>> >>> The best workaround is to use "+scl false" and "+sfwi" with a value of >>> 500 or a bit smaller. See the discussion last month about it, >>> >>> >>> http://erlang.org/pipermail/erlang-questions/2014-September/081017.html >>> >>> My colleague Joe Blomstedt wrote a demo program that can cause scheduler >>> collapse to happen pretty quickly. It might be useful for judging how >>> well any fix works ... at Basho we had a terrible time trying to >>> reproduce this bug before Joe found a semi-reliable trigger. >>> >>> https://github.com/basho/nifwait >>> >>> It is discussed in this email thread (which is broken across two URLs, >>> sorry). >>> >>> http://erlang.org/pipermail/erlang-questions/2012-October/069503.html >>> (first message only, I don't know why) >>> http://erlang.org/pipermail/erlang-questions/2012-October/069585.html >>> (the rest of the thread) >>> >>> If your analysis is correct ... then hopefully this can lead quickly to >>> a Real, Final, 100% Correct Fix. I'm tired of diagnosing systems that >>> suffer scheduler collapse and then discover that the customer forgot to >>> add the magic +sfwi and +scl flags in their runtime configuration to >>> work around that !@#$! bug. >>> >>> -Scott >> >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Rickard Green, Erlang/OTP, Ericsson AB From curtis@REDACTED Wed Nov 5 01:18:12 2014 From: curtis@REDACTED (Curtis J Schofield (ram9)) Date: Tue, 4 Nov 2014 16:18:12 -0800 Subject: [erlang-questions] What can I only do in Erlang? Message-ID: I am working on getting Erlang/Elixir adopted into our company. I've been asked to provide a simple example of something that your can only do in erlang. My reason for pushing to adopt this in our tech stack is that much of the system code I've written I feel I could write easier in Erlang and do it faster. I feel like one of the unique offerings of erlang relates to distributed computing or OTP reliable systems. Does anyone have thoughts on this question ? "What can I do in Erlang that I can't do in another language?" Ideally it would be something small that I could use as a demonstration. Thank you for any thoughts. Curtis --- Modern Yoga vs Traditional Yoga http://swamij.com/traditional-yoga.htm#swamirama From mark.nijhof@REDACTED Wed Nov 5 01:24:41 2014 From: mark.nijhof@REDACTED (Mark Nijhof) Date: Wed, 5 Nov 2014 01:24:41 +0100 Subject: [erlang-questions] What can I only do in Erlang? In-Reply-To: References: Message-ID: Cheeky, but the only thing you can do in Erlang and not in any other language is write Erlang code (except for Elixir I guess) ;) Besides that I think the whole let it crash philosophy is what makes Erlang stand-out from the others, most if not all language features do exist in other languages. And OTP is of course a huge corner stone that you don't find in many other languages. -Mark On Wed, Nov 5, 2014 at 1:18 AM, Curtis J Schofield (ram9) wrote: > I am working on getting Erlang/Elixir adopted into our company. I've > been asked to provide a simple example of something that your can only > do in erlang. > > My reason for pushing to adopt this in our tech stack is that much of > the system code I've written I feel I could write easier in Erlang and > do it faster. I feel like one of the unique offerings of erlang > relates to distributed computing or OTP reliable systems. > > > Does anyone have thoughts on this question ? "What can I do in Erlang > that I can't do in another language?" > > Ideally it would be something small that I could use as a demonstration. > > Thank you for any thoughts. > > Curtis > > > > > --- > Modern Yoga vs Traditional Yoga > http://swamij.com/traditional-yoga.htm#swamirama > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Mark Nijhof t: @MarkNijhof s: marknijhof -------------- next part -------------- An HTML attachment was scrubbed... URL: From t@REDACTED Wed Nov 5 01:29:59 2014 From: t@REDACTED (Tristan Sloughter) Date: Tue, 04 Nov 2014 18:29:59 -0600 Subject: [erlang-questions] What can I only do in Erlang? In-Reply-To: References: Message-ID: <1415147399.2895374.187151425.22A46CDB@webmail.messagingengine.com> Yes and no. I don't know of any language that can parse binary as easily as Erlang or be so easily investigated while running in production. Though I'm sure Richard O'Keefe will now bring up multiple languages with both features :) -- Tristan Sloughter t@REDACTED On Tue, Nov 4, 2014, at 06:24 PM, Mark Nijhof wrote: > Cheeky, but the only thing you can do in Erlang and not in any other > language is write Erlang code (except for Elixir I guess) ;) > > Besides that I think the whole let it crash philosophy is what makes > Erlang stand-out from the others, most if not all language features do > exist in other languages. And OTP is of course a huge corner stone > that you don't find in many other languages. > > -Mark > > > On Wed, Nov 5, 2014 at 1:18 AM, Curtis J Schofield (ram9) > wrote: >> I am working on getting Erlang/Elixir adopted into our company. I've >> been asked to provide a simple example of something that your can only >> do in erlang. >> >> My reason for pushing to adopt this in our tech stack is that much of >> the system code I've written I feel I could write easier in Erlang and >> do it faster. I feel like one of the unique offerings of erlang >> relates to distributed computing or OTP reliable systems. >> >> >> Does anyone have thoughts on this question ? "What can I do in Erlang >> that I can't do in another language?" >> >> Ideally it would be something small that I could use as a demonstration. >> >> Thank you for any thoughts. >> >> Curtis >> >> >> >> >> --- >> Modern Yoga vs Traditional Yoga >> http://swamij.com/traditional-yoga.htm#swamirama >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > > > -- > Mark Nijhof > t: @MarkNijhof[1] > s: marknijhof > > _________________________________________________ > erlang-questions mailing list erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions Links: 1. https://twitter.com/MarkNijhof -------------- next part -------------- An HTML attachment was scrubbed... URL: From kolorahl@REDACTED Wed Nov 5 01:31:35 2014 From: kolorahl@REDACTED (Tyler Margison) Date: Tue, 4 Nov 2014 16:31:35 -0800 Subject: [erlang-questions] What can I only do in Erlang? In-Reply-To: References: Message-ID: I think the question that should be asked is not "what can I only do in Erlang?" Theoretically, with enough time and development, anything you can do in one language you could do in another. You could even write customer parsers/interpreters/compilers to combine semantics and structures from various languages, or to completely embed one language inside of another. I believe that a more appropriate question would be "what does Erlang do better than other languages?" Echoing Mark, I would say the "let it crash" philosophy is definitely high on the list. There's also the built-in means of communicating between distributed nodes. I personally like Erlang's concept of processes, which I don't see a lot of other languages natively implement, though other languages are getting community add-ons for such behavior (like Akka and Celluloid). On Tue, Nov 4, 2014 at 4:24 PM, Mark Nijhof wrote: > Cheeky, but the only thing you can do in Erlang and not in any other > language is write Erlang code (except for Elixir I guess) ;) > > Besides that I think the whole let it crash philosophy is what makes > Erlang stand-out from the others, most if not all language features do > exist in other languages. And OTP is of course a huge corner stone that you > don't find in many other languages. > > -Mark > > > On Wed, Nov 5, 2014 at 1:18 AM, Curtis J Schofield (ram9) > wrote: > >> I am working on getting Erlang/Elixir adopted into our company. I've >> been asked to provide a simple example of something that your can only >> do in erlang. >> >> My reason for pushing to adopt this in our tech stack is that much of >> the system code I've written I feel I could write easier in Erlang and >> do it faster. I feel like one of the unique offerings of erlang >> relates to distributed computing or OTP reliable systems. >> >> >> Does anyone have thoughts on this question ? "What can I do in Erlang >> that I can't do in another language?" >> >> Ideally it would be something small that I could use as a demonstration. >> >> Thank you for any thoughts. >> >> Curtis >> >> >> >> >> --- >> Modern Yoga vs Traditional Yoga >> http://swamij.com/traditional-yoga.htm#swamirama >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > > > -- > Mark Nijhof > t: @MarkNijhof > s: marknijhof > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From felixgallo@REDACTED Wed Nov 5 01:34:25 2014 From: felixgallo@REDACTED (Felix Gallo) Date: Tue, 4 Nov 2014 16:34:25 -0800 Subject: [erlang-questions] What can I only do in Erlang? In-Reply-To: References: Message-ID: http://joearms.github.io/2013/11/21/My-favorite-erlang-program.html is a fun starting point. Set up three servers in a cluster, put the anything server on each, and then change each one around. Next, start a supervised gen_server on each that crashes when you send it the wrong input. Send it the wrong input. Show the logs. Have each save some state when sent messages. remsh to each node and sys:get_state each of the processes. Now go into the code and add a function head that handles the wrong input, using pattern matching. hot-reload the module and call it to show that the hot reload worked. Man, that sounds like so much fun. Good luck! F. On Tue, Nov 4, 2014 at 4:18 PM, Curtis J Schofield (ram9) wrote: > I am working on getting Erlang/Elixir adopted into our company. I've > been asked to provide a simple example of something that your can only > do in erlang. > > My reason for pushing to adopt this in our tech stack is that much of > the system code I've written I feel I could write easier in Erlang and > do it faster. I feel like one of the unique offerings of erlang > relates to distributed computing or OTP reliable systems. > > > Does anyone have thoughts on this question ? "What can I do in Erlang > that I can't do in another language?" > > Ideally it would be something small that I could use as a demonstration. > > Thank you for any thoughts. > > Curtis > > > > > --- > Modern Yoga vs Traditional Yoga > http://swamij.com/traditional-yoga.htm#swamirama > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfidelman@REDACTED Wed Nov 5 02:11:24 2014 From: mfidelman@REDACTED (Miles Fidelman) Date: Tue, 04 Nov 2014 20:11:24 -0500 Subject: [erlang-questions] What can I only do in Erlang? In-Reply-To: References: Message-ID: <5459793C.6020909@meetinghouse.net> Curtis J Schofield (ram9) wrote: > I am working on getting Erlang/Elixir adopted into our company. I've > been asked to provide a simple example of something that your can only > do in erlang. > > My reason for pushing to adopt this in our tech stack is that much of > the system code I've written I feel I could write easier in Erlang and > do it faster. I feel like one of the unique offerings of erlang > relates to distributed computing or OTP reliable systems. > > > Does anyone have thoughts on this question ? "What can I do in Erlang > that I can't do in another language?" > > Ideally it would be something small that I could use as a demonstration. > > Highly reliable, massive concurrency. Not sure how you'd do a small demonstration though. -- In theory, there is no difference between theory and practice. In practice, there is. .... Yogi Berra From zxq9@REDACTED Wed Nov 5 02:57:28 2014 From: zxq9@REDACTED (zxq9) Date: Wed, 05 Nov 2014 10:57:28 +0900 Subject: [erlang-questions] What can I only do in Erlang? In-Reply-To: References: Message-ID: <1442743.qSh8HrgRRI@burrito> Hi Curtis, Anything can be done in any language, so the technical answer is "nothing". But the spirit of the question begs for an answer, especially in a company situation with deadlines. My company does mostly business data systems and custom extensions to certain products of ours. Originally we used a mix of Python (probably 70%), PL/pgSQL (maybe 10%), Guile/Scheme (probably around 10%), C (5%ish) and a smattering of Bash, Perl, Scala or Java, etc. where necessary. I am extremely familiar with Python and Guile, and between those two a one-dimensional solution to any conceivable problem can be reduced to triviality. They are great languages. They have great libraries. They tie into the rest of the *nix world smoothly. Super tools, *vastly* superior in every way (for us) than more prolific tools such as Java. But when it came time for us to deal with some new feature requests -- like internal chat + whiteboard utilities that tie directly into company dashboards, message systems that can't go down (outside of an earthquake or something), and meshing Android, iOS, Windows and Linux native clients together... Doing everything in the web wasn't cutting it for various reasons and most importantly: The way we wanted to solve our problems required that we write a service management layer. A service management system is *not* something you just do over the weekend. Its a big deal. The APIs to it from various languages can be wonderful or they can infect every subordinate project with cancer. This stuff just isn't easy. Luckily, pretty early on in the design phase I started looking around at alternatives and remembered Erlang and how easy it was to do certain things. I can back to it, toyed with it a bit, and in a few weeks wrote stub examples of the sort of things we wanted to do the way we had always wished were possible. And I didn't have to write any service layer. The Erlang environment is its own service layer. We *could* have been like Google and written our own at great expense, but it turns out that's simply not necessary. We still do a lot of Python (client-side), Scala (Android) and Guile (internally), but almost the entire world of business logic, network services, messaging systems, and just about all other back-ends for anything that involves the network is either in Erlang or will soon be. Its just a lot easier. Why? Binaries: I can't explain how much easier it is to deal with binaries in Erlang than C. This alone is reason enough to swap all your network code. In particular, if you deal with sockets at all, Erlang is probably the right tool. Sockets: In addition to the comments above about handling the data once you've received it, writing "from scratch" socket handling code in Erlang is easier than using any socket libraries I've dealt with. Especially with regard to cleanup when something goes wrong. ASN.1: Cryptographically validated (signed, encrypted, whatever) data structures are usually easiest in ASN.1, and Erlang has *great* facilities here as well. "Let it crash": Its not about being lazy or inviting instability, its about writing code that only deals with the problem, instead of inventing 50 other problems in the form of data validation everywhere. Here your error code and your core problem-solving code are clearly separated from one another, and that means you can identify and factor out similarities in them independent of one another. Once we understood this, our code got better and much smaller. Supervision: Automatic restarts to a known state. Wow. Limiting how bad things can get is amazingly powerful. But you won't realize this until you screw something up really bad, and then realize how much worse it would have been without a supervision tree helping you. Inspection: IBM's JVM has some pretty amazing tools for inspecting itself while running and great crash and debug tools. Awesome stuff. Erlang's are even better. This takes time to learn your way around, though, at least it has for me. At the very simplest end of things, though, go start an Erlang shell and type "observer:start()". If you guys write any server software at all, doing this during a presentation should be convincing enough to get other folks in your company curious (might not be strong enough to sell them on it unless you demonstrate monitoring of remote notes that are doing interesting things live, but it should definitely get them interested). Messaging is not RPC: This is probably not an idea you can easily explain in a quick presentation to management, but the way Erlang enforces this distinction is important. It means you don't have to change the way you think when you're messaging between Erlang processes and when you're networking messaging over sockets to non-Erlang things. Because binary data is easy in Erlang the difference between network traffic and process-process traffic (which may involve the network, but its invisible to you) melts away because you don't get distracted for weeks writing heavy duty network code and then turning back to your comparatively trivial problem. Dialyzer: I haven't seen a better middleground solution to type checking in a dynamically typed language than this. The real utility of it comes once you learn to concentrate your side-effecty code in small, specific areas, and mostly write purely functional code. You can't "verify" your programs, maybe, but you can verify a function at a time and have a very high degree of confidence in most of your code -- and almost never have to revisit large parts of it in maintenance. ...and a long list of other things. Oh yeah, concurrency... its really easy in Erlang, but I assume you've heard. What can you *do* in it? You can avoid writing your own service framework. All the applications frameworks, service layer frameworks, and nearly anything else with the word "framework" in it are attempts to re-write OTP in other languages. We were doing this in Python without realizing it. Handling binaries ("How can we make validation easy?"), handling sockets ("how can we make this easy?"), segregating failures ("how should be isolate these services as processes?"), isolating data ("where do we need to deep copy to make sure we're not screwing ourselves elsewhere?"), restarting components ("we need a list of services and known states so we can restart stuff when it dies..."), multi-processing ("what approach to pooling do we need and how should the processes talk across each other?") and a slew of other service layer type issues are just non-issues in Erlang. These problems are usually harder to solve than the business problem that faces you to begin with. Having that taken care of is great. So its not about what you can "only do in Erlang", its really more about what you don't have to do in Erlang. This is my feeling based on my experience dealing with business software. I don't know what your case looks like, but for us it eliminated the need for a whole category of work, and that was a *big* win. It didn't eliminate all other languages, but it gave us the freedom to focus only on the problems those languages are good for once again. OTOH, Erlang involves a longish learning process. As your development team learns you will notice a distinct change in coding style their first few weeks and a year later. There are several evolutions to understanding the natural way of solving certain problems in Erlang, and OTP is, in my view, its own distinct family of Erlang (and usually the right one). In any case, your early, naive implementations of services in Erlang will be at least as robust (from an external observer) as "proper" implementations in other languages would be, so there is no loss there. As you learn, though, things get better. One parting thought: Erlang's VM can be used by several languages. My personal favorite is Erlang itself, but just like the JVM, Erlang it its own ecosystem and you don't have to marry just one language. Some types of problems may be ugly to read in Erlang but quiet elegantly expressed in Elixir or Lisp Flavored Erlang. I can't stand Java, but Scala is decent and Clojure is wonderful -- its not about the syntax, its about knowing your platform. I didn't answer your question or give you a 10-line demonstration of how to solve the question of Life, the Universe and Everything. Hopefully this gives you some points to think over when discussing the issue with other developers and management. As for encouraging adoption, I recommend you solve a problem in your spare time that your company has, in Erlang, and demonstrate how easy it is to maintain compared to whatever the accepted solution being worked on is. That's a lot safer thing to your management than "throwing money at a new toy, hoping for a payoff". Incidentally, this approach will yield a lot of insights about comparative software and personal revelations (perhaps not just to you) about how your company's leadership thinks and make decisions. If you pick your battle (and sample problem) wisely, it could turn an adversarial situation into an all-around win. (from: "Must convince them! If they say yes I'm happy but panicked to make the pig fly and they are wary of anything smelling sour. If they say no I'm frustrated at how short-sighted The Man is and they are worried I'm looking for an early exit." to: "If I can demonstrate a better solution that saves them on an ongoing development cost I look great, they save money, and we all appreciate a new tool.") Just my $2. -Craig On Tuesday 04 November 2014 16:18:12 Curtis J Schofield wrote: > I am working on getting Erlang/Elixir adopted into our company. I've > been asked to provide a simple example of something that your can only > do in erlang. > > My reason for pushing to adopt this in our tech stack is that much of > the system code I've written I feel I could write easier in Erlang and > do it faster. I feel like one of the unique offerings of erlang > relates to distributed computing or OTP reliable systems. > > > Does anyone have thoughts on this question ? "What can I do in Erlang > that I can't do in another language?" > > Ideally it would be something small that I could use as a demonstration. From lloyd@REDACTED Wed Nov 5 03:49:31 2014 From: lloyd@REDACTED (Lloyd R. Prentice) Date: Tue, 4 Nov 2014 21:49:31 -0500 Subject: [erlang-questions] 1000+ users; 30+ data tables/user Message-ID: <0F0A8AE9-10F5-4B8D-9945-EB7DD7CE86E8@writersglen.com> Hello, This is a naive question reflecting my inexperience with databases. I'm planning to offer my users a set of management/planning tools. Each user would be storing/retrieving user-specific data involving as many as 30 data tables. --- Data fits well into Erlang records. --- We're not talking huge volumes of data per user. --- Nor do I expect much data analysis. --- Data integrity and availability are essential. --- Users may, however, wish to bundle up their data a some point and migrate to a different system. I'm attracted to mnesia because of it's it's tight integration with Erlang and it's replication features. I'm also considering riak. My first thought was that every user would own his/her own database. But this seems to rule out mnesia since: "Mnesia is a truly distributed DBMS and the schema is a system table that is replicated on all nodes in a Mnesia system. The function will fail if a schema is already present on any of the nodes in NodeList." http://www.erlang.org/doc/apps/mnesia/Mnesia_chap3.html An option would be to store data for all users in each of the 30 tables. But is there a better solution altogether? I'd much appreciate suggestions and guidance from wiser heads. Many thanks. LRP Sent from my iPad From ok@REDACTED Wed Nov 5 04:36:00 2014 From: ok@REDACTED (Richard A. O'Keefe) Date: Wed, 5 Nov 2014 16:36:00 +1300 Subject: [erlang-questions] What can I only do in Erlang? In-Reply-To: <1415147399.2895374.187151425.22A46CDB@webmail.messagingengine.com> References: <1415147399.2895374.187151425.22A46CDB@webmail.messagingengine.com> Message-ID: <9CAFC080-1DDC-4197-B741-97B733A19457@cs.otago.ac.nz> On 5/11/2014, at 1:29 pm, Tristan Sloughter wrote: > Yes and no. I don't know of any language that can parse binary as easily as Erlang or be so easily investigated while running in production. > > Though I'm sure Richard O'Keefe will now bring up multiple languages with both features :) Bitstring matching is available for Caml: https://code.google.com/p/bitstring/ It was originally copied from Erlang. From xiande.duan@REDACTED Wed Nov 5 03:26:05 2014 From: xiande.duan@REDACTED (Duan, Xiande) Date: Wed, 5 Nov 2014 02:26:05 +0000 Subject: [erlang-questions] What can I only do in Erlang? In-Reply-To: <1442743.qSh8HrgRRI@burrito> References: <1442743.qSh8HrgRRI@burrito> Message-ID: <5A0DB9EC001ECD4DA29B9F40F5A35F40181B44@SHSMSX103.ccr.corp.intel.com> Pretty great sharing! thanks. -----Original Message----- From: erlang-questions-bounces@REDACTED [mailto:erlang-questions-bounces@REDACTED] On Behalf Of zxq9 Sent: Wednesday, November 5, 2014 9:57 AM To: erlang-questions@REDACTED Subject: Re: [erlang-questions] What can I only do in Erlang? Hi Curtis, Anything can be done in any language, so the technical answer is "nothing". But the spirit of the question begs for an answer, especially in a company situation with deadlines. My company does mostly business data systems and custom extensions to certain products of ours. Originally we used a mix of Python (probably 70%), PL/pgSQL (maybe 10%), Guile/Scheme (probably around 10%), C (5%ish) and a smattering of Bash, Perl, Scala or Java, etc. where necessary. I am extremely familiar with Python and Guile, and between those two a one-dimensional solution to any conceivable problem can be reduced to triviality. They are great languages. They have great libraries. They tie into the rest of the *nix world smoothly. Super tools, *vastly* superior in every way (for us) than more prolific tools such as Java. But when it came time for us to deal with some new feature requests -- like internal chat + whiteboard utilities that tie directly into company dashboards, message systems that can't go down (outside of an earthquake or something), and meshing Android, iOS, Windows and Linux native clients together... Doing everything in the web wasn't cutting it for various reasons and most importantly: The way we wanted to solve our problems required that we write a service management layer. A service management system is *not* something you just do over the weekend. Its a big deal. The APIs to it from various languages can be wonderful or they can infect every subordinate project with cancer. This stuff just isn't easy. Luckily, pretty early on in the design phase I started looking around at alternatives and remembered Erlang and how easy it was to do certain things. I can back to it, toyed with it a bit, and in a few weeks wrote stub examples of the sort of things we wanted to do the way we had always wished were possible. And I didn't have to write any service layer. The Erlang environment is its own service layer. We *could* have been like Google and written our own at great expense, but it turns out that's simply not necessary. We still do a lot of Python (client-side), Scala (Android) and Guile (internally), but almost the entire world of business logic, network services, messaging systems, and just about all other back-ends for anything that involves the network is either in Erlang or will soon be. Its just a lot easier. Why? Binaries: I can't explain how much easier it is to deal with binaries in Erlang than C. This alone is reason enough to swap all your network code. In particular, if you deal with sockets at all, Erlang is probably the right tool. Sockets: In addition to the comments above about handling the data once you've received it, writing "from scratch" socket handling code in Erlang is easier than using any socket libraries I've dealt with. Especially with regard to cleanup when something goes wrong. ASN.1: Cryptographically validated (signed, encrypted, whatever) data structures are usually easiest in ASN.1, and Erlang has *great* facilities here as well. "Let it crash": Its not about being lazy or inviting instability, its about writing code that only deals with the problem, instead of inventing 50 other problems in the form of data validation everywhere. Here your error code and your core problem-solving code are clearly separated from one another, and that means you can identify and factor out similarities in them independent of one another. Once we understood this, our code got better and much smaller. Supervision: Automatic restarts to a known state. Wow. Limiting how bad things can get is amazingly powerful. But you won't realize this until you screw something up really bad, and then realize how much worse it would have been without a supervision tree helping you. Inspection: IBM's JVM has some pretty amazing tools for inspecting itself while running and great crash and debug tools. Awesome stuff. Erlang's are even better. This takes time to learn your way around, though, at least it has for me. At the very simplest end of things, though, go start an Erlang shell and type "observer:start()". If you guys write any server software at all, doing this during a presentation should be convincing enough to get other folks in your company curious (might not be strong enough to sell them on it unless you demonstrate monitoring of remote notes that are doing interesting things live, but it should definitely get them interested). Messaging is not RPC: This is probably not an idea you can easily explain in a quick presentation to management, but the way Erlang enforces this distinction is important. It means you don't have to change the way you think when you're messaging between Erlang processes and when you're networking messaging over sockets to non-Erlang things. Because binary data is easy in Erlang the difference between network traffic and process-process traffic (which may involve the network, but its invisible to you) melts away because you don't get distracted for weeks writing heavy duty network code and then turning back to your comparatively trivial problem. Dialyzer: I haven't seen a better middleground solution to type checking in a dynamically typed language than this. The real utility of it comes once you learn to concentrate your side-effecty code in small, specific areas, and mostly write purely functional code. You can't "verify" your programs, maybe, but you can verify a function at a time and have a very high degree of confidence in most of your code -- and almost never have to revisit large parts of it in maintenance. ...and a long list of other things. Oh yeah, concurrency... its really easy in Erlang, but I assume you've heard. What can you *do* in it? You can avoid writing your own service framework. All the applications frameworks, service layer frameworks, and nearly anything else with the word "framework" in it are attempts to re-write OTP in other languages. We were doing this in Python without realizing it. Handling binaries ("How can we make validation easy?"), handling sockets ("how can we make this easy?"), segregating failures ("how should be isolate these services as processes?"), isolating data ("where do we need to deep copy to make sure we're not screwing ourselves elsewhere?"), restarting components ("we need a list of services and known states so we can restart stuff when it dies..."), multi-processing ("what approach to pooling do we need and how should the processes talk across each other?") and a slew of other service layer type issues are just non-issues in Erlang. These problems are usually harder to solve than the business problem that faces you to begin with. Having that taken care of is great. So its not about what you can "only do in Erlang", its really more about what you don't have to do in Erlang. This is my feeling based on my experience dealing with business software. I don't know what your case looks like, but for us it eliminated the need for a whole category of work, and that was a *big* win. It didn't eliminate all other languages, but it gave us the freedom to focus only on the problems those languages are good for once again. OTOH, Erlang involves a longish learning process. As your development team learns you will notice a distinct change in coding style their first few weeks and a year later. There are several evolutions to understanding the natural way of solving certain problems in Erlang, and OTP is, in my view, its own distinct family of Erlang (and usually the right one). In any case, your early, naive implementations of services in Erlang will be at least as robust (from an external observer) as "proper" implementations in other languages would be, so there is no loss there. As you learn, though, things get better. One parting thought: Erlang's VM can be used by several languages. My personal favorite is Erlang itself, but just like the JVM, Erlang it its own ecosystem and you don't have to marry just one language. Some types of problems may be ugly to read in Erlang but quiet elegantly expressed in Elixir or Lisp Flavored Erlang. I can't stand Java, but Scala is decent and Clojure is wonderful -- its not about the syntax, its about knowing your platform. I didn't answer your question or give you a 10-line demonstration of how to solve the question of Life, the Universe and Everything. Hopefully this gives you some points to think over when discussing the issue with other developers and management. As for encouraging adoption, I recommend you solve a problem in your spare time that your company has, in Erlang, and demonstrate how easy it is to maintain compared to whatever the accepted solution being worked on is. That's a lot safer thing to your management than "throwing money at a new toy, hoping for a payoff". Incidentally, this approach will yield a lot of insights about comparative software and personal revelations (perhaps not just to you) about how your company's leadership thinks and make decisions. If you pick your battle (and sample problem) wisely, it could turn an adversarial situation into an all-around win. (from: "Must convince them! If they say yes I'm happy but panicked to make the pig fly and they are wary of anything smelling sour. If they say no I'm frustrated at how short-sighted The Man is and they are worried I'm looking for an early exit." to: "If I can demonstrate a better solution that saves them on an ongoing development cost I look great, they save money, and we all appreciate a new tool.") Just my $2. -Craig On Tuesday 04 November 2014 16:18:12 Curtis J Schofield wrote: > I am working on getting Erlang/Elixir adopted into our company. I've > been asked to provide a simple example of something that your can only > do in erlang. > > My reason for pushing to adopt this in our tech stack is that much of > the system code I've written I feel I could write easier in Erlang and > do it faster. I feel like one of the unique offerings of erlang > relates to distributed computing or OTP reliable systems. > > > Does anyone have thoughts on this question ? "What can I do in Erlang > that I can't do in another language?" > > Ideally it would be something small that I could use as a demonstration. _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions From essen@REDACTED Wed Nov 5 10:21:04 2014 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Wed, 05 Nov 2014 11:21:04 +0200 Subject: [erlang-questions] What can I only do in Erlang? In-Reply-To: References: Message-ID: <5459EC00.2070807@ninenines.eu> On 11/05/2014 02:18 AM, Curtis J Schofield (ram9) wrote: > Does anyone have thoughts on this question ? "What can I do in Erlang > that I can't do in another language?" There is that one thing that only Erlang can do, that management types will surely react to: 500 million users, 32 engineers. But then again they probably won't believe you. -- Lo?c Hoguin http://ninenines.eu From vincent.dephily@REDACTED Wed Nov 5 11:01:11 2014 From: vincent.dephily@REDACTED (Vincent de Phily) Date: Wed, 05 Nov 2014 11:01:11 +0100 Subject: [erlang-questions] What can I only do in Erlang? In-Reply-To: References: Message-ID: <1627196.17CvjEyTxT@moltowork> On Wednesday 05 November 2014 01:24:41 Mark Nijhof wrote: > Besides that I think the whole let it crash philosophy is what makes Erlang > stand-out from the others, most if not all language features do exist in > other languages. And OTP is of course a huge corner stone that you don't > find in many other languages. I think the Beatles phrased it best: When I find myself in buggy projects Father Armstrong comes to me Speaking words of wisdom, let it crash And in my hour of darkness His book is standing right on my desk Reading words of wisdom, let it crash Let it crash, let it crash Let it crash, let it crash Whisper words of wisdom, Let it crash And when all the burned-out coders Living in the world agree There will be a handler, let it crash For though the input may not be proper There is still a way to recover There will be no downtime, let it crash Let it crash, let it crash Let it crash, let it crash The supervisor will restart it, let it crash :p -- Vincent de Phily From lucafinzicontini@REDACTED Wed Nov 5 10:58:54 2014 From: lucafinzicontini@REDACTED (Luca Finzi Contini) Date: Wed, 5 Nov 2014 10:58:54 +0100 Subject: [erlang-questions] ErlIDE on Mac OS X Mavericks - anyone using it successfully? Message-ID: Hi all, is there anyone using ErlIDE / Eclipse on Mac OS X Mavericks? I have successfully installed it on Windows but I usually work on a Mac. I tried with Eclipse 3.7, 4.2.2, 4.4, with no luck. The IDE goes 'not responding' after a few keystrokes on the source editor. In the rare moments of awakens it has *wonderful* hot code update features which are really good especially for those accustomed to Eclipse in the Java world. I tried to follow the suggestions here: https://github.com/erlide/erlide/wiki/Troubleshooting especially when it says: "For Macs, an alternative solution is to rename your computer from foo.whatever to just foo (or anyhting without any dots)." If I do this, apparently I cannot start an erlang instance with long name as in erl -name foo The currently installed Java version is 1.8.0. Has anyone managed to have ErlIDE up and running smoothly on Mac? Thanks in advance. Luca. -------------- next part -------------- An HTML attachment was scrubbed... URL: From xapwing@REDACTED Wed Nov 5 11:24:48 2014 From: xapwing@REDACTED (Arie van Wingerden) Date: Wed, 5 Nov 2014 11:24:48 +0100 Subject: [erlang-questions] What can I only do in Erlang? In-Reply-To: <1627196.17CvjEyTxT@moltowork> References: <1627196.17CvjEyTxT@moltowork> Message-ID: Aaaaah. That is *really nice* !!! 2014-11-05 11:01 GMT+01:00 Vincent de Phily < vincent.dephily@REDACTED>: > On Wednesday 05 November 2014 01:24:41 Mark Nijhof wrote: > > Besides that I think the whole let it crash philosophy is what makes > Erlang > > stand-out from the others, most if not all language features do exist in > > other languages. And OTP is of course a huge corner stone that you don't > > find in many other languages. > > I think the Beatles phrased it best: > > > > When I find myself in buggy projects > Father Armstrong comes to me > Speaking words of wisdom, let it crash > > And in my hour of darkness > His book is standing right on my desk > Reading words of wisdom, let it crash > > Let it crash, let it crash > Let it crash, let it crash > Whisper words of wisdom, Let it crash > > And when all the burned-out coders > Living in the world agree > There will be a handler, let it crash > > For though the input may not be proper > There is still a way to recover > There will be no downtime, let it crash > > Let it crash, let it crash > Let it crash, let it crash > The supervisor will restart it, let it crash > > > > > :p > > -- > Vincent de Phily > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jmrepetti@REDACTED Wed Nov 5 11:59:57 2014 From: jmrepetti@REDACTED (=?UTF-8?B?SnVhbiBNYXTDrWFz?=) Date: Wed, 5 Nov 2014 11:59:57 +0100 Subject: [erlang-questions] What can I only do in Erlang? In-Reply-To: References: Message-ID: Hey Curtis, you may find some inspiration here: http://thisotplife.tumblr.com/ "simple example of something that your can only do in erlang." I don't know about other languages but message passing between processes in Erlang is really simple. Regards, Matias. On Wed, Nov 5, 2014 at 1:18 AM, Curtis J Schofield (ram9) wrote: > I am working on getting Erlang/Elixir adopted into our company. I've > been asked to provide a simple example of something that your can only > do in erlang. > > My reason for pushing to adopt this in our tech stack is that much of > the system code I've written I feel I could write easier in Erlang and > do it faster. I feel like one of the unique offerings of erlang > relates to distributed computing or OTP reliable systems. > > > Does anyone have thoughts on this question ? "What can I do in Erlang > that I can't do in another language?" > > Ideally it would be something small that I could use as a demonstration. > > Thank you for any thoughts. > > Curtis > > > > > --- > Modern Yoga vs Traditional Yoga > http://swamij.com/traditional-yoga.htm#swamirama > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Mat?as -------------- next part -------------- An HTML attachment was scrubbed... URL: From jmrepetti@REDACTED Wed Nov 5 12:07:05 2014 From: jmrepetti@REDACTED (=?UTF-8?B?SnVhbiBNYXTDrWFz?=) Date: Wed, 5 Nov 2014 12:07:05 +0100 Subject: [erlang-questions] Question regarding documentation. Message-ID: I'm going through the Erlang documentation. Here in http://www.erlang.org/doc/reference_manual/typespec.html I've found: | UserDefined %% *described in Section 6.3* I think the section should be 7.3. Is ok to report this here in the mailing list? Regards, Mat?as -------------- next part -------------- An HTML attachment was scrubbed... URL: From bog495@REDACTED Wed Nov 5 13:08:22 2014 From: bog495@REDACTED (Bogdan Andu) Date: Wed, 5 Nov 2014 14:08:22 +0200 Subject: [erlang-questions] 'ssl_otp_session_cache' ets table Message-ID: Hi, I performed a series of test regarding the an Erlang SSL server. In this setup a major role is played by the table called 'ssl_otp_session_cache', and of course the processes using it. The problem is that the size of table increases constantly and, because an ets table does not automatically deallocate memory unless the object are deleted from that table, the size of table remains the same even if there no ssl connections to server. For example, with a single client running 'ad infinitum' the table increases at a rate of 5 MBytes/hour. In 12 hours there are allocate around 60 MB of memory only for this table. Some info about this: $ erl -sname console@REDACTED -remsh n1@REDACTED -setcookie operator Erlang/OTP 17 [erts-6.2] [source] [64-bit] [smp:2:2] [async-threads:10] [kernel-poll:false] Eshell V6.2 (abort with ^G) (n1@REDACTED)1> ets:i(). id name type size mem owner ---------------------------------------------------------------------------- 12 cookies set 0 291 auth 4111 code set 410 26132 code_server 8208 code_names set 58 7459 code_server 12307 httpc_manager__session_cookie_db bag 0 291 httpc_manager 16404 ssl_otp_cacertificate_db set 0 291 ssl_manager 20501 ssl_otp_ca_file_ref set 0 291 ssl_manager 24598 ssl_otp_pem_cache set 3 360 ssl_manager * 28695 ssl_otp_session_cache ordered_set 138057 8421893 ssl_manager* 32797 dets duplicate_bag 2 308 dets 40990 ign_requests set 0 291 inet_gethost_native 45087 ign_req_index set 0 291 inet_gethost_native 2261635104 shell_records ordered_set 0 81 <0.30638.37> ac_tab ac_tab set 33 2216 application_controller code_map code_map set 100 2791 <0.72.0> config config set 12 892 <0.72.0> dets_owners dets_owners set 1 298 dets dets_registry dets_registry set 1 299 dets file_io_servers file_io_servers set 1 344 file_server_2 global_locks global_locks set 0 291 global_name_server global_names global_names set 0 291 global_name_server global_names_ext global_names_ext set 0 291 global_name_server global_pid_ids global_pid_ids bag 0 291 global_name_server global_pid_names global_pid_names bag 0 291 global_name_server httpc_manager__handler_db httpc_manager__handler_db set 0 291 httpc_manager httpc_manager__session_db httpc_manager__session_db set 0 291 httpc_manager inet_cache inet_cache bag 0 291 inet_db inet_db inet_db set 29 600 inet_db inet_hosts_byaddr inet_hosts_byaddr bag 0 291 inet_db inet_hosts_byname inet_hosts_byname bag 0 291 inet_db inet_hosts_file_byaddr inet_hosts_file_byaddr bag 0 291 inet_db inet_hosts_file_byname inet_hosts_file_byname bag 0 291 inet_db models models set 3 28952 <0.72.0> sys_dist sys_dist set 1 334 net_kernel ok (n1@REDACTED)5> ets:info(ssl_otp_session_cache). undefined (n1@REDACTED)7> (8421893*8)/1024. 65796.0390625 (n1@REDACTED)8> memory(). [{total,92699464}, {processes,8964000}, {processes_used,8963152}, {system,83735464}, {atom,429569}, {atom_used,421768}, {binary,199040}, {code,10411520}, {ets,69163032}] The memory allocated to table 'ssl_otp_session_cache' is roughly 64 MB in 12 hours. On an OpenBSD platform such process gets killed immediately it hits some memory and/or CPU limits. To make this test on OpenBSD I had to put 'infinit' to memory, otherwise the Erlang VM would be killed. How can one control , tweak or configure this table such that it does not accumulate such data at such high rate. I seems the table being created private, and there is no way to ets:delete_all_objects/1 from table manually. I know that this table caches some SSL data related to clients, but the client has the same IP address, and I wonder why is neccesary to store a lot of SSL connection info about the same client when only the ephemeral peer port differs? How the size of this table can be held in reasonable limits and the rate it's size increases ? Please if somebody shed some light on these issues. Thank you, Bogdan -------------- next part -------------- An HTML attachment was scrubbed... URL: From vladdu55@REDACTED Wed Nov 5 13:11:41 2014 From: vladdu55@REDACTED (Vlad Dumitrescu) Date: Wed, 5 Nov 2014 13:11:41 +0100 Subject: [erlang-questions] ErlIDE on Mac OS X Mavericks - anyone using it successfully? In-Reply-To: References: Message-ID: Hi Luca, Yes, I know a couple of people that use erlide on OSX. All the problems that I heard of were related to the computer name issue. Since I don't have a Mac, it's difficult to give you the exact steps to fix the situation. I assume that you have tried all the things suggested on the wiki page? If you rename your computer to "foo", that name can't be used with -name. You have to use the fully qualified value -- and it has to be possible to resolve that name, most easily by adding the entry to .hosts. Have you added the new name (and the FQDN) to .hosts? best regards, Vlad On Wed, Nov 5, 2014 at 10:58 AM, Luca Finzi Contini < lucafinzicontini@REDACTED> wrote: > Hi all, > is there anyone using ErlIDE / Eclipse on Mac OS X Mavericks? > I have successfully installed it on Windows but I usually work on a Mac. > I tried with Eclipse 3.7, 4.2.2, 4.4, with no luck. > The IDE goes 'not responding' after a few keystrokes on the source editor. > In the rare moments of awakens it has *wonderful* hot code update features > which are really good especially for those accustomed to Eclipse in the > Java world. > I tried to follow the suggestions here: > > https://github.com/erlide/erlide/wiki/Troubleshooting > > especially when it says: > "For Macs, an alternative solution is to rename your computer from > foo.whatever to just foo (or anyhting without any dots)." > > If I do this, apparently I cannot start an erlang instance with long name > as in > > erl -name foo > > The currently installed Java version is 1.8.0. > > Has anyone managed to have ErlIDE up and running smoothly on Mac? > Thanks in advance. > Luca. > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tony@REDACTED Wed Nov 5 14:15:03 2014 From: tony@REDACTED (Tony Rogvall) Date: Wed, 5 Nov 2014 14:15:03 +0100 Subject: [erlang-questions] 'ssl_otp_session_cache' ets table In-Reply-To: References: Message-ID: I think the ssl session times is the problem here, and the lack of a maximum size. You can change the session time in the ssl environment: session_lifetime The default is set to 24 hours (in seconds) (if I read it correctly, in ssl_manager.erl) I guess that a session_cache_size could be a nice thing to have, limiting the growth of the session cache. In other words you have to estimate the life time of your clients and try to find a reasonable session_lifetime to match that, without blowing up the system. Maybe the ssl_session_cache_api could be used to implement a strategy with a max size cache. Retire session least recently used, while performing the update? There is a time_stamp in the session that that could be used for this purpose. OTP: Why is a fixed limit not implemented in the standard ssl_session_cache? Could this be a target for DOS attacks? /Tony > On 5 nov 2014, at 13:08, Bogdan Andu wrote: > > > > Hi, > > I performed a series of test regarding the an Erlang SSL server. > > In this setup a major role is played by the table called 'ssl_otp_session_cache', and of course the processes using it. > > The problem is that the size of table increases constantly and, because an ets table does not automatically deallocate memory unless the object are deleted from that table, the size of table remains the same even if there no ssl connections to server. > > For example, with a single client running 'ad infinitum' the table increases at a rate of 5 MBytes/hour. In 12 hours there are allocate around 60 MB of memory only for this table. > > Some info about this: > $ erl -sname console@REDACTED -remsh n1@REDACTED -setcookie operator > Erlang/OTP 17 [erts-6.2] [source] [64-bit] [smp:2:2] [async-threads:10] [kernel-poll:false] > > Eshell V6.2 (abort with ^G) > > > (n1@REDACTED)1> ets:i(). > id name type size mem owner > ---------------------------------------------------------------------------- > 12 cookies set 0 291 auth > 4111 code set 410 26132 code_server > 8208 code_names set 58 7459 code_server > 12307 httpc_manager__session_cookie_db bag 0 291 httpc_manager > 16404 ssl_otp_cacertificate_db set 0 291 ssl_manager > 20501 ssl_otp_ca_file_ref set 0 291 ssl_manager > 24598 ssl_otp_pem_cache set 3 360 ssl_manager > 28695 ssl_otp_session_cache ordered_set 138057 8421893 ssl_manager > 32797 dets duplicate_bag 2 308 dets > 40990 ign_requests set 0 291 inet_gethost_native > 45087 ign_req_index set 0 291 inet_gethost_native > 2261635104 shell_records ordered_set 0 81 <0.30638.37> > ac_tab ac_tab set 33 2216 application_controller > code_map code_map set 100 2791 <0.72.0> > config config set 12 892 <0.72.0> > dets_owners dets_owners set 1 298 dets > dets_registry dets_registry set 1 299 dets > file_io_servers file_io_servers set 1 344 file_server_2 > global_locks global_locks set 0 291 global_name_server > global_names global_names set 0 291 global_name_server > global_names_ext global_names_ext set 0 291 global_name_server > global_pid_ids global_pid_ids bag 0 291 global_name_server > global_pid_names global_pid_names bag 0 291 global_name_server > httpc_manager__handler_db httpc_manager__handler_db set 0 291 httpc_manager > httpc_manager__session_db httpc_manager__session_db set 0 291 httpc_manager > inet_cache inet_cache bag 0 291 inet_db > inet_db inet_db set 29 600 inet_db > inet_hosts_byaddr inet_hosts_byaddr bag 0 291 inet_db > inet_hosts_byname inet_hosts_byname bag 0 291 inet_db > inet_hosts_file_byaddr inet_hosts_file_byaddr bag 0 291 inet_db > inet_hosts_file_byname inet_hosts_file_byname bag 0 291 inet_db > models models set 3 28952 <0.72.0> > sys_dist sys_dist set 1 334 net_kernel > ok > (n1@REDACTED)5> ets:info(ssl_otp_session_cache). > undefined > (n1@REDACTED)7> (8421893*8)/1024. > 65796.0390625 > (n1@REDACTED)8> memory(). > [{total,92699464}, > {processes,8964000}, > {processes_used,8963152}, > {system,83735464}, > {atom,429569}, > {atom_used,421768}, > {binary,199040}, > {code,10411520}, > {ets,69163032}] > > The memory allocated to table 'ssl_otp_session_cache' is roughly 64 MB in 12 hours. > > On an OpenBSD platform such process gets killed immediately it hits some memory and/or CPU limits. > > To make this test on OpenBSD I had to put 'infinit' to memory, otherwise the Erlang VM would be killed. > > How can one control , tweak or configure this table such that it does not accumulate such data at such high rate. > > I seems the table being created private, and there is no way to ets:delete_all_objects/1 from table manually. > > I know that this table caches some SSL data related to clients, but the client has the same IP address, > and I wonder why is neccesary to store a lot of SSL connection info about the same client when only the ephemeral peer port > differs? > > How the size of this table can be held in reasonable limits and the rate it's size increases ? > > Please if somebody shed some light on these issues. > > Thank you, > > Bogdan > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions "Installing applications can lead to corruption over time. Applications gradually write over each other's libraries, partial upgrades occur, user and system errors happen, and minute changes may be unnoticeable and difficult to fix" From tony@REDACTED Wed Nov 5 14:50:22 2014 From: tony@REDACTED (Tony Rogvall) Date: Wed, 5 Nov 2014 14:50:22 +0100 Subject: [erlang-questions] erlang:restart_timer Message-ID: Hi! From time to time I implement protocols and services that need watch dogs and/or retransmit timers. I will normally implement this using a BIF timer: WdtTimer = erlang:start_timer(Time, self(), wdt) This will send a message {timeout,WdtTimer, wdt} after Time milliseconds. When using a watchdog the code will normally try to avoid that the wdt is ever sent. The idea is that if you get the wdt message you are to late and need to crash / restart or something. So after som executing some "watched" code you want to re-arm the watch dog (feed it) This is normally done by: erlang:cancel_timer(WdtTimer), NewWdtTimer = erlang:start_timer(Time, self(), wdt) If you match the {timeout,WdtTimer,wdt} message and ignore {timeout,OldWdtTimer,wdt} messages, you could skip the cancel_timer. But that will generate plenty of waste. Protocols using retransmit timeouts also use a scheme like the above. I propose a new BIF: erlang:restart_timer(Ref, Time) -> false | RemainingTime This will restart a time if it is active and return the time that remained when it was restarted. If restart_timer return false either the timer has already time out or the reference is not a timer. The benefit is a speedup with at least 2.5 times. comparing: erlang:restart_timer(Ref, Time) vs erlang:cancel_timer(Ref) Ref1 = erlang:start_timer(Time, self(), Message) The cost saving is that restart_timer preserve already stored messages. And also internal timer structures are preserved. Resulting in less overhead. The interface is simple and do not require a new reference every restart, this could also save the time and heap updating a server state. I have an implementation that perform well, and could probably be optimized even further. What about it? I'd like to hear from people actually implementing protocols, not only from the "normal" responders. /Tony From bog495@REDACTED Wed Nov 5 14:51:19 2014 From: bog495@REDACTED (Bogdan Andu) Date: Wed, 5 Nov 2014 15:51:19 +0200 Subject: [erlang-questions] 'ssl_otp_session_cache' ets table In-Reply-To: References: Message-ID: with this scenario you can easily DOS yourself. you don t even need DDOS, DOS is enough. In fact I managed to almost DOS myself - in test environment, of course With a few hundred concurrent ssl connections to the server one can easily run out of memory sooner than might think. What about few thousands or millions of concurrent ssl connections? In 24 hours a lot of data can be accumulated from a lot of clients Of course one can distribute the application and load balance it, but still the problem is not solved. The only solution I came up with was that to use a tcp reversed proxy as a a ssl off-loader a nd speak with Erlang plain text. This way the memory remains constant (with little variations, of course) . If plain connections are involved the total Ets memory usage does not exceed a value around 600 - 700 Kilobytes of RAM. And also I am able to confirm the fact that ssl connections opened from Erlang vm (so, as client) does not affect the Ets: memory usage. On Wed, Nov 5, 2014 at 3:15 PM, Tony Rogvall wrote: > I think the ssl session times is the problem here, and the lack of a > maximum size. > > You can change the session time in the ssl environment: session_lifetime > The default is set to 24 hours (in seconds) (if I read it correctly, in > ssl_manager.erl) > > I guess that a session_cache_size could be a nice thing to have, > limiting the growth of the session cache. > > In other words you have to estimate the life time of your clients and > try to find a reasonable session_lifetime to match that, without blowing up > the system. > > Maybe the ssl_session_cache_api could be used to implement a strategy with > a max size > cache. Retire session least recently used, while performing the update? > There is a time_stamp in the session that that could be used for this > purpose. > > OTP: Why is a fixed limit not implemented in the standard > ssl_session_cache? > Could this be a target for DOS attacks? > > /Tony > > > On 5 nov 2014, at 13:08, Bogdan Andu wrote: > > > > > > > > Hi, > > > > I performed a series of test regarding the an Erlang SSL server. > > > > In this setup a major role is played by the table called > 'ssl_otp_session_cache', and of course the processes using it. > > > > The problem is that the size of table increases constantly and, because > an ets table does not automatically deallocate memory unless the object are > deleted from that table, the size of table remains the same even if there > no ssl connections to server. > > > > For example, with a single client running 'ad infinitum' the table > increases at a rate of 5 MBytes/hour. In 12 hours there are allocate around > 60 MB of memory only for this table. > > > > Some info about this: > > $ erl -sname console@REDACTED -remsh n1@REDACTED -setcookie operator > > Erlang/OTP 17 [erts-6.2] [source] [64-bit] [smp:2:2] [async-threads:10] > [kernel-poll:false] > > > > Eshell V6.2 (abort with ^G) > > > > > > (n1@REDACTED)1> ets:i(). > > id name type size mem owner > > > ---------------------------------------------------------------------------- > > 12 cookies set 0 291 auth > > 4111 code set 410 26132 code_server > > 8208 code_names set 58 7459 code_server > > 12307 httpc_manager__session_cookie_db bag 0 291 > httpc_manager > > 16404 ssl_otp_cacertificate_db set 0 291 > ssl_manager > > 20501 ssl_otp_ca_file_ref set 0 291 ssl_manager > > 24598 ssl_otp_pem_cache set 3 360 ssl_manager > > 28695 ssl_otp_session_cache ordered_set 138057 8421893 > ssl_manager > > 32797 dets duplicate_bag 2 308 dets > > 40990 ign_requests set 0 291 > inet_gethost_native > > 45087 ign_req_index set 0 291 > inet_gethost_native > > 2261635104 shell_records ordered_set 0 81 > <0.30638.37> > > ac_tab ac_tab set 33 2216 > application_controller > > code_map code_map set 100 2791 <0.72.0> > > config config set 12 892 <0.72.0> > > dets_owners dets_owners set 1 298 dets > > dets_registry dets_registry set 1 299 dets > > file_io_servers file_io_servers set 1 344 file_server_2 > > global_locks global_locks set 0 291 > global_name_server > > global_names global_names set 0 291 > global_name_server > > global_names_ext global_names_ext set 0 291 > global_name_server > > global_pid_ids global_pid_ids bag 0 291 > global_name_server > > global_pid_names global_pid_names bag 0 291 > global_name_server > > httpc_manager__handler_db httpc_manager__handler_db set 0 291 > httpc_manager > > httpc_manager__session_db httpc_manager__session_db set 0 291 > httpc_manager > > inet_cache inet_cache bag 0 291 inet_db > > inet_db inet_db set 29 600 inet_db > > inet_hosts_byaddr inet_hosts_byaddr bag 0 291 inet_db > > inet_hosts_byname inet_hosts_byname bag 0 291 inet_db > > inet_hosts_file_byaddr inet_hosts_file_byaddr bag 0 291 > inet_db > > inet_hosts_file_byname inet_hosts_file_byname bag 0 291 > inet_db > > models models set 3 28952 <0.72.0> > > sys_dist sys_dist set 1 334 net_kernel > > ok > > (n1@REDACTED)5> ets:info(ssl_otp_session_cache). > > undefined > > (n1@REDACTED)7> (8421893*8)/1024. > > 65796.0390625 > > (n1@REDACTED)8> memory(). > > [{total,92699464}, > > {processes,8964000}, > > {processes_used,8963152}, > > {system,83735464}, > > {atom,429569}, > > {atom_used,421768}, > > {binary,199040}, > > {code,10411520}, > > {ets,69163032}] > > > > The memory allocated to table 'ssl_otp_session_cache' is roughly 64 MB > in 12 hours. > > > > On an OpenBSD platform such process gets killed immediately it hits some > memory and/or CPU limits. > > > > To make this test on OpenBSD I had to put 'infinit' to memory, otherwise > the Erlang VM would be killed. > > > > How can one control , tweak or configure this table such that it does > not accumulate such data at such high rate. > > > > I seems the table being created private, and there is no way to > ets:delete_all_objects/1 from table manually. > > > > I know that this table caches some SSL data related to clients, but the > client has the same IP address, > > and I wonder why is neccesary to store a lot of SSL connection info > about the same client when only the ephemeral peer port > > differs? > > > > How the size of this table can be held in reasonable limits and the rate > it's size increases ? > > > > Please if somebody shed some light on these issues. > > > > Thank you, > > > > Bogdan > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > "Installing applications can lead to corruption over time. Applications > gradually write over each other's libraries, partial upgrades occur, user > and system errors happen, and minute changes may be unnoticeable and > difficult to fix" > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Wed Nov 5 14:59:51 2014 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Wed, 05 Nov 2014 15:59:51 +0200 Subject: [erlang-questions] erlang:restart_timer In-Reply-To: References: Message-ID: <545A2D57.5010501@ninenines.eu> > I propose a new BIF: > > erlang:restart_timer(Ref, Time) -> false | RemainingTime This is definitely something that could see use in a few places inside Cowboy. > This will restart a time if it is active and return the time that remained when it was restarted. > If restart_timer return false either the timer has already time out or the reference is not a timer. In some cases a "start_or_restart_timer" scenario could also be useful. Otherwise I'm stuck having to check the return value and having to call start_timer as before, making it not as useful as it could be. For example if I have a timeout to close the socket if no data is received, and I have both data and the timeout in my mailbox, I don't really want to process the data and then immediately close the connection. There's data, so the connection is still up, so I want to restart the timer even if it already expired. > The benefit is a speedup with at least 2.5 times. comparing: > > erlang:restart_timer(Ref, Time) > vs > erlang:cancel_timer(Ref) > Ref1 = erlang:start_timer(Time, self(), Message) > > The cost saving is that restart_timer preserve already stored messages. And also internal timer structures are > preserved. Resulting in less overhead. > > The interface is simple and do not require a new reference every restart, this could also save the time > and heap updating a server state. That part sounds good. -- Lo?c Hoguin http://ninenines.eu From lucafinzicontini@REDACTED Wed Nov 5 13:52:41 2014 From: lucafinzicontini@REDACTED (Luca Finzi Contini) Date: Wed, 5 Nov 2014 13:52:41 +0100 Subject: [erlang-questions] Fwd: ErlIDE on Mac OS X Mavericks - anyone using it successfully? In-Reply-To: References: Message-ID: ---------- Forwarded message ---------- From: Luca Finzi Contini Date: 2014-11-05 13:52 GMT+01:00 Subject: Re: [erlang-questions] ErlIDE on Mac OS X Mavericks - anyone using it successfully? To: Vlad Dumitrescu Hi Vlad, thank you so much for your availability! I will collect all possible debug information about my config and maybe prepare a short video of what I get when I use it so that you could understand it better. In order to better support you, is there any specific log file either coming from Eclipse, ErlIde, mac itself, that would be useful for you? Thanks in advance, Luca. 2014-11-05 13:11 GMT+01:00 Vlad Dumitrescu : > Hi Luca, > > Yes, I know a couple of people that use erlide on OSX. All the problems > that I heard of were related to the computer name issue. Since I don't have > a Mac, it's difficult to give you the exact steps to fix the situation. I > assume that you have tried all the things suggested on the wiki page? > > If you rename your computer to "foo", that name can't be used with -name. > You have to use the fully qualified value -- and it has to be possible to > resolve that name, most easily by adding the entry to .hosts. > > Have you added the new name (and the FQDN) to .hosts? > > best regards, > Vlad > > > On Wed, Nov 5, 2014 at 10:58 AM, Luca Finzi Contini < > lucafinzicontini@REDACTED> wrote: > >> Hi all, >> is there anyone using ErlIDE / Eclipse on Mac OS X Mavericks? >> I have successfully installed it on Windows but I usually work on a Mac. >> I tried with Eclipse 3.7, 4.2.2, 4.4, with no luck. >> The IDE goes 'not responding' after a few keystrokes on the source >> editor. >> In the rare moments of awakens it has *wonderful* hot code update >> features which are really good especially for those accustomed to Eclipse >> in the Java world. >> I tried to follow the suggestions here: >> >> https://github.com/erlide/erlide/wiki/Troubleshooting >> >> especially when it says: >> "For Macs, an alternative solution is to rename your computer from >> foo.whatever to just foo (or anyhting without any dots)." >> >> If I do this, apparently I cannot start an erlang instance with long name >> as in >> >> erl -name foo >> >> The currently installed Java version is 1.8.0. >> >> Has anyone managed to have ErlIDE up and running smoothly on Mac? >> Thanks in advance. >> Luca. >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gerald.weber@REDACTED Wed Nov 5 15:30:37 2014 From: gerald.weber@REDACTED (Gerald Weber) Date: Wed, 5 Nov 2014 14:30:37 +0000 Subject: [erlang-questions] design question from a beginner Message-ID: Hi Coders, finally i have a first real world project to support my efforts in learning erlang and OTP: Trying to write an application to load-test a Java backend. This java daemon talks json over HTTP (jetty). The backend requires a login (GET request, no HTTP Authentication), some register stuff for subscriptions and finally a loop to receive all of the events (the step waiting for events is blocking) So my vision is: - 1x application -- 1x supervisor --- 1x (?) gen_server ---- n x Y The gen_server should wait for a msg like {start_workers,50} and should then start 50 instances of Y connecting to the java backend, run some httpc:request calls and receive the return values (which is json) I have really no clear idea which OTP principle can be used for Y, maybe my mindset tells me gen_server is a process to "serve data" and not to ask for data. What would be the best bet ? Or is my simple understanding that Y screams for an gen_fsm including the httpc calls not that wrong ? Can the supervisor be told to start x instances of Y on request ? thx and sorry for my maybe too easy questions. gerald -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas@REDACTED Wed Nov 5 17:24:49 2014 From: lukas@REDACTED (Lukas Larsson) Date: Wed, 5 Nov 2014 17:24:49 +0100 Subject: [erlang-questions] How to support {cpu_time, true} in fprof? In-Reply-To: References: Message-ID: Hello, cpu_time measurements have lost their value since the introduction of SMP as the timestamps gotten from it are very hard/impossible to reason about when you have multiple threads consuming CPU. At the moment it is disabled on Linux, and I'm thinking about removing it completely in the SMP emulator. What usecase do you have for it? Lukas On Tue, Nov 4, 2014 at 10:18 AM, Park, Sungjin wrote: > Hi, > > I get not_supported error when I do fprof:trace([start,{cpu_time,true}]). > Doc says that it means my platform doesn't have high-resolution cpu clock > feature but nothing about how to enable it. I couldn't find any help from > google either. > > > -- > Park, Sungjin > > ------------------------------------------------------------------------------------------------------------------- > Peculiar travel suggestions are dancing lessons from god. > -- The Books of Bokonon > > ------------------------------------------------------------------------------------------------------------------- > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mononcqc@REDACTED Wed Nov 5 17:51:10 2014 From: mononcqc@REDACTED (Fred Hebert) Date: Wed, 5 Nov 2014 11:51:10 -0500 Subject: [erlang-questions] What can I only do in Erlang? In-Reply-To: <1442743.qSh8HrgRRI@burrito> References: <1442743.qSh8HrgRRI@burrito> Message-ID: <20141105165106.GF2531@ferdair.local> This is a good post and I'd like to expand on a view points: On 11/05, zxq9 wrote: > Hi Curtis, > > > Binaries: [...] > > Sockets: [...] > Particularly interesting about sockets is also the ease of switching options around. I remember working on a product with a different company, where a rewrite to use kernel polling on their end took them a few days to weeks (I can't remember precisely). For us, using Erlang, it was a question of booting the node with the '+K true' option to the VM. We can then write code the way we want it, whether we poll with 'recv' or whether we want the code in the socket to be delivered as messages so it looks more like push than pull. Yet, even in the 'push' model, you can have control flow via choosing when to switch between the passive and active modes. On top of this, there are metrics being kept on each socket, about the size and packets being transfered. That's kind of alright, but the real power shows when you know that Ports (the type sockets are implemented in) behave like processes, can be linked, and introspected. That means I can write custom one-liners in a production shell during an incident and go see who owns a connection that might be going bad, is doing too much input or output, and so on, without the need to go and write special instrumentation. That's cool. > ASN.1: [...] > > "Let it crash": [...] > > Supervision: Automatic restarts to a known state. Wow. Limiting how bad things > can get is amazingly powerful. But you won't realize this until you screw > something up really bad, and then realize how much worse it would have been > without a supervision tree helping you. > I wrote on this particular point at http://ferd.ca/it-s-about-the-guarantees.html (and in http://erlang-in-anger.com) and it makes a crazy difference when your systems are engineered with these principles in mind, rather than just "he he restart". It's the difference between a solid crash-recovery mode, and looping your call in a 'try ... catch'. Any system that provides supervisors but cannot provide a fallback to a stable known state are missing the point. > Inspection: [...] > Yes! Introspection is currently my *favorite* thing about using Erlang in production environments. I can't say how many weird behaviours or bugs I've identified and fixed through introspection features. I've posted a few war stories before here and there, in talks, and added some of them to Erlang In Anger, but there's no easy way to explain the enthusiasm I get. The tasks of thread pulling that used to be painful and terrible wading through tonnes of garbage are now things I can look forward to in the same domain as puzzle solving. And that stuff *will* happen. As much as we like bug prevention, we're all aware of the law of diminishing returns that more or less gets applied in a way that tells you your product (if not life-critical) will be shipped with hard-to-fix bugs. When they'll become critical (or too frequent), then they'll be worth figuring out and fixing. In Programming Forth (Stephen Pelc, 2011), the author says "Debugging isn't an art, it's a science!" and provides the following (ASCII-fied) diagram: find a problem ---> gather data ---> form hypothesis ----, .--------------------------------------------------------' '-> design experiment ---> prove hypothesis --> fix problem Which then loops back on itself. By far, the easiest bits are 'finding the problem'. The difficult ones are to form the *right* hypothesis that will let you design a proper experiment and prove your fix works. It's especially true of Heisenbugs that ruin your life by disappearing as you observe them. So how do you go about it? *GATHER MORE DATA*. The more data you have, the easiest it becomes to form good hypotheses. Traditionally, there's four big ways to do it in the server world: - Gather system metrics - Add logs and read them carefully - Try to replicate it locally - Get a core dump and debug that Those are all available in Erlang, but they're often impractical: - System metrics are often wide and won't help with fine-grained bugs, but will help provide context - logs can generate lots and lots of expansive and useless data, and logging itself may cause the bug to stop happening. - Replicating it locally without any prior information is more or less blind programming. Take shots in the dark until you figure out you've killed the right monster. - Core dumps are post-facto items. They often show you the bad state, but rarely how to get there. More recently, systemtap/dtrace have come into the picture. These help a lot for some classes of bugs. However, I have not yet felt the need to run either in production. Why? Because Erlang comes with tracing tools that trace *every god damn thing* for you out of the box. Process spawns? It's in. Messages sent and received? It's in. Function calls filtered with specific arguments? it's in! Garbage collections? it's in. Processes that got scheduled for too long? it's in. Sockets that have their buffers full? It's in. It's all out of the box. It's all available anywhere, and it's all usable in production (assuming you are using a library with the right safeguards). I've had a fantastically annoying case in the HTTP Proxy/router we operate where a customer was complaining of bugs in some specific responses formatted our way, and wanted to know if *we* modified it, or if some middleman between us and the customer did it. Of course the problem was that connections were encrypted and whatnot, so the usual tcpdump isn't doable without some serious hassle about keys and decrypting and whatnot. How did we go about it? We traced the processes that were handling their responses, and I found the right socket attached to their account for this long request. I dropped a trace from the shell: recon_trace:calls( {gen_tcp, send, fun([Port,_Data]) when Port == TheirPort -> ok end}, 10 ). And I could get all the data we'd send over their connection only (no other customer or socket was being introspected). It took me 5 minutes to figure out a once-in-a-billion bug with that stuff. Do you have any idea how long and how much development effort this could have taken otherwise? It would certainly take more than minutes. The introspection goes even further with OTP processes where you can inspect the state of running processes, but also *swap* it out with different ones. Ever wondered if making a buffer bigger for an important customer would make their life easier? I did it in 2 minutes over a cluster of 40 nodes, and rolled it back after the experiment, without the need to add special cose for different buffer sizes, configuration values and changes to data models to target oly *that* customer, etc. The experiment was unsuccessful, by the way, and buffer sizes were better off the same. What took me 2 minutes to try live saved our team days of development, scaffolding and deploys. This kind of introspection on that level is a thing I've never seen in other languages, and they make my life so much easier. I don't even want to touch other concurrent languages that don't have these features, because my development time will be stolen into operations time. I could probably tell more about a lot of things, but I'm not feeling like writign a novel today. Regards, Fred. From garry@REDACTED Wed Nov 5 18:13:04 2014 From: garry@REDACTED (Garry Hodgson) Date: Wed, 05 Nov 2014 12:13:04 -0500 Subject: [erlang-questions] What can I only do in Erlang? In-Reply-To: References: Message-ID: <545A5AA0.4010906@research.att.com> on the theme of "much easier in erlang", here's what i'd do: write a little python or shell program that loops forever, sleeping for 10 seconds and writing a message including its pid. next write a gen_server that spawns one of those as a port process, reads those messages and logs them. include startup and "it died" log entries. now write an erlang/otp app that uses supervision tree to spawn a few of those. fire it up, tail the logs, show it all works, with erlang managing apps in other languages. now kill one of the external process and watch it magically restart, with shiny new pid as proof. for extra credit, run this on two nodes, using distributed application feature. change log message to include hostname as well as pid. show it running, then kill erlang on one of the nodes. show it still runnning. start it up again, then kill the other. repeat as needed to get the point across. i'm guessing this would all be a few dozen lines of code. you could do it in other languages, but i really doubt it would be as easy, or as robust. On 11/04/2014 07:18 PM, Curtis J Schofield (ram9) wrote: > I am working on getting Erlang/Elixir adopted into our company. I've > been asked to provide a simple example of something that your can only > do in erlang. > > My reason for pushing to adopt this in our tech stack is that much of > the system code I've written I feel I could write easier in Erlang and > do it faster. I feel like one of the unique offerings of erlang > relates to distributed computing or OTP reliable systems. > > > Does anyone have thoughts on this question ? "What can I do in Erlang > that I can't do in another language?" > > Ideally it would be something small that I could use as a demonstration. > > Thank you for any thoughts. > > Curtis > > > > > --- > Modern Yoga vs Traditional Yoga > http://swamij.com/traditional-yoga.htm#swamirama > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- Garry Hodgson AT&T Chief Security Office (CSO) "This e-mail and any files transmitted with it are AT&T property, are confidential, and are intended solely for the use of the individual or entity to whom this e-mail is addressed. If you are not one of the named recipient(s) or otherwise have reason to believe that you have received this message in error, please notify the sender and delete this message immediately from your computer. Any other use, retention, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited." From garry@REDACTED Wed Nov 5 18:25:07 2014 From: garry@REDACTED (Garry Hodgson) Date: Wed, 05 Nov 2014 12:25:07 -0500 Subject: [erlang-questions] What can I only do in Erlang? In-Reply-To: <545A5AA0.4010906@research.att.com> References: <545A5AA0.4010906@research.att.com> Message-ID: <545A5D73.7040407@research.att.com> fwiw, i've got a system that's been running in production for years now that works pretty much like this. an "overseer" distributed application runs on one node of a cluster of machines, owning the virtual ip and delegating requests to other nodes. we don't do hot code upgrades, since some of the code is not erlang, but by upgrading nodes individually we install new code releases on 6 week release cycles without bringing the service down. it's pretty damn cool. On 11/05/2014 12:13 PM, Garry Hodgson wrote: > on the theme of "much easier in erlang", here's what i'd do: > > write a little python or shell program that loops forever, > sleeping for 10 seconds and writing a message including its pid. > > next write a gen_server that spawns one of those as > a port process, reads those messages and logs > them. include startup and "it died" log entries. > > now write an erlang/otp app that uses supervision tree to > spawn a few of those. fire it up, tail the logs, show it all > works, with erlang managing apps in other languages. > now kill one of the external process and watch it magically > restart, with shiny new pid as proof. > > for extra credit, run this on two nodes, using distributed > application feature. change log message to include hostname > as well as pid. show it running, then kill erlang on one > of the nodes. show it still runnning. start it up again, > then kill the other. repeat as needed to get the point across. > > i'm guessing this would all be a few dozen lines of code. > you could do it in other languages, but i really doubt it would > be as easy, or as robust. > > > > On 11/04/2014 07:18 PM, Curtis J Schofield (ram9) wrote: >> I am working on getting Erlang/Elixir adopted into our company. I've >> been asked to provide a simple example of something that your can only >> do in erlang. >> >> My reason for pushing to adopt this in our tech stack is that much of >> the system code I've written I feel I could write easier in Erlang and >> do it faster. I feel like one of the unique offerings of erlang >> relates to distributed computing or OTP reliable systems. >> >> >> Does anyone have thoughts on this question ? "What can I do in Erlang >> that I can't do in another language?" >> >> Ideally it would be something small that I could use as a demonstration. >> >> Thank you for any thoughts. >> >> Curtis >> >> >> >> >> --- >> Modern Yoga vs Traditional Yoga >> http://swamij.com/traditional-yoga.htm#swamirama >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > -- Garry Hodgson AT&T Chief Security Office (CSO) "This e-mail and any files transmitted with it are AT&T property, are confidential, and are intended solely for the use of the individual or entity to whom this e-mail is addressed. If you are not one of the named recipient(s) or otherwise have reason to believe that you have received this message in error, please notify the sender and delete this message immediately from your computer. Any other use, retention, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited." From curtis@REDACTED Wed Nov 5 18:58:13 2014 From: curtis@REDACTED (Curtis J Schofield (ram9)) Date: Wed, 5 Nov 2014 09:58:13 -0800 Subject: [erlang-questions] What can I only do in Erlang? In-Reply-To: References: Message-ID: Oh My! The outpouring of answers is SO appreciated! I'm very excited to go through and try these suggestion and update my proposal document with much of this intelligence. On Tue, Nov 4, 2014 at 4:18 PM, Curtis J Schofield (ram9) wrote: > I am working on getting Erlang/Elixir adopted into our company. I've > been asked to provide a simple example of something that your can only > do in erlang. > > My reason for pushing to adopt this in our tech stack is that much of > the system code I've written I feel I could write easier in Erlang and > do it faster. I feel like one of the unique offerings of erlang > relates to distributed computing or OTP reliable systems. > > > Does anyone have thoughts on this question ? "What can I do in Erlang > that I can't do in another language?" > > Ideally it would be something small that I could use as a demonstration. > > Thank you for any thoughts. > > Curtis > > > > > --- > Modern Yoga vs Traditional Yoga > http://swamij.com/traditional-yoga.htm#swamirama -- --- Modern Yoga vs Traditional Yoga http://swamij.com/traditional-yoga.htm#swamirama From chandrashekhar.mullaparthi@REDACTED Wed Nov 5 19:03:14 2014 From: chandrashekhar.mullaparthi@REDACTED (Chandru) Date: Wed, 5 Nov 2014 18:03:14 +0000 Subject: [erlang-questions] erlang:restart_timer In-Reply-To: References: Message-ID: On 5 November 2014 13:50, Tony Rogvall wrote: > Hi! > > From time to time I implement protocols and services that need watch dogs > and/or retransmit timers. > I will normally implement this using a BIF timer: > > > > I propose a new BIF: > > erlang:restart_timer(Ref, Time) -> false | RemainingTime > > This will restart a time if it is active and return the time that remained > when it was restarted. > If restart_timer return false either the timer has already time out or the > reference is not a timer. > > I second this proposal, quite a useful feature to have when implementing protocols. Fewer lines of code and the code becomes more readable. Chandru -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard.youngkin@REDACTED Wed Nov 5 19:15:36 2014 From: richard.youngkin@REDACTED (Youngkin, Rich) Date: Wed, 5 Nov 2014 11:15:36 -0700 Subject: [erlang-questions] ErlIDE on Mac OS X Mavericks - anyone using it successfully? Message-ID: > > Hi all, > > is there anyone using ErlIDE / Eclipse on Mac OS X Mavericks? > I have successfully installed it on Windows but I usually work on a Mac. > I tried with Eclipse 3.7, 4.2.2, 4.4, with no luck. > The IDE goes 'not responding' after a few keystrokes on the source editor. > In the rare moments of awakens it has *wonderful* hot code update features > which are really good especially for those accustomed to Eclipse in the > Java world. > I tried to follow the suggestions here: > https://github.com/erlide/erlide/wiki/Troubleshooting > especially when it says: > "For Macs, an alternative solution is to rename your computer from > foo.whatever to just foo (or anyhting without any dots)." > If I do this, apparently I cannot start an erlang instance with long name > as in > erl -name foo > The currently installed Java version is 1.8.0. > Has anyone managed to have ErlIDE up and running smoothly on Mac? > Thanks in advance. > Luca. > > Hi Luca, I'm using it with some success. I have had problems with it locking up. If my memory serves me correctly, it hung after I stopped a running Erlang shell by typing q(). at the command line. It behaves better if you use the Eclipse "stop" button to exit the shell. To unlock Eclipse, I took the nuclear option, killed all the beam processes. This has the unfortunate side effect of stopping ErlIDE. I haven't put any effort in trying to get the long host name to work. I'd be interested if anyone has any advice on how to get this to work. All that said, I do use ErlIDE, but mostly just for the on-the-fly compilation, syntax checking, and some local debugging. Cheers, Rich -------------- next part -------------- An HTML attachment was scrubbed... URL: From schuster@REDACTED Wed Nov 5 19:24:43 2014 From: schuster@REDACTED (Jonathan Schuster) Date: Wed, 5 Nov 2014 13:24:43 -0500 Subject: [erlang-questions] Looking for tricky protocol examples for verification research In-Reply-To: <8536865.C0t6tqKRTt@changa> References: <9261345.Oa5XLBjXf2@burrito> <8536865.C0t6tqKRTt@changa> Message-ID: On Fri, Oct 31, 2014 at 12:20 PM, zxq9 wrote: > In a duck typed language like Python we can treat anything as some type if > it > implements the basic methods expected of that type. An outside process > doesn't > need to care (sure we could introspect, but needing to is usually regarded > as > a sign of using the wrong tool). > > Erlang processes can be thought of the same way if we accept that in Erlang > "process" means something much closer to what languages like Python mean by > "object" than is commonly imagined. If I have a game character process it > might have an AI controller, or it might have a player controller. The > character process doesn't need to know or care, and we could actively swap > out > AI for player control on the fly and the character process wouldn't care, > so > long as all controllers, regardless of subtype (if we decide to use this > term > "type"), adhere to the "controller" protocol. > Yes, this is exactly the sort of abstraction I'm looking for. The main questions I have right now are (1) where are the real-life examples of such protocols, and (2) what is the most useful enforcement mechanism? A type system would probably be the most familiar to programmers, but it's not necessarily the most practical. Question 2 is really a research question, and the answer to question 1 should help me solve question 2. I haven't had a chance yet to look into the examples already mentioned on this thread, but hopefully there are examples there or in other programs I'm examining (e.g. ejabberd probably has several instances of this kind of problem). I agree that behaviors might be a good place to look. This does seem to be the closest thing Erlang has to built-in protocol specifications, so maybe I can extend those in an interesting way. Sorry for the late response; been a busy few days. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ingela.andin@REDACTED Wed Nov 5 19:52:10 2014 From: ingela.andin@REDACTED (Ingela Andin) Date: Wed, 5 Nov 2014 19:52:10 +0100 Subject: [erlang-questions] 'ssl_otp_session_cache' ets table In-Reply-To: References: Message-ID: Hi! 2014-11-05 14:15 GMT+01:00 Tony Rogvall : > I think the ssl session times is the problem here, and the lack of a > maximum size. > > You can change the session time in the ssl environment: session_lifetime > The default is set to 24 hours (in seconds) (if I read it correctly, in > ssl_manager.erl) > Default values are always hard. It is the maximum recommended time for a session to live according to the spec. > I guess that a session_cache_size could be a nice thing to have, > limiting the growth of the session cache. > > Yes I agree. > In other words you have to estimate the life time of your clients and > try to find a reasonable session_lifetime to match that, without blowing up > the system. > > Maybe the ssl_session_cache_api could be used to implement a strategy with > a max size > cache. Retire session least recently used, while performing the update? > There is a time_stamp in the session that that could be used for this > purpose. > > OTP: Why is a fixed limit not implemented in the standard > ssl_session_cache? > Could this be a target for DOS attacks? > We are aware of the problem, and it is on our todo list. One reason it has not had top priority is that on the server side there are often other mechanisms like firewalls and webbserver settings that limits the problem. And the reason why it was not implemented in the first place is that implementations by nature are iterative and at first you are faced with a lot of bigger problems to solve and then you need to iterate and fine tune and fix things that you now have a better understanding of. In current master there is a change to the session table that limits the growth on the client side if the client behaves inappropriate. It also splits the session table into a server and a client table which is a better implementation as the same Erlang node can be both a client and a server at the same time. So if someone feels like contributing a max limit please base it on the master branch, otherwise I suspect someone compiling about it did raise the priority level a little. Regards Ingela Erlang/OTP team - Ericsson AB > /Tony > > > On 5 nov 2014, at 13:08, Bogdan Andu wrote: > > > > > > > > Hi, > > > > I performed a series of test regarding the an Erlang SSL server. > > > > In this setup a major role is played by the table called > 'ssl_otp_session_cache', and of course the processes using it. > > > > The problem is that the size of table increases constantly and, because > an ets table does not automatically deallocate memory unless the object are > deleted from that table, the size of table remains the same even if there > no ssl connections to server. > > > > For example, with a single client running 'ad infinitum' the table > increases at a rate of 5 MBytes/hour. In 12 hours there are allocate around > 60 MB of memory only for this table. > > > > Some info about this: > > $ erl -sname console@REDACTED -remsh n1@REDACTED -setcookie operator > > Erlang/OTP 17 [erts-6.2] [source] [64-bit] [smp:2:2] [async-threads:10] > [kernel-poll:false] > > > > Eshell V6.2 (abort with ^G) > > > > > > (n1@REDACTED)1> ets:i(). > > id name type size mem owner > > > ---------------------------------------------------------------------------- > > 12 cookies set 0 291 auth > > 4111 code set 410 26132 code_server > > 8208 code_names set 58 7459 code_server > > 12307 httpc_manager__session_cookie_db bag 0 291 > httpc_manager > > 16404 ssl_otp_cacertificate_db set 0 291 > ssl_manager > > 20501 ssl_otp_ca_file_ref set 0 291 ssl_manager > > 24598 ssl_otp_pem_cache set 3 360 ssl_manager > > 28695 ssl_otp_session_cache ordered_set 138057 8421893 > ssl_manager > > 32797 dets duplicate_bag 2 308 dets > > 40990 ign_requests set 0 291 > inet_gethost_native > > 45087 ign_req_index set 0 291 > inet_gethost_native > > 2261635104 shell_records ordered_set 0 81 > <0.30638.37> > > ac_tab ac_tab set 33 2216 > application_controller > > code_map code_map set 100 2791 <0.72.0> > > config config set 12 892 <0.72.0> > > dets_owners dets_owners set 1 298 dets > > dets_registry dets_registry set 1 299 dets > > file_io_servers file_io_servers set 1 344 file_server_2 > > global_locks global_locks set 0 291 > global_name_server > > global_names global_names set 0 291 > global_name_server > > global_names_ext global_names_ext set 0 291 > global_name_server > > global_pid_ids global_pid_ids bag 0 291 > global_name_server > > global_pid_names global_pid_names bag 0 291 > global_name_server > > httpc_manager__handler_db httpc_manager__handler_db set 0 291 > httpc_manager > > httpc_manager__session_db httpc_manager__session_db set 0 291 > httpc_manager > > inet_cache inet_cache bag 0 291 inet_db > > inet_db inet_db set 29 600 inet_db > > inet_hosts_byaddr inet_hosts_byaddr bag 0 291 inet_db > > inet_hosts_byname inet_hosts_byname bag 0 291 inet_db > > inet_hosts_file_byaddr inet_hosts_file_byaddr bag 0 291 > inet_db > > inet_hosts_file_byname inet_hosts_file_byname bag 0 291 > inet_db > > models models set 3 28952 <0.72.0> > > sys_dist sys_dist set 1 334 net_kernel > > ok > > (n1@REDACTED)5> ets:info(ssl_otp_session_cache). > > undefined > > (n1@REDACTED)7> (8421893*8)/1024. > > 65796.0390625 > > (n1@REDACTED)8> memory(). > > [{total,92699464}, > > {processes,8964000}, > > {processes_used,8963152}, > > {system,83735464}, > > {atom,429569}, > > {atom_used,421768}, > > {binary,199040}, > > {code,10411520}, > > {ets,69163032}] > > > > The memory allocated to table 'ssl_otp_session_cache' is roughly 64 MB > in 12 hours. > > > > On an OpenBSD platform such process gets killed immediately it hits some > memory and/or CPU limits. > > > > To make this test on OpenBSD I had to put 'infinit' to memory, otherwise > the Erlang VM would be killed. > > > > How can one control , tweak or configure this table such that it does > not accumulate such data at such high rate. > > > > I seems the table being created private, and there is no way to > ets:delete_all_objects/1 from table manually. > > > > I know that this table caches some SSL data related to clients, but the > client has the same IP address, > > and I wonder why is neccesary to store a lot of SSL connection info > about the same client when only the ephemeral peer port > > differs? > > > > How the size of this table can be held in reasonable limits and the rate > it's size increases ? > > > > Please if somebody shed some light on these issues. > > > > Thank you, > > > > Bogdan > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > "Installing applications can lead to corruption over time. Applications > gradually write over each other's libraries, partial upgrades occur, user > and system errors happen, and minute changes may be unnoticeable and > difficult to fix" > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chandrashekhar.mullaparthi@REDACTED Wed Nov 5 22:21:44 2014 From: chandrashekhar.mullaparthi@REDACTED (Chandru) Date: Wed, 5 Nov 2014 21:21:44 +0000 Subject: [erlang-questions] design question from a beginner In-Reply-To: References: Message-ID: Hi Gerald, You don't really need all that OTP stuff for the use case you have. You can use them, but it would be a bit artificial. Attached is a module I use for load testing of ibrowse - hopefully it will give you some ideas. regards, Chandru On 5 November 2014 14:30, Gerald Weber wrote: > Hi Coders, > > > > finally i have a first real world project to support my efforts in > learning erlang and OTP: > > Trying to write an application to load-test a Java backend. This java > daemon talks json over HTTP (jetty). > > > > The backend requires a login (GET request, no HTTP Authentication), some > register stuff for subscriptions > > and finally a loop to receive all of the events (the step waiting for > events is blocking) > > > > So my vision is: > > - 1x application > > -- 1x supervisor > > --- 1x (?) gen_server > > ---- n x Y > > > > The gen_server should wait for a msg like {start_workers,50} and should > then start 50 instances of Y > > connecting to the java backend, run some httpc:request calls and receive > the return values (which is json) > > > > I have really no clear idea which OTP principle can be used for Y, maybe > my mindset tells me gen_server is a process to "serve data" > > and not to ask for data. What would be the best bet ? > > Or is my simple understanding that Y screams for an gen_fsm including the > httpc calls not that wrong ? > > > > Can the supervisor be told to start x instances of Y on request ? > > > > thx and sorry for my maybe too easy questions. > > gerald > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ibrowse_load_test.erl Type: text/x-erlang Size: 7529 bytes Desc: not available URL: From xiande.duan@REDACTED Thu Nov 6 02:11:06 2014 From: xiande.duan@REDACTED (Duan, Xiande) Date: Thu, 6 Nov 2014 01:11:06 +0000 Subject: [erlang-questions] What can I only do in Erlang? In-Reply-To: <20141105165106.GF2531@ferdair.local> References: <1442743.qSh8HrgRRI@burrito> <20141105165106.GF2531@ferdair.local> Message-ID: <5A0DB9EC001ECD4DA29B9F40F5A35F40181E43@SHSMSX103.ccr.corp.intel.com> Amazing story about 'inspection'. -----Original Message----- From: erlang-questions-bounces@REDACTED [mailto:erlang-questions-bounces@REDACTED] On Behalf Of Fred Hebert Sent: Thursday, November 6, 2014 12:51 AM To: zxq9 Cc: erlang-questions@REDACTED Subject: Re: [erlang-questions] What can I only do in Erlang? This is a good post and I'd like to expand on a view points: On 11/05, zxq9 wrote: > Hi Curtis, > > > Binaries: [...] > > Sockets: [...] > Particularly interesting about sockets is also the ease of switching options around. I remember working on a product with a different company, where a rewrite to use kernel polling on their end took them a few days to weeks (I can't remember precisely). For us, using Erlang, it was a question of booting the node with the '+K true' option to the VM. We can then write code the way we want it, whether we poll with 'recv' or whether we want the code in the socket to be delivered as messages so it looks more like push than pull. Yet, even in the 'push' model, you can have control flow via choosing when to switch between the passive and active modes. On top of this, there are metrics being kept on each socket, about the size and packets being transfered. That's kind of alright, but the real power shows when you know that Ports (the type sockets are implemented in) behave like processes, can be linked, and introspected. That means I can write custom one-liners in a production shell during an incident and go see who owns a connection that might be going bad, is doing too much input or output, and so on, without the need to go and write special instrumentation. That's cool. > ASN.1: [...] > > "Let it crash": [...] > > Supervision: Automatic restarts to a known state. Wow. Limiting how > bad things can get is amazingly powerful. But you won't realize this > until you screw something up really bad, and then realize how much > worse it would have been without a supervision tree helping you. > I wrote on this particular point at http://ferd.ca/it-s-about-the-guarantees.html (and in http://erlang-in-anger.com) and it makes a crazy difference when your systems are engineered with these principles in mind, rather than just "he he restart". It's the difference between a solid crash-recovery mode, and looping your call in a 'try ... catch'. Any system that provides supervisors but cannot provide a fallback to a stable known state are missing the point. > Inspection: [...] > Yes! Introspection is currently my *favorite* thing about using Erlang in production environments. I can't say how many weird behaviours or bugs I've identified and fixed through introspection features. I've posted a few war stories before here and there, in talks, and added some of them to Erlang In Anger, but there's no easy way to explain the enthusiasm I get. The tasks of thread pulling that used to be painful and terrible wading through tonnes of garbage are now things I can look forward to in the same domain as puzzle solving. And that stuff *will* happen. As much as we like bug prevention, we're all aware of the law of diminishing returns that more or less gets applied in a way that tells you your product (if not life-critical) will be shipped with hard-to-fix bugs. When they'll become critical (or too frequent), then they'll be worth figuring out and fixing. In Programming Forth (Stephen Pelc, 2011), the author says "Debugging isn't an art, it's a science!" and provides the following (ASCII-fied) diagram: find a problem ---> gather data ---> form hypothesis ----, .--------------------------------------------------------' '-> design experiment ---> prove hypothesis --> fix problem Which then loops back on itself. By far, the easiest bits are 'finding the problem'. The difficult ones are to form the *right* hypothesis that will let you design a proper experiment and prove your fix works. It's especially true of Heisenbugs that ruin your life by disappearing as you observe them. So how do you go about it? *GATHER MORE DATA*. The more data you have, the easiest it becomes to form good hypotheses. Traditionally, there's four big ways to do it in the server world: - Gather system metrics - Add logs and read them carefully - Try to replicate it locally - Get a core dump and debug that Those are all available in Erlang, but they're often impractical: - System metrics are often wide and won't help with fine-grained bugs, but will help provide context - logs can generate lots and lots of expansive and useless data, and logging itself may cause the bug to stop happening. - Replicating it locally without any prior information is more or less blind programming. Take shots in the dark until you figure out you've killed the right monster. - Core dumps are post-facto items. They often show you the bad state, but rarely how to get there. More recently, systemtap/dtrace have come into the picture. These help a lot for some classes of bugs. However, I have not yet felt the need to run either in production. Why? Because Erlang comes with tracing tools that trace *every god damn thing* for you out of the box. Process spawns? It's in. Messages sent and received? It's in. Function calls filtered with specific arguments? it's in! Garbage collections? it's in. Processes that got scheduled for too long? it's in. Sockets that have their buffers full? It's in. It's all out of the box. It's all available anywhere, and it's all usable in production (assuming you are using a library with the right safeguards). I've had a fantastically annoying case in the HTTP Proxy/router we operate where a customer was complaining of bugs in some specific responses formatted our way, and wanted to know if *we* modified it, or if some middleman between us and the customer did it. Of course the problem was that connections were encrypted and whatnot, so the usual tcpdump isn't doable without some serious hassle about keys and decrypting and whatnot. How did we go about it? We traced the processes that were handling their responses, and I found the right socket attached to their account for this long request. I dropped a trace from the shell: recon_trace:calls( {gen_tcp, send, fun([Port,_Data]) when Port == TheirPort -> ok end}, 10 ). And I could get all the data we'd send over their connection only (no other customer or socket was being introspected). It took me 5 minutes to figure out a once-in-a-billion bug with that stuff. Do you have any idea how long and how much development effort this could have taken otherwise? It would certainly take more than minutes. The introspection goes even further with OTP processes where you can inspect the state of running processes, but also *swap* it out with different ones. Ever wondered if making a buffer bigger for an important customer would make their life easier? I did it in 2 minutes over a cluster of 40 nodes, and rolled it back after the experiment, without the need to add special cose for different buffer sizes, configuration values and changes to data models to target oly *that* customer, etc. The experiment was unsuccessful, by the way, and buffer sizes were better off the same. What took me 2 minutes to try live saved our team days of development, scaffolding and deploys. This kind of introspection on that level is a thing I've never seen in other languages, and they make my life so much easier. I don't even want to touch other concurrent languages that don't have these features, because my development time will be stolen into operations time. I could probably tell more about a lot of things, but I'm not feeling like writign a novel today. Regards, Fred. _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions From zxq9@REDACTED Thu Nov 6 02:24:29 2014 From: zxq9@REDACTED (zxq9) Date: Thu, 06 Nov 2014 10:24:29 +0900 Subject: [erlang-questions] Looking for tricky protocol examples for verification research In-Reply-To: References: <8536865.C0t6tqKRTt@changa> Message-ID: <2736348.o5O1zzTexJ@burrito> On Wednesday 05 November 2014 13:24:43 Jonathan Schuster wrote: > On Fri, Oct 31, 2014 at 12:20 PM, zxq9 wrote: > > Erlang processes can be thought of the same way if we accept that in > > Erlang > > "process" means something much closer to what languages like Python mean > > by > > "object" than is commonly imagined. If I have a game character process it > > might have an AI controller, or it might have a player controller. The > > character process doesn't need to know or care, and we could actively swap > > out > > AI for player control on the fly and the character process wouldn't care, > > so > > long as all controllers, regardless of subtype (if we decide to use this > > term > > "type"), adhere to the "controller" protocol. > > Yes, this is exactly the sort of abstraction I'm looking for. The main > questions I have right now are (1) where are the real-life examples of such > protocols, and (2) what is the most useful enforcement mechanism? A type > system would probably be the most familiar to programmers, but it's not > necessarily the most practical. Question 2 is really a research question, > and the answer to question 1 should help me solve question 2. > > I haven't had a chance yet to look into the examples already mentioned on > this thread, but hopefully there are examples there or in other programs > I'm examining (e.g. ejabberd probably has several instances of this kind of > problem). > > I agree that behaviors might be a good place to look. This does seem to be > the closest thing Erlang has to built-in protocol specifications, so maybe > I can extend those in an interesting way. I'm working on a demonstration system right now that is full of this particular case. Actually, its the source of the AI/player controller swap-out example. The protocols are dead simple, though, but even so a verification mechanism would be nice. While writing my own system I'm confident I won't mess up an easy protocol that I'm intimately familiar with (and if I do I'll know immediately), but game system operators who want to extend existing games by writing new AIs, interesting game items, new mobs, etc. will have to write new modules. These modules will have to adhere to the protocols that already exist, and having a way to declare them in code and having the compiler check that they do receive and send the correct protocol would be nice. Indication of protocol compliance must permit pluralities. An AI only adheres to the "mob controller" protocol, but a player controller must adhere to the network messaging, chat system and mob controller protocols at once (and more if the system is extended in almost any way). One trick to verification is synchronous messaging with a third-process response. For example, how to verify that: A sends B a synchronous message {From, Ref, Message}. Process B does something with it, and passes {From, Ref, Data} on to process C. Process C responds to A with {Ref, Response2} which satisfies A's synchronous response expectation. >From A's perspective this is synchronous, to B and C it is not. This case does not occur in function returns, so I don't think it has been thought through much in terms of external type or protocol assignment -- at least I've never seen it mentioned before. Good luck with your project. Let me know if you make any progress -- it would be a good tool when writing software in teams or writing code intended for extension by other Erlangers (and being a wizard should not be a requirement to extend code written for this purpose!). -Craig From bog495@REDACTED Thu Nov 6 08:53:53 2014 From: bog495@REDACTED (Bogdan Andu) Date: Thu, 6 Nov 2014 09:53:53 +0200 Subject: [erlang-questions] 'ssl_otp_session_cache' ets table In-Reply-To: References: Message-ID: Hi, on my production servers I have relayd (on OpenBSD) daemon as a reverse proxy to some webservers where one can fine tune some connection parameters, as well as some ssl parameters. I give a snippet from a relayd.conf configuration file on one of my production server: ............. # # Relay and protocol for HTTP layer 7 loadbalancing and SSL acceleration # http protocol www_ssl_prot { header append "$REMOTE_ADDR" to "X-Forwarded-For" header append "$SERVER_ADDR:$SERVER_PORT" to "X-Forwarded-By" header change "Connection" to "close" response header change "Server" to "Apache 0.1" # Various TCP performance options tcp { nodelay, sack, socket buffer 65536, backlog 128 } ssl { no sslv2, no sslv3, tlsv1, ciphers "HIGH" } ssl session cache disable } the last directive tells relayd not to use ssl cache. This configuration is working for years and relayd was restarted once by accident - my fault. SO, y question is: can we have this configurable in Erlang, in other words, we might be able to start an erlang vm such as: erl -ssl session_cache 'disable' -name x@REDACTED .... The ssl option session_cache can be set to disabled by default and can take values either disable or enabled. Just to let you know... The statistics of a node running from yesterday when I opened this thread of discussion, using a single client: Node: 'n1@REDACTED' (Connected) (17/6.2) unix (openbsd 5.4.0) CPU:2 SMP +A:10 Time: local time 09:48:48, up for 000:20:27:49, 0ms latency, Processes: total 681 (RQ 0) at 610275 RpI using 11526.4k (11805.8k allocated) Memory: Sys 94835.5k, Atom 407.7k/419.5k, Bin 176.4k, Code 9934.7k, Ets 80739.2k ........... So, can be this made configurable? On Wed, Nov 5, 2014 at 8:52 PM, Ingela Andin wrote: > Hi! > > 2014-11-05 14:15 GMT+01:00 Tony Rogvall : > >> I think the ssl session times is the problem here, and the lack of a >> maximum size. >> >> You can change the session time in the ssl environment: session_lifetime >> The default is set to 24 hours (in seconds) (if I read it correctly, in >> ssl_manager.erl) >> > > Default values are always hard. It is the maximum recommended time for a > session to live > according to the spec. > > >> I guess that a session_cache_size could be a nice thing to have, >> limiting the growth of the session cache. >> >> > Yes I agree. > > >> In other words you have to estimate the life time of your clients and >> try to find a reasonable session_lifetime to match that, without blowing >> up >> the system. >> >> Maybe the ssl_session_cache_api could be used to implement a strategy >> with a max size >> cache. Retire session least recently used, while performing the update? >> There is a time_stamp in the session that that could be used for this >> purpose. >> >> OTP: Why is a fixed limit not implemented in the standard >> ssl_session_cache? >> Could this be a target for DOS attacks? >> > > > We are aware of the problem, and it is on our todo list. One reason it has > not had top priority is that on the server side there are often other > mechanisms > like firewalls and webbserver settings that limits the problem. And the > reason why it was not implemented in the first place is that > implementations by nature > are iterative and at first you are faced with a lot of bigger problems to > solve and then you need to iterate and fine tune and fix things that you > now have a better understanding of. > > In current master there is a change to the session table that limits the > growth on the client side if the client behaves inappropriate. It also > splits the session table into > a server and a client table which is a better implementation as the same > Erlang node can be both a client and a server at the same time. So if > someone feels like contributing a max limit please base it on the master > branch, otherwise I suspect someone compiling about it did raise the > priority level a little. > > > Regards Ingela Erlang/OTP team - Ericsson AB > > > >> /Tony >> >> > On 5 nov 2014, at 13:08, Bogdan Andu wrote: >> > >> > >> > >> > Hi, >> > >> > I performed a series of test regarding the an Erlang SSL server. >> > >> > In this setup a major role is played by the table called >> 'ssl_otp_session_cache', and of course the processes using it. >> > >> > The problem is that the size of table increases constantly and, because >> an ets table does not automatically deallocate memory unless the object are >> deleted from that table, the size of table remains the same even if there >> no ssl connections to server. >> > >> > For example, with a single client running 'ad infinitum' the table >> increases at a rate of 5 MBytes/hour. In 12 hours there are allocate around >> 60 MB of memory only for this table. >> > >> > Some info about this: >> > $ erl -sname console@REDACTED -remsh n1@REDACTED -setcookie operator >> > Erlang/OTP 17 [erts-6.2] [source] [64-bit] [smp:2:2] [async-threads:10] >> [kernel-poll:false] >> > >> > Eshell V6.2 (abort with ^G) >> > >> > >> > (n1@REDACTED)1> ets:i(). >> > id name type size mem owner >> > >> ---------------------------------------------------------------------------- >> > 12 cookies set 0 291 auth >> > 4111 code set 410 26132 code_server >> > 8208 code_names set 58 7459 code_server >> > 12307 httpc_manager__session_cookie_db bag 0 291 >> httpc_manager >> > 16404 ssl_otp_cacertificate_db set 0 291 >> ssl_manager >> > 20501 ssl_otp_ca_file_ref set 0 291 ssl_manager >> > 24598 ssl_otp_pem_cache set 3 360 ssl_manager >> > 28695 ssl_otp_session_cache ordered_set 138057 8421893 >> ssl_manager >> > 32797 dets duplicate_bag 2 308 dets >> > 40990 ign_requests set 0 291 >> inet_gethost_native >> > 45087 ign_req_index set 0 291 >> inet_gethost_native >> > 2261635104 shell_records ordered_set 0 81 >> <0.30638.37> >> > ac_tab ac_tab set 33 2216 >> application_controller >> > code_map code_map set 100 2791 <0.72.0> >> > config config set 12 892 <0.72.0> >> > dets_owners dets_owners set 1 298 dets >> > dets_registry dets_registry set 1 299 dets >> > file_io_servers file_io_servers set 1 344 file_server_2 >> > global_locks global_locks set 0 291 >> global_name_server >> > global_names global_names set 0 291 >> global_name_server >> > global_names_ext global_names_ext set 0 291 >> global_name_server >> > global_pid_ids global_pid_ids bag 0 291 >> global_name_server >> > global_pid_names global_pid_names bag 0 291 >> global_name_server >> > httpc_manager__handler_db httpc_manager__handler_db set 0 291 >> httpc_manager >> > httpc_manager__session_db httpc_manager__session_db set 0 291 >> httpc_manager >> > inet_cache inet_cache bag 0 291 inet_db >> > inet_db inet_db set 29 600 inet_db >> > inet_hosts_byaddr inet_hosts_byaddr bag 0 291 inet_db >> > inet_hosts_byname inet_hosts_byname bag 0 291 inet_db >> > inet_hosts_file_byaddr inet_hosts_file_byaddr bag 0 291 >> inet_db >> > inet_hosts_file_byname inet_hosts_file_byname bag 0 291 >> inet_db >> > models models set 3 28952 <0.72.0> >> > sys_dist sys_dist set 1 334 net_kernel >> > ok >> > (n1@REDACTED)5> ets:info(ssl_otp_session_cache). >> > undefined >> > (n1@REDACTED)7> (8421893*8)/1024. >> > 65796.0390625 >> > (n1@REDACTED)8> memory(). >> > [{total,92699464}, >> > {processes,8964000}, >> > {processes_used,8963152}, >> > {system,83735464}, >> > {atom,429569}, >> > {atom_used,421768}, >> > {binary,199040}, >> > {code,10411520}, >> > {ets,69163032}] >> > >> > The memory allocated to table 'ssl_otp_session_cache' is roughly 64 MB >> in 12 hours. >> > >> > On an OpenBSD platform such process gets killed immediately it hits >> some memory and/or CPU limits. >> > >> > To make this test on OpenBSD I had to put 'infinit' to memory, >> otherwise the Erlang VM would be killed. >> > >> > How can one control , tweak or configure this table such that it does >> not accumulate such data at such high rate. >> > >> > I seems the table being created private, and there is no way to >> ets:delete_all_objects/1 from table manually. >> > >> > I know that this table caches some SSL data related to clients, but the >> client has the same IP address, >> > and I wonder why is neccesary to store a lot of SSL connection info >> about the same client when only the ephemeral peer port >> > differs? >> > >> > How the size of this table can be held in reasonable limits and the >> rate it's size increases ? >> > >> > Please if somebody shed some light on these issues. >> > >> > Thank you, >> > >> > Bogdan >> > >> > >> > _______________________________________________ >> > erlang-questions mailing list >> > erlang-questions@REDACTED >> > http://erlang.org/mailman/listinfo/erlang-questions >> >> "Installing applications can lead to corruption over time. Applications >> gradually write over each other's libraries, partial upgrades occur, user >> and system errors happen, and minute changes may be unnoticeable and >> difficult to fix" >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ingela.andin@REDACTED Thu Nov 6 11:28:35 2014 From: ingela.andin@REDACTED (Ingela Andin) Date: Thu, 6 Nov 2014 11:28:35 +0100 Subject: [erlang-questions] 'ssl_otp_session_cache' ets table In-Reply-To: References: Message-ID: Hi! 2014-11-06 8:53 GMT+01:00 Bogdan Andu : > Hi, > > on my production servers I have relayd (on OpenBSD) daemon as a reverse > proxy to some webservers > where one can fine tune some connection parameters, as well as some ssl > parameters. > > I give a snippet from a relayd.conf configuration file on one of my > production server: > > ............. > # > # Relay and protocol for HTTP layer 7 loadbalancing and SSL acceleration > # > http protocol www_ssl_prot { > header append "$REMOTE_ADDR" to "X-Forwarded-For" > header append "$SERVER_ADDR:$SERVER_PORT" to "X-Forwarded-By" > header change "Connection" to "close" > > response header change "Server" to "Apache 0.1" > > # Various TCP performance options > tcp { nodelay, sack, socket buffer 65536, backlog 128 } > > ssl { no sslv2, no sslv3, tlsv1, ciphers "HIGH" } > ssl session cache disable > } > > the last directive tells relayd not to use ssl cache. > > This configuration is working for years and relayd was restarted once by > accident - my fault. > > SO, y question is: > > can we have this configurable in Erlang, in other words, we might be able > to start an erlang vm such as: > > erl -ssl session_cache 'disable' -name x@REDACTED .... > > The ssl option session_cache can be set to disabled by default and can > take values either disable or enabled. > > > You can already disable the reuse of the sessions using the server option *{reuse_sessions, boolean()}* which default to true. The thing we plan to do is to have a configurable limit on the table size when sessions are reused. Regards Ingela Erlang/OTP team Ericsson AB > Just to let you know... > > The statistics of a node running from yesterday when I opened this thread > of discussion, using a single client: > > Node: 'n1@REDACTED' (Connected) (17/6.2) unix (openbsd 5.4.0) CPU:2 SMP > +A:10 > Time: local time 09:48:48, up for 000:20:27:49, 0ms latency, > Processes: total 681 (RQ 0) at 610275 RpI using 11526.4k (11805.8k > allocated) > Memory: Sys 94835.5k, Atom 407.7k/419.5k, Bin 176.4k, Code 9934.7k, Ets > 80739.2k > > ........... > > So, can be this made configurable? > > > > On Wed, Nov 5, 2014 at 8:52 PM, Ingela Andin > wrote: > >> Hi! >> >> 2014-11-05 14:15 GMT+01:00 Tony Rogvall : >> >>> I think the ssl session times is the problem here, and the lack of a >>> maximum size. >>> >>> You can change the session time in the ssl environment: session_lifetime >>> The default is set to 24 hours (in seconds) (if I read it correctly, in >>> ssl_manager.erl) >>> >> >> Default values are always hard. It is the maximum recommended time for a >> session to live >> according to the spec. >> >> >>> I guess that a session_cache_size could be a nice thing to have, >>> limiting the growth of the session cache. >>> >>> >> Yes I agree. >> >> >>> In other words you have to estimate the life time of your clients and >>> try to find a reasonable session_lifetime to match that, without blowing >>> up >>> the system. >>> >>> Maybe the ssl_session_cache_api could be used to implement a strategy >>> with a max size >>> cache. Retire session least recently used, while performing the update? >>> There is a time_stamp in the session that that could be used for this >>> purpose. >>> >>> OTP: Why is a fixed limit not implemented in the standard >>> ssl_session_cache? >>> Could this be a target for DOS attacks? >>> >> >> >> We are aware of the problem, and it is on our todo list. One reason it >> has not had top priority is that on the server side there are often other >> mechanisms >> like firewalls and webbserver settings that limits the problem. And the >> reason why it was not implemented in the first place is that >> implementations by nature >> are iterative and at first you are faced with a lot of bigger problems to >> solve and then you need to iterate and fine tune and fix things that you >> now have a better understanding of. >> >> In current master there is a change to the session table that limits the >> growth on the client side if the client behaves inappropriate. It also >> splits the session table into >> a server and a client table which is a better implementation as the same >> Erlang node can be both a client and a server at the same time. So if >> someone feels like contributing a max limit please base it on the master >> branch, otherwise I suspect someone compiling about it did raise the >> priority level a little. >> >> >> Regards Ingela Erlang/OTP team - Ericsson AB >> >> >> >>> /Tony >>> >>> > On 5 nov 2014, at 13:08, Bogdan Andu wrote: >>> > >>> > >>> > >>> > Hi, >>> > >>> > I performed a series of test regarding the an Erlang SSL server. >>> > >>> > In this setup a major role is played by the table called >>> 'ssl_otp_session_cache', and of course the processes using it. >>> > >>> > The problem is that the size of table increases constantly and, >>> because an ets table does not automatically deallocate memory unless the >>> object are deleted from that table, the size of table remains the same even >>> if there no ssl connections to server. >>> > >>> > For example, with a single client running 'ad infinitum' the table >>> increases at a rate of 5 MBytes/hour. In 12 hours there are allocate around >>> 60 MB of memory only for this table. >>> > >>> > Some info about this: >>> > $ erl -sname console@REDACTED -remsh n1@REDACTED -setcookie operator >>> > Erlang/OTP 17 [erts-6.2] [source] [64-bit] [smp:2:2] >>> [async-threads:10] [kernel-poll:false] >>> > >>> > Eshell V6.2 (abort with ^G) >>> > >>> > >>> > (n1@REDACTED)1> ets:i(). >>> > id name type size mem owner >>> > >>> ---------------------------------------------------------------------------- >>> > 12 cookies set 0 291 auth >>> > 4111 code set 410 26132 code_server >>> > 8208 code_names set 58 7459 code_server >>> > 12307 httpc_manager__session_cookie_db bag 0 291 >>> httpc_manager >>> > 16404 ssl_otp_cacertificate_db set 0 291 >>> ssl_manager >>> > 20501 ssl_otp_ca_file_ref set 0 291 ssl_manager >>> > 24598 ssl_otp_pem_cache set 3 360 ssl_manager >>> > 28695 ssl_otp_session_cache ordered_set 138057 8421893 >>> ssl_manager >>> > 32797 dets duplicate_bag 2 308 dets >>> > 40990 ign_requests set 0 291 >>> inet_gethost_native >>> > 45087 ign_req_index set 0 291 >>> inet_gethost_native >>> > 2261635104 shell_records ordered_set 0 81 >>> <0.30638.37> >>> > ac_tab ac_tab set 33 2216 >>> application_controller >>> > code_map code_map set 100 2791 <0.72.0> >>> > config config set 12 892 <0.72.0> >>> > dets_owners dets_owners set 1 298 dets >>> > dets_registry dets_registry set 1 299 dets >>> > file_io_servers file_io_servers set 1 344 file_server_2 >>> > global_locks global_locks set 0 291 >>> global_name_server >>> > global_names global_names set 0 291 >>> global_name_server >>> > global_names_ext global_names_ext set 0 291 >>> global_name_server >>> > global_pid_ids global_pid_ids bag 0 291 >>> global_name_server >>> > global_pid_names global_pid_names bag 0 291 >>> global_name_server >>> > httpc_manager__handler_db httpc_manager__handler_db set 0 291 >>> httpc_manager >>> > httpc_manager__session_db httpc_manager__session_db set 0 291 >>> httpc_manager >>> > inet_cache inet_cache bag 0 291 inet_db >>> > inet_db inet_db set 29 600 inet_db >>> > inet_hosts_byaddr inet_hosts_byaddr bag 0 291 inet_db >>> > inet_hosts_byname inet_hosts_byname bag 0 291 inet_db >>> > inet_hosts_file_byaddr inet_hosts_file_byaddr bag 0 291 >>> inet_db >>> > inet_hosts_file_byname inet_hosts_file_byname bag 0 291 >>> inet_db >>> > models models set 3 28952 <0.72.0> >>> > sys_dist sys_dist set 1 334 net_kernel >>> > ok >>> > (n1@REDACTED)5> ets:info(ssl_otp_session_cache). >>> > undefined >>> > (n1@REDACTED)7> (8421893*8)/1024. >>> > 65796.0390625 >>> > (n1@REDACTED)8> memory(). >>> > [{total,92699464}, >>> > {processes,8964000}, >>> > {processes_used,8963152}, >>> > {system,83735464}, >>> > {atom,429569}, >>> > {atom_used,421768}, >>> > {binary,199040}, >>> > {code,10411520}, >>> > {ets,69163032}] >>> > >>> > The memory allocated to table 'ssl_otp_session_cache' is roughly 64 MB >>> in 12 hours. >>> > >>> > On an OpenBSD platform such process gets killed immediately it hits >>> some memory and/or CPU limits. >>> > >>> > To make this test on OpenBSD I had to put 'infinit' to memory, >>> otherwise the Erlang VM would be killed. >>> > >>> > How can one control , tweak or configure this table such that it does >>> not accumulate such data at such high rate. >>> > >>> > I seems the table being created private, and there is no way to >>> ets:delete_all_objects/1 from table manually. >>> > >>> > I know that this table caches some SSL data related to clients, but >>> the client has the same IP address, >>> > and I wonder why is neccesary to store a lot of SSL connection info >>> about the same client when only the ephemeral peer port >>> > differs? >>> > >>> > How the size of this table can be held in reasonable limits and the >>> rate it's size increases ? >>> > >>> > Please if somebody shed some light on these issues. >>> > >>> > Thank you, >>> > >>> > Bogdan >>> > >>> > >>> > _______________________________________________ >>> > erlang-questions mailing list >>> > erlang-questions@REDACTED >>> > http://erlang.org/mailman/listinfo/erlang-questions >>> >>> "Installing applications can lead to corruption over time. Applications >>> gradually write over each other's libraries, partial upgrades occur, user >>> and system errors happen, and minute changes may be unnoticeable and >>> difficult to fix" >>> >>> >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kunthar@REDACTED Thu Nov 6 13:24:26 2014 From: kunthar@REDACTED (Gokhan Boranalp) Date: Thu, 6 Nov 2014 14:24:26 +0200 Subject: [erlang-questions] capn proto Message-ID: I am just wondering anyone else interested to implement CapnProto? http://kentonv.github.io/capnproto/ Especially, in regard of protocol section for Riak ;-) -- BR, \|/ Kunthar From sergej.jurecko@REDACTED Thu Nov 6 13:36:08 2014 From: sergej.jurecko@REDACTED (=?UTF-8?Q?Sergej_Jure=C4=8Dko?=) Date: Thu, 6 Nov 2014 13:36:08 +0100 Subject: [erlang-questions] capn proto In-Reply-To: References: Message-ID: Well there already is: https://github.com/kaos/ecapnp Sergej On Thu, Nov 6, 2014 at 1:24 PM, Gokhan Boranalp wrote: > I am just wondering anyone else interested to implement CapnProto? > http://kentonv.github.io/capnproto/ > Especially, in regard of protocol section for Riak ;-) > > -- > BR, > \|/ Kunthar > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bog495@REDACTED Thu Nov 6 14:39:09 2014 From: bog495@REDACTED (Bogdan Andu) Date: Thu, 6 Nov 2014 15:39:09 +0200 Subject: [erlang-questions] 'ssl_otp_session_cache' ets table In-Reply-To: References: Message-ID: Hi, i have tried to pass to execute th following forms of commands: erl -ssl session_lifetime 60 erl -ssl session_lifetime '60' erl -ssl session_lifetime '[60]' in order to set a ssl session_lifetime of 60 seconds. in init of ssl_manager , after session_lifetime + 5 seconds a process is created to validate or invalidate sessions. So, after 65 seconds a foldl is called on sessions table to check sessions after formula: Now - TimeStamp < LifeTime However, even after I stop the client, the table 'ssl_otp_session_cache' 's size remains the same. After that once every ?CLEAN_SESSION_DB which is 60 seconds , the table is sweeped to delete any expired sessions, but also nothing happens. On Thu, Nov 6, 2014 at 12:28 PM, Ingela Andin wrote: > Hi! > > > 2014-11-06 8:53 GMT+01:00 Bogdan Andu : > >> Hi, >> >> on my production servers I have relayd (on OpenBSD) daemon as a reverse >> proxy to some webservers >> where one can fine tune some connection parameters, as well as some ssl >> parameters. >> >> I give a snippet from a relayd.conf configuration file on one of my >> production server: >> >> ............. >> # >> # Relay and protocol for HTTP layer 7 loadbalancing and SSL acceleration >> # >> http protocol www_ssl_prot { >> header append "$REMOTE_ADDR" to "X-Forwarded-For" >> header append "$SERVER_ADDR:$SERVER_PORT" to "X-Forwarded-By" >> header change "Connection" to "close" >> >> response header change "Server" to "Apache 0.1" >> >> # Various TCP performance options >> tcp { nodelay, sack, socket buffer 65536, backlog 128 } >> >> ssl { no sslv2, no sslv3, tlsv1, ciphers "HIGH" } >> ssl session cache disable >> } >> >> the last directive tells relayd not to use ssl cache. >> >> This configuration is working for years and relayd was restarted once by >> accident - my fault. >> >> SO, y question is: >> >> can we have this configurable in Erlang, in other words, we might be able >> to start an erlang vm such as: >> >> erl -ssl session_cache 'disable' -name x@REDACTED .... >> >> The ssl option session_cache can be set to disabled by default and can >> take values either disable or enabled. >> >> >> > You can already disable the reuse of the sessions using the server option *{reuse_sessions, > boolean()}* which default to true. > The thing we plan to do is to have a configurable limit on the table size > when sessions are reused. > > Regards Ingela Erlang/OTP team Ericsson AB > > > > > > >> Just to let you know... >> >> The statistics of a node running from yesterday when I opened this >> thread of discussion, using a single client: >> >> Node: 'n1@REDACTED' (Connected) (17/6.2) unix (openbsd 5.4.0) CPU:2 SMP >> +A:10 >> Time: local time 09:48:48, up for 000:20:27:49, 0ms latency, >> Processes: total 681 (RQ 0) at 610275 RpI using 11526.4k (11805.8k >> allocated) >> Memory: Sys 94835.5k, Atom 407.7k/419.5k, Bin 176.4k, Code 9934.7k, Ets >> 80739.2k >> >> ........... >> >> So, can be this made configurable? >> >> >> >> On Wed, Nov 5, 2014 at 8:52 PM, Ingela Andin >> wrote: >> >>> Hi! >>> >>> 2014-11-05 14:15 GMT+01:00 Tony Rogvall : >>> >>>> I think the ssl session times is the problem here, and the lack of a >>>> maximum size. >>>> >>>> You can change the session time in the ssl environment: >>>> session_lifetime >>>> The default is set to 24 hours (in seconds) (if I read it correctly, in >>>> ssl_manager.erl) >>>> >>> >>> Default values are always hard. It is the maximum recommended time for a >>> session to live >>> according to the spec. >>> >>> >>>> I guess that a session_cache_size could be a nice thing to have, >>>> limiting the growth of the session cache. >>>> >>>> >>> Yes I agree. >>> >>> >>>> In other words you have to estimate the life time of your clients and >>>> try to find a reasonable session_lifetime to match that, without >>>> blowing up >>>> the system. >>>> >>>> Maybe the ssl_session_cache_api could be used to implement a strategy >>>> with a max size >>>> cache. Retire session least recently used, while performing the update? >>>> There is a time_stamp in the session that that could be used for this >>>> purpose. >>>> >>>> OTP: Why is a fixed limit not implemented in the standard >>>> ssl_session_cache? >>>> Could this be a target for DOS attacks? >>>> >>> >>> >>> We are aware of the problem, and it is on our todo list. One reason it >>> has not had top priority is that on the server side there are often other >>> mechanisms >>> like firewalls and webbserver settings that limits the problem. And the >>> reason why it was not implemented in the first place is that >>> implementations by nature >>> are iterative and at first you are faced with a lot of bigger problems >>> to solve and then you need to iterate and fine tune and fix things that you >>> now have a better understanding of. >>> >>> In current master there is a change to the session table that limits the >>> growth on the client side if the client behaves inappropriate. It also >>> splits the session table into >>> a server and a client table which is a better implementation as the same >>> Erlang node can be both a client and a server at the same time. So if >>> someone feels like contributing a max limit please base it on the master >>> branch, otherwise I suspect someone compiling about it did raise the >>> priority level a little. >>> >>> >>> Regards Ingela Erlang/OTP team - Ericsson AB >>> >>> >>> >>>> /Tony >>>> >>>> > On 5 nov 2014, at 13:08, Bogdan Andu wrote: >>>> > >>>> > >>>> > >>>> > Hi, >>>> > >>>> > I performed a series of test regarding the an Erlang SSL server. >>>> > >>>> > In this setup a major role is played by the table called >>>> 'ssl_otp_session_cache', and of course the processes using it. >>>> > >>>> > The problem is that the size of table increases constantly and, >>>> because an ets table does not automatically deallocate memory unless the >>>> object are deleted from that table, the size of table remains the same even >>>> if there no ssl connections to server. >>>> > >>>> > For example, with a single client running 'ad infinitum' the table >>>> increases at a rate of 5 MBytes/hour. In 12 hours there are allocate around >>>> 60 MB of memory only for this table. >>>> > >>>> > Some info about this: >>>> > $ erl -sname console@REDACTED -remsh n1@REDACTED -setcookie operator >>>> > Erlang/OTP 17 [erts-6.2] [source] [64-bit] [smp:2:2] >>>> [async-threads:10] [kernel-poll:false] >>>> > >>>> > Eshell V6.2 (abort with ^G) >>>> > >>>> > >>>> > (n1@REDACTED)1> ets:i(). >>>> > id name type size mem owner >>>> > >>>> ---------------------------------------------------------------------------- >>>> > 12 cookies set 0 291 auth >>>> > 4111 code set 410 26132 code_server >>>> > 8208 code_names set 58 7459 code_server >>>> > 12307 httpc_manager__session_cookie_db bag 0 291 >>>> httpc_manager >>>> > 16404 ssl_otp_cacertificate_db set 0 291 >>>> ssl_manager >>>> > 20501 ssl_otp_ca_file_ref set 0 291 ssl_manager >>>> > 24598 ssl_otp_pem_cache set 3 360 ssl_manager >>>> > 28695 ssl_otp_session_cache ordered_set 138057 8421893 >>>> ssl_manager >>>> > 32797 dets duplicate_bag 2 308 dets >>>> > 40990 ign_requests set 0 291 >>>> inet_gethost_native >>>> > 45087 ign_req_index set 0 291 >>>> inet_gethost_native >>>> > 2261635104 shell_records ordered_set 0 81 >>>> <0.30638.37> >>>> > ac_tab ac_tab set 33 2216 >>>> application_controller >>>> > code_map code_map set 100 2791 <0.72.0> >>>> > config config set 12 892 <0.72.0> >>>> > dets_owners dets_owners set 1 298 dets >>>> > dets_registry dets_registry set 1 299 dets >>>> > file_io_servers file_io_servers set 1 344 file_server_2 >>>> > global_locks global_locks set 0 291 >>>> global_name_server >>>> > global_names global_names set 0 291 >>>> global_name_server >>>> > global_names_ext global_names_ext set 0 291 >>>> global_name_server >>>> > global_pid_ids global_pid_ids bag 0 291 >>>> global_name_server >>>> > global_pid_names global_pid_names bag 0 291 >>>> global_name_server >>>> > httpc_manager__handler_db httpc_manager__handler_db set 0 >>>> 291 httpc_manager >>>> > httpc_manager__session_db httpc_manager__session_db set 0 >>>> 291 httpc_manager >>>> > inet_cache inet_cache bag 0 291 inet_db >>>> > inet_db inet_db set 29 600 inet_db >>>> > inet_hosts_byaddr inet_hosts_byaddr bag 0 291 inet_db >>>> > inet_hosts_byname inet_hosts_byname bag 0 291 inet_db >>>> > inet_hosts_file_byaddr inet_hosts_file_byaddr bag 0 291 >>>> inet_db >>>> > inet_hosts_file_byname inet_hosts_file_byname bag 0 291 >>>> inet_db >>>> > models models set 3 28952 <0.72.0> >>>> > sys_dist sys_dist set 1 334 net_kernel >>>> > ok >>>> > (n1@REDACTED)5> ets:info(ssl_otp_session_cache). >>>> > undefined >>>> > (n1@REDACTED)7> (8421893*8)/1024. >>>> > 65796.0390625 >>>> > (n1@REDACTED)8> memory(). >>>> > [{total,92699464}, >>>> > {processes,8964000}, >>>> > {processes_used,8963152}, >>>> > {system,83735464}, >>>> > {atom,429569}, >>>> > {atom_used,421768}, >>>> > {binary,199040}, >>>> > {code,10411520}, >>>> > {ets,69163032}] >>>> > >>>> > The memory allocated to table 'ssl_otp_session_cache' is roughly 64 >>>> MB in 12 hours. >>>> > >>>> > On an OpenBSD platform such process gets killed immediately it hits >>>> some memory and/or CPU limits. >>>> > >>>> > To make this test on OpenBSD I had to put 'infinit' to memory, >>>> otherwise the Erlang VM would be killed. >>>> > >>>> > How can one control , tweak or configure this table such that it does >>>> not accumulate such data at such high rate. >>>> > >>>> > I seems the table being created private, and there is no way to >>>> ets:delete_all_objects/1 from table manually. >>>> > >>>> > I know that this table caches some SSL data related to clients, but >>>> the client has the same IP address, >>>> > and I wonder why is neccesary to store a lot of SSL connection info >>>> about the same client when only the ephemeral peer port >>>> > differs? >>>> > >>>> > How the size of this table can be held in reasonable limits and the >>>> rate it's size increases ? >>>> > >>>> > Please if somebody shed some light on these issues. >>>> > >>>> > Thank you, >>>> > >>>> > Bogdan >>>> > >>>> > >>>> > _______________________________________________ >>>> > erlang-questions mailing list >>>> > erlang-questions@REDACTED >>>> > http://erlang.org/mailman/listinfo/erlang-questions >>>> >>>> "Installing applications can lead to corruption over time. Applications >>>> gradually write over each other's libraries, partial upgrades occur, user >>>> and system errors happen, and minute changes may be unnoticeable and >>>> difficult to fix" >>>> >>>> >>>> >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From egil@REDACTED Thu Nov 6 15:27:09 2014 From: egil@REDACTED (=?windows-1252?Q?Bj=F6rn-Egil_Dahlberg?=) Date: Thu, 6 Nov 2014 15:27:09 +0100 Subject: [erlang-questions] erlang:restart_timer In-Reply-To: References: Message-ID: <545B853D.7060805@erlang.org> Hi! Ah, a new BIF .. third time's a charm? =) Rickard is currently reworking the timer implementation so it is less problematic. The current implementation has some issues with scalability when a system has a lot of timers. The upcoming implementation can conceptually be thought of as processes with asynchronous signals sent between them. This includes the timer wheel as process (conceptually). I'm just mentioning this because a bif such as erlang:restart_timer/2 would be affected by this rewrite. In the current timer implementation it is trivial to implement restart_timer/2 and we see the function as a beneficial addition to the existing timer API. Rickard promised to fix everything in the rewrite. =) +1 // Bj?rn-Egil On 2014-11-05 14:50, Tony Rogvall wrote: > Hi! > > >From time to time I implement protocols and services that need watch dogs and/or retransmit timers. > I will normally implement this using a BIF timer: > > WdtTimer = erlang:start_timer(Time, self(), wdt) > > This will send a message {timeout,WdtTimer, wdt} after Time milliseconds. > When using a watchdog the code will normally try to avoid that the wdt is ever sent. > The idea is that if you get the wdt message you are to late and need to crash / restart or something. > So after som executing some "watched" code you want to re-arm the watch dog (feed it) > This is normally done by: > > erlang:cancel_timer(WdtTimer), > NewWdtTimer = erlang:start_timer(Time, self(), wdt) > > If you match the {timeout,WdtTimer,wdt} message and ignore {timeout,OldWdtTimer,wdt} messages, > you could skip the cancel_timer. But that will generate plenty of waste. > > Protocols using retransmit timeouts also use a scheme like the above. > > I propose a new BIF: > > erlang:restart_timer(Ref, Time) -> false | RemainingTime > > This will restart a time if it is active and return the time that remained when it was restarted. > If restart_timer return false either the timer has already time out or the reference is not a timer. > > The benefit is a speedup with at least 2.5 times. comparing: > > erlang:restart_timer(Ref, Time) > vs > erlang:cancel_timer(Ref) > Ref1 = erlang:start_timer(Time, self(), Message) > > The cost saving is that restart_timer preserve already stored messages. And also internal timer structures are > preserved. Resulting in less overhead. > > The interface is simple and do not require a new reference every restart, this could also save the time > and heap updating a server state. > > I have an implementation that perform well, and could probably be optimized even further. > > What about it? I'd like to hear from people actually implementing protocols, not only from the "normal" responders. > > > /Tony > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > From sean@REDACTED Thu Nov 6 15:28:19 2014 From: sean@REDACTED (Sean Cribbs) Date: Thu, 6 Nov 2014 08:28:19 -0600 Subject: [erlang-questions] capn proto In-Reply-To: References: Message-ID: Note that cap'n proto is NOT Protocol Buffers, and Riak does not use it. You cannot use ecapnp to talk to Riak. On Thu, Nov 6, 2014 at 6:36 AM, Sergej Jure?ko wrote: > Well there already is: https://github.com/kaos/ecapnp > > > Sergej > > On Thu, Nov 6, 2014 at 1:24 PM, Gokhan Boranalp wrote: > >> I am just wondering anyone else interested to implement CapnProto? >> http://kentonv.github.io/capnproto/ >> Especially, in regard of protocol section for Riak ;-) >> >> -- >> BR, >> \|/ Kunthar >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -- Sean Cribbs Sr. Software Engineer Basho Technologies, Inc. http://basho.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ingela.andin@REDACTED Thu Nov 6 15:40:09 2014 From: ingela.andin@REDACTED (Ingela Andin) Date: Thu, 6 Nov 2014 15:40:09 +0100 Subject: [erlang-questions] Removing SSL v3 support from the ssl module In-Reply-To: <16095605.Z2g7XaJzps@moltowork> References: <20141015091030.GA52663@k2r.org> <368069533.74861.1413365696097.JavaMail.zimbra@tpip.net> <16095605.Z2g7XaJzps@moltowork> Message-ID: Hi! 2014-10-22 17:29 GMT+02:00 Vincent de Phily < vincent.dephily@REDACTED>: > On Wednesday 15 October 2014 19:41:23 Steve Vinoski wrote: > > On Wed, Oct 15, 2014 at 5:34 AM, Andreas Schultz > wrote: > > > Hi, > > > > > > ----- On 15 Oct, 2014, at 11:10, Kenji Rikitake kenji@REDACTED wrote: > > > > I'd be glad if how to remove SSL v3 support from OTP ssl module is > > > > provided by the OTP Team, to prevent getting trapped into the POODLE > > > > bug. (I think it won't be that hard, regarding what I've found from > the > > > > ssl module source code. The keyword atom is "sslv3".) > > > > > > Add {versions, ['tlsv1.2', 'tls1.1', 'tls1']} to your SSL options to > > > restrict > > > the version choice. > > > > Slight correction: {versions, ['tlsv1.2', 'tlsv1.1', 'tlsv1']} > > I suggest going with > > proplists:get_value(available,ssl:versions()) -- [sslv3] > > to future-proof your code a bit. I'm not sure what the difference between > 'supported' and 'available' is (a clarification in the docs would be nice), > neither of them seem to be affected by the command-line argument to > restrict > versions. > > It was proably a quoting problem, or that you did not start the ssl application before callinge ssl:versions(). ! > erl -ssl protocol_version '[tlsv1]' Erlang/OTP 18 [DEVELOPMENT] [erts-7.0] [source-7ed6eb5] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] Eshell V7.0 (abort with ^G) 1> ssl:start(). ok 2> ssl:versions(). [{ssl_app,"5.3.7"}, {supported,[tlsv1]}, {available,['tlsv1.2','tlsv1.1',tlsv1,sslv3]}] > erl -ssl protocol_version "['tlsv1.2', 'tlsv1.1']" Erlang/OTP 18 [DEVELOPMENT] [erts-7.0] [source-7ed6eb5] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] Eshell V7.0 (abort with ^G) 1> ssl:start(). ok 2> ssl:versions(). [{ssl_app,"5.3.7"}, {supported,['tlsv1.2','tlsv1.1']}, {available,['tlsv1.2','tlsv1.1',tlsv1,sslv3]}] 3> Maybe we should call them configured_default (supported) and system_default (available) ? We will think about it. We might exclude sslv3 from the system default and make it only available through configuration. Regards Ingela Erlang/OTP team - Ericsson AB > -- > Vincent de Phily > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ingela.andin@REDACTED Thu Nov 6 15:51:25 2014 From: ingela.andin@REDACTED (Ingela Andin) Date: Thu, 6 Nov 2014 15:51:25 +0100 Subject: [erlang-questions] 'ssl_otp_session_cache' ets table In-Reply-To: References: Message-ID: Hi! 2014-11-06 14:39 GMT+01:00 Bogdan Andu : > Hi, > > i have tried to pass to execute th following forms of commands: > > erl -ssl session_lifetime 60 > > erl -ssl session_lifetime '60' > > erl -ssl session_lifetime '[60]' > > > in order to set a ssl session_lifetime of 60 seconds. > > in init of ssl_manager , after session_lifetime + 5 seconds a process is > created to validate or invalidate sessions. > So, after 65 seconds a foldl is called on sessions table to check sessions > after formula: > Now - TimeStamp < LifeTime > > However, even after I stop the client, the table 'ssl_otp_session_cache' > 's size remains the same. > > > After that once every ?CLEAN_SESSION_DB which is 60 seconds , the table > is sweeped to delete any expired sessions, > > but also nothing happens. > > There is also a delay in the actual deletion. In the first sweep the sessions will only be invalidated as there may already be spawned connection handlers that needs to read the session data before we may delete it. It is a performance trade off. Regards Ingela Erlang/OTP team - Ericsson AB > > On Thu, Nov 6, 2014 at 12:28 PM, Ingela Andin > wrote: > >> Hi! >> >> >> 2014-11-06 8:53 GMT+01:00 Bogdan Andu : >> >>> Hi, >>> >>> on my production servers I have relayd (on OpenBSD) daemon as a reverse >>> proxy to some webservers >>> where one can fine tune some connection parameters, as well as some ssl >>> parameters. >>> >>> I give a snippet from a relayd.conf configuration file on one of my >>> production server: >>> >>> ............. >>> # >>> # Relay and protocol for HTTP layer 7 loadbalancing and SSL acceleration >>> # >>> http protocol www_ssl_prot { >>> header append "$REMOTE_ADDR" to "X-Forwarded-For" >>> header append "$SERVER_ADDR:$SERVER_PORT" to "X-Forwarded-By" >>> header change "Connection" to "close" >>> >>> response header change "Server" to "Apache 0.1" >>> >>> # Various TCP performance options >>> tcp { nodelay, sack, socket buffer 65536, backlog 128 } >>> >>> ssl { no sslv2, no sslv3, tlsv1, ciphers "HIGH" } >>> ssl session cache disable >>> } >>> >>> the last directive tells relayd not to use ssl cache. >>> >>> This configuration is working for years and relayd was restarted once by >>> accident - my fault. >>> >>> SO, y question is: >>> >>> can we have this configurable in Erlang, in other words, we might be >>> able to start an erlang vm such as: >>> >>> erl -ssl session_cache 'disable' -name x@REDACTED .... >>> >>> The ssl option session_cache can be set to disabled by default and can >>> take values either disable or enabled. >>> >>> >>> >> You can already disable the reuse of the sessions using the server option *{reuse_sessions, >> boolean()}* which default to true. >> The thing we plan to do is to have a configurable limit on the table size >> when sessions are reused. >> >> Regards Ingela Erlang/OTP team Ericsson AB >> >> >> >> >> >> >>> Just to let you know... >>> >>> The statistics of a node running from yesterday when I opened this >>> thread of discussion, using a single client: >>> >>> Node: 'n1@REDACTED' (Connected) (17/6.2) unix (openbsd 5.4.0) CPU:2 SMP >>> +A:10 >>> Time: local time 09:48:48, up for 000:20:27:49, 0ms latency, >>> Processes: total 681 (RQ 0) at 610275 RpI using 11526.4k (11805.8k >>> allocated) >>> Memory: Sys 94835.5k, Atom 407.7k/419.5k, Bin 176.4k, Code 9934.7k, Ets >>> 80739.2k >>> >>> ........... >>> >>> So, can be this made configurable? >>> >>> >>> >>> On Wed, Nov 5, 2014 at 8:52 PM, Ingela Andin >>> wrote: >>> >>>> Hi! >>>> >>>> 2014-11-05 14:15 GMT+01:00 Tony Rogvall : >>>> >>>>> I think the ssl session times is the problem here, and the lack of a >>>>> maximum size. >>>>> >>>>> You can change the session time in the ssl environment: >>>>> session_lifetime >>>>> The default is set to 24 hours (in seconds) (if I read it correctly, >>>>> in ssl_manager.erl) >>>>> >>>> >>>> Default values are always hard. It is the maximum recommended time for >>>> a session to live >>>> according to the spec. >>>> >>>> >>>>> I guess that a session_cache_size could be a nice thing to have, >>>>> limiting the growth of the session cache. >>>>> >>>>> >>>> Yes I agree. >>>> >>>> >>>>> In other words you have to estimate the life time of your clients and >>>>> try to find a reasonable session_lifetime to match that, without >>>>> blowing up >>>>> the system. >>>>> >>>>> Maybe the ssl_session_cache_api could be used to implement a strategy >>>>> with a max size >>>>> cache. Retire session least recently used, while performing the update? >>>>> There is a time_stamp in the session that that could be used for this >>>>> purpose. >>>>> >>>>> OTP: Why is a fixed limit not implemented in the standard >>>>> ssl_session_cache? >>>>> Could this be a target for DOS attacks? >>>>> >>>> >>>> >>>> We are aware of the problem, and it is on our todo list. One reason it >>>> has not had top priority is that on the server side there are often other >>>> mechanisms >>>> like firewalls and webbserver settings that limits the problem. And >>>> the reason why it was not implemented in the first place is that >>>> implementations by nature >>>> are iterative and at first you are faced with a lot of bigger problems >>>> to solve and then you need to iterate and fine tune and fix things that you >>>> now have a better understanding of. >>>> >>>> In current master there is a change to the session table that limits >>>> the growth on the client side if the client behaves inappropriate. It also >>>> splits the session table into >>>> a server and a client table which is a better implementation as the >>>> same Erlang node can be both a client and a server at the same time. So if >>>> someone feels like contributing a max limit please base it on the master >>>> branch, otherwise I suspect someone compiling about it did raise the >>>> priority level a little. >>>> >>>> >>>> Regards Ingela Erlang/OTP team - Ericsson AB >>>> >>>> >>>> >>>>> /Tony >>>>> >>>>> > On 5 nov 2014, at 13:08, Bogdan Andu wrote: >>>>> > >>>>> > >>>>> > >>>>> > Hi, >>>>> > >>>>> > I performed a series of test regarding the an Erlang SSL server. >>>>> > >>>>> > In this setup a major role is played by the table called >>>>> 'ssl_otp_session_cache', and of course the processes using it. >>>>> > >>>>> > The problem is that the size of table increases constantly and, >>>>> because an ets table does not automatically deallocate memory unless the >>>>> object are deleted from that table, the size of table remains the same even >>>>> if there no ssl connections to server. >>>>> > >>>>> > For example, with a single client running 'ad infinitum' the table >>>>> increases at a rate of 5 MBytes/hour. In 12 hours there are allocate around >>>>> 60 MB of memory only for this table. >>>>> > >>>>> > Some info about this: >>>>> > $ erl -sname console@REDACTED -remsh n1@REDACTED -setcookie operator >>>>> > Erlang/OTP 17 [erts-6.2] [source] [64-bit] [smp:2:2] >>>>> [async-threads:10] [kernel-poll:false] >>>>> > >>>>> > Eshell V6.2 (abort with ^G) >>>>> > >>>>> > >>>>> > (n1@REDACTED)1> ets:i(). >>>>> > id name type size mem owner >>>>> > >>>>> ---------------------------------------------------------------------------- >>>>> > 12 cookies set 0 291 auth >>>>> > 4111 code set 410 26132 code_server >>>>> > 8208 code_names set 58 7459 code_server >>>>> > 12307 httpc_manager__session_cookie_db bag 0 291 >>>>> httpc_manager >>>>> > 16404 ssl_otp_cacertificate_db set 0 291 >>>>> ssl_manager >>>>> > 20501 ssl_otp_ca_file_ref set 0 291 >>>>> ssl_manager >>>>> > 24598 ssl_otp_pem_cache set 3 360 ssl_manager >>>>> > 28695 ssl_otp_session_cache ordered_set 138057 8421893 >>>>> ssl_manager >>>>> > 32797 dets duplicate_bag 2 308 dets >>>>> > 40990 ign_requests set 0 291 >>>>> inet_gethost_native >>>>> > 45087 ign_req_index set 0 291 >>>>> inet_gethost_native >>>>> > 2261635104 shell_records ordered_set 0 81 >>>>> <0.30638.37> >>>>> > ac_tab ac_tab set 33 2216 >>>>> application_controller >>>>> > code_map code_map set 100 2791 <0.72.0> >>>>> > config config set 12 892 <0.72.0> >>>>> > dets_owners dets_owners set 1 298 dets >>>>> > dets_registry dets_registry set 1 299 dets >>>>> > file_io_servers file_io_servers set 1 344 >>>>> file_server_2 >>>>> > global_locks global_locks set 0 291 >>>>> global_name_server >>>>> > global_names global_names set 0 291 >>>>> global_name_server >>>>> > global_names_ext global_names_ext set 0 291 >>>>> global_name_server >>>>> > global_pid_ids global_pid_ids bag 0 291 >>>>> global_name_server >>>>> > global_pid_names global_pid_names bag 0 291 >>>>> global_name_server >>>>> > httpc_manager__handler_db httpc_manager__handler_db set 0 >>>>> 291 httpc_manager >>>>> > httpc_manager__session_db httpc_manager__session_db set 0 >>>>> 291 httpc_manager >>>>> > inet_cache inet_cache bag 0 291 inet_db >>>>> > inet_db inet_db set 29 600 inet_db >>>>> > inet_hosts_byaddr inet_hosts_byaddr bag 0 291 inet_db >>>>> > inet_hosts_byname inet_hosts_byname bag 0 291 inet_db >>>>> > inet_hosts_file_byaddr inet_hosts_file_byaddr bag 0 291 >>>>> inet_db >>>>> > inet_hosts_file_byname inet_hosts_file_byname bag 0 291 >>>>> inet_db >>>>> > models models set 3 28952 <0.72.0> >>>>> > sys_dist sys_dist set 1 334 net_kernel >>>>> > ok >>>>> > (n1@REDACTED)5> ets:info(ssl_otp_session_cache). >>>>> > undefined >>>>> > (n1@REDACTED)7> (8421893*8)/1024. >>>>> > 65796.0390625 >>>>> > (n1@REDACTED)8> memory(). >>>>> > [{total,92699464}, >>>>> > {processes,8964000}, >>>>> > {processes_used,8963152}, >>>>> > {system,83735464}, >>>>> > {atom,429569}, >>>>> > {atom_used,421768}, >>>>> > {binary,199040}, >>>>> > {code,10411520}, >>>>> > {ets,69163032}] >>>>> > >>>>> > The memory allocated to table 'ssl_otp_session_cache' is roughly 64 >>>>> MB in 12 hours. >>>>> > >>>>> > On an OpenBSD platform such process gets killed immediately it hits >>>>> some memory and/or CPU limits. >>>>> > >>>>> > To make this test on OpenBSD I had to put 'infinit' to memory, >>>>> otherwise the Erlang VM would be killed. >>>>> > >>>>> > How can one control , tweak or configure this table such that it >>>>> does not accumulate such data at such high rate. >>>>> > >>>>> > I seems the table being created private, and there is no way to >>>>> ets:delete_all_objects/1 from table manually. >>>>> > >>>>> > I know that this table caches some SSL data related to clients, but >>>>> the client has the same IP address, >>>>> > and I wonder why is neccesary to store a lot of SSL connection info >>>>> about the same client when only the ephemeral peer port >>>>> > differs? >>>>> > >>>>> > How the size of this table can be held in reasonable limits and the >>>>> rate it's size increases ? >>>>> > >>>>> > Please if somebody shed some light on these issues. >>>>> > >>>>> > Thank you, >>>>> > >>>>> > Bogdan >>>>> > >>>>> > >>>>> > _______________________________________________ >>>>> > erlang-questions mailing list >>>>> > erlang-questions@REDACTED >>>>> > http://erlang.org/mailman/listinfo/erlang-questions >>>>> >>>>> "Installing applications can lead to corruption over time. >>>>> Applications gradually write over each other's libraries, partial upgrades >>>>> occur, user and system errors happen, and minute changes may be >>>>> unnoticeable and difficult to fix" >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> erlang-questions mailing list >>>>> erlang-questions@REDACTED >>>>> http://erlang.org/mailman/listinfo/erlang-questions >>>>> >>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bog495@REDACTED Thu Nov 6 17:20:50 2014 From: bog495@REDACTED (Bogdan Andu) Date: Thu, 6 Nov 2014 18:20:50 +0200 Subject: [erlang-questions] 'ssl_otp_session_cache' ets table In-Reply-To: References: Message-ID: yes, in function ssl_manager:invalidate_session/4, the call erlang:send_after(delay_time(), self(), {delayed_clean_session, Key}), does the actual deletion of session from cache where delay_time/0 function returns ?CLEAN_SESSION_DB by default, which is 60 seconds On Thu, Nov 6, 2014 at 4:51 PM, Ingela Andin wrote: > Hi! > > > 2014-11-06 14:39 GMT+01:00 Bogdan Andu : > >> Hi, >> >> i have tried to pass to execute th following forms of commands: >> >> erl -ssl session_lifetime 60 >> >> erl -ssl session_lifetime '60' >> >> erl -ssl session_lifetime '[60]' >> >> >> in order to set a ssl session_lifetime of 60 seconds. >> >> in init of ssl_manager , after session_lifetime + 5 seconds a process is >> created to validate or invalidate sessions. >> So, after 65 seconds a foldl is called on sessions table to check >> sessions after formula: >> Now - TimeStamp < LifeTime >> >> However, even after I stop the client, the table 'ssl_otp_session_cache' >> 's size remains the same. >> >> >> After that once every ?CLEAN_SESSION_DB which is 60 seconds , the table >> is sweeped to delete any expired sessions, >> >> but also nothing happens. >> >> > There is also a delay in the actual deletion. In the first sweep the > sessions will only be invalidated as there may already be > spawned connection handlers that needs to read the session data before we > may delete it. It is a performance trade off. > > Regards Ingela Erlang/OTP team - Ericsson AB > > > >> >> On Thu, Nov 6, 2014 at 12:28 PM, Ingela Andin >> wrote: >> >>> Hi! >>> >>> >>> 2014-11-06 8:53 GMT+01:00 Bogdan Andu : >>> >>>> Hi, >>>> >>>> on my production servers I have relayd (on OpenBSD) daemon as a reverse >>>> proxy to some webservers >>>> where one can fine tune some connection parameters, as well as some ssl >>>> parameters. >>>> >>>> I give a snippet from a relayd.conf configuration file on one of my >>>> production server: >>>> >>>> ............. >>>> # >>>> # Relay and protocol for HTTP layer 7 loadbalancing and SSL acceleration >>>> # >>>> http protocol www_ssl_prot { >>>> header append "$REMOTE_ADDR" to "X-Forwarded-For" >>>> header append "$SERVER_ADDR:$SERVER_PORT" to "X-Forwarded-By" >>>> header change "Connection" to "close" >>>> >>>> response header change "Server" to "Apache 0.1" >>>> >>>> # Various TCP performance options >>>> tcp { nodelay, sack, socket buffer 65536, backlog 128 } >>>> >>>> ssl { no sslv2, no sslv3, tlsv1, ciphers "HIGH" } >>>> ssl session cache disable >>>> } >>>> >>>> the last directive tells relayd not to use ssl cache. >>>> >>>> This configuration is working for years and relayd was restarted once >>>> by accident - my fault. >>>> >>>> SO, y question is: >>>> >>>> can we have this configurable in Erlang, in other words, we might be >>>> able to start an erlang vm such as: >>>> >>>> erl -ssl session_cache 'disable' -name x@REDACTED .... >>>> >>>> The ssl option session_cache can be set to disabled by default and can >>>> take values either disable or enabled. >>>> >>>> >>>> >>> You can already disable the reuse of the sessions using the server >>> option *{reuse_sessions, boolean()}* which default to true. >>> The thing we plan to do is to have a configurable limit on the table >>> size when sessions are reused. >>> >>> Regards Ingela Erlang/OTP team Ericsson AB >>> >>> >>> >>> >>> >>> >>>> Just to let you know... >>>> >>>> The statistics of a node running from yesterday when I opened this >>>> thread of discussion, using a single client: >>>> >>>> Node: 'n1@REDACTED' (Connected) (17/6.2) unix (openbsd 5.4.0) CPU:2 >>>> SMP +A:10 >>>> Time: local time 09:48:48, up for 000:20:27:49, 0ms latency, >>>> Processes: total 681 (RQ 0) at 610275 RpI using 11526.4k (11805.8k >>>> allocated) >>>> Memory: Sys 94835.5k, Atom 407.7k/419.5k, Bin 176.4k, Code 9934.7k, Ets >>>> 80739.2k >>>> >>>> ........... >>>> >>>> So, can be this made configurable? >>>> >>>> >>>> >>>> On Wed, Nov 5, 2014 at 8:52 PM, Ingela Andin >>>> wrote: >>>> >>>>> Hi! >>>>> >>>>> 2014-11-05 14:15 GMT+01:00 Tony Rogvall : >>>>> >>>>>> I think the ssl session times is the problem here, and the lack of a >>>>>> maximum size. >>>>>> >>>>>> You can change the session time in the ssl environment: >>>>>> session_lifetime >>>>>> The default is set to 24 hours (in seconds) (if I read it correctly, >>>>>> in ssl_manager.erl) >>>>>> >>>>> >>>>> Default values are always hard. It is the maximum recommended time for >>>>> a session to live >>>>> according to the spec. >>>>> >>>>> >>>>>> I guess that a session_cache_size could be a nice thing to have, >>>>>> limiting the growth of the session cache. >>>>>> >>>>>> >>>>> Yes I agree. >>>>> >>>>> >>>>>> In other words you have to estimate the life time of your clients and >>>>>> try to find a reasonable session_lifetime to match that, without >>>>>> blowing up >>>>>> the system. >>>>>> >>>>>> Maybe the ssl_session_cache_api could be used to implement a strategy >>>>>> with a max size >>>>>> cache. Retire session least recently used, while performing the >>>>>> update? >>>>>> There is a time_stamp in the session that that could be used for this >>>>>> purpose. >>>>>> >>>>>> OTP: Why is a fixed limit not implemented in the standard >>>>>> ssl_session_cache? >>>>>> Could this be a target for DOS attacks? >>>>>> >>>>> >>>>> >>>>> We are aware of the problem, and it is on our todo list. One reason it >>>>> has not had top priority is that on the server side there are often other >>>>> mechanisms >>>>> like firewalls and webbserver settings that limits the problem. And >>>>> the reason why it was not implemented in the first place is that >>>>> implementations by nature >>>>> are iterative and at first you are faced with a lot of bigger problems >>>>> to solve and then you need to iterate and fine tune and fix things that you >>>>> now have a better understanding of. >>>>> >>>>> In current master there is a change to the session table that limits >>>>> the growth on the client side if the client behaves inappropriate. It also >>>>> splits the session table into >>>>> a server and a client table which is a better implementation as the >>>>> same Erlang node can be both a client and a server at the same time. So if >>>>> someone feels like contributing a max limit please base it on the master >>>>> branch, otherwise I suspect someone compiling about it did raise the >>>>> priority level a little. >>>>> >>>>> >>>>> Regards Ingela Erlang/OTP team - Ericsson AB >>>>> >>>>> >>>>> >>>>>> /Tony >>>>>> >>>>>> > On 5 nov 2014, at 13:08, Bogdan Andu wrote: >>>>>> > >>>>>> > >>>>>> > >>>>>> > Hi, >>>>>> > >>>>>> > I performed a series of test regarding the an Erlang SSL server. >>>>>> > >>>>>> > In this setup a major role is played by the table called >>>>>> 'ssl_otp_session_cache', and of course the processes using it. >>>>>> > >>>>>> > The problem is that the size of table increases constantly and, >>>>>> because an ets table does not automatically deallocate memory unless the >>>>>> object are deleted from that table, the size of table remains the same even >>>>>> if there no ssl connections to server. >>>>>> > >>>>>> > For example, with a single client running 'ad infinitum' the table >>>>>> increases at a rate of 5 MBytes/hour. In 12 hours there are allocate around >>>>>> 60 MB of memory only for this table. >>>>>> > >>>>>> > Some info about this: >>>>>> > $ erl -sname console@REDACTED -remsh n1@REDACTED -setcookie >>>>>> operator >>>>>> > Erlang/OTP 17 [erts-6.2] [source] [64-bit] [smp:2:2] >>>>>> [async-threads:10] [kernel-poll:false] >>>>>> > >>>>>> > Eshell V6.2 (abort with ^G) >>>>>> > >>>>>> > >>>>>> > (n1@REDACTED)1> ets:i(). >>>>>> > id name type size mem owner >>>>>> > >>>>>> ---------------------------------------------------------------------------- >>>>>> > 12 cookies set 0 291 auth >>>>>> > 4111 code set 410 26132 code_server >>>>>> > 8208 code_names set 58 7459 code_server >>>>>> > 12307 httpc_manager__session_cookie_db bag 0 291 >>>>>> httpc_manager >>>>>> > 16404 ssl_otp_cacertificate_db set 0 291 >>>>>> ssl_manager >>>>>> > 20501 ssl_otp_ca_file_ref set 0 291 >>>>>> ssl_manager >>>>>> > 24598 ssl_otp_pem_cache set 3 360 ssl_manager >>>>>> > 28695 ssl_otp_session_cache ordered_set 138057 8421893 >>>>>> ssl_manager >>>>>> > 32797 dets duplicate_bag 2 308 >>>>>> dets >>>>>> > 40990 ign_requests set 0 291 >>>>>> inet_gethost_native >>>>>> > 45087 ign_req_index set 0 291 >>>>>> inet_gethost_native >>>>>> > 2261635104 shell_records ordered_set 0 81 >>>>>> <0.30638.37> >>>>>> > ac_tab ac_tab set 33 2216 >>>>>> application_controller >>>>>> > code_map code_map set 100 2791 <0.72.0> >>>>>> > config config set 12 892 <0.72.0> >>>>>> > dets_owners dets_owners set 1 298 dets >>>>>> > dets_registry dets_registry set 1 299 dets >>>>>> > file_io_servers file_io_servers set 1 344 >>>>>> file_server_2 >>>>>> > global_locks global_locks set 0 291 >>>>>> global_name_server >>>>>> > global_names global_names set 0 291 >>>>>> global_name_server >>>>>> > global_names_ext global_names_ext set 0 291 >>>>>> global_name_server >>>>>> > global_pid_ids global_pid_ids bag 0 291 >>>>>> global_name_server >>>>>> > global_pid_names global_pid_names bag 0 291 >>>>>> global_name_server >>>>>> > httpc_manager__handler_db httpc_manager__handler_db set 0 >>>>>> 291 httpc_manager >>>>>> > httpc_manager__session_db httpc_manager__session_db set 0 >>>>>> 291 httpc_manager >>>>>> > inet_cache inet_cache bag 0 291 inet_db >>>>>> > inet_db inet_db set 29 600 inet_db >>>>>> > inet_hosts_byaddr inet_hosts_byaddr bag 0 291 inet_db >>>>>> > inet_hosts_byname inet_hosts_byname bag 0 291 inet_db >>>>>> > inet_hosts_file_byaddr inet_hosts_file_byaddr bag 0 291 >>>>>> inet_db >>>>>> > inet_hosts_file_byname inet_hosts_file_byname bag 0 291 >>>>>> inet_db >>>>>> > models models set 3 28952 <0.72.0> >>>>>> > sys_dist sys_dist set 1 334 net_kernel >>>>>> > ok >>>>>> > (n1@REDACTED)5> ets:info(ssl_otp_session_cache). >>>>>> > undefined >>>>>> > (n1@REDACTED)7> (8421893*8)/1024. >>>>>> > 65796.0390625 >>>>>> > (n1@REDACTED)8> memory(). >>>>>> > [{total,92699464}, >>>>>> > {processes,8964000}, >>>>>> > {processes_used,8963152}, >>>>>> > {system,83735464}, >>>>>> > {atom,429569}, >>>>>> > {atom_used,421768}, >>>>>> > {binary,199040}, >>>>>> > {code,10411520}, >>>>>> > {ets,69163032}] >>>>>> > >>>>>> > The memory allocated to table 'ssl_otp_session_cache' is roughly 64 >>>>>> MB in 12 hours. >>>>>> > >>>>>> > On an OpenBSD platform such process gets killed immediately it hits >>>>>> some memory and/or CPU limits. >>>>>> > >>>>>> > To make this test on OpenBSD I had to put 'infinit' to memory, >>>>>> otherwise the Erlang VM would be killed. >>>>>> > >>>>>> > How can one control , tweak or configure this table such that it >>>>>> does not accumulate such data at such high rate. >>>>>> > >>>>>> > I seems the table being created private, and there is no way to >>>>>> ets:delete_all_objects/1 from table manually. >>>>>> > >>>>>> > I know that this table caches some SSL data related to clients, but >>>>>> the client has the same IP address, >>>>>> > and I wonder why is neccesary to store a lot of SSL connection info >>>>>> about the same client when only the ephemeral peer port >>>>>> > differs? >>>>>> > >>>>>> > How the size of this table can be held in reasonable limits and the >>>>>> rate it's size increases ? >>>>>> > >>>>>> > Please if somebody shed some light on these issues. >>>>>> > >>>>>> > Thank you, >>>>>> > >>>>>> > Bogdan >>>>>> > >>>>>> > >>>>>> > _______________________________________________ >>>>>> > erlang-questions mailing list >>>>>> > erlang-questions@REDACTED >>>>>> > http://erlang.org/mailman/listinfo/erlang-questions >>>>>> >>>>>> "Installing applications can lead to corruption over time. >>>>>> Applications gradually write over each other's libraries, partial upgrades >>>>>> occur, user and system errors happen, and minute changes may be >>>>>> unnoticeable and difficult to fix" >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> erlang-questions mailing list >>>>>> erlang-questions@REDACTED >>>>>> http://erlang.org/mailman/listinfo/erlang-questions >>>>>> >>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From community-manager@REDACTED Thu Nov 6 17:21:40 2014 From: community-manager@REDACTED (Bruce Yinhe) Date: Thu, 6 Nov 2014 17:21:40 +0100 Subject: [erlang-questions] New Erlang job openings Message-ID: Hi, We have currently 27 Erlang-related job openings. https://erlangcentral.org/jobs To subscribe for weekly updates on new Erlang jobs - Register on Erlang Central. https://erlangcentral.org/login/?action=register To post an Erlang job free of charge: https://erlangcentral.org/jobs/add *Europe:* Software Engineer, Functional Programming at Klarna - Stockholm, Sweden https://erlangcentral.org/software-engineer-functional-programming/ Senior Erlang Software Engineer at Cisco - Stockholm, Sweden https://erlangcentral.org/senior-erlang-software-engineer-cisco-tail-f/ Senior Server Developer at Jongla - Helsinki, Finland https://erlangcentral.org/senior-server-developer-jongla/ Backend Engineer at Centralway - Zurich, Switzerland https://erlangcentral.org/backend-engineer-centralway/ Senior Software Engineer ? Core Tech at DemonWare - Dublin, Ireland https://erlangcentral.org/senior-software-engineer-core-tech-demonware/ Polyglot Engineer (RabbitMQ) at LShift - London, UK https://erlangcentral.org/polyglot-engineer-lshift/ Distributed Systems Developer (enterprise-scale) at Computer People - Staffordshire, England https://erlangcentral.org/distributed-systems-developer-enterprise-scale-computer-people/ Senior Erlang Developer at Darwin Recruitment - UK https://erlangcentral.org/senior-erlang-developer-darwin-recruitment/ Software developer at Oxford Knight - London, UK https://erlangcentral.org/software-developer-java-c-scala-clojure-erlang-functional-programming-london/ Erlang Consultants ? multiple locations at Erlang Solutions - London, Budapest, Krakow, Stockholm, Copenhagen https://erlangcentral.org/erlang-developers-wanted-multiple-locations/ *Americas:* Erlang Developer at Sqor Sports - San Francisco, CA https://erlangcentral.org/erlang-developer-sqor/ Erlang Developer at Inaka - Buenos Aires https://erlangcentral.org/erlang-developer-inaka/ Infrastructure Engineer at Getaround - San Francisco, CA https://erlangcentral.org/infrastructure-engineer-getaround/ Erlang/OTP Expert at A10 Networks - San Jose, CA https://erlangcentral.org/erlangotp-expert-a10networks/ Erlang Developer at A10 Networks - San Jose, CA https://erlangcentral.org/erlang-developer-1-a10networks/ Senior Software Engineer, Erlang Specialist at AdRoll - San Francisco, CA https://erlangcentral.org/senior-software-engineer-erlang-specialist-adroll/ Senior Erlang Engineer at Machine Zone - Palo Alto, CA https://erlangcentral.org/senior-erlang-engineer-machine-zone/ Erlang Back End Engineer - Santa Monica, CA https://erlangcentral.org/erlang-developer-tigertext/ Erlang Engineer at CyberCoders - Sunnyvale, CA; Mountain View, CA https://erlangcentral.org/cybercoders-healthcare-mobile-powerhouse-seeks-erlang-engineer- Principal Erlang Developer at CyberCoders - Redwood City, CA https://erlangcentral.org/cybercoders-principal-erlang-developer-ground-breaking-cloud-start-up/ Backend Engineer ? Relo Offered at CyberCoders - Los Angeles, CA; Santa Monica, CA https://erlangcentral.org/backend-engineer-relo-offered-erlang-java-otp-python-rubyonrails-elixir-cybercoders/ Erlang Developer at True North - San Antonio, TX https://erlangcentral.org/erlang-developer-true-north/ Elixir / Erlang Developer at Vitamin Talent - Austin, Texas (open to remote) https://erlangcentral.org/elixir-erlang-developer-vitamin-talent/ Platform Engineer ? Erlang / Java at GMS Advisors - New York, NY https://erlangcentral.org/platform-engineer-erlang-java-gms-advisors/ Cheers, Bruce Yinhe erlangcentral.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From charleshixsn@REDACTED Thu Nov 6 19:56:27 2014 From: charleshixsn@REDACTED (Charles Hixson) Date: Thu, 06 Nov 2014 10:56:27 -0800 Subject: [erlang-questions] newbie question Message-ID: <545BC45B.9020207@earthlink.net> Given the routine: -module(chunks). -export([chunks/1]). chunks(str)-> chunks(str, []). chunks("", lst)-> list:reverse(lst); chunks(str, lst)-> P1 = "^([[:alnum:]]+|[[:space:]]+|[[:punct:]]+|[[:cntrl:]]+|.)", [[], M1, T1] = re:split(str, P1, [{return, list}]), [M1 | chunks(T1, P1)]. and the shell: 47> c(chunks). {ok,chunks} Why do I get: 49> chunks:chunks("Able was I ere I saw Elba."). ** exception error: no function clause matching chunks:chunks("Able was I ere I saw Elba.") (chunks.erl, line 15) (Well, forget the like 15. I took out a bunch of comments.) From james@REDACTED Thu Nov 6 20:09:03 2014 From: james@REDACTED (James Aimonetti) Date: Thu, 6 Nov 2014 11:09:03 -0800 Subject: [erlang-questions] newbie question In-Reply-To: <545BC45B.9020207@earthlink.net> References: <545BC45B.9020207@earthlink.net> Message-ID: <545BC74F.5090508@2600hz.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 str should be Str. Variables start with a capital letter. Le 2014-11-06 10:56, Charles Hixson a ?crit : > Given the routine: -module(chunks). -export([chunks/1]). > chunks(str)-> chunks(str, []). > > chunks("", lst)-> list:reverse(lst); chunks(str, lst)-> P1 = > "^([[:alnum:]]+|[[:space:]]+|[[:punct:]]+|[[:cntrl:]]+|.)", [[], > M1, T1] = re:split(str, P1, [{return, list}]), [M1 | chunks(T1, > P1)]. > > and the shell: 47> c(chunks). {ok,chunks} > > Why do I get: 49> chunks:chunks("Able was I ere I saw Elba."). ** > exception error: no function clause matching chunks:chunks("Able > was I ere I saw Elba.") (chunks.erl, line 15) (Well, forget the > like 15. I took out a bunch of comments.) > _______________________________________________ erlang-questions > mailing list erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > - -- James Aimonetti Lead Systems Architect / Impressionable Scallywag "I thought I fixed that" 2600Hz | http://2600hz.com sip:james@REDACTED tel:415.886.7905 irc:mc_ @ freenode -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJUW8dPAAoJENTKa+JPXCVgv6UH/Ailn5lOqGv3JWQM9P7FfcYs VM4Ig0/aRhc+qdfUhwIkOj/gimczq0KnnJcImX9aIuJ7kykkGV/B4aLQHLqqaxFN 5xIU6JIohS7TIgSoesAKWkq4DcMoFVUrVkTHWqBb7hTotWeZSoyy6daVBurkYZs8 omd27lVeQ1Oz8nDHU/y1YhqCJsL9U4qk3VGFLQjv8J9MrWNYsBcAtuMJqWcsV0NW BqnAMcK1ERLq5Sn30pJ1Ap4tzR6yZfI2pMcyN0uo5F8pdJYtdZUwS1Hg4D2cuL7R /jsTgCRfbLHTRJEcI160ROWGzx7DtHnoZtoLDpc9Cfq/O1CGKXklNrbG7mOM4Z4= =0Vrm -----END PGP SIGNATURE----- From fly@REDACTED Thu Nov 6 20:26:37 2014 From: fly@REDACTED (Fred Youhanaie) Date: Thu, 06 Nov 2014 19:26:37 +0000 Subject: [erlang-questions] newbie question In-Reply-To: <545BC74F.5090508@2600hz.com> References: <545BC45B.9020207@earthlink.net> <545BC74F.5090508@2600hz.com> Message-ID: <545BCB6D.3040409@anydata.co.uk> and of course the same goes for lst in chunks/2 :) On 06/11/14 19:09, James Aimonetti wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > str should be Str. Variables start with a capital letter. > > Le 2014-11-06 10:56, Charles Hixson a ?crit : >> Given the routine: -module(chunks). -export([chunks/1]). >> chunks(str)-> chunks(str, []). >> >> chunks("", lst)-> list:reverse(lst); chunks(str, lst)-> P1 = >> "^([[:alnum:]]+|[[:space:]]+|[[:punct:]]+|[[:cntrl:]]+|.)", [[], >> M1, T1] = re:split(str, P1, [{return, list}]), [M1 | chunks(T1, >> P1)]. >> >> and the shell: 47> c(chunks). {ok,chunks} >> >> Why do I get: 49> chunks:chunks("Able was I ere I saw Elba."). ** >> exception error: no function clause matching chunks:chunks("Able >> was I ere I saw Elba.") (chunks.erl, line 15) (Well, forget the >> like 15. I took out a bunch of comments.) From eriksoe@REDACTED Thu Nov 6 20:58:35 2014 From: eriksoe@REDACTED (=?UTF-8?Q?Erik_S=C3=B8e_S=C3=B8rensen?=) Date: Thu, 6 Nov 2014 20:58:35 +0100 Subject: [erlang-questions] newbie question In-Reply-To: <545BCB6D.3040409@anydata.co.uk> References: <545BC45B.9020207@earthlink.net> <545BC74F.5090508@2600hz.com> <545BCB6D.3040409@anydata.co.uk> Message-ID: Hi there - Apart from the atom-vs-Variable issue, I think you got your recursion wrong. [M1 | chunks(T1, P1)] doesn't make too much sense (that I can see) - especially the "recurse with P1 for Lst" part. I think you mean either chunks(T1, [M1|Lst]) for a tail recursive loop, or chunks("") -> []; chunks(Str) -> ..., [M1 | chunks(T1)] for "normal" recursion on the rest. At present, you code looks like a confusion of the two (apart from the Lst/P1 difference). Regards, and have fun on the learning curve, Erik Den 06/11/2014 20.27 skrev "Fred Youhanaie" : > > and of course the same goes for lst in chunks/2 :) > > On 06/11/14 19:09, James Aimonetti wrote: > >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA1 >> >> str should be Str. Variables start with a capital letter. >> >> Le 2014-11-06 10:56, Charles Hixson a ?crit : >> >>> Given the routine: -module(chunks). -export([chunks/1]). >>> chunks(str)-> chunks(str, []). >>> >>> chunks("", lst)-> list:reverse(lst); chunks(str, lst)-> P1 = >>> "^([[:alnum:]]+|[[:space:]]+|[[:punct:]]+|[[:cntrl:]]+|.)", [[], >>> M1, T1] = re:split(str, P1, [{return, list}]), [M1 | chunks(T1, >>> P1)]. >>> >>> and the shell: 47> c(chunks). {ok,chunks} >>> >>> Why do I get: 49> chunks:chunks("Able was I ere I saw Elba."). ** >>> exception error: no function clause matching chunks:chunks("Able >>> was I ere I saw Elba.") (chunks.erl, line 15) (Well, forget the >>> like 15. I took out a bunch of comments.) >>> >> _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charleshixsn@REDACTED Thu Nov 6 21:33:14 2014 From: charleshixsn@REDACTED (Charles Hixson) Date: Thu, 06 Nov 2014 12:33:14 -0800 Subject: [erlang-questions] newbie question In-Reply-To: References: <545BC45B.9020207@earthlink.net> <545BC74F.5090508@2600hz.com> <545BCB6D.3040409@anydata.co.uk> Message-ID: <545BDB0A.9010700@earthlink.net> On 11/06/2014 11:58 AM, Erik S?e S?rensen wrote: > > Hi there - > > Apart from the atom-vs-Variable issue, I think you got your recursion > wrong. > > [M1 | chunks(T1, P1)] > > doesn't make too much sense (that I can see) - especially the "recurse > with P1 for Lst" part. > > I think you mean either > > chunks(T1, [M1|Lst]) > > for a tail recursive loop, or > > chunks("") -> []; > > chunks(Str) -> ..., [M1 | chunks(T1)] > > for "normal" recursion on the rest. At present, you code looks like a > confusion of the two (apart from the Lst/P1 difference). > > > Regards, and have fun on the learning curve, > > Erik > > > Den 06/11/2014 20.27 skrev "Fred Youhanaie" >: > > > and of course the same goes for lst in chunks/2 :) > > On 06/11/14 19:09, James Aimonetti wrote: > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > str should be Str. Variables start with a capital letter. > > Le 2014-11-06 10:56, Charles Hixson a ?crit : > > Given the routine: -module(chunks). -export([chunks/1]). > chunks(str)-> chunks(str, []). > > chunks("", lst)-> list:reverse(lst); chunks(str, lst)-> P1 = > "^([[:alnum:]]+|[[:space:]]+|[[:punct:]]+|[[:cntrl:]]+|.)", [[], > M1, T1] = re:split(str, P1, [{return, list}]), [M1 | > chunks(T1, > P1)]. > > and the shell: 47> c(chunks). {ok,chunks} > > Why do I get: 49> chunks:chunks("Able was I ere I saw > Elba."). ** > exception error: no function clause matching > chunks:chunks("Able > was I ere I saw Elba.") (chunks.erl, line 15) (Well, > forget the > like 15. I took out a bunch of comments.) > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions Thank you much (all of you!). Using upper case of "variables" is going to take a bit of learning. -------------- next part -------------- An HTML attachment was scrubbed... URL: From stu.bailey@REDACTED Fri Nov 7 02:33:14 2014 From: stu.bailey@REDACTED (Stu Bailey) Date: Thu, 6 Nov 2014 17:33:14 -0800 Subject: [erlang-questions] Performance question Message-ID: I found binary:replace(BinChunk,<<"\n">>,<<>>,[global]). *significantly *slower than remove_pattern(BinChunk,<<>>,<<"\n">>). with remove_pattern(<<>>,Acc,_BinPat) -> Acc; remove_pattern(Bin,Acc,BinPat)-> <> = Bin, case Byte == BinPat of true -> remove_pattern(Rest,Acc,BinPat); false -> remove_pattern(Rest,<>,BinPat) end. That was surprising to me. The built-in binary:replace() was much much slower for larger BinChunk with lots of <<"\n">> sprinkled through. Thoughts? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ahe.sanath@REDACTED Fri Nov 7 03:07:04 2014 From: ahe.sanath@REDACTED (Sanath Prasanna) Date: Fri, 7 Nov 2014 07:37:04 +0530 Subject: [erlang-questions] =?utf-8?b?RXVybyBTeW1ib2wgKOKCrCkgY29udmVyc2lv?= =?utf-8?q?n_from_HTTP_API_to_SMPP_through_mochiweb?= Message-ID: Hi, I used mochiweb web server to handle http request. I submit messages with euro - ? as "%E2%82%AC" thorugh HTTP API. I need to forward this to SMPP connction. Before that I want to convert that to 1B65. Otherwise ? character is not display in mobile. How to do that ? Appreciate if some can help to do that. Br, Sanath Roberto -------------- next part -------------- An HTML attachment was scrubbed... URL: From kolorahl@REDACTED Fri Nov 7 04:41:03 2014 From: kolorahl@REDACTED (Tyler Margison) Date: Thu, 6 Nov 2014 19:41:03 -0800 Subject: [erlang-questions] Performance question In-Reply-To: References: Message-ID: I'm not sure if this would affect performance or not, but I believe binary:replace looks at variable-sized patterns whereas your remove_pattern function only work for one-byte patterns. What if the pattern is larger than 1-byte in length? It definitely works for <<"\n">> but I don't think it would work for <<"\r\n">>. Perhaps there is a performance degradation when calculating pattern length? Again, I'm not sure if that's part of the performance problem, but the two functions you propose are not quite synonymous. On Thu, Nov 6, 2014 at 5:33 PM, Stu Bailey wrote: > I found > > binary:replace(BinChunk,<<"\n">>,<<>>,[global]). > > *significantly *slower than > > remove_pattern(BinChunk,<<>>,<<"\n">>). > > with > > remove_pattern(<<>>,Acc,_BinPat) -> > Acc; > remove_pattern(Bin,Acc,BinPat)-> > <> = Bin, > case Byte == BinPat of > true -> remove_pattern(Rest,Acc,BinPat); > false -> remove_pattern(Rest,<>,BinPat) > end. > > That was surprising to me. The built-in binary:replace() was much much > slower for larger BinChunk with lots of <<"\n">> sprinkled through. > > Thoughts? > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bob@REDACTED Fri Nov 7 06:22:47 2014 From: bob@REDACTED (Bob Ippolito) Date: Thu, 6 Nov 2014 21:22:47 -0800 Subject: [erlang-questions] =?utf-8?b?RXVybyBTeW1ib2wgKOKCrCkgY29udmVyc2lv?= =?utf-8?q?n_from_HTTP_API_to_SMPP_through_mochiweb?= In-Reply-To: References: Message-ID: Use the unicode module to decode the UTF8 if you need a string of Unicode code points. If you need to decode a URL encoded string then you can use mochiweb_util for that. On Thursday, November 6, 2014, Sanath Prasanna wrote: > Hi, > I used mochiweb web server to handle http request. I submit messages with > euro - ? as "%E2%82%AC" thorugh HTTP API. I need to forward this to SMPP > connction. Before that I want to convert that to 1B65. Otherwise ? > character is not display in mobile. > How to do that ? Appreciate if some can help to do that. > Br, > Sanath Roberto > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ahe.sanath@REDACTED Fri Nov 7 09:21:09 2014 From: ahe.sanath@REDACTED (Sanath Prasanna) Date: Fri, 7 Nov 2014 13:51:09 +0530 Subject: [erlang-questions] =?utf-8?b?RXVybyBTeW1ib2wgKOKCrCkgY29udmVyc2lv?= =?utf-8?q?n_from_HTTP_API_to_SMPP_through_mochiweb?= In-Reply-To: References: Message-ID: Hi Bob, Tx for quick reply. I go through & unable to find direct function to do that.If you never mind, can you elaborate with example.? Br, Sanath Roberto On Fri, Nov 7, 2014 at 10:52 AM, Bob Ippolito wrote: > Use the unicode module to decode the UTF8 if you need a string of Unicode > code points. If you need to decode a URL encoded string then you can use > mochiweb_util for that. > > > On Thursday, November 6, 2014, Sanath Prasanna > wrote: > >> Hi, >> I used mochiweb web server to handle http request. I submit messages with >> euro - ? as "%E2%82%AC" thorugh HTTP API. I need to forward this to SMPP >> connction. Before that I want to convert that to 1B65. Otherwise ? >> character is not display in mobile. >> How to do that ? Appreciate if some can help to do that. >> Br, >> Sanath Roberto >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ingela.andin@REDACTED Fri Nov 7 09:37:32 2014 From: ingela.andin@REDACTED (Ingela Andin) Date: Fri, 7 Nov 2014 09:37:32 +0100 Subject: [erlang-questions] 'ssl_otp_session_cache' ets table In-Reply-To: References: Message-ID: Hi! Which version of the ssl application are you using? The session cleanup was broken in one of the older versions. Regards Ingela Erlang/OTP - team Ericsson AB 2014-11-06 17:20 GMT+01:00 Bogdan Andu : > yes, > > in function ssl_manager:invalidate_session/4, the call > > erlang:send_after(delay_time(), self(), {delayed_clean_session, Key}), > does the actual deletion of session from cache > > where delay_time/0 function returns ?CLEAN_SESSION_DB by default, which > is 60 seconds > > > > > On Thu, Nov 6, 2014 at 4:51 PM, Ingela Andin > wrote: > >> Hi! >> >> >> 2014-11-06 14:39 GMT+01:00 Bogdan Andu : >> >>> Hi, >>> >>> i have tried to pass to execute th following forms of commands: >>> >>> erl -ssl session_lifetime 60 >>> >>> erl -ssl session_lifetime '60' >>> >>> erl -ssl session_lifetime '[60]' >>> >>> >>> in order to set a ssl session_lifetime of 60 seconds. >>> >>> in init of ssl_manager , after session_lifetime + 5 seconds a process is >>> created to validate or invalidate sessions. >>> So, after 65 seconds a foldl is called on sessions table to check >>> sessions after formula: >>> Now - TimeStamp < LifeTime >>> >>> However, even after I stop the client, the table 'ssl_otp_session_cache' >>> 's size remains the same. >>> >>> >>> After that once every ?CLEAN_SESSION_DB which is 60 seconds , the table >>> is sweeped to delete any expired sessions, >>> >>> but also nothing happens. >>> >>> >> There is also a delay in the actual deletion. In the first sweep the >> sessions will only be invalidated as there may already be >> spawned connection handlers that needs to read the session data before we >> may delete it. It is a performance trade off. >> >> Regards Ingela Erlang/OTP team - Ericsson AB >> >> >> >>> >>> On Thu, Nov 6, 2014 at 12:28 PM, Ingela Andin >>> wrote: >>> >>>> Hi! >>>> >>>> >>>> 2014-11-06 8:53 GMT+01:00 Bogdan Andu : >>>> >>>>> Hi, >>>>> >>>>> on my production servers I have relayd (on OpenBSD) daemon as a >>>>> reverse proxy to some webservers >>>>> where one can fine tune some connection parameters, as well as some >>>>> ssl parameters. >>>>> >>>>> I give a snippet from a relayd.conf configuration file on one of my >>>>> production server: >>>>> >>>>> ............. >>>>> # >>>>> # Relay and protocol for HTTP layer 7 loadbalancing and SSL >>>>> acceleration >>>>> # >>>>> http protocol www_ssl_prot { >>>>> header append "$REMOTE_ADDR" to "X-Forwarded-For" >>>>> header append "$SERVER_ADDR:$SERVER_PORT" to "X-Forwarded-By" >>>>> header change "Connection" to "close" >>>>> >>>>> response header change "Server" to "Apache 0.1" >>>>> >>>>> # Various TCP performance options >>>>> tcp { nodelay, sack, socket buffer 65536, backlog 128 } >>>>> >>>>> ssl { no sslv2, no sslv3, tlsv1, ciphers "HIGH" } >>>>> ssl session cache disable >>>>> } >>>>> >>>>> the last directive tells relayd not to use ssl cache. >>>>> >>>>> This configuration is working for years and relayd was restarted once >>>>> by accident - my fault. >>>>> >>>>> SO, y question is: >>>>> >>>>> can we have this configurable in Erlang, in other words, we might be >>>>> able to start an erlang vm such as: >>>>> >>>>> erl -ssl session_cache 'disable' -name x@REDACTED .... >>>>> >>>>> The ssl option session_cache can be set to disabled by default and can >>>>> take values either disable or enabled. >>>>> >>>>> >>>>> >>>> You can already disable the reuse of the sessions using the server >>>> option *{reuse_sessions, boolean()}* which default to true. >>>> The thing we plan to do is to have a configurable limit on the table >>>> size when sessions are reused. >>>> >>>> Regards Ingela Erlang/OTP team Ericsson AB >>>> >>>> >>>> >>>> >>>> >>>> >>>>> Just to let you know... >>>>> >>>>> The statistics of a node running from yesterday when I opened this >>>>> thread of discussion, using a single client: >>>>> >>>>> Node: 'n1@REDACTED' (Connected) (17/6.2) unix (openbsd 5.4.0) CPU:2 >>>>> SMP +A:10 >>>>> Time: local time 09:48:48, up for 000:20:27:49, 0ms latency, >>>>> Processes: total 681 (RQ 0) at 610275 RpI using 11526.4k (11805.8k >>>>> allocated) >>>>> Memory: Sys 94835.5k, Atom 407.7k/419.5k, Bin 176.4k, Code 9934.7k, >>>>> Ets 80739.2k >>>>> >>>>> ........... >>>>> >>>>> So, can be this made configurable? >>>>> >>>>> >>>>> >>>>> On Wed, Nov 5, 2014 at 8:52 PM, Ingela Andin >>>>> wrote: >>>>> >>>>>> Hi! >>>>>> >>>>>> 2014-11-05 14:15 GMT+01:00 Tony Rogvall : >>>>>> >>>>>>> I think the ssl session times is the problem here, and the lack of a >>>>>>> maximum size. >>>>>>> >>>>>>> You can change the session time in the ssl environment: >>>>>>> session_lifetime >>>>>>> The default is set to 24 hours (in seconds) (if I read it correctly, >>>>>>> in ssl_manager.erl) >>>>>>> >>>>>> >>>>>> Default values are always hard. It is the maximum recommended time >>>>>> for a session to live >>>>>> according to the spec. >>>>>> >>>>>> >>>>>>> I guess that a session_cache_size could be a nice thing to have, >>>>>>> limiting the growth of the session cache. >>>>>>> >>>>>>> >>>>>> Yes I agree. >>>>>> >>>>>> >>>>>>> In other words you have to estimate the life time of your clients and >>>>>>> try to find a reasonable session_lifetime to match that, without >>>>>>> blowing up >>>>>>> the system. >>>>>>> >>>>>>> Maybe the ssl_session_cache_api could be used to implement a >>>>>>> strategy with a max size >>>>>>> cache. Retire session least recently used, while performing the >>>>>>> update? >>>>>>> There is a time_stamp in the session that that could be used for >>>>>>> this purpose. >>>>>>> >>>>>>> OTP: Why is a fixed limit not implemented in the standard >>>>>>> ssl_session_cache? >>>>>>> Could this be a target for DOS attacks? >>>>>>> >>>>>> >>>>>> >>>>>> We are aware of the problem, and it is on our todo list. One reason >>>>>> it has not had top priority is that on the server side there are often >>>>>> other mechanisms >>>>>> like firewalls and webbserver settings that limits the problem. And >>>>>> the reason why it was not implemented in the first place is that >>>>>> implementations by nature >>>>>> are iterative and at first you are faced with a lot of bigger >>>>>> problems to solve and then you need to iterate and fine tune and fix things >>>>>> that you now have a better understanding of. >>>>>> >>>>>> In current master there is a change to the session table that limits >>>>>> the growth on the client side if the client behaves inappropriate. It also >>>>>> splits the session table into >>>>>> a server and a client table which is a better implementation as the >>>>>> same Erlang node can be both a client and a server at the same time. So if >>>>>> someone feels like contributing a max limit please base it on the master >>>>>> branch, otherwise I suspect someone compiling about it did raise the >>>>>> priority level a little. >>>>>> >>>>>> >>>>>> Regards Ingela Erlang/OTP team - Ericsson AB >>>>>> >>>>>> >>>>>> >>>>>>> /Tony >>>>>>> >>>>>>> > On 5 nov 2014, at 13:08, Bogdan Andu wrote: >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > Hi, >>>>>>> > >>>>>>> > I performed a series of test regarding the an Erlang SSL server. >>>>>>> > >>>>>>> > In this setup a major role is played by the table called >>>>>>> 'ssl_otp_session_cache', and of course the processes using it. >>>>>>> > >>>>>>> > The problem is that the size of table increases constantly and, >>>>>>> because an ets table does not automatically deallocate memory unless the >>>>>>> object are deleted from that table, the size of table remains the same even >>>>>>> if there no ssl connections to server. >>>>>>> > >>>>>>> > For example, with a single client running 'ad infinitum' the table >>>>>>> increases at a rate of 5 MBytes/hour. In 12 hours there are allocate around >>>>>>> 60 MB of memory only for this table. >>>>>>> > >>>>>>> > Some info about this: >>>>>>> > $ erl -sname console@REDACTED -remsh n1@REDACTED -setcookie >>>>>>> operator >>>>>>> > Erlang/OTP 17 [erts-6.2] [source] [64-bit] [smp:2:2] >>>>>>> [async-threads:10] [kernel-poll:false] >>>>>>> > >>>>>>> > Eshell V6.2 (abort with ^G) >>>>>>> > >>>>>>> > >>>>>>> > (n1@REDACTED)1> ets:i(). >>>>>>> > id name type size mem owner >>>>>>> > >>>>>>> ---------------------------------------------------------------------------- >>>>>>> > 12 cookies set 0 291 auth >>>>>>> > 4111 code set 410 26132 >>>>>>> code_server >>>>>>> > 8208 code_names set 58 7459 >>>>>>> code_server >>>>>>> > 12307 httpc_manager__session_cookie_db bag 0 >>>>>>> 291 httpc_manager >>>>>>> > 16404 ssl_otp_cacertificate_db set 0 291 >>>>>>> ssl_manager >>>>>>> > 20501 ssl_otp_ca_file_ref set 0 291 >>>>>>> ssl_manager >>>>>>> > 24598 ssl_otp_pem_cache set 3 360 >>>>>>> ssl_manager >>>>>>> > 28695 ssl_otp_session_cache ordered_set 138057 8421893 >>>>>>> ssl_manager >>>>>>> > 32797 dets duplicate_bag 2 308 >>>>>>> dets >>>>>>> > 40990 ign_requests set 0 291 >>>>>>> inet_gethost_native >>>>>>> > 45087 ign_req_index set 0 291 >>>>>>> inet_gethost_native >>>>>>> > 2261635104 shell_records ordered_set 0 81 >>>>>>> <0.30638.37> >>>>>>> > ac_tab ac_tab set 33 2216 >>>>>>> application_controller >>>>>>> > code_map code_map set 100 2791 <0.72.0> >>>>>>> > config config set 12 892 <0.72.0> >>>>>>> > dets_owners dets_owners set 1 298 dets >>>>>>> > dets_registry dets_registry set 1 299 dets >>>>>>> > file_io_servers file_io_servers set 1 344 >>>>>>> file_server_2 >>>>>>> > global_locks global_locks set 0 291 >>>>>>> global_name_server >>>>>>> > global_names global_names set 0 291 >>>>>>> global_name_server >>>>>>> > global_names_ext global_names_ext set 0 291 >>>>>>> global_name_server >>>>>>> > global_pid_ids global_pid_ids bag 0 291 >>>>>>> global_name_server >>>>>>> > global_pid_names global_pid_names bag 0 291 >>>>>>> global_name_server >>>>>>> > httpc_manager__handler_db httpc_manager__handler_db set 0 >>>>>>> 291 httpc_manager >>>>>>> > httpc_manager__session_db httpc_manager__session_db set 0 >>>>>>> 291 httpc_manager >>>>>>> > inet_cache inet_cache bag 0 291 inet_db >>>>>>> > inet_db inet_db set 29 600 inet_db >>>>>>> > inet_hosts_byaddr inet_hosts_byaddr bag 0 291 inet_db >>>>>>> > inet_hosts_byname inet_hosts_byname bag 0 291 inet_db >>>>>>> > inet_hosts_file_byaddr inet_hosts_file_byaddr bag 0 291 >>>>>>> inet_db >>>>>>> > inet_hosts_file_byname inet_hosts_file_byname bag 0 291 >>>>>>> inet_db >>>>>>> > models models set 3 28952 <0.72.0> >>>>>>> > sys_dist sys_dist set 1 334 net_kernel >>>>>>> > ok >>>>>>> > (n1@REDACTED)5> ets:info(ssl_otp_session_cache). >>>>>>> > undefined >>>>>>> > (n1@REDACTED)7> (8421893*8)/1024. >>>>>>> > 65796.0390625 >>>>>>> > (n1@REDACTED)8> memory(). >>>>>>> > [{total,92699464}, >>>>>>> > {processes,8964000}, >>>>>>> > {processes_used,8963152}, >>>>>>> > {system,83735464}, >>>>>>> > {atom,429569}, >>>>>>> > {atom_used,421768}, >>>>>>> > {binary,199040}, >>>>>>> > {code,10411520}, >>>>>>> > {ets,69163032}] >>>>>>> > >>>>>>> > The memory allocated to table 'ssl_otp_session_cache' is roughly >>>>>>> 64 MB in 12 hours. >>>>>>> > >>>>>>> > On an OpenBSD platform such process gets killed immediately it >>>>>>> hits some memory and/or CPU limits. >>>>>>> > >>>>>>> > To make this test on OpenBSD I had to put 'infinit' to memory, >>>>>>> otherwise the Erlang VM would be killed. >>>>>>> > >>>>>>> > How can one control , tweak or configure this table such that it >>>>>>> does not accumulate such data at such high rate. >>>>>>> > >>>>>>> > I seems the table being created private, and there is no way to >>>>>>> ets:delete_all_objects/1 from table manually. >>>>>>> > >>>>>>> > I know that this table caches some SSL data related to clients, >>>>>>> but the client has the same IP address, >>>>>>> > and I wonder why is neccesary to store a lot of SSL connection >>>>>>> info about the same client when only the ephemeral peer port >>>>>>> > differs? >>>>>>> > >>>>>>> > How the size of this table can be held in reasonable limits and >>>>>>> the rate it's size increases ? >>>>>>> > >>>>>>> > Please if somebody shed some light on these issues. >>>>>>> > >>>>>>> > Thank you, >>>>>>> > >>>>>>> > Bogdan >>>>>>> > >>>>>>> > >>>>>>> > _______________________________________________ >>>>>>> > erlang-questions mailing list >>>>>>> > erlang-questions@REDACTED >>>>>>> > http://erlang.org/mailman/listinfo/erlang-questions >>>>>>> >>>>>>> "Installing applications can lead to corruption over time. >>>>>>> Applications gradually write over each other's libraries, partial upgrades >>>>>>> occur, user and system errors happen, and minute changes may be >>>>>>> unnoticeable and difficult to fix" >>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> erlang-questions mailing list >>>>>>> erlang-questions@REDACTED >>>>>>> http://erlang.org/mailman/listinfo/erlang-questions >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bob@REDACTED Fri Nov 7 10:03:05 2014 From: bob@REDACTED (Bob Ippolito) Date: Fri, 7 Nov 2014 01:03:05 -0800 Subject: [erlang-questions] =?utf-8?b?RXVybyBTeW1ib2wgKOKCrCkgY29udmVyc2lv?= =?utf-8?q?n_from_HTTP_API_to_SMPP_through_mochiweb?= In-Reply-To: References: Message-ID: 1> unicode:characters_to_list(iolist_to_binary(mochiweb_util:unquote("%E2%82%AC"))). [8364] On Fri, Nov 7, 2014 at 12:21 AM, Sanath Prasanna wrote: > Hi Bob, > Tx for quick reply. I go through & unable to find direct function to do > that.If you never mind, can you elaborate with example.? > Br, > Sanath Roberto > > On Fri, Nov 7, 2014 at 10:52 AM, Bob Ippolito wrote: > >> Use the unicode module to decode the UTF8 if you need a string of Unicode >> code points. If you need to decode a URL encoded string then you can use >> mochiweb_util for that. >> >> >> On Thursday, November 6, 2014, Sanath Prasanna >> wrote: >> >>> Hi, >>> I used mochiweb web server to handle http request. I submit messages >>> with euro - ? as "%E2%82%AC" thorugh HTTP API. I need to forward this to >>> SMPP connction. Before that I want to convert that to 1B65. Otherwise ? >>> character is not display in mobile. >>> How to do that ? Appreciate if some can help to do that. >>> Br, >>> Sanath Roberto >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Fri Nov 7 10:44:15 2014 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Fri, 07 Nov 2014 11:44:15 +0200 Subject: [erlang-questions] Performance question In-Reply-To: References: Message-ID: <545C946F.4010605@ninenines.eu> binary:split and binary:replace, unlike other functions of the binary module, are normal Erlang functions. They also process a list of options before doing the actual work, so there's an obvious overhead compared to not doing that. In addition as has been pointed out, your code is more specialized so that helps too. On 11/07/2014 03:33 AM, Stu Bailey wrote: > I found > > binary:replace(BinChunk,<<"\n">>,<<>>,[global]). > > /significantly /slower than > > remove_pattern(BinChunk,<<>>,<<"\n">>). > > with > > remove_pattern(<<>>,Acc,_BinPat) -> > Acc; > remove_pattern(Bin,Acc,BinPat)-> > <> = Bin, > case Byte == BinPat of > true -> remove_pattern(Rest,Acc,BinPat); > false -> remove_pattern(Rest,<>,BinPat) > end. > > That was surprising to me. The built-in binary:replace() was much much > slower for larger BinChunk with lots of <<"\n">> sprinkled through. > > Thoughts? > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin http://ninenines.eu From bog495@REDACTED Fri Nov 7 11:58:59 2014 From: bog495@REDACTED (Bogdan Andu) Date: Fri, 7 Nov 2014 12:58:59 +0200 Subject: [erlang-questions] 'ssl_otp_session_cache' ets table In-Reply-To: References: Message-ID: I work with R14B04 as production and OTP 17.3 as experimental. R14B04 also deletes all entries after 24 hours. 17.3 also deletes all entries after 24 hours. I think is it has to do with passing the environment parameter like this: erl -ssl session_lifetime 60 ... I tried many variants and may be I miss something. may be I am mistaken but it is possible that only the default value, 24 hours, is only considered as the environment variable is not passed correctly by me. i think the environment parameters are the problem here because maybe i specify them incorrctly. could you advise what is the proper method of correctly specifying session_lifetime, because according to manual is like : erl -ssl session_lifetime 60 ... a ssl session must expire afte 60 seconds + delayed 60 seconds = 2 minutes. intstead entries from 'ssl_otp_session_cache' be deleted every 2 minutes, they remain 24 hours . after 22 hours I have: Node: 'n1@REDACTED' (Connected) (17/6.2) unix (openbsd 5.4.0) CPU:2 SMP +A:10 Time: local time 12:50:23, up for 000:21:54:49, 0ms latency, Processes: total 676 (RQ 0) at 21121 RpI using 10978.0k (11007.4k allocated) Memory: Sys 14898.8k, Atom 410.8k/419.5k, Bin 112.9k, Code 10053.7k, Ets 720.6k and for ets:i(). ........... 28695 ssl_otp_session_cache ordered_set 115 7134 ssl_manager .......... The vm was configured with 60 seconds session lifetime and the client was running 60 seconds an I expected after 2 minutes to see less entries in the table but they remained untouched I will see what happens after 24 hours mark and I'll let you know and another thing... even if i pass the options: {reuse_session, fun(_,_,_,_) -> false end}, {reuse_sessions, false}, both or individually , to the ssl:listen/2 function the table is filled with sessions and they expire after 24 hours, only that cpu is working harder, 7-8% which means the sessions a not reused and the ssl handshake is happening every time. Bogdan On Fri, Nov 7, 2014 at 10:37 AM, Ingela Andin wrote: > Hi! > > Which version of the ssl application are you using? The session cleanup > was broken in one of the older versions. > > Regards Ingela Erlang/OTP - team Ericsson AB > > > 2014-11-06 17:20 GMT+01:00 Bogdan Andu : > >> yes, >> >> in function ssl_manager:invalidate_session/4, the call >> >> erlang:send_after(delay_time(), self(), {delayed_clean_session, Key}), >> does the actual deletion of session from cache >> >> where delay_time/0 function returns ?CLEAN_SESSION_DB by default, which >> is 60 seconds >> >> >> >> >> On Thu, Nov 6, 2014 at 4:51 PM, Ingela Andin >> wrote: >> >>> Hi! >>> >>> >>> 2014-11-06 14:39 GMT+01:00 Bogdan Andu : >>> >>>> Hi, >>>> >>>> i have tried to pass to execute th following forms of commands: >>>> >>>> erl -ssl session_lifetime 60 >>>> >>>> erl -ssl session_lifetime '60' >>>> >>>> erl -ssl session_lifetime '[60]' >>>> >>>> >>>> in order to set a ssl session_lifetime of 60 seconds. >>>> >>>> in init of ssl_manager , after session_lifetime + 5 seconds a process >>>> is created to validate or invalidate sessions. >>>> So, after 65 seconds a foldl is called on sessions table to check >>>> sessions after formula: >>>> Now - TimeStamp < LifeTime >>>> >>>> However, even after I stop the client, the table >>>> 'ssl_otp_session_cache' 's size remains the same. >>>> >>>> >>>> After that once every ?CLEAN_SESSION_DB which is 60 seconds , the >>>> table is sweeped to delete any expired sessions, >>>> >>>> but also nothing happens. >>>> >>>> >>> There is also a delay in the actual deletion. In the first sweep the >>> sessions will only be invalidated as there may already be >>> spawned connection handlers that needs to read the session data before >>> we may delete it. It is a performance trade off. >>> >>> Regards Ingela Erlang/OTP team - Ericsson AB >>> >>> >>> >>>> >>>> On Thu, Nov 6, 2014 at 12:28 PM, Ingela Andin >>>> wrote: >>>> >>>>> Hi! >>>>> >>>>> >>>>> 2014-11-06 8:53 GMT+01:00 Bogdan Andu : >>>>> >>>>>> Hi, >>>>>> >>>>>> on my production servers I have relayd (on OpenBSD) daemon as a >>>>>> reverse proxy to some webservers >>>>>> where one can fine tune some connection parameters, as well as some >>>>>> ssl parameters. >>>>>> >>>>>> I give a snippet from a relayd.conf configuration file on one of my >>>>>> production server: >>>>>> >>>>>> ............. >>>>>> # >>>>>> # Relay and protocol for HTTP layer 7 loadbalancing and SSL >>>>>> acceleration >>>>>> # >>>>>> http protocol www_ssl_prot { >>>>>> header append "$REMOTE_ADDR" to "X-Forwarded-For" >>>>>> header append "$SERVER_ADDR:$SERVER_PORT" to "X-Forwarded-By" >>>>>> header change "Connection" to "close" >>>>>> >>>>>> response header change "Server" to "Apache 0.1" >>>>>> >>>>>> # Various TCP performance options >>>>>> tcp { nodelay, sack, socket buffer 65536, backlog 128 } >>>>>> >>>>>> ssl { no sslv2, no sslv3, tlsv1, ciphers "HIGH" } >>>>>> ssl session cache disable >>>>>> } >>>>>> >>>>>> the last directive tells relayd not to use ssl cache. >>>>>> >>>>>> This configuration is working for years and relayd was restarted once >>>>>> by accident - my fault. >>>>>> >>>>>> SO, y question is: >>>>>> >>>>>> can we have this configurable in Erlang, in other words, we might be >>>>>> able to start an erlang vm such as: >>>>>> >>>>>> erl -ssl session_cache 'disable' -name x@REDACTED .... >>>>>> >>>>>> The ssl option session_cache can be set to disabled by default and >>>>>> can take values either disable or enabled. >>>>>> >>>>>> >>>>>> >>>>> You can already disable the reuse of the sessions using the server >>>>> option *{reuse_sessions, boolean()}* which default to true. >>>>> The thing we plan to do is to have a configurable limit on the table >>>>> size when sessions are reused. >>>>> >>>>> Regards Ingela Erlang/OTP team Ericsson AB >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> Just to let you know... >>>>>> >>>>>> The statistics of a node running from yesterday when I opened this >>>>>> thread of discussion, using a single client: >>>>>> >>>>>> Node: 'n1@REDACTED' (Connected) (17/6.2) unix (openbsd 5.4.0) CPU:2 >>>>>> SMP +A:10 >>>>>> Time: local time 09:48:48, up for 000:20:27:49, 0ms latency, >>>>>> Processes: total 681 (RQ 0) at 610275 RpI using 11526.4k (11805.8k >>>>>> allocated) >>>>>> Memory: Sys 94835.5k, Atom 407.7k/419.5k, Bin 176.4k, Code 9934.7k, >>>>>> Ets 80739.2k >>>>>> >>>>>> ........... >>>>>> >>>>>> So, can be this made configurable? >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Nov 5, 2014 at 8:52 PM, Ingela Andin >>>>>> wrote: >>>>>> >>>>>>> Hi! >>>>>>> >>>>>>> 2014-11-05 14:15 GMT+01:00 Tony Rogvall : >>>>>>> >>>>>>>> I think the ssl session times is the problem here, and the lack of >>>>>>>> a maximum size. >>>>>>>> >>>>>>>> You can change the session time in the ssl environment: >>>>>>>> session_lifetime >>>>>>>> The default is set to 24 hours (in seconds) (if I read it >>>>>>>> correctly, in ssl_manager.erl) >>>>>>>> >>>>>>> >>>>>>> Default values are always hard. It is the maximum recommended time >>>>>>> for a session to live >>>>>>> according to the spec. >>>>>>> >>>>>>> >>>>>>>> I guess that a session_cache_size could be a nice thing to have, >>>>>>>> limiting the growth of the session cache. >>>>>>>> >>>>>>>> >>>>>>> Yes I agree. >>>>>>> >>>>>>> >>>>>>>> In other words you have to estimate the life time of your clients >>>>>>>> and >>>>>>>> try to find a reasonable session_lifetime to match that, without >>>>>>>> blowing up >>>>>>>> the system. >>>>>>>> >>>>>>>> Maybe the ssl_session_cache_api could be used to implement a >>>>>>>> strategy with a max size >>>>>>>> cache. Retire session least recently used, while performing the >>>>>>>> update? >>>>>>>> There is a time_stamp in the session that that could be used for >>>>>>>> this purpose. >>>>>>>> >>>>>>>> OTP: Why is a fixed limit not implemented in the standard >>>>>>>> ssl_session_cache? >>>>>>>> Could this be a target for DOS attacks? >>>>>>>> >>>>>>> >>>>>>> >>>>>>> We are aware of the problem, and it is on our todo list. One reason >>>>>>> it has not had top priority is that on the server side there are often >>>>>>> other mechanisms >>>>>>> like firewalls and webbserver settings that limits the problem. And >>>>>>> the reason why it was not implemented in the first place is that >>>>>>> implementations by nature >>>>>>> are iterative and at first you are faced with a lot of bigger >>>>>>> problems to solve and then you need to iterate and fine tune and fix things >>>>>>> that you now have a better understanding of. >>>>>>> >>>>>>> In current master there is a change to the session table that limits >>>>>>> the growth on the client side if the client behaves inappropriate. It also >>>>>>> splits the session table into >>>>>>> a server and a client table which is a better implementation as the >>>>>>> same Erlang node can be both a client and a server at the same time. So if >>>>>>> someone feels like contributing a max limit please base it on the master >>>>>>> branch, otherwise I suspect someone compiling about it did raise the >>>>>>> priority level a little. >>>>>>> >>>>>>> >>>>>>> Regards Ingela Erlang/OTP team - Ericsson AB >>>>>>> >>>>>>> >>>>>>> >>>>>>>> /Tony >>>>>>>> >>>>>>>> > On 5 nov 2014, at 13:08, Bogdan Andu wrote: >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > Hi, >>>>>>>> > >>>>>>>> > I performed a series of test regarding the an Erlang SSL server. >>>>>>>> > >>>>>>>> > In this setup a major role is played by the table called >>>>>>>> 'ssl_otp_session_cache', and of course the processes using it. >>>>>>>> > >>>>>>>> > The problem is that the size of table increases constantly and, >>>>>>>> because an ets table does not automatically deallocate memory unless the >>>>>>>> object are deleted from that table, the size of table remains the same even >>>>>>>> if there no ssl connections to server. >>>>>>>> > >>>>>>>> > For example, with a single client running 'ad infinitum' the >>>>>>>> table increases at a rate of 5 MBytes/hour. In 12 hours there are allocate >>>>>>>> around 60 MB of memory only for this table. >>>>>>>> > >>>>>>>> > Some info about this: >>>>>>>> > $ erl -sname console@REDACTED -remsh n1@REDACTED -setcookie >>>>>>>> operator >>>>>>>> > Erlang/OTP 17 [erts-6.2] [source] [64-bit] [smp:2:2] >>>>>>>> [async-threads:10] [kernel-poll:false] >>>>>>>> > >>>>>>>> > Eshell V6.2 (abort with ^G) >>>>>>>> > >>>>>>>> > >>>>>>>> > (n1@REDACTED)1> ets:i(). >>>>>>>> > id name type size mem owner >>>>>>>> > >>>>>>>> ---------------------------------------------------------------------------- >>>>>>>> > 12 cookies set 0 291 auth >>>>>>>> > 4111 code set 410 26132 >>>>>>>> code_server >>>>>>>> > 8208 code_names set 58 7459 >>>>>>>> code_server >>>>>>>> > 12307 httpc_manager__session_cookie_db bag 0 >>>>>>>> 291 httpc_manager >>>>>>>> > 16404 ssl_otp_cacertificate_db set 0 291 >>>>>>>> ssl_manager >>>>>>>> > 20501 ssl_otp_ca_file_ref set 0 291 >>>>>>>> ssl_manager >>>>>>>> > 24598 ssl_otp_pem_cache set 3 360 >>>>>>>> ssl_manager >>>>>>>> > 28695 ssl_otp_session_cache ordered_set 138057 >>>>>>>> 8421893 ssl_manager >>>>>>>> > 32797 dets duplicate_bag 2 308 >>>>>>>> dets >>>>>>>> > 40990 ign_requests set 0 291 >>>>>>>> inet_gethost_native >>>>>>>> > 45087 ign_req_index set 0 291 >>>>>>>> inet_gethost_native >>>>>>>> > 2261635104 shell_records ordered_set 0 81 >>>>>>>> <0.30638.37> >>>>>>>> > ac_tab ac_tab set 33 2216 >>>>>>>> application_controller >>>>>>>> > code_map code_map set 100 2791 <0.72.0> >>>>>>>> > config config set 12 892 <0.72.0> >>>>>>>> > dets_owners dets_owners set 1 298 dets >>>>>>>> > dets_registry dets_registry set 1 299 dets >>>>>>>> > file_io_servers file_io_servers set 1 344 >>>>>>>> file_server_2 >>>>>>>> > global_locks global_locks set 0 291 >>>>>>>> global_name_server >>>>>>>> > global_names global_names set 0 291 >>>>>>>> global_name_server >>>>>>>> > global_names_ext global_names_ext set 0 291 >>>>>>>> global_name_server >>>>>>>> > global_pid_ids global_pid_ids bag 0 291 >>>>>>>> global_name_server >>>>>>>> > global_pid_names global_pid_names bag 0 291 >>>>>>>> global_name_server >>>>>>>> > httpc_manager__handler_db httpc_manager__handler_db set 0 >>>>>>>> 291 httpc_manager >>>>>>>> > httpc_manager__session_db httpc_manager__session_db set 0 >>>>>>>> 291 httpc_manager >>>>>>>> > inet_cache inet_cache bag 0 291 inet_db >>>>>>>> > inet_db inet_db set 29 600 inet_db >>>>>>>> > inet_hosts_byaddr inet_hosts_byaddr bag 0 291 inet_db >>>>>>>> > inet_hosts_byname inet_hosts_byname bag 0 291 inet_db >>>>>>>> > inet_hosts_file_byaddr inet_hosts_file_byaddr bag 0 291 >>>>>>>> inet_db >>>>>>>> > inet_hosts_file_byname inet_hosts_file_byname bag 0 291 >>>>>>>> inet_db >>>>>>>> > models models set 3 28952 <0.72.0> >>>>>>>> > sys_dist sys_dist set 1 334 >>>>>>>> net_kernel >>>>>>>> > ok >>>>>>>> > (n1@REDACTED)5> ets:info(ssl_otp_session_cache). >>>>>>>> > undefined >>>>>>>> > (n1@REDACTED)7> (8421893*8)/1024. >>>>>>>> > 65796.0390625 >>>>>>>> > (n1@REDACTED)8> memory(). >>>>>>>> > [{total,92699464}, >>>>>>>> > {processes,8964000}, >>>>>>>> > {processes_used,8963152}, >>>>>>>> > {system,83735464}, >>>>>>>> > {atom,429569}, >>>>>>>> > {atom_used,421768}, >>>>>>>> > {binary,199040}, >>>>>>>> > {code,10411520}, >>>>>>>> > {ets,69163032}] >>>>>>>> > >>>>>>>> > The memory allocated to table 'ssl_otp_session_cache' is roughly >>>>>>>> 64 MB in 12 hours. >>>>>>>> > >>>>>>>> > On an OpenBSD platform such process gets killed immediately it >>>>>>>> hits some memory and/or CPU limits. >>>>>>>> > >>>>>>>> > To make this test on OpenBSD I had to put 'infinit' to memory, >>>>>>>> otherwise the Erlang VM would be killed. >>>>>>>> > >>>>>>>> > How can one control , tweak or configure this table such that it >>>>>>>> does not accumulate such data at such high rate. >>>>>>>> > >>>>>>>> > I seems the table being created private, and there is no way to >>>>>>>> ets:delete_all_objects/1 from table manually. >>>>>>>> > >>>>>>>> > I know that this table caches some SSL data related to clients, >>>>>>>> but the client has the same IP address, >>>>>>>> > and I wonder why is neccesary to store a lot of SSL connection >>>>>>>> info about the same client when only the ephemeral peer port >>>>>>>> > differs? >>>>>>>> > >>>>>>>> > How the size of this table can be held in reasonable limits and >>>>>>>> the rate it's size increases ? >>>>>>>> > >>>>>>>> > Please if somebody shed some light on these issues. >>>>>>>> > >>>>>>>> > Thank you, >>>>>>>> > >>>>>>>> > Bogdan >>>>>>>> > >>>>>>>> > >>>>>>>> > _______________________________________________ >>>>>>>> > erlang-questions mailing list >>>>>>>> > erlang-questions@REDACTED >>>>>>>> > http://erlang.org/mailman/listinfo/erlang-questions >>>>>>>> >>>>>>>> "Installing applications can lead to corruption over time. >>>>>>>> Applications gradually write over each other's libraries, partial upgrades >>>>>>>> occur, user and system errors happen, and minute changes may be >>>>>>>> unnoticeable and difficult to fix" >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> erlang-questions mailing list >>>>>>>> erlang-questions@REDACTED >>>>>>>> http://erlang.org/mailman/listinfo/erlang-questions >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From benmmurphy@REDACTED Fri Nov 7 12:28:09 2014 From: benmmurphy@REDACTED (Ben Murphy) Date: Fri, 7 Nov 2014 11:28:09 +0000 Subject: [erlang-questions] 'ssl_otp_session_cache' ets table In-Reply-To: References: Message-ID: If you are desperate you can implement your own ssl session cache that does nothing. erl -ssl session_cb dummy_session_cache https://github.com/mitadmin/ionstorm/blob/master/dummy_session_cache.erl I think this was working for me with R1403. On Fri, Nov 7, 2014 at 10:58 AM, Bogdan Andu wrote: > I work with R14B04 as production and OTP 17.3 as experimental. > > R14B04 also deletes all entries after 24 hours. > > 17.3 also deletes all entries after 24 hours. > > I think is it has to do with passing the environment parameter like this: > > erl -ssl session_lifetime 60 ... > > I tried many variants and may be I miss something. > > may be I am mistaken but it is possible that only the default value, 24 > hours, > is only considered as the environment variable is not passed correctly by > me. > > i think the environment parameters are the problem here because maybe i > specify them incorrctly. > > could you advise what is the proper method of correctly specifying > session_lifetime, because according to > manual is like : > > erl -ssl session_lifetime 60 ... > > a ssl session must expire afte 60 seconds + delayed 60 seconds = 2 minutes. > > intstead entries from 'ssl_otp_session_cache' be deleted every 2 minutes, > they remain 24 hours . > > after 22 hours I have: > > Node: 'n1@REDACTED' (Connected) (17/6.2) unix (openbsd 5.4.0) CPU:2 SMP > +A:10 > Time: local time 12:50:23, up for 000:21:54:49, 0ms latency, > Processes: total 676 (RQ 0) at 21121 RpI using 10978.0k (11007.4k allocated) > Memory: Sys 14898.8k, Atom 410.8k/419.5k, Bin 112.9k, Code 10053.7k, Ets > 720.6k > > > and for ets:i(). > > ........... > 28695 ssl_otp_session_cache ordered_set 115 7134 > ssl_manager > .......... > > The vm was configured with 60 seconds session lifetime and the client was > running 60 seconds > an I expected after 2 minutes to see less entries in the table but they > remained untouched > > > I will see what happens after 24 hours mark and I'll let you know > > and another thing... > > even if i pass the options: > {reuse_session, fun(_,_,_,_) -> false end}, > {reuse_sessions, false}, > > both or individually , to the ssl:listen/2 function the table is filled with > sessions and they expire after 24 hours, > only that cpu is working harder, 7-8% which means the sessions a not reused > and the ssl > handshake is happening every time. > > > Bogdan > > On Fri, Nov 7, 2014 at 10:37 AM, Ingela Andin > wrote: >> >> Hi! >> >> Which version of the ssl application are you using? The session cleanup >> was broken in one of the older versions. >> >> Regards Ingela Erlang/OTP - team Ericsson AB >> >> >> 2014-11-06 17:20 GMT+01:00 Bogdan Andu : >>> >>> yes, >>> >>> in function ssl_manager:invalidate_session/4, the call >>> >>> erlang:send_after(delay_time(), self(), {delayed_clean_session, Key}), >>> does the actual deletion of session from cache >>> >>> where delay_time/0 function returns ?CLEAN_SESSION_DB by default, which >>> is 60 seconds >>> >>> >>> >>> >>> On Thu, Nov 6, 2014 at 4:51 PM, Ingela Andin >>> wrote: >>>> >>>> Hi! >>>> >>>> >>>> 2014-11-06 14:39 GMT+01:00 Bogdan Andu : >>>>> >>>>> Hi, >>>>> >>>>> i have tried to pass to execute th following forms of commands: >>>>> >>>>> erl -ssl session_lifetime 60 >>>>> >>>>> erl -ssl session_lifetime '60' >>>>> >>>>> erl -ssl session_lifetime '[60]' >>>>> >>>>> >>>>> in order to set a ssl session_lifetime of 60 seconds. >>>>> >>>>> in init of ssl_manager , after session_lifetime + 5 seconds a process >>>>> is created to validate or invalidate sessions. >>>>> So, after 65 seconds a foldl is called on sessions table to check >>>>> sessions after formula: >>>>> Now - TimeStamp < LifeTime >>>>> >>>>> However, even after I stop the client, the table >>>>> 'ssl_otp_session_cache' 's size remains the same. >>>>> >>>>> >>>>> After that once every ?CLEAN_SESSION_DB which is 60 seconds , the table >>>>> is sweeped to delete any expired sessions, >>>>> >>>>> but also nothing happens. >>>>> >>>> >>>> There is also a delay in the actual deletion. In the first sweep the >>>> sessions will only be invalidated as there may already be >>>> spawned connection handlers that needs to read the session data before >>>> we may delete it. It is a performance trade off. >>>> >>>> Regards Ingela Erlang/OTP team - Ericsson AB >>>> >>>> >>>>> >>>>> >>>>> On Thu, Nov 6, 2014 at 12:28 PM, Ingela Andin >>>>> wrote: >>>>>> >>>>>> Hi! >>>>>> >>>>>> >>>>>> 2014-11-06 8:53 GMT+01:00 Bogdan Andu : >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> on my production servers I have relayd (on OpenBSD) daemon as a >>>>>>> reverse proxy to some webservers >>>>>>> where one can fine tune some connection parameters, as well as some >>>>>>> ssl parameters. >>>>>>> >>>>>>> I give a snippet from a relayd.conf configuration file on one of my >>>>>>> production server: >>>>>>> >>>>>>> ............. >>>>>>> # >>>>>>> # Relay and protocol for HTTP layer 7 loadbalancing and SSL >>>>>>> acceleration >>>>>>> # >>>>>>> http protocol www_ssl_prot { >>>>>>> header append "$REMOTE_ADDR" to "X-Forwarded-For" >>>>>>> header append "$SERVER_ADDR:$SERVER_PORT" to "X-Forwarded-By" >>>>>>> header change "Connection" to "close" >>>>>>> >>>>>>> response header change "Server" to "Apache 0.1" >>>>>>> >>>>>>> # Various TCP performance options >>>>>>> tcp { nodelay, sack, socket buffer 65536, backlog 128 } >>>>>>> >>>>>>> ssl { no sslv2, no sslv3, tlsv1, ciphers "HIGH" } >>>>>>> ssl session cache disable >>>>>>> } >>>>>>> >>>>>>> the last directive tells relayd not to use ssl cache. >>>>>>> >>>>>>> This configuration is working for years and relayd was restarted once >>>>>>> by accident - my fault. >>>>>>> >>>>>>> SO, y question is: >>>>>>> >>>>>>> can we have this configurable in Erlang, in other words, we might be >>>>>>> able to start an erlang vm such as: >>>>>>> >>>>>>> erl -ssl session_cache 'disable' -name x@REDACTED .... >>>>>>> >>>>>>> The ssl option session_cache can be set to disabled by default and >>>>>>> can take values either disable or enabled. >>>>>>> >>>>>>> >>>>>> >>>>>> You can already disable the reuse of the sessions using the server >>>>>> option {reuse_sessions, boolean()} which default to true. >>>>>> The thing we plan to do is to have a configurable limit on the table >>>>>> size when sessions are reused. >>>>>> >>>>>> Regards Ingela Erlang/OTP team Ericsson AB >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> >>>>>>> Just to let you know... >>>>>>> >>>>>>> The statistics of a node running from yesterday when I opened this >>>>>>> thread of discussion, using a single client: >>>>>>> >>>>>>> Node: 'n1@REDACTED' (Connected) (17/6.2) unix (openbsd 5.4.0) CPU:2 >>>>>>> SMP +A:10 >>>>>>> Time: local time 09:48:48, up for 000:20:27:49, 0ms latency, >>>>>>> Processes: total 681 (RQ 0) at 610275 RpI using 11526.4k (11805.8k >>>>>>> allocated) >>>>>>> Memory: Sys 94835.5k, Atom 407.7k/419.5k, Bin 176.4k, Code 9934.7k, >>>>>>> Ets 80739.2k >>>>>>> >>>>>>> ........... >>>>>>> >>>>>>> So, can be this made configurable? >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Wed, Nov 5, 2014 at 8:52 PM, Ingela Andin >>>>>>> wrote: >>>>>>>> >>>>>>>> Hi! >>>>>>>> >>>>>>>> 2014-11-05 14:15 GMT+01:00 Tony Rogvall : >>>>>>>>> >>>>>>>>> I think the ssl session times is the problem here, and the lack of >>>>>>>>> a maximum size. >>>>>>>>> >>>>>>>>> You can change the session time in the ssl environment: >>>>>>>>> session_lifetime >>>>>>>>> The default is set to 24 hours (in seconds) (if I read it >>>>>>>>> correctly, in ssl_manager.erl) >>>>>>>> >>>>>>>> >>>>>>>> Default values are always hard. It is the maximum recommended time >>>>>>>> for a session to live >>>>>>>> according to the spec. >>>>>>>> >>>>>>>>> >>>>>>>>> I guess that a session_cache_size could be a nice thing to have, >>>>>>>>> limiting the growth of the session cache. >>>>>>>>> >>>>>>>> >>>>>>>> Yes I agree. >>>>>>>> >>>>>>>>> >>>>>>>>> In other words you have to estimate the life time of your clients >>>>>>>>> and >>>>>>>>> try to find a reasonable session_lifetime to match that, without >>>>>>>>> blowing up >>>>>>>>> the system. >>>>>>>>> >>>>>>>>> Maybe the ssl_session_cache_api could be used to implement a >>>>>>>>> strategy with a max size >>>>>>>>> cache. Retire session least recently used, while performing the >>>>>>>>> update? >>>>>>>>> There is a time_stamp in the session that that could be used for >>>>>>>>> this purpose. >>>>>>>>> >>>>>>>>> OTP: Why is a fixed limit not implemented in the standard >>>>>>>>> ssl_session_cache? >>>>>>>>> Could this be a target for DOS attacks? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> We are aware of the problem, and it is on our todo list. One reason >>>>>>>> it has not had top priority is that on the server side there are often other >>>>>>>> mechanisms >>>>>>>> like firewalls and webbserver settings that limits the problem. And >>>>>>>> the reason why it was not implemented in the first place is that >>>>>>>> implementations by nature >>>>>>>> are iterative and at first you are faced with a lot of bigger >>>>>>>> problems to solve and then you need to iterate and fine tune and fix things >>>>>>>> that you now have a better understanding of. >>>>>>>> >>>>>>>> In current master there is a change to the session table that limits >>>>>>>> the growth on the client side if the client behaves inappropriate. It also >>>>>>>> splits the session table into >>>>>>>> a server and a client table which is a better implementation as the >>>>>>>> same Erlang node can be both a client and a server at the same time. So if >>>>>>>> someone feels like contributing a max limit please base it on the master >>>>>>>> branch, otherwise I suspect someone compiling about it did raise the >>>>>>>> priority level a little. >>>>>>>> >>>>>>>> >>>>>>>> Regards Ingela Erlang/OTP team - Ericsson AB >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> /Tony >>>>>>>>> >>>>>>>>> > On 5 nov 2014, at 13:08, Bogdan Andu wrote: >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > Hi, >>>>>>>>> > >>>>>>>>> > I performed a series of test regarding the an Erlang SSL server. >>>>>>>>> > >>>>>>>>> > In this setup a major role is played by the table called >>>>>>>>> > 'ssl_otp_session_cache', and of course the processes using it. >>>>>>>>> > >>>>>>>>> > The problem is that the size of table increases constantly and, >>>>>>>>> > because an ets table does not automatically deallocate memory unless the >>>>>>>>> > object are deleted from that table, the size of table remains the same even >>>>>>>>> > if there no ssl connections to server. >>>>>>>>> > >>>>>>>>> > For example, with a single client running 'ad infinitum' the >>>>>>>>> > table increases at a rate of 5 MBytes/hour. In 12 hours there are allocate >>>>>>>>> > around 60 MB of memory only for this table. >>>>>>>>> > >>>>>>>>> > Some info about this: >>>>>>>>> > $ erl -sname console@REDACTED -remsh n1@REDACTED -setcookie >>>>>>>>> > operator >>>>>>>>> > Erlang/OTP 17 [erts-6.2] [source] [64-bit] [smp:2:2] >>>>>>>>> > [async-threads:10] [kernel-poll:false] >>>>>>>>> > >>>>>>>>> > Eshell V6.2 (abort with ^G) >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > (n1@REDACTED)1> ets:i(). >>>>>>>>> > id name type size mem owner >>>>>>>>> > >>>>>>>>> > ---------------------------------------------------------------------------- >>>>>>>>> > 12 cookies set 0 291 auth >>>>>>>>> > 4111 code set 410 26132 >>>>>>>>> > code_server >>>>>>>>> > 8208 code_names set 58 7459 >>>>>>>>> > code_server >>>>>>>>> > 12307 httpc_manager__session_cookie_db bag 0 >>>>>>>>> > 291 httpc_manager >>>>>>>>> > 16404 ssl_otp_cacertificate_db set 0 291 >>>>>>>>> > ssl_manager >>>>>>>>> > 20501 ssl_otp_ca_file_ref set 0 291 >>>>>>>>> > ssl_manager >>>>>>>>> > 24598 ssl_otp_pem_cache set 3 360 >>>>>>>>> > ssl_manager >>>>>>>>> > 28695 ssl_otp_session_cache ordered_set 138057 8421893 >>>>>>>>> > ssl_manager >>>>>>>>> > 32797 dets duplicate_bag 2 308 >>>>>>>>> > dets >>>>>>>>> > 40990 ign_requests set 0 291 >>>>>>>>> > inet_gethost_native >>>>>>>>> > 45087 ign_req_index set 0 291 >>>>>>>>> > inet_gethost_native >>>>>>>>> > 2261635104 shell_records ordered_set 0 81 >>>>>>>>> > <0.30638.37> >>>>>>>>> > ac_tab ac_tab set 33 2216 >>>>>>>>> > application_controller >>>>>>>>> > code_map code_map set 100 2791 <0.72.0> >>>>>>>>> > config config set 12 892 <0.72.0> >>>>>>>>> > dets_owners dets_owners set 1 298 dets >>>>>>>>> > dets_registry dets_registry set 1 299 dets >>>>>>>>> > file_io_servers file_io_servers set 1 344 >>>>>>>>> > file_server_2 >>>>>>>>> > global_locks global_locks set 0 291 >>>>>>>>> > global_name_server >>>>>>>>> > global_names global_names set 0 291 >>>>>>>>> > global_name_server >>>>>>>>> > global_names_ext global_names_ext set 0 291 >>>>>>>>> > global_name_server >>>>>>>>> > global_pid_ids global_pid_ids bag 0 291 >>>>>>>>> > global_name_server >>>>>>>>> > global_pid_names global_pid_names bag 0 291 >>>>>>>>> > global_name_server >>>>>>>>> > httpc_manager__handler_db httpc_manager__handler_db set 0 >>>>>>>>> > 291 httpc_manager >>>>>>>>> > httpc_manager__session_db httpc_manager__session_db set 0 >>>>>>>>> > 291 httpc_manager >>>>>>>>> > inet_cache inet_cache bag 0 291 inet_db >>>>>>>>> > inet_db inet_db set 29 600 inet_db >>>>>>>>> > inet_hosts_byaddr inet_hosts_byaddr bag 0 291 >>>>>>>>> > inet_db >>>>>>>>> > inet_hosts_byname inet_hosts_byname bag 0 291 >>>>>>>>> > inet_db >>>>>>>>> > inet_hosts_file_byaddr inet_hosts_file_byaddr bag 0 291 >>>>>>>>> > inet_db >>>>>>>>> > inet_hosts_file_byname inet_hosts_file_byname bag 0 291 >>>>>>>>> > inet_db >>>>>>>>> > models models set 3 28952 <0.72.0> >>>>>>>>> > sys_dist sys_dist set 1 334 >>>>>>>>> > net_kernel >>>>>>>>> > ok >>>>>>>>> > (n1@REDACTED)5> ets:info(ssl_otp_session_cache). >>>>>>>>> > undefined >>>>>>>>> > (n1@REDACTED)7> (8421893*8)/1024. >>>>>>>>> > 65796.0390625 >>>>>>>>> > (n1@REDACTED)8> memory(). >>>>>>>>> > [{total,92699464}, >>>>>>>>> > {processes,8964000}, >>>>>>>>> > {processes_used,8963152}, >>>>>>>>> > {system,83735464}, >>>>>>>>> > {atom,429569}, >>>>>>>>> > {atom_used,421768}, >>>>>>>>> > {binary,199040}, >>>>>>>>> > {code,10411520}, >>>>>>>>> > {ets,69163032}] >>>>>>>>> > >>>>>>>>> > The memory allocated to table 'ssl_otp_session_cache' is roughly >>>>>>>>> > 64 MB in 12 hours. >>>>>>>>> > >>>>>>>>> > On an OpenBSD platform such process gets killed immediately it >>>>>>>>> > hits some memory and/or CPU limits. >>>>>>>>> > >>>>>>>>> > To make this test on OpenBSD I had to put 'infinit' to memory, >>>>>>>>> > otherwise the Erlang VM would be killed. >>>>>>>>> > >>>>>>>>> > How can one control , tweak or configure this table such that it >>>>>>>>> > does not accumulate such data at such high rate. >>>>>>>>> > >>>>>>>>> > I seems the table being created private, and there is no way to >>>>>>>>> > ets:delete_all_objects/1 from table manually. >>>>>>>>> > >>>>>>>>> > I know that this table caches some SSL data related to clients, >>>>>>>>> > but the client has the same IP address, >>>>>>>>> > and I wonder why is neccesary to store a lot of SSL connection >>>>>>>>> > info about the same client when only the ephemeral peer port >>>>>>>>> > differs? >>>>>>>>> > >>>>>>>>> > How the size of this table can be held in reasonable limits and >>>>>>>>> > the rate it's size increases ? >>>>>>>>> > >>>>>>>>> > Please if somebody shed some light on these issues. >>>>>>>>> > >>>>>>>>> > Thank you, >>>>>>>>> > >>>>>>>>> > Bogdan >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > _______________________________________________ >>>>>>>>> > erlang-questions mailing list >>>>>>>>> > erlang-questions@REDACTED >>>>>>>>> > http://erlang.org/mailman/listinfo/erlang-questions >>>>>>>>> >>>>>>>>> "Installing applications can lead to corruption over time. >>>>>>>>> Applications gradually write over each other's libraries, partial upgrades >>>>>>>>> occur, user and system errors happen, and minute changes may be unnoticeable >>>>>>>>> and difficult to fix" >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> erlang-questions mailing list >>>>>>>>> erlang-questions@REDACTED >>>>>>>>> http://erlang.org/mailman/listinfo/erlang-questions >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From essen@REDACTED Fri Nov 7 14:26:40 2014 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Fri, 07 Nov 2014 15:26:40 +0200 Subject: [erlang-questions] [ANN] erlang.mk 1.2.0 + small Cowboy/Ranch updates Message-ID: <545CC890.5090606@ninenines.eu> Hello, I have just pushed a major update to erlang.mk that I feel requires a proper announcement. erlang.mk is a fast and highly configurable GNU Make build and testing tool for Erlang: https://github.com/ninenines/erlang.mk erlang.mk 1.2.0 is made available thanks to 49 contributors. Most of the changes listed below are from user contributions; everything else is based on user feedback. Great job everyone! * Add "make erlang-mk" to automatically update erlang.mk to the latest * Add "make shell" to quickly have an Erlang shell with paths set * Add "make elvis" to run Elvis on your project * We now have 55 packages in the index * Add a script to add packages to the index easily * Add support for Mercurial * Optionally fill "id" key with "git describe" output * Compile dependencies with only +debug_info set * Add a fallback if wget is missing * Make sure tests can find the project's include files * Only download the index file when strictly necessary * Don't delete the index file on distclean if it's custom * Allow overriding the method and directories analyzed by Dialyzer * Improve output, templates, documentation We are currently working on adding a eunit plugin and escript build support (similar to rebar escriptize). Both are user contributions and should land in soon. Feedback on the eunit plugin is welcome. In addition I would like to hijack this email to announce Ranch 1.1.0 (https://github.com/ninenines/ranch/blob/master/CHANGELOG.md) and Cowboy 1.0.1 (https://github.com/ninenines/cowboy/blob/1.0.x/CHANGELOG.md). The Cowlib library also changed (https://github.com/ninenines/cowlib/blob/master/CHANGELOG.md). All these changes are small and you should be able to update from 1.0 without modifying your code. Enjoy! -- Lo?c Hoguin http://ninenines.eu From max.lapshin@REDACTED Fri Nov 7 14:45:27 2014 From: max.lapshin@REDACTED (Max Lapshin) Date: Fri, 7 Nov 2014 16:45:27 +0300 Subject: [erlang-questions] [ANN] erlang.mk 1.2.0 + small Cowboy/Ranch updates In-Reply-To: <545CC890.5090606@ninenines.eu> References: <545CC890.5090606@ninenines.eu> Message-ID: Loic, I've asked you but already forgot: is there any "standard" way to compile .c to .so with erlang.mk? There is a tricky thing in finding out location of erl_nif.h -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Fri Nov 7 14:47:27 2014 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Fri, 07 Nov 2014 15:47:27 +0200 Subject: [erlang-questions] [ANN] erlang.mk 1.2.0 + small Cowboy/Ranch updates In-Reply-To: References: <545CC890.5090606@ninenines.eu> Message-ID: <545CCD6F.8020405@ninenines.eu> There is a c_src plugin (optional, currently disabled by default), it's documented in the README. This is a project that use it: https://github.com/ninenines/esdl2 On 11/07/2014 03:45 PM, Max Lapshin wrote: > Loic, I've asked you but already forgot: is there any "standard" way to > compile .c to .so with erlang.mk ? > There is a tricky thing in finding out location of erl_nif.h -- Lo?c Hoguin http://ninenines.eu From ingela.andin@REDACTED Fri Nov 7 15:20:27 2014 From: ingela.andin@REDACTED (Ingela Andin) Date: Fri, 7 Nov 2014 15:20:27 +0100 Subject: [erlang-questions] 'ssl_otp_session_cache' ets table In-Reply-To: References: Message-ID: Hi! The following release note is from the first R15 version of ssl: "Invalidation handling of sessions could cause the time_stamp field in the session record to be set to undefined crashing the session clean up process. This did not affect the connections but would result in that the session table would grow." 2014-11-07 11:58 GMT+01:00 Bogdan Andu : > I work with R14B04 as production and OTP 17.3 as experimental. > > R14B04 also deletes all entries after 24 hours. > > 17.3 also deletes all entries after 24 hours. > > I think is it has to do with passing the environment parameter like this: > > erl -ssl session_lifetime 60 ... > > I tried many variants and may be I miss something. > > may be I am mistaken but it is possible that only the default value, 24 > hours, > is only considered as the environment variable is not passed correctly by > me. > > i think the environment parameters are the problem here because maybe i > specify them incorrctly. > > could you advise what is the proper method of correctly specifying > session_lifetime, because according to > manual is like : > > erl -ssl session_lifetime 60 ... > You can check with application:get_env(ssl, session_lifetime). to see if you succeed setting the variable. > > a ssl session must expire afte 60 seconds + delayed 60 seconds = 2 minutes. > > intstead entries from 'ssl_otp_session_cache' be deleted every 2 minutes, > they remain 24 hours . > > after 22 hours I have: > > Node: 'n1@REDACTED' (Connected) (17/6.2) unix (openbsd 5.4.0) CPU:2 SMP > +A:10 > Time: local time 12:50:23, up for 000:21:54:49, 0ms latency, > Processes: total 676 (RQ 0) at 21121 RpI using 10978.0k (11007.4k > allocated) > Memory: Sys 14898.8k, Atom 410.8k/419.5k, Bin 112.9k, Code 10053.7k, Ets > 720.6k > > > and for ets:i(). > > ........... > 28695 ssl_otp_session_cache ordered_set 115 7134 > ssl_manager > .......... > > The vm was configured with 60 seconds session lifetime and the client was > running 60 seconds > an I expected after 2 minutes to see less entries in the table but they > remained untouched > > > I will see what happens after 24 hours mark and I'll let you know > > and another thing... > > even if i pass the options: > {reuse_session, fun(_,_,_,_) -> false end}, > {reuse_sessions, false}, > > both or individually , to the ssl:listen/2 function the table is filled > with sessions and they expire after 24 hours, > only that cpu is working harder, 7-8% which means the sessions a not > reused and the ssl > handshake is happening every time. > > Was that for both R14 and R17? Regards Ingela Erlang/OTP team - Ericsson AB > > Bogdan > > On Fri, Nov 7, 2014 at 10:37 AM, Ingela Andin > wrote: > >> Hi! >> >> Which version of the ssl application are you using? The session cleanup >> was broken in one of the older versions. >> >> Regards Ingela Erlang/OTP - team Ericsson AB >> >> >> 2014-11-06 17:20 GMT+01:00 Bogdan Andu : >> >>> yes, >>> >>> in function ssl_manager:invalidate_session/4, the call >>> >>> erlang:send_after(delay_time(), self(), {delayed_clean_session, Key}), >>> does the actual deletion of session from cache >>> >>> where delay_time/0 function returns ?CLEAN_SESSION_DB by default, which >>> is 60 seconds >>> >>> >>> >>> >>> On Thu, Nov 6, 2014 at 4:51 PM, Ingela Andin >>> wrote: >>> >>>> Hi! >>>> >>>> >>>> 2014-11-06 14:39 GMT+01:00 Bogdan Andu : >>>> >>>>> Hi, >>>>> >>>>> i have tried to pass to execute th following forms of commands: >>>>> >>>>> erl -ssl session_lifetime 60 >>>>> >>>>> erl -ssl session_lifetime '60' >>>>> >>>>> erl -ssl session_lifetime '[60]' >>>>> >>>>> >>>>> in order to set a ssl session_lifetime of 60 seconds. >>>>> >>>>> in init of ssl_manager , after session_lifetime + 5 seconds a process >>>>> is created to validate or invalidate sessions. >>>>> So, after 65 seconds a foldl is called on sessions table to check >>>>> sessions after formula: >>>>> Now - TimeStamp < LifeTime >>>>> >>>>> However, even after I stop the client, the table >>>>> 'ssl_otp_session_cache' 's size remains the same. >>>>> >>>>> >>>>> After that once every ?CLEAN_SESSION_DB which is 60 seconds , the >>>>> table is sweeped to delete any expired sessions, >>>>> >>>>> but also nothing happens. >>>>> >>>>> >>>> There is also a delay in the actual deletion. In the first sweep the >>>> sessions will only be invalidated as there may already be >>>> spawned connection handlers that needs to read the session data before >>>> we may delete it. It is a performance trade off. >>>> >>>> Regards Ingela Erlang/OTP team - Ericsson AB >>>> >>>> >>>> >>>>> >>>>> On Thu, Nov 6, 2014 at 12:28 PM, Ingela Andin >>>>> wrote: >>>>> >>>>>> Hi! >>>>>> >>>>>> >>>>>> 2014-11-06 8:53 GMT+01:00 Bogdan Andu : >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> on my production servers I have relayd (on OpenBSD) daemon as a >>>>>>> reverse proxy to some webservers >>>>>>> where one can fine tune some connection parameters, as well as some >>>>>>> ssl parameters. >>>>>>> >>>>>>> I give a snippet from a relayd.conf configuration file on one of my >>>>>>> production server: >>>>>>> >>>>>>> ............. >>>>>>> # >>>>>>> # Relay and protocol for HTTP layer 7 loadbalancing and SSL >>>>>>> acceleration >>>>>>> # >>>>>>> http protocol www_ssl_prot { >>>>>>> header append "$REMOTE_ADDR" to "X-Forwarded-For" >>>>>>> header append "$SERVER_ADDR:$SERVER_PORT" to "X-Forwarded-By" >>>>>>> header change "Connection" to "close" >>>>>>> >>>>>>> response header change "Server" to "Apache 0.1" >>>>>>> >>>>>>> # Various TCP performance options >>>>>>> tcp { nodelay, sack, socket buffer 65536, backlog 128 } >>>>>>> >>>>>>> ssl { no sslv2, no sslv3, tlsv1, ciphers "HIGH" } >>>>>>> ssl session cache disable >>>>>>> } >>>>>>> >>>>>>> the last directive tells relayd not to use ssl cache. >>>>>>> >>>>>>> This configuration is working for years and relayd was restarted >>>>>>> once by accident - my fault. >>>>>>> >>>>>>> SO, y question is: >>>>>>> >>>>>>> can we have this configurable in Erlang, in other words, we might be >>>>>>> able to start an erlang vm such as: >>>>>>> >>>>>>> erl -ssl session_cache 'disable' -name x@REDACTED .... >>>>>>> >>>>>>> The ssl option session_cache can be set to disabled by default and >>>>>>> can take values either disable or enabled. >>>>>>> >>>>>>> >>>>>>> >>>>>> You can already disable the reuse of the sessions using the server >>>>>> option *{reuse_sessions, boolean()}* which default to true. >>>>>> The thing we plan to do is to have a configurable limit on the table >>>>>> size when sessions are reused. >>>>>> >>>>>> Regards Ingela Erlang/OTP team Ericsson AB >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> Just to let you know... >>>>>>> >>>>>>> The statistics of a node running from yesterday when I opened this >>>>>>> thread of discussion, using a single client: >>>>>>> >>>>>>> Node: 'n1@REDACTED' (Connected) (17/6.2) unix (openbsd 5.4.0) CPU:2 >>>>>>> SMP +A:10 >>>>>>> Time: local time 09:48:48, up for 000:20:27:49, 0ms latency, >>>>>>> Processes: total 681 (RQ 0) at 610275 RpI using 11526.4k (11805.8k >>>>>>> allocated) >>>>>>> Memory: Sys 94835.5k, Atom 407.7k/419.5k, Bin 176.4k, Code 9934.7k, >>>>>>> Ets 80739.2k >>>>>>> >>>>>>> ........... >>>>>>> >>>>>>> So, can be this made configurable? >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Wed, Nov 5, 2014 at 8:52 PM, Ingela Andin >>>>>> > wrote: >>>>>>> >>>>>>>> Hi! >>>>>>>> >>>>>>>> 2014-11-05 14:15 GMT+01:00 Tony Rogvall : >>>>>>>> >>>>>>>>> I think the ssl session times is the problem here, and the lack of >>>>>>>>> a maximum size. >>>>>>>>> >>>>>>>>> You can change the session time in the ssl environment: >>>>>>>>> session_lifetime >>>>>>>>> The default is set to 24 hours (in seconds) (if I read it >>>>>>>>> correctly, in ssl_manager.erl) >>>>>>>>> >>>>>>>> >>>>>>>> Default values are always hard. It is the maximum recommended time >>>>>>>> for a session to live >>>>>>>> according to the spec. >>>>>>>> >>>>>>>> >>>>>>>>> I guess that a session_cache_size could be a nice thing to have, >>>>>>>>> limiting the growth of the session cache. >>>>>>>>> >>>>>>>>> >>>>>>>> Yes I agree. >>>>>>>> >>>>>>>> >>>>>>>>> In other words you have to estimate the life time of your clients >>>>>>>>> and >>>>>>>>> try to find a reasonable session_lifetime to match that, without >>>>>>>>> blowing up >>>>>>>>> the system. >>>>>>>>> >>>>>>>>> Maybe the ssl_session_cache_api could be used to implement a >>>>>>>>> strategy with a max size >>>>>>>>> cache. Retire session least recently used, while performing the >>>>>>>>> update? >>>>>>>>> There is a time_stamp in the session that that could be used for >>>>>>>>> this purpose. >>>>>>>>> >>>>>>>>> OTP: Why is a fixed limit not implemented in the standard >>>>>>>>> ssl_session_cache? >>>>>>>>> Could this be a target for DOS attacks? >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> We are aware of the problem, and it is on our todo list. One reason >>>>>>>> it has not had top priority is that on the server side there are often >>>>>>>> other mechanisms >>>>>>>> like firewalls and webbserver settings that limits the problem. >>>>>>>> And the reason why it was not implemented in the first place is that >>>>>>>> implementations by nature >>>>>>>> are iterative and at first you are faced with a lot of bigger >>>>>>>> problems to solve and then you need to iterate and fine tune and fix things >>>>>>>> that you now have a better understanding of. >>>>>>>> >>>>>>>> In current master there is a change to the session table that >>>>>>>> limits the growth on the client side if the client behaves inappropriate. >>>>>>>> It also splits the session table into >>>>>>>> a server and a client table which is a better implementation as the >>>>>>>> same Erlang node can be both a client and a server at the same time. So if >>>>>>>> someone feels like contributing a max limit please base it on the master >>>>>>>> branch, otherwise I suspect someone compiling about it did raise the >>>>>>>> priority level a little. >>>>>>>> >>>>>>>> >>>>>>>> Regards Ingela Erlang/OTP team - Ericsson AB >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> /Tony >>>>>>>>> >>>>>>>>> > On 5 nov 2014, at 13:08, Bogdan Andu wrote: >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > Hi, >>>>>>>>> > >>>>>>>>> > I performed a series of test regarding the an Erlang SSL server. >>>>>>>>> > >>>>>>>>> > In this setup a major role is played by the table called >>>>>>>>> 'ssl_otp_session_cache', and of course the processes using it. >>>>>>>>> > >>>>>>>>> > The problem is that the size of table increases constantly and, >>>>>>>>> because an ets table does not automatically deallocate memory unless the >>>>>>>>> object are deleted from that table, the size of table remains the same even >>>>>>>>> if there no ssl connections to server. >>>>>>>>> > >>>>>>>>> > For example, with a single client running 'ad infinitum' the >>>>>>>>> table increases at a rate of 5 MBytes/hour. In 12 hours there are allocate >>>>>>>>> around 60 MB of memory only for this table. >>>>>>>>> > >>>>>>>>> > Some info about this: >>>>>>>>> > $ erl -sname console@REDACTED -remsh n1@REDACTED -setcookie >>>>>>>>> operator >>>>>>>>> > Erlang/OTP 17 [erts-6.2] [source] [64-bit] [smp:2:2] >>>>>>>>> [async-threads:10] [kernel-poll:false] >>>>>>>>> > >>>>>>>>> > Eshell V6.2 (abort with ^G) >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > (n1@REDACTED)1> ets:i(). >>>>>>>>> > id name type size mem owner >>>>>>>>> > >>>>>>>>> ---------------------------------------------------------------------------- >>>>>>>>> > 12 cookies set 0 291 auth >>>>>>>>> > 4111 code set 410 26132 >>>>>>>>> code_server >>>>>>>>> > 8208 code_names set 58 7459 >>>>>>>>> code_server >>>>>>>>> > 12307 httpc_manager__session_cookie_db bag 0 >>>>>>>>> 291 httpc_manager >>>>>>>>> > 16404 ssl_otp_cacertificate_db set 0 291 >>>>>>>>> ssl_manager >>>>>>>>> > 20501 ssl_otp_ca_file_ref set 0 291 >>>>>>>>> ssl_manager >>>>>>>>> > 24598 ssl_otp_pem_cache set 3 360 >>>>>>>>> ssl_manager >>>>>>>>> > 28695 ssl_otp_session_cache ordered_set 138057 >>>>>>>>> 8421893 ssl_manager >>>>>>>>> > 32797 dets duplicate_bag 2 308 >>>>>>>>> dets >>>>>>>>> > 40990 ign_requests set 0 291 >>>>>>>>> inet_gethost_native >>>>>>>>> > 45087 ign_req_index set 0 291 >>>>>>>>> inet_gethost_native >>>>>>>>> > 2261635104 shell_records ordered_set 0 81 >>>>>>>>> <0.30638.37> >>>>>>>>> > ac_tab ac_tab set 33 2216 >>>>>>>>> application_controller >>>>>>>>> > code_map code_map set 100 2791 <0.72.0> >>>>>>>>> > config config set 12 892 <0.72.0> >>>>>>>>> > dets_owners dets_owners set 1 298 dets >>>>>>>>> > dets_registry dets_registry set 1 299 dets >>>>>>>>> > file_io_servers file_io_servers set 1 344 >>>>>>>>> file_server_2 >>>>>>>>> > global_locks global_locks set 0 291 >>>>>>>>> global_name_server >>>>>>>>> > global_names global_names set 0 291 >>>>>>>>> global_name_server >>>>>>>>> > global_names_ext global_names_ext set 0 291 >>>>>>>>> global_name_server >>>>>>>>> > global_pid_ids global_pid_ids bag 0 291 >>>>>>>>> global_name_server >>>>>>>>> > global_pid_names global_pid_names bag 0 291 >>>>>>>>> global_name_server >>>>>>>>> > httpc_manager__handler_db httpc_manager__handler_db set 0 >>>>>>>>> 291 httpc_manager >>>>>>>>> > httpc_manager__session_db httpc_manager__session_db set 0 >>>>>>>>> 291 httpc_manager >>>>>>>>> > inet_cache inet_cache bag 0 291 inet_db >>>>>>>>> > inet_db inet_db set 29 600 inet_db >>>>>>>>> > inet_hosts_byaddr inet_hosts_byaddr bag 0 291 >>>>>>>>> inet_db >>>>>>>>> > inet_hosts_byname inet_hosts_byname bag 0 291 >>>>>>>>> inet_db >>>>>>>>> > inet_hosts_file_byaddr inet_hosts_file_byaddr bag 0 291 >>>>>>>>> inet_db >>>>>>>>> > inet_hosts_file_byname inet_hosts_file_byname bag 0 291 >>>>>>>>> inet_db >>>>>>>>> > models models set 3 28952 <0.72.0> >>>>>>>>> > sys_dist sys_dist set 1 334 >>>>>>>>> net_kernel >>>>>>>>> > ok >>>>>>>>> > (n1@REDACTED)5> ets:info(ssl_otp_session_cache). >>>>>>>>> > undefined >>>>>>>>> > (n1@REDACTED)7> (8421893*8)/1024. >>>>>>>>> > 65796.0390625 >>>>>>>>> > (n1@REDACTED)8> memory(). >>>>>>>>> > [{total,92699464}, >>>>>>>>> > {processes,8964000}, >>>>>>>>> > {processes_used,8963152}, >>>>>>>>> > {system,83735464}, >>>>>>>>> > {atom,429569}, >>>>>>>>> > {atom_used,421768}, >>>>>>>>> > {binary,199040}, >>>>>>>>> > {code,10411520}, >>>>>>>>> > {ets,69163032}] >>>>>>>>> > >>>>>>>>> > The memory allocated to table 'ssl_otp_session_cache' is roughly >>>>>>>>> 64 MB in 12 hours. >>>>>>>>> > >>>>>>>>> > On an OpenBSD platform such process gets killed immediately it >>>>>>>>> hits some memory and/or CPU limits. >>>>>>>>> > >>>>>>>>> > To make this test on OpenBSD I had to put 'infinit' to memory, >>>>>>>>> otherwise the Erlang VM would be killed. >>>>>>>>> > >>>>>>>>> > How can one control , tweak or configure this table such that it >>>>>>>>> does not accumulate such data at such high rate. >>>>>>>>> > >>>>>>>>> > I seems the table being created private, and there is no way to >>>>>>>>> ets:delete_all_objects/1 from table manually. >>>>>>>>> > >>>>>>>>> > I know that this table caches some SSL data related to clients, >>>>>>>>> but the client has the same IP address, >>>>>>>>> > and I wonder why is neccesary to store a lot of SSL connection >>>>>>>>> info about the same client when only the ephemeral peer port >>>>>>>>> > differs? >>>>>>>>> > >>>>>>>>> > How the size of this table can be held in reasonable limits and >>>>>>>>> the rate it's size increases ? >>>>>>>>> > >>>>>>>>> > Please if somebody shed some light on these issues. >>>>>>>>> > >>>>>>>>> > Thank you, >>>>>>>>> > >>>>>>>>> > Bogdan >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > _______________________________________________ >>>>>>>>> > erlang-questions mailing list >>>>>>>>> > erlang-questions@REDACTED >>>>>>>>> > http://erlang.org/mailman/listinfo/erlang-questions >>>>>>>>> >>>>>>>>> "Installing applications can lead to corruption over time. >>>>>>>>> Applications gradually write over each other's libraries, partial upgrades >>>>>>>>> occur, user and system errors happen, and minute changes may be >>>>>>>>> unnoticeable and difficult to fix" >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> erlang-questions mailing list >>>>>>>>> erlang-questions@REDACTED >>>>>>>>> http://erlang.org/mailman/listinfo/erlang-questions >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthias@REDACTED Fri Nov 7 15:40:13 2014 From: matthias@REDACTED (Matthias Lang) Date: Fri, 7 Nov 2014 15:40:13 +0100 Subject: [erlang-questions] erlang:restart_timer In-Reply-To: References: Message-ID: <20141107144013.GA20188@corelatus.se> On Wednesday, November 05, Tony Rogvall wrote: > I propose a new BIF: > > erlang:restart_timer(Ref, Time) -> false | RemainingTime ... > What about it? I'd like to hear from people actually implementing > protocols, not only from the "normal" responders. TL;DR: Looks logical, will make some of my code slightly simpler, but I don't have enough of this code for it to make a practical difference. All the code where performance really matters is in C, assembler or in an FPGA. I took a look through my production (i.e. not side projects or tests) code. I have two places in my Erlang code which use the pattern you described, i.e. a recurring event (e.g. receiving a packet) resets a timer; the timer is "always" running. Neither is performance-sensitive. I have far more places in the code where I do something and start a timer in case the operation hangs. There's no restarting in that situation. Most of those timers are of the 'receive....after' sort. I also took a look through Vance Shipley's LAPD code. I can't find a git repo of it, but here's a google code one: https://code.google.com/p/lapderl/source/browse/?r=9#svn%2Fbranches%2FMOTIVITY%2Fsrc. As far as I can see (I took a quick look), all the timers it uses are the 'gen_fsm' ones, i.e. he calls gen_fsm:cancel_timer(). One example is LAPD T203, the maximum time allowed without a frame. It gets reset every time you send a packet. Matt From darach@REDACTED Fri Nov 7 15:51:20 2014 From: darach@REDACTED (Darach Ennis) Date: Fri, 7 Nov 2014 14:51:20 +0000 Subject: [erlang-questions] [ANN] erlang.mk 1.2.0 + small Cowboy/Ranch updates In-Reply-To: <545CCD6F.8020405@ninenines.eu> References: <545CC890.5090606@ninenines.eu> <545CCD6F.8020405@ninenines.eu> Message-ID: Hi guys, This would be great(er) as a default but good to know! Cheers, Darach. On 7 Nov 2014 13:47, "Lo?c Hoguin" wrote: > There is a c_src plugin (optional, currently disabled by default), it's > documented in the README. This is a project that use it: > https://github.com/ninenines/esdl2 > > On 11/07/2014 03:45 PM, Max Lapshin wrote: > >> Loic, I've asked you but already forgot: is there any "standard" way to >> compile .c to .so with erlang.mk ? >> There is a tricky thing in finding out location of erl_nif.h >> > > -- > Lo?c Hoguin > http://ninenines.eu > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Fri Nov 7 15:54:48 2014 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Fri, 07 Nov 2014 16:54:48 +0200 Subject: [erlang-questions] [ANN] erlang.mk 1.2.0 + small Cowboy/Ranch updates In-Reply-To: References: <545CC890.5090606@ninenines.eu> <545CCD6F.8020405@ninenines.eu> Message-ID: <545CDD38.9090600@ninenines.eu> Someone sent a patch today that would allow including it by default (among other improvements). So it should be included by default in a few days. On 11/07/2014 04:51 PM, Darach Ennis wrote: > Hi guys, > > This would be great(er) as a default but good to know! > > Cheers, > > Darach. > > On 7 Nov 2014 13:47, "Lo?c Hoguin" > wrote: > > There is a c_src plugin (optional, currently disabled by default), > it's documented in the README. This is a project that use it: > https://github.com/ninenines/__esdl2 > > > On 11/07/2014 03:45 PM, Max Lapshin wrote: > > Loic, I've asked you but already forgot: is there any > "standard" way to > compile .c to .so with erlang.mk > ? > There is a tricky thing in finding out location of erl_nif.h > > > -- > Lo?c Hoguin > http://ninenines.eu > _________________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/__listinfo/erlang-questions > > -- Lo?c Hoguin http://ninenines.eu From darach@REDACTED Fri Nov 7 15:55:50 2014 From: darach@REDACTED (Darach Ennis) Date: Fri, 7 Nov 2014 14:55:50 +0000 Subject: [erlang-questions] [ANN] erlang.mk 1.2.0 + small Cowboy/Ranch updates In-Reply-To: <545CDD38.9090600@ninenines.eu> References: <545CC890.5090606@ninenines.eu> <545CCD6F.8020405@ninenines.eu> <545CDD38.9090600@ninenines.eu> Message-ID: <3 :) On Fri, Nov 7, 2014 at 2:54 PM, Lo?c Hoguin wrote: > Someone sent a patch today that would allow including it by default (among > other improvements). So it should be included by default in a few days. > > On 11/07/2014 04:51 PM, Darach Ennis wrote: > >> Hi guys, >> >> This would be great(er) as a default but good to know! >> >> Cheers, >> >> Darach. >> >> On 7 Nov 2014 13:47, "Lo?c Hoguin" > > wrote: >> >> There is a c_src plugin (optional, currently disabled by default), >> it's documented in the README. This is a project that use it: >> https://github.com/ninenines/__esdl2 >> >> >> On 11/07/2014 03:45 PM, Max Lapshin wrote: >> >> Loic, I've asked you but already forgot: is there any >> "standard" way to >> compile .c to .so with erlang.mk >> ? >> There is a tricky thing in finding out location of erl_nif.h >> >> >> -- >> Lo?c Hoguin >> http://ninenines.eu >> _________________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/__listinfo/erlang-questions >> >> >> > -- > Lo?c Hoguin > http://ninenines.eu > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rich.neswold@REDACTED Fri Nov 7 16:30:14 2014 From: rich.neswold@REDACTED (Rich Neswold) Date: Fri, 7 Nov 2014 09:30:14 -0600 Subject: [erlang-questions] erlang:restart_timer In-Reply-To: References: Message-ID: On Wed, Nov 5, 2014 at 7:50 AM, Tony Rogvall wrote: > I propose a new BIF: > > erlang:restart_timer(Ref, Time) -> false | RemainingTime > > This will restart a time if it is active and return the time that remained > when it was restarted. > If restart_timer return false either the timer has already time out or the > reference is not a timer. > I like the proposal. However, is it too "imperative" of an interface? If a process saves a timer reference and the timer never seems to fire, you have to determine whether it was cancelled or reset. One of the strengths of immutability is that you can always tell where a variable received its value. By reusing the same timer ref, a long lived value may go through many state changes which may make it harder to find bugs related to the timer. I guess erlang:cancel_timer/1 already changes the state of a timer, so this proposal isn't setting the precedent, just expanding it. To keep it more functional, it should be more like: erlang:restart_timer(ORef, Time) -> { NRef, false | RemainingTime } The old timer gets cancelled, any messages from the old timer are removed from the queue, and then new timer is created. A new ref is always returned and the second value indicates the end state of the previous timer: 'false', if it expired, or the time remaining. -- Rich -------------- next part -------------- An HTML attachment was scrubbed... URL: From kevprg@REDACTED Fri Nov 7 16:00:54 2014 From: kevprg@REDACTED (kab) Date: Fri, 7 Nov 2014 10:00:54 -0500 Subject: [erlang-questions] newbie question In-Reply-To: References: <545BC45B.9020207@earthlink.net> <545BC74F.5090508@2600hz.com> <545BCB6D.3040409@anydata.co.uk> Message-ID: <20141107145805.GA4786@localhost.localdomain> I'm a noob trying to learn erlang too. I took Charles script and applied the feedback. The only part I'm not grokking is why re:split wants a 3 part list on the left hand side. I expected it to want two parts; one for the string matched and the second for the remaining string. Is there something about the re that is matching an empty string? -module(chunks). main(_) -> io:format("Chunks Forward~n"), CF = chunks_fwd("Able was I ere I saw Elba."), lists:foreach(fun(A) -> io:format("~p~n", [A]) end, CF), io:format("Chunks Reverse~n"), CR = chunks_rev("Able was I ere I saw Elba."), lists:foreach(fun(A) -> io:format("~p~n", [A]) end, CR) . chunks_fwd("") -> []; chunks_fwd(Str)-> P1 = "^([[:alnum:]]+|[[:space:]]+|[[:punct:]]+|[[:cntrl:]]+|.)", [[], M1, T1] = re:split(Str, P1, [{return, list}]), [M1 | chunks_fwd(T1)] . chunks_rev(Str) -> chunks_rev(Str, []). chunks_rev("", Lst) -> Lst; chunks_rev(Str, Lst)-> P1 = "^([[:alnum:]]+|[[:space:]]+|[[:punct:]]+|[[:cntrl:]]+|.)", [[], M1, T1] = re:split(Str, P1, [{return, list}]), chunks_rev(T1, [M1|Lst]) . On Thu, Nov 06, 2014 at 08:58:35PM +0100, Erik S?e S?rensen wrote: > Hi there - > > Apart from the atom-vs-Variable issue, I think you got your recursion wrong. > > [M1 | chunks(T1, P1)] > > doesn't make too much sense (that I can see) - especially the "recurse with > P1 for Lst" part. > > I think you mean either > > chunks(T1, [M1|Lst]) > > for a tail recursive loop, or > > chunks("") -> []; > > chunks(Str) -> ..., [M1 | chunks(T1)] > > for "normal" recursion on the rest. At present, you code looks like a > confusion of the two (apart from the Lst/P1 difference). > > > Regards, and have fun on the learning curve, > > Erik > > > Den 06/11/2014 20.27 skrev "Fred Youhanaie" : > > > > > and of course the same goes for lst in chunks/2 :) > > > > On 06/11/14 19:09, James Aimonetti wrote: > > > >> -----BEGIN PGP SIGNED MESSAGE----- > >> Hash: SHA1 > >> > >> str should be Str. Variables start with a capital letter. > >> > >> Le 2014-11-06 10:56, Charles Hixson a ?crit : > >> > >>> Given the routine: -module(chunks). -export([chunks/1]). > >>> chunks(str)-> chunks(str, []). > >>> > >>> chunks("", lst)-> list:reverse(lst); chunks(str, lst)-> P1 = > >>> "^([[:alnum:]]+|[[:space:]]+|[[:punct:]]+|[[:cntrl:]]+|.)", [[], > >>> M1, T1] = re:split(str, P1, [{return, list}]), [M1 | chunks(T1, > >>> P1)]. > >>> > >>> and the shell: 47> c(chunks). {ok,chunks} > >>> > >>> Why do I get: 49> chunks:chunks("Able was I ere I saw Elba."). ** > >>> exception error: no function clause matching chunks:chunks("Able > >>> was I ere I saw Elba.") (chunks.erl, line 15) (Well, forget the > >>> like 15. I took out a bunch of comments.) > >>> > >> _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From bob@REDACTED Fri Nov 7 17:34:30 2014 From: bob@REDACTED (Bob Ippolito) Date: Fri, 7 Nov 2014 08:34:30 -0800 Subject: [erlang-questions] =?utf-8?b?RXVybyBTeW1ib2wgKOKCrCkgY29udmVyc2lv?= =?utf-8?q?n_from_HTTP_API_to_SMPP_through_mochiweb?= In-Reply-To: References: Message-ID: I don't know anything about SMPP encoding, so I don't have that answer for you, but I've told you everything you need to know related to mochiweb and UTF8. Now that you have a list of unicode codepoints I'm sure you can find a way to translate that to SMPP bytes, even if you have to implement the encoding table yourself. On Fri, Nov 7, 2014 at 1:36 AM, Sanath Prasanna wrote: > Dear Rob, > Tx for example. I got binary values (E2,82,AC) in my HTTP server module > when submit above using URL. (when used InputParas = Req:parse_qs() ) > I want to replace E2,82,AC from 1B,65 in http server module, before > sending to SMPP module. (which is responsible to deliver SMS) > Br, > Sanath Robert > > > On Fri, Nov 7, 2014 at 2:33 PM, Bob Ippolito wrote: > >> 1> >> unicode:characters_to_list(iolist_to_binary(mochiweb_util:unquote("%E2%82%AC"))). >> [8364] >> >> On Fri, Nov 7, 2014 at 12:21 AM, Sanath Prasanna >> wrote: >> >>> Hi Bob, >>> Tx for quick reply. I go through & unable to find direct function to do >>> that.If you never mind, can you elaborate with example.? >>> Br, >>> Sanath Roberto >>> >>> On Fri, Nov 7, 2014 at 10:52 AM, Bob Ippolito wrote: >>> >>>> Use the unicode module to decode the UTF8 if you need a string of >>>> Unicode code points. If you need to decode a URL encoded string then you >>>> can use mochiweb_util for that. >>>> >>>> >>>> On Thursday, November 6, 2014, Sanath Prasanna >>>> wrote: >>>> >>>>> Hi, >>>>> I used mochiweb web server to handle http request. I submit messages >>>>> with euro - ? as "%E2%82%AC" thorugh HTTP API. I need to forward this to >>>>> SMPP connction. Before that I want to convert that to 1B65. Otherwise ? >>>>> character is not display in mobile. >>>>> How to do that ? Appreciate if some can help to do that. >>>>> Br, >>>>> Sanath Roberto >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From icfp.publicity@REDACTED Fri Nov 7 17:43:40 2014 From: icfp.publicity@REDACTED (David Van Horn) Date: Fri, 7 Nov 2014 11:43:40 -0500 Subject: [erlang-questions] ICFP 2015: Call for Papers Message-ID: ===================================================================== 20th ACM SIGPLAN International Conference on Functional Programming ICFP 2015 Vancouver, Canada, August 31 - September 2, 2015 http://www.icfpconference.org/icfp2015 ===================================================================== Important Dates ~~~~~~~~~~~~~~~ Submissions due: Friday, February 27 2015, 23:59 UTC-11 Author response: Tuesday, April 21, 2015 through Thursday, 23 April, 2015 Notification: Friday, May 1, 2015 Final copy due: Friday, June 12, 2015 Scope ~~~~~ ICFP 2015 seeks original papers on the art and science of functional programming. Submissions are invited on all topics from principles to practice, from foundations to features, and from abstraction to application. The scope includes all languages that encourage functional programming, including both purely applicative and imperative languages, as well as languages with objects, concurrency, or parallelism. Topics of interest include (but are not limited to): * Language Design: concurrency, parallelism, and distribution; modules; components and composition; metaprogramming; type systems; interoperability; domain-specific languages; and relations to imperative, object-oriented, or logic programming. * Implementation: abstract machines; virtual machines; interpretation; compilation; compile-time and run-time optimization; garbage collection and memory management; multi-threading; exploiting parallel hardware; interfaces to foreign functions, services, components, or low-level machine resources. * Software-Development Techniques: algorithms and data structures; design patterns; specification; verification; validation; proof assistants; debugging; testing; tracing; profiling. * Foundations: formal semantics; lambda calculus; rewriting; type theory; monads; continuations; control; state; effects; program verification; dependent types. * Analysis and Transformation: control-flow; data-flow; abstract interpretation; partial evaluation; program calculation. * Applications: symbolic computing; formal-methods tools; artificial intelligence; systems programming; distributed-systems and web programming; hardware design; databases; XML processing; scientific and numerical computing; graphical user interfaces; multimedia and 3D graphics programming; scripting; system administration; security. * Education: teaching introductory programming; parallel programming; mathematical proof; algebra. * Functional Pearls: elegant, instructive, and fun essays on functional programming. * Experience Reports: short papers that provide evidence that functional programming really works or describe obstacles that have kept it from working. If you are concerned about the appropriateness of some topic, do not hesitate to contact the program chair. Abbreviated instructions for authors ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ * By Friday, 27 February 2015, 23:59 UTC-11 (anywhere in the world), submit a full paper of at most 12 pages (6 pages for an Experience Report) in standard ACM conference format, including bibliography, figures, and appendices. The deadlines will be strictly enforced and papers exceeding the page limits will be summarily rejected. * Authors have the option to attach supplementary material to a submission, on the understanding that reviewers may choose not to look at it. * Each submission must adhere to SIGPLAN's republication policy, as explained on the web at http://www.sigplan.org/Resources/Policies/Republication * Authors of resubmitted (but previously rejected) papers have the option to attach an annotated copy of the reviews of their previous submission(s), explaining how they have addressed these previous reviews in the present submission. If a reviewer identifies him/herself as a reviewer of this previous submission and wishes to see how his/her comments have been addressed, the program chair will communicate to this reviewer the annotated copy of his/her previous review. Otherwise, no reviewer will read the annotated copies of the previous reviews. Overall, a submission will be evaluated according to its relevance, correctness, significance, originality, and clarity. It should explain its contributions in both general and technical terms, clearly identifying what has been accomplished, explaining why it is significant, and comparing it with previous work. The technical content should be accessible to a broad audience. Functional Pearls and Experience Reports are separate categories of papers that need not report original research results and must be marked as such at the time of submission. Detailed guidelines on both categories are on the conference web site. Proceedings will be published by ACM Press. Authors of accepted submissions will have a choice of one of three ways to manage their publication rights. These choices are described at http://authors.acm.org/main.html Presentations will be videotaped and released online if the presenter consents. The proceedings will be freely available for download from the ACM Digital Library from one week before the start of the conference until two weeks after the conference. Formatting: Submissions must be in PDF format printable in black and white on US Letter sized paper and interpretable by Ghostscript. Papers must adhere to the standard ACM conference format: two columns, nine-point font on a ten-point baseline, with columns 20pc (3.33in) wide and 54pc (9in) tall, with a column gutter of 2pc (0.33in). A suitable document template for LaTeX is available at http://www.acm.org/sigs/sigplan/authorInformation.htm Submission: Submissions will be accepted on the web using a link that will be posted at http://icfpconference.org/icfp2015/ Improved versions of a paper may be submitted at any point before the submission deadline using the same web interface. Author response: Authors will have a 72-hour period, starting at 0:00 UTC on Tuesday, 21 April 2015, to read reviews and respond to them. ACM Author-Izer is a unique service that enables ACM authors to generate and post links on either their home page or institutional repository for visitors to download the definitive version of their articles from the ACM Digital Library at no charge. Downloads through Author-Izer links are captured in official ACM statistics, improving the accuracy of usage and impact measurements. Consistently linking the definitive version of ACM article should reduce user confusion over article versioning. After your article has been published and assigned to your ACM Author Profile page, please visit http://www.acm.org/publications/acm-author-izer-service to learn how to create your links for free downloads from the ACM DL. Publication date: The official publication date of accepted papers is the date the proceedings are made available in the ACM Digital Library. This date may be up to two weeks prior to the first day of the conference. The official publication date affects the deadline for any patent filings related to published work. General Chair: Kathleen Fisher Tufts University (USA) Program Chair: John Reppy University of Chicago (USA) Program Committee: Amal Ahmed Northeastern University (USA) Jean-Philippe Bernardy Chalmers University of Technology (Sweden) Matthias Blume Google (USA) William Byrd University of Utah (USA) Andy Gill University of Kansas (USA) Neal Glew Google (USA) Fritz Henglein University of Copenhagen (Denmark) Gabriele Keller University of New South Wales and NICTA (Australia) Andrew Kennedy Microsoft Research Cambridge (UK) Neelakantan Krishnaswami Birmingham University (UK) Daan Leijen Microsoft Research Redmond (USA) Keiko Nakata Institute of Cybernetics at Tallinn University of Technology (Estonia) Mike Rainey INRIA Rocquencourt (France) Andreas Rossberg Google (Germany) Manuel Serrano INRIA Sophia Antipolis (France) Simon Thompson University of Kent (UK) David Van Horn University of Maryland (USA) Stephanie Weirich University of Pennsylvania (USA) From stu.bailey@REDACTED Fri Nov 7 18:22:36 2014 From: stu.bailey@REDACTED (Stu Bailey) Date: Fri, 7 Nov 2014 09:22:36 -0800 Subject: [erlang-questions] Performance question In-Reply-To: <545C946F.4010605@ninenines.eu> References: <545C946F.4010605@ninenines.eu> Message-ID: I'm not planning to spend a lot of time on this right now, but the binary:replace(...) was chewing a tremendous amount of system time CPU load (and actually never finished before I got frustrated and killed it) and my function was reporting the CPU load as 99% user time (not system time) and finished in a reasonable time. I assume the high system time usage for binary:replace(..) is because binary:replace(...) is doing something manic with system calls for memory management or something? On Fri, Nov 7, 2014 at 1:44 AM, Lo?c Hoguin wrote: > binary:split and binary:replace, unlike other functions of the binary > module, are normal Erlang functions. They also process a list of options > before doing the actual work, so there's an obvious overhead compared to > not doing that. In addition as has been pointed out, your code is more > specialized so that helps too. > > On 11/07/2014 03:33 AM, Stu Bailey wrote: > >> I found >> >> binary:replace(BinChunk,<<"\n">>,<<>>,[global]). >> >> /significantly /slower than >> >> remove_pattern(BinChunk,<<>>,<<"\n">>). >> >> with >> >> remove_pattern(<<>>,Acc,_BinPat) -> >> Acc; >> remove_pattern(Bin,Acc,BinPat)-> >> <> = Bin, >> case Byte == BinPat of >> true -> remove_pattern(Rest,Acc,BinPat); >> false -> remove_pattern(Rest,<>,BinPat) >> end. >> >> That was surprising to me. The built-in binary:replace() was much much >> slower for larger BinChunk with lots of <<"\n">> sprinkled through. >> >> Thoughts? >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -- > Lo?c Hoguin > http://ninenines.eu > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stu.bailey@REDACTED Fri Nov 7 18:33:29 2014 From: stu.bailey@REDACTED (Stu Bailey) Date: Fri, 7 Nov 2014 09:33:29 -0800 Subject: [erlang-questions] Performance question In-Reply-To: References: <545C946F.4010605@ninenines.eu> Message-ID: FYI, if you want to try to replicate it, I was processing ~80 chunks of binary where each chunk was about ~250,000,000 bytes. I think you'll see the difference on just one chunk. I happen to running on a 8-core MacBook Pro with 16GB Ram and therefore spawned a process per chunk to grab all the resources on all the cores. With the hand written function, it worked like a charm...yay Erlang! :-) I love seeing a few lines of code effectively use all processing power available. Heats the machine up quite a bit too. :-) On Fri, Nov 7, 2014 at 9:22 AM, Stu Bailey wrote: > I'm not planning to spend a lot of time on this right now, but the > binary:replace(...) was chewing a tremendous amount of system time CPU load > (and actually never finished before I got frustrated and killed it) and my > function was reporting the CPU load as 99% user time (not system time) and > finished in a reasonable time. I assume the high system time usage for > binary:replace(..) is because binary:replace(...) is doing something manic > with system calls for memory management or something? > > > On Fri, Nov 7, 2014 at 1:44 AM, Lo?c Hoguin wrote: > >> binary:split and binary:replace, unlike other functions of the binary >> module, are normal Erlang functions. They also process a list of options >> before doing the actual work, so there's an obvious overhead compared to >> not doing that. In addition as has been pointed out, your code is more >> specialized so that helps too. >> >> On 11/07/2014 03:33 AM, Stu Bailey wrote: >> >>> I found >>> >>> binary:replace(BinChunk,<<"\n">>,<<>>,[global]). >>> >>> /significantly /slower than >>> >>> remove_pattern(BinChunk,<<>>,<<"\n">>). >>> >>> with >>> >>> remove_pattern(<<>>,Acc,_BinPat) -> >>> Acc; >>> remove_pattern(Bin,Acc,BinPat)-> >>> <> = Bin, >>> case Byte == BinPat of >>> true -> remove_pattern(Rest,Acc,BinPat); >>> false -> remove_pattern(Rest,<>,BinPat) >>> end. >>> >>> That was surprising to me. The built-in binary:replace() was much much >>> slower for larger BinChunk with lots of <<"\n">> sprinkled through. >>> >>> Thoughts? >>> >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >> -- >> Lo?c Hoguin >> http://ninenines.eu >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Fri Nov 7 18:44:04 2014 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Fri, 07 Nov 2014 19:44:04 +0200 Subject: [erlang-questions] Performance question In-Reply-To: References: <545C946F.4010605@ninenines.eu> Message-ID: <545D04E4.4060902@ninenines.eu> Based on the code at https://github.com/erlang/otp/blob/maint/lib/stdlib/src/binary.erl#L268 It does a lot of splitting, and then a lot more splitting, and then call iolist_to_binary. It looks very inefficient. Your solution is the fastest way to do it. You also benefit from match context optimization and so your code is very fast. The only thing that could make it faster is if memory was allocated only once for the resulting binary (instead of realloc a few times)... but maybe there's already an optimization like this? On 11/07/2014 07:33 PM, Stu Bailey wrote: > FYI, if you want to try to replicate it, I was processing ~80 chunks of > binary where each chunk was about ~250,000,000 bytes. I think you'll > see the difference on just one chunk. I happen to running on a 8-core > MacBook Pro with 16GB Ram and therefore spawned a process per chunk to > grab all the resources on all the cores. With the hand written > function, it worked like a charm...yay Erlang! :-) I love seeing a few > lines of code effectively use all processing power available. Heats the > machine up quite a bit too. :-) > > On Fri, Nov 7, 2014 at 9:22 AM, Stu Bailey > wrote: > > I'm not planning to spend a lot of time on this right now, but the > binary:replace(...) was chewing a tremendous amount of system time > CPU load (and actually never finished before I got frustrated and > killed it) and my function was reporting the CPU load as 99% user > time (not system time) and finished in a reasonable time. I assume > the high system time usage for binary:replace(..) is because > binary:replace(...) is doing something manic with system calls for > memory management or something? > > > On Fri, Nov 7, 2014 at 1:44 AM, Lo?c Hoguin > wrote: > > binary:split and binary:replace, unlike other functions of the > binary module, are normal Erlang functions. They also process a > list of options before doing the actual work, so there's an > obvious overhead compared to not doing that. In addition as has > been pointed out, your code is more specialized so that helps too. > > On 11/07/2014 03:33 AM, Stu Bailey wrote: > > I found > > binary:replace(BinChunk,<<"\n"__>>,<<>>,[global]). > > /significantly /slower than > > remove_pattern(BinChunk,<<>>,<__<"\n">>). > > with > > remove_pattern(<<>>,Acc,___BinPat) -> > Acc; > remove_pattern(Bin,Acc,BinPat)__-> > <> = Bin, > case Byte == BinPat of > true -> remove_pattern(Rest,Acc,__BinPat); > false -> > remove_pattern(Rest,<>,BinPat) > end. > > That was surprising to me. The built-in binary:replace() > was much much > slower for larger BinChunk with lots of <<"\n">> sprinkled > through. > > Thoughts? > > > _________________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/__listinfo/erlang-questions > > > > -- > Lo?c Hoguin > http://ninenines.eu > > > -- Lo?c Hoguin http://ninenines.eu From stu.bailey@REDACTED Fri Nov 7 18:50:31 2014 From: stu.bailey@REDACTED (Stu Bailey) Date: Fri, 7 Nov 2014 09:50:31 -0800 Subject: [erlang-questions] Performance question In-Reply-To: <545D04E4.4060902@ninenines.eu> References: <545C946F.4010605@ninenines.eu> <545D04E4.4060902@ninenines.eu> Message-ID: Thank you for the feedback. That's very helpful. On Fri, Nov 7, 2014 at 9:44 AM, Lo?c Hoguin wrote: > Based on the code at > > https://github.com/erlang/otp/blob/maint/lib/stdlib/src/binary.erl#L268 > > It does a lot of splitting, and then a lot more splitting, and then call > iolist_to_binary. It looks very inefficient. > > Your solution is the fastest way to do it. You also benefit from match > context optimization and so your code is very fast. The only thing that > could make it faster is if memory was allocated only once for the resulting > binary (instead of realloc a few times)... but maybe there's already an > optimization like this? > > On 11/07/2014 07:33 PM, Stu Bailey wrote: > >> FYI, if you want to try to replicate it, I was processing ~80 chunks of >> binary where each chunk was about ~250,000,000 bytes. I think you'll >> see the difference on just one chunk. I happen to running on a 8-core >> MacBook Pro with 16GB Ram and therefore spawned a process per chunk to >> grab all the resources on all the cores. With the hand written >> function, it worked like a charm...yay Erlang! :-) I love seeing a few >> lines of code effectively use all processing power available. Heats the >> machine up quite a bit too. :-) >> >> On Fri, Nov 7, 2014 at 9:22 AM, Stu Bailey > > wrote: >> >> I'm not planning to spend a lot of time on this right now, but the >> binary:replace(...) was chewing a tremendous amount of system time >> CPU load (and actually never finished before I got frustrated and >> killed it) and my function was reporting the CPU load as 99% user >> time (not system time) and finished in a reasonable time. I assume >> the high system time usage for binary:replace(..) is because >> binary:replace(...) is doing something manic with system calls for >> memory management or something? >> >> >> On Fri, Nov 7, 2014 at 1:44 AM, Lo?c Hoguin > > wrote: >> >> binary:split and binary:replace, unlike other functions of the >> binary module, are normal Erlang functions. They also process a >> list of options before doing the actual work, so there's an >> obvious overhead compared to not doing that. In addition as has >> been pointed out, your code is more specialized so that helps too. >> >> On 11/07/2014 03:33 AM, Stu Bailey wrote: >> >> I found >> >> binary:replace(BinChunk,<<"\n"__>>,<<>>,[global]). >> >> /significantly /slower than >> >> remove_pattern(BinChunk,<<>>,<__<"\n">>). >> >> with >> >> remove_pattern(<<>>,Acc,___BinPat) -> >> Acc; >> remove_pattern(Bin,Acc,BinPat)__-> >> <> = Bin, >> case Byte == BinPat of >> true -> remove_pattern(Rest,Acc,__BinPat); >> false -> >> remove_pattern(Rest,<>,BinPat) >> end. >> >> That was surprising to me. The built-in binary:replace() >> was much much >> slower for larger BinChunk with lots of <<"\n">> sprinkled >> through. >> >> Thoughts? >> >> >> _________________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED > erlang.org> >> http://erlang.org/mailman/__listinfo/erlang-questions >> >> >> >> -- >> Lo?c Hoguin >> http://ninenines.eu >> >> >> >> > -- > Lo?c Hoguin > http://ninenines.eu > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dm.klionsky@REDACTED Fri Nov 7 21:23:40 2014 From: dm.klionsky@REDACTED (Dmitry Klionsky) Date: Fri, 07 Nov 2014 23:23:40 +0300 Subject: [erlang-questions] =?windows-1252?q?Euro_Symbol_=28=80=29_convers?= =?windows-1252?q?ion_from_HTTP_API_to_SMPP_through_mochiweb?= In-Reply-To: References: Message-ID: <545D2A4C.50505@gmail.com> On 11/07/2014 05:07 AM, Sanath Prasanna wrote: > Hi, > I used mochiweb web server to handle http request. I submit messages > with euro - ? as "%E2%82%AC" thorugh HTTP API. I need to forward this > to SMPP connction. Before that I want to convert that to 1B65. > Otherwise ? character is not display in mobile. > How to do that ? Appreciate if some can help to do that. > Br, > Sanath Roberto > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions Hi, You need to convert from UTF8 to GSM 03.38. Use this library https://github.com/iamaleksey/gsm0338. 3> gsm0338:from_utf8("?"). <<"\ee">> -- Best regards, Dmitry Klionsky -------------- next part -------------- An HTML attachment was scrubbed... URL: From darach@REDACTED Fri Nov 7 22:08:07 2014 From: darach@REDACTED (Darach Ennis) Date: Fri, 7 Nov 2014 21:08:07 +0000 Subject: [erlang-questions] [ANN] jch-erl 0.1 - Fast minimal memory consistent hashing for Erlang/OTP. Message-ID: Hey folk. jch-erl 0.1 freshly baked. NIF wrapper for Jump Consistent Hash algorithm by John Lamping and Eric Veach developed at Google, Inc. Paper: "A Fast, Minimal Memory, Consistent Hash Algorithm". http://arxiv.org/ftp/arxiv/papers/1406/1406.2294.pdf This implementation uses the xorshift64* pseudo-random number generator rather than the linear congruential generator in the paper as it is reasonably fast but, more importantly, memory efficient. Usage A single function jch:ch/2 is offered. Simply pass in a 64 bit long (or less) integer key argument followed by the desired bucket or continuum partition size. The function returns the partition allocated for the key. Performance is very stable as bucket size increases and distribution across the ring is stable (standard deviation for a reasonable sample size is typically <2% relative to the mean). Example % ERL_LIBS=deps erl +sfwi 1 +scl false -pa ebin Erlang R16B02 (erts-5.10.3) [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] [dtrace] Eshell V5.10.3 (abort with ^G) 1> jch:ch(1,128). 29 2> jch:ch(13,128). 121 3> jch:ch(13,128). 121 4> jch:ch(trunc(math:pow(2,64))-1,128). 78 5> jch:ch(trunc(math:pow(2,64)),128). %% off by 1 mate ** exception error: bad argument in function jch:ch/2 called as jch:ch(18446744073709551616,128) 6> %% off by 1 mate 6> Access times are typically sub-microsecond. Performance tests of the erlang NIF and underlying C function are included. Example performance for a space of 1 billion buckets: - 1B Buckets. Hash performance. ch1b N: 10 Min: -----1 Max: -----2 Median: -----1 Avg: -----1 Elapsed: 16 ch1b N: 100 Min: -----0 Max: -----1 Median: -----1 Avg: -----1 Elapsed: 102 ch1b N: 1000 Min: -----0 Max: -----9 Median: -----0 Avg: -----0 Elapsed: 1178 ch1b N: 10000 Min: -----0 Max: ----13 Median: -----0 Avg: -----0 Elapsed: 10146 ch1b N: 100000 Min: -----0 Max: ----59 Median: -----0 Avg: -----0 Elapsed: 109772 ch1b N: 1000000 Min: -----0 Max: ---886 Median: -----0 Avg: -----0 Elapsed: 1048561 ch1b N: 10000000 Min: -----0 Max: --3734 Median: -----0 Avg: -----0 Elapsed: 11757297 And a sample of how keys partition over the space: - 32 Buckets. 10M hashes Uniform Distribution Check. 312292 312457 312660 312522 313805 311987 311431 312299 311975 312834 312233 312125 313391 312222 312170 312389 311811 313134 312986 312434 312002 313060 312784 312699 312114 312170 312385 312721 313163 311883 313060 312802 oOo| Min: 311431 Max: 313805 Median: 312389 Avg: 312500 Elapsed: 15748372 Worst: 99.2435 Med: 99.5488 Avg: 99.5841 RSD: 0.9139 The RSD here is the standard deviation relative to the mean (avg) which is less than 1% after 10M hash operations on random uniform test data. That's pretty good. A pint (or fancy fizzy water, or cup of pant wetting tea...) to anyone who can strip off an order of magnitude off the max outlier latencies and explain through scheduler, gc or emulator tuning what insight I missed or runtime vagary i've yet to master. Mainly written to toy with micro-benchmarking and get practice with tuning the emulator. Enjoy! Cheers, Darach. -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto@REDACTED Sat Nov 8 11:02:26 2014 From: roberto@REDACTED (Roberto Ostinelli) Date: Sat, 8 Nov 2014 11:02:26 +0100 Subject: [erlang-questions] [ANN] jch-erl 0.1 - Fast minimal memory consistent hashing for Erlang/OTP. In-Reply-To: References: Message-ID: I might still be sleepy, but? enjoy where? :) > Enjoy! From darach@REDACTED Sat Nov 8 13:09:34 2014 From: darach@REDACTED (Darach Ennis) Date: Sat, 8 Nov 2014 12:09:34 +0000 Subject: [erlang-questions] [ANN] jch-erl 0.1 - Fast minimal memory consistent hashing for Erlang/OTP. In-Reply-To: References: Message-ID: https://github.com/darach/jch-erl Forgot we have no packaging system so URLs are still needed. Part of our charm... ;) https://github.com/darach/jch-erl Thanks for the heads up! Cheers, Darach. On 8 Nov 2014 10:02, "Roberto Ostinelli" wrote: > I might still be sleepy, but? enjoy where? :) > > > Enjoy! > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmkolesnikov@REDACTED Sat Nov 8 13:13:33 2014 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Sat, 8 Nov 2014 14:13:33 +0200 Subject: [erlang-questions] Performance question In-Reply-To: References: <545C946F.4010605@ninenines.eu> <545D04E4.4060902@ninenines.eu> Message-ID: Hello, The problem is ?global? here. It turns out that ?processing? of large binary chunk at once is not very efficient using binary:replace or binary:split. The ?custom? algorithm works better. Like you?ve shown this to us. However, If you are pipelining the binary operation with something else (e.g. parsing, counting, whats-not) then binary bif?s wins. I am not able to speak about binary:replace, I?ve never used that at my code but binary:split without global rocks. It was capable to out-perform a custom accumulator functions. Best Regards, Dmitry > On 07 Nov 2014, at 19:50, Stu Bailey wrote: > > Thank you for the feedback. That's very helpful. > > > On Fri, Nov 7, 2014 at 9:44 AM, Lo?c Hoguin > wrote: > Based on the code at > > https://github.com/erlang/otp/blob/maint/lib/stdlib/src/binary.erl#L268 > > It does a lot of splitting, and then a lot more splitting, and then call iolist_to_binary. It looks very inefficient. > > Your solution is the fastest way to do it. You also benefit from match context optimization and so your code is very fast. The only thing that could make it faster is if memory was allocated only once for the resulting binary (instead of realloc a few times)... but maybe there's already an optimization like this? > > On 11/07/2014 07:33 PM, Stu Bailey wrote: > FYI, if you want to try to replicate it, I was processing ~80 chunks of > binary where each chunk was about ~250,000,000 bytes. I think you'll > see the difference on just one chunk. I happen to running on a 8-core > MacBook Pro with 16GB Ram and therefore spawned a process per chunk to > grab all the resources on all the cores. With the hand written > function, it worked like a charm...yay Erlang! :-) I love seeing a few > lines of code effectively use all processing power available. Heats the > machine up quite a bit too. :-) > > On Fri, Nov 7, 2014 at 9:22 AM, Stu Bailey > >> wrote: > > I'm not planning to spend a lot of time on this right now, but the > binary:replace(...) was chewing a tremendous amount of system time > CPU load (and actually never finished before I got frustrated and > killed it) and my function was reporting the CPU load as 99% user > time (not system time) and finished in a reasonable time. I assume > the high system time usage for binary:replace(..) is because > binary:replace(...) is doing something manic with system calls for > memory management or something? > > > On Fri, Nov 7, 2014 at 1:44 AM, Lo?c Hoguin > >> wrote: > > binary:split and binary:replace, unlike other functions of the > binary module, are normal Erlang functions. They also process a > list of options before doing the actual work, so there's an > obvious overhead compared to not doing that. In addition as has > been pointed out, your code is more specialized so that helps too. > > On 11/07/2014 03:33 AM, Stu Bailey wrote: > > I found > > binary:replace(BinChunk,<<"\n"__>>,<<>>,[global]). > > /significantly /slower than > > remove_pattern(BinChunk,<<>>,<__<"\n">>). > > with > > remove_pattern(<<>>,Acc,___BinPat) -> > Acc; > remove_pattern(Bin,Acc,BinPat)__-> > <> = Bin, > case Byte == BinPat of > true -> remove_pattern(Rest,Acc,__BinPat); > false -> > remove_pattern(Rest,<>,BinPat) > end. > > That was surprising to me. The built-in binary:replace() > was much much > slower for larger BinChunk with lots of <<"\n">> sprinkled > through. > > Thoughts? > > > _________________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > > http://erlang.org/mailman/__listinfo/erlang-questions > > > > > -- > Lo?c Hoguin > http://ninenines.eu > > > > > -- > Lo?c Hoguin > http://ninenines.eu > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Sat Nov 8 14:10:33 2014 From: max.lapshin@REDACTED (Max Lapshin) Date: Sat, 8 Nov 2014 16:10:33 +0300 Subject: [erlang-questions] Set program name Message-ID: When one launches erlang, ps or top shows long path to beam.smp, then many useless command line options, then many-many -pa commands and only in the very end something like -s So it is rather hard to distinguish one erlang program from another. It would be very good to have an API for changing current process name. Looks like this feature is very complicated to port between different OS. For example in Linux there must be combined two methods: change argv[0] and call prctl. In MacOS it is required to summon a devil ^W call some tricky code like: https://github.com/openwebos/nodejs/blob/7272321dffae49f67c3d128aedb27bebc3790398/src/platform_darwin_proctitle.cc This code partially works under macos, it changes process name in ActivityMonitor, but doesn't change in top/ps But it seems that it is always required to change argv[0] Have anyone tried to do it? It seems that original argc,argv are lost in erl_start function. -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric.pailleau@REDACTED Sat Nov 8 14:38:47 2014 From: eric.pailleau@REDACTED (Eric Pailleau) Date: Sat, 08 Nov 2014 14:38:47 +0100 Subject: [erlang-questions] Set program name Message-ID: Hi, I personnaly use a wrapper script which is group leader of beams. Imply that the node do not daemonize itself but only the wrapper. Another reason to this, is that my wrapper traps Posix signal and drive erl. example , a kill on the wrapper call a clean node stop before exiting itself. In the ps , i grep only my application names and excluding all beams. The wrapper create its pidfile as any application for integration in init.d or upstart... my 2 cents. ? Envoy? depuis mon mobile ? Eric From max.lapshin@REDACTED Sat Nov 8 15:11:35 2014 From: max.lapshin@REDACTED (Max Lapshin) Date: Sat, 8 Nov 2014 17:11:35 +0300 Subject: [erlang-questions] Set program name In-Reply-To: References: Message-ID: I could change process name in MacOS by following NIF: https://gist.github.com/maxlapshin/910e860665ad40144729 but it doesn't work for linux due to lack of global argv -------------- next part -------------- An HTML attachment was scrubbed... URL: From bchesneau@REDACTED Sat Nov 8 17:49:28 2014 From: bchesneau@REDACTED (Benoit Chesneau) Date: Sat, 8 Nov 2014 17:49:28 +0100 Subject: [erlang-questions] Looking for tricky protocol examples for verification research In-Reply-To: References: <30091.1414656461@snookles.snookles.com> Message-ID: On Thu, Oct 30, 2014 at 10:51 AM, Ulf Wiger wrote: > > On 30 Oct 2014, at 09:07, Scott Lystig Fritchie > wrote: > > > Elsewhere, verifying the good/bad health of the gen_leader protocol. > > Its implementations have splintered, so I'm not quite sure which one is > > the highest quality today, perhaps this one? Ulf Wiger and/or Andrew > > Thompson might also have some opinions ? > > I do. It?s probably not the majority opinion, but I?m leaning towards > ?locks_leader? [1] as the best leader-election approach, with the huge > proviso that it hasn?t seen much - if any - real action*, as far as I > know. ;-) > how would you compare it to raft? Also can we add/remove dynamically nodes which wasn't really possible with gen_leader? > > I haven't seen any postings to this list by Augusto Becciu. > > I remember him posting back in 2010 [2]. > > BR, > Ulf W > > [1] https://github.com/uwiger/locks/blob/master/doc/locks_leader.md > [2] http://erlang.org/pipermail/erlang-questions/2010-October/054088.html > > * If anyone has some feedback about ?locks? in the field, please let me > know. > You can email me in confidence if you want. Personally, I?ve used locks in > the > kvdb database, and successfully negotiated both the kvdb test suite and the > Exosense server test suite (which also does things like passing transaction > contexts between processes). The ?uw-locks-integration? branch of kvdb has > not yet been merged into master, though, and while the idea is to > eventually > use locks_leader in kvdb (or a Raft version), I?ve not yet gone there. > > Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. > http://feuerlabs.com > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric.pailleau@REDACTED Sat Nov 8 18:26:48 2014 From: eric.pailleau@REDACTED (PAILLEAU Eric) Date: Sat, 08 Nov 2014 18:26:48 +0100 Subject: [erlang-questions] Set program name In-Reply-To: References: Message-ID: <545E5258.7030104@wanadoo.fr> Le 08/11/2014 15:11, Max Lapshin a ?crit : > I could change process name in MacOS by following NIF: > > https://gist.github.com/maxlapshin/910e860665ad40144729 > > but it doesn't work for linux due to lack of global argv I found this http://www.uofr.net/~greg/processname.html this may help you ? Regards From shayan@REDACTED Sat Nov 8 22:41:01 2014 From: shayan@REDACTED (Shayan Pooya) Date: Sat, 8 Nov 2014 16:41:01 -0500 Subject: [erlang-questions] performance issue with cowboy Message-ID: Hello, I have ported Disco to use cowboy instead of mochiweb. Mochiweb has been quite stable and we did not have any issues with it recently. However, cowboy uses the sendfile syscall which avoids a lot of overhead when serving static files and I thought that could lower the memory footprint of Disco when serving large files and free up some cycles. Disco+cowboy is now fully functional and all of the integration tests pass (minus some minor missing features). I wanted to get a sense of the performance before making the switch and used a couple of different Disco jobs (which pressure the web-servers) to do a comparison and the Disco version with cowboy consistently takes 10 to 20 percent longer to finish. I haven't done much for optimizing the usage of cowboy yet, but the micro-benchmarks that I find on the web are contrary to what I am seeing here and I wanted to know if anyone can offer some guidance for getting better results with cowboy. I am using cowboy 1.0.1 and the Cowboy patch for Disco is available at https://github.com/pooya/disco/commit/5499d696438 just in case. Regards. -------------- next part -------------- An HTML attachment was scrubbed... URL: From z@REDACTED Sat Nov 8 23:44:26 2014 From: z@REDACTED (Danil Zagoskin) Date: Sun, 9 Nov 2014 02:44:26 +0400 Subject: [erlang-questions] performance issue with cowboy In-Reply-To: References: Message-ID: Hi! Did you do any profiling? Reading the patch I noticed that you have replaced single [Mochiweb's Req:stream_body + file:write] with iterative [cowboy_req:body + file:write]. So your process now writes to file (possibly blocking on it) every MTU (typically 1.5K) bytes instead of single write ? about 70K syscalls per second on gigabit network. I'd suggest adding a large buffer (e.g. 1 MB) to not access the file too often. On Sun, Nov 9, 2014 at 12:41 AM, Shayan Pooya wrote: > Hello, > > I have ported Disco to use cowboy instead of mochiweb. Mochiweb has been > quite stable and we did not have any issues with it recently. However, > cowboy uses the sendfile syscall which avoids a lot of overhead when > serving static files and I thought that could lower the memory footprint of > Disco when serving large files and free up some cycles. Disco+cowboy is now > fully functional and all of the integration tests pass (minus some minor > missing features). > > I wanted to get a sense of the performance before making the switch and > used a couple of different Disco jobs (which pressure the web-servers) to > do a comparison and the Disco version with cowboy consistently takes 10 to > 20 percent longer to finish. I haven't done much for optimizing the usage > of cowboy yet, but the micro-benchmarks that I find on the web are contrary > to what I am seeing here and I wanted to know if anyone can offer some > guidance for getting better results with cowboy. > > I am using cowboy 1.0.1 and the Cowboy patch for Disco is available at > https://github.com/pooya/disco/commit/5499d696438 just in case. > > Regards. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -- Danil Zagoskin | z@REDACTED -------------- next part -------------- An HTML attachment was scrubbed... URL: From shayan@REDACTED Sun Nov 9 15:37:53 2014 From: shayan@REDACTED (Shayan Pooya) Date: Sun, 9 Nov 2014 09:37:53 -0500 Subject: [erlang-questions] performance issue with cowboy In-Reply-To: References: Message-ID: Hey Danil, Thanks for the response. Everything is buffered in the kernel-space anyway and there is no use for a user-space buffer in this case. On Sat, Nov 8, 2014 at 5:44 PM, Danil Zagoskin wrote: > Hi! > > Did you do any profiling? > > Reading the patch I noticed that you have replaced single > [Mochiweb's Req:stream_body + file:write] with iterative [cowboy_req:body + > file:write]. So your process now writes to file (possibly blocking on it) > every MTU (typically 1.5K) bytes instead of single write ? about 70K > syscalls per second on gigabit network. > I'd suggest adding a large buffer (e.g. 1 MB) to not access the file too > often. > > > On Sun, Nov 9, 2014 at 12:41 AM, Shayan Pooya wrote: > >> Hello, >> >> I have ported Disco to use cowboy instead of mochiweb. Mochiweb has been >> quite stable and we did not have any issues with it recently. However, >> cowboy uses the sendfile syscall which avoids a lot of overhead when >> serving static files and I thought that could lower the memory footprint of >> Disco when serving large files and free up some cycles. Disco+cowboy is now >> fully functional and all of the integration tests pass (minus some minor >> missing features). >> >> I wanted to get a sense of the performance before making the switch and >> used a couple of different Disco jobs (which pressure the web-servers) to >> do a comparison and the Disco version with cowboy consistently takes 10 to >> 20 percent longer to finish. I haven't done much for optimizing the usage >> of cowboy yet, but the micro-benchmarks that I find on the web are contrary >> to what I am seeing here and I wanted to know if anyone can offer some >> guidance for getting better results with cowboy. >> >> I am using cowboy 1.0.1 and the Cowboy patch for Disco is available at >> https://github.com/pooya/disco/commit/5499d696438 just in case. >> >> Regards. >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > > > -- > Danil Zagoskin | z@REDACTED > -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Sun Nov 9 15:45:37 2014 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Sun, 09 Nov 2014 16:45:37 +0200 Subject: [erlang-questions] performance issue with cowboy In-Reply-To: References: Message-ID: <545F7E11.2040706@ninenines.eu> Sure but everything you do through file:* goes through a single Erlang process except if you use the 'raw' option. On 11/09/2014 04:37 PM, Shayan Pooya wrote: > Hey Danil, > > Thanks for the response. Everything is buffered in the kernel-space > anyway and there is no use for a user-space buffer in this case. > > On Sat, Nov 8, 2014 at 5:44 PM, Danil Zagoskin > wrote: > > Hi! > > Did you do any profiling? > > Reading the patch I noticed that you have replaced single > [Mochiweb's Req:stream_body + file:write] with > iterative [cowboy_req:body + file:write]. So your process now writes > to file (possibly blocking on it) every MTU (typically 1.5K) bytes > instead of single write ? about 70K syscalls per second on gigabit > network. > I'd suggest adding a large buffer (e.g. 1 MB) to not access the file > too often. > > > On Sun, Nov 9, 2014 at 12:41 AM, Shayan Pooya > wrote: > > Hello, > > I have ported Disco to use cowboy instead of mochiweb. Mochiweb > has been quite stable and we did not have any issues with it > recently. However, cowboy uses the sendfile syscall which avoids > a lot of overhead when serving static files and I thought that > could lower the memory footprint of Disco when serving large > files and free up some cycles. Disco+cowboy is now fully > functional and all of the integration tests pass (minus some > minor missing features). > > I wanted to get a sense of the performance before making the > switch and used a couple of different Disco jobs (which pressure > the web-servers) to do a comparison and the Disco version with > cowboy consistently takes 10 to 20 percent longer to finish. I > haven't done much for optimizing the usage of cowboy yet, but > the micro-benchmarks that I find on the web are contrary to what > I am seeing here and I wanted to know if anyone can offer some > guidance for getting better results with cowboy. > > I am using cowboy 1.0.1 and the Cowboy patch for Disco is > available at https://github.com/pooya/disco/commit/5499d696438 > just in case. > > Regards. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > > -- > Danil Zagoskin | z@REDACTED > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin http://ninenines.eu From ulf@REDACTED Sun Nov 9 15:53:22 2014 From: ulf@REDACTED (Ulf Wiger) Date: Sun, 9 Nov 2014 15:53:22 +0100 Subject: [erlang-questions] Looking for tricky protocol examples for verification research In-Reply-To: References: <30091.1414656461@snookles.snookles.com> Message-ID: <8E3FD71B-63A5-45E6-AD1E-457890D77A22@feuerlabs.com> On 08 Nov 2014, at 17:49, Benoit Chesneau wrote: > > > On Thu, Oct 30, 2014 at 10:51 AM, Ulf Wiger wrote: > > On 30 Oct 2014, at 09:07, Scott Lystig Fritchie wrote: > > > Elsewhere, verifying the good/bad health of the gen_leader protocol. > > Its implementations have splintered, so I'm not quite sure which one is > > the highest quality today, perhaps this one? Ulf Wiger and/or Andrew > > Thompson might also have some opinions ? > > I do. It?s probably not the majority opinion, but I?m leaning towards > ?locks_leader? [1] as the best leader-election approach, with the huge > proviso that it hasn?t seen much - if any - real action*, as far as I know. ;-) > > how would you compare it to raft? Also can we add/remove dynamically nodes which wasn't really possible with gen_leader? Raft does include leader election, but that?s only part of what it does. I assume the Raft leader election algorithm will work correctly even in an Erlang setting, but some differences between Raft and locks: 1. Raft uses backoff and randomized sleep to resolve election conflicts; locks uses a form of reactive deadlock resolution that converges immediately. 2. Raft makes use of replication history to elect the most suitable leader. This is difficult for locks, since it will fall back on a global ordering of candidates when needed. For fun, I started trying to implement Raft on top of the locks_leader implementation. I got so far as to add a ?surrender? operation in locks_leader, so the newly elected leader can decide to make another process the leader. This is where I stopped, waiting for a time where I could do some more careful analysis of the problem space. BR, Ulf W Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. http://feuerlabs.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From shayan@REDACTED Sun Nov 9 16:06:10 2014 From: shayan@REDACTED (Shayan Pooya) Date: Sun, 9 Nov 2014 10:06:10 -0500 Subject: [erlang-questions] performance issue with cowboy In-Reply-To: <545F7E11.2040706@ninenines.eu> References: <545F7E11.2040706@ninenines.eu> Message-ID: Almost all of the file operations in Disco use raw mode. For the functions that do not have raw mode, we use the prim_file module directly. Regards. On Sun, Nov 9, 2014 at 9:45 AM, Lo?c Hoguin wrote: > Sure but everything you do through file:* goes through a single Erlang > process except if you use the 'raw' option. > > On 11/09/2014 04:37 PM, Shayan Pooya wrote: > >> Hey Danil, >> >> Thanks for the response. Everything is buffered in the kernel-space >> anyway and there is no use for a user-space buffer in this case. >> >> On Sat, Nov 8, 2014 at 5:44 PM, Danil Zagoskin > > wrote: >> >> Hi! >> >> Did you do any profiling? >> >> Reading the patch I noticed that you have replaced single >> [Mochiweb's Req:stream_body + file:write] with >> iterative [cowboy_req:body + file:write]. So your process now writes >> to file (possibly blocking on it) every MTU (typically 1.5K) bytes >> instead of single write ? about 70K syscalls per second on gigabit >> network. >> I'd suggest adding a large buffer (e.g. 1 MB) to not access the file >> too often. >> >> >> On Sun, Nov 9, 2014 at 12:41 AM, Shayan Pooya > > wrote: >> >> Hello, >> >> I have ported Disco to use cowboy instead of mochiweb. Mochiweb >> has been quite stable and we did not have any issues with it >> recently. However, cowboy uses the sendfile syscall which avoids >> a lot of overhead when serving static files and I thought that >> could lower the memory footprint of Disco when serving large >> files and free up some cycles. Disco+cowboy is now fully >> functional and all of the integration tests pass (minus some >> minor missing features). >> >> I wanted to get a sense of the performance before making the >> switch and used a couple of different Disco jobs (which pressure >> the web-servers) to do a comparison and the Disco version with >> cowboy consistently takes 10 to 20 percent longer to finish. I >> haven't done much for optimizing the usage of cowboy yet, but >> the micro-benchmarks that I find on the web are contrary to what >> I am seeing here and I wanted to know if anyone can offer some >> guidance for getting better results with cowboy. >> >> I am using cowboy 1.0.1 and the Cowboy patch for Disco is >> available at https://github.com/pooya/disco/commit/5499d696438 >> just in case. >> >> Regards. >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> >> >> -- >> Danil Zagoskin | z@REDACTED >> >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -- > Lo?c Hoguin > http://ninenines.eu > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lucafinzicontini@REDACTED Sun Nov 9 17:54:41 2014 From: lucafinzicontini@REDACTED (Luca Finzi Contini) Date: Sun, 9 Nov 2014 17:54:41 +0100 Subject: [erlang-questions] ErlIDE on Mac OS X Mavericks - anyone using it successfully? In-Reply-To: References: Message-ID: Hi all, for those of you interested in this subject, and thanks to the help of Vlad (vladdu55@REDACTED), I came up with a solution to work on my Macbook Pro. Root cause was: Java 8! Switching to Java 7 magically solved all the issues. Here is what I did in detail. 0) my machine's 'name' is 'legolas'. 1) in System Preferences, it is displayed as "Computer Name: legolas" but underneath this, it reads: "Computers on your local network can access your computer at : *legolas.local*". So I changed the hostname to be this fully qualified name: legolas:~ finzic$ sudo hostname -s legolas.local legolas:~ finzic$ hostname legolas.local 2) I modified the /etc/hosts to include *legolas* and *legolas.local* on the same line as 127.0.0.1 Looking at the log files, it seems that both the short-name and the long-name version are needed to have it working. 3) Most importantly I found that the default JDK used was 1.8.0, so I set JAVA_HOME to the 1.7.0 folder and launched Eclipse via terminal: legolas:MacOS finzic$ /usr/libexec/java_home -V Matching Java Virtual Machines (4): 1.8.0_05, x86_64: "Java SE 8" /Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home 1.7.0_55, x86_64: "Java SE 7" /Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home 1.6.0_65-b14-462, x86_64: "Java SE 6" /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home 1.6.0_65-b14-462, i386: "Java SE 6" /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home /Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home legolas:MacOS finzic$ echo $JAVA_HOME /Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home legolas:MacOS finzic$ This way, everything worked extra-smoothly. This means that all the issues of timeouts in communication between Java and Erlang in my opinion were caused by Eclipse running Java 8. Thank to Vlad once more for all the help. Luca. 2014-11-05 10:58 GMT+01:00 Luca Finzi Contini : > Hi all, > is there anyone using ErlIDE / Eclipse on Mac OS X Mavericks? > I have successfully installed it on Windows but I usually work on a Mac. > I tried with Eclipse 3.7, 4.2.2, 4.4, with no luck. > The IDE goes 'not responding' after a few keystrokes on the source editor. > In the rare moments of awakens it has *wonderful* hot code update features > which are really good especially for those accustomed to Eclipse in the > Java world. > I tried to follow the suggestions here: > > https://github.com/erlide/erlide/wiki/Troubleshooting > > especially when it says: > "For Macs, an alternative solution is to rename your computer from > foo.whatever to just foo (or anyhting without any dots)." > > If I do this, apparently I cannot start an erlang instance with long name > as in > > erl -name foo > > The currently installed Java version is 1.8.0. > > Has anyone managed to have ErlIDE up and running smoothly on Mac? > Thanks in advance. > Luca. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charleshixsn@REDACTED Sun Nov 9 21:00:49 2014 From: charleshixsn@REDACTED (Charles Hixson) Date: Sun, 09 Nov 2014 12:00:49 -0800 Subject: [erlang-questions] Mnesia vs. ets+dets Message-ID: <545FC7F1.2090607@earthlink.net> I'm going to be running on a single node, at least for the foreseeable future, if that matters. I expect the data to grow to be too large to fit into RAM, OTOH, only a small subset will be needed at any one time. Also I expect active items to have frequent updates, but most items to rarely be updated (only when a purge is contemplated). There doesn't seem to be any way to control the frequency with which Mnesia synchronizes to disk...but I also don't see any information about how often it does so. Once every few minutes wouldn't be a problem, but on every change would be prohibitive. On the plus side, Mnesia will automatically handle splitting the data between tables (when the data grows too large to fit into one dets file). One way of handling the problem is to operate out of an ets database which pulls in data as needed from various dets files and occasionally synchronizes with them. This starts looking complex, and Mnesia is clearly intended to solve problems *like* this. I just can't determine that it will handle *this* problem. If I could set the rate of synchronization between RAM and disk, it would be the clear best approach...but I haven't even found any documentation on that. OTOH, I'm quite new to Erlang (I've just written one successful program) so I could easily be missing something that's extremely obvious to someone more experienced. Any comments or suggestions? From eric.pailleau@REDACTED Sun Nov 9 22:45:00 2014 From: eric.pailleau@REDACTED (Eric Pailleau) Date: Sun, 09 Nov 2014 22:45:00 +0100 Subject: [erlang-questions] Mnesia vs. ets+dets Message-ID: Hi, did you look at change_table_copy/3 an dump_tables/1 ? This could let you decide when data have to go on disc (timer or/and sasl overload request). regards ? Envoy? depuis mon mobile ? Eric From oleg@REDACTED Sun Nov 9 22:24:35 2014 From: oleg@REDACTED (Oleg Tsarev) Date: Mon, 10 Nov 2014 01:24:35 +0400 Subject: [erlang-questions] Lager & Common Test question Message-ID: Hello! I use lager + common test. I configured lager for output to console. During test lager calls print output to my terminal, and test-related log is empty. io:format works as I want - no output in my terminal, only in test-related log. How I can configure lager for see lager outputs in the log file instead of terminal? Best regards, Oleg -------------- next part -------------- An HTML attachment was scrubbed... URL: From charleshixsn@REDACTED Mon Nov 10 02:26:45 2014 From: charleshixsn@REDACTED (Charles Hixson) Date: Sun, 09 Nov 2014 17:26:45 -0800 Subject: [erlang-questions] Mnesia vs. ets+dets In-Reply-To: References: <545FC7F1.2090607@earthlink.net> Message-ID: <54601455.3060709@earthlink.net> On 11/09/2014 12:56 PM, T Ty wrote: > I usually advocate using Mnesia because any schema you come up with > you will later want some form of fault tolerance and as Joe Armstrong > keeps harping, you can't do fault tolerance with one node. > > That said, Mnesia does not automatically split data between tables. > You have to explicitly setup fragmented tables for it. > > To control how often Mnesia dumps from RAM to disc see: > > -mnesia dc_dump_limit Number. Controls how often disc_copies tables > are dumped from memory. Tables are dumped when filesize(Log) > > (filesize(Tab)/Dc_dump_limit). Lower values reduces cpu overhead but > increases disk space and startup times. The default is 4. > > > > On Sun, Nov 9, 2014 at 8:00 PM, Charles Hixson > > wrote: > > I'm going to be running on a single node, at least for the > foreseeable future, if that matters. > > I expect the data to grow to be too large to fit into RAM, OTOH, > only a small subset will be needed at any one time. Also I expect > active items to have frequent updates, but most items to rarely be > updated (only when a purge is contemplated). There doesn't seem > to be any way to control the frequency with which Mnesia > synchronizes to disk...but I also don't see any information about > how often it does so. Once every few minutes wouldn't be a > problem, but on every change would be prohibitive. On the plus > side, Mnesia will automatically handle splitting the data between > tables (when the data grows too large to fit into one dets file). > > One way of handling the problem is to operate out of an ets > database which pulls in data as needed from various dets files and > occasionally synchronizes with them. This starts looking complex, > and Mnesia is clearly intended to solve problems *like* this. I > just can't determine that it will handle *this* problem. If I > could set the rate of synchronization between RAM and disk, it > would be the clear best approach...but I haven't even found any > documentation on that. > > OTOH, I'm quite new to Erlang (I've just written one successful > program) so I could easily be missing something that's extremely > obvious to someone more experienced. > > Any comments or suggestions? > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ahe.sanath@REDACTED Mon Nov 10 07:20:20 2014 From: ahe.sanath@REDACTED (Sanath Prasanna) Date: Mon, 10 Nov 2014 11:50:20 +0530 Subject: [erlang-questions] =?utf-8?b?RXVybyBTeW1ib2wgKOKCrCkgY29udmVyc2lv?= =?utf-8?q?n_from_HTTP_API_to_SMPP_through_mochiweb?= In-Reply-To: <545D2A4C.50505@gmail.com> References: <545D2A4C.50505@gmail.com> Message-ID: Hi Dimtry, Tx a lot for your help.It is working. Br, Sanath Roberto On Sat, Nov 8, 2014 at 1:53 AM, Dmitry Klionsky wrote: > On 11/07/2014 05:07 AM, Sanath Prasanna wrote: > > Hi, > I used mochiweb web server to handle http request. I submit messages with > euro - ? as "%E2%82%AC" thorugh HTTP API. I need to forward this to SMPP > connction. Before that I want to convert that to 1B65. Otherwise ? > character is not display in mobile. > How to do that ? Appreciate if some can help to do that. > Br, > Sanath Roberto > > > _______________________________________________ > erlang-questions mailing listerlang-questions@REDACTED://erlang.org/mailman/listinfo/erlang-questions > > > Hi, > > You need to convert from UTF8 to GSM 03.38. Use this library > https://github.com/iamaleksey/gsm0338. > > 3> gsm0338:from_utf8("?"). > <<"\ee">> > > -- > Best regards, > Dmitry Klionsky > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charleshixsn@REDACTED Mon Nov 10 08:19:37 2014 From: charleshixsn@REDACTED (Charles Hixson) Date: Sun, 09 Nov 2014 23:19:37 -0800 Subject: [erlang-questions] Mnesia vs. ets+dets In-Reply-To: <54601455.3060709@earthlink.net> References: <545FC7F1.2090607@earthlink.net> <54601455.3060709@earthlink.net> Message-ID: <54606709.6060800@earthlink.net> Sorry, something seems to have happened to what I originally sent. So this is essentially what got cut: Those controls tell me what I needed to know about the original question. Every 3 minutes to save seems reasonable, though I'd probably want to increase the number of allowed writes without flush to 100,000. But while looking things up I read some things that seemed to mean that Mnesia wanted to either hold the entire database in memory (disc_copies) or not hold any cache (disc_only_copies). This doesn't sound good, as what I need is a cache of active records in RAM. I can still do this by using disc_only_copies and using it to fill an ets table which I occasionally flush to Mnesia, but this seems unexpectedly clumsy. Am I reading the documentation incorrectly? I really don't expect to be able to fit the entire table(s) into RAM. On 11/09/2014 05:26 PM, Charles Hixson wrote: > > On 11/09/2014 12:56 PM, T Ty wrote: >> I usually advocate using Mnesia because any schema you come up with >> you will later want some form of fault tolerance and as Joe Armstrong >> keeps harping, you can't do fault tolerance with one node. >> >> .... > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From Maxim.Minin@REDACTED Mon Nov 10 08:29:07 2014 From: Maxim.Minin@REDACTED (Minin Maxim) Date: Mon, 10 Nov 2014 08:29:07 +0100 Subject: [erlang-questions] Lager & Common Test question Message-ID: You have to configure lager's backends for your purpose - see https://github.com/basho/lager#configuration Regards Maxim Von: erlang-questions-bounces@REDACTED [mailto:erlang-questions-bounces@REDACTED] Im Auftrag von Oleg Tsarev Gesendet: Sonntag, 9. November 2014 22:25 An: erlang-questions@REDACTED Betreff: [erlang-questions] Lager & Common Test question Hello! I use lager + common test. I configured lager for output to console. During test lager calls print output to my terminal, and test-related log is empty. io:format works as I want - no output in my terminal, only in test-related log. How I can configure lager for see lager outputs in the log file instead of terminal? Best regards, Oleg -------------- next part -------------- An HTML attachment was scrubbed... URL: From bog495@REDACTED Mon Nov 10 10:54:49 2014 From: bog495@REDACTED (Bogdan Andu) Date: Mon, 10 Nov 2014 11:54:49 +0200 Subject: [erlang-questions] 'ssl_otp_session_cache' ets table In-Reply-To: References: Message-ID: only for R17 I tested this . if you do not do cache at all the cpu usage increases and dummy cache module is out of questions. so a balance must be founded between session reuse (which a re good) and processor usage. application:get_env(ssl, session_lifetime). returns ok the values. If I have time I'll dig into the code to see what happens deeper. On Fri, Nov 7, 2014 at 4:20 PM, Ingela Andin wrote: > Hi! > > > The following release note is from the first R15 version of ssl: > > "Invalidation handling of sessions could cause the time_stamp field in the > session record to be set to undefined crashing the session clean up > process. This did not affect the connections but would result in that the > session table would grow." > > > 2014-11-07 11:58 GMT+01:00 Bogdan Andu : > >> I work with R14B04 as production and OTP 17.3 as experimental. >> >> R14B04 also deletes all entries after 24 hours. >> >> 17.3 also deletes all entries after 24 hours. >> >> I think is it has to do with passing the environment parameter like this: >> >> erl -ssl session_lifetime 60 ... >> >> I tried many variants and may be I miss something. >> >> may be I am mistaken but it is possible that only the default value, 24 >> hours, >> is only considered as the environment variable is not passed correctly by >> me. >> >> i think the environment parameters are the problem here because maybe i >> specify them incorrctly. >> >> could you advise what is the proper method of correctly specifying >> session_lifetime, because according to >> manual is like : >> >> erl -ssl session_lifetime 60 ... >> > > > You can check with application:get_env(ssl, session_lifetime). to see if > you succeed setting the variable. > > > >> >> a ssl session must expire afte 60 seconds + delayed 60 seconds = 2 >> minutes. >> >> intstead entries from 'ssl_otp_session_cache' be deleted every 2 minutes, >> they remain 24 hours . >> >> after 22 hours I have: >> >> Node: 'n1@REDACTED' (Connected) (17/6.2) unix (openbsd 5.4.0) CPU:2 SMP >> +A:10 >> Time: local time 12:50:23, up for 000:21:54:49, 0ms latency, >> Processes: total 676 (RQ 0) at 21121 RpI using 10978.0k (11007.4k >> allocated) >> Memory: Sys 14898.8k, Atom 410.8k/419.5k, Bin 112.9k, Code 10053.7k, Ets >> 720.6k >> >> >> and for ets:i(). >> >> ........... >> 28695 ssl_otp_session_cache ordered_set 115 7134 >> ssl_manager >> .......... >> >> The vm was configured with 60 seconds session lifetime and the client was >> running 60 seconds >> an I expected after 2 minutes to see less entries in the table but they >> remained untouched >> >> >> I will see what happens after 24 hours mark and I'll let you know >> >> and another thing... >> >> even if i pass the options: >> {reuse_session, fun(_,_,_,_) -> false end}, >> {reuse_sessions, false}, >> >> both or individually , to the ssl:listen/2 function the table is filled >> with sessions and they expire after 24 hours, >> only that cpu is working harder, 7-8% which means the sessions a not >> reused and the ssl >> handshake is happening every time. >> >> > Was that for both R14 and R17? > > > Regards Ingela Erlang/OTP team - Ericsson AB > > > > > >> >> Bogdan >> >> On Fri, Nov 7, 2014 at 10:37 AM, Ingela Andin >> wrote: >> >>> Hi! >>> >>> Which version of the ssl application are you using? The session cleanup >>> was broken in one of the older versions. >>> >>> Regards Ingela Erlang/OTP - team Ericsson AB >>> >>> >>> 2014-11-06 17:20 GMT+01:00 Bogdan Andu : >>> >>>> yes, >>>> >>>> in function ssl_manager:invalidate_session/4, the call >>>> >>>> erlang:send_after(delay_time(), self(), {delayed_clean_session, Key}), >>>> does the actual deletion of session from cache >>>> >>>> where delay_time/0 function returns ?CLEAN_SESSION_DB by default, >>>> which is 60 seconds >>>> >>>> >>>> >>>> >>>> On Thu, Nov 6, 2014 at 4:51 PM, Ingela Andin >>>> wrote: >>>> >>>>> Hi! >>>>> >>>>> >>>>> 2014-11-06 14:39 GMT+01:00 Bogdan Andu : >>>>> >>>>>> Hi, >>>>>> >>>>>> i have tried to pass to execute th following forms of commands: >>>>>> >>>>>> erl -ssl session_lifetime 60 >>>>>> >>>>>> erl -ssl session_lifetime '60' >>>>>> >>>>>> erl -ssl session_lifetime '[60]' >>>>>> >>>>>> >>>>>> in order to set a ssl session_lifetime of 60 seconds. >>>>>> >>>>>> in init of ssl_manager , after session_lifetime + 5 seconds a process >>>>>> is created to validate or invalidate sessions. >>>>>> So, after 65 seconds a foldl is called on sessions table to check >>>>>> sessions after formula: >>>>>> Now - TimeStamp < LifeTime >>>>>> >>>>>> However, even after I stop the client, the table >>>>>> 'ssl_otp_session_cache' 's size remains the same. >>>>>> >>>>>> >>>>>> After that once every ?CLEAN_SESSION_DB which is 60 seconds , the >>>>>> table is sweeped to delete any expired sessions, >>>>>> >>>>>> but also nothing happens. >>>>>> >>>>>> >>>>> There is also a delay in the actual deletion. In the first sweep the >>>>> sessions will only be invalidated as there may already be >>>>> spawned connection handlers that needs to read the session data before >>>>> we may delete it. It is a performance trade off. >>>>> >>>>> Regards Ingela Erlang/OTP team - Ericsson AB >>>>> >>>>> >>>>> >>>>>> >>>>>> On Thu, Nov 6, 2014 at 12:28 PM, Ingela Andin >>>>> > wrote: >>>>>> >>>>>>> Hi! >>>>>>> >>>>>>> >>>>>>> 2014-11-06 8:53 GMT+01:00 Bogdan Andu : >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> on my production servers I have relayd (on OpenBSD) daemon as a >>>>>>>> reverse proxy to some webservers >>>>>>>> where one can fine tune some connection parameters, as well as some >>>>>>>> ssl parameters. >>>>>>>> >>>>>>>> I give a snippet from a relayd.conf configuration file on one of my >>>>>>>> production server: >>>>>>>> >>>>>>>> ............. >>>>>>>> # >>>>>>>> # Relay and protocol for HTTP layer 7 loadbalancing and SSL >>>>>>>> acceleration >>>>>>>> # >>>>>>>> http protocol www_ssl_prot { >>>>>>>> header append "$REMOTE_ADDR" to "X-Forwarded-For" >>>>>>>> header append "$SERVER_ADDR:$SERVER_PORT" to >>>>>>>> "X-Forwarded-By" >>>>>>>> header change "Connection" to "close" >>>>>>>> >>>>>>>> response header change "Server" to "Apache 0.1" >>>>>>>> >>>>>>>> # Various TCP performance options >>>>>>>> tcp { nodelay, sack, socket buffer 65536, backlog 128 } >>>>>>>> >>>>>>>> ssl { no sslv2, no sslv3, tlsv1, ciphers "HIGH" } >>>>>>>> ssl session cache disable >>>>>>>> } >>>>>>>> >>>>>>>> the last directive tells relayd not to use ssl cache. >>>>>>>> >>>>>>>> This configuration is working for years and relayd was restarted >>>>>>>> once by accident - my fault. >>>>>>>> >>>>>>>> SO, y question is: >>>>>>>> >>>>>>>> can we have this configurable in Erlang, in other words, we might >>>>>>>> be able to start an erlang vm such as: >>>>>>>> >>>>>>>> erl -ssl session_cache 'disable' -name x@REDACTED .... >>>>>>>> >>>>>>>> The ssl option session_cache can be set to disabled by default and >>>>>>>> can take values either disable or enabled. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> You can already disable the reuse of the sessions using the server >>>>>>> option *{reuse_sessions, boolean()}* which default to true. >>>>>>> The thing we plan to do is to have a configurable limit on the table >>>>>>> size when sessions are reused. >>>>>>> >>>>>>> Regards Ingela Erlang/OTP team Ericsson AB >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> Just to let you know... >>>>>>>> >>>>>>>> The statistics of a node running from yesterday when I opened this >>>>>>>> thread of discussion, using a single client: >>>>>>>> >>>>>>>> Node: 'n1@REDACTED' (Connected) (17/6.2) unix (openbsd 5.4.0) >>>>>>>> CPU:2 SMP +A:10 >>>>>>>> Time: local time 09:48:48, up for 000:20:27:49, 0ms latency, >>>>>>>> Processes: total 681 (RQ 0) at 610275 RpI using 11526.4k (11805.8k >>>>>>>> allocated) >>>>>>>> Memory: Sys 94835.5k, Atom 407.7k/419.5k, Bin 176.4k, Code 9934.7k, >>>>>>>> Ets 80739.2k >>>>>>>> >>>>>>>> ........... >>>>>>>> >>>>>>>> So, can be this made configurable? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Nov 5, 2014 at 8:52 PM, Ingela Andin < >>>>>>>> ingela.andin@REDACTED> wrote: >>>>>>>> >>>>>>>>> Hi! >>>>>>>>> >>>>>>>>> 2014-11-05 14:15 GMT+01:00 Tony Rogvall : >>>>>>>>> >>>>>>>>>> I think the ssl session times is the problem here, and the lack >>>>>>>>>> of a maximum size. >>>>>>>>>> >>>>>>>>>> You can change the session time in the ssl environment: >>>>>>>>>> session_lifetime >>>>>>>>>> The default is set to 24 hours (in seconds) (if I read it >>>>>>>>>> correctly, in ssl_manager.erl) >>>>>>>>>> >>>>>>>>> >>>>>>>>> Default values are always hard. It is the maximum recommended time >>>>>>>>> for a session to live >>>>>>>>> according to the spec. >>>>>>>>> >>>>>>>>> >>>>>>>>>> I guess that a session_cache_size could be a nice thing to have, >>>>>>>>>> limiting the growth of the session cache. >>>>>>>>>> >>>>>>>>>> >>>>>>>>> Yes I agree. >>>>>>>>> >>>>>>>>> >>>>>>>>>> In other words you have to estimate the life time of your clients >>>>>>>>>> and >>>>>>>>>> try to find a reasonable session_lifetime to match that, without >>>>>>>>>> blowing up >>>>>>>>>> the system. >>>>>>>>>> >>>>>>>>>> Maybe the ssl_session_cache_api could be used to implement a >>>>>>>>>> strategy with a max size >>>>>>>>>> cache. Retire session least recently used, while performing the >>>>>>>>>> update? >>>>>>>>>> There is a time_stamp in the session that that could be used for >>>>>>>>>> this purpose. >>>>>>>>>> >>>>>>>>>> OTP: Why is a fixed limit not implemented in the standard >>>>>>>>>> ssl_session_cache? >>>>>>>>>> Could this be a target for DOS attacks? >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> We are aware of the problem, and it is on our todo list. One >>>>>>>>> reason it has not had top priority is that on the server side there are >>>>>>>>> often other mechanisms >>>>>>>>> like firewalls and webbserver settings that limits the problem. >>>>>>>>> And the reason why it was not implemented in the first place is that >>>>>>>>> implementations by nature >>>>>>>>> are iterative and at first you are faced with a lot of bigger >>>>>>>>> problems to solve and then you need to iterate and fine tune and fix things >>>>>>>>> that you now have a better understanding of. >>>>>>>>> >>>>>>>>> In current master there is a change to the session table that >>>>>>>>> limits the growth on the client side if the client behaves inappropriate. >>>>>>>>> It also splits the session table into >>>>>>>>> a server and a client table which is a better implementation as >>>>>>>>> the same Erlang node can be both a client and a server at the same time. So >>>>>>>>> if someone feels like contributing a max limit please base it on the >>>>>>>>> master branch, otherwise I suspect someone compiling about it did raise >>>>>>>>> the priority level a little. >>>>>>>>> >>>>>>>>> >>>>>>>>> Regards Ingela Erlang/OTP team - Ericsson AB >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> /Tony >>>>>>>>>> >>>>>>>>>> > On 5 nov 2014, at 13:08, Bogdan Andu wrote: >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > Hi, >>>>>>>>>> > >>>>>>>>>> > I performed a series of test regarding the an Erlang SSL server. >>>>>>>>>> > >>>>>>>>>> > In this setup a major role is played by the table called >>>>>>>>>> 'ssl_otp_session_cache', and of course the processes using it. >>>>>>>>>> > >>>>>>>>>> > The problem is that the size of table increases constantly and, >>>>>>>>>> because an ets table does not automatically deallocate memory unless the >>>>>>>>>> object are deleted from that table, the size of table remains the same even >>>>>>>>>> if there no ssl connections to server. >>>>>>>>>> > >>>>>>>>>> > For example, with a single client running 'ad infinitum' the >>>>>>>>>> table increases at a rate of 5 MBytes/hour. In 12 hours there are allocate >>>>>>>>>> around 60 MB of memory only for this table. >>>>>>>>>> > >>>>>>>>>> > Some info about this: >>>>>>>>>> > $ erl -sname console@REDACTED -remsh n1@REDACTED -setcookie >>>>>>>>>> operator >>>>>>>>>> > Erlang/OTP 17 [erts-6.2] [source] [64-bit] [smp:2:2] >>>>>>>>>> [async-threads:10] [kernel-poll:false] >>>>>>>>>> > >>>>>>>>>> > Eshell V6.2 (abort with ^G) >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > (n1@REDACTED)1> ets:i(). >>>>>>>>>> > id name type size mem owner >>>>>>>>>> > >>>>>>>>>> ---------------------------------------------------------------------------- >>>>>>>>>> > 12 cookies set 0 291 auth >>>>>>>>>> > 4111 code set 410 26132 >>>>>>>>>> code_server >>>>>>>>>> > 8208 code_names set 58 7459 >>>>>>>>>> code_server >>>>>>>>>> > 12307 httpc_manager__session_cookie_db bag 0 >>>>>>>>>> 291 httpc_manager >>>>>>>>>> > 16404 ssl_otp_cacertificate_db set 0 291 >>>>>>>>>> ssl_manager >>>>>>>>>> > 20501 ssl_otp_ca_file_ref set 0 291 >>>>>>>>>> ssl_manager >>>>>>>>>> > 24598 ssl_otp_pem_cache set 3 360 >>>>>>>>>> ssl_manager >>>>>>>>>> > 28695 ssl_otp_session_cache ordered_set 138057 >>>>>>>>>> 8421893 ssl_manager >>>>>>>>>> > 32797 dets duplicate_bag 2 308 >>>>>>>>>> dets >>>>>>>>>> > 40990 ign_requests set 0 291 >>>>>>>>>> inet_gethost_native >>>>>>>>>> > 45087 ign_req_index set 0 291 >>>>>>>>>> inet_gethost_native >>>>>>>>>> > 2261635104 shell_records ordered_set 0 81 >>>>>>>>>> <0.30638.37> >>>>>>>>>> > ac_tab ac_tab set 33 2216 >>>>>>>>>> application_controller >>>>>>>>>> > code_map code_map set 100 2791 >>>>>>>>>> <0.72.0> >>>>>>>>>> > config config set 12 892 >>>>>>>>>> <0.72.0> >>>>>>>>>> > dets_owners dets_owners set 1 298 dets >>>>>>>>>> > dets_registry dets_registry set 1 299 dets >>>>>>>>>> > file_io_servers file_io_servers set 1 344 >>>>>>>>>> file_server_2 >>>>>>>>>> > global_locks global_locks set 0 291 >>>>>>>>>> global_name_server >>>>>>>>>> > global_names global_names set 0 291 >>>>>>>>>> global_name_server >>>>>>>>>> > global_names_ext global_names_ext set 0 291 >>>>>>>>>> global_name_server >>>>>>>>>> > global_pid_ids global_pid_ids bag 0 291 >>>>>>>>>> global_name_server >>>>>>>>>> > global_pid_names global_pid_names bag 0 291 >>>>>>>>>> global_name_server >>>>>>>>>> > httpc_manager__handler_db httpc_manager__handler_db set 0 >>>>>>>>>> 291 httpc_manager >>>>>>>>>> > httpc_manager__session_db httpc_manager__session_db set 0 >>>>>>>>>> 291 httpc_manager >>>>>>>>>> > inet_cache inet_cache bag 0 291 inet_db >>>>>>>>>> > inet_db inet_db set 29 600 inet_db >>>>>>>>>> > inet_hosts_byaddr inet_hosts_byaddr bag 0 291 >>>>>>>>>> inet_db >>>>>>>>>> > inet_hosts_byname inet_hosts_byname bag 0 291 >>>>>>>>>> inet_db >>>>>>>>>> > inet_hosts_file_byaddr inet_hosts_file_byaddr bag 0 >>>>>>>>>> 291 inet_db >>>>>>>>>> > inet_hosts_file_byname inet_hosts_file_byname bag 0 >>>>>>>>>> 291 inet_db >>>>>>>>>> > models models set 3 28952 >>>>>>>>>> <0.72.0> >>>>>>>>>> > sys_dist sys_dist set 1 334 >>>>>>>>>> net_kernel >>>>>>>>>> > ok >>>>>>>>>> > (n1@REDACTED)5> ets:info(ssl_otp_session_cache). >>>>>>>>>> > undefined >>>>>>>>>> > (n1@REDACTED)7> (8421893*8)/1024. >>>>>>>>>> > 65796.0390625 >>>>>>>>>> > (n1@REDACTED)8> memory(). >>>>>>>>>> > [{total,92699464}, >>>>>>>>>> > {processes,8964000}, >>>>>>>>>> > {processes_used,8963152}, >>>>>>>>>> > {system,83735464}, >>>>>>>>>> > {atom,429569}, >>>>>>>>>> > {atom_used,421768}, >>>>>>>>>> > {binary,199040}, >>>>>>>>>> > {code,10411520}, >>>>>>>>>> > {ets,69163032}] >>>>>>>>>> > >>>>>>>>>> > The memory allocated to table 'ssl_otp_session_cache' is >>>>>>>>>> roughly 64 MB in 12 hours. >>>>>>>>>> > >>>>>>>>>> > On an OpenBSD platform such process gets killed immediately it >>>>>>>>>> hits some memory and/or CPU limits. >>>>>>>>>> > >>>>>>>>>> > To make this test on OpenBSD I had to put 'infinit' to memory, >>>>>>>>>> otherwise the Erlang VM would be killed. >>>>>>>>>> > >>>>>>>>>> > How can one control , tweak or configure this table such that >>>>>>>>>> it does not accumulate such data at such high rate. >>>>>>>>>> > >>>>>>>>>> > I seems the table being created private, and there is no way to >>>>>>>>>> ets:delete_all_objects/1 from table manually. >>>>>>>>>> > >>>>>>>>>> > I know that this table caches some SSL data related to clients, >>>>>>>>>> but the client has the same IP address, >>>>>>>>>> > and I wonder why is neccesary to store a lot of SSL connection >>>>>>>>>> info about the same client when only the ephemeral peer port >>>>>>>>>> > differs? >>>>>>>>>> > >>>>>>>>>> > How the size of this table can be held in reasonable limits and >>>>>>>>>> the rate it's size increases ? >>>>>>>>>> > >>>>>>>>>> > Please if somebody shed some light on these issues. >>>>>>>>>> > >>>>>>>>>> > Thank you, >>>>>>>>>> > >>>>>>>>>> > Bogdan >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > _______________________________________________ >>>>>>>>>> > erlang-questions mailing list >>>>>>>>>> > erlang-questions@REDACTED >>>>>>>>>> > http://erlang.org/mailman/listinfo/erlang-questions >>>>>>>>>> >>>>>>>>>> "Installing applications can lead to corruption over time. >>>>>>>>>> Applications gradually write over each other's libraries, partial upgrades >>>>>>>>>> occur, user and system errors happen, and minute changes may be >>>>>>>>>> unnoticeable and difficult to fix" >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> erlang-questions mailing list >>>>>>>>>> erlang-questions@REDACTED >>>>>>>>>> http://erlang.org/mailman/listinfo/erlang-questions >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rtrlists@REDACTED Mon Nov 10 11:06:39 2014 From: rtrlists@REDACTED (Robert Raschke) Date: Mon, 10 Nov 2014 10:06:39 +0000 Subject: [erlang-questions] Lager & Common Test question In-Reply-To: References: Message-ID: Hi Oleg, a very recent addition to the lager beckends appears to be a common_test one: https://github.com/basho/lager/blob/master/src/lager_common_test_backend.erl So, you could either run against the HEAD, or alternatively (as I do), copy the file next to your CT suites and tweak as you require (I changed the ct:pal/1 calls to ct:log/1). My CT sys.config now contains: { lager, [ { handlers, [ { lager_console_backend, error }, { lager_common_test_backend, debug } ] } ] } So, for me, everything goes into the CT logs, and only errors to the console. Hope this helps, Robby On 9 November 2014 21:24, Oleg Tsarev wrote: > Hello! > > I use lager + common test. > I configured lager for output to console. > > During test lager calls print output to my terminal, and test-related log > is empty. > io:format works as I want - no output in my terminal, only in test-related > log. > > How I can configure lager for see lager outputs in the log file instead of > terminal? > > Best regards, Oleg > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From oleg@REDACTED Mon Nov 10 10:17:10 2014 From: oleg@REDACTED (Oleg Tsarev) Date: Mon, 10 Nov 2014 13:17:10 +0400 Subject: [erlang-questions] Lager & Common Test question In-Reply-To: References: Message-ID: Hi, I have found solution already - it is common_test_backend from lager. By the way, on link that you reference you have no words about common_test_backend. I found it by chance. Best Regards, Oleg On Mon, Nov 10, 2014 at 10:29 AM, Minin Maxim wrote: > You have to configure lager's backends for your purpose - see > https://github.com/basho/lager#configuration > > > > Regards > > Maxim > > > > *Von:* erlang-questions-bounces@REDACTED [mailto: > erlang-questions-bounces@REDACTED] *Im Auftrag von *Oleg Tsarev > *Gesendet:* Sonntag, 9. November 2014 22:25 > *An:* erlang-questions@REDACTED > *Betreff:* [erlang-questions] Lager & Common Test question > > > > Hello! > > > > I use lager + common test. > > I configured lager for output to console. > > > > During test lager calls print output to my terminal, and test-related log > is empty. > > io:format works as I want - no output in my terminal, only in test-related > log. > > > > How I can configure lager for see lager outputs in the log file instead of > terminal? > > > > Best regards, Oleg > -------------- next part -------------- An HTML attachment was scrubbed... URL: From imantc@REDACTED Mon Nov 10 10:29:00 2014 From: imantc@REDACTED (Imants Cekusins) Date: Mon, 10 Nov 2014 10:29:00 +0100 Subject: [erlang-questions] ssl: a possible bug Message-ID: An example at http://www.erlang.org/doc/apps/ssl/using_ssl.html 2.2 Setting up connections Minmal example results in error: (server@REDACTED)4> ssl:ssl_accept(Socket). {error, {keyfile,{badmatch,{error,{asn1,{invalid_length,6}}}}}} OTP 17.3 From oleg@REDACTED Mon Nov 10 11:20:04 2014 From: oleg@REDACTED (Oleg Tsarev) Date: Mon, 10 Nov 2014 14:20:04 +0400 Subject: [erlang-questions] Lager & Common Test question In-Reply-To: References: Message-ID: Yes, see my previous letter - it is exactly what I want and it works :) Thank you! On Mon, Nov 10, 2014 at 1:06 PM, Robert Raschke wrote: > Hi Oleg, > > a very recent addition to the lager beckends appears to be a common_test > one: > https://github.com/basho/lager/blob/master/src/lager_common_test_backend.erl > > So, you could either run against the HEAD, or alternatively (as I do), > copy the file next to your CT suites and tweak as you require (I changed > the ct:pal/1 calls to ct:log/1). > > My CT sys.config now contains: > > { lager, [ > { handlers, [ > { lager_console_backend, error }, > { lager_common_test_backend, debug } > ] } > ] } > > So, for me, everything goes into the CT logs, and only errors to the > console. > > Hope this helps, > Robby > > > On 9 November 2014 21:24, Oleg Tsarev wrote: > >> Hello! >> >> I use lager + common test. >> I configured lager for output to console. >> >> During test lager calls print output to my terminal, and test-related log >> is empty. >> io:format works as I want - no output in my terminal, only in >> test-related log. >> >> How I can configure lager for see lager outputs in the log file instead >> of terminal? >> >> Best regards, Oleg >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From imantc@REDACTED Mon Nov 10 11:52:03 2014 From: imantc@REDACTED (Imants Cekusins) Date: Mon, 10 Nov 2014 11:52:03 +0100 Subject: [erlang-questions] ssl: a possible bug In-Reply-To: References: Message-ID: This is not a bug. Please ignore my previous message If the private key file is password protected, need to specify the password: As per http://www.erlang.org/doc/apps/ssl/using_ssl.html 2 server> {ok, ListenSocket} = ssl:listen(9999, [{certfile, "cert.pem"}, {keyfile, "key.pem"},{reuseaddr, true}]). {ok,{sslsocket, [...]}} With password specified: {ok, ListenSocket} = ssl:listen(9999,[{keyfile,Pem_file_name},{password,Password_value},{reuseaddr,true}]). On 10 November 2014 10:29, wrote: > An example at > http://www.erlang.org/doc/apps/ssl/using_ssl.html > > 2.2 Setting up connections > Minmal example > > results in error: > (server@REDACTED)4> ssl:ssl_accept(Socket). > {error, > {keyfile,{badmatch,{error,{asn1,{invalid_length,6}}}}}} > > > OTP 17.3 From roger@REDACTED Mon Nov 10 16:50:45 2014 From: roger@REDACTED (Roger Lipscombe) Date: Mon, 10 Nov 2014 15:50:45 +0000 Subject: [erlang-questions] compile:forms is sometimes slow Message-ID: I'm using 'meck' in my unit tests, and -- on a particular machine, of which more later -- it's slow, sometimes. It appears to be impossible to tell eunit (via rebar) to relax its 5 second timeout, without annotating every test and fixture (which I don't want to do), so I'm trying to figure out why it's so slow. I've tracked down the slowness, and it appears to be in compile:forms/2. Usually, it takes around 2-3ms, but occasionally, it takes ~6 seconds. Any ideas why? Oh, the machine? It's a VirtualBox VM, 2GB RAM, 6 cores, VMDK is on an SSD. It's noticeably slower than the host, but not painfully so. It seems like the performance just drops off a cliff under certain conditions. Regards, Roger. From federico.carrone@REDACTED Mon Nov 10 17:19:39 2014 From: federico.carrone@REDACTED (Federico Carrone) Date: Mon, 10 Nov 2014 16:19:39 +0000 Subject: [erlang-questions] Request for comments: gut, a template printing, aka scaffolding, tool for Erlang. Like rails generate or yeoman Message-ID: Hi, I have created a small tool for setting up projects (ranch, cowboy projects, etc) and scaffolding files (gen_servers, supervisors, cowboy handlers, gen_fsm, etc) in erlang: https://unbalancedparentheses.github.io/gut/ The generators are searched and downloaded from github (like yeoman) so thtat anybody can create a generator and use it with gut. Version 1 will be released this week after I got some input and comments from the community. Please test it and let me know what you think! Obviously this tool is meant for erlang newcomers or people like me that like the tools available in the nodejs, clojure, ruby and python communities. Thanks, Federico Carrone. *roadmap:* Present - v0.8: Basic functionality 12 November - v0.9: Better logs 15 November - v1.0: Support many variables in generators. Better documentation for example on how to create your own generators. Odds and ends. -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew@REDACTED Mon Nov 10 17:32:14 2014 From: andrew@REDACTED (Andrew Thompson) Date: Mon, 10 Nov 2014 11:32:14 -0500 Subject: [erlang-questions] Lager & Common Test question In-Reply-To: References: Message-ID: <20141110163214.GY26437@hijacked.us> On Mon, Nov 10, 2014 at 02:20:04PM +0400, Oleg Tsarev wrote: > Yes, see my previous letter - it is exactly what I want and it works :) > Thank you! The common_test background was a contribution, if anyone has any documentation they feel it would benefit from, please feel free to make a PR. I haven't used it, so I'm not in a position to document it. Andrew From sean@REDACTED Mon Nov 10 18:00:01 2014 From: sean@REDACTED (Functional Jobs) Date: Mon, 10 Nov 2014 12:00:01 -0500 Subject: [erlang-questions] New Functional Programming Job Opportunities Message-ID: <5460ef16568d6@functionaljobs.com> Here are some functional programming job opportunities that were posted recently: Senior Software Engineer at McGraw-Hill Education http://functionaljobs.com/jobs/8760-senior-software-engineer-at-mcgraw-hill-education Cheers, Sean Murphy FunctionalJobs.com From eric.pailleau@REDACTED Mon Nov 10 18:43:56 2014 From: eric.pailleau@REDACTED (Eric Pailleau) Date: Mon, 10 Nov 2014 18:43:56 +0100 Subject: [erlang-questions] Mnesia vs. ets+dets Message-ID: <4dnqv441tqqa3fh52v7hqvg4.1415641436349@email.android.com> I'm afraid you found the answer by yourself. You have to dump on disk older entries from RAM, but this need to code it. by the way, coding in Erlang is cheap. if so, do a module and share ! ? Envoy? depuis mon mobile ? Eric From richard.youngkin@REDACTED Mon Nov 10 21:26:56 2014 From: richard.youngkin@REDACTED (Youngkin, Rich) Date: Mon, 10 Nov 2014 13:26:56 -0700 Subject: [erlang-questions] SSL error Message-ID: Hi all, I'm having an issue sending an http request to a service and I'm not sure if it's a problem with the certificate or in Erlang. The app is using R15B03. I've included the Erlang error message below. I tried sending the request via curl and it looks fine to my not-very-proficient-in-ssl eyes. Any ideas where I should focus - Erlang or the certificate? Thanks, Rich I used "some*" to replace actual values... 2014-11-07T17:55:31Z ERRORED sub=00000ce5-cb71-83fc-3b95-76d3eed24d94 msg=00000ce5-cbbe-f055-3b95-76d3eed24d94 del_attempt=00000ce5-cbbe-f055-3b95-76d3eed24d94 reason={'EXIT', {{{function_clause, [{pubkey_cert,digest_type, [{1,2,840,113549,1,1,12}], [{file,"pubkey_cert.erl"},{line,377}]}, {pubkey_cert,extract_verify_data,2, [{file,"pubkey_cert.erl"},{line,358}]}, {pubkey_cert,verify_signature,4, [{file,"pubkey_cert.erl"},{line,362}]}, {pubkey_cert,validate_signature,6, [{file,"pubkey_cert.erl"},{line,160}]}, {public_key,validate,2,[{file,"public_key.erl"},{line,647}]}, {public_key,path_validation,2,[{file,"public_key.erl"},{line,591}]}, {ssl_handshake,certify,7,[{file,"ssl_handshake.erl"},{line,218}]}, {ssl_connection,certify,2, [{file,"ssl_connection.erl"},{line,514}]}]}, {gen_fsm,sync_send_all_state_event,[<0.9974.31>,start,10000]}}, {gen_server,call, [<0.8657.31>, {send_req, {{url,"https:// cert.somedomain.com/someresource";, "cert.somedomain.com ",443,undefined,undefined, "/message_receiver",https}, [{"Content-Type","application/x-www-form-urlencoded"}], post, ... curl -i -v -X POST https://cert.somedomain.com/someresource * About to connect() to cert.somedomain.comport 443 (#0) * Trying 54.225.73.122... connected * successfully set certificate verify locations: * CAfile: none CApath: /etc/ssl/certs * SSLv3, TLS handshake, Client hello (1): * SSLv3, TLS handshake, Server hello (2): * SSLv3, TLS handshake, CERT (11): * SSLv3, TLS handshake, Server key exchange (12): * SSLv3, TLS handshake, Server finished (14): * SSLv3, TLS handshake, Client key exchange (16): * SSLv3, TLS change cipher, Client hello (1): * SSLv3, TLS handshake, Finished (20): * SSLv3, TLS change cipher, Client hello (1): * SSLv3, TLS handshake, Finished (20): * SSL connection using ECDHE-RSA-AES128-SHA * Server certificate: * subject: C=US; postalCode=SomeZipCode; ST=NJ; L=Old Tappan; street=200 Old Tappan Rd; O=SomeOrgName; OU=Web Security; OU=Enterprise SSL Wildcard; CN=*.somedomain.com * start date: 2014-10-09 00:00:00 GMT * expire date: 2017-10-08 23:59:59 GMT * subjectAltName: cert.somedomain.com matched * issuer: C=US; ST=DE; L=Wilmington; O=Corporation Service Company; CN=Trusted Secure Certificate Authority 5 * SSL certificate verify ok. > POST /message_receiver HTTP/1.1 > User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3 > Host: cert.somedomain.com > Accept: */* > < HTTP/1.1 200 OK HTTP/1.1 200 OK < Cache-Control: max-age=0, private, must-revalidate Cache-Control: max-age=0, private, must-revalidate < Content-Type: text/html; charset=utf-8 Content-Type: text/html; charset=utf-8 < Date: Mon, 10 Nov 2014 20:13:26 GMT Date: Mon, 10 Nov 2014 20:13:26 GMT < ETag: "7215ee9c7d9dc229d2921a40e899ec5f" ETag: "7215ee9c7d9dc229d2921a40e899ec5f" < Server: Apache Server: Apache < Set-Cookie: _lc_session=03e19badfa917af689f53b27a2297f8a; path=/; HttpOnly Set-Cookie: _lc_session=03e19badfa917af689f53b27a2297f8a; path=/; HttpOnly < Status: 200 OK Status: 200 OK < X-Request-Id: bcd72a89933142d8557af9c386701494 X-Request-Id: bcd72a89933142d8557af9c386701494 < X-UA-Compatible: IE=Edge,chrome=1 X-UA-Compatible: IE=Edge,chrome=1 < Content-Length: 1 Content-Length: 1 < Connection: keep-alive Connection: keep-alive < * Connection #0 to host cert.somedomain.com left intact * Closing connection #0 * SSLv3, TLS alert, Client hello (1): -------------- next part -------------- An HTML attachment was scrubbed... URL: From oleg@REDACTED Tue Nov 11 01:02:31 2014 From: oleg@REDACTED (Oleg Tsarev) Date: Tue, 11 Nov 2014 04:02:31 +0400 Subject: [erlang-questions] lager/io:format and sets Message-ID: Hello, I am logging some record with set. State = #state{client_id = ClientId, session_id = SessionId, user_id = UserId, channel_id_set = sets:new(), mailbox = []}, lager:debug("args ~p state ~p", [Args, lager:pr(State, ?MODULE)]), As result I received: **** User 2014-11-11 02:54:02.166 **** 02:54:02.166 ct@REDACTED debug qlp_client:init:89 args {0,"sid0",0} state #state{client_id=0,session_id="sid0",user_id=0,channel_id_set={set,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},mailbox=[]} Same with io:format 10> io:format("test ~p", [sets:new()]). test {set,0,16,16,8,80,48, {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}, {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}} How I can fix the output of the set in lager and io:format? Thank you very much, Oleg -------------- next part -------------- An HTML attachment was scrubbed... URL: From cian@REDACTED Tue Nov 11 02:13:47 2014 From: cian@REDACTED (Cian Synnott) Date: Tue, 11 Nov 2014 01:13:47 +0000 Subject: [erlang-questions] lager/io:format and sets In-Reply-To: References: Message-ID: On Tue, Nov 11, 2014 at 12:02 AM, Oleg Tsarev wrote: > How I can fix the output of the set in lager and io:format? > I don't see a "built in" way to fix this. To roll your own, you could "patch over" the record with sets:to_list/1 to make a printable version. E.g. something like printable(#state{channel_id_set=Set}=State) -> State#state{channel_id_set=sets:to_list(Set)}. lager:debug("args ~p, state ~p", [Args, lager:pr(printable(State), ?MODULE)), It's not super. You might need to change the record's type annotations to avoid upsetting Dialyzer. Cian From ryankbrown@REDACTED Tue Nov 11 04:40:53 2014 From: ryankbrown@REDACTED (Ryan Brown) Date: Mon, 10 Nov 2014 20:40:53 -0700 Subject: [erlang-questions] SSL error for valid(?) certificate Message-ID: <546185532a77c_73fa1855d3041733b@mxhero-gmail-gateway.mail> Hi all. I have a client that appears to have a legitimate ssl cert but I am getting the error seen here (http://pastebin.com/2TTGAQDm) when attempting to post to it's endpoint. It is dying in public_cert.erl in a place that does not seem to offer me any insight. Just curious if anybody had seen anything like this before. Thanks. -rb -------------- next part -------------- An HTML attachment was scrubbed... URL: From jerome@REDACTED Tue Nov 11 07:15:26 2014 From: jerome@REDACTED (=?windows-1252?Q?J=E9r=F4me_Desquilbet?=) Date: Tue, 11 Nov 2014 07:15:26 +0100 Subject: [erlang-questions] SSL error for valid(?) certificate In-Reply-To: <546185532a77c_73fa1855d3041733b@mxhero-gmail-gateway.mail> References: <546185532a77c_73fa1855d3041733b@mxhero-gmail-gateway.mail> Message-ID: <5461A97E.8060901@desquilbet.org> Why are you tracking http://pastebin.com/2TTGAQDm with a thing called mxHero? J. Le 11/11/2014 04:40, Ryan Brown a ?crit : > Hi all. I have a client that appears to have a legitimate ssl cert but I > am getting the error seen here ( http://pastebin.com/2TTGAQDm > > ) when attempting to post to it's endpoint. It is dying in > public_cert.erl in a place that does not seem to offer me any insight. > Just curious if anybody had seen anything like this before. > > Thanks. > > -rb > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From z@REDACTED Tue Nov 11 07:59:40 2014 From: z@REDACTED (Danil Zagoskin) Date: Tue, 11 Nov 2014 10:59:40 +0400 Subject: [erlang-questions] SSL error for valid(?) certificate In-Reply-To: <5461A97E.8060901@desquilbet.org> References: <546185532a77c_73fa1855d3041733b@mxhero-gmail-gateway.mail> <5461A97E.8060901@desquilbet.org> Message-ID: Seems like you have an old OTP ? there is no pubkey_cert:digest_type/1 exported these days SSL app gets fixed and improved every major release, so if possibe, try to reproduce this crash in OTP 17.3. On Tue, Nov 11, 2014 at 9:15 AM, J?r?me Desquilbet wrote: > Why are you tracking > http://pastebin.com/2TTGAQDm > with a thing called mxHero? > J. > > Le 11/11/2014 04:40, Ryan Brown a ?crit : > >> Hi all. I have a client that appears to have a legitimate ssl cert but I >> am getting the error seen here ( http://pastebin.com/2TTGAQDm >> >> ) when attempting to post to it's endpoint. It is dying in >> public_cert.erl in a place that does not seem to offer me any insight. >> Just curious if anybody had seen anything like this before. >> >> Thanks. >> >> -rb >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Danil Zagoskin | z@REDACTED -------------- next part -------------- An HTML attachment was scrubbed... URL: From caisonglu@REDACTED Tue Nov 11 08:03:28 2014 From: caisonglu@REDACTED (songlu cai) Date: Tue, 11 Nov 2014 15:03:28 +0800 Subject: [erlang-questions] Unbalanced Scheduler Problem caused by Using out of date active_no_runq In-Reply-To: References: <28140.1414655602@snookles.snookles.com> Message-ID: Hi Richard, Thank you for your help & quick reply! Follow your suggestion, I do another one-week test with four versions of erlang. V1: R16B03.origin with collapse problem V2: R16B03.fix with my own fix patch V3: R16B03.backport from github you provide V4: R17B03 from erlang.com During the one-week test, V1 collapses twice and lasts for nearly two days. V2 & V3 & V4 behave the same, balanced as normal. So I will use R16B03.backport to fix my problem. Thanks again. Best Regards, Songlu Cai 2014-11-05 6:14 GMT+08:00 Rickard Green : > Hi Songlu Cai, > > Thanks for your work on this! > > Although it is not an error that NONEMPTY and INACTIVE flag is set at > the same time it should occur rather infrequently. Assuming that this > would be the case, your fix should have little effect on the behavior, > so the root cause should be elsewhere. I've found a potential race > when the NONEMPTY flag could end up on the run-queue of a waiting > scheduler which could cause the problem you are seeing. > > When implementing support for balancing on scheduler utilization for > OTP 17.0 I rewrote the code that set the NONEMPTY flag. This rewrite > removes that potential race. I've back-ported this rewrite based on > R16B03-1. It can be found in the > rickard/R16B03-1/load_balance/OTP-11385 branch > < > https://github.com/rickard-green/otp/tree/rickard/R16B03-1/load_balance/OTP-11385 > > > of my github repo . Please > try it out and see if it solves your problem (use the same > configuration as before). > > Something similar to your fix should perhaps in the end be introduced > anyway (one also has to check the SUSPENDED flag though), since there > are no good reason to prevent the activation just because the > run-queue happen to be non-empty. It would, however, be good to see > that the root issue has been fixed first before introducing this. > > Regards, > Rickard Green, Erlang/OTP, Ericsson AB > > On Mon, Nov 3, 2014 at 4:18 PM, songlu cai wrote: > > Hi Scott, > > > > Last week I fix the bug in a simple way, then I run the fixed-version > > compared with the old unbalanced-version. > > The two nodes are with same pressure & timeline. > > The unbalanced(collapsed) state comes up several times on the old > version, > > but never show up on the fixed version. > > The pressure spreads averagely among 24 schedulers on the fixed version > > (especially when with high pressure). > > In fact, the fixed version is with higher pressure when old version runs > > into the unbalanced state. > > Because the old version is only with 4 schedulers, and easily gets to the > > bottleneck, its cpu is 400%, at the same time, fixed version is with cpu > > 1200%. > > So I can be sure that the root cause of unbalanced scheduler (scheduler > > collapse) is "using out of date active_no_runq", just as analyzed before. > > > > I just modify the chk_wake_sched function, code diff as below: > > > > Index: emulator/beam/erl_process.c > > =================================================================== > > --- emulator/beam/erl_process.c (revision 298281) > > +++ emulator/beam/erl_process.c (working copy) > > @@ -2694,6 +2694,16 @@ > > return 0; > > wrq = ERTS_RUNQ_IX(ix); > > flags = ERTS_RUNQ_FLGS_GET(wrq); > > + > > + if ( activate && > > + (flags & ERTS_RUNQ_FLG_NONEMPTY) && > > + (flags & ERTS_RUNQ_FLG_INACTIVE)) { > > + if (try_inc_no_active_runqs(ix+1)) > > + (void) ERTS_RUNQ_FLGS_UNSET(wrq, ERTS_RUNQ_FLG_INACTIVE); > > + wake_scheduler(wrq, 0); > > + return 1; > > + } > > + > > if (!(flags & (ERTS_RUNQ_FLG_SUSPENDED|ERTS_RUNQ_FLG_NONEMPTY))) { > > if (activate) { > > if (try_inc_no_active_runqs(ix+1)) > > > > It saves the scheduler from the weird state. It is not a perfect fix, > but an > > effective one. > > Scott, would you please apply this patch to R16B03 and run your test case > > again? > > Thank you very much and waiting for your reply. > > And I will run it for a week to ensure that we do fix the problem. > > > > Best Regards, > > Songlu Cai > > > > 2014-10-31 10:55 GMT+08:00 songlu cai : > >> > >> Hi Scott, > >> > >> > >> > >> Thanks for your attention & quick reply. > >> > >> It seems that quite a few people suffer from this problem. > >> > >> > >> > >> Scott>The best workaround is to use "+scl false" and "+sfwi" with a > value > >> of 500 or a bit smaller > >> > >> 1, we set +sfwi 500 > >> > >> 2,at first, we set +scl false, but it causes unbalanced runq length > among > >> all runqs on R16B03, then we set +scl true (by default), so +scl false > is > >> not a safe choice on R16B03 > >> > >> > >> > >> Our test cmdline: > >> > >> /home/Xxx/erts-5.10.4/bin/beam.smp -zdbbl 8192 -sbt db -sbwt very_short > >> -swt low -sfwi 500 -MBmmsbc 100 -MHmmsbc 100 -MBmmmbc 100 -MHmmmbc 100 > >> -MMscs 20480 -MBsmbcs 10240 -MHsbct 2048 -W w -e 50000 -Q 1000000 -hmbs > >> 46422 -hms 2586 -P 1000000 -A 16 -K true -d -Bi -sct > >> > L23T0C0P0N0:L22T1C1P0N0:L21T2C2P0N0:L20T3C3P0N0:L19T4C4P0N0:L18T5C5P0N0:L17T6C0P1N1:L16T7C1P1N1:L15T8C2P1N1:L14T9C3P1N1:L13T10C4P1N1:L12T11C5P1N1:L11T12C0P0N0:L10T13C1P0N0:L9T14C2P0N0:L8T15C3P0N0:L7T16C4P0N0:L6T17C5P0N0:L5T18C0P1N1:L4T19C1P1N1:L3T20C2P1N1:L2T21C3P1N1:L1T22C4P1N1:L0T23C5P1N1 > >> -swct medium -- -root /home/Xxx/dir -progname Xxx -- -home /root -- > -boot > >> /home/Xxx/dir -mode interactive -config /home/Xxx/sys.config > -shutdown_time > >> 30000 -heart -setcookie Xxx -name proxy@REDACTED ? console > >> > >> > >> > >> And , apart from everything else, INACTIVE|NONEMPTY is not a normal > state > >> of runq flag. > >> > >> Next few days, I will fix the not-yet-be-sure bug in my way based on > >> R16B03 and run the test cases again. > >> > >> > >> Best Regards, > >> > >> Songlu Cai > >> > >> > >> 2014-10-30 15:53 GMT+08:00 Scott Lystig Fritchie >: > >>> > >>> songlu cai wrote: > >>> > >>> slc> How to fix: > >>> > >>> slc> [...] > >>> > >>> slc> 3, Or Another Way? > >>> > >>> Wow, that's quite a diagnosis. I'm not a good judge of the race > >>> condition that you've found or your fix. I can provide some context, > >>> however, in case that you weren't aware of it. It might help to create > >>> a Real, Final, 100% Correct Fix ... something which does not exist > right > >>> now. > >>> > >>> The best workaround is to use "+scl false" and "+sfwi" with a value of > >>> 500 or a bit smaller. See the discussion last month about it, > >>> > >>> > >>> > http://erlang.org/pipermail/erlang-questions/2014-September/081017.html > >>> > >>> My colleague Joe Blomstedt wrote a demo program that can cause > scheduler > >>> collapse to happen pretty quickly. It might be useful for judging how > >>> well any fix works ... at Basho we had a terrible time trying to > >>> reproduce this bug before Joe found a semi-reliable trigger. > >>> > >>> https://github.com/basho/nifwait > >>> > >>> It is discussed in this email thread (which is broken across two URLs, > >>> sorry). > >>> > >>> > http://erlang.org/pipermail/erlang-questions/2012-October/069503.html > >>> (first message only, I don't know why) > >>> > http://erlang.org/pipermail/erlang-questions/2012-October/069585.html > >>> (the rest of the thread) > >>> > >>> If your analysis is correct ... then hopefully this can lead quickly to > >>> a Real, Final, 100% Correct Fix. I'm tired of diagnosing systems that > >>> suffer scheduler collapse and then discover that the customer forgot to > >>> add the magic +sfwi and +scl flags in their runtime configuration to > >>> work around that !@#$! bug. > >>> > >>> -Scott > >> > >> > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > > -- > Rickard Green, Erlang/OTP, Ericsson AB > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Ola.Backstrom@REDACTED Tue Nov 11 09:33:40 2014 From: Ola.Backstrom@REDACTED (=?iso-8859-1?Q?Ola_B=E4ckstr=F6m?=) Date: Tue, 11 Nov 2014 08:33:40 +0000 Subject: [erlang-questions] lager/io:format and sets Message-ID: <9551e5a5098840f9a10e2d720307ce56@exsrvfe.ls.local> I would not recommend to change the internal representation just for making pretty printing simpler. I'd either make a private conversion function, or if this concerns an otp genserver implement the gen_server optional behaviour format_status/2. You'd have to do the conversion of the State yourself, and then call the function every time you want to log the state. The advantage of making the conversion in the format_status/2 function is that also crash reports will contain converted state. Perhaps also in other tools too, like observer, I haven't tried. /Ola -----Original Message----- From: erlang-questions-bounces@REDACTED [mailto:erlang-questions-bounces@REDACTED] On Behalf Of Cian Synnott Sent: den 11 november 2014 02:14 To: Oleg Tsarev Cc: erlang-questions@REDACTED Subject: Re: [erlang-questions] lager/io:format and sets >On Tue, Nov 11, 2014 at 12:02 AM, Oleg Tsarev wrote: >> How I can fix the output of the set in lager and io:format? >> >I don't see a "built in" way to fix this. To roll your own, you could "patch over" the record with sets:to_list/1 to make a printable version. E.g. something like > >printable(#state{channel_id_set=Set}=State) -> > State#state{channel_id_set=sets:to_list(Set)}. > >lager:debug("args ~p, state ~p", [Args, lager:pr(printable(State), ?MODULE)), > >It's not super. You might need to change the record's type annotations to avoid upsetting Dialyzer. > >Cian From ingela.andin@REDACTED Tue Nov 11 10:17:08 2014 From: ingela.andin@REDACTED (Ingela Andin) Date: Tue, 11 Nov 2014 10:17:08 +0100 Subject: [erlang-questions] SSL error In-Reply-To: References: Message-ID: Hi! That is a really old bug, you need to have public_key-0.17 at least to avoid it. Latest is public_key-0.22.1 Regards Ingela Erlang/OTP team - Ericsson AB 2014-11-10 21:26 GMT+01:00 Youngkin, Rich : > Hi all, > > I'm having an issue sending an http request to a service and I'm not sure > if it's a problem with the certificate or in Erlang. The app is using > R15B03. I've included the Erlang error message below. I tried sending the > request via curl and it looks fine to my not-very-proficient-in-ssl eyes. > Any ideas where I should focus - Erlang or the certificate? > > Thanks, > Rich > > I used "some*" to replace actual values... > > 2014-11-07T17:55:31Z ERRORED sub=00000ce5-cb71-83fc-3b95-76d3eed24d94 > msg=00000ce5-cbbe-f055-3b95-76d3eed24d94 del_attempt=00000ce5-cbbe-f055-3b95-76d3eed24d94 > reason={'EXIT', {{{function_clause, [{pubkey_cert,digest_type, > [{1,2,840,113549,1,1,12}], [{file,"pubkey_cert.erl"},{line,377}]}, > {pubkey_cert,extract_verify_data,2, [{file,"pubkey_cert.erl"},{line,358}]}, > {pubkey_cert,verify_signature,4, [{file,"pubkey_cert.erl"},{line,362}]}, > {pubkey_cert,validate_signature,6, [{file,"pubkey_cert.erl"},{line,160}]}, > {public_key,validate,2,[{file,"public_key.erl"},{line,647}]}, > {public_key,path_validation,2,[{file,"public_key.erl"},{line,591}]}, > {ssl_handshake,certify,7,[{file,"ssl_handshake.erl"},{line,218}]}, > {ssl_connection,certify,2, [{file,"ssl_connection.erl"},{line,514}]}]}, > {gen_fsm,sync_send_all_state_event,[<0.9974.31>,start,10000]}}, > {gen_server,call, [<0.8657.31>, {send_req, {{url,"https:// > > cert.somedomain.com/someresource";, "cert.somedomain.com > ",443,undefined,undefined, > "/message_receiver",https}, [{"Content-Type","application/x-www-form-urlencoded"}], > post, ... > > > curl -i -v -X POST https://cert.somedomain.com/someresource > * About to connect() to cert.somedomain.comport 443 (#0) > * Trying 54.225.73.122... connected > * successfully set certificate verify locations: > * CAfile: none > CApath: /etc/ssl/certs > * SSLv3, TLS handshake, Client hello (1): > * SSLv3, TLS handshake, Server hello (2): > * SSLv3, TLS handshake, CERT (11): > * SSLv3, TLS handshake, Server key exchange (12): > * SSLv3, TLS handshake, Server finished (14): > * SSLv3, TLS handshake, Client key exchange (16): > * SSLv3, TLS change cipher, Client hello (1): > * SSLv3, TLS handshake, Finished (20): > * SSLv3, TLS change cipher, Client hello (1): > * SSLv3, TLS handshake, Finished (20): > * SSL connection using ECDHE-RSA-AES128-SHA > * Server certificate: > * subject: C=US; postalCode=SomeZipCode; ST=NJ; L=Old Tappan; street=200 > Old Tappan Rd; O=SomeOrgName; OU=Web Security; OU=Enterprise SSL Wildcard; > CN=*.somedomain.com > * start date: 2014-10-09 00:00:00 GMT > * expire date: 2017-10-08 23:59:59 GMT > * subjectAltName: cert.somedomain.com matched > * issuer: C=US; ST=DE; L=Wilmington; O=Corporation Service Company; > CN=Trusted Secure Certificate Authority 5 > * SSL certificate verify ok. > > POST /message_receiver HTTP/1.1 > > User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 > OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3 > > Host: cert.somedomain.com > > Accept: */* > > > < HTTP/1.1 200 OK > HTTP/1.1 200 OK > < Cache-Control: max-age=0, private, must-revalidate > Cache-Control: max-age=0, private, must-revalidate > < Content-Type: text/html; charset=utf-8 > Content-Type: text/html; charset=utf-8 > < Date: Mon, 10 Nov 2014 20:13:26 GMT > Date: Mon, 10 Nov 2014 20:13:26 GMT > < ETag: "7215ee9c7d9dc229d2921a40e899ec5f" > ETag: "7215ee9c7d9dc229d2921a40e899ec5f" > < Server: Apache > Server: Apache > < Set-Cookie: _lc_session=03e19badfa917af689f53b27a2297f8a; path=/; > HttpOnly > Set-Cookie: _lc_session=03e19badfa917af689f53b27a2297f8a; path=/; HttpOnly > < Status: 200 OK > Status: 200 OK > < X-Request-Id: bcd72a89933142d8557af9c386701494 > X-Request-Id: bcd72a89933142d8557af9c386701494 > < X-UA-Compatible: IE=Edge,chrome=1 > X-UA-Compatible: IE=Edge,chrome=1 > < Content-Length: 1 > Content-Length: 1 > < Connection: keep-alive > Connection: keep-alive > > < > * Connection #0 to host cert.somedomain.com left intact > * Closing connection #0 > * SSLv3, TLS alert, Client hello (1): > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ingela.andin@REDACTED Tue Nov 11 10:22:13 2014 From: ingela.andin@REDACTED (Ingela Andin) Date: Tue, 11 Nov 2014 10:22:13 +0100 Subject: [erlang-questions] SSL error for valid(?) certificate In-Reply-To: References: <546185532a77c_73fa1855d3041733b@mxhero-gmail-gateway.mail> <5461A97E.8060901@desquilbet.org> Message-ID: Hi! Seems to be the same error as in the thread "SSL error" e.i. you have a very old version of public_key and you need to upgrade it to atleast public_key-0.17 which is quite old too. Regards Ingela Erlang/OTP Team - Ericsson AB 2014-11-11 7:59 GMT+01:00 Danil Zagoskin : > Seems like you have an old OTP ? there is no pubkey_cert:digest_type/1 > exported these days > SSL app gets fixed and improved every major release, so if possibe, try to > reproduce this crash in OTP 17.3. > > On Tue, Nov 11, 2014 at 9:15 AM, J?r?me Desquilbet > wrote: > >> Why are you tracking >> http://pastebin.com/2TTGAQDm >> with a thing called mxHero? >> J. >> >> Le 11/11/2014 04:40, Ryan Brown a ?crit : >> >>> Hi all. I have a client that appears to have a legitimate ssl cert but I >>> am getting the error seen here ( http://pastebin.com/2TTGAQDm >>> >>> ) when attempting to post to it's endpoint. It is dying in >>> public_cert.erl in a place that does not seem to offer me any insight. >>> Just curious if anybody had seen anything like this before. >>> >>> Thanks. >>> >>> -rb >>> >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > > > -- > Danil Zagoskin | z@REDACTED > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aschultz@REDACTED Tue Nov 11 10:29:16 2014 From: aschultz@REDACTED (Andreas Schultz) Date: Tue, 11 Nov 2014 09:29:16 +0000 (UTC) Subject: [erlang-questions] SSL error for valid(?) certificate In-Reply-To: <642375167.943022.1415698149267.JavaMail.zimbra@tpip.net> References: <546185532a77c_73fa1855d3041733b@mxhero-gmail-gateway.mail> Message-ID: <66536814.943023.1415698156208.JavaMail.zimbra@tpip.net> Hi, ----- On 11 Nov, 2014, at 04:40, Ryan Brown ryankbrown@REDACTED wrote: > Hi all. I have a client that appears to have a legitimate ssl cert but I am > getting the error seen here ( http://pastebin.com/2TTGAQDm ) when attempting to > post to it's endpoint. It is dying in public_cert.erl in a place that does not > seem to offer me any insight. Just curious if anybody had seen anything like > this before. The error indicates that your certificate is signed with sha384WithRSAEncryption. I'm not sure what the earliest version that supports that algorithm is, but any version that supports TLS 1.2 should have it. That is definitely R16B03, R15B03 might also work. Also, make sure the underlying OpenSSL version has sha384 support enabled. Andreas > > Thanks. > > -rb > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- -- Dipl. Inform. Andreas Schultz From semmitmondo@REDACTED Tue Nov 11 10:24:25 2014 From: semmitmondo@REDACTED (semmit mondo) Date: Tue, 11 Nov 2014 10:24:25 +0100 (CET) Subject: [erlang-questions] beginner architectural question Message-ID: Hi, I have an application with a structural pattern that comes up again andagain, and I would like to ask you how would you solve this. My app hastwo kinds of processes: a singleton that is a server collecting data, andworkers that do some calculation and send the result to the server.There can be any number of workers and there's usually one datacollector server. I think this is a very simple and common architecture.The problem with this is that the calculators have to know the pid of thecollector and the whole thing has to be supervised. The simplest solution is that under a main supervisor I spawn the collectorserver and register it with a name. Next to the collector I start anothersupervisor that manages the pool of the calculators and those processessimply use the collectors name as the message destination when they'redone with the jod. The main supervisor uses one_for_one strategy, whilethe subsupervisor is a simple_one_for_one. I feel that this is not the best I c ould do. Another solution could be touse rest_for_one in the top supervisor, not register a name for thecollector, but push down the pid of the top supervisor to the sub-supervisor, and then in the init function of the sub-supervisor I canask the top supervisor (with a which_children() call) to give me thepid of the collector. So I can give it to the calculators as an argument.Seems a bit difficult, but has the benefit not using a registered namefor a purely internal use case. There's also a wrong solution with a top supervisor with no staticchildren. In that case I can start the children manually. First I canstart the collector, then I have got the pid of it and can use it whenstarting the calculators. The problem here is that when the collectorgets restarted, the calculators won't have the right pid. Even if thetop supervisor uses rest_for_one strategy, and the calculators getrestarted, there's no mechanism to start calculators with the newpid of the collector. (I can't give the supervisor a function ! thatdescribes the way how to restart the whole thing, can I?) This wrong approach can be fixed if the top level supervisor exitswhen the collector or the sub-supervisor fails (that's a {one_for_one,0, 1} supervisor setup) and a plus layer of supervision above thissupervisor gets into play. But that's just too many layers andsupervisors for a simple thing like this, am I right? I believe that the whole thing boils down to that I have not muchcontrol over the order of child starts and I can't move informationbetween children of the same supervisor. It would be nice if asupervisor could supply some sort of a local name registry for theprocesses below them but as far as I know there's only a global(node local) one that could be used here. Which architecture would you choose? Are there other alternativestructures I haven't thought of? What do you thing which one is thebest approach and why? I'm sorry for the long and beginner question. -------------- next part -------------- An HTML attachment was scrubbed... URL: From n.oxyde@REDACTED Tue Nov 11 11:59:30 2014 From: n.oxyde@REDACTED (Anthony Ramine) Date: Tue, 11 Nov 2014 11:59:30 +0100 Subject: [erlang-questions] Request for comments: gut, a template printing, aka scaffolding, tool for Erlang. Like rails generate or yeoman In-Reply-To: References: Message-ID: <676D92DB-7BCA-4A87-AB3C-6CA9F1B67C8E@gmail.com> What about renaming the 'Stalin' example tree in your readme file? Le 10 nov. 2014 ? 17:19, Federico Carrone a ?crit : > Hi, > > I have created a small tool for setting up projects (ranch, cowboy projects, etc) and scaffolding files (gen_servers, supervisors, cowboy handlers, gen_fsm, etc) in erlang: https://unbalancedparentheses.github.io/gut/ > > The generators are searched and downloaded from github (like yeoman) so thtat anybody can create a generator and use it with gut. > > Version 1 will be released this week after I got some input and comments from the community. > > Please test it and let me know what you think! Obviously this tool is meant for erlang newcomers or people like me that like the tools available in the nodejs, clojure, ruby and python communities. > > Thanks, > Federico Carrone. > > roadmap: > Present - v0.8: Basic functionality > 12 November - v0.9: Better logs > 15 November - v1.0: Support many variables in generators. Better documentation for example on how to create your own generators. Odds and ends. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From maxim@REDACTED Tue Nov 11 12:02:43 2014 From: maxim@REDACTED (Maxim Sokhatsky) Date: Tue, 11 Nov 2014 13:02:43 +0200 Subject: [erlang-questions] beginner architectural question In-Reply-To: References: Message-ID: Hello! Here is best practice for PubSub. In init of your collector you register process pid with symbolic name. In Erlang applications the underlying implementation usually relies on ETS. You may think of it as an variable in ETS table. ? ? ?collector:init([]) -> pubsub:reg(?collector?,self()). In calculators you should not use an unique pid, but a given symbolic name of the only subscriber in your system, the ?collector?. ? ? ?calculator:reduce(Data) -> pubsub:send(?collector?,Data). This fits the GPROC pubsub and RABBITMQ. --? Maxim From daniel.abrahamsson@REDACTED Tue Nov 11 13:49:49 2014 From: daniel.abrahamsson@REDACTED (Daniel Abrahamsson) Date: Tue, 11 Nov 2014 13:49:49 +0100 Subject: [erlang-questions] SSL error for valid(?) certificate In-Reply-To: <66536814.943023.1415698156208.JavaMail.zimbra@tpip.net> References: <546185532a77c_73fa1855d3041733b@mxhero-gmail-gateway.mail> <642375167.943022.1415698149267.JavaMail.zimbra@tpip.net> <66536814.943023.1415698156208.JavaMail.zimbra@tpip.net> Message-ID: R15B03 unfortunately does not support sha384WithRSAEncryption. I've had to patch in my own OTP builds of R15. //Daniel On Tue, Nov 11, 2014 at 10:29 AM, Andreas Schultz wrote: > Hi, > > ----- On 11 Nov, 2014, at 04:40, Ryan Brown ryankbrown@REDACTED wrote: > > > Hi all. I have a client that appears to have a legitimate ssl cert but I > am > > getting the error seen here ( http://pastebin.com/2TTGAQDm ) when > attempting to > > post to it's endpoint. It is dying in public_cert.erl in a place that > does not > > seem to offer me any insight. Just curious if anybody had seen anything > like > > this before. > > The error indicates that your certificate is signed with > sha384WithRSAEncryption. > I'm not sure what the earliest version that supports that algorithm is, > but any version > that supports TLS 1.2 should have it. That is definitely R16B03, R15B03 > might also > work. > Also, make sure the underlying OpenSSL version has sha384 support enabled. > > Andreas > > > > > Thanks. > > > > -rb > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > -- > -- > Dipl. Inform. > Andreas Schultz > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sverker.eriksson@REDACTED Tue Nov 11 14:41:08 2014 From: sverker.eriksson@REDACTED (Sverker Eriksson) Date: Tue, 11 Nov 2014 14:41:08 +0100 Subject: [erlang-questions] [ANN] jch-erl 0.1 - Fast minimal memory consistent hashing for Erlang/OTP. In-Reply-To: References: Message-ID: <546211F4.8060705@erix.ericsson.se> Maybe not worth a pint, but... If you want to ensure the first argument can be 64 bit long you should use enif_get_uint64. Type 'long' is only 64 bit on most 64bit OS (not Windows). /Sverker, Erlang/OTP On 11/07/2014 10:08 PM, Darach Ennis wrote: > Hey folk. > > jch-erl 0.1 freshly baked. > > NIF wrapper for Jump Consistent Hash algorithm by John Lamping and Eric > Veach developed at Google, Inc. Paper: "A Fast, Minimal Memory, Consistent > Hash Algorithm". > > http://arxiv.org/ftp/arxiv/papers/1406/1406.2294.pdf > > This implementation uses the xorshift64* pseudo-random number generator > rather than the linear congruential generator in the paper as it is > reasonably fast but, more importantly, memory efficient. > > Usage > > A single function jch:ch/2 is offered. Simply pass in a 64 bit long (or > less) integer key argument followed by the desired bucket or continuum > partition size. The function returns the partition allocated for the key. > > Performance is very stable as bucket size increases and distribution across > the ring is stable (standard deviation for a reasonable sample size is > typically <2% relative to the mean). > > Example > > % ERL_LIBS=deps erl +sfwi 1 +scl false -pa ebin > Erlang R16B02 (erts-5.10.3) [source] [64-bit] [smp:8:8] [async-threads:10] > [hipe] [kernel-poll:false] [dtrace] > > Eshell V5.10.3 (abort with ^G) > 1> jch:ch(1,128). > 29 > 2> jch:ch(13,128). > 121 > 3> jch:ch(13,128). > 121 > 4> jch:ch(trunc(math:pow(2,64))-1,128). > 78 > 5> jch:ch(trunc(math:pow(2,64)),128). %% off by 1 mate > ** exception error: bad argument > in function jch:ch/2 > called as jch:ch(18446744073709551616,128) > 6> %% off by 1 mate > 6> > > Access times are typically sub-microsecond. Performance tests of the erlang > NIF > and underlying C function are included. > > Example performance for a space of 1 billion buckets: > > - 1B Buckets. Hash performance. > ch1b N: 10 Min: -----1 Max: -----2 Median: -----1 Avg: -----1 Elapsed: 16 > ch1b N: 100 Min: -----0 Max: -----1 Median: -----1 Avg: -----1 Elapsed: 102 > ch1b N: 1000 Min: -----0 Max: -----9 Median: -----0 Avg: -----0 Elapsed: > 1178 > ch1b N: 10000 Min: -----0 Max: ----13 Median: -----0 Avg: -----0 Elapsed: > 10146 > ch1b N: 100000 Min: -----0 Max: ----59 Median: -----0 Avg: -----0 Elapsed: > 109772 > ch1b N: 1000000 Min: -----0 Max: ---886 Median: -----0 Avg: -----0 Elapsed: > 1048561 > ch1b N: 10000000 Min: -----0 Max: --3734 Median: -----0 Avg: -----0 > Elapsed: 11757297 > > And a sample of how keys partition over the space: > > - 32 Buckets. 10M hashes Uniform Distribution Check. > 312292 312457 312660 312522 313805 311987 311431 312299 > 311975 312834 312233 312125 313391 312222 312170 312389 > 311811 313134 312986 312434 312002 313060 312784 312699 > 312114 312170 312385 312721 313163 311883 313060 312802 > oOo| Min: 311431 Max: 313805 Median: 312389 Avg: 312500 Elapsed: 15748372 > Worst: 99.2435 Med: 99.5488 Avg: 99.5841 RSD: 0.9139 > > The RSD here is the standard deviation relative to the mean (avg) which is > less than 1% > after 10M hash operations on random uniform test data. That's pretty good. > > A pint (or fancy fizzy water, or cup of pant wetting tea...) to anyone who > can strip off an order of magnitude off the max outlier latencies and > explain through scheduler, gc or emulator tuning what insight I missed or > runtime vagary i've yet to master. Mainly written to toy with > micro-benchmarking and get practice with tuning the emulator. > > Enjoy! > > Cheers, > > Darach. > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard.youngkin@REDACTED Tue Nov 11 15:45:10 2014 From: richard.youngkin@REDACTED (Youngkin, Rich) Date: Tue, 11 Nov 2014 07:45:10 -0700 Subject: [erlang-questions] SSL error In-Reply-To: References: Message-ID: Thanks, Ingela. I suspected it might be something like that. Cheers, Rich On Tue, Nov 11, 2014 at 2:17 AM, Ingela Andin wrote: > Hi! > > That is a really old bug, you need to have public_key-0.17 at least to > avoid it. Latest is public_key-0.22.1 > > Regards Ingela Erlang/OTP team - Ericsson AB > > 2014-11-10 21:26 GMT+01:00 Youngkin, Rich : > >> Hi all, >> >> I'm having an issue sending an http request to a service and I'm not sure >> if it's a problem with the certificate or in Erlang. The app is using >> R15B03. I've included the Erlang error message below. I tried sending the >> request via curl and it looks fine to my not-very-proficient-in-ssl eyes. >> Any ideas where I should focus - Erlang or the certificate? >> >> Thanks, >> Rich >> >> I used "some*" to replace actual values... >> >> 2014-11-07T17:55:31Z ERRORED sub=00000ce5-cb71-83fc-3b95-76d3eed24d94 >> msg=00000ce5-cbbe-f055-3b95-76d3eed24d94 del_attempt=00000ce5-cbbe-f055-3b95-76d3eed24d94 >> reason={'EXIT', {{{function_clause, [{pubkey_cert,digest_type, >> [{1,2,840,113549,1,1,12}], [{file,"pubkey_cert.erl"},{line,377}]}, >> {pubkey_cert,extract_verify_data,2, [{file,"pubkey_cert.erl"},{line,358}]}, >> {pubkey_cert,verify_signature,4, [{file,"pubkey_cert.erl"},{line,362}]}, >> {pubkey_cert,validate_signature,6, [{file,"pubkey_cert.erl"},{line,160}]}, >> {public_key,validate,2,[{file,"public_key.erl"},{line,647}]}, >> {public_key,path_validation,2,[{file,"public_key.erl"},{line,591}]}, >> {ssl_handshake,certify,7,[{file,"ssl_handshake.erl"},{line,218}]}, >> {ssl_connection,certify,2, [{file,"ssl_connection.erl"},{line,514}]}]}, >> {gen_fsm,sync_send_all_state_event,[<0.9974.31>,start,10000]}}, >> {gen_server,call, [<0.8657.31>, {send_req, {{url,"https:// >> >> cert.somedomain.com/someresource";, "cert.somedomain.com >> ",443,undefined,undefined, >> "/message_receiver",https}, [{"Content-Type","application/x-www-form-urlencoded"}], >> post, ... >> >> >> curl -i -v -X POST https://cert.somedomain.com/someresource >> * About to connect() to cert.somedomain.comport 443 (#0) >> * Trying 54.225.73.122... connected >> * successfully set certificate verify locations: >> * CAfile: none >> CApath: /etc/ssl/certs >> * SSLv3, TLS handshake, Client hello (1): >> * SSLv3, TLS handshake, Server hello (2): >> * SSLv3, TLS handshake, CERT (11): >> * SSLv3, TLS handshake, Server key exchange (12): >> * SSLv3, TLS handshake, Server finished (14): >> * SSLv3, TLS handshake, Client key exchange (16): >> * SSLv3, TLS change cipher, Client hello (1): >> * SSLv3, TLS handshake, Finished (20): >> * SSLv3, TLS change cipher, Client hello (1): >> * SSLv3, TLS handshake, Finished (20): >> * SSL connection using ECDHE-RSA-AES128-SHA >> * Server certificate: >> * subject: C=US; postalCode=SomeZipCode; ST=NJ; L=Old Tappan; >> street=200 Old Tappan Rd; O=SomeOrgName; OU=Web Security; OU=Enterprise SSL >> Wildcard; CN=*.somedomain.com >> * start date: 2014-10-09 00:00:00 GMT >> * expire date: 2017-10-08 23:59:59 GMT >> * subjectAltName: cert.somedomain.com matched >> * issuer: C=US; ST=DE; L=Wilmington; O=Corporation Service Company; >> CN=Trusted Secure Certificate Authority 5 >> * SSL certificate verify ok. >> > POST /message_receiver HTTP/1.1 >> > User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 >> OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3 >> > Host: cert.somedomain.com >> > Accept: */* >> > >> < HTTP/1.1 200 OK >> HTTP/1.1 200 OK >> < Cache-Control: max-age=0, private, must-revalidate >> Cache-Control: max-age=0, private, must-revalidate >> < Content-Type: text/html; charset=utf-8 >> Content-Type: text/html; charset=utf-8 >> < Date: Mon, 10 Nov 2014 20:13:26 GMT >> Date: Mon, 10 Nov 2014 20:13:26 GMT >> < ETag: "7215ee9c7d9dc229d2921a40e899ec5f" >> ETag: "7215ee9c7d9dc229d2921a40e899ec5f" >> < Server: Apache >> Server: Apache >> < Set-Cookie: _lc_session=03e19badfa917af689f53b27a2297f8a; path=/; >> HttpOnly >> Set-Cookie: _lc_session=03e19badfa917af689f53b27a2297f8a; path=/; HttpOnly >> < Status: 200 OK >> Status: 200 OK >> < X-Request-Id: bcd72a89933142d8557af9c386701494 >> X-Request-Id: bcd72a89933142d8557af9c386701494 >> < X-UA-Compatible: IE=Edge,chrome=1 >> X-UA-Compatible: IE=Edge,chrome=1 >> < Content-Length: 1 >> Content-Length: 1 >> < Connection: keep-alive >> Connection: keep-alive >> >> < >> * Connection #0 to host cert.somedomain.com left intact >> * Closing connection #0 >> * SSLv3, TLS alert, Client hello (1): >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From federico.carrone@REDACTED Tue Nov 11 16:14:43 2014 From: federico.carrone@REDACTED (Federico Carrone) Date: Tue, 11 Nov 2014 12:14:43 -0300 Subject: [erlang-questions] Request for comments: gut, a template printing, aka scaffolding, tool for Erlang. Like rails generate or yeoman In-Reply-To: <676D92DB-7BCA-4A87-AB3C-6CA9F1B67C8E@gmail.com> References: <676D92DB-7BCA-4A87-AB3C-6CA9F1B67C8E@gmail.com> Message-ID: I forgot about that, I only put it because it was the first thing that came to my mind (I am watching The Americans ). I need to put a better name, I will think it while I finish a few issues that need improvement. Thanks Anthony :). On Tue, Nov 11, 2014 at 7:59 AM, Anthony Ramine wrote: > What about renaming the 'Stalin' example tree in your readme file? > > Le 10 nov. 2014 ? 17:19, Federico Carrone a > ?crit : > > > Hi, > > > > I have created a small tool for setting up projects (ranch, cowboy > projects, etc) and scaffolding files (gen_servers, supervisors, cowboy > handlers, gen_fsm, etc) in erlang: > https://unbalancedparentheses.github.io/gut/ > > > > The generators are searched and downloaded from github (like yeoman) so > thtat anybody can create a generator and use it with gut. > > > > Version 1 will be released this week after I got some input and comments > from the community. > > > > Please test it and let me know what you think! Obviously this tool is > meant for erlang newcomers or people like me that like the tools available > in the nodejs, clojure, ruby and python communities. > > > > Thanks, > > Federico Carrone. > > > > roadmap: > > Present - v0.8: Basic functionality > > 12 November - v0.9: Better logs > > 15 November - v1.0: Support many variables in generators. Better > documentation for example on how to create your own generators. Odds and > ends. > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > -- http://federicocarrone.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ryankbrown@REDACTED Tue Nov 11 16:27:01 2014 From: ryankbrown@REDACTED (Ryan Brown) Date: Tue, 11 Nov 2014 08:27:01 -0700 Subject: [erlang-questions] SSL error for valid(?) certificate In-Reply-To: <66536814.943023.1415698156208.JavaMail.zimbra@tpip.net> References: <546185532a77c_73fa1855d3041733b@mxhero-gmail-gateway.mail> <642375167.943022.1415698149267.JavaMail.zimbra@tpip.net> <66536814.943023.1415698156208.JavaMail.zimbra@tpip.net> Message-ID: Thank you all for the (helpful) responses. Turns-out a colleague of mine asked essentially the same question. We'll get the ssl mod patched right-away. Best. On Tuesday, November 11, 2014, Andreas Schultz wrote: > Hi, > > ----- On 11 Nov, 2014, at 04:40, Ryan Brown ryankbrown@REDACTED > wrote: > > > Hi all. I have a client that appears to have a legitimate ssl cert but I > am > > getting the error seen here ( http://pastebin.com/2TTGAQDm ) when > attempting to > > post to it's endpoint. It is dying in public_cert.erl in a place that > does not > > seem to offer me any insight. Just curious if anybody had seen anything > like > > this before. > > The error indicates that your certificate is signed with > sha384WithRSAEncryption. > I'm not sure what the earliest version that supports that algorithm is, > but any version > that supports TLS 1.2 should have it. That is definitely R16B03, R15B03 > might also > work. > Also, make sure the underlying OpenSSL version has sha384 support enabled. > > Andreas > > > > > Thanks. > > > > -rb > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > -- > -- > Dipl. Inform. > Andreas Schultz > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.parker@REDACTED Tue Nov 11 16:22:21 2014 From: dave.parker@REDACTED (David Parker) Date: Tue, 11 Nov 2014 10:22:21 -0500 Subject: [erlang-questions] Issues with running smoke test after compiling OTP 17.3 from source Message-ID: Hi everybody, I?m trying to compile otp_src_17.3 and have run into the following errors when running the smoke test: $ERL_TOP/bin/erl -s ts install -s ts smoke_test batch -s init stop Relevant lines: 1> Testing tests.emulator_test.time_SUITE: Starting test, 11 test cases 1> - - - - - - - - - - - - - - - - - - - - - - - - - - time_SUITE:test_univ_to_local failed on line 123 Reason: {badmatch,{{1996,1,30},{7,45,7}}} - - - - - - - - - - - - - - - - - - - - - - - - - - 1> Testing tests.emulator_test.time_SUITE: *** FAILED test case 1 of 11 *** 1> - - - - - - - - - - - - - - - - - - - - - - - - - - time_SUITE:test_local_to_univ failed on line 135 Reason: {badmatch,{{1996,1,30},{18,45,7}}} - - - - - - - - - - - - - - - - - - - - - - - - - - 1> Testing tests.emulator_test.time_SUITE: *** FAILED test case 2 of 11 *** 1> - - - - - - - - - - - - - - - - - - - - - - - - - - time_SUITE:consistency failed on line 209 Reason: {badmatch,error} - - - - - - - - - - - - - - - - - - - - - - - - - - I?m following the documentation from http://www.erlang.org/doc/installation_guide/INSTALL.html and i get these errors after I compile and run the smoke test. I haven?t any idea what they mean or how to fix them. Can anybody help? I?m compiling on a CENTOS 5 machine running in vagrant. Linux vagrant-centos-5 2.6.18-398.el5 #1 SMP Tue Sep 16 20:50:52 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux I have also tried compiling on a bare metal CENTOS 5 machine with a similar config, with the same results. Any help would be appreciated, let me know if you need more info, I?ll provide whatever I can. ? Dave -------------- next part -------------- An HTML attachment was scrubbed... URL: From magnus@REDACTED Tue Nov 11 18:36:28 2014 From: magnus@REDACTED (Magnus Henoch) Date: Tue, 11 Nov 2014 17:36:28 +0000 Subject: [erlang-questions] Issues with running smoke test after compiling OTP 17.3 from source In-Reply-To: (David Parker's message of "Tue, 11 Nov 2014 10:22:21 -0500") References: Message-ID: Looking at erts/emulator/test/time_SUITE.erl, it seems like this test will only work correctly in Central European or South African time zones. It checks the output of "date +%Z", and if it starts with "SAST", the test assumes that the time zone is UTC+2 all year; otherwise it assumes UTC+1 in the winter and UTC+2 in the summer (northern hemisphere). You can change the time zone that the test assumes by modifying the -define(timezone, 1) and -define(dst_timezone, 2) lines in the time_SUITE.erl file. Regards, Magnus From g@REDACTED Tue Nov 11 19:28:32 2014 From: g@REDACTED (Garrett Smith) Date: Tue, 11 Nov 2014 12:28:32 -0600 Subject: [erlang-questions] beginner architectural question In-Reply-To: References: Message-ID: Hi Semmit, On Tue, Nov 11, 2014 at 3:24 AM, semmit mondo wrote: > Hi, > > I have an application with a structural pattern that comes up again and > again, and I would like to ask you how would you solve this. My app has > two kinds of processes: a singleton that is a server collecting data, and > workers that do some calculation and send the result to the server. > There can be any number of workers and there's usually one data > collector server. I think this is a very simple and common architecture. > The problem with this is that the calculators have to know the pid of the > collector and the whole thing has to be supervised. They don't have to know the pid. That's what the registry is for. > The simplest solution is that under a main supervisor I spawn the collector > server and register it with a name. Next to the collector I start another > supervisor that manages the pool of the calculators and those processes > simply use the collectors name as the message destination when they're > done with the jod. The main supervisor uses one_for_one strategy, while > the subsupervisor is a simple_one_for_one. Exactly right. But you actually never spawn/start anything - you specify the child config for the supervisor and it starts the processes. The simple_one_for_one supervisor to start collectors. You should never call spawn/start_link for any of these yourself. > I feel that this is not the best I could do. Another solution could be to > use rest_for_one in the top supervisor, not register a name for the > collector, but push down the pid of the top supervisor to the sub- > supervisor, and then in the init function of the sub-supervisor I can > ask the top supervisor (with a which_children() call) to give me the > pid of the collector. So I can give it to the calculators as an argument. > Seems a bit difficult, but has the benefit not using a registered name > for a purely internal use case. I don't understand the problem you're trying to solve here. The process registry is the canonical way to avoid the pid lookup problem. You wrap this so called "singleton" in a facade module (API) and just make calls without any thought to where they'll be sent. Internally of course, you use the process name (typically the name of the module that implements the process behavior) to your calls to gen_server:cast/call. > There's also a wrong solution with a top supervisor with no static > children. In that case I can start the children manually. First I can > start the collector, then I have got the pid of it and can use it when > starting the calculators. The problem here is that when the collector > gets restarted, the calculators won't have the right pid. Even if the > top supervisor uses rest_for_one strategy, and the calculators get > restarted, there's no mechanism to start calculators with the new > pid of the collector. (I can't give the supervisor a function that > describes the way how to restart the whole thing, can I?) You are making an excellent case for the process registry. Try it! > This wrong approach can be fixed if the top level supervisor exits > when the collector or the sub-supervisor fails (that's a {one_for_one, > 0, 1} supervisor setup) and a plus layer of supervision above this > supervisor gets into play. But that's just too many layers and > supervisors for a simple thing like this, am I right? My eyes have glazed over - I think you're solving problems of your own imaginings. Use the process registry. > I believe that the whole thing boils down to that I have not much > control over the order of child starts and I can't move information > between children of the same supervisor. It would be nice if a > supervisor could supply some sort of a local name registry for the > processes below them but as far as I know there's only a global > (node local) one that could be used here. Don't try to wire these things up - you'll drive yourself crazy. Create an independent gen_server (named/registered) - you can think of this as a singleton. I prefer to think of it as a "service". This thing hides behind a module interface that doesn't reveal anything about its implementation (i.e. whether it's a process, how the process is looked up, etc.) Run this collector service under the root supervisor. Let it go. Trust it. It'll be fine. Create a simple_one_for_one supervisor, also managed by the root. This guy is a "factory" for the calculators/workers. Provide a "start_calculator" function that calls the supervisor's start_child function. Each calculator does it's thing and uses your collector module (facade) to report back data. > Which architecture would you choose? Are there other alternative > structures I haven't thought of? What do you thing which one is the > best approach and why? Keep it simple. Run your processes under supervision. The singleton/server pattern is straight forward - use a named/registered process and provide its functionality via a module that doesn't require any server pid. Super simple. The worker pattern is straight forward - use a simple one for one supervisor to start the children. The simplest model is to "fire and forget" with these things. Try this and see what happens. Fix problems that you encounter - not problems that you think you might encounter or feel or believe. Garrett From g@REDACTED Tue Nov 11 19:31:04 2014 From: g@REDACTED (Garrett Smith) Date: Tue, 11 Nov 2014 12:31:04 -0600 Subject: [erlang-questions] beginner architectural question In-Reply-To: References: Message-ID: On Tue, Nov 11, 2014 at 12:28 PM, Garrett Smith wrote: > > The simple_one_for_one supervisor to start collectors. Sorry - I meant the calculators/workers. From g@REDACTED Tue Nov 11 19:41:29 2014 From: g@REDACTED (Garrett Smith) Date: Tue, 11 Nov 2014 12:41:29 -0600 Subject: [erlang-questions] 1000+ users; 30+ data tables/user In-Reply-To: <0F0A8AE9-10F5-4B8D-9945-EB7DD7CE86E8@writersglen.com> References: <0F0A8AE9-10F5-4B8D-9945-EB7DD7CE86E8@writersglen.com> Message-ID: Hi Lloyd, Sorry for the late reply here - I was interested in this thread last week when you sent it but was in sleep deprived conference mode and never got back to it. On Tue, Nov 4, 2014 at 8:49 PM, Lloyd R. Prentice wrote: > Hello, > > This is a naive question reflecting my inexperience with databases. > > I'm planning to offer my users a set of management/planning tools. Each user would be storing/retrieving user-specific data involving as many as 30 data tables. > > --- Data fits well into Erlang records. > --- We're not talking huge volumes of data per user. > --- Nor do I expect much data analysis. > --- Data integrity and availability are essential. > --- Users may, however, wish to bundle up their data a some point and migrate to a different system. > > I'm attracted to mnesia because of it's it's tight integration with Erlang and it's replication features. I'm also considering riak. > > My first thought was that every user would own his/her own database. But this seems to rule out > mnesia since: > > "Mnesia is a truly distributed DBMS and the schema is a system table that is replicated on all nodes in a Mnesia system. The function will fail if a schema is already present on any of the nodes in NodeList." > http://www.erlang.org/doc/apps/mnesia/Mnesia_chap3.html > > An option would be to store data for all users in each of the 30 tables. But is there a better solution altogether? > > I'd much appreciate suggestions and guidance from wiser heads. I think we've all had this problem - a new project and nothing to hold us back but our own imagination. So the question... which database to pick. Which indeed? There are like 100 amazing options! My suggestion here is to stop this line of thinking immediately :) I would instead plan to throw your early work away. Pick something that is the fastest and easiest imaginable for you to make progress on your app. Treat it as a "this will almost certainly not be what I end up with". _For me_ this means one of these: - Hard coded values or config files - Dets - SQLite - MySQL The point is to keep it as simple as possible and just get stuff working. When I'm starting on something new, I just don't know enough about anything to make the right decision - so I deliberately make the right _wrong_ decision - that is, the decision that will let me move forward quickly and get to the real problems. I might throw it away later, or I might keep it. But in any case, I'm sure as hell not going to spend a lot of time on it. Not until I'm facing real, hard, visible problems that I can use to inform my next steps. Garrett From erlang@REDACTED Tue Nov 11 21:02:14 2014 From: erlang@REDACTED (Joe Armstrong) Date: Tue, 11 Nov 2014 21:02:14 +0100 Subject: [erlang-questions] 1000+ users; 30+ data tables/user In-Reply-To: References: <0F0A8AE9-10F5-4B8D-9945-EB7DD7CE86E8@writersglen.com> Message-ID: On Tue, Nov 11, 2014 at 7:41 PM, Garrett Smith wrote: > Hi Lloyd, > > Sorry for the late reply here - I was interested in this thread last > week when you sent it but was in sleep deprived conference mode and > never got back to it. > > On Tue, Nov 4, 2014 at 8:49 PM, Lloyd R. Prentice wrote: >> Hello, >> >> This is a naive question reflecting my inexperience with databases. >> >> I'm planning to offer my users a set of management/planning tools. Each user would be storing/retrieving user-specific data involving as many as 30 data tables. >> >> --- Data fits well into Erlang records. >> --- We're not talking huge volumes of data per user. >> --- Nor do I expect much data analysis. >> --- Data integrity and availability are essential. >> --- Users may, however, wish to bundle up their data a some point and migrate to a different system. >> >> I'm attracted to mnesia because of it's it's tight integration with Erlang and it's replication features. I'm also considering riak. >> >> My first thought was that every user would own his/her own database. But this seems to rule out >> mnesia since: >> >> "Mnesia is a truly distributed DBMS and the schema is a system table that is replicated on all nodes in a Mnesia system. The function will fail if a schema is already present on any of the nodes in NodeList." >> http://www.erlang.org/doc/apps/mnesia/Mnesia_chap3.html >> >> An option would be to store data for all users in each of the 30 tables. But is there a better solution altogether? >> >> I'd much appreciate suggestions and guidance from wiser heads. > > I think we've all had this problem - a new project and nothing to hold > us back but our own imagination. So the question... which database to > pick. Which indeed? There are like 100 amazing options! > > My suggestion here is to stop this line of thinking immediately :) > > I would instead plan to throw your early work away. Pick something > that is the fastest and easiest imaginable for you to make progress on > your app. Treat it as a "this will almost certainly not be what I end > up with". > > _For me_ this means one of these: > > - Hard coded values or config files > - Dets > - SQLite > - MySQL > > The point is to keep it as simple as possible and just get stuff working. Excellent advice. For many systems - I use one file per user. The file contains term_to_binary(X) where X is whatever I feel like to represent the use data. (or you can use text files - then you can run amazing things like grep and find on them :-) The OS caches file access and I can easily analyse/dump the files. I've *never* got to the point where I need to change the file system for a database (but then again I've not built a really big system - and this works fine for several thousand files/users) If and when the design problems are solved you can change representations *if it is necessary* - choosing a database right at the start is "premature optimisation" - if you ever get to this point then the choice of representation should be dictated by measurement and not guesswork. /Joe > When I'm starting on something new, I just don't know enough about > anything to make the right decision - so I deliberately make the right > _wrong_ decision - that is, the decision that will let me move forward > quickly and get to the real problems. I might throw it away later, or > I might keep it. But in any case, I'm sure as hell not going to spend > a lot of time on it. Not until I'm facing real, hard, visible problems > that I can use to inform my next steps. > > Garrett > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From lloyd@REDACTED Tue Nov 11 22:28:19 2014 From: lloyd@REDACTED (lloyd@REDACTED) Date: Tue, 11 Nov 2014 16:28:19 -0500 (EST) Subject: [erlang-questions] 1000+ users; 30+ data tables/user In-Reply-To: References: <0F0A8AE9-10F5-4B8D-9945-EB7DD7CE86E8@writersglen.com> Message-ID: <1415741299.554228471@apps.rackspace.com> Thank you, Joe, You've broadened my horizons and I love the simplification implied. For sake of clarification and deeper understanding: -- Would, then, one directory and n files per user work? One respondent to my question cited possible limits imposed by innodes and atoms. But I see that my work station provides 30138368 inodes and Erlang docs tell me that Erlang provides 1048576 atoms, so I'm not convinced that these limits are a problem. -- Would one directory and n dets tables per user work? -- Can sufficient availability be achieved with conventional backup strategies? Yes, I know the answers are application specific, but are there metrics/rules of thumb/guidelines that can give me back-of-the-envelope answers short of all-out test-and-measurement? Many, many thanks for your insights. Lloyd -----Original Message----- From: "Joe Armstrong" Sent: Tuesday, November 11, 2014 3:02pm To: "Garrett Smith" Cc: "Lloyd R. Prentice" , "erlang-questions" Subject: Re: [erlang-questions] 1000+ users; 30+ data tables/user On Tue, Nov 11, 2014 at 7:41 PM, Garrett Smith wrote: > Hi Lloyd, > > Sorry for the late reply here - I was interested in this thread last > week when you sent it but was in sleep deprived conference mode and > never got back to it. > > On Tue, Nov 4, 2014 at 8:49 PM, Lloyd R. Prentice wrote: >> Hello, >> >> This is a naive question reflecting my inexperience with databases. >> >> I'm planning to offer my users a set of management/planning tools. Each user would be storing/retrieving user-specific data involving as many as 30 data tables. >> >> --- Data fits well into Erlang records. >> --- We're not talking huge volumes of data per user. >> --- Nor do I expect much data analysis. >> --- Data integrity and availability are essential. >> --- Users may, however, wish to bundle up their data a some point and migrate to a different system. >> >> I'm attracted to mnesia because of it's it's tight integration with Erlang and it's replication features. I'm also considering riak. >> >> My first thought was that every user would own his/her own database. But this seems to rule out >> mnesia since: >> >> "Mnesia is a truly distributed DBMS and the schema is a system table that is replicated on all nodes in a Mnesia system. The function will fail if a schema is already present on any of the nodes in NodeList." >> http://www.erlang.org/doc/apps/mnesia/Mnesia_chap3.html >> >> An option would be to store data for all users in each of the 30 tables. But is there a better solution altogether? >> >> I'd much appreciate suggestions and guidance from wiser heads. > > I think we've all had this problem - a new project and nothing to hold > us back but our own imagination. So the question... which database to > pick. Which indeed? There are like 100 amazing options! > > My suggestion here is to stop this line of thinking immediately :) > > I would instead plan to throw your early work away. Pick something > that is the fastest and easiest imaginable for you to make progress on > your app. Treat it as a "this will almost certainly not be what I end > up with". > > _For me_ this means one of these: > > - Hard coded values or config files > - Dets > - SQLite > - MySQL > > The point is to keep it as simple as possible and just get stuff working. Excellent advice. For many systems - I use one file per user. The file contains term_to_binary(X) where X is whatever I feel like to represent the use data. (or you can use text files - then you can run amazing things like grep and find on them :-) The OS caches file access and I can easily analyse/dump the files. I've *never* got to the point where I need to change the file system for a database (but then again I've not built a really big system - and this works fine for several thousand files/users) If and when the design problems are solved you can change representations *if it is necessary* - choosing a database right at the start is "premature optimisation" - if you ever get to this point then the choice of representation should be dictated by measurement and not guesswork. /Joe > When I'm starting on something new, I just don't know enough about > anything to make the right decision - so I deliberately make the right > _wrong_ decision - that is, the decision that will let me move forward > quickly and get to the real problems. I might throw it away later, or > I might keep it. But in any case, I'm sure as hell not going to spend > a lot of time on it. Not until I'm facing real, hard, visible problems > that I can use to inform my next steps. > > Garrett > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric.meadows.jonsson@REDACTED Wed Nov 12 00:53:22 2014 From: eric.meadows.jonsson@REDACTED (=?UTF-8?Q?Eric_Meadows=2DJ=C3=B6nsson?=) Date: Wed, 12 Nov 2014 00:53:22 +0100 Subject: [erlang-questions] Unknown message in httpc_manager:handle_info Message-ID: We are sometimes getting this error report with httpc: =ERROR REPORT==== 6-Nov-2014::14:27:14 === Unknown message in httpc_manager:handle_info {#Ref<0.0.0.709>,ok} I haven't been able to figure out why it happens or who sends the erroneous message. Has anyone seen this before or know what may be causing it? -- Eric Meadows-J?nsson -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmkolesnikov@REDACTED Wed Nov 12 10:16:21 2014 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Wed, 12 Nov 2014 11:16:21 +0200 Subject: [erlang-questions] gen_tcp:send timeout Message-ID: <0430803B-EEEE-40AF-9F8A-3C1FFD2A3B2F@gmail.com> Hello, I am looking on timeout issue with gen_tcp:send. I?ve not discovered yet the reason but my sender process is stacked at {current_function,{prim_inet,send,3}} The timeout will help me but I?am confused on the way to achieve it * gen_tcp provides an option {send_timeout, xxx}. The documentation says "The fact that the send call does not accept a timeout option, is because timeouts on send is handled through the socket option send_timeout." * there was a recommendation for Heroku to use erlang port directly to handle timeout send(S, Data, Timeout) -> true = erlang:port_command(S, Data, []), receive {inet_reply, S, Status} -> Status after Timeout -> {error, timeout} end. What is your experience to on this issue? What is the best way to handle it in production? Best Regards, Dmitry From erlang@REDACTED Wed Nov 12 14:15:05 2014 From: erlang@REDACTED (Joe Armstrong) Date: Wed, 12 Nov 2014 14:15:05 +0100 Subject: [erlang-questions] 1000+ users; 30+ data tables/user In-Reply-To: <1415741299.554228471@apps.rackspace.com> References: <0F0A8AE9-10F5-4B8D-9945-EB7DD7CE86E8@writersglen.com> <1415741299.554228471@apps.rackspace.com> Message-ID: On Tue, Nov 11, 2014 at 10:28 PM, wrote: > Thank you, Joe, > > > > You've broadened my horizons and I love the simplification implied. > For sake of clarification and deeper understanding: >> > -- Would, then, one directory and n files per user work? > Well I'd have to say "try it and see" - what I do know is that sooner or later it will break - all things break eventually if you stress them enough - but I don't know where. So make it - break it and fix it. I used 1 directory per user in my latest prototype. > > > One respondent to my question cited possible limits imposed by innodes > and atoms. But I > > see that my work station provides 30138368 inodes and Erlang docs > tell me that Erlang > provides 1048576 atoms, so I'm not convinced that these limits are a > problem. You can always start a second erlang node - buy a few exta GB of memory or an SSD If you get enough users that memory of any kind is a problem (be it RAM, local storage or off-site storage) then you will have made a successful product :-) The choice of OS, the amount of RAM and whether the disk is a SSD or spinning lump of iron might made more differences than architectural differences. (I've just swapped a magnetic disk in my five year old macbook pro for an SSD and now it's going crazy fast - a lot of algorithms I wrote years ago and ditched because they were too slow are now feasible) Personally I'd start with an API - sufficiently complex to meet your needs. Something like: storage:open(Who) -> Handle | EXIT storage:read(Handle, Key) -> {ok, Val} | error storage:write(Handle, Key, Val) -> ok storage:close(handle) -> ok Then implement this in the simplest possible way - it could be the file system, an ets table, a dets table or a old man in a cave who carves this on a big stone - the point is you're not supposed to know. Then write your code - scale it and see where it breaks - try to break it and see what happens. Once it's broken you can fix it. If it never breaks then heave a sigh of relief and ship it. > > -- Would one directory and n dets tables per user work? Yes - but again it will work for K dets tables whose average size is M bytes - but we don't know K and M - and no amount of guessing will work - it depends upon your machine. So now let me break my golden rule and guess .... Let's suppose you have 10K users (and if you get 10K users for anything - congratulations - every new thing on the planet wants 10K users, so you're doing great). Lets suppose each user has 10KBytes of data (which seems like a lot for many apps - or could be really small if you're exchanging images) (aside if they are exchanging images than assume 10K of meta data is all that has to be kept in RAM and all the images will be stored in the file system) So we now have 10K users with 10K of meta data = 100M of data - but a modern laptop comes with 4G of memory - so my 100M of data is 2.5% of RAM - so now *everything* will fit into RAM so you could have all user data in a single ets table and do ets:tab2file and friends to store this. Even a little rasperry PI mdel B with a 0.5G should be OK - but you might break it. If your users store big things (like images) then you will run out of storage far faster than you run out of processing power (say 100 images of 5MB = 0.5 GB/user x 10K users = 5TB) So here the meta data of 100M can be kept in memory but the data (images) would be better stored off-site (in say S3 storage) With 1GByte of meta data you could have 40K users - easy - but it would be unwise to have these on one machine - so really early on (say at 10 users) you should be replicating data across machines to recover from a crash ... As I said, I dislike guessing. > > > -- Can sufficient availability be achieved with conventional backup > strategies? "conventional" means nothing to me - you have to think through the crash scenarios and ask "what do I want to happen" - is it acceptable to loose data, is it acceptable to loose latency, then build your application accordingly. I don't really understand the difference between fault-tolerance and backup. Backup seems to mean "I don't trust my fault tolerance" - storage should be reliable meaning that if I store stuff I can retrieve it later - so backup means "I don't trust my storage layer". Backup could be used in the sense of "a snapshot" - so if I deliberately delete a file then I can go back to an earlier snapshot and recover it. This should not mean that the storage abstraction is broken. Rather than a snapshot it would be better if the storage abstraction had no "delete" operation so deleting something just removes it from the current namespace so I can't see it any more. Personally, it seems far easier to assume infinite write append stores, with a trail somewhere else for crash resilience (which is how most back-end stores work anyway) > > Yes, I know the answers are application specific, but are there > metrics/rules of thumb/guidelines that can give me back-of-the-envelope > answers short of all-out test-and-measurement? My rule of thumb is that human textual data (by that I mean, text that you type into a computer) is easy to process. So anything that it typed into the machine through a keyboard can be stored in RAM - Back of envelope - suppose I program like crazy and write 10K of code per day (that's a lot) and do this for 200 days a year - so I'm up in 2MBytes/year - so 1GB will last for 500 years .... Data produced by devices - photos videos etc. is what takes up space - we need 100s of GB to TB's of storage depending upon the resolution of the devices - but this data will be stored somewhere in the cloud - since they have far lower prices than we can achieve at home. I'd reckon on keeping all meta data in RAM locally - all big data (images, whatever) in the cloud with locally cached copies. I'd replicate the meta data across K machines (for some K) You don't need to do all-out measurements, think in terms of "measurements that are appropriate to this phase in the design" - start with a guess, measure to see if you are correct and refine your estimates as you go along. Estimating how long something takes is the the area were I am most likely to be wrong. Today I am very reluctant to guess how long something I have never done will take. I have been so wrong so many times in the past that it is laughable. A few weeks ago I had a problem that I thought would take 5 minutes to solve - it took three weeks ... if stuff worked as it said on the label life would be simple - but often strange and unexpected things happen when you do something that you have never done before. Life would however be rather boring if we could predict the future :-) Cheers /Joe > > > > Many, many thanks for your insights. > > > > Lloyd > > > > > > > > > > > > -----Original Message----- > From: "Joe Armstrong" > Sent: Tuesday, November 11, 2014 3:02pm > To: "Garrett Smith" > Cc: "Lloyd R. Prentice" , "erlang-questions" > > Subject: Re: [erlang-questions] 1000+ users; 30+ data tables/user > > On Tue, Nov 11, 2014 at 7:41 PM, Garrett Smith wrote: >> Hi Lloyd, >> >> Sorry for the late reply here - I was interested in this thread last >> week when you sent it but was in sleep deprived conference mode and >> never got back to it. >> >> On Tue, Nov 4, 2014 at 8:49 PM, Lloyd R. Prentice >> wrote: >>> Hello, >>> >>> This is a naive question reflecting my inexperience with databases. >>> >>> I'm planning to offer my users a set of management/planning tools. Each >>> user would be storing/retrieving user-specific data involving as many as 30 >>> data tables. >>> >>> --- Data fits well into Erlang records. >>> --- We're not talking huge volumes of data per user. >>> --- Nor do I expect much data analysis. >>> --- Data integrity and availability are essential. >>> --- Users may, however, wish to bundle up their data a some point and >>> migrate to a different system. >>> >>> I'm attracted to mnesia because of it's it's tight integration with >>> Erlang and it's replication features. I'm also considering riak. >>> >>> My first thought was that every user would own his/her own database. But >>> this seems to rule out >>> mnesia since: >>> >>> "Mnesia is a truly distributed DBMS and the schema is a system table that >>> is replicated on all nodes in a Mnesia system. The function will fail if a >>> schema is already present on any of the nodes in NodeList." >>> http://www.erlang.org/doc/apps/mnesia/Mnesia_chap3.html >>> >>> An option would be to store data for all users in each of the 30 tables. >>> But is there a better solution altogether? >>> >>> I'd much appreciate suggestions and guidance from wiser heads. >> >> I think we've all had this problem - a new project and nothing to hold >> us back but our own imagination. So the question... which database to >> pick. Which indeed? There are like 100 amazing options! >> >> My suggestion here is to stop this line of thinking immediately :) >> >> I would instead plan to throw your early work away. Pick something >> that is the fastest and easiest imaginable for you to make progress on >> your app. Treat it as a "this will almost certainly not be what I end >> up with". >> >> _For me_ this means one of these: >> >> - Hard coded values or config files >> - Dets >> - SQLite >> - MySQL >> >> The point is to keep it as simple as possible and just get stuff working. > > Excellent advice. > > For many systems - I use one file per user. The file contains > term_to_binary(X) > where X is whatever I feel like to represent the use data. > > (or you can use text files - then you can run amazing things like grep and > find > on them :-) > > The OS caches file access and I can easily analyse/dump the files. > > I've *never* got to the point where I need to change the file system for > a database (but then again I've not built a really big system - and > this works fine > for several thousand files/users) > > If and when the design problems are solved you can change representations > *if it is necessary* - choosing a database right at the start is > "premature optimisation" - if you ever get to this point then the > choice of representation > should be dictated by measurement and not guesswork. > > /Joe > >> When I'm starting on something new, I just don't know enough about >> anything to make the right decision - so I deliberately make the right >> _wrong_ decision - that is, the decision that will let me move forward >> quickly and get to the real problems. I might throw it away later, or >> I might keep it. But in any case, I'm sure as hell not going to spend >> a lot of time on it. Not until I'm facing real, hard, visible problems >> that I can use to inform my next steps. >> >> Garrett >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions From 114999420@REDACTED Wed Nov 12 11:18:22 2014 From: 114999420@REDACTED (=?utf-8?B?YWRhbSBjaGFu?=) Date: Wed, 12 Nov 2014 18:18:22 +0800 Subject: [erlang-questions] A problem with exit erlang node. Message-ID: Hi List, I have a problem to stop or exit a erlang node.? When I called erlang:halt(), the node is fake dead, and the cpu goes up to 100%. Here is the situation: I'm running OTP_R15B02 on Centos 6.3.? I have 3 nodes named 'server', 'unite' and 'gateway' which connected to each other. The 'gateway' node listens to a port , receives socket datas from client, and then transfers to 'server' and 'unite'. The response data from 'server' and 'unite' will send back to client through 'gateway' node too. When I want to stop all these 3 nodes, the 'gateway' node CAN NOT exit completely sometimes (small probability) ?. The nodes is running in screen of linux, the starting scripts like this: [start_all.sh] ... /usr/bin/screen -dmS server -s $ScriptPath/start_server.sh $Log?? ... /usr/bin/screen -dmS unite -s $ScriptPath/start_unite.sh $Log? ... /usr/bin/screen -dmS gateway -s $ScriptPath/start_gateway.sh $Log? [start_gateway.sh] #!/bin/bash cd /data/web/server/server/config ulimit -s 262140 erl -kernel inet_dist_listen_min 40001 -kernel inet_dist_listen_max 40100 +P 1024000 +K true -smp disable -name gateway@REDACTED -setcookie abc -boot start_sasl -config gs_main -pa ../ebin -s gs_main start -extra 192.168.7.100? 9001 2 I stop the nodes in the order of 'gateway' -> 'unite' -> 'server' The stop scripts like this: [stop_all.sh] #!/bin/bash cd /data/web/server/server/scripts/ chmod +x stop_gateway.sh chmod +x stop_unite.sh chmod +x stop_server.sh ./stop_gateway.sh ./stop_unite.sh ./stop_server.sh? [stop_gateway.sh] #!/bin/bash cd /data/web/server/server/config erl -noshell -hidden -name stop_gateway@REDACTED -setcookie abc -pa ../ebin -eval "rpc:call('gateway@REDACTED', gs_main, stop, [])." -s c q? [gs_main.erl] -define(SERVER_APPS, [sasl, gs_main]).? ... stop() -> ok = stop_applications(?SERVER_APPS), erlang:halt().? The 'server' and 'unite' node can exit completely every time, and the screen which is running the node also exit too. But the 'gateway' node sometimes (small probability?) can't exit, the screen remains too: [root@REDACTED logs]# screen -ls There are screens on: 20107.gateway (Detached) [root@REDACTED logs]# ps -ef | grep gateway root 20107 1 0 Nov10 ? 00:00:00 /usr/bin/SCREEN -dmS gateway -s /data/web/server/server/scripts/start_gateway.sh -L -c /data/web/server/server/var/logs/screenrc_gateway root 20110 20107 0 Nov10 pts/7 00:00:00 /bin/bash /data/web/server/server/scripts/start_gateway.sh root 20111 20110 90 Nov10 pts/7 1-19:56:53 /usr/local/lib/erlang/erts-5.9.2/bin/beam -P 1024000 -K true -- -root /usr/local/lib/erlang -progname erl -- -home /root -- -kernel inet_dist_listen_min 40001 -kernel inet_dist_listen_max 40100 -smp disable -name gateway@REDACTED -setcookie abc -boot start_sasl -config gs_main -pa ../ebin -s gs_main start -extra 192.168.7.100 9001 2? [root@REDACTED logs]# strace -c -p 20111 Process 20111 attached - interrupt to quit ^CProcess 20111 detached? strace command has no effect here. And one CPU core keeps running at 100%. At the end of the 'gateway' node's log, it says the application is exited: [gateway.log] =INFO REPORT==== 11-Nov-2014::10:21:18 === application: gs_main exited: stopped type: temporary? It seems that some endless loop occured after the printing of the =INFO REPORT=. The application is not really exited, or the 'ps -ef | grep gateway' command won't find the 20111 process. Any ideas? Thanks in advance. ------------------ Adam Chan -------------- next part -------------- An HTML attachment was scrubbed... URL: From darach@REDACTED Wed Nov 12 19:44:36 2014 From: darach@REDACTED (Darach Ennis) Date: Wed, 12 Nov 2014 18:44:36 +0000 Subject: [erlang-questions] [ANN] jch-erl 0.1 - Fast minimal memory consistent hashing for Erlang/OTP. In-Reply-To: <546211F4.8060705@erix.ericsson.se> References: <546211F4.8060705@erix.ericsson.se> Message-ID: Hi Sverker. I did not know that as I don't use Windows platforms often these days. Fixed, and I learnt something - defo worth a pint! :) Cheers, Darach. On Tue, Nov 11, 2014 at 1:41 PM, Sverker Eriksson < sverker.eriksson@REDACTED> wrote: > Maybe not worth a pint, but... > If you want to ensure the first argument can be 64 bit long > you should use enif_get_uint64. > Type 'long' is only 64 bit on most 64bit OS (not Windows). > > /Sverker, Erlang/OTP > > > > On 11/07/2014 10:08 PM, Darach Ennis wrote: > > Hey folk. > > jch-erl 0.1 freshly baked. > > NIF wrapper for Jump Consistent Hash algorithm by John Lamping and Eric > Veach developed at Google, Inc. Paper: "A Fast, Minimal Memory, Consistent > Hash Algorithm". > http://arxiv.org/ftp/arxiv/papers/1406/1406.2294.pdf > > This implementation uses the xorshift64* pseudo-random number generator > rather than the linear congruential generator in the paper as it is > reasonably fast but, more importantly, memory efficient. > > Usage > > A single function jch:ch/2 is offered. Simply pass in a 64 bit long (or > less) integer key argument followed by the desired bucket or continuum > partition size. The function returns the partition allocated for the key. > > Performance is very stable as bucket size increases and distribution across > the ring is stable (standard deviation for a reasonable sample size is > typically <2% relative to the mean). > > Example > > % ERL_LIBS=deps erl +sfwi 1 +scl false -pa ebin > Erlang R16B02 (erts-5.10.3) [source] [64-bit] [smp:8:8] [async-threads:10] > [hipe] [kernel-poll:false] [dtrace] > > Eshell V5.10.3 (abort with ^G) > 1> jch:ch(1,128). > 29 > 2> jch:ch(13,128). > 121 > 3> jch:ch(13,128). > 121 > 4> jch:ch(trunc(math:pow(2,64))-1,128). > 78 > 5> jch:ch(trunc(math:pow(2,64)),128). %% off by 1 mate > ** exception error: bad argument > in function jch:ch/2 > called as jch:ch(18446744073709551616,128) > 6> %% off by 1 mate > 6> > > Access times are typically sub-microsecond. Performance tests of the erlang > NIF > and underlying C function are included. > > Example performance for a space of 1 billion buckets: > > - 1B Buckets. Hash performance. > ch1b N: 10 Min: -----1 Max: -----2 Median: -----1 Avg: -----1 Elapsed: 16 > ch1b N: 100 Min: -----0 Max: -----1 Median: -----1 Avg: -----1 Elapsed: 102 > ch1b N: 1000 Min: -----0 Max: -----9 Median: -----0 Avg: -----0 Elapsed: > 1178 > ch1b N: 10000 Min: -----0 Max: ----13 Median: -----0 Avg: -----0 Elapsed: > 10146 > ch1b N: 100000 Min: -----0 Max: ----59 Median: -----0 Avg: -----0 Elapsed: > 109772 > ch1b N: 1000000 Min: -----0 Max: ---886 Median: -----0 Avg: -----0 Elapsed: > 1048561 > ch1b N: 10000000 Min: -----0 Max: --3734 Median: -----0 Avg: -----0 > Elapsed: 11757297 > > And a sample of how keys partition over the space: > > - 32 Buckets. 10M hashes Uniform Distribution Check. > 312292 312457 312660 312522 313805 311987 311431 312299 > 311975 312834 312233 312125 313391 312222 312170 312389 > 311811 313134 312986 312434 312002 313060 312784 312699 > 312114 312170 312385 312721 313163 311883 313060 312802 > oOo| Min: 311431 Max: 313805 Median: 312389 Avg: 312500 Elapsed: 15748372 > Worst: 99.2435 Med: 99.5488 Avg: 99.5841 RSD: 0.9139 > > The RSD here is the standard deviation relative to the mean (avg) which is > less than 1% > after 10M hash operations on random uniform test data. That's pretty good. > > A pint (or fancy fizzy water, or cup of pant wetting tea...) to anyone who > can strip off an order of magnitude off the max outlier latencies and > explain through scheduler, gc or emulator tuning what insight I missed or > runtime vagary i've yet to master. Mainly written to toy with > micro-benchmarking and get practice with tuning the emulator. > > Enjoy! > > Cheers, > > Darach. > > > > > _______________________________________________ > erlang-questions mailing listerlang-questions@REDACTED://erlang.org/mailman/listinfo/erlang-questions > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lloyd@REDACTED Wed Nov 12 19:48:50 2014 From: lloyd@REDACTED (lloyd@REDACTED) Date: Wed, 12 Nov 2014 13:48:50 -0500 (EST) Subject: [erlang-questions] 1000+ users; 30+ data tables/user In-Reply-To: References: <0F0A8AE9-10F5-4B8D-9945-EB7DD7CE86E8@writersglen.com> <1415741299.554228471@apps.rackspace.com> Message-ID: <1415818130.687210148@apps.rackspace.com> Joe, Thank you for sharing your experience and bringing common sense to a muddle mind. I bought an S-100 system too many moons ago to count because I wanted to use it for word processing. I knew nothing about computers so everything was magic and terrified me. And, of course, back then nothing worked out of the box. I learned to program in dribs and drabs, supervised programmers, but never programmed professionally. We were working with severely limited system resources so counting bytes and cycles was essential to the task. So, I maintain a kind of computer phobia. Every time I sit down I have deep dread in the pit of my stomach that it's simply not going to work. Your experience and common sense are a tonic. I should know that limits on processing, storage, and data transport are fast approaching infinite; quite likely far beyond the capacity of my imagination to stress them. I have always regarded the computer as a tool to make muscle and mind work easier and, next to speech, the most effective medium of communication ever. And, until dedicating myself to learning Erlang, I have long regretted my programming aphasia. I'm pushing now to create an Erlang application worthy of the outstanding tools that you and other Erlang wizards have created. Your guidance, Garrett Smith's encouragement, and Jesse Gumm's tutoring have done wonders for my confidence. Thanks again to you and the amazing Erlang community, Lloyd -----Original Message----- From: "Joe Armstrong" Sent: Wednesday, November 12, 2014 8:15am To: "Lloyd Prentice" Cc: "Garrett Smith" , "erlang-questions" Subject: Re: [erlang-questions] 1000+ users; 30+ data tables/user On Tue, Nov 11, 2014 at 10:28 PM, wrote: > Thank you, Joe, > > > > You've broadened my horizons and I love the simplification implied. > For sake of clarification and deeper understanding: >> > -- Would, then, one directory and n files per user work? > Well I'd have to say "try it and see" - what I do know is that sooner or later it will break - all things break eventually if you stress them enough - but I don't know where. So make it - break it and fix it. I used 1 directory per user in my latest prototype. > > > One respondent to my question cited possible limits imposed by innodes > and atoms. But I > > see that my work station provides 30138368 inodes and Erlang docs > tell me that Erlang > provides 1048576 atoms, so I'm not convinced that these limits are a > problem. You can always start a second erlang node - buy a few exta GB of memory or an SSD If you get enough users that memory of any kind is a problem (be it RAM, local storage or off-site storage) then you will have made a successful product :-) The choice of OS, the amount of RAM and whether the disk is a SSD or spinning lump of iron might made more differences than architectural differences. (I've just swapped a magnetic disk in my five year old macbook pro for an SSD and now it's going crazy fast - a lot of algorithms I wrote years ago and ditched because they were too slow are now feasible) Personally I'd start with an API - sufficiently complex to meet your needs. Something like: storage:open(Who) -> Handle | EXIT storage:read(Handle, Key) -> {ok, Val} | error storage:write(Handle, Key, Val) -> ok storage:close(handle) -> ok Then implement this in the simplest possible way - it could be the file system, an ets table, a dets table or a old man in a cave who carves this on a big stone - the point is you're not supposed to know. Then write your code - scale it and see where it breaks - try to break it and see what happens. Once it's broken you can fix it. If it never breaks then heave a sigh of relief and ship it. > > -- Would one directory and n dets tables per user work? Yes - but again it will work for K dets tables whose average size is M bytes - but we don't know K and M - and no amount of guessing will work - it depends upon your machine. So now let me break my golden rule and guess .... Let's suppose you have 10K users (and if you get 10K users for anything - congratulations - every new thing on the planet wants 10K users, so you're doing great). Lets suppose each user has 10KBytes of data (which seems like a lot for many apps - or could be really small if you're exchanging images) (aside if they are exchanging images than assume 10K of meta data is all that has to be kept in RAM and all the images will be stored in the file system) So we now have 10K users with 10K of meta data = 100M of data - but a modern laptop comes with 4G of memory - so my 100M of data is 2.5% of RAM - so now *everything* will fit into RAM so you could have all user data in a single ets table and do ets:tab2file and friends to store this. Even a little rasperry PI mdel B with a 0.5G should be OK - but you might break it. If your users store big things (like images) then you will run out of storage far faster than you run out of processing power (say 100 images of 5MB = 0.5 GB/user x 10K users = 5TB) So here the meta data of 100M can be kept in memory but the data (images) would be better stored off-site (in say S3 storage) With 1GByte of meta data you could have 40K users - easy - but it would be unwise to have these on one machine - so really early on (say at 10 users) you should be replicating data across machines to recover from a crash ... As I said, I dislike guessing. > > > -- Can sufficient availability be achieved with conventional backup > strategies? "conventional" means nothing to me - you have to think through the crash scenarios and ask "what do I want to happen" - is it acceptable to loose data, is it acceptable to loose latency, then build your application accordingly. I don't really understand the difference between fault-tolerance and backup. Backup seems to mean "I don't trust my fault tolerance" - storage should be reliable meaning that if I store stuff I can retrieve it later - so backup means "I don't trust my storage layer". Backup could be used in the sense of "a snapshot" - so if I deliberately delete a file then I can go back to an earlier snapshot and recover it. This should not mean that the storage abstraction is broken. Rather than a snapshot it would be better if the storage abstraction had no "delete" operation so deleting something just removes it from the current namespace so I can't see it any more. Personally, it seems far easier to assume infinite write append stores, with a trail somewhere else for crash resilience (which is how most back-end stores work anyway) > > Yes, I know the answers are application specific, but are there > metrics/rules of thumb/guidelines that can give me back-of-the-envelope > answers short of all-out test-and-measurement? My rule of thumb is that human textual data (by that I mean, text that you type into a computer) is easy to process. So anything that it typed into the machine through a keyboard can be stored in RAM - Back of envelope - suppose I program like crazy and write 10K of code per day (that's a lot) and do this for 200 days a year - so I'm up in 2MBytes/year - so 1GB will last for 500 years .... Data produced by devices - photos videos etc. is what takes up space - we need 100s of GB to TB's of storage depending upon the resolution of the devices - but this data will be stored somewhere in the cloud - since they have far lower prices than we can achieve at home. I'd reckon on keeping all meta data in RAM locally - all big data (images, whatever) in the cloud with locally cached copies. I'd replicate the meta data across K machines (for some K) You don't need to do all-out measurements, think in terms of "measurements that are appropriate to this phase in the design" - start with a guess, measure to see if you are correct and refine your estimates as you go along. Estimating how long something takes is the the area were I am most likely to be wrong. Today I am very reluctant to guess how long something I have never done will take. I have been so wrong so many times in the past that it is laughable. A few weeks ago I had a problem that I thought would take 5 minutes to solve - it took three weeks ... if stuff worked as it said on the label life would be simple - but often strange and unexpected things happen when you do something that you have never done before. Life would however be rather boring if we could predict the future :-) Cheers /Joe > > > > Many, many thanks for your insights. > > > > Lloyd > > > > > > > > > > > > -----Original Message----- > From: "Joe Armstrong" > Sent: Tuesday, November 11, 2014 3:02pm > To: "Garrett Smith" > Cc: "Lloyd R. Prentice" , "erlang-questions" > > Subject: Re: [erlang-questions] 1000+ users; 30+ data tables/user > > On Tue, Nov 11, 2014 at 7:41 PM, Garrett Smith wrote: >> Hi Lloyd, >> >> Sorry for the late reply here - I was interested in this thread last >> week when you sent it but was in sleep deprived conference mode and >> never got back to it. >> >> On Tue, Nov 4, 2014 at 8:49 PM, Lloyd R. Prentice >> wrote: >>> Hello, >>> >>> This is a naive question reflecting my inexperience with databases. >>> >>> I'm planning to offer my users a set of management/planning tools. Each >>> user would be storing/retrieving user-specific data involving as many as 30 >>> data tables. >>> >>> --- Data fits well into Erlang records. >>> --- We're not talking huge volumes of data per user. >>> --- Nor do I expect much data analysis. >>> --- Data integrity and availability are essential. >>> --- Users may, however, wish to bundle up their data a some point and >>> migrate to a different system. >>> >>> I'm attracted to mnesia because of it's it's tight integration with >>> Erlang and it's replication features. I'm also considering riak. >>> >>> My first thought was that every user would own his/her own database. But >>> this seems to rule out >>> mnesia since: >>> >>> "Mnesia is a truly distributed DBMS and the schema is a system table that >>> is replicated on all nodes in a Mnesia system. The function will fail if a >>> schema is already present on any of the nodes in NodeList." >>> http://www.erlang.org/doc/apps/mnesia/Mnesia_chap3.html >>> >>> An option would be to store data for all users in each of the 30 tables. >>> But is there a better solution altogether? >>> >>> I'd much appreciate suggestions and guidance from wiser heads. >> >> I think we've all had this problem - a new project and nothing to hold >> us back but our own imagination. So the question... which database to >> pick. Which indeed? There are like 100 amazing options! >> >> My suggestion here is to stop this line of thinking immediately :) >> >> I would instead plan to throw your early work away. Pick something >> that is the fastest and easiest imaginable for you to make progress on >> your app. Treat it as a "this will almost certainly not be what I end >> up with". >> >> _For me_ this means one of these: >> >> - Hard coded values or config files >> - Dets >> - SQLite >> - MySQL >> >> The point is to keep it as simple as possible and just get stuff working. > > Excellent advice. > > For many systems - I use one file per user. The file contains > term_to_binary(X) > where X is whatever I feel like to represent the use data. > > (or you can use text files - then you can run amazing things like grep and > find > on them :-) > > The OS caches file access and I can easily analyse/dump the files. > > I've *never* got to the point where I need to change the file system for > a database (but then again I've not built a really big system - and > this works fine > for several thousand files/users) > > If and when the design problems are solved you can change representations > *if it is necessary* - choosing a database right at the start is > "premature optimisation" - if you ever get to this point then the > choice of representation > should be dictated by measurement and not guesswork. > > /Joe > >> When I'm starting on something new, I just don't know enough about >> anything to make the right decision - so I deliberately make the right >> _wrong_ decision - that is, the decision that will let me move forward >> quickly and get to the real problems. I might throw it away later, or >> I might keep it. But in any case, I'm sure as hell not going to spend >> a lot of time on it. Not until I'm facing real, hard, visible problems >> that I can use to inform my next steps. >> >> Garrett >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions From eric.pailleau@REDACTED Wed Nov 12 22:37:11 2014 From: eric.pailleau@REDACTED (Eric Pailleau) Date: Wed, 12 Nov 2014 22:37:11 +0100 Subject: [erlang-questions] A problem with exit erlang node. Message-ID: [gs_main.erl] -define(SERVER_APPS, [sasl, gs_main]). ... stop() -> ok = stop_applications(?SERVER_APPS), erlang:halt(). I would reverse this list for stopping first gs _main , then sasl. Is gs_main depending of sasl in .app file ? Don t know if it is the problem... regards ? Envoy? depuis mon mobile ? Eric adam chan <114999420@REDACTED> a ?crit?: >Hi List, > > >I have a problem to stop or exit a erlang node.? >When I called erlang:halt(), the node is fake dead, and the cpu goes up to 100%. > > >Here is the situation: >I'm running OTP_R15B02 on Centos 6.3.? > > >I have 3 nodes named 'server', 'unite' and 'gateway' which connected to each other. >The 'gateway' node listens to a port , receives socket datas from client, and then transfers to 'server' and 'unite'. >The response data from 'server' and 'unite' will send back to client through 'gateway' node too. > >When I want to stop all these 3 nodes, the 'gateway' node CAN NOT exit completely sometimes (small probability) ?. >The nodes is running in screen of linux, the starting scripts like this: > > >[start_all.sh] >... >/usr/bin/screen -dmS server -s $ScriptPath/start_server.sh $Log?? >... >/usr/bin/screen -dmS unite -s $ScriptPath/start_unite.sh $Log? >... >/usr/bin/screen -dmS gateway -s $ScriptPath/start_gateway.sh $Log? > > >[start_gateway.sh] >#!/bin/bash >cd /data/web/server/server/config >ulimit -s 262140 >erl -kernel inet_dist_listen_min 40001 -kernel inet_dist_listen_max 40100 +P 1024000 +K true -smp disable -name gateway@REDACTED -setcookie abc -boot start_sasl -config gs_main -pa ../ebin -s gs_main start -extra 192.168.7.100? 9001 2 > > > > > >I stop the nodes in the order of 'gateway' -> 'unite' -> 'server' >The stop scripts like this: >[stop_all.sh] >#!/bin/bash >cd /data/web/server/server/scripts/ >chmod +x stop_gateway.sh >chmod +x stop_unite.sh >chmod +x stop_server.sh >./stop_gateway.sh >./stop_unite.sh >./stop_server.sh? > > > >[stop_gateway.sh] >#!/bin/bash >cd /data/web/server/server/config >erl -noshell -hidden -name stop_gateway@REDACTED -setcookie abc -pa ../ebin -eval "rpc:call('gateway@REDACTED', gs_main, stop, [])." -s c q? > > > >[gs_main.erl] >-define(SERVER_APPS, [sasl, gs_main]).? >... >stop() -> > ok = stop_applications(?SERVER_APPS), > erlang:halt().? > > > > > >The 'server' and 'unite' node can exit completely every time, and the screen which is running the node also exit too. >But the 'gateway' node sometimes (small probability?) can't exit, the screen remains too: > > >[root@REDACTED logs]# screen -ls >There are screens on: > 20107.gateway (Detached) > > >[root@REDACTED logs]# ps -ef | grep gateway >root 20107 1 0 Nov10 ? 00:00:00 /usr/bin/SCREEN -dmS gateway -s /data/web/server/server/scripts/start_gateway.sh -L -c /data/web/server/server/var/logs/screenrc_gateway >root 20110 20107 0 Nov10 pts/7 00:00:00 /bin/bash /data/web/server/server/scripts/start_gateway.sh >root 20111 20110 90 Nov10 pts/7 1-19:56:53 /usr/local/lib/erlang/erts-5.9.2/bin/beam -P 1024000 -K true -- -root /usr/local/lib/erlang -progname erl -- -home /root -- -kernel inet_dist_listen_min 40001 -kernel inet_dist_listen_max 40100 -smp disable -name gateway@REDACTED -setcookie abc -boot start_sasl -config gs_main -pa ../ebin -s gs_main start -extra 192.168.7.100 9001 2? > > > >[root@REDACTED logs]# strace -c -p 20111 >Process 20111 attached - interrupt to quit >^CProcess 20111 detached? > > > >strace command has no effect here. And one CPU core keeps running at 100%. >At the end of the 'gateway' node's log, it says the application is exited: >[gateway.log] >=INFO REPORT==== 11-Nov-2014::10:21:18 === > application: gs_main > exited: stopped > type: temporary? > > > >It seems that some endless loop occured after the printing of the =INFO REPORT=. > >The application is not really exited, or the 'ps -ef | grep gateway' command won't find the 20111 process. > > >Any ideas? >Thanks in advance. > > >------------------ >Adam Chan >_______________________________________________ >erlang-questions mailing list >erlang-questions@REDACTED >http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From maxim@REDACTED Wed Nov 12 23:16:14 2014 From: maxim@REDACTED (Maxim Sokhatsky) Date: Thu, 13 Nov 2014 00:16:14 +0200 Subject: [erlang-questions] beam.smp crash reproducibly Message-ID: Hello! tl;dr We?ve managed to put beam.smp in crash reproducibly. ? ? ? It is resistant to reproduce on 17.3 and 16B02 versions. ? ? ? This happens under heavy load. No HiPE. No NIFs. ulimit is ok. We have very simple application that consumes RabbitMQ queue and store data in mnesia's disc_copies. For that purposes we use RabbitMQ client stack amqp_client/rabbit_common wrapped in our simple library synrc/mqs (300 LOC) along with very simple wrapper over mnesia synrc/kvs (200 LOC). We have 16GB RAM on powerful machine and performance is good. However after reaching memory consumption near 10GB the system goes to core. We used the original Ubuntu 12.04 package R16B02 which was without symbol information needed to bug report. So we?ve built with KERL Erlang 17.3 from sources and situation hadn?t change. Here is GBD session we did retrieve from core file along with detailed information about application, build procedure, etc.: ? ? ? ? ?1. https://gist.github.com/5HT/e35d58b76bc25680e17b ? ? ? ? ?2.?https://gist.github.com/5HT/224c569df807f1e337aa We heard that 17.3 have some unstable memory allocators. But crashes was also reproducible on R16B02. So we decided not to panic and ask in community the recipe how to perform further checks and plan in calm the regression test kit.? As you can see crash core files contains information about allocators and gc. We think the problem is there. That leads us with following questions to community: ? ? ? ? ?1. Which memory allocators you suggest us to try at first? ? ? ? ? ?2. What other steps we should perform? ? ? ? ? ?3. How do you think what is the real cause of the problem? Sources are really simple, I can send them by request to Ericsson OTP team. The application itself weight only?1332 LOC. ? Synrc Research Center ? Maxim From imantc@REDACTED Thu Nov 13 00:35:37 2014 From: imantc@REDACTED (Imants Cekusins) Date: Thu, 13 Nov 2014 00:35:37 +0100 Subject: [erlang-questions] A problem with exit erlang node. In-Reply-To: References: Message-ID: Could the 'gateway' node close its open sockets before shutting down? Maybe pause for a second after pinging an exit signal to the gateway node, stopping the apps but before calling erlang:halt()? On 12 Nov 2014 17:14, "adam chan" <114999420@REDACTED> wrote: Hi List, I have a problem to stop or exit a erlang node. When I called erlang:halt(), the node is fake dead, and the cpu goes up to 100%. Here is the situation: I'm running OTP_R15B02 on Centos 6.3. I have 3 nodes named 'server', 'unite' and 'gateway' which connected to each other. The 'gateway' node listens to a port , receives socket datas from client, and then transfers to 'server' and 'unite'. The response data from 'server' and 'unite' will send back to client through 'gateway' node too. When I want to stop all these 3 nodes, the 'gateway' node CAN NOT exit completely sometimes (small probability) . The nodes is running in screen of linux, the starting scripts like this: *[start_all.sh]* ... /usr/bin/screen -dmS server -s $ScriptPath/start_server.sh $Log ... /usr/bin/screen -dmS unite -s $ScriptPath/start_unite.sh $Log ... /usr/bin/screen -dmS gateway -s $ScriptPath/start_gateway.sh $Log *[start_gateway.sh]* #!/bin/bash cd /data/web/server/server/config ulimit -s 262140 erl -kernel inet_dist_listen_min 40001 -kernel inet_dist_listen_max 40100 +P 1024000 +K true -smp disable -name gateway@REDACTED -setcookie abc -boot start_sasl -config gs_main -pa ../ebin -s gs_main start -extra 192.168.7.100 9001 2 I stop the nodes in the order of 'gateway' -> 'unite' -> 'server' The stop scripts like this: *[stop_all.sh]* #!/bin/bash cd /data/web/server/server/scripts/ chmod +x stop_gateway.sh chmod +x stop_unite.sh chmod +x stop_server.sh ./stop_gateway.sh ./stop_unite.sh ./stop_server.sh *[stop_gateway.sh]* #!/bin/bash cd /data/web/server/server/config erl -noshell -hidden -name stop_gateway@REDACTED -setcookie abc -pa ../ebin -eval "rpc:call('gateway@REDACTED', gs_main, stop, [])." -s c q *[gs_main.erl]* -define(SERVER_APPS, [sasl, gs_main]). ... stop() -> ok = stop_applications(?SERVER_APPS), erlang:halt(). The 'server' and 'unite' node can exit completely every time, and the screen which is running the node also exit too. But the 'gateway' node sometimes (small probability) can't exit, the screen remains too: *[root@REDACTED logs]# screen -ls* There are screens on: 20107.gateway (Detached) *[root@REDACTED logs]# ps -ef | grep gateway* root 20107 1 0 Nov10 ? 00:00:00 /usr/bin/SCREEN -dmS gateway -s /data/web/server/server/scripts/start_gateway.sh -L -c /data/web/server/server/var/logs/screenrc_gateway root 20110 20107 0 Nov10 pts/7 00:00:00 /bin/bash /data/web/server/server/scripts/start_gateway.sh root 20111 20110 90 Nov10 pts/7 1-19:56:53 /usr/local/lib/erlang/erts-5.9.2/bin/beam -P 1024000 -K true -- -root /usr/local/lib/erlang -progname erl -- -home /root -- -kernel inet_dist_listen_min 40001 -kernel inet_dist_listen_max 40100 -smp disable -name gateway@REDACTED -setcookie abc -boot start_sasl -config gs_main -pa ../ebin -s gs_main start -extra 192.168.7.100 9001 2 *[root@REDACTED logs]# strace -c -p 20111* Process 20111 attached - interrupt to quit ^CProcess 20111 detached strace command has no effect here. And one CPU core keeps running at 100%. At the end of the 'gateway' node's log, it says the application is exited: *[gateway.log]* =INFO REPORT==== 11-Nov-2014::10:21:18 === application: gs_main exited: stopped type: temporary It seems that some endless loop occured after the printing of the =INFO REPORT=. The application is not really exited, or the 'ps -ef | grep gateway' command won't find the 20111 process. Any ideas? Thanks in advance. ------------------ Adam Chan _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From charleshixsn@REDACTED Thu Nov 13 02:32:00 2014 From: charleshixsn@REDACTED (Charles Hixson) Date: Wed, 12 Nov 2014 17:32:00 -0800 Subject: [erlang-questions] Does anyone have experience with LETS? Message-ID: <54640A10.4030508@earthlink.net> https://github.com/norton/lets If this is good, it may solve some problems, but I'm quite new to Erlang, and won't be able to judge how good it is. From holger@REDACTED Thu Nov 13 11:09:03 2014 From: holger@REDACTED (Holger =?iso-8859-1?Q?Wei=DF?=) Date: Thu, 13 Nov 2014 11:09:03 +0100 Subject: [erlang-questions] gen_tcp:send timeout In-Reply-To: <0430803B-EEEE-40AF-9F8A-3C1FFD2A3B2F@gmail.com> References: <0430803B-EEEE-40AF-9F8A-3C1FFD2A3B2F@gmail.com> Message-ID: <20141113100903.GA706946@zedat.fu-berlin.de> * Dmitry Kolesnikov [2014-11-12 11:16]: > I've not discovered yet the reason but my sender process is stacked at > {current_function,{prim_inet,send,3}} > > The timeout will help me but I'am confused on the way to achieve it > > * gen_tcp provides an option {send_timeout, xxx}. The documentation says > "The fact that the send call does not accept a timeout option, is > because timeouts on send is handled through the socket option > send_timeout." I'm confused, too. Is your process getting stuck in prim_inet:send/3 even though you specified this option? Or is your question whether/how to use it? Holger From dmkolesnikov@REDACTED Thu Nov 13 11:21:07 2014 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Thu, 13 Nov 2014 12:21:07 +0200 Subject: [erlang-questions] gen_tcp:send timeout In-Reply-To: <20141113100903.GA706946@zedat.fu-berlin.de> References: <0430803B-EEEE-40AF-9F8A-3C1FFD2A3B2F@gmail.com> <20141113100903.GA706946@zedat.fu-berlin.de> Message-ID: <6C785308-FE73-48F3-B583-ED362A0C336C@gmail.com> My question is about best practice. If send_timeout exists at socket option why you need to use ports directly unless there is some undocumented issues with send_timeout Best Regards, Dmitry >-|-|-(*> > On 13.11.2014, at 12.09, Holger Wei? wrote: > > * Dmitry Kolesnikov [2014-11-12 11:16]: >> I've not discovered yet the reason but my sender process is stacked at >> {current_function,{prim_inet,send,3}} >> >> The timeout will help me but I'am confused on the way to achieve it >> >> * gen_tcp provides an option {send_timeout, xxx}. The documentation says >> "The fact that the send call does not accept a timeout option, is >> because timeouts on send is handled through the socket option >> send_timeout." > > I'm confused, too. Is your process getting stuck in prim_inet:send/3 > even though you specified this option? Or is your question whether/how > to use it? > > Holger > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From essen@REDACTED Thu Nov 13 11:23:51 2014 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Thu, 13 Nov 2014 12:23:51 +0200 Subject: [erlang-questions] gen_tcp:send timeout In-Reply-To: <6C785308-FE73-48F3-B583-ED362A0C336C@gmail.com> References: <0430803B-EEEE-40AF-9F8A-3C1FFD2A3B2F@gmail.com> <20141113100903.GA706946@zedat.fu-berlin.de> <6C785308-FE73-48F3-B583-ED362A0C336C@gmail.com> Message-ID: <546486B7.6060606@ninenines.eu> I believe Heroku is using the port directly so the send call does not block. The send call is synchronous, and they could not afford to wait so they went with an asynchronous solution. On 11/13/2014 12:21 PM, Dmitry Kolesnikov wrote: > My question is about best practice. If send_timeout exists at socket option why you need to use ports directly unless there is some undocumented issues with send_timeout > > Best Regards, > Dmitry >-|-|-(*> > > >> On 13.11.2014, at 12.09, Holger Wei? wrote: >> >> * Dmitry Kolesnikov [2014-11-12 11:16]: >>> I've not discovered yet the reason but my sender process is stacked at >>> {current_function,{prim_inet,send,3}} >>> >>> The timeout will help me but I'am confused on the way to achieve it >>> >>> * gen_tcp provides an option {send_timeout, xxx}. The documentation says >>> "The fact that the send call does not accept a timeout option, is >>> because timeouts on send is handled through the socket option >>> send_timeout." >> >> I'm confused, too. Is your process getting stuck in prim_inet:send/3 >> even though you specified this option? Or is your question whether/how >> to use it? >> >> Holger >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin http://ninenines.eu From 114999420@REDACTED Thu Nov 13 10:25:26 2014 From: 114999420@REDACTED (=?utf-8?B?YWRhbSBjaGFu?=) Date: Thu, 13 Nov 2014 17:25:26 +0800 Subject: [erlang-questions] A problem with exit erlang node. Message-ID: I added some codes to stop the receiver and sender's supervisor, and also sleep(500) before calling erlang:halt() But the problem remains.? [gs_gateway.erl] ... stop() -> supervisor:terminate_child(gs_sup, gs_tcp_client_sup), supervisor:terminate_child(gs_sup, gs_tcp_listener_sup), timer:sleep(500), io:format("gs_gateway stop~n"), ok.? The gs_tcp_client_sup is the supervisor which supervises the receiver process: [gs_tcp_client_sup.erl?] ... start_link(Mod) -> supervisor:start_link({local,?MODULE}, ?MODULE, [Mod]). init([Mod]) -> {ok, {{simple_one_for_one, 10, 10}, [{Mod, {Mod, start_link, []}, temporary, brutal_kill, worker, [Mod]}]}}.? Since I use 'brutal_kill' option, the receiver process will be killed immediately while the gs_tcp_client_sup is terminated. In this case it doesn't call gen_tcp:close(Socket) in the receiver process obviously. It is true that my 'gateway' node may has some bug , though it is working normally. When I strace the normal working 'gateway' node, it shows there are lots of 'recvfrom' errors increasing by time: [root@REDACTED scripts]# strace -c -p 21376 Process 21376 attached - interrupt to quit ^CProcess 21376 detached % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 94.61 0.004370 2 1988 epoll_wait 1.91 0.000088 1 61 writev 1.28 0.000059 1 107 times 1.28 0.000059 1 118 epoll_ctl 0.93 0.000043 0 105 15 recvfrom ------ ----------- ----------- --------- --------- ---------------- 100.00 0.004619 2379 15 total [root@REDACTED scripts]# strace -p 21376 -e 'recvfrom' Process 21376 attached - interrupt to quit recvfrom(28, "\0\0\0\16N!", 6, 0, NULL, NULL) = 6 recvfrom(28, "\0\0\0\307\0\3\r\252", 8, 0, NULL, NULL) = 8 recvfrom(28, 0x7fac5a3d5e00, 6, 0, 0, 0) = -1 EAGAIN (Resource temporarily unavailable) recvfrom(23, "\0\0\0\204\203D\3\5\2D\5\350h\3a\2R\1gR\0\0\0\0\314\0\0\0\0\1h\2"..., 1460, 0, NULL, NULL) = 136 recvfrom(23, "\0\0\0001\203D\3\5\2D\5\350h\3a\2R\1gR\0\0\0\0\314\0\0\0\0\1h\2"..., 1460, 0, NULL, NULL) = 53 recvfrom(28, "\0\0\0\31N\"", 6, 0, NULL, NULL) = 6 recvfrom(28, "\0\0\0\0\0\0\0006\0\3dev\0\1\0\3\16\v", 19, 0, NULL, NULL) = 19 recvfrom(28, 0x7fac5a3d5e00, 6, 0, 0, 0) = -1 EAGAIN (Resource temporarily unavailable) recvfrom(23, "\0\0\0V\203D\3\5\2D\5\350h\3a\2R\1gR\0\0\0\0\314\0\0\0\0\1h\2"..., 1460, 0, NULL, NULL) = 170 recvfrom(28, "\0\0\0\16N!", 6, 0, NULL, NULL) = 6 recvfrom(28, "\0\0\0\307\0\3\r\251", 8, 0, NULL, NULL) = 8 recvfrom(28, 0x7fac5a3d5e00, 6, 0, 0, 0) = -1 EAGAIN (Resource temporarily unavailable) ... But I don't know how to dig deeper here to find out why there are recvfrom errors. In additional, I use the pstack command to detect the stack of the fack dead 'gateway' node which is not exit normally: [root@REDACTED scripts]# pstack 27304 #0 0x0000000000496233 in erts_deliver_time () #1 0x000000000051ce2c in erts_check_io_kp () #2 0x0000000000516a4d in erl_sys_schedule () #3 0x000000000048d406 in schedule () #4 0x00000000004fcb34 in process_main () #5 0x0000000000450fd5 in erl_start () #6 0x0000000000435add in main ()? [root@REDACTED scripts]# pstack 27304 #0 0x000000000051bf2d in erts_poll_interrupt_kp () #1 0x000000000048d431 in schedule () #2 0x00000000004fcb34 in process_main () #3 0x0000000000450fd5 in erl_start () #4 0x0000000000435add in main ()? It seems that the erlang scheduler is still working after erlang:halt(). So, any new suggestion? Thanks~ ------------------ Original ------------------ From: "Imants Cekusins";; Date: Thu, Nov 13, 2014 07:35 AM To: "adam chan"<114999420@REDACTED>; Cc: "erlang-questions"; Subject: Re: [erlang-questions] A problem with exit erlang node. Could the 'gateway' node close its open sockets before shutting down? Maybe pause for a second after pinging an exit signal to the gateway node, stopping the apps but before calling erlang:halt()? On 12 Nov 2014 17:14, "adam chan" <114999420@REDACTED> wrote: Hi List, I have a problem to stop or exit a erlang node. When I called erlang:halt(), the node is fake dead, and the cpu goes up to 100%. Here is the situation: I'm running OTP_R15B02 on Centos 6.3. I have 3 nodes named 'server', 'unite' and 'gateway' which connected to each other. The 'gateway' node listens to a port , receives socket datas from client, and then transfers to 'server' and 'unite'. The response data from 'server' and 'unite' will send back to client through 'gateway' node too. When I want to stop all these 3 nodes, the 'gateway' node CAN NOT exit completely sometimes (small probability) . The nodes is running in screen of linux, the starting scripts like this: [start_all.sh] ... /usr/bin/screen -dmS server -s $ScriptPath/start_server.sh $Log ... /usr/bin/screen -dmS unite -s $ScriptPath/start_unite.sh $Log ... /usr/bin/screen -dmS gateway -s $ScriptPath/start_gateway.sh $Log [start_gateway.sh] #!/bin/bash cd /data/web/server/server/config ulimit -s 262140 erl -kernel inet_dist_listen_min 40001 -kernel inet_dist_listen_max 40100 +P 1024000 +K true -smp disable -name gateway@REDACTED -setcookie abc -boot start_sasl -config gs_main -pa ../ebin -s gs_main start -extra 192.168.7.100 9001 2 I stop the nodes in the order of 'gateway' -> 'unite' -> 'server' The stop scripts like this: [stop_all.sh] #!/bin/bash cd /data/web/server/server/scripts/ chmod +x stop_gateway.sh chmod +x stop_unite.sh chmod +x stop_server.sh ./stop_gateway.sh ./stop_unite.sh ./stop_server.sh [stop_gateway.sh] #!/bin/bash cd /data/web/server/server/config erl -noshell -hidden -name stop_gateway@REDACTED -setcookie abc -pa ../ebin -eval "rpc:call('gateway@REDACTED', gs_main, stop, [])." -s c q [gs_main.erl] -define(SERVER_APPS, [sasl, gs_main]). ... stop() -> ok = stop_applications(?SERVER_APPS), erlang:halt(). The 'server' and 'unite' node can exit completely every time, and the screen which is running the node also exit too. But the 'gateway' node sometimes (small probability) can't exit, the screen remains too: [root@REDACTED logs]# screen -ls There are screens on: 20107.gateway (Detached) [root@REDACTED logs]# ps -ef | grep gateway root 20107 1 0 Nov10 ? 00:00:00 /usr/bin/SCREEN -dmS gateway -s /data/web/server/server/scripts/start_gateway.sh -L -c /data/web/server/server/var/logs/screenrc_gateway root 20110 20107 0 Nov10 pts/7 00:00:00 /bin/bash /data/web/server/server/scripts/start_gateway.sh root 20111 20110 90 Nov10 pts/7 1-19:56:53 /usr/local/lib/erlang/erts-5.9.2/bin/beam -P 1024000 -K true -- -root /usr/local/lib/erlang -progname erl -- -home /root -- -kernel inet_dist_listen_min 40001 -kernel inet_dist_listen_max 40100 -smp disable -name gateway@REDACTED -setcookie abc -boot start_sasl -config gs_main -pa ../ebin -s gs_main start -extra 192.168.7.100 9001 2 [root@REDACTED logs]# strace -c -p 20111 Process 20111 attached - interrupt to quit ^CProcess 20111 detached strace command has no effect here. And one CPU core keeps running at 100%. At the end of the 'gateway' node's log, it says the application is exited: [gateway.log] =INFO REPORT==== 11-Nov-2014::10:21:18 === application: gs_main exited: stopped type: temporary It seems that some endless loop occured after the printing of the =INFO REPORT=. The application is not really exited, or the 'ps -ef | grep gateway' command won't find the 20111 process. Any ideas? Thanks in advance. ------------------ Adam Chan _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmkolesnikov@REDACTED Thu Nov 13 14:53:22 2014 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Thu, 13 Nov 2014 15:53:22 +0200 Subject: [erlang-questions] Does anyone have experience with LETS? In-Reply-To: <54640A10.4030508@earthlink.net> References: <54640A10.4030508@earthlink.net> Message-ID: I?ve decided to use eleveldb directly. https://github.com/basho/eleveldb - Dmitry > On 13 Nov 2014, at 03:32, Charles Hixson wrote: > > https://github.com/norton/lets > > If this is good, it may solve some problems, but I'm quite new to Erlang, and won't be able to judge how good it is. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From community-manager@REDACTED Thu Nov 13 17:14:03 2014 From: community-manager@REDACTED (Bruce Yinhe) Date: Thu, 13 Nov 2014 17:14:03 +0100 Subject: [erlang-questions] Calling for help to improve contents on erlang.org Message-ID: Hello, We are aware that parts of erlang.org need improvement. For example http://www.erlang.org/article/tag/examples and http://www.erlang.org/course/course.html are outdated. We would like to see a number of small code examples for beginners. The purpose of these examples is to provide an attractive and useful introduction for people who are interested in adopting the Erlang programming language. Please send us your input. We would like to call for help from the community since OTP team does not have too much time and it is not possible to submit pull requests for editorial of erlang.org as of now. Any other suggestions for erlang.org are always welcome. Best regards, Bruce Yinhe community-manager@REDACTED -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Thu Nov 13 17:22:49 2014 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Thu, 13 Nov 2014 18:22:49 +0200 Subject: [erlang-questions] Calling for help to improve contents on erlang.org In-Reply-To: References: Message-ID: <5464DAD9.1060608@ninenines.eu> Hi, This response is addressed to OTP Team, as Bruce already knows what I am saying here. Please make the erlang.org code and content open-source, and add a link to this repository at the bottom of all erlang.org pages. This is not only easier to contribute to for programmers, but will make people who read contribute spontaneously (even if they don't end up using Erlang) and will also address the concerns of those who do not want to contribute to closed software/platforms. On 11/13/2014 06:14 PM, Bruce Yinhe wrote: > Hello, > > We are aware that parts of erlang.org need > improvement. For example http://www.erlang.org/article/tag/examples and > http://www.erlang.org/course/course.html are outdated. We would like to > see a number of small code examples for beginners. The purpose of these > examples is to provide an attractive and useful introduction for people > who are interested in adopting the Erlang programming language. > > Please send us your input. We would like to call for help from the > community since OTP team does not have too much time and it is not > possible to submit pull requests for editorial of erlang.org > as of now. > > Any other suggestions for erlang.org are always welcome. > > Best regards, > Bruce Yinhe > community-manager@REDACTED > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin http://ninenines.eu From charleshixsn@REDACTED Thu Nov 13 19:20:09 2014 From: charleshixsn@REDACTED (Charles Hixson) Date: Thu, 13 Nov 2014 10:20:09 -0800 Subject: [erlang-questions] Does anyone have experience with LETS? In-Reply-To: References: <54640A10.4030508@earthlink.net> Message-ID: <5464F659.9080702@earthlink.net> Riak and eleveldb both look very interesting as solutions to the memory limitation problems. It's clearly going to take a bit of study, but thanks very much. On 11/13/2014 05:53 AM, Dmitry Kolesnikov wrote: > I?ve decided to use eleveldb directly. > https://github.com/basho/eleveldb > > - Dmitry > >> On 13 Nov 2014, at 03:32, Charles Hixson > > wrote: >> >> https://github.com/norton/lets >> >> If this is good, it may solve some problems, but I'm quite new to >> Erlang, and won't be able to judge how good it is. >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From davidw@REDACTED Thu Nov 13 20:12:31 2014 From: davidw@REDACTED (David Welton) Date: Thu, 13 Nov 2014 20:12:31 +0100 Subject: [erlang-questions] Calling for help to improve contents on erlang.org In-Reply-To: References: Message-ID: > We are aware that parts of erlang.org need improvement. For example > http://www.erlang.org/article/tag/examples and > http://www.erlang.org/course/course.html are outdated. We would like to see > a number of small code examples for beginners. The purpose of these examples > is to provide an attractive and useful introduction for people who are > interested in adopting the Erlang programming language. > > Please send us your input. We would like to call for help from the community > since OTP team does not have too much time and it is not possible to submit > pull requests for editorial of erlang.org as of now. > > Any other suggestions for erlang.org are always welcome. I completely agree with Lo?c: open source it so that people can work with it and help out. Include instructions on how to generate or recreate the environment that the site runs, so people can run it locally. Another important thing would be to add some logos (perhaps with stories/use-cases if possible) of well-known Erlang users in a prominent place: WhatsApp, Heroku, Ericsson and so on. It'd be useful to see how other communities manage that, because you don't want "David's Erlang Consulting" on that logo page or it's going to take something away from it. This is a "social proof" signal, so you want to showcase people that are using Erlang to do cool stuff. Thanks! -- David N. Welton http://www.dedasys.com/ From mjtruog@REDACTED Thu Nov 13 20:13:58 2014 From: mjtruog@REDACTED (Michael Truog) Date: Thu, 13 Nov 2014 11:13:58 -0800 Subject: [erlang-questions] Calling for help to improve contents on erlang.org In-Reply-To: <5464DAD9.1060608@ninenines.eu> References: <5464DAD9.1060608@ninenines.eu> Message-ID: <546502F6.5030705@gmail.com> On 11/13/2014 08:22 AM, Lo?c Hoguin wrote: > Hi, > > This response is addressed to OTP Team, as Bruce already knows what I am saying here. > > Please make the erlang.org code and content open-source, and add a link to this repository at the bottom of all erlang.org pages. > > This is not only easier to contribute to for programmers, but will make people who read contribute spontaneously (even if they don't end up using Erlang) and will also address the concerns of those who do not want to contribute to closed software/platforms. As part of this process it would be good to have public usage of a bug tracker to provide a transparent development process. There are submissions of issues on github which get attached to pull requests and there is an empty page at the "What you could do" section of http://www.erlang.org/development/ . Currently, decision making for Erlang has been "The OTP committee has spoken, and it will be so!" which follows a waterfall development model (a blessing process in the cathedral, as in http://en.wikipedia.org/wiki/The_Cathedral_and_the_Bazaar) and pursues the idea that Erlang is a proprietary R&D product. If Erlang is open-source a public bug tracker would have entries for every entry found in the release notes (http://www.erlang.org/download/otp_src_17.3.readme) when an issue is assigned an "OTP-#####" identifier. If development is more transparent then it will be simpler for the development to occur outside of Ericsson as a community effort. Otherwise, Ericsson politics or problems can adversely affect Erlang as a project when Ericsson process is the bottleneck on development, and that risk can impact other companies/people making them less likely to feel comfortable with using Erlang (due to the business risk of being tied to unknown events at Ericsson). With more transparency on bugs/features it will be easier to avoid creating errors in Erlang because more community involvement will lead to more testing and oversight of development. Pursuing that transparency makes it easier to diagnose problems and determine the Erlang release which should be in production (i.e., "What Erlang release does Ericsson put into production?") and package managers (or port trees). > > On 11/13/2014 06:14 PM, Bruce Yinhe wrote: >> Hello, >> >> We are aware that parts of erlang.org need >> improvement. For example http://www.erlang.org/article/tag/examples and >> http://www.erlang.org/course/course.html are outdated. We would like to >> see a number of small code examples for beginners. The purpose of these >> examples is to provide an attractive and useful introduction for people >> who are interested in adopting the Erlang programming language. >> >> Please send us your input. We would like to call for help from the >> community since OTP team does not have too much time and it is not >> possible to submit pull requests for editorial of erlang.org >> as of now. >> >> Any other suggestions for erlang.org are always welcome. >> >> Best regards, >> Bruce Yinhe >> community-manager@REDACTED >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > From t@REDACTED Thu Nov 13 20:14:42 2014 From: t@REDACTED (Tristan Sloughter) Date: Thu, 13 Nov 2014 13:14:42 -0600 Subject: [erlang-questions] Calling for help to improve contents on erlang.org In-Reply-To: References: Message-ID: <1415906082.2417474.190699173.6DB24BEA@webmail.messagingengine.com> Not to say that https://erlangcentral.org/ doesn't need more material, but it feels like this would duplicate efforts. Why not just have erlang.org link over to Erlang central for examples and such? -- Tristan Sloughter t@REDACTED On Thu, Nov 13, 2014, at 01:12 PM, David Welton wrote: > > We are aware that parts of erlang.org need improvement. For example > > http://www.erlang.org/article/tag/examples and > > http://www.erlang.org/course/course.html are outdated. We would like to see > > a number of small code examples for beginners. The purpose of these examples > > is to provide an attractive and useful introduction for people who are > > interested in adopting the Erlang programming language. > > > > Please send us your input. We would like to call for help from the community > > since OTP team does not have too much time and it is not possible to submit > > pull requests for editorial of erlang.org as of now. > > > > Any other suggestions for erlang.org are always welcome. > > I completely agree with Lo?c: open source it so that people can work > with it and help out. Include instructions on how to generate or > recreate the environment that the site runs, so people can run it > locally. > > Another important thing would be to add some logos (perhaps with > stories/use-cases if possible) of well-known Erlang users in a > prominent place: WhatsApp, Heroku, Ericsson and so on. It'd be useful > to see how other communities manage that, because you don't want > "David's Erlang Consulting" on that logo page or it's going to take > something away from it. This is a "social proof" signal, so you want > to showcase people that are using Erlang to do cool stuff. > > Thanks! > -- > David N. Welton > > http://www.dedasys.com/ > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From mononcqc@REDACTED Thu Nov 13 20:22:52 2014 From: mononcqc@REDACTED (Fred Hebert) Date: Thu, 13 Nov 2014 14:22:52 -0500 Subject: [erlang-questions] gen_tcp:send timeout In-Reply-To: <546486B7.6060606@ninenines.eu> References: <0430803B-EEEE-40AF-9F8A-3C1FFD2A3B2F@gmail.com> <20141113100903.GA706946@zedat.fu-berlin.de> <6C785308-FE73-48F3-B583-ED362A0C336C@gmail.com> <546486B7.6060606@ninenines.eu> Message-ID: <20141113192251.GC93129@ferdair.local> We do that specifically in some critical section of logplex, but in general, where we could, we've tried to architect things so that the process that sends on a buffer is not the one that also needs to apply back-pressure or load-shedding for the entire stack. In places where we've had no choice, we indeed had to use the port directly (with whatever the non-blocking flag was) to make sure it wouldn't mess with the flow of data. Regards, Fred. On 11/13, Lo?c Hoguin wrote: > I believe Heroku is using the port directly so the send call does not block. > The send call is synchronous, and they could not afford to wait so they went > with an asynchronous solution. > > On 11/13/2014 12:21 PM, Dmitry Kolesnikov wrote: > >My question is about best practice. If send_timeout exists at socket option why you need to use ports directly unless there is some undocumented issues with send_timeout > > > >Best Regards, > >Dmitry >-|-|-(*> > > > > > >>On 13.11.2014, at 12.09, Holger Wei? wrote: > >> > >>* Dmitry Kolesnikov [2014-11-12 11:16]: > >>>I've not discovered yet the reason but my sender process is stacked at > >>>{current_function,{prim_inet,send,3}} > >>> > >>>The timeout will help me but I'am confused on the way to achieve it > >>> > >>>* gen_tcp provides an option {send_timeout, xxx}. The documentation says > >>> "The fact that the send call does not accept a timeout option, is > >>> because timeouts on send is handled through the socket option > >>> send_timeout." > >> > >>I'm confused, too. Is your process getting stuck in prim_inet:send/3 > >>even though you specified this option? Or is your question whether/how > >>to use it? > >> > >>Holger > >>_______________________________________________ > >>erlang-questions mailing list > >>erlang-questions@REDACTED > >>http://erlang.org/mailman/listinfo/erlang-questions > >_______________________________________________ > >erlang-questions mailing list > >erlang-questions@REDACTED > >http://erlang.org/mailman/listinfo/erlang-questions > > > > -- > Lo?c Hoguin > http://ninenines.eu > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From imantc@REDACTED Thu Nov 13 21:44:22 2014 From: imantc@REDACTED (Imants Cekusins) Date: Thu, 13 Nov 2014 21:44:22 +0100 Subject: [erlang-questions] Beam archives (~ namespace) Message-ID: Would it be possible to implement *.beam archives (something like *.jar's)? This would help a lot to deal with name clashes. E.g., a bunch of files are packaged in one good_program.bear file Then the code would show: -import(good_program). good_program_module:function(). i.e., archives are in the code path but need to be specified explicitly before use. ? From darach@REDACTED Thu Nov 13 21:59:14 2014 From: darach@REDACTED (Darach Ennis) Date: Thu, 13 Nov 2014 20:59:14 +0000 Subject: [erlang-questions] Beam archives (~ namespace) In-Reply-To: References: Message-ID: Hi Imants, Erlang supports .ez (zip based) archives that supports something like that... http://erlang.org/doc/man/code.html It works quite well for pluggable behaviours etc. Cheers, Darach. On 13 Nov 2014 20:44, "Imants Cekusins" wrote: > Would it be possible to implement *.beam archives (something like *.jar's)? > > This would help a lot to deal with name clashes. > > E.g., a bunch of files are packaged in one good_program.bear file > > Then the code would show: > -import(good_program). > > good_program_module:function(). > > i.e., archives are in the code path but need to be specified > explicitly before use. > > ? > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ulf@REDACTED Thu Nov 13 23:41:57 2014 From: ulf@REDACTED (Ulf Wiger) Date: Thu, 13 Nov 2014 23:41:57 +0100 Subject: [erlang-questions] Does anyone have experience with LETS? In-Reply-To: <54640A10.4030508@earthlink.net> References: <54640A10.4030508@earthlink.net> Message-ID: Basho has had to work very hard to get around some nasty blocking issues with leveldb. We used to experience those issues in eleveldb before, but the 2.0 version works very well. My guess is that LETS has not addressed these issues, at least to the level that Basho has. For an idea of the problem, watch this presentation: https://www.youtube.com/watch?v=vo88IdglU_8 BR, Ulf On 13 Nov 2014, at 02:32, Charles Hixson wrote: > https://github.com/norton/lets > > If this is good, it may solve some problems, but I'm quite new to Erlang, and won't be able to judge how good it is. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. http://feuerlabs.com From roberto@REDACTED Fri Nov 14 00:09:13 2014 From: roberto@REDACTED (Roberto Ostinelli) Date: Fri, 14 Nov 2014 00:09:13 +0100 Subject: [erlang-questions] load balancing Message-ID: Dear list, I have many servers in a cluster that handle long-lived TCP connections with multiple clients. I'd like to collect some opinions on what you'd believe to be the best options to balance the load on these servers: - Implement a protocol that a client has to first ask to which server to connect to, and then go for it - Use SRV records - Have all traffic through some proxy like HAProxy (but then, doesn't this mean that a single box has to handle millions of connections?) I do realize that this is not necessarily related to Erlang, but as we often have to deal with backend servers with long-lived TCP connections I thought there might be the correct experience here. Hopefully some kind sould will provide me with some suggestions :) Best, r. -------------- next part -------------- An HTML attachment was scrubbed... URL: From vladimir.ralev@REDACTED Fri Nov 14 02:03:34 2014 From: vladimir.ralev@REDACTED (Vladimir Ralev) Date: Fri, 14 Nov 2014 03:03:34 +0200 Subject: [erlang-questions] load balancing In-Reply-To: References: Message-ID: Hi, I do this a lot and here are two cents about common issues I see. It all depends on the protocol (reconnects/retransmits/statefulness/session-based) and if you have control over the clients. The best load balancing is always on the client side. If you have control over the clients just code them to search and pick their server with your custom timeout rules and recovery. The next best thing is if the clients have SRV awareness or even a simple multiple A DNS record awareness that would be a very reasonable solution Proper SRV support from random apps is rare in my experience, even common client for protocols like SIP, HTTP and XMPP and bad with SRV. Normal DNS A dispenser is sufficient most of the time, but doesn't handle failover cases very well. There is no advantage in using a load balancer unless your protocol is stateful and requires app-level affinity or sticky sessions, however there are a few exceptions. TCP itself is stateful however if your protocol knows to reconnect and heartbeat independently you can ignore that. In the context of Erlang, you may have to consider how are you going to handle production maintenance operations such as moving nodes around and detecting cluster in bad state. Load balancers can be reliable and do great for mid-session failover cases, but it's more expensive and difficult. If you are using a load balancer it has to be able to maintain existing TCP connections until they terminate but not send new connections to the serviced nodes. Another production operation you need to consider is IP-takeover to replace a node at some IP address, Erlang doesn't handle this very well on a running system and it has to be done with virtual adapters or redirection at the OS level. If your protocol stamps IP addresses or names inside the payload like SIP/SDP you will need application awareness of the load balancing and IP takeover operations. Many solutions for SIP or HTTP are hybrid with active/passive DNS pointing to N IP load balancers feeding M app-level load balancers to assure protocol specific sticky-ness (HTTP session/Call-ID). On Fri, Nov 14, 2014 at 1:09 AM, Roberto Ostinelli wrote: > Dear list, > I have many servers in a cluster that handle long-lived TCP connections > with multiple clients. > > I'd like to collect some opinions on what you'd believe to be the best > options to balance the load on these servers: > > - Implement a protocol that a client has to first ask to which server to > connect to, and then go for it > - Use SRV records > - Have all traffic through some proxy like HAProxy (but then, doesn't > this mean that a single box has to handle millions of connections?) > > I do realize that this is not necessarily related to Erlang, but as we > often have to deal with backend servers with long-lived TCP connections I > thought there might be the correct experience here. > > Hopefully some kind sould will provide me with some suggestions :) > > Best, > r. > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Fri Nov 14 05:11:33 2014 From: max.lapshin@REDACTED (Max Lapshin) Date: Fri, 14 Nov 2014 07:11:33 +0300 Subject: [erlang-questions] load balancing In-Reply-To: References: Message-ID: > Implement a protocol that a client has to first ask to which server to connect to, and then go for it It is the best one if you can add it: 1) it doesn't have DNS lag 2) it doesn't require such SPOF like HAProxy 3) you will have problem how to design resolver, but it will be simpler and it is possible to do it reliable and scalable. -------------- next part -------------- An HTML attachment was scrubbed... URL: From felixgallo@REDACTED Fri Nov 14 05:59:16 2014 From: felixgallo@REDACTED (Felix Gallo) Date: Thu, 13 Nov 2014 20:59:16 -0800 Subject: [erlang-questions] load balancing In-Reply-To: References: Message-ID: haproxy is really, really excellent at load balancing and providing HA switchover. It has a number of modes, can do weighting, heartbeats, has a vibrant community, and is incredibly stable. Before you write your own, I'd try that. On Thu, Nov 13, 2014 at 8:11 PM, Max Lapshin wrote: > > Implement a protocol that a client has to first ask to which server to > connect to, and then go for it > > It is the best one if you can add it: > > 1) it doesn't have DNS lag > 2) it doesn't require such SPOF like HAProxy > 3) you will have problem how to design resolver, but it will be simpler > and it is possible to do it reliable and scalable. > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fritchie@REDACTED Fri Nov 14 09:38:16 2014 From: fritchie@REDACTED (Scott Lystig Fritchie) Date: Fri, 14 Nov 2014 02:38:16 -0600 Subject: [erlang-questions] Does anyone have experience with LETS? In-Reply-To: Message of "Thu, 13 Nov 2014 23:41:57 +0100." Message-ID: <43792.1415954296@snookles.snookles.com> Ulf Wiger wrote: uw> Basho has had to work very hard to get around some nasty blocking uw> issues with leveldb. Indeed, there's been a tremendous amount of work on LevelDB ... so much so that Basho's fork of LevelDB[1] (which is separate from the Erlang binding[2]) is really a separate & independent fork of Google's original version. Matthew has made a lot of changes to make LevelDB useful in Riak's concurrent environment, which uses *many* LevelDB instances simultaneously. The Google folks aren't interested in that kind of use case. Matthew still has a good working relationship with the parent project, AFAIK, so they're still accepting patches as we find and fix bugs. Basho has a strong interest in continuing development & performance enhancements. That Basho fork will remain active for the indefinite future. -Scott [1] https://github.com/basho/leveldb [2] https://github.com/basho/eleveldb From darach@REDACTED Fri Nov 14 21:22:46 2014 From: darach@REDACTED (Darach Ennis) Date: Fri, 14 Nov 2014 20:22:46 +0000 Subject: [erlang-questions] [ANN] hdr_histogram_erl 0.1.0 High Dynamic Range Histogram Library for Erlang/OTP Message-ID: Where: https://github.com/darach/hdr_histogram_erl What: The HDR histogram library is an Erlang native interface function wrapper of Mike Barker's C port of Gil Tene's HDR Histogram java utility. Details here: http://hdrhistogram.org/ A high dynamic range histogram is one that supports recording and analyzing sampled data points across a configurable range with configurable precision within that range. The precision is expressed as a number of significant figures in the recording. This HDR histogram implementation is designed for recording histograms of value measurements in latency sensitive environments. Although the native recording times can be as low as single digit nanoseconds there is added overhead in this wrapper/binding due to both the frontend overhead of converting from native C to the NIF interface, and the erlang overhead incurred calling into the NIFs. C'est la vie, I suppose. A distinct advantage of this histogram implementation is constant space and recording (time) overhead with an ability to recycle and reset instances whilst reclaiming already allocated space for reuse thereby reducing allocation cost and garbage collection overhead in the BEAM where repeated or continuous usage is likely. For example, a gen_server recording metrics continuously and resetting and logging histogram dumps on a periodic or other windowed basis. API docs here - https://github.com/darach/hdr_histogram_erl/blob/master/doc/hdr_histogram.md There are simple Erlang and Elixir examples in the repository. Enjoy! -------------- next part -------------- An HTML attachment was scrubbed... URL: From bchesneau@REDACTED Sat Nov 15 08:14:47 2014 From: bchesneau@REDACTED (Benoit Chesneau) Date: Sat, 15 Nov 2014 08:14:47 +0100 Subject: [erlang-questions] DTLS/SRTP for WebRTC In-Reply-To: References: Message-ID: On Sunday, November 2, 2014, Ingela Andin wrote: > Hi! > > 2014-10-30 22:33 GMT+01:00 pablo platt >: > >> Hi, >> >> What is the state of DTLS in OTP? >> > > Alas Ericsson has prioritised quite a few other things higher than DTLS > just lately so the implementation has been delayed. > I hope it will get back on track soon but I am afraid you can no count on > it in a short term perspective. > > Regards Ingela Erlang/OTP team - Ericsson AB > > what is missing currently? is there a list of tasks and known bus available? - benoit > > >> >> I'm interested in using DTLS/SRTP in Erlang for WebRTC. >> >> I've found two implementations: >> >> Clean code but I couldn't make it work because it's missing SRTP support. >> https://github.com/RoadRunnr/otp/tree/new_crypto_dtls >> >> This works quite well. I'm able to connect clients to the server: >> https://groups.google.com/d/msg/discuss-webrtc/MP-1sCrOljA/qAs4VK-18y4J >> >> The problem is that some clients can't connect and I'm getting HANDSHAKE >> packets while in the CIPHER state in the following order: >> HANDSHAKE, SERVER_HELLO >> HANDSHAKE, CERTIFICATE >> HANDSHAKE, SERVER_KEY_EXCHANGE >> HANDSHAKE, CERTIFICATE_REQUEST >> HANDSHAKE, SERVER_HELLO_DONE >> >> I'm assuming that is related to wrong state change or retransmission. >> The code only implement retransmission for two packet types and have the >> following comment: >> %% problem remainning: different strategy should be adopted while >> state==hello or cipher. >> >> Does anyone use DTLS/SRTP in Erlang and help solve this issue or >> recommend another solution? >> >> Thanks >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -- Sent from my Mobile -------------- next part -------------- An HTML attachment was scrubbed... URL: From ulf@REDACTED Sat Nov 15 09:46:57 2014 From: ulf@REDACTED (Ulf Wiger) Date: Sat, 15 Nov 2014 09:46:57 +0100 Subject: [erlang-questions] Does anyone have experience with LETS? In-Reply-To: <43792.1415954296@snookles.snookles.com> References: Message of "Thu, 13 Nov 2014 23:41:57 +0100." <43792.1415954296@snookles.snookles.com> Message-ID: Scott, I can testify that it doesn't take that many instances for leveldb till cause problems. Before upgrading to eleveldb 2.0, we experienced some lethal blocking* conditions using only one instance via kvdb. With 2.0, we've seen none of that. BR, Ulf W * Mainly when running on EC2, where disk access is decidedly slower than with the SSD in your average laptop. ---- Scott Lystig Fritchie skrev ---- >Ulf Wiger wrote: > >uw> Basho has had to work very hard to get around some nasty blocking >uw> issues with leveldb. > >Indeed, there's been a tremendous amount of work on LevelDB ... so much >so that Basho's fork of LevelDB[1] (which is separate from the Erlang >binding[2]) is really a separate & independent fork of Google's original >version. Matthew has made a lot of changes to make LevelDB useful in >Riak's concurrent environment, which uses *many* LevelDB instances >simultaneously. The Google folks aren't interested in that kind of use >case. > >Matthew still has a good working relationship with the parent project, >AFAIK, so they're still accepting patches as we find and fix bugs. >Basho has a strong interest in continuing development & performance >enhancements. That Basho fork will remain active for the indefinite >future. > >-Scott > >[1] https://github.com/basho/leveldb >[2] https://github.com/basho/eleveldb > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kevprg@REDACTED Sat Nov 15 15:07:23 2014 From: kevprg@REDACTED (kab) Date: Sat, 15 Nov 2014 09:07:23 -0500 Subject: [erlang-questions] newbie question In-Reply-To: <20141107145805.GA4786@localhost.localdomain> References: <545BC45B.9020207@earthlink.net> <545BC74F.5090508@2600hz.com> <545BCB6D.3040409@anydata.co.uk> <20141107145805.GA4786@localhost.localdomain> Message-ID: <20141115140723.GD23509@localhost.localdomain> resending I'm a noob trying to learn erlang too. I took Charles script and applied the feedback. The only part I'm not grokking is why re:split wants a 3 part list on the left hand side. I expected it to want two parts; one for the string matched and the second for the remaining string. Is there something about the re that is matching an empty string? -module(chunks). main(_) -> io:format("Chunks Forward~n"), CF = chunks_fwd("Able was I ere I saw Elba."), lists:foreach(fun(A) -> io:format("~p~n", [A]) end, CF), io:format("Chunks Reverse~n"), CR = chunks_rev("Able was I ere I saw Elba."), lists:foreach(fun(A) -> io:format("~p~n", [A]) end, CR) . chunks_fwd("") -> []; chunks_fwd(Str)-> P1 = "^([[:alnum:]]+|[[:space:]]+|[[:punct:]]+|[[:cntrl:]]+|.)", [[], M1, T1] = re:split(Str, P1, [{return, list}]), [M1 | chunks_fwd(T1)] . chunks_rev(Str) -> chunks_rev(Str, []). chunks_rev("", Lst) -> Lst; chunks_rev(Str, Lst)-> P1 = "^([[:alnum:]]+|[[:space:]]+|[[:punct:]]+|[[:cntrl:]]+|.)", [[], M1, T1] = re:split(Str, P1, [{return, list}]), chunks_rev(T1, [M1|Lst]) . On Thu, Nov 06, 2014 at 08:58:35PM +0100, Erik S?e S?rensen wrote: > Hi there - > > Apart from the atom-vs-Variable issue, I think you got your recursion wrong. > > [M1 | chunks(T1, P1)] > > doesn't make too much sense (that I can see) - especially the "recurse with > P1 for Lst" part. > > I think you mean either > > chunks(T1, [M1|Lst]) > > for a tail recursive loop, or > > chunks("") -> []; > > chunks(Str) -> ..., [M1 | chunks(T1)] > > for "normal" recursion on the rest. At present, you code looks like a > confusion of the two (apart from the Lst/P1 difference). > > > Regards, and have fun on the learning curve, > > Erik > > > Den 06/11/2014 20.27 skrev "Fred Youhanaie" : > > > > > and of course the same goes for lst in chunks/2 :) > > > > On 06/11/14 19:09, James Aimonetti wrote: > > > >> -----BEGIN PGP SIGNED MESSAGE----- > >> Hash: SHA1 > >> > >> str should be Str. Variables start with a capital letter. > >> > >> Le 2014-11-06 10:56, Charles Hixson a ?crit : > >> > >>> Given the routine: -module(chunks). -export([chunks/1]). > >>> chunks(str)-> chunks(str, []). > >>> > >>> chunks("", lst)-> list:reverse(lst); chunks(str, lst)-> P1 = > >>> "^([[:alnum:]]+|[[:space:]]+|[[:punct:]]+|[[:cntrl:]]+|.)", [[], > >>> M1, T1] = re:split(str, P1, [{return, list}]), [M1 | chunks(T1, > >>> P1)]. > >>> > >>> and the shell: 47> c(chunks). {ok,chunks} > >>> > >>> Why do I get: 49> chunks:chunks("Able was I ere I saw Elba."). ** > >>> exception error: no function clause matching chunks:chunks("Able > >>> was I ere I saw Elba.") (chunks.erl, line 15) (Well, forget the > >>> like 15. I took out a bunch of comments.) > >>> > >> _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- Kevin Baker - PnTOps irc: kbaker phone: +1 978 392 2442 From s.kostyushkin@REDACTED Sat Nov 15 11:33:41 2014 From: s.kostyushkin@REDACTED (=?UTF-8?B?0KHQtdGA0LPRltC5INCa0L7RgdGC0Y7RiNC60ZbQvQ==?=) Date: Sat, 15 Nov 2014 12:33:41 +0200 Subject: [erlang-questions] [ANN] jamdb_sybase (Erlang driver for SAP Sybase ASE) Message-ID: Hello, I want to make an official announce of jamdb_sybase. https://github.com/erlangbureau/jamdb_sybase jamdb_sybase is driver for SAP Sybase ASE written in pure Erlang. Eighteen months ago, I needed to work with Sybase Adaptive Server Enterprise. ODBC is not approached because require additional configuration at the operating system level. So there was this project. This driver more than a year works in highload production, on several projects, and established itself well. Though it still needs improvement, so send your commits if you need this project like me. -------------- next part -------------- An HTML attachment was scrubbed... URL: From fly@REDACTED Sat Nov 15 18:31:59 2014 From: fly@REDACTED (Fred Youhanaie) Date: Sat, 15 Nov 2014 17:31:59 +0000 Subject: [erlang-questions] newbie question In-Reply-To: <20141115140723.GD23509@localhost.localdomain> References: <545BC45B.9020207@earthlink.net> <545BC74F.5090508@2600hz.com> <545BCB6D.3040409@anydata.co.uk> <20141107145805.GA4786@localhost.localdomain> <20141115140723.GD23509@localhost.localdomain> Message-ID: <54678E0F.8080104@anydata.co.uk> I believe the first (empty) element of the returned list corresponds to the unmatched part at the start of the string, outside the sub-expression. BTW, re:split/2,3 may return longer lists, if there are more sub-expressions in the pattern. Cheers Fred On 15/11/14 14:07, kab wrote: > resending > > I'm a noob trying to learn erlang too. I took Charles script and > applied the feedback. The only part I'm not grokking is why re:split > wants a 3 part list on the left hand side. I expected it to want two > parts; one for the string matched and the second for the remaining > string. Is there something about the re that is matching an empty > string? > > > -module(chunks). > > main(_) -> > io:format("Chunks Forward~n"), > CF = chunks_fwd("Able was I ere I saw Elba."), > lists:foreach(fun(A) -> io:format("~p~n", [A]) end, CF), > io:format("Chunks Reverse~n"), > CR = chunks_rev("Able was I ere I saw Elba."), > lists:foreach(fun(A) -> io:format("~p~n", [A]) end, CR) > . > > chunks_fwd("") -> []; > chunks_fwd(Str)-> > P1 = "^([[:alnum:]]+|[[:space:]]+|[[:punct:]]+|[[:cntrl:]]+|.)", > [[], M1, T1] = re:split(Str, P1, [{return, list}]), > [M1 | chunks_fwd(T1)] > . > > chunks_rev(Str) -> > chunks_rev(Str, []). > chunks_rev("", Lst) -> Lst; > chunks_rev(Str, Lst)-> > P1 = "^([[:alnum:]]+|[[:space:]]+|[[:punct:]]+|[[:cntrl:]]+|.)", > [[], M1, T1] = re:split(Str, P1, [{return, list}]), > chunks_rev(T1, [M1|Lst]) > . > From yelt@REDACTED Sun Nov 16 03:35:52 2014 From: yelt@REDACTED (._) Date: Sun, 16 Nov 2014 10:35:52 +0800 Subject: [erlang-questions] =?gb2312?b?UXVlc3Rpb25zIGFib3V0IGhvdyB0byBs?= =?gb2312?b?aW5rIGVybF9pbnRlcmZhY2UubGlio78=?= Message-ID: Hi all, I meet a problem when compiling port examples from the otp document. I use VS 2010 and erl6.2(64bit). The compiler complains that it cannot find symbols, such as _erl_free_term, _erl_free_compound . Below is the command line: Link: C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\bin\link.exe /ERRORREPORT:PROMPT /OUT:"C:\Projects\extprg\Debug\extprg.exe" /NOLOGO /LIBPATH:"C:\Projects\extprg\extprg\..\..\..\..\..\..\Program Files\erl6.2\usr\lib" erl_interface.lib ei.lib kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib /MANIFEST /ManifestFile:"Debug\extprg.exe.intermediate.manifest" /MANIFESTUAC:"level='asInvoker' uiAccess='false'" /DEBUG /PDB:"C:\Projects\extprg\Debug\extprg.pdb" /TLBID:1 /DYNAMICBASE /NXCOMPAT /IMPLIB:"C:\Projects\extprg\Debug\extprg.lib" /MACHINE:X86 Debug\complex.obj Debug\ei.obj Debug\erl_comm.obj I use dumpbin.exe on erl_interface.lib, there is a entry of _erl_free_term. I don?t now how to fix this problem, may someone give me some clues? Or, should I use erl-32-bit? -------------- next part -------------- An HTML attachment was scrubbed... URL: From sergej.jurecko@REDACTED Sun Nov 16 10:08:30 2014 From: sergej.jurecko@REDACTED (Sergej Jurecko) Date: Sun, 16 Nov 2014 10:08:30 +0100 Subject: [erlang-questions] =?utf-8?q?Questions_about_how_to_link_erl=5Fin?= =?utf-8?b?dGVyZmFjZS5saWLvvJ8=?= In-Reply-To: References: Message-ID: <27DBF08E-D8ED-4E37-9A8B-E5688011FAEC@gmail.com> Best to use rebar. Add erts bin folder to system path, compile rebar on windows (copy rebar and rebar.cmd files to your project), create rebar.config and compile from visual studio console with rebar. Sergej On 16 Nov 2014, at 03:35, ._ wrote: > Hi all, > I meet a problem when compiling port examples from the otp document. > > I use VS 2010 and erl6.2(64bit). The compiler complains that it cannot find symbols, such as _erl_free_term, _erl_free_compound . > > Below is the command line: > > Link: > C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\bin\link.exe > /ERRORREPORT:PROMPT > /OUT:"C:\Projects\extprg\Debug\extprg.exe" > /NOLOGO > /LIBPATH:"C:\Projects\extprg\extprg\..\..\..\..\..\..\Program Files\erl6.2\usr\lib" > erl_interface.lib ei.lib kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib > /MANIFEST > /ManifestFile:"Debug\extprg.exe.intermediate.manifest" > /MANIFESTUAC:"level='asInvoker' uiAccess='false'" > /DEBUG > /PDB:"C:\Projects\extprg\Debug\extprg.pdb" > /TLBID:1 > /DYNAMICBASE > /NXCOMPAT > /IMPLIB:"C:\Projects\extprg\Debug\extprg.lib" > /MACHINE:X86 Debug\complex.obj Debug\ei.obj Debug\erl_comm.obj > > I use dumpbin.exe on erl_interface.lib, there is a entry of _erl_free_term. > > I don?t now how to fix this problem, may someone give me some clues? Or, should I use erl-32-bit? > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From xapwing@REDACTED Sun Nov 16 13:52:45 2014 From: xapwing@REDACTED (Arie van Wingerden) Date: Sun, 16 Nov 2014 13:52:45 +0100 Subject: [erlang-questions] Erlang for Power Grid Message-ID: Hi, I'm just curious about the usage of Erlang for Power Grid related applications. I already saw a video with a German guy talking about a Grid using Erlang. Are there more occasions of that? And is Erlang being used for monitoring of Smart Meters somewhere? TIA. -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.elsgaard@REDACTED Sun Nov 16 14:45:31 2014 From: thomas.elsgaard@REDACTED (Thomas Elsgaard) Date: Sun, 16 Nov 2014 10:45:31 -0300 Subject: [erlang-questions] Erlang for Power Grid In-Reply-To: References: Message-ID: Den s?ndag den 16. november 2014 skrev Arie van Wingerden : > Hi, I'm just curious about the usage of Erlang for Power Grid related > applications. > > I already saw a video with a German guy talking about a Grid using Erlang. > > Are there more occasions of that? > > And is Erlang being used for monitoring of Smart Meters somewhere? > > TIA. > Hi Arie, AS i remember, http://www.cyantechnology.com/smart-electricity-metering/ is using erlang for the smart meter data collection. They presented it at the London 2012 Erlang Factory Lite Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From fritchie@REDACTED Mon Nov 17 04:51:05 2014 From: fritchie@REDACTED (Scott Lystig Fritchie) Date: Sun, 16 Nov 2014 21:51:05 -0600 Subject: [erlang-questions] Does anyone have experience with LETS? In-Reply-To: Message of "Sat, 15 Nov 2014 09:46:57 +0100." Message-ID: <201411170351.sAH3p5KB008027@snookles.snookles.com> Ulf, I discovered that we have been in good company. According to the speaker at this RICON talk a few weeks ago, both Yahoo and Facebook discovered the exact same things. :-| https://www.youtube.com/watch?v=8cYp52oeDx4&list=PL9Jh2HsAWHxLco7V1SjU9hUzP53CBZOYO&index=20 -Scott From bchesneau@REDACTED Mon Nov 17 09:50:43 2014 From: bchesneau@REDACTED (Benoit Chesneau) Date: Mon, 17 Nov 2014 09:50:43 +0100 Subject: [erlang-questions] aes gcm 128 Message-ID: Is there any chance that this patch can fall in 17.4 ? : https://github.com/erlang/otp/commit/7603a4029514a644c8323028b06acdc33e45b286 Or is this only planned for the 18 ? - benoit -------------- next part -------------- An HTML attachment was scrubbed... URL: From olivier.boudeville@REDACTED Mon Nov 17 10:12:08 2014 From: olivier.boudeville@REDACTED (Olivier BOUDEVILLE) Date: Mon, 17 Nov 2014 10:12:08 +0100 Subject: [erlang-questions] RE Erlang for Power Grid In-Reply-To: References: Message-ID: Hi, Erlang has also been used in order to *simulate* upcoming smart metering infrastructures (rather than to implement them) by a few of our projects, using Sim-Diasca (http://sim-diasca.com/) - notably in 2011 in the context of the CLEVER project which benefited from a bit of public exposure ( http://www.cleveronline.org/). Best, Olivier Boudeville. --------------------------- Olivier Boudeville EDF R&D : 1, avenue du G?n?ral de Gaulle, 92140 Clamart, France D?partement SINETICS, groupe ASICS (I2A), bureau B-226 Office : +33 1 47 65 59 58 / Mobile : +33 6 16 83 37 22 / Fax : +33 1 47 65 27 13 xapwing@REDACTED Envoy? par : erlang-questions-bounces@REDACTED 16/11/2014 14:25 A erlang-questions@REDACTED cc Objet [erlang-questions] Erlang for Power Grid Hi, I'm just curious about the usage of Erlang for Power Grid related applications. I already saw a video with a German guy talking about a Grid using Erlang. Are there more occasions of that? And is Erlang being used for monitoring of Smart Meters somewhere? TIA._______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont ?tablis ? l'intention exclusive des destinataires et les informations qui y figurent sont strictement confidentielles. Toute utilisation de ce Message non conforme ? sa destination, toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse. Si vous n'?tes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez re?u ce Message par erreur, merci de le supprimer de votre syst?me, ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Nous vous remercions ?galement d'en avertir imm?diatement l'exp?diteur par retour du message. Il est impossible de garantir que les communications par messagerie ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute erreur ou virus. ____________________________________________________ This message and any attachments (the 'Message') are intended solely for the addressees. The information contained in this Message is confidential. Any use of information contained in this Message not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval. If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return message. E-mail communication cannot be guaranteed to be timely secure, error or virus-free. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ingela.andin@REDACTED Mon Nov 17 11:30:49 2014 From: ingela.andin@REDACTED (Ingela Andin) Date: Mon, 17 Nov 2014 11:30:49 +0100 Subject: [erlang-questions] DTLS/SRTP for WebRTC In-Reply-To: References: Message-ID: Hi! 2014-11-15 8:14 GMT+01:00 Benoit Chesneau : > > > On Sunday, November 2, 2014, Ingela Andin wrote: > >> Hi! >> >> 2014-10-30 22:33 GMT+01:00 pablo platt : >> >>> Hi, >>> >>> What is the state of DTLS in OTP? >>> >> >> Alas Ericsson has prioritised quite a few other things higher than DTLS >> just lately so the implementation has been delayed. >> I hope it will get back on track soon but I am afraid you can no count on >> it in a short term perspective. >> >> Regards Ingela Erlang/OTP team - Ericsson AB >> >> > > what is missing currently? is there a list of tasks and > known bus available? > > No there is no such public list. The big show stopper is that the implementation of the DTLS-connection process is not finished. To try to summarize We do not want bugs in DTLS connections to affect TLS-connections so the connection processes must be separately implemented. DTLS is defined as a diff against TLS and that is how we want to structure the implementation. The TLS connection process is implemented by tls_connection.erl and the DTLS connection shall be in dtls_connection.erl which is only in a sketch state. Both implementations use ssl_connection.erl to handle common parts, however some calls in the DTLS code might not be quite up to date. The same is true for tls_handshake/record/alert, dtls_hanshake/record/alert , ssl_handhake/record/alert. There might be a need of further refactoring to make this approach work the whole way. Something the we will also will need is the heart beat extension (RFC 6520) which is fairly separate so it could be easy for some one to contribute the basic support for this. Our next goal is to make a test suite against openSSL DTLS so that we can have a test driven development of the DTLS connection process. Regards Ingela Erlang/OTP team - Ericsson A > - benoit > >> >> >>> >>> I'm interested in using DTLS/SRTP in Erlang for WebRTC. >>> >>> I've found two implementations: >>> >>> Clean code but I couldn't make it work because it's missing SRTP support. >>> https://github.com/RoadRunnr/otp/tree/new_crypto_dtls >>> >>> This works quite well. I'm able to connect clients to the server: >>> https://groups.google.com/d/msg/discuss-webrtc/MP-1sCrOljA/qAs4VK-18y4J >>> >>> The problem is that some clients can't connect and I'm getting HANDSHAKE >>> packets while in the CIPHER state in the following order: >>> HANDSHAKE, SERVER_HELLO >>> HANDSHAKE, CERTIFICATE >>> HANDSHAKE, SERVER_KEY_EXCHANGE >>> HANDSHAKE, CERTIFICATE_REQUEST >>> HANDSHAKE, SERVER_HELLO_DONE >>> >>> I'm assuming that is related to wrong state change or retransmission. >>> The code only implement retransmission for two packet types and have the >>> following comment: >>> %% problem remainning: different strategy should be adopted while >>> state==hello or cipher. >>> >>> Does anyone use DTLS/SRTP in Erlang and help solve this issue or >>> recommend another solution? >>> >>> Thanks >>> >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >> > > -- > Sent from my Mobile > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aschultz@REDACTED Mon Nov 17 12:10:40 2014 From: aschultz@REDACTED (Andreas Schultz) Date: Mon, 17 Nov 2014 11:10:40 +0000 (UTC) Subject: [erlang-questions] DTLS/SRTP for WebRTC In-Reply-To: <651666163.987017.1416222640143.JavaMail.zimbra@tpip.net> References: Message-ID: <1561626763.987018.1416222640677.JavaMail.zimbra@tpip.net> Hi, I've started some work on the test suite and some other bits and pieces of the DTLS support. My current (non working) state is at: https://github.com/RoadRunnr/otp/tree/R17/dtls what I've done so far: * remove all test case that use raw tcp from the DTLS tests * change the ssl.erl interface to take pid's for connection socket (required for the UDP wrapper) * elevate ChangeCipherSpec to an FSM level event (check the commit message for the reasoning behind that) * forward port changes from the TLS code to the DTLS code The biggest outstanding pieces of work are: * hook DTLS fragment assembler into next_tls_record logic (or rewrite/refactor that part) * figure out how to (re)implement the DTLS flight logic and the associated re-transmission logic Ingela mentioned a DTLS test suite and I really would love to have one that goes beyond what TLS currently does. For DTLS the really difficult tests to write will be the once testing the reassembly and re-transmission logic. Getting the actual implementation right is already quite difficult, but getting good test cases that cover all the different permutations of fragment loss and reordering is IMHO more challenging. Andreas ----- On 17 Nov, 2014, at 11:30, Ingela Andin ingela.andin@REDACTED wrote: > Hi! > > > 2014-11-15 8:14 GMT+01:00 Benoit Chesneau < bchesneau@REDACTED > : > > > > > On Sunday, November 2, 2014, Ingela Andin < ingela.andin@REDACTED > wrote: > > > > Hi! > > 2014-10-30 22:33 GMT+01:00 pablo platt < pablo.platt@REDACTED > : > > > > Hi, > > What is the state of DTLS in OTP? > > Alas Ericsson has prioritised quite a few other things higher than DTLS just > lately so the implementation has been delayed. > I hope it will get back on track soon but I am afraid you can no count on it in > a short term perspective. > > Regards Ingela Erlang/OTP team - Ericsson AB > > > > what is missing currently? is there a list of tasks and known bus available? > > > No there is no such public list. The big show stopper is that the implementation > of the DTLS-connection process is not finished. To try to summarize > > We do not want bugs in DTLS connections to affect TLS-connections so the > connection processes must be separately implemented. DTLS is > defined as a diff against TLS and that is how we want to structure the > implementation. > > The TLS connection process is implemented by tls_connection.erl and the DTLS > connection shall be in dtls_connection.erl which is only in a sketch state. > Both implementations use ssl_connection.erl to handle common parts, however some > calls in the DTLS code might not be quite up to date. > The same is true for tls_handshake/record/alert, dtls_hanshake/record/alert , > ssl_handhake/record/alert. There might be a need of further refactoring > to make this approach work the whole way. > > Something the we will also will need is the heart beat extension (RFC 6520) > which is fairly separate so it could be easy for some one to contribute the > basic support > for this. > > Our next goal is to make a test suite against openSSL DTLS so that we can have a > test driven development of the DTLS connection process. > > Regards Ingela Erlang/OTP team - Ericsson A > > > > > - benoit > > > > > > > > I'm interested in using DTLS/SRTP in Erlang for WebRTC. > > I've found two implementations: > > Clean code but I couldn't make it work because it's missing SRTP support. > https://github.com/RoadRunnr/otp/tree/new_crypto_dtls > > This works quite well. I'm able to connect clients to the server: > https://groups.google.com/d/msg/discuss-webrtc/MP-1sCrOljA/qAs4VK-18y4J > > The problem is that some clients can't connect and I'm getting HANDSHAKE packets > while in the CIPHER state in the following order: > HANDSHAKE, SERVER_HELLO > HANDSHAKE, CERTIFICATE > HANDSHAKE, SERVER_KEY_EXCHANGE > HANDSHAKE, CERTIFICATE_REQUEST > HANDSHAKE, SERVER_HELLO_DONE > > I'm assuming that is related to wrong state change or retransmission. > The code only implement retransmission for two packet types and have the > following comment: > %% problem remainning: different strategy should be adopted while state==hello > or cipher. > > Does anyone use DTLS/SRTP in Erlang and help solve this issue or recommend > another solution? > > Thanks > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > > -- > Sent from my Mobile > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- -- Dipl. Inform. Andreas Schultz From michael@REDACTED Mon Nov 17 13:53:07 2014 From: michael@REDACTED (Michael Gebetsroither) Date: Mon, 17 Nov 2014 13:53:07 +0100 Subject: [erlang-questions] Does anyone have experience with LETS? In-Reply-To: <201411170351.sAH3p5KB008027@snookles.snookles.com> References: <201411170351.sAH3p5KB008027@snookles.snookles.com> Message-ID: <5469EFB3.8070307@mgeb.org> On 2014-11-17 04:51, Scott Lystig Fritchie wrote: > Ulf, I discovered that we have been in good company. According to the > speaker at this RICON talk a few weeks ago, both Yahoo and Facebook > discovered the exact same things. :-| > > https://www.youtube.com/watch?v=8cYp52oeDx4&list=PL9Jh2HsAWHxLco7V1SjU9hUzP53CBZOYO&index=20 And they created rocksdb based on levelDB which is fit for such server workloads. http://rocksdb.org/ Michael Gebetsroither From ingela.andin@REDACTED Mon Nov 17 14:19:45 2014 From: ingela.andin@REDACTED (Ingela Andin) Date: Mon, 17 Nov 2014 14:19:45 +0100 Subject: [erlang-questions] aes gcm 128 In-Reply-To: References: Message-ID: Hi! 2014-11-17 9:50 GMT+01:00 Benoit Chesneau : > Is there any chance that this patch can fall in 17.4 ? : > > > https://github.com/erlang/otp/commit/7603a4029514a644c8323028b06acdc33e45b286 > > We have not received this as a pull-request so at the moment it will not be considered. If it is submitted and tested it could be considered for 17.4. > > Or is this only planned for the 18 ? > - benoit > > Mainly we want new features for 18, but considering the impact and size of a patch it can be somewhat flexible. Regards Ingela Erlang/OTP Team - Ericsson AB > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bchesneau@REDACTED Mon Nov 17 14:22:44 2014 From: bchesneau@REDACTED (Benoit Chesneau) Date: Mon, 17 Nov 2014 14:22:44 +0100 Subject: [erlang-questions] aes gcm 128 In-Reply-To: References: Message-ID: hrm, You mean : https://github.com/erlang/otp/pull/372 ? It says it have been merged, also the linked code is from the main repo. Did I miss something? - benoit On Mon, Nov 17, 2014 at 2:19 PM, Ingela Andin wrote: > Hi! > > > 2014-11-17 9:50 GMT+01:00 Benoit Chesneau : > >> Is there any chance that this patch can fall in 17.4 ? : >> >> >> https://github.com/erlang/otp/commit/7603a4029514a644c8323028b06acdc33e45b286 >> >> > We have not received this as a pull-request so at the moment it will not > be considered. > If it is submitted and tested it could be considered for 17.4. > > >> >> Or is this only planned for the 18 ? >> - benoit >> >> > Mainly we want new features for 18, but considering the impact and size of > a patch it can be somewhat flexible. > > > Regards Ingela Erlang/OTP Team - Ericsson AB > > > >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ingela.andin@REDACTED Mon Nov 17 15:36:05 2014 From: ingela.andin@REDACTED (Ingela Andin) Date: Mon, 17 Nov 2014 15:36:05 +0100 Subject: [erlang-questions] aes gcm 128 In-Reply-To: References: Message-ID: Hi! 2014-11-17 14:22 GMT+01:00 Benoit Chesneau : > hrm, > > You mean : > > https://github.com/erlang/otp/pull/372 ? > > It says it have been merged, also the linked code is from the main repo. > Did I miss something? > > That is not the same link that you posted last time, that only seems to point to the last commit of that pull-request. Pull-request 372 will not be back ported for 17.4. If you need the changes earlier why not try to cherry-pick them? Regards Ingela Erlang/OTP Team - Ericsson AB > - benoit > > On Mon, Nov 17, 2014 at 2:19 PM, Ingela Andin > wrote: > >> Hi! >> >> >> 2014-11-17 9:50 GMT+01:00 Benoit Chesneau : >> >>> Is there any chance that this patch can fall in 17.4 ? : >>> >>> >>> https://github.com/erlang/otp/commit/7603a4029514a644c8323028b06acdc33e45b286 >>> >>> >> We have not received this as a pull-request so at the moment it will not >> be considered. >> If it is submitted and tested it could be considered for 17.4. >> >> >>> >>> Or is this only planned for the 18 ? >>> - benoit >>> >>> >> Mainly we want new features for 18, but considering the impact and size >> of a patch it can be somewhat flexible. >> >> >> Regards Ingela Erlang/OTP Team - Ericsson AB >> >> >> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yelt@REDACTED Mon Nov 17 15:13:47 2014 From: yelt@REDACTED (._) Date: Mon, 17 Nov 2014 22:13:47 +0800 Subject: [erlang-questions] =?gb2312?b?tPC4tDogIFF1ZXN0aW9ucyBhYm91dCBo?= =?gb2312?b?b3cgdG8gbGluayBlcmxfaW50ZXJmYWNlLmxpYqO/?= In-Reply-To: <27DBF08E-D8ED-4E37-9A8B-E5688011FAEC@gmail.com> References: <27DBF08E-D8ED-4E37-9A8B-E5688011FAEC@gmail.com> Message-ID: Hi Serge Jurecko?? Thank you very much, but sadly it does not work for me. Let me put it simpler. Code goes like below: #include "erl_interface.h" #include "ei.h" typedef unsigned char byte; int main() { erl_init(NULL, 0); return 1; } Using Cygwin or VS2010??compile command looks like below: gcc ei.c -o extprg -I/cygdrive/c/"Program Files"/erl6.2/usr/include -L/cygdrive/c/"Program Files"/erl6.2/lib/erl_interface-3.7.18/lib -lerl_interface ?Clei error information is: /tmp/ccrvLdwc.o:ei.c:(.text+0x1e): undefined reference to `_erl_init' I find someone has the same problem at stackoverflow: http://stackoverflow.com/questions/14185290/erl-interface-linker-error Someone gives a solution in that thread, but it does not work for me too... ;( ??????: Sergej Jurecko [mailto:sergej.jurecko@REDACTED] ????????: 2014??11??16?? 17:09 ??????: ._ ????: erlang-questions@REDACTED ????: Re: [erlang-questions] Questions about how to link erl_interface.lib?? Best to use rebar. Add erts bin folder to system path, compile rebar on windows (copy rebar and rebar.cmd files to your project), create rebar.config and compile from visual studio console with rebar. Sergej On 16 Nov 2014, at 03:35, ._ wrote: _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From rvirding@REDACTED Mon Nov 17 16:53:15 2014 From: rvirding@REDACTED (Robert Virding) Date: Mon, 17 Nov 2014 16:53:15 +0100 Subject: [erlang-questions] Dialyzer type/spec info question Message-ID: >From where does dialyzer get the user added type and spec info? From the AST, or from Core erlang which contains the same type/spec data? And why? Can I control it? An extra question: why doesn't dialyzer allow me to mix input from both .erl and .beam files? Or does it and I have missed that? Robert -------------- next part -------------- An HTML attachment was scrubbed... URL: From aronisstav@REDACTED Mon Nov 17 17:03:39 2014 From: aronisstav@REDACTED (Stavros Aronis) Date: Mon, 17 Nov 2014 17:03:39 +0100 Subject: [erlang-questions] Dialyzer type/spec info question In-Reply-To: References: Message-ID: Hi Robert, Dialyzer gets this info from beam_lib:chunks(File, [abstract_code]) which corresponds to the AST and is included in beam files if +debug_info is used while compiling. To my knowledge there is no reason to not let sources be combined (except perhaps when building a PLT), but the implementation seems to require uniformity. Relevant functions: dialyzer_utils:get_abstract_code_from_{src,beam}/{1,2}, get_spec_info/3, get_record_and_type_info/1. Regards, Stavros On Mon, Nov 17, 2014 at 4:53 PM, Robert Virding wrote: > From where does dialyzer get the user added type and spec info? From the > AST, or from Core erlang which contains the same type/spec data? And why? > Can I control it? > > An extra question: why doesn't dialyzer allow me to mix input from both > .erl and .beam files? Or does it and I have missed that? > > Robert > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rvirding@REDACTED Mon Nov 17 17:11:13 2014 From: rvirding@REDACTED (Robert Virding) Date: Mon, 17 Nov 2014 17:11:13 +0100 Subject: [erlang-questions] Dialyzer type/spec info question In-Reply-To: References: Message-ID: >From where does it get it if I have .erl files as input? From the AST as well? Do you know if there is any difference in the data itself between the AST and core? I am guessing not but want to check. If there is no difference then the info could be gotten from Core. This would make it easier to use dialyzer together with LFE. Robert On 17 November 2014 17:03, Stavros Aronis wrote: > Hi Robert, > > Dialyzer gets this info from beam_lib:chunks(File, [abstract_code]) which > corresponds to the AST and is included in beam files if +debug_info is used > while compiling. > > To my knowledge there is no reason to not let sources be combined (except > perhaps when building a PLT), but the implementation seems to require > uniformity. > > Relevant functions: > dialyzer_utils:get_abstract_code_from_{src,beam}/{1,2}, get_spec_info/3, > get_record_and_type_info/1. > > Regards, > > Stavros > > On Mon, Nov 17, 2014 at 4:53 PM, Robert Virding > wrote: > >> From where does dialyzer get the user added type and spec info? From the >> AST, or from Core erlang which contains the same type/spec data? And why? >> Can I control it? >> >> An extra question: why doesn't dialyzer allow me to mix input from both >> .erl and .beam files? Or does it and I have missed that? >> >> Robert >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jose.valim@REDACTED Mon Nov 17 17:22:40 2014 From: jose.valim@REDACTED (=?UTF-8?Q?Jos=C3=A9_Valim?=) Date: Mon, 17 Nov 2014 17:22:40 +0100 Subject: [erlang-questions] Dialyzer type/spec info question In-Reply-To: References: Message-ID: We have expressed interest too in reading typespecs from core. If I remember correctly though, the OTP team did not reach a consensus about storing Core AST in beam files. If there is a consensus and a name for a chunk where we can store Core AST in beam files, I would gladly sent a patch to dialyzer and compiler (so we have a pass that stores Core AST). On Monday, November 17, 2014, Robert Virding wrote: > From where does it get it if I have .erl files as input? From the AST as > well? Do you know if there is any difference in the data itself between the > AST and core? I am guessing not but want to check. If there is no > difference then the info could be gotten from Core. This would make it > easier to use dialyzer together with LFE. > > Robert > > > On 17 November 2014 17:03, Stavros Aronis > wrote: > >> Hi Robert, >> >> Dialyzer gets this info from beam_lib:chunks(File, [abstract_code]) which >> corresponds to the AST and is included in beam files if +debug_info is used >> while compiling. >> >> To my knowledge there is no reason to not let sources be combined (except >> perhaps when building a PLT), but the implementation seems to require >> uniformity. >> >> Relevant functions: >> dialyzer_utils:get_abstract_code_from_{src,beam}/{1,2}, get_spec_info/3, >> get_record_and_type_info/1. >> >> Regards, >> >> Stavros >> >> On Mon, Nov 17, 2014 at 4:53 PM, Robert Virding > > wrote: >> >>> From where does dialyzer get the user added type and spec info? From the >>> AST, or from Core erlang which contains the same type/spec data? And why? >>> Can I control it? >>> >>> An extra question: why doesn't dialyzer allow me to mix input from both >>> .erl and .beam files? Or does it and I have missed that? >>> >>> Robert >>> >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >> > -- *Jos? Valim* www.plataformatec.com.br Skype: jv.ptec Founder and Lead Developer -------------- next part -------------- An HTML attachment was scrubbed... URL: From aronisstav@REDACTED Mon Nov 17 17:32:34 2014 From: aronisstav@REDACTED (Stavros Aronis) Date: Mon, 17 Nov 2014 17:32:34 +0100 Subject: [erlang-questions] Dialyzer type/spec info question In-Reply-To: References: Message-ID: For .erl files, dialyzer calls the compiler with 'to_pp', which stops the compilation before code is converted to Core, and reads attributes from there. Judging from v3_core, line 170 the two representations should be compatible. /Stavros On Mon, Nov 17, 2014 at 5:11 PM, Robert Virding wrote: > From where does it get it if I have .erl files as input? From the AST as > well? Do you know if there is any difference in the data itself between the > AST and core? I am guessing not but want to check. If there is no > difference then the info could be gotten from Core. This would make it > easier to use dialyzer together with LFE. > > Robert > > > On 17 November 2014 17:03, Stavros Aronis wrote: > >> Hi Robert, >> >> Dialyzer gets this info from beam_lib:chunks(File, [abstract_code]) which >> corresponds to the AST and is included in beam files if +debug_info is used >> while compiling. >> >> To my knowledge there is no reason to not let sources be combined (except >> perhaps when building a PLT), but the implementation seems to require >> uniformity. >> >> Relevant functions: >> dialyzer_utils:get_abstract_code_from_{src,beam}/{1,2}, get_spec_info/3, >> get_record_and_type_info/1. >> >> Regards, >> >> Stavros >> >> On Mon, Nov 17, 2014 at 4:53 PM, Robert Virding >> wrote: >> >>> From where does dialyzer get the user added type and spec info? From the >>> AST, or from Core erlang which contains the same type/spec data? And why? >>> Can I control it? >>> >>> An extra question: why doesn't dialyzer allow me to mix input from both >>> .erl and .beam files? Or does it and I have missed that? >>> >>> Robert >>> >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rvirding@REDACTED Mon Nov 17 17:39:52 2014 From: rvirding@REDACTED (Robert Virding) Date: Mon, 17 Nov 2014 17:39:52 +0100 Subject: [erlang-questions] Dialyzer type/spec info question In-Reply-To: References: Message-ID: Ah, so then in principle I can make my own specialised versions of the dialyzer load modules which takes the core which the LFE compiler generates and extracts the core and the type/spec data dialyzer needs. This without any Erlang AST which LFE never generates anyway. Now we are getting somewhere useful. With this we don't really need to be able to store core in the .beam files if we can accept using source files as input. Robert On 17 November 2014 17:32, Stavros Aronis wrote: > For .erl files, dialyzer calls the compiler with 'to_pp', which stops the > compilation before code is converted to Core, and reads attributes from > there. > > Judging from v3_core, line 170 the two representations should be > compatible. > > /Stavros > > On Mon, Nov 17, 2014 at 5:11 PM, Robert Virding > wrote: > >> From where does it get it if I have .erl files as input? From the AST as >> well? Do you know if there is any difference in the data itself between the >> AST and core? I am guessing not but want to check. If there is no >> difference then the info could be gotten from Core. This would make it >> easier to use dialyzer together with LFE. >> >> Robert >> >> >> On 17 November 2014 17:03, Stavros Aronis wrote: >> >>> Hi Robert, >>> >>> Dialyzer gets this info from beam_lib:chunks(File, [abstract_code]) >>> which corresponds to the AST and is included in beam files if +debug_info >>> is used while compiling. >>> >>> To my knowledge there is no reason to not let sources be combined >>> (except perhaps when building a PLT), but the implementation seems to >>> require uniformity. >>> >>> Relevant functions: >>> dialyzer_utils:get_abstract_code_from_{src,beam}/{1,2}, get_spec_info/3, >>> get_record_and_type_info/1. >>> >>> Regards, >>> >>> Stavros >>> >>> On Mon, Nov 17, 2014 at 4:53 PM, Robert Virding >>> wrote: >>> >>>> From where does dialyzer get the user added type and spec info? From >>>> the AST, or from Core erlang which contains the same type/spec data? And >>>> why? Can I control it? >>>> >>>> An extra question: why doesn't dialyzer allow me to mix input from both >>>> .erl and .beam files? Or does it and I have missed that? >>>> >>>> Robert >>>> >>>> >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jose.valim@REDACTED Mon Nov 17 17:49:40 2014 From: jose.valim@REDACTED (=?UTF-8?Q?Jos=C3=A9_Valim?=) Date: Mon, 17 Nov 2014 17:49:40 +0100 Subject: [erlang-questions] Dialyzer type/spec info question In-Reply-To: References: Message-ID: Robert, what would a specialized version of dialyzer entail? A fork? A specific front-end? Curious. :) On Monday, November 17, 2014, Robert Virding wrote: > Ah, so then in principle I can make my own specialised versions of the > dialyzer load modules which takes the core which the LFE compiler generates > and extracts the core and the type/spec data dialyzer needs. This without > any Erlang AST which LFE never generates anyway. > > Now we are getting somewhere useful. With this we don't really need to be > able to store core in the .beam files if we can accept using source files > as input. > > Robert > > > On 17 November 2014 17:32, Stavros Aronis > wrote: > >> For .erl files, dialyzer calls the compiler with 'to_pp', which stops the >> compilation before code is converted to Core, and reads attributes from >> there. >> >> Judging from v3_core, line 170 the two representations should be >> compatible. >> >> /Stavros >> >> On Mon, Nov 17, 2014 at 5:11 PM, Robert Virding > > wrote: >> >>> From where does it get it if I have .erl files as input? From the AST as >>> well? Do you know if there is any difference in the data itself between the >>> AST and core? I am guessing not but want to check. If there is no >>> difference then the info could be gotten from Core. This would make it >>> easier to use dialyzer together with LFE. >>> >>> Robert >>> >>> >>> On 17 November 2014 17:03, Stavros Aronis >> > wrote: >>> >>>> Hi Robert, >>>> >>>> Dialyzer gets this info from beam_lib:chunks(File, [abstract_code]) >>>> which corresponds to the AST and is included in beam files if +debug_info >>>> is used while compiling. >>>> >>>> To my knowledge there is no reason to not let sources be combined >>>> (except perhaps when building a PLT), but the implementation seems to >>>> require uniformity. >>>> >>>> Relevant functions: >>>> dialyzer_utils:get_abstract_code_from_{src,beam}/{1,2}, get_spec_info/3, >>>> get_record_and_type_info/1. >>>> >>>> Regards, >>>> >>>> Stavros >>>> >>>> On Mon, Nov 17, 2014 at 4:53 PM, Robert Virding >>> > wrote: >>>> >>>>> From where does dialyzer get the user added type and spec info? From >>>>> the AST, or from Core erlang which contains the same type/spec data? And >>>>> why? Can I control it? >>>>> >>>>> An extra question: why doesn't dialyzer allow me to mix input from >>>>> both .erl and .beam files? Or does it and I have missed that? >>>>> >>>>> Robert >>>>> >>>>> >>>>> _______________________________________________ >>>>> erlang-questions mailing list >>>>> erlang-questions@REDACTED >>>>> >>>>> http://erlang.org/mailman/listinfo/erlang-questions >>>>> >>>>> >>>> >>> >> > -- *Jos? Valim* www.plataformatec.com.br Skype: jv.ptec Founder and Lead Developer -------------- next part -------------- An HTML attachment was scrubbed... URL: From rvirding@REDACTED Mon Nov 17 18:19:12 2014 From: rvirding@REDACTED (Robert Virding) Date: Mon, 17 Nov 2014 18:19:12 +0100 Subject: [erlang-questions] Dialyzer type/spec info question In-Reply-To: References: Message-ID: Something much naughtier and more basic. - Make LFE local copies of the relevant dialyzer modules which have been hacked to be able to handle LFE files as input (with a new --lfe option) - Run dialyzer using -pa to push the local module directory first so as to use these special modules instead of the standard ones This will make it possible to use dialyzer on LFE files and other .erl or .beam files as well. Maybe with a little more hacking fix have multiple file types. Of course if dialyzer was to be opened up so that it would use the filename extension to pick the right versions of the functions Stavros mentioned then this would be much simpler. Wink, wink, nudge nudge. :-) This is my plan anyway. Robert P.S. Having LFE generate erlang AST has never been an option as it is very close to core and it would take a lot of effort to go back to erlang which would then be undone by the erlang compiler. On 17 November 2014 17:49, Jos? Valim wrote: > Robert, what would a specialized version of dialyzer entail? A fork? A > specific front-end? Curious. :) > > On Monday, November 17, 2014, Robert Virding wrote: > >> Ah, so then in principle I can make my own specialised versions of the >> dialyzer load modules which takes the core which the LFE compiler generates >> and extracts the core and the type/spec data dialyzer needs. This without >> any Erlang AST which LFE never generates anyway. >> >> Now we are getting somewhere useful. With this we don't really need to be >> able to store core in the .beam files if we can accept using source files >> as input. >> >> Robert >> >> >> On 17 November 2014 17:32, Stavros Aronis wrote: >> >>> For .erl files, dialyzer calls the compiler with 'to_pp', which stops >>> the compilation before code is converted to Core, and reads attributes from >>> there. >>> >>> Judging from v3_core, line 170 the two representations should be >>> compatible. >>> >>> /Stavros >>> >>> On Mon, Nov 17, 2014 at 5:11 PM, Robert Virding >>> wrote: >>> >>>> From where does it get it if I have .erl files as input? From the AST >>>> as well? Do you know if there is any difference in the data itself between >>>> the AST and core? I am guessing not but want to check. If there is no >>>> difference then the info could be gotten from Core. This would make it >>>> easier to use dialyzer together with LFE. >>>> >>>> Robert >>>> >>>> >>>> On 17 November 2014 17:03, Stavros Aronis wrote: >>>> >>>>> Hi Robert, >>>>> >>>>> Dialyzer gets this info from beam_lib:chunks(File, [abstract_code]) >>>>> which corresponds to the AST and is included in beam files if +debug_info >>>>> is used while compiling. >>>>> >>>>> To my knowledge there is no reason to not let sources be combined >>>>> (except perhaps when building a PLT), but the implementation seems to >>>>> require uniformity. >>>>> >>>>> Relevant functions: >>>>> dialyzer_utils:get_abstract_code_from_{src,beam}/{1,2}, get_spec_info/3, >>>>> get_record_and_type_info/1. >>>>> >>>>> Regards, >>>>> >>>>> Stavros >>>>> >>>>> On Mon, Nov 17, 2014 at 4:53 PM, Robert Virding >>>>> wrote: >>>>> >>>>>> From where does dialyzer get the user added type and spec info? From >>>>>> the AST, or from Core erlang which contains the same type/spec data? And >>>>>> why? Can I control it? >>>>>> >>>>>> An extra question: why doesn't dialyzer allow me to mix input from >>>>>> both .erl and .beam files? Or does it and I have missed that? >>>>>> >>>>>> Robert >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> erlang-questions mailing list >>>>>> erlang-questions@REDACTED >>>>>> http://erlang.org/mailman/listinfo/erlang-questions >>>>>> >>>>>> >>>>> >>>> >>> >> > > -- > > > *Jos? Valim* > www.plataformatec.com.br > Skype: jv.ptec > Founder and Lead Developer > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rvirding@REDACTED Mon Nov 17 18:26:01 2014 From: rvirding@REDACTED (Robert Virding) Date: Mon, 17 Nov 2014 18:26:01 +0100 Subject: [erlang-questions] Dialyzer type/spec info question In-Reply-To: References: Message-ID: The reason I would like to be able to handle multiple file types is to simplify checking systems with parts which are written in different languages. Robert On 17 November 2014 18:19, Robert Virding wrote: > Something much naughtier and more basic. > > - Make LFE local copies of the relevant dialyzer modules which have been > hacked to be able to handle LFE files as input (with a new --lfe option) > - Run dialyzer using -pa to push the local module directory first so as to > use these special modules instead of the standard ones > > This will make it possible to use dialyzer on LFE files and other .erl or > .beam files as well. Maybe with a little more hacking fix have multiple > file types. Of course if dialyzer was to be opened up so that it would use > the filename extension to pick the right versions of the functions Stavros > mentioned then this would be much simpler. Wink, wink, nudge nudge. :-) > > This is my plan anyway. > > Robert > > P.S. Having LFE generate erlang AST has never been an option as it is very > close to core and it would take a lot of effort to go back to erlang which > would then be undone by the erlang compiler. > > > On 17 November 2014 17:49, Jos? Valim > wrote: > >> Robert, what would a specialized version of dialyzer entail? A fork? A >> specific front-end? Curious. :) >> >> On Monday, November 17, 2014, Robert Virding wrote: >> >>> Ah, so then in principle I can make my own specialised versions of the >>> dialyzer load modules which takes the core which the LFE compiler generates >>> and extracts the core and the type/spec data dialyzer needs. This without >>> any Erlang AST which LFE never generates anyway. >>> >>> Now we are getting somewhere useful. With this we don't really need to >>> be able to store core in the .beam files if we can accept using source >>> files as input. >>> >>> Robert >>> >>> >>> On 17 November 2014 17:32, Stavros Aronis wrote: >>> >>>> For .erl files, dialyzer calls the compiler with 'to_pp', which stops >>>> the compilation before code is converted to Core, and reads attributes from >>>> there. >>>> >>>> Judging from v3_core, line 170 the two representations should be >>>> compatible. >>>> >>>> /Stavros >>>> >>>> On Mon, Nov 17, 2014 at 5:11 PM, Robert Virding >>>> wrote: >>>> >>>>> From where does it get it if I have .erl files as input? From the AST >>>>> as well? Do you know if there is any difference in the data itself between >>>>> the AST and core? I am guessing not but want to check. If there is no >>>>> difference then the info could be gotten from Core. This would make it >>>>> easier to use dialyzer together with LFE. >>>>> >>>>> Robert >>>>> >>>>> >>>>> On 17 November 2014 17:03, Stavros Aronis >>>>> wrote: >>>>> >>>>>> Hi Robert, >>>>>> >>>>>> Dialyzer gets this info from beam_lib:chunks(File, [abstract_code]) >>>>>> which corresponds to the AST and is included in beam files if +debug_info >>>>>> is used while compiling. >>>>>> >>>>>> To my knowledge there is no reason to not let sources be combined >>>>>> (except perhaps when building a PLT), but the implementation seems to >>>>>> require uniformity. >>>>>> >>>>>> Relevant functions: >>>>>> dialyzer_utils:get_abstract_code_from_{src,beam}/{1,2}, get_spec_info/3, >>>>>> get_record_and_type_info/1. >>>>>> >>>>>> Regards, >>>>>> >>>>>> Stavros >>>>>> >>>>>> On Mon, Nov 17, 2014 at 4:53 PM, Robert Virding >>>>>> wrote: >>>>>> >>>>>>> From where does dialyzer get the user added type and spec info? From >>>>>>> the AST, or from Core erlang which contains the same type/spec data? And >>>>>>> why? Can I control it? >>>>>>> >>>>>>> An extra question: why doesn't dialyzer allow me to mix input from >>>>>>> both .erl and .beam files? Or does it and I have missed that? >>>>>>> >>>>>>> Robert >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> erlang-questions mailing list >>>>>>> erlang-questions@REDACTED >>>>>>> http://erlang.org/mailman/listinfo/erlang-questions >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >> -- >> >> >> *Jos? Valim* >> www.plataformatec.com.br >> Skype: jv.ptec >> Founder and Lead Developer >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kunthar@REDACTED Mon Nov 17 22:03:53 2014 From: kunthar@REDACTED (Gokhan Boranalp) Date: Mon, 17 Nov 2014 23:03:53 +0200 Subject: [erlang-questions] Does anyone have experience with LETS? In-Reply-To: <5469EFB3.8070307@mgeb.org> References: <201411170351.sAH3p5KB008027@snookles.snookles.com> <5469EFB3.8070307@mgeb.org> Message-ID: RocksDB is an embeddable persistent key-value store for fast storage. RocksDB can also be the foundation for a client-server database but our current focus is on embedded workloads. *What is RocksDB suitable for?* RocksDB can be used by applications that need low latency database accesses. A user-facing application that stores the viewing history and state of users of a website can potentially store this content on RocksDB. A spam detection application that needs fast access to big data sets can use RocksDB. A graph-search query that needs to scan a data set in realtime can use RocksDB. RocksDB can be used to cache data from Hadoop, thereby allowing applications to query Hadoop data in realtime. A message-queue that supports a high number of inserts and deletes can use RocksDB. On Mon, Nov 17, 2014 at 2:53 PM, Michael Gebetsroither wrote: > On 2014-11-17 04:51, Scott Lystig Fritchie wrote: >> >> Ulf, I discovered that we have been in good company. According to the >> speaker at this RICON talk a few weeks ago, both Yahoo and Facebook >> discovered the exact same things. :-| >> >> >> https://www.youtube.com/watch?v=8cYp52oeDx4&list=PL9Jh2HsAWHxLco7V1SjU9hUzP53CBZOYO&index=20 > > > And they created rocksdb based on levelDB which is fit for such server > workloads. > > http://rocksdb.org/ > > > Michael Gebetsroither > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- BR, \|/ Kunthar From arif@REDACTED Mon Nov 17 22:38:06 2014 From: arif@REDACTED (arif@REDACTED) Date: Mon, 17 Nov 2014 22:38:06 +0100 Subject: [erlang-questions] can flush the shell but can't receive from it Message-ID: <0a30511b7227afecb8c84138df45a840@riseup.net> Hi, I have the simplest of applications that consists of just the skeleton gen_server, registered locally as "server", with one callback implemented: handle_info({From, A, B}, State) -> From ! {self(), A + B}, {noreply, State}; i.e. a simple adder The supervisor is the skeleton one_for_one supervisor. The application, app, the skeleton application. Here is what I do: 7> application:start(app). ok 8> server ! {self(), 1, 2}. {<0.32.0>,1,2} 9> receive X -> X end. {<0.62.0>,3} 10> server ! {self(), a, b}. =ERROR REPORT==== 17-Nov-2014::22:18:42 === ** Generic server server terminating ** Last message in was {<0.32.0>,a,b} ** When Server state == {state} ** Reason for termination == ** {badarith,[{server,handle_info,2,[{file,"server.erl"},{line,98}]}, {gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,599}]}, {proc_lib,init_p_do_apply,3, [{file,"proc_lib.erl"},{line,239}]}]} {<0.32.0>,a,b} 11> server ! {self(), 3, 4}. {<0.32.0>,3,4} 12> receive X -> X end. and here the shell blocks. The message ({<0.67.0>,7}) is in the shell's mailbox alright. I can see it by running observer. If, instead of receive X -> X end, I do a flush(), the message _is_ flushed. It is reproducible on R 17, R15B01, on linux and on windows. What am I doing wrong? Thanks and best regards Arif From imantc@REDACTED Mon Nov 17 22:44:59 2014 From: imantc@REDACTED (Imants Cekusins) Date: Mon, 17 Nov 2014 22:44:59 +0100 Subject: [erlang-questions] can flush the shell but can't receive from it In-Reply-To: <0a30511b7227afecb8c84138df45a840@riseup.net> References: <0a30511b7227afecb8c84138df45a840@riseup.net> Message-ID: Why not try to send / receive { A, B } instead of A, B ? On 17 Nov 2014 22:38, wrote: > Hi, > > I have the simplest of applications that consists of just the skeleton > gen_server, registered locally as "server", with one callback implemented: > > handle_info({From, A, B}, State) -> > From ! {self(), A + B}, > {noreply, State}; > > i.e. a simple adder > > The supervisor is the skeleton one_for_one supervisor. The application, > app, the skeleton application. > > Here is what I do: > > 7> application:start(app). > ok > 8> server ! {self(), 1, 2}. > {<0.32.0>,1,2} > 9> receive X -> X end. > {<0.62.0>,3} > 10> server ! {self(), a, b}. > > =ERROR REPORT==== 17-Nov-2014::22:18:42 === > ** Generic server server terminating > ** Last message in was {<0.32.0>,a,b} > ** When Server state == {state} > ** Reason for termination == > ** {badarith,[{server,handle_info,2,[{file,"server.erl"},{line,98}]}, > {gen_server,handle_msg,5,[{file,"gen_server.erl"},{line, > 599}]}, > {proc_lib,init_p_do_apply,3, > [{file,"proc_lib.erl"},{line,239}]}]} > {<0.32.0>,a,b} > 11> server ! {self(), 3, 4}. > {<0.32.0>,3,4} > 12> receive X -> X end. > > and here the shell blocks. > The message ({<0.67.0>,7}) is in the shell's mailbox alright. I can see it > by running observer. > > If, instead of receive X -> X end, I do a flush(), the message _is_ > flushed. > > It is reproducible on R 17, R15B01, on linux and on windows. > > What am I doing wrong? > > Thanks and best regards > Arif > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinoski@REDACTED Mon Nov 17 22:46:04 2014 From: vinoski@REDACTED (Steve Vinoski) Date: Mon, 17 Nov 2014 16:46:04 -0500 Subject: [erlang-questions] can flush the shell but can't receive from it In-Reply-To: <0a30511b7227afecb8c84138df45a840@riseup.net> References: <0a30511b7227afecb8c84138df45a840@riseup.net> Message-ID: On Mon, Nov 17, 2014 at 4:38 PM, wrote: > Hi, > > I have the simplest of applications that consists of just the skeleton > gen_server, registered locally as "server", with one callback implemented: > > handle_info({From, A, B}, State) -> > From ! {self(), A + B}, > {noreply, State}; > > i.e. a simple adder > > The supervisor is the skeleton one_for_one supervisor. The application, > app, the skeleton application. > > Here is what I do: > > 7> application:start(app). > ok > 8> server ! {self(), 1, 2}. > {<0.32.0>,1,2} > 9> receive X -> X end. > {<0.62.0>,3} > Here you bind X to the tuple value consisting of the server pid and the value 3. > 10> server ! {self(), a, b}. > > =ERROR REPORT==== 17-Nov-2014::22:18:42 === > ** Generic server server terminating > ** Last message in was {<0.32.0>,a,b} > ** When Server state == {state} > ** Reason for termination == > ** {badarith,[{server,handle_info,2,[{file,"server.erl"},{line,98}]}, > {gen_server,handle_msg,5,[{file,"gen_server.erl"},{line, > 599}]}, > {proc_lib,init_p_do_apply,3, > [{file,"proc_lib.erl"},{line,239}]}]} > {<0.32.0>,a,b} > 11> server ! {self(), 3, 4}. > {<0.32.0>,3,4} > 12> receive X -> X end. > > and here the shell blocks. > It blocks because you're asking to receive the value of X you bound earlier, which doesn't match what's in the mailbox. --steve -------------- next part -------------- An HTML attachment was scrubbed... URL: From imantc@REDACTED Mon Nov 17 22:55:02 2014 From: imantc@REDACTED (Imants Cekusins) Date: Mon, 17 Nov 2014 22:55:02 +0100 Subject: [erlang-questions] can flush the shell but can't receive from it In-Reply-To: References: <0a30511b7227afecb8c84138df45a840@riseup.net> Message-ID: Btw, a, b are atoms which can't be added, no? On 17 Nov 2014 22:46, "Steve Vinoski" wrote: > > > On Mon, Nov 17, 2014 at 4:38 PM, wrote: > >> Hi, >> >> I have the simplest of applications that consists of just the skeleton >> gen_server, registered locally as "server", with one callback implemented: >> >> handle_info({From, A, B}, State) -> >> From ! {self(), A + B}, >> {noreply, State}; >> >> i.e. a simple adder >> >> The supervisor is the skeleton one_for_one supervisor. The application, >> app, the skeleton application. >> >> Here is what I do: >> >> 7> application:start(app). >> ok >> 8> server ! {self(), 1, 2}. >> {<0.32.0>,1,2} >> 9> receive X -> X end. >> {<0.62.0>,3} >> > > Here you bind X to the tuple value consisting of the server pid and the > value 3. > > >> 10> server ! {self(), a, b}. >> >> =ERROR REPORT==== 17-Nov-2014::22:18:42 === >> ** Generic server server terminating >> ** Last message in was {<0.32.0>,a,b} >> ** When Server state == {state} >> ** Reason for termination == >> ** {badarith,[{server,handle_info,2,[{file,"server.erl"},{line,98}]}, >> {gen_server,handle_msg,5,[{file,"gen_server.erl"},{line, >> 599}]}, >> {proc_lib,init_p_do_apply,3, >> [{file,"proc_lib.erl"},{line,239}]}]} >> {<0.32.0>,a,b} >> 11> server ! {self(), 3, 4}. >> {<0.32.0>,3,4} >> 12> receive X -> X end. >> >> and here the shell blocks. >> > > It blocks because you're asking to receive the value of X you bound > earlier, which doesn't match what's in the mailbox. > > --steve > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric.pailleau@REDACTED Mon Nov 17 22:58:28 2014 From: eric.pailleau@REDACTED (Eric Pailleau) Date: Mon, 17 Nov 2014 22:58:28 +0100 Subject: [erlang-questions] can flush the shell but can't receive from it Message-ID: ?It blocks because you're asking to receive the value of X you bound earlier, which doesn't match what's in the mailbox.? BTW, it is a good example that could go into Erlang.org , chapter ?common caveat? ... ? Envoy? depuis mon mobile ? Eric Steve Vinoski a ?crit?: >On Mon, Nov 17, 2014 at 4:38 PM, wrote: > >> Hi, >> >> I have the simplest of applications that consists of just the skeleton >> gen_server, registered locally as "server", with one callback implemented: >> >> handle_info({From, A, B}, State) -> >> From ! {self(), A + B}, >> {noreply, State}; >> >> i.e. a simple adder >> >> The supervisor is the skeleton one_for_one supervisor. The application, >> app, the skeleton application. >> >> Here is what I do: >> >> 7> application:start(app). >> ok >> 8> server ! {self(), 1, 2}. >> {<0.32.0>,1,2} >> 9> receive X -> X end. >> {<0.62.0>,3} >> > >Here you bind X to the tuple value consisting of the server pid and the >value 3. > > >> 10> server ! {self(), a, b}. >> >> =ERROR REPORT==== 17-Nov-2014::22:18:42 === >> ** Generic server server terminating >> ** Last message in was {<0.32.0>,a,b} >> ** When Server state == {state} >> ** Reason for termination == >> ** {badarith,[{server,handle_info,2,[{file,"server.erl"},{line,98}]}, >> {gen_server,handle_msg,5,[{file,"gen_server.erl"},{line, >> 599}]}, >> {proc_lib,init_p_do_apply,3, >> [{file,"proc_lib.erl"},{line,239}]}]} >> {<0.32.0>,a,b} >> 11> server ! {self(), 3, 4}. >> {<0.32.0>,3,4} >> 12> receive X -> X end. >> >> and here the shell blocks. >> > >It blocks because you're asking to receive the value of X you bound >earlier, which doesn't match what's in the mailbox. > >--steve > >_______________________________________________ >erlang-questions mailing list >erlang-questions@REDACTED >http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From tty.erlang@REDACTED Mon Nov 17 23:59:11 2014 From: tty.erlang@REDACTED (T Ty) Date: Mon, 17 Nov 2014 22:59:11 +0000 Subject: [erlang-questions] Does anyone have experience with LETS? In-Reply-To: References: <201411170351.sAH3p5KB008027@snookles.snookles.com> <5469EFB3.8070307@mgeb.org> Message-ID: I have a very basic Erlang binding for it. https://github.com/ttyerlsol/rockse On Mon, Nov 17, 2014 at 9:03 PM, Gokhan Boranalp wrote: > RocksDB is an embeddable persistent key-value store for fast storage. > RocksDB can also be the foundation for a client-server database but > our current focus is on embedded workloads. > > *What is RocksDB suitable for?* > > RocksDB can be used by applications that need low latency database > accesses. A user-facing application that stores the viewing history > and state of users of a website can potentially store this content on > RocksDB. > A spam detection application that needs fast access to big data sets > can use RocksDB. A graph-search query that needs to scan a data set in > realtime can use RocksDB. RocksDB can be used to cache data from > Hadoop, thereby allowing applications to query Hadoop data in > realtime. A message-queue that supports a high number of inserts and > deletes can use RocksDB. > > On Mon, Nov 17, 2014 at 2:53 PM, Michael Gebetsroither > wrote: > > On 2014-11-17 04:51, Scott Lystig Fritchie wrote: > >> > >> Ulf, I discovered that we have been in good company. According to the > >> speaker at this RICON talk a few weeks ago, both Yahoo and Facebook > >> discovered the exact same things. :-| > >> > >> > >> > https://www.youtube.com/watch?v=8cYp52oeDx4&list=PL9Jh2HsAWHxLco7V1SjU9hUzP53CBZOYO&index=20 > > > > > > And they created rocksdb based on levelDB which is fit for such server > > workloads. > > > > http://rocksdb.org/ > > > > > > Michael Gebetsroither > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > > -- > BR, > \|/ Kunthar > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rvirding@REDACTED Tue Nov 18 01:14:27 2014 From: rvirding@REDACTED (Robert Virding) Date: Tue, 18 Nov 2014 01:14:27 +0100 Subject: [erlang-questions] Dialyzer type/spec info question In-Reply-To: References: Message-ID: Initial tests show that this idea works, but dialyzer is putting up a fight. :-) Robert On 17 November 2014 18:26, Robert Virding wrote: > The reason I would like to be able to handle multiple file types is to > simplify checking systems with parts which are written in different > languages. > > Robert > > > On 17 November 2014 18:19, Robert Virding wrote: > >> Something much naughtier and more basic. >> >> - Make LFE local copies of the relevant dialyzer modules which have been >> hacked to be able to handle LFE files as input (with a new --lfe option) >> - Run dialyzer using -pa to push the local module directory first so as >> to use these special modules instead of the standard ones >> >> This will make it possible to use dialyzer on LFE files and other .erl or >> .beam files as well. Maybe with a little more hacking fix have multiple >> file types. Of course if dialyzer was to be opened up so that it would use >> the filename extension to pick the right versions of the functions Stavros >> mentioned then this would be much simpler. Wink, wink, nudge nudge. :-) >> >> This is my plan anyway. >> >> Robert >> >> P.S. Having LFE generate erlang AST has never been an option as it is >> very close to core and it would take a lot of effort to go back to erlang >> which would then be undone by the erlang compiler. >> >> >> On 17 November 2014 17:49, Jos? Valim >> wrote: >> >>> Robert, what would a specialized version of dialyzer entail? A fork? A >>> specific front-end? Curious. :) >>> >>> On Monday, November 17, 2014, Robert Virding wrote: >>> >>>> Ah, so then in principle I can make my own specialised versions of the >>>> dialyzer load modules which takes the core which the LFE compiler generates >>>> and extracts the core and the type/spec data dialyzer needs. This without >>>> any Erlang AST which LFE never generates anyway. >>>> >>>> Now we are getting somewhere useful. With this we don't really need to >>>> be able to store core in the .beam files if we can accept using source >>>> files as input. >>>> >>>> Robert >>>> >>>> >>>> On 17 November 2014 17:32, Stavros Aronis wrote: >>>> >>>>> For .erl files, dialyzer calls the compiler with 'to_pp', which stops >>>>> the compilation before code is converted to Core, and reads attributes from >>>>> there. >>>>> >>>>> Judging from v3_core, line 170 the two representations should be >>>>> compatible. >>>>> >>>>> /Stavros >>>>> >>>>> On Mon, Nov 17, 2014 at 5:11 PM, Robert Virding >>>>> wrote: >>>>> >>>>>> From where does it get it if I have .erl files as input? From the AST >>>>>> as well? Do you know if there is any difference in the data itself between >>>>>> the AST and core? I am guessing not but want to check. If there is no >>>>>> difference then the info could be gotten from Core. This would make it >>>>>> easier to use dialyzer together with LFE. >>>>>> >>>>>> Robert >>>>>> >>>>>> >>>>>> On 17 November 2014 17:03, Stavros Aronis >>>>>> wrote: >>>>>> >>>>>>> Hi Robert, >>>>>>> >>>>>>> Dialyzer gets this info from beam_lib:chunks(File, [abstract_code]) >>>>>>> which corresponds to the AST and is included in beam files if +debug_info >>>>>>> is used while compiling. >>>>>>> >>>>>>> To my knowledge there is no reason to not let sources be combined >>>>>>> (except perhaps when building a PLT), but the implementation seems to >>>>>>> require uniformity. >>>>>>> >>>>>>> Relevant functions: >>>>>>> dialyzer_utils:get_abstract_code_from_{src,beam}/{1,2}, get_spec_info/3, >>>>>>> get_record_and_type_info/1. >>>>>>> >>>>>>> Regards, >>>>>>> >>>>>>> Stavros >>>>>>> >>>>>>> On Mon, Nov 17, 2014 at 4:53 PM, Robert Virding >>>>>>> wrote: >>>>>>> >>>>>>>> From where does dialyzer get the user added type and spec info? >>>>>>>> From the AST, or from Core erlang which contains the same type/spec data? >>>>>>>> And why? Can I control it? >>>>>>>> >>>>>>>> An extra question: why doesn't dialyzer allow me to mix input from >>>>>>>> both .erl and .beam files? Or does it and I have missed that? >>>>>>>> >>>>>>>> Robert >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> erlang-questions mailing list >>>>>>>> erlang-questions@REDACTED >>>>>>>> http://erlang.org/mailman/listinfo/erlang-questions >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >>> -- >>> >>> >>> *Jos? Valim* >>> www.plataformatec.com.br >>> Skype: jv.ptec >>> Founder and Lead Developer >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang@REDACTED Tue Nov 18 11:10:42 2014 From: erlang@REDACTED (Joe Armstrong) Date: Tue, 18 Nov 2014 11:10:42 +0100 Subject: [erlang-questions] is there a generic file fetcher anywhere? Message-ID: Hello, Has anybody written a "generic file fetcher" - I'd like to fetch files or collections of files from various remote locations (Git, SVG, HTTP, FTP etc) handling proxies etc. on the way Is there any generic code to do this? I took a quick peek at the rebar source code since I know it can fetch from git but it just does a os:cmd("git fetch origin ...") It strikes me that it would be a good idea to isolate code like this into a separate library and make it more generic. This would be rather useful Cheers /Joe From junioru_ovidiu@REDACTED Tue Nov 18 10:59:15 2014 From: junioru_ovidiu@REDACTED (Ovidiu Iulian) Date: Tue, 18 Nov 2014 09:59:15 +0000 (UTC) Subject: [erlang-questions] Integrate single sign on PHP into Erlang Message-ID: <251183591.1921911.1416304755141.JavaMail.yahoo@jws106141.mail.bf1.yahoo.com> Hi Everyone, thank you for this opportunity and help. I'm a newbie in Erlang. We have a single sign on written in Php and we need to integrate it with Erlang. Is there any documentation for this, ever done something like this? thx -------------- next part -------------- An HTML attachment was scrubbed... URL: From alcosholik@REDACTED Tue Nov 18 11:25:54 2014 From: alcosholik@REDACTED (Alexei Sholik) Date: Tue, 18 Nov 2014 12:25:54 +0200 Subject: [erlang-questions] is there a generic file fetcher anywhere? In-Reply-To: References: Message-ID: Things like HTTP and Git operate on different kinds of entities. One does not simply fetch a file from a Git repository, the whole repository has to be cloned (or shallow-cloned). Do you have in mind something like "here's my and , please fetch whatever this points to"? On Tue, Nov 18, 2014 at 12:10 PM, Joe Armstrong wrote: > Hello, > > Has anybody written a "generic file fetcher" - I'd like to fetch files > or collections of files from various remote locations (Git, SVG, HTTP, FTP > etc) > handling proxies etc. on the way > > Is there any generic code to do this? > > I took a quick peek at the rebar source code since I know it can fetch > from git > but it just does a os:cmd("git fetch origin ...") > > It strikes me that it would be a good idea to isolate code like this into a > separate library and make it more generic. This would be rather useful > > Cheers > > /Joe > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From peerst@REDACTED Tue Nov 18 11:36:57 2014 From: peerst@REDACTED (Peer Stritzinger) Date: Tue, 18 Nov 2014 11:36:57 +0100 Subject: [erlang-questions] Dialyzer type/spec info question References: Message-ID: On 2014-11-17 17:26:01 +0000, Robert Virding said: > The reason I would like to be able to handle multiple file types is to > simplify checking systems with parts which are written in different > languages. +1 This would be useful to my IEC61499 PLC language compiler too, especially since I have some native (here meaning in Erlang) implemented behaviour callback modules which can then be checked if they implement a matching interface to the PLC interface part. And since I'm also generating core like LFE opening up this interface to Dialyzer would help me greatly. Especially so since I'm compiling a statically typed language anyway and could derive the typespecs of the underlying Erlang types from this information. Cheers, -- Peer > > Robert > > > On 17 November 2014 18:19, Robert Virding wrote: > Something much naughtier and more basic. > > - Make LFE local copies of the relevant dialyzer modules which have > been hacked to be able to handle LFE files as input (with a new --lfe > option) > - Run dialyzer using -pa to push the local module directory first so as > to use these special modules instead of the standard ones > > This will make it possible to use dialyzer on LFE files and other .erl > or .beam files as well. Maybe with a little more hacking fix have > multiple file types. Of course if dialyzer was to be opened up so that > it would use the filename extension to pick the right versions of the > functions Stavros mentioned then this would be much simpler. Wink, > wink, nudge nudge. :-) > > This is my plan anyway. > > Robert > > P.S. Having LFE generate erlang AST has never been an option as it is > very close to core and it would take a lot of effort to go back to > erlang which would then be undone by the erlang compiler. > > > On 17 November 2014 17:49, Jos? Valim wrote: > Robert, what would a specialized version of dialyzer entail? A fork? A > specific front-end? Curious. :) > > On Monday, November 17, 2014, Robert Virding wrote: > Ah, so then in principle I can make my own specialised versions of the > dialyzer load modules which takes the core which the LFE compiler > generates and extracts the core and the type/spec data dialyzer needs. > This without any Erlang AST which LFE never generates anyway. > > Now we are getting somewhere useful. With this we don't really need to > be able to store core in the .beam files if we can accept using source > files as input. > > Robert > > > On 17 November 2014 17:32, Stavros Aronis wrote: > For .erl files, dialyzer calls the compiler with 'to_pp', which stops > the compilation before code is converted to Core, and reads attributes > from there. > > Judging from v3_core, line 170 the two representations should be compatible. > > /Stavros > > On Mon, Nov 17, 2014 at 5:11 PM, Robert Virding wrote: > From where does it get it if I have .erl files as input? From the AST > as well? Do you know if there is any difference in the data itself > between the AST and core? I am guessing not but want to check. If there > is no difference then the info could be gotten from Core. This would > make it easier to use dialyzer together with LFE. > > Robert > > > On 17 November 2014 17:03, Stavros Aronis wrote: > Hi Robert, > > Dialyzer gets this info from beam_lib:chunks(File, [abstract_code]) > which corresponds to the AST and is included in beam files if > +debug_info is used while compiling. > > To my knowledge there is no reason to not let sources be combined > (except perhaps when building a PLT), but the implementation seems to > require uniformity. > > Relevant functions: > dialyzer_utils:get_abstract_code_from_{src,beam}/{1,2}, > get_spec_info/3, get_record_and_type_info/1. > > Regards, > > Stavros > > On Mon, Nov 17, 2014 at 4:53 PM, Robert Virding wrote: > From where does dialyzer get the user added type and spec info? From > the AST, or from Core erlang which contains the same type/spec data? > And why? Can I control it? > > An extra question: why doesn't dialyzer allow me to mix input from both > .erl and .beam files? Or does it and I have missed that? > > Robert > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > > > -- > > > Jos? Valim > www.plataformatec.com.br > Skype: jv.ptec > Founder and Lead Developer From erlang@REDACTED Tue Nov 18 11:43:19 2014 From: erlang@REDACTED (Joe Armstrong) Date: Tue, 18 Nov 2014 11:43:19 +0100 Subject: [erlang-questions] is there a generic file fetcher anywhere? In-Reply-To: References: Message-ID: On Tue, Nov 18, 2014 at 11:25 AM, Alexei Sholik wrote: > Things like HTTP and Git operate on different kinds of entities. Precisely - what I want to do is abstract out the differences - I see no reason why I should not extract a specific file from a git archive given its hash. > One does > not simply fetch a file from a Git repository, the whole repository has to > be cloned (or shallow-cloned). Why not? - often I do this (ie clone a repository, extract one file) then throw the rest away :-) Git is just a key-value blob store whose keys are content hashes - and a few fancy commands with silly names to hide this from you. > > Do you have in mind something like "here's my and , please > fetch whatever this points to"? I was thinking more of things that are publically available /Joe > > On Tue, Nov 18, 2014 at 12:10 PM, Joe Armstrong wrote: >> >> Hello, >> >> Has anybody written a "generic file fetcher" - I'd like to fetch files >> or collections of files from various remote locations (Git, SVG, HTTP, FTP >> etc) >> handling proxies etc. on the way >> >> Is there any generic code to do this? >> >> I took a quick peek at the rebar source code since I know it can fetch >> from git >> but it just does a os:cmd("git fetch origin ...") >> >> It strikes me that it would be a good idea to isolate code like this into >> a >> separate library and make it more generic. This would be rather useful >> >> Cheers >> >> /Joe >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > From zxq9@REDACTED Tue Nov 18 11:55:18 2014 From: zxq9@REDACTED (zxq9) Date: Tue, 18 Nov 2014 19:55:18 +0900 Subject: [erlang-questions] is there a generic file fetcher anywhere? In-Reply-To: References: Message-ID: <4132242.QKUHN5x9AA@changa> Maybe the more interesting question up front is whether there is a generic file finder? At the very least addressing would be a minor challenge (one GitHib has tried to pretend doesn't exist through the use of gigantic URLs), even if discovery were trivial. I vaguely remember something about gopher discovery and retrieval extensions that could cope with whatever you wrote them to, and it not being easy to manage addressing or discovery. But that was a long time ago. -Craig On 2014?11?18? ??? 11:43:19 Joe Armstrong wrote: > On Tue, Nov 18, 2014 at 11:25 AM, Alexei Sholik wrote: > > Things like HTTP and Git operate on different kinds of entities. > > Precisely - what I want to do is abstract out the differences - I see > no reason why I should not > extract a specific file from a git archive given its hash. > > > One does > > not simply fetch a file from a Git repository, the whole repository has to > > be cloned (or shallow-cloned). > > Why not? - often I do this (ie clone a repository, extract one file) > then throw the rest away :-) > > Git is just a key-value blob store whose keys are content hashes - and > a few fancy commands with > silly names to hide this from you. > > > Do you have in mind something like "here's my and , > > please fetch whatever this points to"? > > I was thinking more of things that are publically available > > > /Joe > > > On Tue, Nov 18, 2014 at 12:10 PM, Joe Armstrong wrote: > >> Hello, > >> > >> Has anybody written a "generic file fetcher" - I'd like to fetch files > >> or collections of files from various remote locations (Git, SVG, HTTP, > >> FTP > >> etc) > >> handling proxies etc. on the way > >> > >> Is there any generic code to do this? > >> > >> I took a quick peek at the rebar source code since I know it can fetch > >> from git > >> but it just does a os:cmd("git fetch origin ...") > >> > >> It strikes me that it would be a good idea to isolate code like this into > >> a > >> separate library and make it more generic. This would be rather useful > >> > >> Cheers > >> > >> /Joe > >> _______________________________________________ > >> erlang-questions mailing list > >> erlang-questions@REDACTED > >> http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From mfidelman@REDACTED Tue Nov 18 12:37:28 2014 From: mfidelman@REDACTED (Miles Fidelman) Date: Tue, 18 Nov 2014 06:37:28 -0500 Subject: [erlang-questions] is there a generic file fetcher anywhere? In-Reply-To: References: Message-ID: <546B2F78.7010606@meetinghouse.net> Joe Armstrong wrote: > Hello, > > Has anybody written a "generic file fetcher" - I'd like to fetch files > or collections of files from various remote locations (Git, SVG, HTTP, FTP etc) > handling proxies etc. on the way > Sounds like you're looking for an erlang version of curl (which does, I believe, talk git's API). Maybe run curl through a port? Miles Fidelman -- In theory, there is no difference between theory and practice. In practice, there is. .... Yogi Berra From g@REDACTED Tue Nov 18 15:28:21 2014 From: g@REDACTED (Garrett Smith) Date: Tue, 18 Nov 2014 08:28:21 -0600 Subject: [erlang-questions] is there a generic file fetcher anywhere? In-Reply-To: <546B2F78.7010606@meetinghouse.net> References: <546B2F78.7010606@meetinghouse.net> Message-ID: On Tue, Nov 18, 2014 at 5:37 AM, Miles Fidelman wrote: > Joe Armstrong wrote: >> >> Hello, >> >> Has anybody written a "generic file fetcher" - I'd like to fetch files >> or collections of files from various remote locations (Git, SVG, HTTP, FTP >> etc) >> handling proxies etc. on the way >> > > Sounds like you're looking for an erlang version of curl (which does, I > believe, talk git's API). > > Maybe run curl through a port? This From erik.sette@REDACTED Tue Nov 18 13:24:58 2014 From: erik.sette@REDACTED (Erik Sette) Date: Tue, 18 Nov 2014 07:24:58 -0500 Subject: [erlang-questions] is there a generic file fetcher anywhere? In-Reply-To: <546B2F78.7010606@meetinghouse.net> References: <546B2F78.7010606@meetinghouse.net> Message-ID: I would also enjoy a general solution myself. I actually modified my local copy of rebar to get files (artifacts) from artifactory via http. I now can specify in my rebar.config something like: {http, Url, [{auth, {basic, {User, Password}}}, tgz]}, [raw]} and it will either look for the exact Url or get the "latest" version it finds. Erik On Tue, Nov 18, 2014 at 6:37 AM, Miles Fidelman wrote: > Joe Armstrong wrote: > >> Hello, >> >> Has anybody written a "generic file fetcher" - I'd like to fetch files >> or collections of files from various remote locations (Git, SVG, HTTP, >> FTP etc) >> handling proxies etc. on the way >> >> > Sounds like you're looking for an erlang version of curl (which does, I > believe, talk git's API). > > Maybe run curl through a port? > > Miles Fidelman > > -- > In theory, there is no difference between theory and practice. > In practice, there is. .... Yogi Berra > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From viktor@REDACTED Tue Nov 18 16:13:26 2014 From: viktor@REDACTED (=?ISO-8859-1?Q?Viktor_S=F6derqvist?=) Date: Tue, 18 Nov 2014 16:13:26 +0100 Subject: [erlang-questions] GPL and others licenses Message-ID: Hello. What are the implications of using GPL for an Erlang application? Say, I'm considering using GPL 3 in a project for whatever reason. >From what I've understood EPL is incompatible with GPL. Does that mean my library application cannot be distibuted together with Erlang/OTP itself in a release? Can module loading in the BEAM be considered to be "linking"? If that is so, an LGPL application can be used and distributed together with any other Erlang code, right? I remember reading somewhere that in Java they consider the class loading to be dynamic linking. EPL is based on MPL 1.0 which is GPL incompatible. Mozilla have since then updated their license to MPL 2.0 which is GPL compatible. Is there any chance EPL will be updated and be GPL compatible in the future? Viktor From mpmiszczyk@REDACTED Tue Nov 18 16:27:02 2014 From: mpmiszczyk@REDACTED (Marcin Miszczyk) Date: Tue, 18 Nov 2014 16:27:02 +0100 Subject: [erlang-questions] PLT from source. Message-ID: Is it possible to create plt file from source files? It seems it should be possible, but I can not make it work. -------------- next part -------------- An HTML attachment was scrubbed... URL: From k.petrauskas@REDACTED Tue Nov 18 20:06:24 2014 From: k.petrauskas@REDACTED (Karolis Petrauskas) Date: Tue, 18 Nov 2014 21:06:24 +0200 Subject: [erlang-questions] Distributed application and netsplit Message-ID: Hello, I am trying to implement an application, that should be active on a single node in a cluster. Distributed applications look like a proper tool for implementing such behaviour, so I implemented my app using it. It works as expected (failover ant takeover), when turning on and off appropriate nodes. I tried to check, what happens on a netsplit and a recovery from it. I simulated the netsplit by erlang:disconnect_node/1. The application has been started on both nodes (that was expected), although they were left both running after netsplit was removed (using net_adm:ping/1). So the application is not stopped automatically on the second node. How can I stop it manually? I use the global process registry in that application and the resolve function (passed to global:register_name/3) can be used to detect such situation. I tried the following: * invoke application:stop/1 on the secondary node. The application was stopped on the secondary node, although the failover mechanism was broken. I.e. if I shutdown the primary node after such recovery (from the netsplit), the application is not started on the secondary node anymore. * invoke application:permit/2 with Permission=false and then with Permission=true on the secondary node (looks like workaround). The application was started again, when permission was set to true. So it not works either. How can I stop the application on a secondary node so, that it would be started automatically in the case of failover? Karolis From imantc@REDACTED Tue Nov 18 20:30:31 2014 From: imantc@REDACTED (Imants Cekusins) Date: Tue, 18 Nov 2014 20:30:31 +0100 Subject: [erlang-questions] Distributed application and netsplit In-Reply-To: References: Message-ID: net_kernel:monitor_nodes/1,2 ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From k.petrauskas@REDACTED Tue Nov 18 21:38:39 2014 From: k.petrauskas@REDACTED (Karolis Petrauskas) Date: Tue, 18 Nov 2014 22:38:39 +0200 Subject: [erlang-questions] Distributed application and netsplit In-Reply-To: References: Message-ID: My problem is not how to detect netsplit or recovery from it. My problem is how to stop a distributed application on a secondary node(s) so, that they still would be started in the case of failover automatically (when primary node is down). By primary, I mean the node, that is listed before other (secondary) nodes in the kernel's "distributed" option. Karolis On Tue, Nov 18, 2014 at 9:30 PM, Imants Cekusins wrote: > net_kernel:monitor_nodes/1,2 ? From rpettit@REDACTED Tue Nov 18 21:55:40 2014 From: rpettit@REDACTED (Rick Pettit) Date: Tue, 18 Nov 2014 14:55:40 -0600 Subject: [erlang-questions] Distributed application and netsplit In-Reply-To: References: Message-ID: <148C0C12-566F-4C43-B77E-B5A77D4260E9@vailsys.com> For what it?s worth I understand your question and am also interested in hearing the ?right? way to handle this. :-) -Rick On Nov 18, 2014, at 2:38 PM, Karolis Petrauskas wrote: > My problem is not how to detect netsplit or recovery from it. My > problem is how to stop a distributed application on a secondary > node(s) so, that they still would be started in the case of failover > automatically (when primary node is down). > > By primary, I mean the node, that is listed before other (secondary) > nodes in the kernel's "distributed" option. > > Karolis > > On Tue, Nov 18, 2014 at 9:30 PM, Imants Cekusins wrote: >> net_kernel:monitor_nodes/1,2 ? > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From raould@REDACTED Tue Nov 18 22:00:02 2014 From: raould@REDACTED (Raoul Duke) Date: Tue, 18 Nov 2014 13:00:02 -0800 Subject: [erlang-questions] Distributed application and netsplit In-Reply-To: <148C0C12-566F-4C43-B77E-B5A77D4260E9@vailsys.com> References: <148C0C12-566F-4C43-B77E-B5A77D4260E9@vailsys.com> Message-ID: hello? isn't this like a canonical systems engineer question?! https://www.google.com/search?q=split+brain+interview+question+networking+storage From sean@REDACTED Tue Nov 18 22:03:09 2014 From: sean@REDACTED (Sean Cribbs) Date: Tue, 18 Nov 2014 15:03:09 -0600 Subject: [erlang-questions] io:format validation Message-ID: Where in the OTP source is the validation of format strings done? Specifically, when you have io:format/2,3 or io_lib:format/2,3 in your code and the compiler complains about the number of patterns not matching the number of arguments. I'd like to use this elsewhere, for instance, where a library call eventually calls io_lib:format/2,3 but the caller into the library is not validated. -- Sean Cribbs Sr. Software Engineer Basho Technologies, Inc. http://basho.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From felixgallo@REDACTED Tue Nov 18 22:04:33 2014 From: felixgallo@REDACTED (Felix Gallo) Date: Tue, 18 Nov 2014 13:04:33 -0800 Subject: [erlang-questions] Distributed application and netsplit In-Reply-To: References: <148C0C12-566F-4C43-B77E-B5A77D4260E9@vailsys.com> Message-ID: No, it's a specific question having to do with the failover/takeover mechanisms. If I'm understanding the problem correctly, ferd's page on the topic may be handy -- specifically the bottom part talks about how to configure different nodes to recognize themselves as being less important than others. http://learnyousomeerlang.com/distributed-otp-applications On Tue, Nov 18, 2014 at 1:00 PM, Raoul Duke wrote: > hello? isn't this like a canonical systems engineer question?! > > > https://www.google.com/search?q=split+brain+interview+question+networking+storage > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From k.petrauskas@REDACTED Tue Nov 18 22:23:15 2014 From: k.petrauskas@REDACTED (Karolis Petrauskas) Date: Tue, 18 Nov 2014 23:23:15 +0200 Subject: [erlang-questions] Distributed application and netsplit In-Reply-To: References: <148C0C12-566F-4C43-B77E-B5A77D4260E9@vailsys.com> Message-ID: I have read LYSE and its section on distributed applications in particular. I have configured my nodes so, that I can determine, which node is more important than other. The problem is how to stop the application on the less important node. This problem only occurs after recovery from netsplit. Ferd's page (http://learnyousomeerlang.com/distributed-otp-applications) covers netsplits only by the following note: Note: In terms of distributed programming fallacies, distributed OTP applications assume that when there is a failure, it is likely due to a hardware failure, and not a netsplit. If you deem netsplits more likely than hardware failures, then you have to be aware of the possibility that the application is running both as a backup and main one, and that funny things could happen when the network issue is resolved. Maybe distributed OTP applications aren't the right mechanism for you in these cases. Now I have those "funny things". Karolis On Tue, Nov 18, 2014 at 11:04 PM, Felix Gallo wrote: > No, it's a specific question having to do with the failover/takeover > mechanisms. > > If I'm understanding the problem correctly, ferd's page on the topic may be > handy -- specifically the bottom part talks about how to configure different > nodes to recognize themselves as being less important than others. > > http://learnyousomeerlang.com/distributed-otp-applications > > On Tue, Nov 18, 2014 at 1:00 PM, Raoul Duke wrote: >> >> hello? isn't this like a canonical systems engineer question?! >> >> >> https://www.google.com/search?q=split+brain+interview+question+networking+storage >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From imantc@REDACTED Tue Nov 18 22:32:30 2014 From: imantc@REDACTED (Imants Cekusins) Date: Tue, 18 Nov 2014 22:32:30 +0100 Subject: [erlang-questions] Distributed application and netsplit In-Reply-To: References: <148C0C12-566F-4C43-B77E-B5A77D4260E9@vailsys.com> Message-ID: .. may application:load/2 be relevant? It takes Nodes parameters. If the current node crashes, application controller will restart the app on the next node in order of specified priority. -------------- next part -------------- An HTML attachment was scrubbed... URL: From raould@REDACTED Tue Nov 18 22:32:42 2014 From: raould@REDACTED (Raoul Duke) Date: Tue, 18 Nov 2014 13:32:42 -0800 Subject: [erlang-questions] Distributed application and netsplit In-Reply-To: References: <148C0C12-566F-4C43-B77E-B5A77D4260E9@vailsys.com> Message-ID: > Now I have those "funny things". does that mean > Note: Maybe distributed OTP applications aren't the right > mechanism for you in these cases. ? (not intended as being snarky.) From felixgallo@REDACTED Tue Nov 18 22:38:34 2014 From: felixgallo@REDACTED (Felix Gallo) Date: Tue, 18 Nov 2014 13:38:34 -0800 Subject: [erlang-questions] Distributed application and netsplit In-Reply-To: References: <148C0C12-566F-4C43-B77E-B5A77D4260E9@vailsys.com> Message-ID: It's possible that the takeover logic wasn't implemented on network partition heal because there's no obviously right thing to do in the generic case when two nodes believing themselves to be masters in a distributed system discover that they have independently been making progress owing to a network partition. On the other hand, the master/slave system creates data loss anyway on node failure. So I would personally call what you have found a bug, and try to make a minimum example case and see if anyone from OTP is paying attention. Pragmatically if relatively hackily, it would be a few hours' work to implement a watchdog gen_server that calls nodes(), and on any changes, broadcasts info messages to all affected parties for appropriate resolution. You could even try to deal with reconciliation between a slave and a master in that scenario if you wanted to be extra clever. On Tue, Nov 18, 2014 at 1:23 PM, Karolis Petrauskas wrote: > I have read LYSE and its section on distributed applications in > particular. I have configured my nodes so, that I can determine, which > node is more important than other. The problem is how to stop the > application on the less important node. This problem only occurs after > recovery from netsplit. Ferd's page > (http://learnyousomeerlang.com/distributed-otp-applications) covers > netsplits only by the following note: > > Note: In terms of distributed programming fallacies, distributed OTP > applications assume that when there is a failure, it is likely due to > a hardware failure, and not a netsplit. If you deem netsplits more > likely than hardware failures, then you have to be aware of the > possibility that the application is running both as a backup and main > one, and that funny things could happen when the network issue is > resolved. Maybe distributed OTP applications aren't the right > mechanism for you in these cases. > > Now I have those "funny things". > > Karolis > > On Tue, Nov 18, 2014 at 11:04 PM, Felix Gallo > wrote: > > No, it's a specific question having to do with the failover/takeover > > mechanisms. > > > > If I'm understanding the problem correctly, ferd's page on the topic may > be > > handy -- specifically the bottom part talks about how to configure > different > > nodes to recognize themselves as being less important than others. > > > > http://learnyousomeerlang.com/distributed-otp-applications > > > > On Tue, Nov 18, 2014 at 1:00 PM, Raoul Duke wrote: > >> > >> hello? isn't this like a canonical systems engineer question?! > >> > >> > >> > https://www.google.com/search?q=split+brain+interview+question+networking+storage > >> _______________________________________________ > >> erlang-questions mailing list > >> erlang-questions@REDACTED > >> http://erlang.org/mailman/listinfo/erlang-questions > > > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From k.petrauskas@REDACTED Tue Nov 18 22:44:11 2014 From: k.petrauskas@REDACTED (Karolis Petrauskas) Date: Tue, 18 Nov 2014 23:44:11 +0200 Subject: [erlang-questions] Distributed application and netsplit In-Reply-To: References: <148C0C12-566F-4C43-B77E-B5A77D4260E9@vailsys.com> Message-ID: On Tue, Nov 18, 2014 at 11:32 PM, Raoul Duke wrote: >> Now I have those "funny things". > > does that mean > >> Note: Maybe distributed OTP applications aren't the right >> mechanism for you in these cases. > > ? > > (not intended as being snarky.) I had a hope to avoid that :) The next alternative is to implement this functionality in my application, by starting/stopping relevant parts of the supervision tree. From raould@REDACTED Tue Nov 18 22:52:04 2014 From: raould@REDACTED (Raoul Duke) Date: Tue, 18 Nov 2014 13:52:04 -0800 Subject: [erlang-questions] Distributed application and netsplit In-Reply-To: References: <148C0C12-566F-4C43-B77E-B5A77D4260E9@vailsys.com> Message-ID: On Tue, Nov 18, 2014 at 1:38 PM, Felix Gallo wrote: > It's possible that the takeover logic wasn't implemented on network > partition heal because there's no obviously right thing to do in the generic > case when two nodes believing themselves to be masters in a distributed > system discover that they have independently been making progress owing to a > network partition. On the other hand, the master/slave system creates data > loss anyway on node failure. So I would personally call what you have found > a bug, and try to make a minimum example case and see if anyone from OTP is > paying attention. how is it a bug when, "there's no obviously right thing to do"? (not snarky, just confused/curious/clueless.) From felixgallo@REDACTED Tue Nov 18 23:05:41 2014 From: felixgallo@REDACTED (Felix Gallo) Date: Tue, 18 Nov 2014 14:05:41 -0800 Subject: [erlang-questions] Distributed application and netsplit In-Reply-To: References: <148C0C12-566F-4C43-B77E-B5A77D4260E9@vailsys.com> Message-ID: We're out of CAP world anyway (see my second sentence in that paragraph) so all bets are off. It's more consistent for it to have a single behavior in the event of a cluster membership change (master/slave failover/takeover), rather than to have two separate behaviors (master/slave failover/takeover on node loss, or master/master split-brain on netsplit). So that would be "better" and easier to design against. On Tue, Nov 18, 2014 at 1:52 PM, Raoul Duke wrote: > On Tue, Nov 18, 2014 at 1:38 PM, Felix Gallo wrote: > > It's possible that the takeover logic wasn't implemented on network > > partition heal because there's no obviously right thing to do in the > generic > > case when two nodes believing themselves to be masters in a distributed > > system discover that they have independently been making progress owing > to a > > network partition. On the other hand, the master/slave system creates > data > > loss anyway on node failure. So I would personally call what you have > found > > a bug, and try to make a minimum example case and see if anyone from OTP > is > > paying attention. > > > how is it a bug when, "there's no obviously right thing to do"? (not > snarky, just confused/curious/clueless.) > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From raould@REDACTED Tue Nov 18 23:08:02 2014 From: raould@REDACTED (Raoul Duke) Date: Tue, 18 Nov 2014 14:08:02 -0800 Subject: [erlang-questions] Distributed application and netsplit In-Reply-To: References: <148C0C12-566F-4C43-B77E-B5A77D4260E9@vailsys.com> Message-ID: On Tue, Nov 18, 2014 at 2:05 PM, Felix Gallo wrote: > We're out of CAP world anyway (see my second sentence in that paragraph) so > all bets are off. > > It's more consistent for it to have a single behavior in the event of a > cluster membership change (master/slave failover/takeover), rather than to > have two separate behaviors (master/slave failover/takeover on node loss, or > master/master split-brain on netsplit). So that would be "better" and > easier to design against. er, i still don't get it. if all bets are off then why should the toolkit/framework be choosing sides? From rpettit@REDACTED Tue Nov 18 23:23:27 2014 From: rpettit@REDACTED (Rick Pettit) Date: Tue, 18 Nov 2014 16:23:27 -0600 Subject: [erlang-questions] Distributed application and netsplit In-Reply-To: References: <148C0C12-566F-4C43-B77E-B5A77D4260E9@vailsys.com> Message-ID: <802A948E-4671-4D4F-A7B3-552F15EDAE47@vailsys.com> This seems like a shame, since distributed application gives you 99% of what you need?it just doesn?t handle recovery from the split brain case in the event of a net split. It doesn?t seem unreasonable to try to do what you are doing, adding some application-specific logic to determine a winning node, and one or more losing nodes, and to programmatically ?reset? the losing nodes such that (1) the application is no longer actively running on them, and (2) failover continues to work going forward. I can understand if distributed application doesn?t do this out of the box, since this isn?t a one-size-fits-all sort of problem, but surely there is some way to accomplish this without rewriting most of distributed application, no? By the way, not targeting that question at you Karolis, but rather anyone on the list that may be more familiar with distributed application than we are. -Rick On Nov 18, 2014, at 3:44 PM, Karolis Petrauskas wrote: > On Tue, Nov 18, 2014 at 11:32 PM, Raoul Duke wrote: >>> Now I have those "funny things". >> >> does that mean >> >>> Note: Maybe distributed OTP applications aren't the right >>> mechanism for you in these cases. >> >> ? >> >> (not intended as being snarky.) > > I had a hope to avoid that :) The next alternative is to implement > this functionality in my application, by starting/stopping relevant > parts of the supervision tree. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From sean@REDACTED Tue Nov 18 23:26:56 2014 From: sean@REDACTED (Sean Cribbs) Date: Tue, 18 Nov 2014 16:26:56 -0600 Subject: [erlang-questions] io:format validation In-Reply-To: References: Message-ID: Nevermind, I found it deep in erl_lint. On Tue, Nov 18, 2014 at 3:03 PM, Sean Cribbs wrote: > Where in the OTP source is the validation of format strings done? > Specifically, when you have io:format/2,3 or io_lib:format/2,3 in your code > and the compiler complains about the number of patterns not matching the > number of arguments. > > I'd like to use this elsewhere, for instance, where a library call > eventually calls io_lib:format/2,3 but the caller into the library is not > validated. > > -- > Sean Cribbs > Sr. Software Engineer > Basho Technologies, Inc. > http://basho.com/ > -- Sean Cribbs Sr. Software Engineer Basho Technologies, Inc. http://basho.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthias@REDACTED Tue Nov 18 23:59:07 2014 From: matthias@REDACTED (Matthias Lang) Date: Tue, 18 Nov 2014 23:59:07 +0100 Subject: [erlang-questions] Calling for help to improve contents on erlang.org In-Reply-To: References: Message-ID: <20141118225906.GA32465@corelatus.se> David Welton wrote: > I completely agree with Lo?c: open source it so that people can work > with it and help out. At the risk of hijacking Bruce's request...one part of erlang.org that _is_ open sourced is the FAQ. Source: https://github.com/matthiasl/Erlang-FAQ Built: http://www.erlang.org/faq/faq.html One quick and easy thing anyone can help with is choosing a random section of the FAQ and checking the links. I know, a linkchecker can find 404s. But a linkchecker can't easily find pages that return stuff that's no longer relevant. Nor can it find a suitable replacement link. Fixing up answers which have aged badly is another thing I'd love help with. I work with low-level signalling in telecom networks, so I don't keep up with web frameworks, for instance. Reading erlang-questions and noticing things that would make good entries is a bigger job, but there must be people out there who notice things. I look for subjects that - come up 'often', i.e. the "F" in FAQ. and - can be expressed as a fairly simple question. and - have a straight answer which many people can understand. and - have "staying power", i.e. will still be FAQs in three years. FWIW, I put the FAQ on github in 2009. Since then, six people have contributed through pull requests, including two I've never met. A few more have contributed by email. Matt From raould@REDACTED Wed Nov 19 00:37:18 2014 From: raould@REDACTED (Raoul Duke) Date: Tue, 18 Nov 2014 15:37:18 -0800 Subject: [erlang-questions] Distributed application and netsplit In-Reply-To: <802A948E-4671-4D4F-A7B3-552F15EDAE47@vailsys.com> References: <148C0C12-566F-4C43-B77E-B5A77D4260E9@vailsys.com> <802A948E-4671-4D4F-A7B3-552F15EDAE47@vailsys.com> Message-ID: On Tue, Nov 18, 2014 at 2:23 PM, Rick Pettit wrote: > This seems like a shame, since distributed application gives you 99% of what you need?it just doesn?t handle recovery from the split brain case in the event of a net split. i suspect it is really application-specific. From imantc@REDACTED Wed Nov 19 01:40:53 2014 From: imantc@REDACTED (Imants Cekusins) Date: Wed, 19 Nov 2014 01:40:53 +0100 Subject: [erlang-questions] Distributed application and netsplit In-Reply-To: References: <148C0C12-566F-4C43-B77E-B5A77D4260E9@vailsys.com> <802A948E-4671-4D4F-A7B3-552F15EDAE47@vailsys.com> Message-ID: how about adding lightweight proxy distributed apps: Proxy app instance is started by application controller when a node goes down. This proxy app checks / monitors netsplit and starts / stops the worker app. When connection is restored, 1+ proxy apps run (but no more than # of nodes) but only 1 worker app instance runs. Would it solve the problem? -------------- next part -------------- An HTML attachment was scrubbed... URL: From raould@REDACTED Wed Nov 19 01:52:39 2014 From: raould@REDACTED (Raoul Duke) Date: Tue, 18 Nov 2014 16:52:39 -0800 Subject: [erlang-questions] Distributed application and netsplit In-Reply-To: References: <148C0C12-566F-4C43-B77E-B5A77D4260E9@vailsys.com> <802A948E-4671-4D4F-A7B3-552F15EDAE47@vailsys.com> Message-ID: no matter what, there will be a time when both are making progress - they cannot instantaneously detect healing of the network partition. thus the first requirement is to know for a given application what the correct resolution process is. From imantc@REDACTED Wed Nov 19 02:02:22 2014 From: imantc@REDACTED (Imants Cekusins) Date: Wed, 19 Nov 2014 02:02:22 +0100 Subject: [erlang-questions] Distributed application and netsplit In-Reply-To: References: <148C0C12-566F-4C43-B77E-B5A77D4260E9@vailsys.com> <802A948E-4671-4D4F-A7B3-552F15EDAE47@vailsys.com> Message-ID: does monitor_nodes not keep subscribers up-to-date? The proxy apps could store a list of nodes in order of priority. Each app instance but the one running in the top priority active node would stop their worker. ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From zxq9@REDACTED Wed Nov 19 02:45:37 2014 From: zxq9@REDACTED (zxq9) Date: Wed, 19 Nov 2014 10:45:37 +0900 Subject: [erlang-questions] GPL and others licenses In-Reply-To: References: Message-ID: <5400473.hSIFceOgSg@changa> Hi Viktor. Please note that I am not a lawyer and this is not legal advice. What I write below is my sense of the rules from watching Red Hat legal interpret and arbitrate various licensing situations that have come up over the years in Fedora and RHEL. On 2014?11?18? ??? 16:13:26 Viktor S?derqvist wrote: > Hello. > > What are the implications of using GPL for an Erlang application? Say, > I'm considering using GPL 3 in a project for whatever reason. The code you write is independent of the code that constitutes Erlang itself. This is true no matter the platform or language: your works are your works. The real question comes with distribution. > From what I've understood EPL is incompatible with GPL. Does that mean > my library application cannot be distibuted together with Erlang/OTP > itself in a release? Your code could probably not be included in a release of Erlang/OTP without relicensing to EPL or LGPL (which is up to you, if you are the sole licensing authority), but you can distribute the Erlang runtime under its terms and your code under their own different terms. This is the same thing Linux distributions do: distribute a compilation of interoperating software. > Can module loading in the BEAM be considered to be "linking"? If that > is so, an LGPL application can be used and distributed together with > any other Erlang code, right? I remember reading somewhere that in > Java they consider the class loading to be dynamic linking. The situation should be the same in Erlang. The bytecode is not the same as the runtime it runs on. Another way of thinking about this is to consider the rather large body of GPL programs running on Windows today. The situation would be different if you are distributing binaries made with HiPE, though. In that case you would be distributing something that would almost definitely be considered a statically compiled binary -- not a bundled compilation of cooperating, but different, software. > EPL is based on MPL 1.0 which is GPL incompatible. Mozilla have since > then updated their license to MPL 2.0 which is GPL compatible. Is > there any chance EPL will be updated and be GPL compatible in the > future? I'm curious about this to... But I'm not holding my breath -- or worrying about it too much. The terms of the EPL give us quite a bit of freedom and are *vastly* superior to, say, the little fleet of Oracle licenses that escort their Java product flotilla around everywhere (or the IBM Java licenses, for that matter). Fortunately the runtime and our code that runs on it are two different things, and we are free to distribute bytecode produced by erlc without extra encumberance (which is not the situation with *many* commercial compiler products). Remember, I am not a lawyer. If your company has a business interest in a concrete answer then you would either need to consult a lawyer familiar with software licensing issues or confirm with Ericsson that you are clear. Also, you can avoid the issue entirely by doing what Gentoo does: just provide source, including a bootstrapping script that builds your project releases from source locally. I like the Gentoo method of dodging license conflicts on binary disrtribution. It absolutely forces you to understand everything about your build environment -- its not just customer friendly in a legal sense, its better tech. -Craig From stu.bailey@REDACTED Wed Nov 19 02:49:31 2014 From: stu.bailey@REDACTED (Stu Bailey) Date: Tue, 18 Nov 2014 17:49:31 -0800 Subject: [erlang-questions] arcane library question Message-ID: egd is bundled in the standard distro: http://www.erlang.org/doc/man/egd.html the example leverages the single font which is bundled in percept: http://www.erlang.org/doc/man/percept.html 6x11_latin1.wingsfont Is that font format standard? Is it documented somewhere? Can I drop in other files files? Is there a different 100% Erlang graphics generator which people like prefer? Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From rvirding@REDACTED Wed Nov 19 04:14:42 2014 From: rvirding@REDACTED (Robert Virding) Date: Wed, 19 Nov 2014 04:14:42 +0100 Subject: [erlang-questions] Calling for help to improve contents on erlang.org In-Reply-To: <5464DAD9.1060608@ninenines.eu> References: <5464DAD9.1060608@ninenines.eu> Message-ID: Yes, I definitely agree with Lo?c on this, make the code and content open-source. And with a good licence, don't GPL it. And I agree with Tristan, put all of this in one place, either erlangcentral.org or erlang.org, and have decent links to it from other places. Robert On 13 November 2014 17:22, Lo?c Hoguin wrote: > Hi, > > This response is addressed to OTP Team, as Bruce already knows what I am > saying here. > > Please make the erlang.org code and content open-source, and add a link > to this repository at the bottom of all erlang.org pages. > > This is not only easier to contribute to for programmers, but will make > people who read contribute spontaneously (even if they don't end up using > Erlang) and will also address the concerns of those who do not want to > contribute to closed software/platforms. > > On 11/13/2014 06:14 PM, Bruce Yinhe wrote: > >> Hello, >> >> We are aware that parts of erlang.org need >> improvement. For example http://www.erlang.org/article/tag/examples and >> http://www.erlang.org/course/course.html are outdated. We would like to >> see a number of small code examples for beginners. The purpose of these >> examples is to provide an attractive and useful introduction for people >> who are interested in adopting the Erlang programming language. >> >> Please send us your input. We would like to call for help from the >> community since OTP team does not have too much time and it is not >> possible to submit pull requests for editorial of erlang.org >> as of now. >> >> Any other suggestions for erlang.org are always >> welcome. >> >> Best regards, >> Bruce Yinhe >> community-manager@REDACTED >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -- > Lo?c Hoguin > http://ninenines.eu > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jpcustin@REDACTED Wed Nov 19 00:31:38 2014 From: jpcustin@REDACTED (Jay Custin) Date: Tue, 18 Nov 2014 18:31:38 -0500 Subject: [erlang-questions] Confusion over documentation (OTPROOT vs OTP_ROOT) Message-ID: I needed to get to the bottom of an issue on my system that involved name resolution. I noticed in the Kernel Reference Manual that the net_adm: host_file function it states that the host_file function: Reads the .hosts.erlang file, see the section Files below. Returns the hosts in this file as a list, or returns {error, Reason} if the file could not be read or the Erlang terms on the file could not be interpreted. When you scroll to the bottom and read the Files section you see the following: The .hosts.erlang file consists of a number of host names written as Erlang terms. It is looked for in the current work directory, the user's home directory, and $OTP_ROOT (the root directory of Erlang/OTP), in that order. Note the underscore/underbar '_' in OTP_ROOT If you then read the beginning of the code module documentation, the second paragraph of the Code Path section has the following: Initially, the code path consists of the current working directory and all Erlang object code directories under the library directory $OTPROOT/lib, where $OTPROOT is the installation directory of Erlang/OTP, code:root_dir(). Note the LACK of the underscore/underbar '_' in OTPROOT Luckily we can always fall back on the Kernel Reference Manual section describing the config file which clears it all up with: When starting Erlang in embedded mode, it is assumed that exactly one system configuration file is used, named sys.config. This file should be located in $ROOT/releases/Vsn, where $ROOT is the Erlang/OTP root installation directory and Vsn is the release version. Confused about whether to use OTP or OTP_ in identifying the root installation for Erlang/OTP? Don't use either! Now I suppose by having three different ways for "identifying the root installation directory" that maybe the authors never intended for an end user to define environmental/system variables... then again... Building and Installing Erlang/OTP => make and $ERL_TOP opens with the sentence, "All the makefiles in the entire directory tree use the environment variable ERL_TOP to find the absolute path of the installation." so maybe we need to define ONE environment variable. So OTP_ROOT, OTPROOT, ROOT, or don't bother with any of them? SenseiC bows out. -------------- next part -------------- An HTML attachment was scrubbed... URL: From k.petrauskas@REDACTED Wed Nov 19 07:41:58 2014 From: k.petrauskas@REDACTED (Karolis Petrauskas) Date: Wed, 19 Nov 2014 08:41:58 +0200 Subject: [erlang-questions] Distributed application and netsplit In-Reply-To: References: <148C0C12-566F-4C43-B77E-B5A77D4260E9@vailsys.com> <802A948E-4671-4D4F-A7B3-552F15EDAE47@vailsys.com> Message-ID: I agree, that the recovery can be application specific. The problem here is that there is no way to resolve the netsplit even manually. In my case, I use the global process registry and therefore I am getting notifications in the case of recovery from netsplit. On that event the resolve function (see global:register_name/3) gets pids from both nodes. From them I can determine nodes, where the processes are running. Then I can determine, which of the nodes should keep running and which should be stopped. The only problem in my case is that there is no way to stop the secondary node explicitly. Imants, your suggestions imply rewriting a large portion of dist_ac (http://www.erlang.org/doc/design_principles/distributed_applications.html). I consider that as the plan B. I could make the minimal test case to reproduce my problem. But I am not sure, what to show there? I can show, that distributed application keeps running on both nodes after recovery from netsplit. However I doubt it will be considered a bug. Most likely, it is missing feature to be able to fix the situation explicitly. Karolis On Wed, Nov 19, 2014 at 3:02 AM, Imants Cekusins wrote: > does monitor_nodes not keep subscribers up-to-date? > > The proxy apps could store a list of nodes in order of priority. Each app > instance but the one running in the top priority active node would stop > their worker. > > ? > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From essen@REDACTED Wed Nov 19 10:17:24 2014 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Wed, 19 Nov 2014 11:17:24 +0200 Subject: [erlang-questions] Calling for help to improve contents on erlang.org In-Reply-To: <20141118225906.GA32465@corelatus.se> References: <20141118225906.GA32465@corelatus.se> Message-ID: <546C6024.6020409@ninenines.eu> On 11/19/2014 12:59 AM, Matthias Lang wrote: > > David Welton wrote: > >> I completely agree with Lo?c: open source it so that people can work >> with it and help out. > > At the risk of hijacking Bruce's request...one part of erlang.org that > _is_ open sourced is the FAQ. > > Source: https://github.com/matthiasl/Erlang-FAQ > Built: http://www.erlang.org/faq/faq.html Thanks for the FAQ. I'm probbly dumb. I opened the faq, failed to see a link to github, closed the faq, read your email... > FWIW, I put the FAQ on github in 2009. Since then, six people have > contributed through pull requests, including two I've never met. A > few more have contributed by email. ... finished reading your email, noticed some people contributed by email, tried to find it and finally found the link to github deep inside a question (and to be perfectly honest, almost missed it even after I clicked on the right category, but that's probably because I'm tired as the category is not that big). If there was a link on top or bottom (or even better considering the layout, on the left) it would greatly help dumb people like me to find their way. To give an example, the "fork me on github" thing people put in the corner of pages is a bit childish perhaps but at least you know where to contribute before you even started reading. Same applies to erlang.org when/if it becomes open source. All pages must have a link to show where to contribute. -- Lo?c Hoguin http://ninenines.eu From ml-erlang@REDACTED Wed Nov 19 10:47:51 2014 From: ml-erlang@REDACTED (Matthias Rieber) Date: Wed, 19 Nov 2014 10:47:51 +0100 Subject: [erlang-questions] =?utf-8?q?How_to_configure_inet=5Fdist=5Fliste?= =?utf-8?q?n=5Fmin/max_for_the_Observer?= Message-ID: <38f967e99320824d6f036dfb3bf66c8a@ssl.scheff32.de> Hi, I'd like to trace some processes on an ejabberd node which is behind a NAT. I configured the required port forwards and limited the port range for the dist ports on the ejabberd node with: -kernel inet_dist_listen_min 54000 inet_dist_listen_max 54010. Connection with -remsh and the Observer GUI work. I can see the processes, ets and mnesia tables. No problem so far. When I try to trace processes it turns out that another connection is required. This time, the port is not chosen from the ejabberd range 54000-54010. Is it possible to configure the port range for the Observer? Or maybe there's another approach to do that? Regards, Matthias From wallentin.dahlberg@REDACTED Wed Nov 19 10:55:10 2014 From: wallentin.dahlberg@REDACTED (=?UTF-8?Q?Bj=C3=B6rn=2DEgil_Dahlberg?=) Date: Wed, 19 Nov 2014 10:55:10 +0100 Subject: [erlang-questions] arcane library question In-Reply-To: References: Message-ID: 2014-11-19 2:49 GMT+01:00 Stu Bailey : > egd is bundled in the standard distro: > > http://www.erlang.org/doc/man/egd.html > > the example leverages the single font which is bundled in percept: > > http://www.erlang.org/doc/man/percept.html > > 6x11_latin1.wingsfont > > Is that font format standard? Is it documented somewhere? Can I drop in > other files files? > The font is from Wings3D, see https://github.com/bjorng/wings I don't know if is documented .. I doubt it. You can drop in other wings fonts .. it should work fine. I only needed it to render very simple images for offline webpages and javascript was not an option at the time. Nowadays there are a ton of js solutions for this so egd is no longer developed. -------------- next part -------------- An HTML attachment was scrubbed... URL: From sperber@REDACTED Wed Nov 19 09:30:57 2014 From: sperber@REDACTED (Michael Sperber) Date: Wed, 19 Nov 2014 09:30:57 +0100 Subject: [erlang-questions] Call for Participation: BOB 2015 (January 23, Berlin) Message-ID: Check out the BOB program - many talks and tutorials, and an entire track with Erlang tutorials! ================================================================ BOB 2015 Conference "What happens if we simply use what's best?" January 23.2015 Berlin http://bobkonf.de/2015/ Program: http://bobkonf.de/2015/programm.html Registration: http://bobkonf.de/2015/registration.html ================================================================ BOB is the conference for developers, architects and decision-makers to explore technologies beyond the mainstream in software development, and to find the best tools available to software developers today. Our goal is for all participants of BOB to return home with new insights that enable them to improve their own software development experiences. The program features 14 talks and 8 tutorials on current topics: http://bobkonf.de/2015/programm.html The subject range of talks includes functional programming, microservices, package management, and data management. The tutorials feature introductions to Erlang, Haskell, Swift, and ClojureScript, and their applications. Anil Madhavapeddy will hold the keynote talk - about unikernels and functional programming. Registration is open online: http://bobkonf.de/2015/registration.html NOTE: The early-bird rates expire on Dec. 19, 2014! BOB cooperates with the :clojured conference on the following day. There is a registration discount available for participants of both events. http://www.clojured.de/ From dangud@REDACTED Wed Nov 19 11:40:17 2014 From: dangud@REDACTED (Dan Gudmundsson) Date: Wed, 19 Nov 2014 11:40:17 +0100 Subject: [erlang-questions] How to configure inet_dist_listen_min/max for the Observer In-Reply-To: <38f967e99320824d6f036dfb3bf66c8a@ssl.scheff32.de> References: <38f967e99320824d6f036dfb3bf66c8a@ssl.scheff32.de> Message-ID: > Is it possible to configure the port range for the Observer? Or maybe there's another approach to do that? Not possible from the gui at least, it uses ttb have a look in the manual and send a patch, if possible, on observer :-) /Dan On Wed, Nov 19, 2014 at 10:47 AM, Matthias Rieber wrote: > Hi, > > I'd like to trace some processes on an ejabberd node which is behind a > NAT. I configured the required port forwards and limited the port range for > the dist ports on the ejabberd node with: -kernel inet_dist_listen_min > 54000 inet_dist_listen_max 54010. > > Connection with -remsh and the Observer GUI work. I can see the processes, > ets and mnesia tables. No problem so far. When I try to trace processes it > turns out that another connection is required. This time, the port is not > chosen from the ejabberd range 54000-54010. Is it possible to configure the > port range for the Observer? Or maybe there's another approach to do that? > > Regards, > Matthias > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Wed Nov 19 15:40:39 2014 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Wed, 19 Nov 2014 15:40:39 +0100 Subject: [erlang-questions] Emysql: a cry for help (Harbinger is releasing control) Message-ID: Hi Everyone, You may know that I was maintaining the Emysql code base for Erlang for a while. It is a driver for MySQL which seem to be quite popular. However, as of late, I have not really been maintaining it anymore, and I see no reason to continuing doing so: * I have no passion for MySQL. It is the worst database in existence ever and I utterly hate working with it. * Its client/server protocol is utterly useless in every way. How to parse the response to a command depends on which command you send which requires you to textually parse the command. * Like many other drivers, it conflates the communication protocol with the pooling facility. * The low-level protocol handling is wrong, which leads to a wrong timeout structure in all of the code base. ... and many more things I dislike. However, since so many are stuck with MySQL, there is a genuine need to maintain a driver, however weak it happens to be. Currently, the project is slipping through my fingers. There are several pull requests which has not been addressed but ought to be. If I were involved with the driver at work, I would be able to set aside some time for the maintenance, but frankly, it is not something I am willing to do unless I get paid for it. The code base simply burns me out. So I am throwing the towel into the ring. Don't look at this as a bad thing! What I am doing is to make it official that I have no intention to maintain the driver and someone else has to do it. This is a much better outcome in the long run than me pretending to maintain something which I really don't. I better do this than keep people in limbo as to where the code base is headed. I hope someone picks up the reins. There are several pull requests which all have to be merged. There is a nice test suite, and with little work, Travis CI support is salvageable. And I am here to answer any questions someone might have. In general, however, database drivers are in a very sad state in Erlang-land. But fixing that requires someone who will write a generic driver infrastructure and not doing so in anger because there is another project which needs to be addressed. Loic Hoguin's "bank" project is promising, but I don't know where it stands. And now, to address other projects for which I still do have the passion :) Code is here: github.com/Eonblast/emysql -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Wed Nov 19 15:53:47 2014 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Wed, 19 Nov 2014 16:53:47 +0200 Subject: [erlang-questions] Emysql: a cry for help (Harbinger is releasing control) In-Reply-To: References: Message-ID: <546CAEFB.30407@ninenines.eu> On 11/19/2014 04:40 PM, Jesper Louis Andersen wrote: > In general, however, database drivers are in a very sad state in > Erlang-land. But fixing that requires someone who will write a generic > driver infrastructure and not doing so in anger because there is another > project which needs to be addressed. Loic Hoguin's "bank" project is > promising, but I don't know where it stands. Pretty much the same as you, I don't need to access databases so it is still nothing more than a prototype at this point, with work on a future better interface that has documentation but no code. It's in limbo until further notice. I would welcome patches working toward the better interface and possibly added drivers if someone is interested, and would discuss and review things. Most of my time today is spent on Cowboy 2 and erlang.mk, and when I'm done with these, the Websocket work on Gun will take priority (but that should not take too long). I will probably not get back to Bank until OTP 18.0 (and to be honest, maps changed everything so there was no real interest from me to work on Bank until we got those in Erlang properly). In short: it's still on my radar, just not at the top of the list. -- Lo?c Hoguin http://ninenines.eu From g@REDACTED Wed Nov 19 16:39:26 2014 From: g@REDACTED (Garrett Smith) Date: Wed, 19 Nov 2014 09:39:26 -0600 Subject: [erlang-questions] Emysql: a cry for help (Harbinger is releasing control) In-Reply-To: References: Message-ID: On Wed, Nov 19, 2014 at 8:40 AM, Jesper Louis Andersen wrote: > Hi Everyone, > > You may know that I was maintaining the Emysql code base for Erlang for a > while. It is a driver for MySQL which seem to be quite popular. However, as > of late, I have not really been maintaining it anymore, and I see no reason > to continuing doing so: > > * I have no passion for MySQL. It is the worst database in existence ever > and I utterly hate working with it. > * Its client/server protocol is utterly useless in every way. How to parse > the response to a command depends on which command you send which requires > you to textually parse the command. > * Like many other drivers, it conflates the communication protocol with the > pooling facility. > * The low-level protocol handling is wrong, which leads to a wrong timeout > structure in all of the code base. > > ... and many more things I dislike. However, since so many are stuck with > MySQL, there is a genuine need to maintain a driver, however weak it happens > to be. > > Currently, the project is slipping through my fingers. There are several > pull requests which has not been addressed but ought to be. If I were > involved with the driver at work, I would be able to set aside some time for > the maintenance, but frankly, it is not something I am willing to do unless > I get paid for it. The code base simply burns me out. > > So I am throwing the towel into the ring. Don't look at this as a bad thing! > What I am doing is to make it official that I have no intention to maintain > the driver and someone else has to do it. This is a much better outcome in > the long run than me pretending to maintain something which I really don't. > I better do this than keep people in limbo as to where the code base is > headed. > > I hope someone picks up the reins. There are several pull requests which all > have to be merged. There is a nice test suite, and with little work, Travis > CI support is salvageable. And I am here to answer any questions someone > might have. > > In general, however, database drivers are in a very sad state in > Erlang-land. But fixing that requires someone who will write a generic > driver infrastructure and not doing so in anger because there is another > project which needs to be addressed. Loic Hoguin's "bank" project is > promising, but I don't know where it stands. > > And now, to address other projects for which I still do have the passion :) > > Code is here: > > github.com/Eonblast/emysql I have zero cycles to spend on this, so I volunteer! This is indeed important and I agree completely with you on all counts. This driver is a product of evolution and is fit for a purpose, but, like the dinosaurs before it, I fear it's headed for extinction. It's history is a fascinating chronicle of how things seem to work in Erlang land - and here we are :) Proposing a direction at this point is just talk (from me) since I really don't have a ton of time for this, but I have a spark of interest, which is something. I think this could be an opportunity to move a generic DB interface forward, which seems like a fun project. That's as much as I can say now about the future. I believe I have at least one, maybe two, pending pull requests for this library, so I can take a look at the others as well. I'd be happy to take on the repo at gar1t, but I have a sinking feeling that I'll just be in the same state as you. Not quite as soul crushed, but sounds like that's just a matter of time. If there are any others we can fight for who wants this less, otherwise I'll grab the latest, merge my own PRs and reject all others as "overly complex and out of scope" (kidding, maybe). Who else? From gumm@REDACTED Wed Nov 19 17:18:06 2014 From: gumm@REDACTED (Jesse Gumm) Date: Wed, 19 Nov 2014 10:18:06 -0600 Subject: [erlang-questions] Emysql: a cry for help (Harbinger is releasing control) In-Reply-To: References: Message-ID: Oops, that was meant to be a reply all. -- Jesse Gumm Owner, Sigma Star Systems 414.940.4866 || sigma-star.com || @jessegumm On Nov 19, 2014 10:16 AM, "Jesse Gumm" wrote: > What about creating an organization on github called emysql and move the > repo over to that, promoting a handful of team members whose contributions > you trust to be acceptable. > > Then we don't have to figure out whose repo is "official" every couple of > years, just promote contributors to the repo and magical things happen. > > -Jesse > > -- > Jesse Gumm > Owner, Sigma Star Systems > 414.940.4866 || sigma-star.com || @jessegumm > On Nov 19, 2014 9:39 AM, "Garrett Smith" wrote: > >> On Wed, Nov 19, 2014 at 8:40 AM, Jesper Louis Andersen >> wrote: >> > Hi Everyone, >> > >> > You may know that I was maintaining the Emysql code base for Erlang for >> a >> > while. It is a driver for MySQL which seem to be quite popular. >> However, as >> > of late, I have not really been maintaining it anymore, and I see no >> reason >> > to continuing doing so: >> > >> > * I have no passion for MySQL. It is the worst database in existence >> ever >> > and I utterly hate working with it. >> > * Its client/server protocol is utterly useless in every way. How to >> parse >> > the response to a command depends on which command you send which >> requires >> > you to textually parse the command. >> > * Like many other drivers, it conflates the communication protocol with >> the >> > pooling facility. >> > * The low-level protocol handling is wrong, which leads to a wrong >> timeout >> > structure in all of the code base. >> > >> > ... and many more things I dislike. However, since so many are stuck >> with >> > MySQL, there is a genuine need to maintain a driver, however weak it >> happens >> > to be. >> > >> > Currently, the project is slipping through my fingers. There are several >> > pull requests which has not been addressed but ought to be. If I were >> > involved with the driver at work, I would be able to set aside some >> time for >> > the maintenance, but frankly, it is not something I am willing to do >> unless >> > I get paid for it. The code base simply burns me out. >> > >> > So I am throwing the towel into the ring. Don't look at this as a bad >> thing! >> > What I am doing is to make it official that I have no intention to >> maintain >> > the driver and someone else has to do it. This is a much better outcome >> in >> > the long run than me pretending to maintain something which I really >> don't. >> > I better do this than keep people in limbo as to where the code base is >> > headed. >> > >> > I hope someone picks up the reins. There are several pull requests >> which all >> > have to be merged. There is a nice test suite, and with little work, >> Travis >> > CI support is salvageable. And I am here to answer any questions someone >> > might have. >> > >> > In general, however, database drivers are in a very sad state in >> > Erlang-land. But fixing that requires someone who will write a generic >> > driver infrastructure and not doing so in anger because there is another >> > project which needs to be addressed. Loic Hoguin's "bank" project is >> > promising, but I don't know where it stands. >> > >> > And now, to address other projects for which I still do have the >> passion :) >> > >> > Code is here: >> > >> > github.com/Eonblast/emysql >> >> I have zero cycles to spend on this, so I volunteer! >> >> This is indeed important and I agree completely with you on all >> counts. This driver is a product of evolution and is fit for a >> purpose, but, like the dinosaurs before it, I fear it's headed for >> extinction. >> >> It's history is a fascinating chronicle of how things seem to work in >> Erlang land - and here we are :) >> >> Proposing a direction at this point is just talk (from me) since I >> really don't have a ton of time for this, but I have a spark of >> interest, which is something. I think this could be an opportunity to >> move a generic DB interface forward, which seems like a fun project. >> That's as much as I can say now about the future. >> >> I believe I have at least one, maybe two, pending pull requests for >> this library, so I can take a look at the others as well. I'd be happy >> to take on the repo at gar1t, but I have a sinking feeling that I'll >> just be in the same state as you. Not quite as soul crushed, but >> sounds like that's just a matter of time. >> >> If there are any others we can fight for who wants this less, >> otherwise I'll grab the latest, merge my own PRs and reject all others >> as "overly complex and out of scope" (kidding, maybe). >> >> Who else? >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mjtruog@REDACTED Wed Nov 19 17:37:03 2014 From: mjtruog@REDACTED (Michael Truog) Date: Wed, 19 Nov 2014 08:37:03 -0800 Subject: [erlang-questions] Emysql: a cry for help (Harbinger is releasing control) In-Reply-To: References: Message-ID: <546CC72F.4070401@gmail.com> On 11/19/2014 06:40 AM, Jesper Louis Andersen wrote: > Hi Everyone, > > You may know that I was maintaining the Emysql code base for Erlang for a while. It is a driver for MySQL which seem to be quite popular. However, as of late, I have not really been maintaining it anymore, and I see no reason to continuing doing so: > > * I have no passion for MySQL. It is the worst database in existence ever and I utterly hate working with it. > * Its client/server protocol is utterly useless in every way. How to parse the response to a command depends on which command you send which requires you to textually parse the command. > * Like many other drivers, it conflates the communication protocol with the pooling facility. > * The low-level protocol handling is wrong, which leads to a wrong timeout structure in all of the code base. > > ... and many more things I dislike. However, since so many are stuck with MySQL, there is a genuine need to maintain a driver, however weak it happens to be. > > Currently, the project is slipping through my fingers. There are several pull requests which has not been addressed but ought to be. If I were involved with the driver at work, I would be able to set aside some time for the maintenance, but frankly, it is not something I am willing to do unless I get paid for it. The code base simply burns me out. > > So I am throwing the towel into the ring. Don't look at this as a bad thing! What I am doing is to make it official that I have no intention to maintain the driver and someone else has to do it. This is a much better outcome in the long run than me pretending to maintain something which I really don't. I better do this than keep people in limbo as to where the code base is headed. > > I hope someone picks up the reins. There are several pull requests which all have to be merged. There is a nice test suite, and with little work, Travis CI support is salvageable. And I am here to answer any questions someone might have. > > In general, however, database drivers are in a very sad state in Erlang-land. But fixing that requires someone who will write a generic driver infrastructure and not doing so in anger because there is another project which needs to be addressed. Loic Hoguin's "bank" project is promising, but I don't know where it stands. > > And now, to address other projects for which I still do have the passion :) > > Code is here: > > github.com/Eonblast/emysql > > -- > J. > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions CloudI (http://cloudi.org) has emysql integration along with three postgres drivers and a common interface (in addition to various other database driver integration). - Michael -------------- next part -------------- An HTML attachment was scrubbed... URL: From mjtruog@REDACTED Wed Nov 19 18:08:58 2014 From: mjtruog@REDACTED (Michael Truog) Date: Wed, 19 Nov 2014 09:08:58 -0800 Subject: [erlang-questions] Emysql: a cry for help (Harbinger is releasing control) In-Reply-To: References: Message-ID: <546CCEAA.2060207@gmail.com> On 11/19/2014 08:18 AM, Jesse Gumm wrote: > On Nov 19, 2014 10:16 AM, "Jesse Gumm" > wrote: > > What about creating an organization on github called emysql and move the repo over to that, promoting a handful of team members whose contributions you trust to be acceptable. > > Then we don't have to figure out whose repo is "official" every couple of years, just promote contributors to the repo and magical things happen. > > -Jesse > I agree. This approach makes the most sense. -------------- next part -------------- An HTML attachment was scrubbed... URL: From g@REDACTED Wed Nov 19 18:16:14 2014 From: g@REDACTED (Garrett Smith) Date: Wed, 19 Nov 2014 11:16:14 -0600 Subject: [erlang-questions] Emysql: a cry for help (Harbinger is releasing control) In-Reply-To: <546CCEAA.2060207@gmail.com> References: <546CCEAA.2060207@gmail.com> Message-ID: You realize this will end the long standing sense of adventure and, finally, accomplishment as people search the annals and repos to find "the one". On Nov 19, 2014 11:09 AM, "Michael Truog" wrote: > On 11/19/2014 08:18 AM, Jesse Gumm wrote: > > On Nov 19, 2014 10:16 AM, "Jesse Gumm" wrote: > >> What about creating an organization on github called emysql and move the >> repo over to that, promoting a handful of team members whose contributions >> you trust to be acceptable. >> >> Then we don't have to figure out whose repo is "official" every couple of >> years, just promote contributors to the repo and magical things happen. >> >> -Jesse >> > > I agree. This approach makes the most sense. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From t@REDACTED Wed Nov 19 18:18:20 2014 From: t@REDACTED (Tristan Sloughter) Date: Wed, 19 Nov 2014 11:18:20 -0600 Subject: [erlang-questions] Emysql: a cry for help (Harbinger is releasing control) In-Reply-To: References: <546CCEAA.2060207@gmail.com> Message-ID: <1416417500.697972.192980009.1DC0790A@webmail.messagingengine.com> I hope this helps with that too :) http://packages.rebar3.org/package/emysql/0.4.1 On Wed, Nov 19, 2014, at 11:16 AM, Garrett Smith wrote: > You realize this will end the long standing sense of adventure and, > finally, accomplishment as people search the annals and repos to find > "the one". > On Nov 19, 2014 11:09 AM, "Michael Truog" wrote: >> On 11/19/2014 08:18 AM, Jesse Gumm wrote: >>> On Nov 19, 2014 10:16 AM, "Jesse Gumm" wrote: >>>> What about creating an organization on github called emysql and move the repo over to that, promoting a handful of team members whose contributions you trust to be acceptable. >>>> Then we don't have to figure out whose repo is "official" every couple of years, just promote contributors to the repo and magical things happen. >>>> -Jesse >> >> I agree. This approach makes the most sense. >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > _________________________________________________ > erlang-questions mailing list erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From g@REDACTED Wed Nov 19 18:39:07 2014 From: g@REDACTED (Garrett Smith) Date: Wed, 19 Nov 2014 11:39:07 -0600 Subject: [erlang-questions] Emysql: a cry for help (Harbinger is releasing control) In-Reply-To: <546CCEAA.2060207@gmail.com> References: <546CCEAA.2060207@gmail.com> Message-ID: On Wed, Nov 19, 2014 at 11:08 AM, Michael Truog wrote: > On 11/19/2014 08:18 AM, Jesse Gumm wrote: > > On Nov 19, 2014 10:16 AM, "Jesse Gumm" wrote: >> >> What about creating an organization on github called emysql and move the >> repo over to that, promoting a handful of team members whose contributions >> you trust to be acceptable. >> >> Then we don't have to figure out whose repo is "official" every couple of >> years, just promote contributors to the repo and magical things happen. >> >> -Jesse > > > I agree. This approach makes the most sense. One other option - and this one would minimize moving parts until the library is (wave hands) re-launched as something better - is for Jesper to just add some of us as contributors to deal with the PRs. From raould@REDACTED Wed Nov 19 19:26:45 2014 From: raould@REDACTED (Raoul Duke) Date: Wed, 19 Nov 2014 10:26:45 -0800 Subject: [erlang-questions] Distributed application and netsplit In-Reply-To: References: <148C0C12-566F-4C43-B77E-B5A77D4260E9@vailsys.com> <802A948E-4671-4D4F-A7B3-552F15EDAE47@vailsys.com> Message-ID: > keeps running on both nodes after recovery from netsplit. However I > doubt it will be considered a bug. Most likely, it is missing feature > to be able to fix the situation explicitly. The wording continues to worry me. There is not feature that will fix the situation in the largest sense. You acknowledge that already so I know you know it. :-) So what I would urge us to say is that we want there to be a signal after the network heals. Expectation of how soon after the healing the signal will get through, however, must be carefully constrained. I assume what people would do is a "simply" ping constantly. It could be a star-network ping, it could be a P2P ping, it could be a broadcast ping... Even with this, there are many ways to skin the chicken. It will depend on each situation: What the application's network is like in the first place. I would hazard to guess. From rpettit@REDACTED Wed Nov 19 21:49:19 2014 From: rpettit@REDACTED (Rick Pettit) Date: Wed, 19 Nov 2014 14:49:19 -0600 Subject: [erlang-questions] Distributed application and netsplit In-Reply-To: References: <148C0C12-566F-4C43-B77E-B5A77D4260E9@vailsys.com> <802A948E-4671-4D4F-A7B3-552F15EDAE47@vailsys.com> Message-ID: Raoul, The problem is, once the net split has been detected, how then to manually shutdown the unwanted application instance(s) in such a way that the distributed application failover continues to work from that point forward. The OP already has application-specific code which is able to detect that a net split / split brain scenario has occurred. Furthermore, the OP is ?ok? writing application specific code to attempt to resolve the situation at that point. The question is, what is the ?best? way to go about doing that? Is it not possible to shutdown all of the unwanted application instances in such a way that distributed application failover continues to work from that point forward? Or is it really necessary to restart *all* of the nodes on which the distributed application may run? Clearly that would be, well, less than ideal. Then again, so would rewriting dist_ac... -Rick On Nov 19, 2014, at 12:26 PM, Raoul Duke wrote: >> keeps running on both nodes after recovery from netsplit. However I >> doubt it will be considered a bug. Most likely, it is missing feature >> to be able to fix the situation explicitly. > > > The wording continues to worry me. There is not feature that will fix > the situation in the largest sense. You acknowledge that already so I > know you know it. :-) So what I would urge us to say is that we want > there to be a signal after the network heals. Expectation of how soon > after the healing the signal will get through, however, must be > carefully constrained. > > I assume what people would do is a "simply" ping constantly. It could > be a star-network ping, it could be a P2P ping, it could be a > broadcast ping... Even with this, there are many ways to skin the > chicken. It will depend on each situation: What the application's > network is like in the first place. I would hazard to guess. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From raould@REDACTED Wed Nov 19 22:18:08 2014 From: raould@REDACTED (Raoul Duke) Date: Wed, 19 Nov 2014 13:18:08 -0800 Subject: [erlang-questions] Distributed application and netsplit In-Reply-To: References: <148C0C12-566F-4C43-B77E-B5A77D4260E9@vailsys.com> <802A948E-4671-4D4F-A7B3-552F15EDAE47@vailsys.com> Message-ID: > The problem is, once the net split has been detected, how then to manually shutdown the unwanted application instance(s) in such a way that the distributed application failover continues to work from that point forward. > > The OP already has application-specific code which is able to detect that a net split / split brain scenario has occurred. > > Furthermore, the OP is ?ok? writing application specific code to attempt to resolve the situation at that point. > > The question is, what is the ?best? way to go about doing that? thanks. ok so it sounds like more clearly what we want is to know that OTP has been designed / will be updated to handle this at the level of it "going back to normal" at the OTP level, leaving the actual "sync up" part to the individual applications that use OTP. From jesper.louis.andersen@REDACTED Wed Nov 19 22:29:52 2014 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Wed, 19 Nov 2014 22:29:52 +0100 Subject: [erlang-questions] Emysql: a cry for help (Harbinger is releasing control) In-Reply-To: References: <546CCEAA.2060207@gmail.com> Message-ID: It sounds really awesome you would take over the work by volunteering with no time at all :) However, much as I would like to add you to the repository, it is not mine. It is owned by Henning Diedrich, but I can vouch for you to be added if need be. Also, the immediate work that needs done is to triage all the open PRs and pull those in which makes sense. I don't want to be the judge too much on the direction to take though my style has mostly been to stay backwards compatible at all costs. Mostly because one has to strike a balance deciding if this is development or maintenance. On Wed, Nov 19, 2014 at 6:39 PM, Garrett Smith wrote: > On Wed, Nov 19, 2014 at 11:08 AM, Michael Truog wrote: > > On 11/19/2014 08:18 AM, Jesse Gumm wrote: > > > > On Nov 19, 2014 10:16 AM, "Jesse Gumm" wrote: > >> > >> What about creating an organization on github called emysql and move the > >> repo over to that, promoting a handful of team members whose > contributions > >> you trust to be acceptable. > >> > >> Then we don't have to figure out whose repo is "official" every couple > of > >> years, just promote contributors to the repo and magical things happen. > >> > >> -Jesse > > > > > > I agree. This approach makes the most sense. > > One other option - and this one would minimize moving parts until the > library is (wave hands) re-launched as something better - is for > Jesper to just add some of us as contributors to deal with the PRs. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From imantc@REDACTED Wed Nov 19 22:32:02 2014 From: imantc@REDACTED (Imants Cekusins) Date: Wed, 19 Nov 2014 22:32:02 +0100 Subject: [erlang-questions] Distributed application and netsplit In-Reply-To: References: <148C0C12-566F-4C43-B77E-B5A77D4260E9@vailsys.com> <802A948E-4671-4D4F-A7B3-552F15EDAE47@vailsys.com> Message-ID: This is a small quirk. It is easy to fine-tune to specific requirements once you know about it. Thank you Karolis for highlighting this. -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthias@REDACTED Wed Nov 19 23:18:43 2014 From: matthias@REDACTED (Matthias Lang) Date: Wed, 19 Nov 2014 23:18:43 +0100 Subject: [erlang-questions] Calling for help to improve contents on erlang.org In-Reply-To: <546C6024.6020409@ninenines.eu> References: <20141118225906.GA32465@corelatus.se> <546C6024.6020409@ninenines.eu> Message-ID: <20141119221843.GC6090@corelatus.se> Hi, > I'm probbly dumb. I opened the faq, failed to see a link to github, closed ... > ... finished reading your email, noticed some people contributed by email, > tried to find it and finally found the link to github deep inside a question I started with this back in 1998, before git existed. When I moved it to github, I did it cautiously. One thought behind yesterday's mail was that it's been five years since I did that and maybe a higher profile wouldn't hurt. Matt From rvirding@REDACTED Thu Nov 20 03:53:50 2014 From: rvirding@REDACTED (Robert Virding) Date: Thu, 20 Nov 2014 03:53:50 +0100 Subject: [erlang-questions] Dialyzer type/spec info question In-Reply-To: References: Message-ID: I have now push the first version of dialyzer and LFE on to github in the lfe-dev branch. See the mailing list for more information https://groups.google.com/forum/#!forum/lisp-flavoured-erlang. It is possible but dialyzer is not giving up without a fight. Robert On 18 November 2014 11:36, Peer Stritzinger wrote: > On 2014-11-17 17:26:01 +0000, Robert Virding said: > > The reason I would like to be able to handle multiple file types is to >> simplify checking systems with parts which are written in different >> languages. >> > > +1 > > This would be useful to my IEC61499 PLC language compiler too, especially > since I have some native (here meaning in Erlang) implemented behaviour > callback modules which can then be checked if they implement a matching > interface to the PLC interface part. > > And since I'm also generating core like LFE opening up this interface to > Dialyzer would help me greatly. > > Especially so since I'm compiling a statically typed language anyway and > could derive the typespecs of the underlying Erlang types from this > information. > > Cheers, > -- Peer > > > >> Robert >> >> >> On 17 November 2014 18:19, Robert Virding wrote: >> Something much naughtier and more basic. >> >> - Make LFE local copies of the relevant dialyzer modules which have been >> hacked to be able to handle LFE files as input (with a new --lfe option) >> - Run dialyzer using -pa to push the local module directory first so as >> to use these special modules instead of the standard ones >> >> This will make it possible to use dialyzer on LFE files and other .erl or >> .beam files as well. Maybe with a little more hacking fix have multiple >> file types. Of course if dialyzer was to be opened up so that it would use >> the filename extension to pick the right versions of the functions Stavros >> mentioned then this would be much simpler. Wink, wink, nudge nudge. :-) >> >> This is my plan anyway. >> >> Robert >> >> P.S. Having LFE generate erlang AST has never been an option as it is >> very close to core and it would take a lot of effort to go back to erlang >> which would then be undone by the erlang compiler. >> >> >> On 17 November 2014 17:49, Jos? Valim >> wrote: >> Robert, what would a specialized version of dialyzer entail? A fork? A >> specific front-end? Curious. :) >> >> On Monday, November 17, 2014, Robert Virding wrote: >> Ah, so then in principle I can make my own specialised versions of the >> dialyzer load modules which takes the core which the LFE compiler generates >> and extracts the core and the type/spec data dialyzer needs. This without >> any Erlang AST which LFE never generates anyway. >> >> Now we are getting somewhere useful. With this we don't really need to be >> able to store core in the .beam files if we can accept using source files >> as input. >> >> Robert >> >> >> On 17 November 2014 17:32, Stavros Aronis wrote: >> For .erl files, dialyzer calls the compiler with 'to_pp', which stops the >> compilation before code is converted to Core, and reads attributes from >> there. >> >> Judging from v3_core, line 170 the two representations should be >> compatible. >> >> /Stavros >> >> On Mon, Nov 17, 2014 at 5:11 PM, Robert Virding >> wrote: >> From where does it get it if I have .erl files as input? From the AST as >> well? Do you know if there is any difference in the data itself between the >> AST and core? I am guessing not but want to check. If there is no >> difference then the info could be gotten from Core. This would make it >> easier to use dialyzer together with LFE. >> >> Robert >> >> >> On 17 November 2014 17:03, Stavros Aronis wrote: >> Hi Robert, >> >> Dialyzer gets this info from beam_lib:chunks(File, [abstract_code]) which >> corresponds to the AST and is included in beam files if +debug_info is used >> while compiling. >> >> To my knowledge there is no reason to not let sources be combined (except >> perhaps when building a PLT), but the implementation seems to require >> uniformity. >> >> Relevant functions: dialyzer_utils:get_abstract_code_from_{src,beam}/{1,2}, >> get_spec_info/3, get_record_and_type_info/1. >> >> Regards, >> >> Stavros >> >> On Mon, Nov 17, 2014 at 4:53 PM, Robert Virding >> wrote: >> From where does dialyzer get the user added type and spec info? From the >> AST, or from Core erlang which contains the same type/spec data? And why? >> Can I control it? >> >> An extra question: why doesn't dialyzer allow me to mix input from both >> .erl and .beam files? Or does it and I have missed that? >> >> Robert >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> >> >> >> >> >> -- >> >> >> Jos? Valim >> www.plataformatec.com.br >> Skype: jv.ptec >> Founder and Lead Developer >> > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From msekender@REDACTED Thu Nov 20 04:49:39 2014 From: msekender@REDACTED (Zulfiqer Sekender) Date: Thu, 20 Nov 2014 03:49:39 +0000 Subject: [erlang-questions] Distributed application and netsplit In-Reply-To: References: , , , <148C0C12-566F-4C43-B77E-B5A77D4260E9@vailsys.com>, , , , , , <802A948E-4671-4D4F-A7B3-552F15EDAE47@vailsys.com>, , , , , , , Message-ID: Hi, I am too waiting for this. Could you please let me know in which version of Erlang/OTP, this will be included and approximately when that version can be released? Thanks. - Zulfiqer > Date: Wed, 19 Nov 2014 13:18:08 -0800 > From: raould@REDACTED > To: erlang-questions@REDACTED > Subject: Re: [erlang-questions] Distributed application and netsplit > > > The problem is, once the net split has been detected, how then to manually shutdown the unwanted application instance(s) in such a way that the distributed application failover continues to work from that point forward. > > > > The OP already has application-specific code which is able to detect that a net split / split brain scenario has occurred. > > > > Furthermore, the OP is ?ok? writing application specific code to attempt to resolve the situation at that point. > > > > The question is, what is the ?best? way to go about doing that? > > > > thanks. > > ok so it sounds like more clearly what we want is to know that OTP has > been designed / will be updated to handle this at the level of it > "going back to normal" at the OTP level, leaving the actual "sync up" > part to the individual applications that use OTP. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alexey.Saltanov@REDACTED Thu Nov 20 11:00:19 2014 From: Alexey.Saltanov@REDACTED (Saltanov, Alexey) Date: Thu, 20 Nov 2014 10:00:19 +0000 Subject: [erlang-questions] r16b02 odbc-2.10.17 memmory corruption. Message-ID: <0589296B5BCDDA4A8F5CEB3F8739A815E178716B@SRV-EXCHMBX2.billing.ru> I use Erlang OTP r16b02 with odbc-2.10.17, unixODBC 2.3.1, Oracle Client 11.2.3 on the Red Hat Enterprise Linux Server 6.3 x86_64. And I get some glibc error messages: *** glibc detected *** /home/user/app1/lib/odbc-2.10.17/priv/bin/odbcserver: free(): invalid next size (fast): 0x0000000002192490 *** *** glibc detected *** /home/user/app1/lib/odbc-2.10.17/priv/bin/odbcserver: corrupted double-linked list: 0x00000000013d7110 *** Often odbcserver crashes... Stacktrace 1: Core was generated by `/home/user/app1/lib/odbc-2.10.1'. Program terminated with signal SIGABRT, Aborted. (gdb) bt #0 0x0000003c4160f3cb in raise () from /lib64/libpthread.so.0 #1 0x00007f9d98336442 in skgesigOSCrash () from /u01/app/oracle/product/11.2.0/cl11203/lib/libclntsh.so.11.1 #2 0x00007f9d985d8cbd in kpeDbgSignalHandler () from /u01/app/oracle/product/11.2.0/cl11203/lib/libclntsh.so.11.1 #3 0x00007f9d98336652 in skgesig_sigactionHandler () from /u01/app/oracle/product/11.2.0/cl11203/lib/libclntsh.so.11.1 #4 #5 0x0000003c412328a5 in raise () from /lib64/libc.so.6 #6 0x0000003c41234085 in abort () from /lib64/libc.so.6 #7 0x0000003c4126fa37 in __libc_message () from /lib64/libc.so.6 #8 0x0000003c41275366 in malloc_printerr () from /lib64/libc.so.6 #9 0x0000003c41275770 in malloc_consolidate () from /lib64/libc.so.6 #10 0x0000003c41278605 in _int_malloc () from /lib64/libc.so.6 #11 0x0000003c41279826 in calloc () from /lib64/libc.so.6 #12 0x00007f9d9a08115c in __alloc_stmt () at __handles.c:771 #13 0x00007f9d9a04d1b5 in __SQLAllocHandle (handle_type=, input_handle=0x12e4970, output_handle=0x7fff8b2ca130, requested_version=0) at SQLAllocHandle.c:580 #14 0x0000000000403bac in init_param_statement (status=, state=, num_param_values=, cols=) at odbcserver.c:2372 #15 db_param_query (buffer=0x13d7281 "\203h\003k", state=0x7fff8b2ca120) at odbcserver.c:859 #16 0x0000000000405332 in handle_db_request (state=, reqstring=) at odbcserver.c:424 #17 database_handler (port=) at odbcserver.c:371 #18 0x00000000004059ce in main () at odbcserver.c:296 Stacktrace2: Core was generated by `/home/user/app1/lib/odbc-2.10.1'. Program terminated with signal SIGABRT, Aborted. (gdb) bt #0 0x0000003c4160f3cb in raise () from /lib64/libpthread.so.0 #1 0x00007f3472c1e442 in skgesigOSCrash () from /u01/app/oracle/product/11.2.0/cl11203/lib/libclntsh.so.11.1 #2 0x00007f3472ec0cbd in kpeDbgSignalHandler () from /u01/app/oracle/product/11.2.0/cl11203/lib/libclntsh.so.11.1 #3 0x00007f3472c1e652 in skgesig_sigactionHandler () from /u01/app/oracle/product/11.2.0/cl11203/lib/libclntsh.so.11.1 #4 #5 0x0000003c412328a5 in raise () from /lib64/libc.so.6 #6 0x0000003c41234085 in abort () from /lib64/libc.so.6 #7 0x0000003c4126fa37 in __libc_message () from /lib64/libc.so.6 #8 0x0000003c41275366 in malloc_printerr () from /lib64/libc.so.6 #9 0x0000003c41277e93 in _int_free () from /lib64/libc.so.6 #10 0x00007f3473b8c37a in MEMFree () from /u01/app/oracle/product/11.2.0/cl11203/lib/libsqora.so.11.1 #11 0x00007f3473b83908 in bcoResetParam () from /u01/app/oracle/product/11.2.0/cl11203/lib/libsqora.so.11.1 #12 0x00007f3473b83b00 in bcoResetParams () from /u01/app/oracle/product/11.2.0/cl11203/lib/libsqora.so.11.1 #13 0x00007f3473b3a759 in bcoSQLFreeStmt () from /u01/app/oracle/product/11.2.0/cl11203/lib/libsqora.so.11.1 #14 0x00007f3473b90a90 in bccSQLFreeStmtLckd () from /u01/app/oracle/product/11.2.0/cl11203/lib/libsqora.so.11.1 #15 0x00007f3473b90864 in bccSQLFreeStmt () from /u01/app/oracle/product/11.2.0/cl11203/lib/libsqora.so.11.1 #16 0x00007f3473b9157e in SQLFreeStmt () from /u01/app/oracle/product/11.2.0/cl11203/lib/libsqora.so.11.1 #17 0x00007f34749462f8 in SQLFreeStmt (statement_handle=0x2195480, option=3) at SQLFreeStmt.c:247 #18 0x0000000000403f62 in db_param_query (buffer=, state=0x7fff2dda97d0) at odbcserver.c:911 #19 0x0000000000405332 in handle_db_request (state=, reqstring=) at odbcserver.c:424 #20 database_handler (port=) at odbcserver.c:371 #21 0x00000000004059ce in main () at odbcserver.c:296 Is it known bug? It's looks like odbcserver has invalid memory usage code and r17.3 has no significant changes in the odbc. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: erlang.log Type: application/octet-stream Size: 13031 bytes Desc: erlang.log URL: From benmmurphy@REDACTED Thu Nov 20 12:33:16 2014 From: benmmurphy@REDACTED (Ben Murphy) Date: Thu, 20 Nov 2014 11:33:16 +0000 Subject: [erlang-questions] r16b02 odbc-2.10.17 memmory corruption. In-Reply-To: <0589296B5BCDDA4A8F5CEB3F8739A815E178716B@SRV-EXCHMBX2.billing.ru> References: <0589296B5BCDDA4A8F5CEB3F8739A815E178716B@SRV-EXCHMBX2.billing.ru> Message-ID: The erlang ODBC API lets you shoot yourself in the foot. When you use param_query and {sql_varchar, X} (or other types that have variable size) you MUST ensure that X is greater than the number of bytes in your bound parameters (this might be slightly different for wchar). If it is less than or equal then it will cause heap corruption. On Thu, Nov 20, 2014 at 10:00 AM, Saltanov, Alexey wrote: > I use Erlang OTP r16b02 with odbc-2.10.17, unixODBC 2.3.1, Oracle Client > 11.2.3 on the Red Hat Enterprise Linux Server 6.3 x86_64. > > And I get some glibc error messages: > > *** glibc detected *** /home/user/app1/lib/odbc-2.10.17/priv/bin/odbcserver: > free(): invalid next size (fast): 0x0000000002192490 *** > > *** glibc detected *** /home/user/app1/lib/odbc-2.10.17/priv/bin/odbcserver: > corrupted double-linked list: 0x00000000013d7110 *** > > > > Often odbcserver crashes? > > > > Stacktrace 1: > > Core was generated by `/home/user/app1/lib/odbc-2.10.1'. > > Program terminated with signal SIGABRT, Aborted. > > (gdb) bt > > #0 0x0000003c4160f3cb in raise () from /lib64/libpthread.so.0 > > #1 0x00007f9d98336442 in skgesigOSCrash () from > /u01/app/oracle/product/11.2.0/cl11203/lib/libclntsh.so.11.1 > > #2 0x00007f9d985d8cbd in kpeDbgSignalHandler () from > /u01/app/oracle/product/11.2.0/cl11203/lib/libclntsh.so.11.1 > > #3 0x00007f9d98336652 in skgesig_sigactionHandler () from > /u01/app/oracle/product/11.2.0/cl11203/lib/libclntsh.so.11.1 > > #4 > > #5 0x0000003c412328a5 in raise () from /lib64/libc.so.6 > > #6 0x0000003c41234085 in abort () from /lib64/libc.so.6 > > #7 0x0000003c4126fa37 in __libc_message () from /lib64/libc.so.6 > > #8 0x0000003c41275366 in malloc_printerr () from /lib64/libc.so.6 > > #9 0x0000003c41275770 in malloc_consolidate () from /lib64/libc.so.6 > > #10 0x0000003c41278605 in _int_malloc () from /lib64/libc.so.6 > > #11 0x0000003c41279826 in calloc () from /lib64/libc.so.6 > > #12 0x00007f9d9a08115c in __alloc_stmt () at __handles.c:771 > > #13 0x00007f9d9a04d1b5 in __SQLAllocHandle (handle_type=, > input_handle=0x12e4970, output_handle=0x7fff8b2ca130, requested_version=0) > at SQLAllocHandle.c:580 > > #14 0x0000000000403bac in init_param_statement (status=, > state=, num_param_values=, cols= out>) at odbcserver.c:2372 > > #15 db_param_query (buffer=0x13d7281 "\203h\003k", state=0x7fff8b2ca120) at > odbcserver.c:859 > > #16 0x0000000000405332 in handle_db_request (state=, > reqstring=) at odbcserver.c:424 > > #17 database_handler (port=) at odbcserver.c:371 > > #18 0x00000000004059ce in main () at odbcserver.c:296 > > > > Stacktrace2: > > Core was generated by `/home/user/app1/lib/odbc-2.10.1'. > > Program terminated with signal SIGABRT, Aborted. > > (gdb) bt > > #0 0x0000003c4160f3cb in raise () from /lib64/libpthread.so.0 > > #1 0x00007f3472c1e442 in skgesigOSCrash () from > /u01/app/oracle/product/11.2.0/cl11203/lib/libclntsh.so.11.1 > > #2 0x00007f3472ec0cbd in kpeDbgSignalHandler () from > /u01/app/oracle/product/11.2.0/cl11203/lib/libclntsh.so.11.1 > > #3 0x00007f3472c1e652 in skgesig_sigactionHandler () from > /u01/app/oracle/product/11.2.0/cl11203/lib/libclntsh.so.11.1 > > #4 > > #5 0x0000003c412328a5 in raise () from /lib64/libc.so.6 > > #6 0x0000003c41234085 in abort () from /lib64/libc.so.6 > > #7 0x0000003c4126fa37 in __libc_message () from /lib64/libc.so.6 > > #8 0x0000003c41275366 in malloc_printerr () from /lib64/libc.so.6 > > #9 0x0000003c41277e93 in _int_free () from /lib64/libc.so.6 > > #10 0x00007f3473b8c37a in MEMFree () from > /u01/app/oracle/product/11.2.0/cl11203/lib/libsqora.so.11.1 > > #11 0x00007f3473b83908 in bcoResetParam () from > /u01/app/oracle/product/11.2.0/cl11203/lib/libsqora.so.11.1 > > #12 0x00007f3473b83b00 in bcoResetParams () from > /u01/app/oracle/product/11.2.0/cl11203/lib/libsqora.so.11.1 > > #13 0x00007f3473b3a759 in bcoSQLFreeStmt () from > /u01/app/oracle/product/11.2.0/cl11203/lib/libsqora.so.11.1 > > #14 0x00007f3473b90a90 in bccSQLFreeStmtLckd () from > /u01/app/oracle/product/11.2.0/cl11203/lib/libsqora.so.11.1 > > #15 0x00007f3473b90864 in bccSQLFreeStmt () from > /u01/app/oracle/product/11.2.0/cl11203/lib/libsqora.so.11.1 > > #16 0x00007f3473b9157e in SQLFreeStmt () from > /u01/app/oracle/product/11.2.0/cl11203/lib/libsqora.so.11.1 > > #17 0x00007f34749462f8 in SQLFreeStmt (statement_handle=0x2195480, option=3) > at SQLFreeStmt.c:247 > > #18 0x0000000000403f62 in db_param_query (buffer=, > state=0x7fff2dda97d0) at odbcserver.c:911 > > #19 0x0000000000405332 in handle_db_request (state=, > reqstring=) at odbcserver.c:424 > > #20 database_handler (port=) at odbcserver.c:371 > > #21 0x00000000004059ce in main () at odbcserver.c:296 > > > > Is it known bug? > > It?s looks like odbcserver has invalid memory usage code and r17.3 has no > significant changes in the odbc. > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From Alexey.Saltanov@REDACTED Thu Nov 20 13:05:52 2014 From: Alexey.Saltanov@REDACTED (Saltanov, Alexey) Date: Thu, 20 Nov 2014 12:05:52 +0000 Subject: [erlang-questions] r16b02 odbc-2.10.17 memmory corruption. In-Reply-To: References: <0589296B5BCDDA4A8F5CEB3F8739A815E178716B@SRV-EXCHMBX2.billing.ru> Message-ID: <0589296B5BCDDA4A8F5CEB3F8739A815E17871BC@SRV-EXCHMBX2.billing.ru> Thanks a lot! -----Original Message----- From: Ben Murphy [mailto:benmmurphy@REDACTED] Sent: Thursday, November 20, 2014 2:33 PM To: Saltanov, Alexey Cc: erlang-questions@REDACTED Subject: Re: [erlang-questions] r16b02 odbc-2.10.17 memmory corruption. The erlang ODBC API lets you shoot yourself in the foot. When you use param_query and {sql_varchar, X} (or other types that have variable size) you MUST ensure that X is greater than the number of bytes in your bound parameters (this might be slightly different for wchar). If it is less than or equal then it will cause heap corruption. On Thu, Nov 20, 2014 at 10:00 AM, Saltanov, Alexey wrote: > I use Erlang OTP r16b02 with odbc-2.10.17, unixODBC 2.3.1, Oracle > Client > 11.2.3 on the Red Hat Enterprise Linux Server 6.3 x86_64. > > And I get some glibc error messages: > > *** glibc detected *** /home/user/app1/lib/odbc-2.10.17/priv/bin/odbcserver: > free(): invalid next size (fast): 0x0000000002192490 *** > > *** glibc detected *** /home/user/app1/lib/odbc-2.10.17/priv/bin/odbcserver: > corrupted double-linked list: 0x00000000013d7110 *** > > > > Often odbcserver crashes? > > > > Stacktrace 1: > > Core was generated by `/home/user/app1/lib/odbc-2.10.1'. > > Program terminated with signal SIGABRT, Aborted. > > (gdb) bt > > #0 0x0000003c4160f3cb in raise () from /lib64/libpthread.so.0 > > #1 0x00007f9d98336442 in skgesigOSCrash () from > /u01/app/oracle/product/11.2.0/cl11203/lib/libclntsh.so.11.1 > > #2 0x00007f9d985d8cbd in kpeDbgSignalHandler () from > /u01/app/oracle/product/11.2.0/cl11203/lib/libclntsh.so.11.1 > > #3 0x00007f9d98336652 in skgesig_sigactionHandler () from > /u01/app/oracle/product/11.2.0/cl11203/lib/libclntsh.so.11.1 > > #4 > > #5 0x0000003c412328a5 in raise () from /lib64/libc.so.6 > > #6 0x0000003c41234085 in abort () from /lib64/libc.so.6 > > #7 0x0000003c4126fa37 in __libc_message () from /lib64/libc.so.6 > > #8 0x0000003c41275366 in malloc_printerr () from /lib64/libc.so.6 > > #9 0x0000003c41275770 in malloc_consolidate () from /lib64/libc.so.6 > > #10 0x0000003c41278605 in _int_malloc () from /lib64/libc.so.6 > > #11 0x0000003c41279826 in calloc () from /lib64/libc.so.6 > > #12 0x00007f9d9a08115c in __alloc_stmt () at __handles.c:771 > > #13 0x00007f9d9a04d1b5 in __SQLAllocHandle (handle_type= out>, input_handle=0x12e4970, output_handle=0x7fff8b2ca130, > requested_version=0) at SQLAllocHandle.c:580 > > #14 0x0000000000403bac in init_param_statement (status= out>, state=, num_param_values=, > cols= out>) at odbcserver.c:2372 > > #15 db_param_query (buffer=0x13d7281 "\203h\003k", > state=0x7fff8b2ca120) at > odbcserver.c:859 > > #16 0x0000000000405332 in handle_db_request (state=, > reqstring=) at odbcserver.c:424 > > #17 database_handler (port=) at odbcserver.c:371 > > #18 0x00000000004059ce in main () at odbcserver.c:296 > > > > Stacktrace2: > > Core was generated by `/home/user/app1/lib/odbc-2.10.1'. > > Program terminated with signal SIGABRT, Aborted. > > (gdb) bt > > #0 0x0000003c4160f3cb in raise () from /lib64/libpthread.so.0 > > #1 0x00007f3472c1e442 in skgesigOSCrash () from > /u01/app/oracle/product/11.2.0/cl11203/lib/libclntsh.so.11.1 > > #2 0x00007f3472ec0cbd in kpeDbgSignalHandler () from > /u01/app/oracle/product/11.2.0/cl11203/lib/libclntsh.so.11.1 > > #3 0x00007f3472c1e652 in skgesig_sigactionHandler () from > /u01/app/oracle/product/11.2.0/cl11203/lib/libclntsh.so.11.1 > > #4 > > #5 0x0000003c412328a5 in raise () from /lib64/libc.so.6 > > #6 0x0000003c41234085 in abort () from /lib64/libc.so.6 > > #7 0x0000003c4126fa37 in __libc_message () from /lib64/libc.so.6 > > #8 0x0000003c41275366 in malloc_printerr () from /lib64/libc.so.6 > > #9 0x0000003c41277e93 in _int_free () from /lib64/libc.so.6 > > #10 0x00007f3473b8c37a in MEMFree () from > /u01/app/oracle/product/11.2.0/cl11203/lib/libsqora.so.11.1 > > #11 0x00007f3473b83908 in bcoResetParam () from > /u01/app/oracle/product/11.2.0/cl11203/lib/libsqora.so.11.1 > > #12 0x00007f3473b83b00 in bcoResetParams () from > /u01/app/oracle/product/11.2.0/cl11203/lib/libsqora.so.11.1 > > #13 0x00007f3473b3a759 in bcoSQLFreeStmt () from > /u01/app/oracle/product/11.2.0/cl11203/lib/libsqora.so.11.1 > > #14 0x00007f3473b90a90 in bccSQLFreeStmtLckd () from > /u01/app/oracle/product/11.2.0/cl11203/lib/libsqora.so.11.1 > > #15 0x00007f3473b90864 in bccSQLFreeStmt () from > /u01/app/oracle/product/11.2.0/cl11203/lib/libsqora.so.11.1 > > #16 0x00007f3473b9157e in SQLFreeStmt () from > /u01/app/oracle/product/11.2.0/cl11203/lib/libsqora.so.11.1 > > #17 0x00007f34749462f8 in SQLFreeStmt (statement_handle=0x2195480, > option=3) at SQLFreeStmt.c:247 > > #18 0x0000000000403f62 in db_param_query (buffer=, > state=0x7fff2dda97d0) at odbcserver.c:911 > > #19 0x0000000000405332 in handle_db_request (state=, > reqstring=) at odbcserver.c:424 > > #20 database_handler (port=) at odbcserver.c:371 > > #21 0x00000000004059ce in main () at odbcserver.c:296 > > > > Is it known bug? > > It?s looks like odbcserver has invalid memory usage code and r17.3 has > no significant changes in the odbc. > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From imantc@REDACTED Thu Nov 20 13:36:03 2014 From: imantc@REDACTED (Imants Cekusins) Date: Thu, 20 Nov 2014 13:36:03 +0100 Subject: [erlang-questions] Distributed application and netsplit In-Reply-To: References: <148C0C12-566F-4C43-B77E-B5A77D4260E9@vailsys.com> <802A948E-4671-4D4F-A7B3-552F15EDAE47@vailsys.com> Message-ID: Working two-application (supervisor, worker) sample code is available. It works with nodes shut down and in the case of netsplit. Email me if you are interested. On 20 November 2014 04:49, Zulfiqer Sekender wrote: > Hi, > I am too waiting for this. Could you please let me know in which version of > Erlang/OTP, this will be included and approximately when that version can be > released? > Thanks. > > - > Zulfiqer > >> Date: Wed, 19 Nov 2014 13:18:08 -0800 >> From: raould@REDACTED >> To: erlang-questions@REDACTED >> Subject: Re: [erlang-questions] Distributed application and netsplit >> >> > The problem is, once the net split has been detected, how then to >> > manually shutdown the unwanted application instance(s) in such a way that >> > the distributed application failover continues to work from that point >> > forward. >> > >> > The OP already has application-specific code which is able to detect >> > that a net split / split brain scenario has occurred. >> > >> > Furthermore, the OP is "ok" writing application specific code to attempt >> > to resolve the situation at that point. >> > >> > The question is, what is the "best" way to go about doing that? >> >> >> >> thanks. >> >> ok so it sounds like more clearly what we want is to know that OTP has >> been designed / will be updated to handle this at the level of it >> "going back to normal" at the OTP level, leaving the actual "sync up" >> part to the individual applications that use OTP. >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From leonard.boyce@REDACTED Thu Nov 20 14:42:02 2014 From: leonard.boyce@REDACTED (Leonard Boyce) Date: Thu, 20 Nov 2014 08:42:02 -0500 Subject: [erlang-questions] Distributed application and netsplit In-Reply-To: References: <148C0C12-566F-4C43-B77E-B5A77D4260E9@vailsys.com> <802A948E-4671-4D4F-A7B3-552F15EDAE47@vailsys.com> Message-ID: Hi Imants, I'm certainly interested in this. Maybe instead of dealing with plenty of requests from interested parties now and in the future, it may be a good idea to just add a github repo with the example code? Leonard On Thu, Nov 20, 2014 at 7:36 AM, Imants Cekusins wrote: > Working two-application (supervisor, worker) sample code is available. > > It works with nodes shut down and in the case of netsplit. > > Email me if you are interested. > > On 20 November 2014 04:49, Zulfiqer Sekender wrote: >> Hi, >> I am too waiting for this. Could you please let me know in which version of >> Erlang/OTP, this will be included and approximately when that version can be >> released? >> Thanks. >> >> - >> Zulfiqer >> >>> Date: Wed, 19 Nov 2014 13:18:08 -0800 >>> From: raould@REDACTED >>> To: erlang-questions@REDACTED >>> Subject: Re: [erlang-questions] Distributed application and netsplit >>> >>> > The problem is, once the net split has been detected, how then to >>> > manually shutdown the unwanted application instance(s) in such a way that >>> > the distributed application failover continues to work from that point >>> > forward. >>> > >>> > The OP already has application-specific code which is able to detect >>> > that a net split / split brain scenario has occurred. >>> > >>> > Furthermore, the OP is "ok" writing application specific code to attempt >>> > to resolve the situation at that point. >>> > >>> > The question is, what is the "best" way to go about doing that? >>> >>> >>> >>> thanks. >>> >>> ok so it sounds like more clearly what we want is to know that OTP has >>> been designed / will be updated to handle this at the level of it >>> "going back to normal" at the OTP level, leaving the actual "sync up" >>> part to the individual applications that use OTP. >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From imantc@REDACTED Thu Nov 20 14:51:47 2014 From: imantc@REDACTED (Imants Cekusins) Date: Thu, 20 Nov 2014 14:51:47 +0100 Subject: [erlang-questions] Distributed application and netsplit In-Reply-To: References: <148C0C12-566F-4C43-B77E-B5A77D4260E9@vailsys.com> <802A948E-4671-4D4F-A7B3-552F15EDAE47@vailsys.com> Message-ID: Hi Leonard, Is there maybe a repo where this list's members can upload their code samples? These are only a few files. Not sure if they merit own repo. I email you the files in a separate message. On 20 November 2014 14:42, Leonard Boyce wrote: > Hi Imants, > > I'm certainly interested in this. > > Maybe instead of dealing with plenty of requests from interested > parties now and in the future, it may be a good idea to just add a > github repo with the example code? > > Leonard > > On Thu, Nov 20, 2014 at 7:36 AM, Imants Cekusins wrote: >> Working two-application (supervisor, worker) sample code is available. >> >> It works with nodes shut down and in the case of netsplit. >> >> Email me if you are interested. >> >> On 20 November 2014 04:49, Zulfiqer Sekender wrote: >>> Hi, >>> I am too waiting for this. Could you please let me know in which version of >>> Erlang/OTP, this will be included and approximately when that version can be >>> released? >>> Thanks. >>> >>> - >>> Zulfiqer >>> >>>> Date: Wed, 19 Nov 2014 13:18:08 -0800 >>>> From: raould@REDACTED >>>> To: erlang-questions@REDACTED >>>> Subject: Re: [erlang-questions] Distributed application and netsplit >>>> >>>> > The problem is, once the net split has been detected, how then to >>>> > manually shutdown the unwanted application instance(s) in such a way that >>>> > the distributed application failover continues to work from that point >>>> > forward. >>>> > >>>> > The OP already has application-specific code which is able to detect >>>> > that a net split / split brain scenario has occurred. >>>> > >>>> > Furthermore, the OP is "ok" writing application specific code to attempt >>>> > to resolve the situation at that point. >>>> > >>>> > The question is, what is the "best" way to go about doing that? >>>> >>>> >>>> >>>> thanks. >>>> >>>> ok so it sounds like more clearly what we want is to know that OTP has >>>> been designed / will be updated to handle this at the level of it >>>> "going back to normal" at the OTP level, leaving the actual "sync up" >>>> part to the individual applications that use OTP. >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions From mark.nijhof@REDACTED Thu Nov 20 15:05:13 2014 From: mark.nijhof@REDACTED (Mark Nijhof) Date: Thu, 20 Nov 2014 15:05:13 +0100 Subject: [erlang-questions] Distributed application and netsplit In-Reply-To: References: <148C0C12-566F-4C43-B77E-B5A77D4260E9@vailsys.com> <802A948E-4671-4D4F-A7B3-552F15EDAE47@vailsys.com> Message-ID: Can these be put in a gist.github.com so we can all (or at least me) benefit? On Thu, Nov 20, 2014 at 2:51 PM, Imants Cekusins wrote: > Hi Leonard, > > Is there maybe a repo where this list's members can upload their code > samples? > > These are only a few files. Not sure if they merit own repo. > > I email you the files in a separate message. > > On 20 November 2014 14:42, Leonard Boyce > wrote: > > Hi Imants, > > > > I'm certainly interested in this. > > > > Maybe instead of dealing with plenty of requests from interested > > parties now and in the future, it may be a good idea to just add a > > github repo with the example code? > > > > Leonard > > > > On Thu, Nov 20, 2014 at 7:36 AM, Imants Cekusins > wrote: > >> Working two-application (supervisor, worker) sample code is available. > >> > >> It works with nodes shut down and in the case of netsplit. > >> > >> Email me if you are interested. > >> > >> On 20 November 2014 04:49, Zulfiqer Sekender > wrote: > >>> Hi, > >>> I am too waiting for this. Could you please let me know in which > version of > >>> Erlang/OTP, this will be included and approximately when that version > can be > >>> released? > >>> Thanks. > >>> > >>> - > >>> Zulfiqer > >>> > >>>> Date: Wed, 19 Nov 2014 13:18:08 -0800 > >>>> From: raould@REDACTED > >>>> To: erlang-questions@REDACTED > >>>> Subject: Re: [erlang-questions] Distributed application and netsplit > >>>> > >>>> > The problem is, once the net split has been detected, how then to > >>>> > manually shutdown the unwanted application instance(s) in such a > way that > >>>> > the distributed application failover continues to work from that > point > >>>> > forward. > >>>> > > >>>> > The OP already has application-specific code which is able to detect > >>>> > that a net split / split brain scenario has occurred. > >>>> > > >>>> > Furthermore, the OP is "ok" writing application specific code to > attempt > >>>> > to resolve the situation at that point. > >>>> > > >>>> > The question is, what is the "best" way to go about doing that? > >>>> > >>>> > >>>> > >>>> thanks. > >>>> > >>>> ok so it sounds like more clearly what we want is to know that OTP has > >>>> been designed / will be updated to handle this at the level of it > >>>> "going back to normal" at the OTP level, leaving the actual "sync up" > >>>> part to the individual applications that use OTP. > >>>> _______________________________________________ > >>>> erlang-questions mailing list > >>>> erlang-questions@REDACTED > >>>> http://erlang.org/mailman/listinfo/erlang-questions > >>> > >>> _______________________________________________ > >>> erlang-questions mailing list > >>> erlang-questions@REDACTED > >>> http://erlang.org/mailman/listinfo/erlang-questions > >>> > >> _______________________________________________ > >> erlang-questions mailing list > >> erlang-questions@REDACTED > >> http://erlang.org/mailman/listinfo/erlang-questions > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Mark Nijhof t: @MarkNijhof s: marknijhof -------------- next part -------------- An HTML attachment was scrubbed... URL: From imantc@REDACTED Thu Nov 20 15:51:48 2014 From: imantc@REDACTED (Imants Cekusins) Date: Thu, 20 Nov 2014 15:51:48 +0100 Subject: [erlang-questions] Distributed application and netsplit In-Reply-To: References: <148C0C12-566F-4C43-B77E-B5A77D4260E9@vailsys.com> <802A948E-4671-4D4F-A7B3-552F15EDAE47@vailsys.com> Message-ID: the code is in https://github.com/aminishiki/distr_netsplit.git any comments are welcome! From mark.nijhof@REDACTED Thu Nov 20 15:56:32 2014 From: mark.nijhof@REDACTED (Mark Nijhof) Date: Thu, 20 Nov 2014 15:56:32 +0100 Subject: [erlang-questions] Distributed application and netsplit In-Reply-To: References: <148C0C12-566F-4C43-B77E-B5A77D4260E9@vailsys.com> <802A948E-4671-4D4F-A7B3-552F15EDAE47@vailsys.com> Message-ID: Thank you! On Thu, Nov 20, 2014 at 3:51 PM, Imants Cekusins wrote: > the code is in > > https://github.com/aminishiki/distr_netsplit.git > > any comments are welcome! > -- Mark Nijhof t: @MarkNijhof s: marknijhof -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew.j.stone.1@REDACTED Thu Nov 20 21:09:55 2014 From: andrew.j.stone.1@REDACTED (Andrew Stone) Date: Thu, 20 Nov 2014 15:09:55 -0500 Subject: [erlang-questions] Distributed application and netsplit In-Reply-To: References: <148C0C12-566F-4C43-B77E-B5A77D4260E9@vailsys.com> <802A948E-4671-4D4F-A7B3-552F15EDAE47@vailsys.com> Message-ID: Hi, I've waited for someone to jump in and say this, but it hasn't happened. You really, really don't want to try dealing with netsplits and application failover in an app specific manner. It is not safe, and you will likely lose data. You really need a consensus algorithm like raft or paxos to handle this type of thing safely, or else you will end up with conflicting data on both sides of the partition. It may be quite a large dependency to rely on, but riak 2.0 has strongly consistent keys[1] that you could use to build a lock server to point to the active master server. Alternatively you could use riak_ensemble[2] directly to build a custom solution. Lastly, you can simply choose to not use failover and accept that when the primary goes down you will be offline until it comes back up. The secondary is just there to provide disaster recovery in case the primary is irrecoverable. This is a much safer and simpler solution, and one historically used by conventional databases with both asynchronous and synchronous replication. If you must have some level of fault tolerance/ HA you can use paxos. If your application can handle eventual consistency and the data types fit the model, you could try to use CRDTs [3]. That would allow you 100% availability and even allow writes to happen to both servers at once! I can't stress enough how important it is to not build ad-hoc failover protocols for this purpose. It will bite you. I've been bitten before, and so have many other people relying on this mechanism. While it may seem easier than using a proper distributed systems protocol at first, when you lose customer data in production, you quickly learn that easy isn't best. Best wishes, Andrew [1] http://docs.basho.com/riak/latest/dev/advanced/strong-consistency/ [2] https://github.com/basho/riak_ensemble [3] https://en.wikipedia.org/wiki/Conflict-free_replicated_data_types On Thu, Nov 20, 2014 at 9:56 AM, Mark Nijhof wrote: > Thank you! > > On Thu, Nov 20, 2014 at 3:51 PM, Imants Cekusins wrote: > >> the code is in >> >> https://github.com/aminishiki/distr_netsplit.git >> >> any comments are welcome! >> > > > > -- > Mark Nijhof > t: @MarkNijhof > s: marknijhof > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.goertzen@REDACTED Thu Nov 20 21:30:20 2014 From: daniel.goertzen@REDACTED (Daniel Goertzen) Date: Thu, 20 Nov 2014 14:30:20 -0600 Subject: [erlang-questions] Erlang on BBC Message-ID: Alright, the title is a bit of stretch, but the mention of Erlang in the image at the end of article made my day. http://www.bbc.com/news/blogs-trending-30135148 -------------- next part -------------- An HTML attachment was scrubbed... URL: From mjtruog@REDACTED Thu Nov 20 21:53:43 2014 From: mjtruog@REDACTED (Michael Truog) Date: Thu, 20 Nov 2014 12:53:43 -0800 Subject: [erlang-questions] Distributed application and netsplit In-Reply-To: References: Message-ID: <546E54D7.9010701@gmail.com> On 11/20/2014 12:09 PM, Andrew Stone wrote: > Hi, > > I've waited for someone to jump in and say this, but it hasn't happened. You really, really don't want to try dealing with netsplits and application failover in an app specific manner. It is not safe, and you will likely lose data. You really need a consensus algorithm like raft or paxos to handle this type of thing safely, or else you will end up with conflicting data on both sides of the partition. > > It may be quite a large dependency to rely on, but riak 2.0 has strongly consistent keys[1] that you could use to build a lock server to point to the active master server. Alternatively you could use riak_ensemble[2] directly to build a custom solution. > > Lastly, you can simply choose to not use failover and accept that when the primary goes down you will be offline until it comes back up. The secondary is just there to provide disaster recovery in case the primary is irrecoverable. This is a much safer and simpler solution, and one historically used by conventional databases with both asynchronous and synchronous replication. If you must have some level of fault tolerance/ HA you can use paxos. If your application can handle eventual consistency and the data types fit the model, you could try to use CRDTs [3]. That would allow you 100% availability and even allow writes to happen to both servers at once! > > I can't stress enough how important it is to not build ad-hoc failover protocols for this purpose. It will bite you. I've been bitten before, and so have many other people relying on this mechanism. While it may seem easier than using a proper distributed systems protocol at first, when you lose customer data in production, you quickly learn that easy isn't best. > > Best wishes, > Andrew > > [1] http://docs.basho.com/riak/latest/dev/advanced/strong-consistency/ > [2] https://github.com/basho/riak_ensemble > [3] https://en.wikipedia.org/wiki/Conflict-free_replicated_data_types The perspective above (Andrew's post) is good, but it is the perspective that you must be trying to maintain global state yourself. That means you are trying to create your own database instead of reusing one of the many databases that already exist. I believe development time is better spent reusing the databases that already exist, to handle replication of state as necessary and to deal with the latency inherent in that process. To pursue lower latency fault-tolerance it is better to have master-less processing in the Erlang nodes (not quorum among all instances with replicas for failures, but rather separate instances of source code that are used separately, concurrently). Then all the source code execution that needs to be fault-tolerant can be replicated to separate nodes, so netsplits do not impact the situation (any state the source code uses is only temporary, due to relying completely on the database). If necessary, database nodes could share the same machine as the Erlang node hosts, to avoid the possibility that a switch failure could cause a netsplit which impacts all database connections (assuming the database was one which could handle all the failure scenarios). This is the approach you can take with pg2 usage (http://www.erlang.org/doc/man/pg2.html) or cpg (at https://github.com/okeuday/cpg/) to create process groups that are distributed. If you are looking for higher-level abstractions, there is a service abstraction provided by CloudI (http://cloudi.org) which relies on cpg to keep service processes available on all Erlang nodes, despite any netsplits, pursuing this master-less approach. > > On Thu, Nov 20, 2014 at 9:56 AM, Mark Nijhof > wrote: > > Thank you! > > On Thu, Nov 20, 2014 at 3:51 PM, Imants Cekusins > wrote: > > the code is in > > https://github.com/aminishiki/distr_netsplit.git > > any comments are welcome! > > > > > -- > Mark Nijhof > t: @MarkNijhof > s: marknijhof > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew.j.stone.1@REDACTED Fri Nov 21 00:10:31 2014 From: andrew.j.stone.1@REDACTED (Andrew Stone) Date: Thu, 20 Nov 2014 18:10:31 -0500 Subject: [erlang-questions] Distributed application and netsplit In-Reply-To: <546E54D7.9010701@gmail.com> References: <546E54D7.9010701@gmail.com> Message-ID: I agree that using stateless app servers with a highly available database like Riak is probably a simpler and more robust design than trying to build that logic yourself. If your application can be built that way, it will be a lot easier. -Andrew On Thu, Nov 20, 2014 at 3:53 PM, Michael Truog wrote: > On 11/20/2014 12:09 PM, Andrew Stone wrote: > > Hi, > > I've waited for someone to jump in and say this, but it hasn't happened. > You really, really don't want to try dealing with netsplits and application > failover in an app specific manner. It is not safe, and you will likely > lose data. You really need a consensus algorithm like raft or paxos to > handle this type of thing safely, or else you will end up with conflicting > data on both sides of the partition. > > It may be quite a large dependency to rely on, but riak 2.0 has strongly > consistent keys[1] that you could use to build a lock server to point to > the active master server. Alternatively you could use riak_ensemble[2] > directly to build a custom solution. > > Lastly, you can simply choose to not use failover and accept that when > the primary goes down you will be offline until it comes back up. The > secondary is just there to provide disaster recovery in case the primary is > irrecoverable. This is a much safer and simpler solution, and one > historically used by conventional databases with both asynchronous and > synchronous replication. If you must have some level of fault tolerance/ HA > you can use paxos. If your application can handle eventual consistency and > the data types fit the model, you could try to use CRDTs [3]. That would > allow you 100% availability and even allow writes to happen to both servers > at once! > > I can't stress enough how important it is to not build ad-hoc failover > protocols for this purpose. It will bite you. I've been bitten before, and > so have many other people relying on this mechanism. While it may seem > easier than using a proper distributed systems protocol at first, when you > lose customer data in production, you quickly learn that easy isn't best. > > Best wishes, > Andrew > > [1] http://docs.basho.com/riak/latest/dev/advanced/strong-consistency/ > [2] https://github.com/basho/riak_ensemble > [3] https://en.wikipedia.org/wiki/Conflict-free_replicated_data_types > > The perspective above (Andrew's post) is good, but it is the perspective > that you must be trying to maintain global state yourself. That means you > are trying to create your own database instead of reusing one of the many > databases that already exist. I believe development time is better spent > reusing the databases that already exist, to handle replication of state as > necessary and to deal with the latency inherent in that process. > > To pursue lower latency fault-tolerance it is better to have master-less > processing in the Erlang nodes (not quorum among all instances with > replicas for failures, but rather separate instances of source code that > are used separately, concurrently). Then all the source code execution > that needs to be fault-tolerant can be replicated to separate nodes, so > netsplits do not impact the situation (any state the source code uses is > only temporary, due to relying completely on the database). If necessary, > database nodes could share the same machine as the Erlang node hosts, to > avoid the possibility that a switch failure could cause a netsplit which > impacts all database connections (assuming the database was one which could > handle all the failure scenarios). > > This is the approach you can take with pg2 usage ( > http://www.erlang.org/doc/man/pg2.html) or cpg (at > https://github.com/okeuday/cpg/) to create process groups that are > distributed. If you are looking for higher-level abstractions, there is a > service abstraction provided by CloudI (http://cloudi.org) which relies > on cpg to keep service processes available on all Erlang nodes, despite any > netsplits, pursuing this master-less approach. > > > On Thu, Nov 20, 2014 at 9:56 AM, Mark Nijhof < > mark.nijhof@REDACTED> wrote: > >> Thank you! >> >> On Thu, Nov 20, 2014 at 3:51 PM, Imants Cekusins >> wrote: >> >>> the code is in >>> >>> https://github.com/aminishiki/distr_netsplit.git >>> >>> any comments are welcome! >>> >> >> >> >> -- >> Mark Nijhof >> t: @MarkNijhof >> s: marknijhof >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > > > _______________________________________________ > erlang-questions mailing listerlang-questions@REDACTED://erlang.org/mailman/listinfo/erlang-questions > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From atill@REDACTED Thu Nov 20 23:44:05 2014 From: atill@REDACTED (Andy Till) Date: Thu, 20 Nov 2014 22:44:05 +0000 Subject: [erlang-questions] Configuring Messaging Back Pressure Message-ID: <546E6EB5.90508@mail.com> Short outline of how erts_use_sender_punish works, please comment if there are any errors in the description. erts_use_sender_punish is a flag hard coded to 1 (true), when it is true a process sending messages to another process will have its reduction count reduced by the number of messages in the receivers message queue multiplied by four. Sending messages to processes with zero messages in the queue is free in terms of reductions, but sending messages to load queues is very expensive and will lead the scheduler to context switch to another process more often. In the case where the the relationship between of producers and consumers is one to one, this makes sense. It provides back pressure when part of the system is loaded. In the case where there is one producer to many consumers, if one consumer gets a flood of messages and the producer is punished then the punishment is not only on the producer but on all other consumers because the producer cannot create enough work for them as it used all of its reductions. This makes performance unpredictable when work cannot be spread evenly. Could this be made to be configurable using a process flag? Cheers Andy Code: https://github.com/erlang/otp/blob/a70f0ce9b34e4db61dacb8db24f9ab5671ed4c8a/erts/emulator/beam/erl_init.c#L694 https://github.com/erlang/otp/blob/682a6082159568f40615f03d12d44ee70edd14c6/erts/emulator/beam/bif.c#L2058 https://github.com/erlang/otp/blob/a8e12f7168c14cc765a63a51c838d065412795d7/erts/emulator/beam/erl_message.c#L504 Previous mentions and issues with this feature: http://erlang.org/pipermail/erlang-questions/2011-July/060314.html http://erlang.org/pipermail/erlang-questions/2011-July/060042.html From fritchie@REDACTED Fri Nov 21 07:28:36 2014 From: fritchie@REDACTED (Scott Lystig Fritchie) Date: Fri, 21 Nov 2014 00:28:36 -0600 Subject: [erlang-questions] Distributed application and netsplit In-Reply-To: Message of "Thu, 20 Nov 2014 15:09:55 EST." Message-ID: <98965.1416551316@snookles.snookles.com> Andrew Stone wrote: ajs> Hi, I've waited for someone to jump in and say this, but it hasn't ajs> happened. You really, really don't want to try dealing with ajs> netsplits and application failover in an app specific manner. It is ajs> not safe, and you will likely lose data. Agreed, 106%. As Joe Armstrong, benelovent co-leader of the concurrent world, is fond of saying, it takes at least two machines to detect and recover from a fault in one. In the larger case of split brain management in distributed, asynchronous, message passing environment (like the Universe), there are fundamental limits and impossibilities that nobody is immune from. Good luck with two machines figuring that out. -Scott From darach@REDACTED Fri Nov 21 11:19:58 2014 From: darach@REDACTED (Darach Ennis) Date: Fri, 21 Nov 2014 10:19:58 +0000 Subject: [erlang-questions] Configuring Messaging Back Pressure In-Reply-To: <546E6EB5.90508@mail.com> References: <546E6EB5.90508@mail.com> Message-ID: +1 On 21 Nov 2014 04:53, "Andy Till" wrote: > Short outline of how erts_use_sender_punish works, please comment if there > are any errors in the description. > > erts_use_sender_punish is a flag hard coded to 1 (true), when it is true a > process sending messages to another process will have its reduction count > reduced by the number of messages in the receivers message queue multiplied > by four. Sending messages to processes with zero messages in the queue is > free in terms of reductions, but sending messages to load queues is very > expensive and will lead the scheduler to context switch to another process > more often. > > In the case where the the relationship between of producers and consumers > is one to one, this makes sense. It provides back pressure when part of > the system is loaded. > > In the case where there is one producer to many consumers, if one consumer > gets a flood of messages and the producer is punished then the punishment > is not only on the producer but on all other consumers because the producer > cannot create enough work for them as it used all of its reductions. > > This makes performance unpredictable when work cannot be spread evenly. > > Could this be made to be configurable using a process flag? > > Cheers > > Andy > > Code: > > https://github.com/erlang/otp/blob/a70f0ce9b34e4db61dacb8db24f9ab > 5671ed4c8a/erts/emulator/beam/erl_init.c#L694 > https://github.com/erlang/otp/blob/682a6082159568f40615f03d12d44e > e70edd14c6/erts/emulator/beam/bif.c#L2058 > https://github.com/erlang/otp/blob/a8e12f7168c14cc765a63a51c838d0 > 65412795d7/erts/emulator/beam/erl_message.c#L504 > > Previous mentions and issues with this feature: > > http://erlang.org/pipermail/erlang-questions/2011-July/060314.html > http://erlang.org/pipermail/erlang-questions/2011-July/060042.html > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Fri Nov 21 12:21:40 2014 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Fri, 21 Nov 2014 13:21:40 +0200 Subject: [erlang-questions] The -module line Message-ID: <546F2044.9010908@ninenines.eu> Hi, Something has been bugging me for a very long time now. The -module line is mandatory. It also must be the same as the file name minus the extension .erl or compilation fails. http://erlang.org/doc/reference_manual/modules.html > -module(Module). > Module declaration, defining the name of the module. The name Module, > an atom, should be the same as the file name minus the extension erl. > Otherwise code loading will not work as intended. > This attribute should be specified first and is the only attribute > which is mandatory. Instead of forcing the programmer to write down the file name, minus the .erl extension, and then have the compiler check that the programmer did a good job and can have a cookie, can't we have the compiler simply get the module name from the file name? The compiler is looking there *regardless* so maybe it can save us some time and fetch it itself? The attribute is mandatory, yes. But it can be just another attribute the compiler fills in automatically. Or perhaps I'm missing an obvious reason as to why we have to fill in a line identical to the file name we are feeding the compiler? I would like to recommend a change that will make the compiler fetch and use the file name automatically if the attribute is missing, and error out if it's different than the one in the file. -- Lo?c Hoguin http://ninenines.eu From jon@REDACTED Fri Nov 21 12:36:29 2014 From: jon@REDACTED (Jonathan Schneider) Date: Fri, 21 Nov 2014 11:36:29 +0000 Subject: [erlang-questions] The -module line In-Reply-To: <546F2044.9010908@ninenines.eu> References: <546F2044.9010908@ninenines.eu> Message-ID: Personally I?d like the restriction that the names are the same to be removed altogether. Is there a good reason for it ? Jon On 21 Nov 2014, at 11:21, Lo?c Hoguin wrote: > Hi, > > Something has been bugging me for a very long time now. > > The -module line is mandatory. It also must be the same as the file name minus the extension .erl or compilation fails. From lhfcws@REDACTED Fri Nov 21 08:44:21 2014 From: lhfcws@REDACTED (Lhfcws GMail) Date: Fri, 21 Nov 2014 15:44:21 +0800 Subject: [erlang-questions] What kind of objects can be a message type? Message-ID: Hi, I?m new to Erlang. I have a question about the message passed among the Erlang processes. 1. I create a Socket or FileHandler, can I use ! to send it to another process and still works well? demo code: ListenSocket = gen_tcp:listen(1055, [{active, true}]), Socket = gen_tcp:accept(ListenSocket), From ! {new_socket, Socket}. 2. What if I invoking ?spawn? to pass the Socket to the new process ? demo code: NewProcess = fun(Socket) -> ? end. ListenSocket = gen_tcp:listen(1055, [{active, true}]), Socket = gen_tcp:accept(ListenSocket), spawn(NewProcess, Socket). ======= I know that the message in the receiver is a copy of the one in the sender, but I?m a little confused about the questions above. Could someone just explain this? Thank you. From aggelgian@REDACTED Fri Nov 21 12:41:49 2014 From: aggelgian@REDACTED (Aggelos Giantsios) Date: Fri, 21 Nov 2014 13:41:49 +0200 Subject: [erlang-questions] The -module line In-Reply-To: References: <546F2044.9010908@ninenines.eu> Message-ID: On Fri, Nov 21, 2014 at 1:36 PM, Jonathan Schneider wrote: > Personally I?d like the restriction that the names are the same to be > removed altogether. > > Is there a good reason for it ? > Well I guess if you are searching for a specific module in a folder with many .erl files, you would like to be looking for the *module_name*.erl rather than opening all the files one by one or greping the -module attribute. Aggelos > > Jon > > > On 21 Nov 2014, at 11:21, Lo?c Hoguin wrote: > > > Hi, > > > > Something has been bugging me for a very long time now. > > > > The -module line is mandatory. It also must be the same as the file name > minus the extension .erl or compilation fails. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From olivier.boudeville@REDACTED Fri Nov 21 13:23:46 2014 From: olivier.boudeville@REDACTED (Olivier BOUDEVILLE) Date: Fri, 21 Nov 2014 13:23:46 +0100 Subject: [erlang-questions] Configuring Messaging Back Pressure In-Reply-To: References: <546E6EB5.90508@mail.com> Message-ID: Hi, The reciprocal case (M producers and N consumers, with M >> N) may also be a problem. I can share a simple program which seems to show that the retaliation upon the producers is at least in some cases (where M is the number of cores of the machine minus 1, and N=1) insufficient: the program will crash and burn, exhausting the memory because of the mailbox of the consumer becoming too large. Ideally the back-pressure would increase steadily until the mailbox of the producer decreases below reasonable limits. Currently, I think that the developer has either to tune M by hand (which would depend on the hardware resources...), or to perform some explicit yielding (with timer:sleep(1) on the producers). Neither of them is very satisfactory. More generally, couldn't we hope that the scheduling logic could cope with at least most of the M producers-N consumers scenarios, without resorting to hints from the developer like specifying a process flag or performing explicit yielding? Maybe (for the M >> N case) it is just a matter of heavier punishing (ex: ultimately exponential) whenever the receiving mailbox becomes too large. Best, Olivier. --------------------------- Olivier Boudeville EDF R&D : 1, avenue du G?n?ral de Gaulle, 92140 Clamart, France D?partement SINETICS, groupe ASICS (I2A), bureau B-226 Office : +33 1 47 65 59 58 / Mobile : +33 6 16 83 37 22 / Fax : +33 1 47 65 27 13 darach@REDACTED Envoy? par : erlang-questions-bounces@REDACTED 21/11/2014 11:27 A atill@REDACTED cc erlang-questions@REDACTED Objet Re: [erlang-questions] Configuring Messaging Back Pressure +1 On 21 Nov 2014 04:53, "Andy Till" wrote: Short outline of how erts_use_sender_punish works, please comment if there are any errors in the description. erts_use_sender_punish is a flag hard coded to 1 (true), when it is true a process sending messages to another process will have its reduction count reduced by the number of messages in the receivers message queue multiplied by four. Sending messages to processes with zero messages in the queue is free in terms of reductions, but sending messages to load queues is very expensive and will lead the scheduler to context switch to another process more often. In the case where the the relationship between of producers and consumers is one to one, this makes sense. It provides back pressure when part of the system is loaded. In the case where there is one producer to many consumers, if one consumer gets a flood of messages and the producer is punished then the punishment is not only on the producer but on all other consumers because the producer cannot create enough work for them as it used all of its reductions. This makes performance unpredictable when work cannot be spread evenly. Could this be made to be configurable using a process flag? Cheers Andy Code: https://github.com/erlang/otp/blob/a70f0ce9b34e4db61dacb8db24f9ab5671ed4c8a/erts/emulator/beam/erl_init.c#L694 https://github.com/erlang/otp/blob/682a6082159568f40615f03d12d44ee70edd14c6/erts/emulator/beam/bif.c#L2058 https://github.com/erlang/otp/blob/a8e12f7168c14cc765a63a51c838d065412795d7/erts/emulator/beam/erl_message.c#L504 Previous mentions and issues with this feature: http://erlang.org/pipermail/erlang-questions/2011-July/060314.html http://erlang.org/pipermail/erlang-questions/2011-July/060042.html _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont ?tablis ? l'intention exclusive des destinataires et les informations qui y figurent sont strictement confidentielles. Toute utilisation de ce Message non conforme ? sa destination, toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse. Si vous n'?tes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez re?u ce Message par erreur, merci de le supprimer de votre syst?me, ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Nous vous remercions ?galement d'en avertir imm?diatement l'exp?diteur par retour du message. Il est impossible de garantir que les communications par messagerie ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute erreur ou virus. ____________________________________________________ This message and any attachments (the 'Message') are intended solely for the addressees. The information contained in this Message is confidential. Any use of information contained in this Message not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval. If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return message. E-mail communication cannot be guaranteed to be timely secure, error or virus-free. -------------- next part -------------- An HTML attachment was scrubbed... URL: From lhfcws@REDACTED Fri Nov 21 13:05:47 2014 From: lhfcws@REDACTED (Lhfcws GMail) Date: Fri, 21 Nov 2014 20:05:47 +0800 Subject: [erlang-questions] What kind of objects can be a message type? In-Reply-To: <546F2701.40103@tail-f.com> References: <546F2556.2010303@tail-f.com> <546F2701.40103@tail-f.com> Message-ID: <2D7716C4-E0CF-4228-8887-CEF4A762869B@gmail.com> Thanks a lot! gen_tcp:controlling_process is indeed what I want. And let me confirm your answer: 1. Objects like socket and file handler can be pass in spawn arguments. 2. But they can not be passed by ! message passing. Am I right? -------------- ? 2014?11?21??19:50?Niclas Eklund ??? > Hello again! > > Forgot to answer your second question. Yes, you can spawn a new process that will "own" the connection, but then you need to use this function - http://www.erlang.org/doc/man/gen_tcp.html#controlling_process-2 You typically want to keep the listen process rather lightweight (do very little) since using blocking accept is common and then the listen process needs to be restarted when doing a live code upgrade. See for example this code - https://github.com/erlang/otp/blob/maint/lib/orber/src/orber_iiop_net_accept.erl It's doing a bit more, i.e. only allows a certain number of concurrent incoming requests, but as you can see it's minimal. > > /Nick > > On 11/21/2014 12:43 PM, Niclas Eklund wrote: >> Hi! >> >> I'm afraid that you don't handle the returned values correctly. See this page - http://www.erlang.org/doc/man/gen_tcp.html - for more info. Should be: >> >> server() -> >> {ok, LSock} = gen_tcp:listen(5678, [binary, {packet, 0}, >> {active, false}]), >> {ok, Sock} = gen_tcp:accept(LSock), >> {ok, Bin} = do_recv(Sock, []), >> ok = gen_tcp:close(Sock), >> Bin. >> >> do_recv(Sock, Bs) -> >> case gen_tcp:recv(Sock, 0) of >> {ok, B} -> >> do_recv(Sock, [Bs, B]); >> {error, closed} -> >> {ok, list_to_binary(Bs)} >> end. >> >> /Nick >> >> >> >> On 11/21/2014 08:44 AM, Lhfcws GMail wrote: >>> Hi, I?m new to Erlang. >>> I have a question about the message passed among the Erlang processes. >>> 1. I create a Socket or FileHandler, can I use ! to send it to another process and still works well? >>> >>> demo code: >>> >>> ListenSocket = gen_tcp:listen(1055, [{active, true}]), >>> Socket = gen_tcp:accept(ListenSocket), >>> From ! {new_socket, Socket}. >>> >>> >>> 2. What if I invoking ?spawn? to pass the Socket to the new process ? >>> >>> demo code: >>> >>> NewProcess = fun(Socket) -> ? end. >>> >>> ListenSocket = gen_tcp:listen(1055, [{active, true}]), >>> Socket = gen_tcp:accept(ListenSocket), >>> spawn(NewProcess, Socket). >>> >>> ======= >>> >>> I know that the message in the receiver is a copy of the one in the sender, but I?m a little confused about the questions above. Could someone just explain this? >>> Thank you. >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >> > From erlang@REDACTED Fri Nov 21 14:06:42 2014 From: erlang@REDACTED (Joe Armstrong) Date: Fri, 21 Nov 2014 14:06:42 +0100 Subject: [erlang-questions] The -module line In-Reply-To: <546F2044.9010908@ninenines.eu> References: <546F2044.9010908@ninenines.eu> Message-ID: On Fri, Nov 21, 2014 at 12:21 PM, Lo?c Hoguin wrote: > Hi, > > Something has been bugging me for a very long time now. > > The -module line is mandatory. It also must be the same as the file name > minus the extension .erl or compilation fails. Not quite. Here's an example where the two differ $ cat foo.erl -module(bar). -export([me/0]). me() -> "bar". $erl 1> {ok, Mod, Code} = compile:file("foo.erl",[binary]). {ok,bar, <<70,79,82,49,0,0,2,0,66,69,65,77,65,116,111,109,0,0,0, 46,0,0,0,5,3,98,...>>} 2> code:load_binary(bar,"glurk",Code). {module,bar} 3> bar:me(). "bar" The reasons for this are lost in the mists of time - if memory serves me well it had to do with some deep skulduggery in distributed Erlang. Personally I'd like to have In a file foo.erls -module(bar). ... -module(foo). ... -module(baz). ... Like elixir :-) /Joe > > http://erlang.org/doc/reference_manual/modules.html > >> -module(Module). >> Module declaration, defining the name of the module. The name Module, >> an atom, should be the same as the file name minus the extension erl. >> Otherwise code loading will not work as intended. > >> This attribute should be specified first and is the only attribute >> which is mandatory. > > Instead of forcing the programmer to write down the file name, minus the > .erl extension, and then have the compiler check that the programmer did a > good job and can have a cookie, can't we have the compiler simply get the > module name from the file name? The compiler is looking there *regardless* > so maybe it can save us some time and fetch it itself? > > The attribute is mandatory, yes. But it can be just another attribute the > compiler fills in automatically. > > Or perhaps I'm missing an obvious reason as to why we have to fill in a line > identical to the file name we are feeding the compiler? > > I would like to recommend a change that will make the compiler fetch and use > the file name automatically if the attribute is missing, and error out if > it's different than the one in the file. > > -- > Lo?c Hoguin > http://ninenines.eu > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From essen@REDACTED Fri Nov 21 14:13:56 2014 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Fri, 21 Nov 2014 15:13:56 +0200 Subject: [erlang-questions] The -module line In-Reply-To: References: <546F2044.9010908@ninenines.eu> Message-ID: <546F3A94.4070908@ninenines.eu> On 11/21/2014 03:06 PM, Joe Armstrong wrote: > On Fri, Nov 21, 2014 at 12:21 PM, Lo?c Hoguin wrote: >> Hi, >> >> Something has been bugging me for a very long time now. >> >> The -module line is mandatory. It also must be the same as the file name >> minus the extension .erl or compilation fails. > > Not quite. I should have been more clear, I was talking about the Erlang compiler (erlc), not individual compiler modules. I do not know where the check occurs exactly, but it is done at least when using erlc. -- Lo?c Hoguin http://ninenines.eu From tty.erlang@REDACTED Fri Nov 21 16:25:46 2014 From: tty.erlang@REDACTED (T Ty) Date: Fri, 21 Nov 2014 15:25:46 +0000 Subject: [erlang-questions] What kind of objects can be a message type? In-Reply-To: <2D7716C4-E0CF-4228-8887-CEF4A762869B@gmail.com> References: <546F2556.2010303@tail-f.com> <546F2701.40103@tail-f.com> <2D7716C4-E0CF-4228-8887-CEF4A762869B@gmail.com> Message-ID: You can pass Pid and References in a message using ! This is an example of how trivial writing a FTP-like client in Erlang. On machine 1. File = file:open(FileName, [read]). PidOfProcessOnMachine2 ! {remote_file, File}. On machine 2. receive {remote_file, File} -> {ok, Data} = file:read(File, 1000000), write_to_local_disk(Data) end Now, that said I have only tried it with File /Pid references and not with Socket. On Fri, Nov 21, 2014 at 12:05 PM, Lhfcws GMail wrote: > Thanks a lot! gen_tcp:controlling_process is indeed what I want. > And let me confirm your answer: > 1. Objects like socket and file handler can be pass in spawn arguments. > 2. But they can not be passed by ! message passing. > > Am I right? > -------------- > > ? 2014?11?21??19:50?Niclas Eklund ??? > > > Hello again! > > > > Forgot to answer your second question. Yes, you can spawn a new process > that will "own" the connection, but then you need to use this function - > http://www.erlang.org/doc/man/gen_tcp.html#controlling_process-2 You > typically want to keep the listen process rather lightweight (do very > little) since using blocking accept is common and then the listen process > needs to be restarted when doing a live code upgrade. See for example this > code - > https://github.com/erlang/otp/blob/maint/lib/orber/src/orber_iiop_net_accept.erl > It's doing a bit more, i.e. only allows a certain number of concurrent > incoming requests, but as you can see it's minimal. > > > > /Nick > > > > On 11/21/2014 12:43 PM, Niclas Eklund wrote: > >> Hi! > >> > >> I'm afraid that you don't handle the returned values correctly. See > this page - http://www.erlang.org/doc/man/gen_tcp.html - for more info. > Should be: > >> > >> server() -> > >> {ok, LSock} = gen_tcp:listen(5678, [binary, {packet, 0}, > >> {active, false}]), > >> {ok, Sock} = gen_tcp:accept(LSock), > >> {ok, Bin} = do_recv(Sock, []), > >> ok = gen_tcp:close(Sock), > >> Bin. > >> > >> do_recv(Sock, Bs) -> > >> case gen_tcp:recv(Sock, 0) of > >> {ok, B} -> > >> do_recv(Sock, [Bs, B]); > >> {error, closed} -> > >> {ok, list_to_binary(Bs)} > >> end. > >> > >> /Nick > >> > >> > >> > >> On 11/21/2014 08:44 AM, Lhfcws GMail wrote: > >>> Hi, I?m new to Erlang. > >>> I have a question about the message passed among the Erlang processes. > >>> 1. I create a Socket or FileHandler, can I use ! to send it to another > process and still works well? > >>> > >>> demo code: > >>> > >>> ListenSocket = gen_tcp:listen(1055, [{active, true}]), > >>> Socket = gen_tcp:accept(ListenSocket), > >>> From ! {new_socket, Socket}. > >>> > >>> > >>> 2. What if I invoking ?spawn? to pass the Socket to the new process ? > >>> > >>> demo code: > >>> > >>> NewProcess = fun(Socket) -> ? end. > >>> > >>> ListenSocket = gen_tcp:listen(1055, [{active, true}]), > >>> Socket = gen_tcp:accept(ListenSocket), > >>> spawn(NewProcess, Socket). > >>> > >>> ======= > >>> > >>> I know that the message in the receiver is a copy of the one in the > sender, but I?m a little confused about the questions above. Could someone > just explain this? > >>> Thank you. > >>> _______________________________________________ > >>> erlang-questions mailing list > >>> erlang-questions@REDACTED > >>> http://erlang.org/mailman/listinfo/erlang-questions > >> > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From felixgallo@REDACTED Fri Nov 21 18:04:12 2014 From: felixgallo@REDACTED (Felix Gallo) Date: Fri, 21 Nov 2014 09:04:12 -0800 Subject: [erlang-questions] Distributed application and netsplit In-Reply-To: <98965.1416551316@snookles.snookles.com> References: <98965.1416551316@snookles.snookles.com> Message-ID: With all due respect to the Basho guys and the wars from which their unique and terrible scars were born, the question here was recovering a single gen_server from split-brain. Disregarding for a moment the incongruousness of installing a Riak cluster to try to handle that case, there's nothing Riak could do to even help out, even if both progressing gen_servers were able to instantly save state in a known shared key. Either it's configured as last-write-wins ("lose data") or it's configured to return siblings ("split-brain, YOU handle it"). The underlying problem is that you need conflict resolution, which Riak *does* provide with their recent CRDT work, but that's a function of CRDTs as a concept and organizing your data properly. And while I wholeheartedly agree with the Basho guys that you should go to the literature and proven algorithms rather than inventing your own, there's a very large class of gen_servers for which conflicting progress state is trivially composable (e.g 'union', 'greater_of'). CRDT paper: http://highscalability.com/blog/2010/12/23/paper-crdts-consistency-without-concurrency-control.html great discussion of the issues: http://aphyr.com/posts/285-call-me-maybe-riak On Thu, Nov 20, 2014 at 10:28 PM, Scott Lystig Fritchie < fritchie@REDACTED> wrote: > Andrew Stone wrote: > > ajs> Hi, I've waited for someone to jump in and say this, but it hasn't > ajs> happened. You really, really don't want to try dealing with > ajs> netsplits and application failover in an app specific manner. It is > ajs> not safe, and you will likely lose data. > > Agreed, 106%. > > As Joe Armstrong, benelovent co-leader of the concurrent world, is fond > of saying, it takes at least two machines to detect and recover from a > fault in one. In the larger case of split brain management in > distributed, asynchronous, message passing environment (like the > Universe), there are fundamental limits and impossibilities that nobody > is immune from. Good luck with two machines figuring that out. > > -Scott > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From imantc@REDACTED Fri Nov 21 20:41:24 2014 From: imantc@REDACTED (Imants Cekusins) Date: Fri, 21 Nov 2014 20:41:24 +0100 Subject: [erlang-questions] The -module line In-Reply-To: <546F3A94.4070908@ninenines.eu> References: <546F2044.9010908@ninenines.eu> <546F3A94.4070908@ninenines.eu> Message-ID: It is possible to compile .erl code with module different from file name. In http://www.erlang.org/doc/man/compile.html Search for *no_error_module_mismatch* -------------- next part -------------- An HTML attachment was scrubbed... URL: From imantc@REDACTED Fri Nov 21 21:03:10 2014 From: imantc@REDACTED (Imants Cekusins) Date: Fri, 21 Nov 2014 21:03:10 +0100 Subject: [erlang-questions] Distributed application and netsplit In-Reply-To: References: <98965.1416551316@snookles.snookles.com> Message-ID: As a sidenote, could you say that: a) how to propose and b) when to propose and c) whether to propose all are very important yet separate topics? -------------- next part -------------- An HTML attachment was scrubbed... URL: From raould@REDACTED Fri Nov 21 21:59:05 2014 From: raould@REDACTED (Raoul Duke) Date: Fri, 21 Nov 2014 12:59:05 -0800 Subject: [erlang-questions] Distributed application and netsplit In-Reply-To: References: <98965.1416551316@snookles.snookles.com> Message-ID: yeah, just ask my spouse. From atill@REDACTED Fri Nov 21 23:44:22 2014 From: atill@REDACTED (Andy Till) Date: Fri, 21 Nov 2014 22:44:22 +0000 Subject: [erlang-questions] Configuring Messaging Back Pressure In-Reply-To: References: <546E6EB5.90508@mail.com> Message-ID: <546FC046.6050800@mail.com> I don't think you can rely on the scheduler alone if there are, say ten producers giving work to one consumer asynchronously. Using gen_server:call from the producer to the consumer could help here? Cheers Andy On 21/11/2014 12:23, Olivier BOUDEVILLE wrote: > Hi, > > The reciprocal case (M producers and N consumers, with M >> N) may > also be a problem. > > I can share a simple program which seems to show that the retaliation > upon the producers is at least in some cases (where M is the number of > cores of the machine minus 1, and N=1) insufficient: the program will > crash and burn, exhausting the memory because of the mailbox of the > consumer becoming too large. Ideally the back-pressure would increase > steadily until the mailbox of the producer decreases below reasonable > limits. Currently, I think that the developer has either to tune M by > hand (which would depend on the hardware resources...), or to perform > some explicit yielding (with timer:sleep(1) on the producers). Neither > of them is very satisfactory. > > More generally, couldn't we hope that the scheduling logic could cope > with at least most of the M producers-N consumers scenarios, without > resorting to hints from the developer like specifying a process flag > or performing explicit yielding? Maybe (for the M >> N case) it is > just a matter of heavier punishing (ex: ultimately exponential) > whenever the receiving mailbox becomes too large. > > Best, > > Olivier. > --------------------------- > Olivier Boudeville > > EDF R&D : 1, avenue du G?n?ral de Gaulle, 92140 Clamart, France > D?partement SINETICS, groupe ASICS (I2A), bureau B-226 > Office : +33 1 47 65 59 58 / Mobile : +33 6 16 83 37 22 / Fax : +33 1 > 47 65 27 13 > > > *darach@REDACTED* > Envoy? par : erlang-questions-bounces@REDACTED > > 21/11/2014 11:27 > > > A > atill@REDACTED > cc > erlang-questions@REDACTED > Objet > Re: [erlang-questions] Configuring Messaging Back Pressure > > > > > > > > > > +1 > > On 21 Nov 2014 04:53, "Andy Till" <_atill@REDACTED > > wrote: > Short outline of how erts_use_sender_punish works, please comment if > there are any errors in the description. > > erts_use_sender_punish is a flag hard coded to 1 (true), when it is > true a process sending messages to another process will have its > reduction count reduced by the number of messages in the receivers > message queue multiplied by four. Sending messages to processes with > zero messages in the queue is free in terms of reductions, but sending > messages to load queues is very expensive and will lead the scheduler > to context switch to another process more often. > > In the case where the the relationship between of producers and > consumers is one to one, this makes sense. It provides back pressure > when part of the system is loaded. > > In the case where there is one producer to many consumers, if one > consumer gets a flood of messages and the producer is punished then > the punishment is not only on the producer but on all other consumers > because the producer cannot create enough work for them as it used all > of its reductions. > > This makes performance unpredictable when work cannot be spread evenly. > > Could this be made to be configurable using a process flag? > > Cheers > > Andy > > Code: > _ > __https://github.com/erlang/otp/blob/a70f0ce9b34e4db61dacb8db24f9ab5671ed4c8a/erts/emulator/beam/erl_init.c#L694__ > __https://github.com/erlang/otp/blob/682a6082159568f40615f03d12d44ee70edd14c6/erts/emulator/beam/bif.c#L2058__ > __https://github.com/erlang/otp/blob/a8e12f7168c14cc765a63a51c838d065412795d7/erts/emulator/beam/erl_message.c#L504_ > > Previous mentions and issues with this feature: > _ > __http://erlang.org/pipermail/erlang-questions/2011-July/060314.html__ > __http://erlang.org/pipermail/erlang-questions/2011-July/060042.html_ > > _______________________________________________ > erlang-questions mailing list_ > __erlang-questions@REDACTED _ > __http://erlang.org/mailman/listinfo/erlang-questions________________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont > ?tablis ? l'intention exclusive des destinataires et les informations > qui y figurent sont strictement confidentielles. Toute utilisation de > ce Message non conforme ? sa destination, toute diffusion ou toute > publication totale ou partielle, est interdite sauf autorisation expresse. > > Si vous n'?tes pas le destinataire de ce Message, il vous est interdit > de le copier, de le faire suivre, de le divulguer ou d'en utiliser > tout ou partie. Si vous avez re?u ce Message par erreur, merci de le > supprimer de votre syst?me, ainsi que toutes ses copies, et de n'en > garder aucune trace sur quelque support que ce soit. Nous vous > remercions ?galement d'en avertir imm?diatement l'exp?diteur par > retour du message. > > Il est impossible de garantir que les communications par messagerie > ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de > toute erreur ou virus. > ____________________________________________________ > > This message and any attachments (the 'Message') are intended solely > for the addressees. The information contained in this Message is > confidential. Any use of information contained in this Message not in > accord with its purpose, any dissemination or disclosure, either whole > or partial, is prohibited except formal approval. > > If you are not the addressee, you may not copy, forward, disclose or > use any part of it. If you have received this message in error, please > delete it and all copies from your system and notify the sender > immediately by return message. > > E-mail communication cannot be guaranteed to be timely secure, error > or virus-free. > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From rvirding@REDACTED Sat Nov 22 03:03:32 2014 From: rvirding@REDACTED (Robert Virding) Date: Sat, 22 Nov 2014 03:03:32 +0100 Subject: [erlang-questions] The -module line In-Reply-To: References: <546F2044.9010908@ninenines.eu> <546F3A94.4070908@ninenines.eu> Message-ID: You might run into problems taking the module name from the file name on systems which the file names are case independent. Robert On 21 November 2014 20:41, Imants Cekusins wrote: > It is possible to compile .erl code with module different from file name. > > In > http://www.erlang.org/doc/man/compile.html > > Search for > > *no_error_module_mismatch* > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lhfcws@REDACTED Sat Nov 22 05:07:02 2014 From: lhfcws@REDACTED (Lhfcws GMail) Date: Sat, 22 Nov 2014 12:07:02 +0800 Subject: [erlang-questions] What kind of objects can be a message type? In-Reply-To: References: <546F2556.2010303@tail-f.com> <546F2701.40103@tail-f.com> <2D7716C4-E0CF-4228-8887-CEF4A762869B@gmail.com> Message-ID: <25D4A257-F18C-42EB-97C2-C9EE82B32C65@gmail.com> That?s really fantastic!!! And Socket also works! That?s to say, we can remotely open file or use the same driver by message passing. Thanks again! ? 2014?11?21??23:25?T Ty ??? > You can pass Pid and References in a message using ! > > This is an example of how trivial writing a FTP-like client in Erlang. > > On machine 1. > File = file:open(FileName, [read]). > PidOfProcessOnMachine2 ! {remote_file, File}. > > On machine 2. > receive > {remote_file, File} -> > {ok, Data} = file:read(File, 1000000), > write_to_local_disk(Data) > end > > Now, that said I have only tried it with File /Pid references and not with Socket. > > On Fri, Nov 21, 2014 at 12:05 PM, Lhfcws GMail wrote: > Thanks a lot! gen_tcp:controlling_process is indeed what I want. > And let me confirm your answer: > 1. Objects like socket and file handler can be pass in spawn arguments. > 2. But they can not be passed by ! message passing. > > Am I right? > -------------- > > ? 2014?11?21??19:50?Niclas Eklund ??? > > > Hello again! > > > > Forgot to answer your second question. Yes, you can spawn a new process that will "own" the connection, but then you need to use this function - http://www.erlang.org/doc/man/gen_tcp.html#controlling_process-2 You typically want to keep the listen process rather lightweight (do very little) since using blocking accept is common and then the listen process needs to be restarted when doing a live code upgrade. See for example this code - https://github.com/erlang/otp/blob/maint/lib/orber/src/orber_iiop_net_accept.erl It's doing a bit more, i.e. only allows a certain number of concurrent incoming requests, but as you can see it's minimal. > > > > /Nick > > > > On 11/21/2014 12:43 PM, Niclas Eklund wrote: > >> Hi! > >> > >> I'm afraid that you don't handle the returned values correctly. See this page - http://www.erlang.org/doc/man/gen_tcp.html - for more info. Should be: > >> > >> server() -> > >> {ok, LSock} = gen_tcp:listen(5678, [binary, {packet, 0}, > >> {active, false}]), > >> {ok, Sock} = gen_tcp:accept(LSock), > >> {ok, Bin} = do_recv(Sock, []), > >> ok = gen_tcp:close(Sock), > >> Bin. > >> > >> do_recv(Sock, Bs) -> > >> case gen_tcp:recv(Sock, 0) of > >> {ok, B} -> > >> do_recv(Sock, [Bs, B]); > >> {error, closed} -> > >> {ok, list_to_binary(Bs)} > >> end. > >> > >> /Nick > >> > >> > >> > >> On 11/21/2014 08:44 AM, Lhfcws GMail wrote: > >>> Hi, I?m new to Erlang. > >>> I have a question about the message passed among the Erlang processes. > >>> 1. I create a Socket or FileHandler, can I use ! to send it to another process and still works well? > >>> > >>> demo code: > >>> > >>> ListenSocket = gen_tcp:listen(1055, [{active, true}]), > >>> Socket = gen_tcp:accept(ListenSocket), > >>> From ! {new_socket, Socket}. > >>> > >>> > >>> 2. What if I invoking ?spawn? to pass the Socket to the new process ? > >>> > >>> demo code: > >>> > >>> NewProcess = fun(Socket) -> ? end. > >>> > >>> ListenSocket = gen_tcp:listen(1055, [{active, true}]), > >>> Socket = gen_tcp:accept(ListenSocket), > >>> spawn(NewProcess, Socket). > >>> > >>> ======= > >>> > >>> I know that the message in the receiver is a copy of the one in the sender, but I?m a little confused about the questions above. Could someone just explain this? > >>> Thank you. > >>> _______________________________________________ > >>> erlang-questions mailing list > >>> erlang-questions@REDACTED > >>> http://erlang.org/mailman/listinfo/erlang-questions > >> > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From imantc@REDACTED Sat Nov 22 09:56:03 2014 From: imantc@REDACTED (Imants Cekusins) Date: Sat, 22 Nov 2014 09:56:03 +0100 Subject: [erlang-questions] Distributed application and netsplit In-Reply-To: References: <98965.1416551316@snookles.snookles.com> Message-ID: Karolis asked A) how to solve netsplit when running distributed apps, didn't he? Simply to help understand, is Andrew's point about B) whether to run distributed apps at all; C) distribute, but use the suggested (by Andrew) tools; D) do not use application: load/2 on its own - with or without the suggested (by me) nesplit fix? If C, would the same app misbehaving with D, work ok with C? -------------- next part -------------- An HTML attachment was scrubbed... URL: From tao.zhou2009@REDACTED Sat Nov 22 09:50:23 2014 From: tao.zhou2009@REDACTED (tao.zhou2009) Date: Sat, 22 Nov 2014 16:50:23 +0800 Subject: [erlang-questions] is there a generic file fetcher anywhere? In-Reply-To: References: Message-ID: <5E801830EB184444AC540761BCA29F3B@gmail.com> https://github.com/CocoaPods/cocoapods-downloader A small library that provides downloaders for various source types (HTTP/SVN/Git/Mercurial) this ruby lib may be help? -- tao.zhou2009 Sent with Sparrow (http://www.sparrowmailapp.com/?sig) On Tuesday, November 18, 2014 at 6:10 PM, Joe Armstrong wrote: > Hello, > > Has anybody written a "generic file fetcher" - I'd like to fetch files > or collections of files from various remote locations (Git, SVG, HTTP, FTP etc) > handling proxies etc. on the way > > Is there any generic code to do this? > > I took a quick peek at the rebar source code since I know it can fetch from git > but it just does a os:cmd("git fetch origin ...") > > It strikes me that it would be a good idea to isolate code like this into a > separate library and make it more generic. This would be rather useful > > Cheers > > /Joe > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED (mailto:erlang-questions@REDACTED) > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric.pailleau@REDACTED Sat Nov 22 13:41:43 2014 From: eric.pailleau@REDACTED (PAILLEAU Eric) Date: Sat, 22 Nov 2014 13:41:43 +0100 Subject: [erlang-questions] The -module line In-Reply-To: References: <546F2044.9010908@ninenines.eu> <546F3A94.4070908@ninenines.eu> Message-ID: <54708487.9010107@wanadoo.fr> Hi, It should be more usefull to allow namespace. This could allow running two differents modules with same name. As far hot code change allow to run two same module version at same time, I suppose it could be possible to run two differents module of same name but in different namespace. A big change in Erlang paradigm, but closer of git philosophy, where clones are same module/application but possibly differents. running two differents git instances of same application in same node, could be very usefull to compare perfs or behaviours. Regards Le 22/11/2014 03:03, Robert Virding a ?crit : > You might run into problems taking the module name from the file name on > systems which the file names are case independent. > > Robert > > > On 21 November 2014 20:41, Imants Cekusins > wrote: > > It is possible to compile .erl code with module different from file > name. > > In > http://www.erlang.org/doc/man/compile.html > > Search for > > *no_error_module_mismatch* > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From donpedrothird@REDACTED Sat Nov 22 19:33:19 2014 From: donpedrothird@REDACTED (John Doe) Date: Sat, 22 Nov 2014 22:33:19 +0400 Subject: [erlang-questions] analogue of lists:keyfind and other lists:key*** functions for lists which contain maps Message-ID: It would be nice to extend lists module with a set of optimized functions like lists:key*** for lists containing maps. -------------- next part -------------- An HTML attachment was scrubbed... URL: From quuoms@REDACTED Sun Nov 23 08:12:34 2014 From: quuoms@REDACTED (=?UTF-8?B?5YWo5Y2T?=) Date: Sun, 23 Nov 2014 15:12:34 +0800 Subject: [erlang-questions] About a simple program Message-ID: Hi, I write a simple Erlang program,But it doesnot function like what i think. Here is my source code. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -module(exception). -export([start/0]). generate_exception(1) -> a; generate_exception(2) -> throw(throw_exception); generate_exception(3) -> error(error_exception); generate_exception(4) -> exit(exit_exception); generate_exception(5) -> {'EXIT',a}. start() -> lists:foreach(fun (I) -> try generate_exception(I) of Res -> Res catch throw:T -> T; exit:E -> E; error: Er -> Er after io:format("exec this line:~p~n",[I]) end end,lists:seq(1,5)). %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% I compile it and run it ,the result is: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% exec this line:1 exec this line:2 exec this line:3 exec this line:4 exec this line:5 ok %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% It seems that no exception has been caught, why?Or,I have something wrong? By the way,My Otp version is: Erlang/OTP 17,Install from source code, OS is Ubuntu 14.04. Thank you ! From vladdu55@REDACTED Sun Nov 23 09:35:52 2014 From: vladdu55@REDACTED (Vlad Dumitrescu) Date: Sun, 23 Nov 2014 09:35:52 +0100 Subject: [erlang-questions] About a simple program In-Reply-To: References: Message-ID: Hi! What would you expect the code would do? Try replacing lists:foreach with lists:map and notice that the result is [a, throw_exception, error_exception, exit_exception, {'EXIT',a}]. Maybe this is what you wanted? best regards, Vlad On Sun, Nov 23, 2014 at 8:12 AM, ?? wrote: > Hi, > I write a simple Erlang program,But it doesnot function like what i > think. > Here is my source code. > > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > -module(exception). > -export([start/0]). > > generate_exception(1) -> > a; > generate_exception(2) -> > throw(throw_exception); > generate_exception(3) -> > error(error_exception); > generate_exception(4) -> > exit(exit_exception); > generate_exception(5) -> > {'EXIT',a}. > > start() -> > lists:foreach(fun (I) -> try generate_exception(I) of > Res -> Res > catch > throw:T -> T; > exit:E -> E; > error: Er -> Er > after > io:format("exec this line:~p~n",[I]) > end > end,lists:seq(1,5)). > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > I compile it and run it ,the result is: > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > exec this line:1 > exec this line:2 > exec this line:3 > exec this line:4 > exec this line:5 > ok > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > > It seems that no exception has been caught, why?Or,I have something wrong? > By the way,My Otp version is: Erlang/OTP 17,Install from source code, > OS is Ubuntu 14.04. > > Thank you ! > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fritchie@REDACTED Sun Nov 23 15:16:56 2014 From: fritchie@REDACTED (Scott Lystig Fritchie) Date: Sun, 23 Nov 2014 08:16:56 -0600 Subject: [erlang-questions] Distributed application and netsplit In-Reply-To: Message of "Fri, 21 Nov 2014 09:04:12 PST." Message-ID: <20142.1416752216@snookles.snookles.com> At the risk of sounding like we aren't in violent agreement when it's quite likely that we are indeed agreeing... Felix Gallo wrote: fg> Disregarding for a moment the incongruousness of installing a Riak fg> cluster to try to handle that case, there's nothing Riak could do to fg> even help out, [...] I'm not suggesting that Riak KV could fix that problem. (Though, a small application of riak_ensemble *could*. But those two apps have entirely different availability profiles and behavior/semantic guarantees.) I don't believe that Andrew was suggesting it, either. My apologies if our ties to a common employer was suggesting paths or solutions related to that employer's products. I think that both Andrew and I are suggesting that sometimes you really do need something more than dirt on a wound[1] to avoid bleeding to death[1b]. Many folks use ZooKeeper[2]. The OpenReplica[3] paper suggests an alternative to ZooKeeper, one of many others. I'm likely agreeing with Felix and Andrew both that when the dirt-or-better-than-dirt mechanism has told you about a problem, how you react to that problem has no universal answer[4]. -Scott[6] [1] http://www.urbandictionary.com/define.php?term=rub%20some%20dirt%20on%20it [1b] I'm not suggesting that the Erlang/OTP application controller is dirt. However, it was designed by Ericsson to operate in a hardware environment where it it far more likely to work correctly than in a general data center or (gasp!) EC2 environment. [2] After all, "Rub some ZooKeeper on it" is as good advice as "Rub some dirt on it": both work well in some cases and are ineffective in others. [3] https://ecommons.library.cornell.edu/bitstream/1813/29009/2/OpenReplica.pdf [4] OK, alright, "It depends on your app," is the correct answer. P.S. Many systems use ZooKeeper or other netsplit-tolerant[5] tools to come close to a universal answer, which is "believe that process/machine/actor/thingie and nothing else." And those systems are willing to accept unavailability when those tools can't give an unambiguous answer. [5] For varying (and maddening) definitions of "tolerant". [6] Who apologizes for mixing up two issues when they should have been kept separate. However, like peanut butter and chocolate, or phyllo dough and honey, they are frequently difficult to keep separate in practice. From lukas@REDACTED Sun Nov 23 15:43:23 2014 From: lukas@REDACTED (Lukas Larsson) Date: Sun, 23 Nov 2014 15:43:23 +0100 Subject: [erlang-questions] analogue of lists:keyfind and other lists:key*** functions for lists which contain maps In-Reply-To: References: Message-ID: Hello, I agree with you that in certain situations they would be very convenient. I've myself already written these functions in a couple of modules. How do you imagine the semantics would work? It is fairly straight forward when the list only contains maps or tuples. lists:keyfind(you, name, [#{ name => me, from => here }, #{ name => you, from => there}]) returns #{ name => you, from => there}. But if we start to mix them some interesting corner cases pop up: lists:keyfind(2, 1,[#{ 2 => 3 },{2, 4},#{ 1 => 2 } ]) returns {2,4} or #{ 1 => 2 }? lists:keyfind(1, id, [{1,2}, #{ id => 2}, #{ id => 1}) returns #{ id => 1 } or crashes as id is not a valid key for {1,2}? Or error cases of today: lists:keyfind(2, id, []) today gives badarg, but should return false for a map list? The safest way to introduce these functions would be to create new ones called something like lists:mkeyfind, but if possible it would be very neat to use the same API. It really is unfortunate that lists:keyfind is so forgiving today, you can mix and match whatever you want in the list and keyfind will quietly ignore it. Also we have to consider which of the lists:key* functions we want a map counterpart? Does it make sense for all of them? Thoughts? Ideas? Suggestions? Lukas On Sat, Nov 22, 2014 at 7:33 PM, John Doe wrote: > It would be nice to extend lists module with a set of optimized functions > like lists:key*** for lists containing maps. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sergej.jurecko@REDACTED Sun Nov 23 16:04:18 2014 From: sergej.jurecko@REDACTED (Sergej Jurecko) Date: Sun, 23 Nov 2014 16:04:18 +0100 Subject: [erlang-questions] analogue of lists:keyfind and other lists:key*** functions for lists which contain maps In-Reply-To: References: Message-ID: <46BE5C2F-31CB-414D-A452-80CACD579607@gmail.com> I just converted a ~5k lines of code project into using maps. I wrote myself mapfind(Key,Value,L) and mapstore(Key,Val,L,Map). That was all I needed. I did not care about compatibility with other types in list. Is worrying about mixing tuples/records with maps in a list really necessary? In the 6+ years of writing erlang, I don?t think I?ve had a list of mixed types more than a couple of times. Sergej On 23 Nov 2014, at 15:43, Lukas Larsson wrote: > Hello, > > I agree with you that in certain situations they would be very convenient. I've myself already written these functions in a couple of modules. > > How do you imagine the semantics would work? It is fairly straight forward when the list only contains maps or tuples. > > lists:keyfind(you, name, [#{ name => me, from => here }, #{ name => you, from => there}]) returns #{ name => you, from => there}. > > But if we start to mix them some interesting corner cases pop up: > > lists:keyfind(2, 1,[#{ 2 => 3 },{2, 4},#{ 1 => 2 } ]) returns {2,4} or #{ 1 => 2 }? > lists:keyfind(1, id, [{1,2}, #{ id => 2}, #{ id => 1}) returns #{ id => 1 } or crashes as id is not a valid key for {1,2}? > > Or error cases of today: > > lists:keyfind(2, id, []) today gives badarg, but should return false for a map list? > > The safest way to introduce these functions would be to create new ones called something like lists:mkeyfind, but if possible it would be very neat to use the same API. It really is unfortunate that lists:keyfind is so forgiving today, you can mix and match whatever you want in the list and keyfind will quietly ignore it. > > Also we have to consider which of the lists:key* functions we want a map counterpart? Does it make sense for all of them? > > Thoughts? Ideas? Suggestions? > > Lukas > > > On Sat, Nov 22, 2014 at 7:33 PM, John Doe wrote: > It would be nice to extend lists module with a set of optimized functions like lists:key*** for lists containing maps. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Sun Nov 23 16:26:54 2014 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Sun, 23 Nov 2014 17:26:54 +0200 Subject: [erlang-questions] analogue of lists:keyfind and other lists:key*** functions for lists which contain maps In-Reply-To: <46BE5C2F-31CB-414D-A452-80CACD579607@gmail.com> References: <46BE5C2F-31CB-414D-A452-80CACD579607@gmail.com> Message-ID: <5471FCBE.3040101@ninenines.eu> I would +1 that. A tuple list and a map list are two different things, and it would be very unusual to see one with both. Also would like to add that as they are operating on different types there should be separate functions (mapfind and mapstore are indeed good names). On 11/23/2014 05:04 PM, Sergej Jurecko wrote: > I just converted a ~5k lines of code project into using maps. I wrote > myself mapfind(Key,Value,L) and mapstore(Key,Val,L,Map). That was all I > needed. I did not care about compatibility with other types in list. > > Is worrying about mixing tuples/records with maps in a list really > necessary? In the 6+ years of writing erlang, I don?t think I?ve had a > list of mixed types more than a couple of times. > > > Sergej > > On 23 Nov 2014, at 15:43, Lukas Larsson > wrote: > >> Hello, >> >> I agree with you that in certain situations they would be very >> convenient. I've myself already written these functions in a couple of >> modules. >> >> How do you imagine the semantics would work? It is fairly straight >> forward when the list only contains maps or tuples. >> >> lists:keyfind(you, name, [#{ name => me, from => here }, #{ name => >> you, from => there}]) returns #{ name => you, from => there}. >> >> But if we start to mix them some interesting corner cases pop up: >> >> lists:keyfind(2, 1,[#{ 2 => 3 },{2, 4},#{ 1 => 2 } ]) returns {2,4} or >> #{ 1 => 2 }? >> lists:keyfind(1, id, [{1,2}, #{ id => 2}, #{ id => 1}) returns #{ id >> => 1 } or crashes as id is not a valid key for {1,2}? >> >> Or error cases of today: >> >> lists:keyfind(2, id, []) today gives badarg, but should return false >> for a map list? >> >> The safest way to introduce these functions would be to create new >> ones called something like lists:mkeyfind, but if possible it would be >> very neat to use the same API. It really is unfortunate that >> lists:keyfind is so forgiving today, you can mix and match whatever >> you want in the list and keyfind will quietly ignore it. >> >> Also we have to consider which of the lists:key* functions we want a >> map counterpart? Does it make sense for all of them? >> >> Thoughts? Ideas? Suggestions? >> >> Lukas >> >> >> On Sat, Nov 22, 2014 at 7:33 PM, John Doe > > wrote: >> >> It would be nice to extend lists module with a set of optimized >> functions like lists:key*** for lists containing maps. >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin http://ninenines.eu From denis.justinek@REDACTED Sun Nov 23 20:38:36 2014 From: denis.justinek@REDACTED (Denis Justinek) Date: Sun, 23 Nov 2014 20:38:36 +0100 Subject: [erlang-questions] analogue of lists:keyfind and other lists:key*** functions for lists which contain maps In-Reply-To: <5471FCBE.3040101@ninenines.eu> References: <46BE5C2F-31CB-414D-A452-80CACD579607@gmail.com> <5471FCBE.3040101@ninenines.eu> Message-ID: In case someone finds it useful; Sergej's implementations of mapstore/mapfind are available on github (bkdcore / butil module): https://github.com/biokoda/bkdcore/blob/master/src/butil.erl 2014-11-23 16:26 GMT+01:00 Lo?c Hoguin : > I would +1 that. A tuple list and a map list are two different things, and > it would be very unusual to see one with both. > > Also would like to add that as they are operating on different types there > should be separate functions (mapfind and mapstore are indeed good names). > > On 11/23/2014 05:04 PM, Sergej Jurecko wrote: > >> I just converted a ~5k lines of code project into using maps. I wrote >> myself mapfind(Key,Value,L) and mapstore(Key,Val,L,Map). That was all I >> needed. I did not care about compatibility with other types in list. >> >> Is worrying about mixing tuples/records with maps in a list really >> necessary? In the 6+ years of writing erlang, I don?t think I?ve had a >> list of mixed types more than a couple of times. >> >> >> Sergej >> >> On 23 Nov 2014, at 15:43, Lukas Larsson > > wrote: >> >> Hello, >>> >>> I agree with you that in certain situations they would be very >>> convenient. I've myself already written these functions in a couple of >>> modules. >>> >>> How do you imagine the semantics would work? It is fairly straight >>> forward when the list only contains maps or tuples. >>> >>> lists:keyfind(you, name, [#{ name => me, from => here }, #{ name => >>> you, from => there}]) returns #{ name => you, from => there}. >>> >>> But if we start to mix them some interesting corner cases pop up: >>> >>> lists:keyfind(2, 1,[#{ 2 => 3 },{2, 4},#{ 1 => 2 } ]) returns {2,4} or >>> #{ 1 => 2 }? >>> lists:keyfind(1, id, [{1,2}, #{ id => 2}, #{ id => 1}) returns #{ id >>> => 1 } or crashes as id is not a valid key for {1,2}? >>> >>> Or error cases of today: >>> >>> lists:keyfind(2, id, []) today gives badarg, but should return false >>> for a map list? >>> >>> The safest way to introduce these functions would be to create new >>> ones called something like lists:mkeyfind, but if possible it would be >>> very neat to use the same API. It really is unfortunate that >>> lists:keyfind is so forgiving today, you can mix and match whatever >>> you want in the list and keyfind will quietly ignore it. >>> >>> Also we have to consider which of the lists:key* functions we want a >>> map counterpart? Does it make sense for all of them? >>> >>> Thoughts? Ideas? Suggestions? >>> >>> Lukas >>> >>> >>> On Sat, Nov 22, 2014 at 7:33 PM, John Doe >> > wrote: >>> >>> It would be nice to extend lists module with a set of optimized >>> functions like lists:key*** for lists containing maps. >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -- > Lo?c Hoguin > http://ninenines.eu > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Denis Justinek - @: denis.justinek@REDACTED - www: http://about.me/djustinek -------------- next part -------------- An HTML attachment was scrubbed... URL: From imantc@REDACTED Sun Nov 23 21:18:49 2014 From: imantc@REDACTED (Imants Cekusins) Date: Sun, 23 Nov 2014 21:18:49 +0100 Subject: [erlang-questions] About a simple program In-Reply-To: References: Message-ID: How about replacing throw:T -> T; exit:E -> E; error: Er -> Er With: throw:T -> throw(T); exit:E -> exit(E); error: Er -> error(Er) Does it behave as expected? -------------- next part -------------- An HTML attachment was scrubbed... URL: From imantc@REDACTED Sun Nov 23 21:24:54 2014 From: imantc@REDACTED (Imants Cekusins) Date: Sun, 23 Nov 2014 21:24:54 +0100 Subject: [erlang-questions] About a simple program In-Reply-To: References: Message-ID: .. or replace each clause with io:format("caught this: ~p:~p~n",[I, T]) T|E|Er as the last variable -------------- next part -------------- An HTML attachment was scrubbed... URL: From donpedrothird@REDACTED Sun Nov 23 21:32:18 2014 From: donpedrothird@REDACTED (John Doe) Date: Mon, 24 Nov 2014 00:32:18 +0400 Subject: [erlang-questions] analogue of lists:keyfind and other lists:key*** functions for lists which contain maps In-Reply-To: References: Message-ID: Yes, I was thinking about separate functions, like mapfind, mapmerge etc, not about making a Frankenstein from lists:key** ones. Of course there's no problem to write such simple library myself, but having built-in fast bifs would be just more convenient as the more and more people will model objects by maps instead of proplists, use lists of such objects in their programs and expect CRUD operations over such lists. There are 12 lists:key*** functions at the moment and it looks like it makes sense to replicate all of them as lists:map*** ones. The edge error cases will be "what to do when a map does not contain the key?". In this case lists:key*** behave differently in different functions - lists:keymap throws and exception when list contains a single non-tuple, lists:keydelete allows such lists. 2014-11-23 17:43 GMT+03:00 Lukas Larsson : > Hello, > > I agree with you that in certain situations they would be very convenient. > I've myself already written these functions in a couple of modules. > > How do you imagine the semantics would work? It is fairly straight forward > when the list only contains maps or tuples. > > lists:keyfind(you, name, [#{ name => me, from => here }, #{ name => you, > from => there}]) returns #{ name => you, from => there}. > > But if we start to mix them some interesting corner cases pop up: > > lists:keyfind(2, 1,[#{ 2 => 3 },{2, 4},#{ 1 => 2 } ]) returns {2,4} or #{ > 1 => 2 }? > lists:keyfind(1, id, [{1,2}, #{ id => 2}, #{ id => 1}) returns #{ id => 1 > } or crashes as id is not a valid key for {1,2}? > > Or error cases of today: > > lists:keyfind(2, id, []) today gives badarg, but should return false for a > map list? > > The safest way to introduce these functions would be to create new ones > called something like lists:mkeyfind, but if possible it would be very neat > to use the same API. It really is unfortunate that lists:keyfind is so > forgiving today, you can mix and match whatever you want in the list and > keyfind will quietly ignore it. > > Also we have to consider which of the lists:key* functions we want a map > counterpart? Does it make sense for all of them? > > Thoughts? Ideas? Suggestions? > > Lukas > > > On Sat, Nov 22, 2014 at 7:33 PM, John Doe wrote: > >> It would be nice to extend lists module with a set of optimized functions >> like lists:key*** for lists containing maps. >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Sun Nov 23 21:45:47 2014 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Sun, 23 Nov 2014 22:45:47 +0200 Subject: [erlang-questions] analogue of lists:keyfind and other lists:key*** functions for lists which contain maps In-Reply-To: References: Message-ID: <5472477B.1030102@ninenines.eu> On 11/23/2014 10:32 PM, John Doe wrote: > There are 12 lists:key*** functions at the moment and it looks like it > makes sense to replicate all of them as lists:map*** ones. The edge > error cases will be "what to do when a map does not contain the key?". > In this case lists:key*** behave differently in different functions - > lists:keymap throws and exception when list contains a single non-tuple, > lists:keydelete allows such lists. keysearch and keytake should probably not be considered as they're basically duplicating functionality in other functions (and should be deprecated and removed from OTP at some point in the far future in my opinion). > 2014-11-23 17:43 GMT+03:00 Lukas Larsson >: > > Hello, > > I agree with you that in certain situations they would be very > convenient. I've myself already written these functions in a couple > of modules. > > How do you imagine the semantics would work? It is fairly straight > forward when the list only contains maps or tuples. > > lists:keyfind(you, name, [#{ name => me, from => here }, #{ name => > you, from => there}]) returns #{ name => you, from => there}. > > But if we start to mix them some interesting corner cases pop up: > > lists:keyfind(2, 1,[#{ 2 => 3 },{2, 4},#{ 1 => 2 } ]) returns {2,4} > or #{ 1 => 2 }? > lists:keyfind(1, id, [{1,2}, #{ id => 2}, #{ id => 1}) returns #{ id > => 1 } or crashes as id is not a valid key for {1,2}? > > Or error cases of today: > > lists:keyfind(2, id, []) today gives badarg, but should return false > for a map list? > > The safest way to introduce these functions would be to create new > ones called something like lists:mkeyfind, but if possible it would > be very neat to use the same API. It really is unfortunate that > lists:keyfind is so forgiving today, you can mix and match whatever > you want in the list and keyfind will quietly ignore it. > > Also we have to consider which of the lists:key* functions we want a > map counterpart? Does it make sense for all of them? > > Thoughts? Ideas? Suggestions? > > Lukas > > > On Sat, Nov 22, 2014 at 7:33 PM, John Doe > wrote: > > It would be nice to extend lists module with a set of optimized > functions like lists:key*** for lists containing maps. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin http://ninenines.eu From lhfcws@REDACTED Mon Nov 24 04:51:02 2014 From: lhfcws@REDACTED (Lhfcws GMail) Date: Mon, 24 Nov 2014 11:51:02 +0800 Subject: [erlang-questions] About a simple program In-Reply-To: References: Message-ID: <800983D2-B37D-43F5-A32B-1C2D205C757A@gmail.com> Because you did not output the exit or error, and after block is just like finally block in Java, so it?ll always be executed. Just try modify catch block as follow: throw:T -> io:write(T), io:format("~n"); exit:E -> io:write(E), io:format("~n"); error:Er -> io:write(Er), io:format("~n") ? 2014?11?23??15:12??? ??? > Hi, > I write a simple Erlang program,But it doesnot function like what i think. > Here is my source code. > > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > -module(exception). > -export([start/0]). > > generate_exception(1) -> > a; > generate_exception(2) -> > throw(throw_exception); > generate_exception(3) -> > error(error_exception); > generate_exception(4) -> > exit(exit_exception); > generate_exception(5) -> > {'EXIT',a}. > > start() -> > lists:foreach(fun (I) -> try generate_exception(I) of > Res -> Res > catch > throw:T -> T; > exit:E -> E; > error: Er -> Er > after > io:format("exec this line:~p~n",[I]) > end > end,lists:seq(1,5)). > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > I compile it and run it ,the result is: > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > exec this line:1 > exec this line:2 > exec this line:3 > exec this line:4 > exec this line:5 > ok > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > > It seems that no exception has been caught, why?Or,I have something wrong? > By the way,My Otp version is: Erlang/OTP 17,Install from source code, > OS is Ubuntu 14.04. > > Thank you ! > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From llbgurs@REDACTED Mon Nov 24 08:06:16 2014 From: llbgurs@REDACTED (linbo liao) Date: Mon, 24 Nov 2014 15:06:16 +0800 Subject: [erlang-questions] How to start application before all eunit cases Message-ID: My Erlang project managed by rebar, it divides to different module. -pro |-- rel |-- src |--- module1 |--- module2 |-- test |--- module1_tests.erl |--- module2_tests.erl and for each module*_tests.erl, Use Eunit Fixtures to setup environment. For example, module1_test_() -> {setup, fun setup/0, fun clean_up/1, fun (SetupData) -> [ test_function(SetupData) ] end}. setup() -> pro:start(), ok. clean_up(_) -> pro:stop(). And Makefile is: test: ERL_AFLAGS="-config rel/pro/etc/app.config" $(REBAR) compile eunit skip_deps=true Here I encounter a problem, since I have many test module in test/, each test module will start and stop application for whole executing flow. Sometimes start application will be failed, told not find app.config configure file, not sure why. So I think is there a way to start application before all test module? Thanks, Linbo -------------- next part -------------- An HTML attachment was scrubbed... URL: From askjuise@REDACTED Mon Nov 24 10:29:32 2014 From: askjuise@REDACTED (Alexander Petrovsky) Date: Mon, 24 Nov 2014 13:29:32 +0400 Subject: [erlang-questions] Problem with erlang:memory(processes_used) Message-ID: Hi! On Erlang v.17, the function erlang:memory(process_used) shows that summary size on used memory near 17 (exa?)byte. The erlang application build on the CI server as erlang release and then rolled on the cluster. CI server: > erlang:system_info(system_version). "Erlang/OTP 17 [erts-6.2] [source] [64-bit] [smp:4:4] [async-threads:10] [kernel-poll:false]\n" # uname -a Linux ci.our.net 3.2.0-4-amd64 #1 SMP Debian 3.2.54-2 x86_64 GNU/Linux Target server: > erlang:system_info(system_version). "Erlang/OTP 17 [erts-6.2] [source] [64-bit] [smp:4:4] [async-threads:10] [kernel-poll:true]\n" > erlang:memory(processes_used). 18446744073687889792 > lists:sum([V || {memory, V} <- [erlang:process_info(Pid, memory) || Pid <- erlang:processes()]]). 15523320 # uname -a Linux n2432.our.net 3.2.0-4-amd64 #1 SMP Debian 3.2.63-2 x86_64 GNU/Linux # free -m total used free shared buffers cached Mem: 7978 3397 4581 0 289 1511 -/+ buffers/cache: 1596 6382 Swap: 15625 0 15625 Can someone please help me with this? -- ?????????? ????????? / Alexander Petrovsky, Skype: askjuise Phone: +7 914 8 820 815 -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang@REDACTED Mon Nov 24 10:45:23 2014 From: erlang@REDACTED (Joe Armstrong) Date: Mon, 24 Nov 2014 10:45:23 +0100 Subject: [erlang-questions] Try-catch-after considered harmful. Message-ID: Is it just me, or is something wrong about try-catch-after? I keep reading a lot of articles, and replying to private posts concerning the correct us of try-catch-after. My first response is "don't use use try-catch-otherwise at all" let the process crash and let some other process fix up any damaging consequences of the crash. The best was to write error handling code is to write no error handling code at all. At a first approximation, all the libraries are designed to be used this way. If you open a resource like a file/socket/ets table/dets table/database table then if your program crashes the resource (which is represented by a process) will detect the crash and is supposed to "do the right thing" and clean up behind you. The try-catch syntax was deliberately chosen to be reminiscent of Java, the idea being that if you understood this try-catch consequence in Java you'd easily understand the Erlang code. The problem I see with this is that programmers with previous experience in Java are tempted to blindly convert sequential try-catch code in Java into sequential try-catch code in Erlang, but this is almost always the wrong thing to do. Beginners should be forbidden to use try-catch - the Erlang "way" is to spawn_link a regular process (ie a process that does not trap exits) and just let that process die if anything goes wrong. There should be no error trapping code in the spawned process. Call exit(Why) in every place where the behaviour is unspecified. The process doing the spawn_link should trap exits and report errors in the child processes. This is what all the supervisor and gen_my_aunties_hat stuff does in the standard libraries. If you really-really-really understand what changing the flow of execution in sequential code does then by all means try-catch-after - but the best advice is not to use it at all. There should really be a compiler warning - ** try-catch-after used - this is probably not the right way to do things Cheers /Joe From lukas@REDACTED Mon Nov 24 10:49:35 2014 From: lukas@REDACTED (Lukas Larsson) Date: Mon, 24 Nov 2014 10:49:35 +0100 Subject: [erlang-questions] Problem with erlang:memory(processes_used) In-Reply-To: References: Message-ID: Hello, Could you please provide the output of erlang:system_info({allocator,eheap_alloc}) and erlang:system_info({allocator,fix_alloc}). erlang:memory is based on the total of the first element in the blocks_size tuple of each instance for sbcs and mbcs. Something seems to be calculated or reported incorrectly there. Lukas On Mon, Nov 24, 2014 at 10:29 AM, Alexander Petrovsky wrote: > Hi! > > On Erlang v.17, the function erlang:memory(process_used) shows that > summary size on used memory near 17 (exa?)byte. The erlang application > build on the CI server as erlang release and then rolled on the cluster. > > CI server: > > > erlang:system_info(system_version). > "Erlang/OTP 17 [erts-6.2] [source] [64-bit] [smp:4:4] [async-threads:10] > [kernel-poll:false]\n" > > # uname -a > Linux ci.our.net 3.2.0-4-amd64 #1 SMP Debian 3.2.54-2 x86_64 GNU/Linux > > > Target server: > > > erlang:system_info(system_version). > "Erlang/OTP 17 [erts-6.2] [source] [64-bit] [smp:4:4] [async-threads:10] > [kernel-poll:true]\n" > > > erlang:memory(processes_used). > > 18446744073687889792 > > > lists:sum([V || {memory, V} <- [erlang:process_info(Pid, memory) || Pid > <- erlang:processes()]]). > 15523320 > > # uname -a > Linux n2432.our.net 3.2.0-4-amd64 #1 SMP Debian 3.2.63-2 x86_64 GNU/Linux > > # free -m > total used free shared buffers cached > Mem: 7978 3397 4581 0 289 1511 > -/+ buffers/cache: 1596 6382 > Swap: 15625 0 15625 > > Can someone please help me with this? > > -- > ?????????? ????????? / Alexander Petrovsky, > > Skype: askjuise > Phone: +7 914 8 820 815 > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From olivier.boudeville@REDACTED Mon Nov 24 10:53:50 2014 From: olivier.boudeville@REDACTED (Olivier BOUDEVILLE) Date: Mon, 24 Nov 2014 10:53:50 +0100 Subject: [erlang-questions] Configuring Messaging Back Pressure In-Reply-To: <546FC046.6050800@mail.com> References: <546E6EB5.90508@mail.com> <546FC046.6050800@mail.com> Message-ID: Hi, > I don't think you can rely on the scheduler alone if there are, say > ten producers giving work to one consumer asynchronously. Using > gen_server:call from the producer to the consumer could help here? Indeed, using synchronous operations (not necessarily involving OTP) between the producer(s) and the consumer(s) would balance the processings, at the expense of many more messages being sent. We could imagine sending one "ack" from the consumer(s) only every P messages received (a bit like a TCP window), but the logic would then be more complex to manage. On the other hand, if the scheduler could "naturally" slow down parts of the chain to make the work of loaded processes easier, this would be neat. This is a bit surprising that, even with a reduction count reduced by the number of messages in the receivers message queue multiplied by four, producers are still scheduled at least relatively often. Maybe the VM prefers to keep all cores busy even if it results in having low priority processes keeping on feeding already loaded processes. Maybe it shouldn't, especially if prefering stability over efficiency! (i.e. priority #1: do not saturate loaded processes; then only, priority #2: try to use all cores) Best, Olivier. > Cheers > > Andy > > On 21/11/2014 12:23, Olivier BOUDEVILLE wrote: > Hi, > > The reciprocal case (M producers and N consumers, with M >> N) may > also be a problem. > > I can share a simple program which seems to show that the > retaliation upon the producers is at least in some cases (where M is > the number of cores of the machine minus 1, and N=1) insufficient: > the program will crash and burn, exhausting the memory because of > the mailbox of the consumer becoming too large. Ideally the back- > pressure would increase steadily until the mailbox of the producer > decreases below reasonable limits. Currently, I think that the > developer has either to tune M by hand (which would depend on the > hardware resources...), or to perform some explicit yielding (with > timer:sleep(1) on the producers). Neither of them is very satisfactory. > > More generally, couldn't we hope that the scheduling logic could > cope with at least most of the M producers-N consumers scenarios, > without resorting to hints from the developer like specifying a > process flag or performing explicit yielding? Maybe (for the M >> N > case) it is just a matter of heavier punishing (ex: ultimately > exponential) whenever the receiving mailbox becomes too large. > > Best, > > Olivier. > --------------------------- > Olivier Boudeville > > EDF R&D : 1, avenue du G?n?ral de Gaulle, 92140 Clamart, France > D?partement SINETICS, groupe ASICS (I2A), bureau B-226 > Office : +33 1 47 65 59 58 / Mobile : +33 6 16 83 37 22 / Fax : +33 > 1 47 65 27 13 > > > darach@REDACTED > Envoy? par : erlang-questions-bounces@REDACTED > 21/11/2014 11:27 > > A > > atill@REDACTED > > cc > > erlang-questions@REDACTED > > Objet > > Re: [erlang-questions] Configuring Messaging Back Pressure > > > > > +1 > On 21 Nov 2014 04:53, "Andy Till" wrote: > Short outline of how erts_use_sender_punish works, please comment if > there are any errors in the description. > > erts_use_sender_punish is a flag hard coded to 1 (true), when it is > true a process sending messages to another process will have its > reduction count reduced by the number of messages in the receivers > message queue multiplied by four. Sending messages to processes > with zero messages in the queue is free in terms of reductions, but > sending messages to load queues is very expensive and will lead the > scheduler to context switch to another process more often. > > In the case where the the relationship between of producers and > consumers is one to one, this makes sense. It provides back > pressure when part of the system is loaded. > > In the case where there is one producer to many consumers, if one > consumer gets a flood of messages and the producer is punished then > the punishment is not only on the producer but on all other > consumers because the producer cannot create enough work for them as > it used all of its reductions. > > This makes performance unpredictable when work cannot be spread evenly. > > Could this be made to be configurable using a process flag? > > Cheers > > Andy > > Code: > > https://github.com/erlang/otp/blob/ > a70f0ce9b34e4db61dacb8db24f9ab5671ed4c8a/erts/emulator/beam/erl_init.c#L694 > https://github.com/erlang/otp/blob/ > 682a6082159568f40615f03d12d44ee70edd14c6/erts/emulator/beam/bif.c#L2058 > https://github.com/erlang/otp/blob/ > a8e12f7168c14cc765a63a51c838d065412795d7/erts/emulator/beam/erl_message.c#L504 > > Previous mentions and issues with this feature: > > http://erlang.org/pipermail/erlang-questions/2011-July/060314.html > http://erlang.org/pipermail/erlang-questions/2011-July/060042.html > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont > ?tablis ? l'intention exclusive des destinataires et les > informations qui y figurent sont strictement confidentielles. Toute > utilisation de ce Message non conforme ? sa destination, toute > diffusion ou toute publication totale ou partielle, est interdite > sauf autorisation expresse. > Si vous n'?tes pas le destinataire de ce Message, il vous est > interdit de le copier, de le faire suivre, de le divulguer ou d'en > utiliser tout ou partie. Si vous avez re?u ce Message par erreur, > merci de le supprimer de votre syst?me, ainsi que toutes ses copies, > et de n'en garder aucune trace sur quelque support que ce soit. Nous > vous remercions ?galement d'en avertir imm?diatement l'exp?diteur > par retour du message. > Il est impossible de garantir que les communications par messagerie > ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de > toute erreur ou virus. > ____________________________________________________ > This message and any attachments (the 'Message') are intended solely > for the addressees. The information contained in this Message is > confidential. Any use of information contained in this Message not > in accord with its purpose, any dissemination or disclosure, either > whole or partial, is prohibited except formal approval. > If you are not the addressee, you may not copy, forward, disclose or > use any part of it. If you have received this message in error, > please delete it and all copies from your system and notify the > sender immediately by return message. > E-mail communication cannot be guaranteed to be timely secure, error > or virus-free. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont ?tablis ? l'intention exclusive des destinataires et les informations qui y figurent sont strictement confidentielles. Toute utilisation de ce Message non conforme ? sa destination, toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse. Si vous n'?tes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez re?u ce Message par erreur, merci de le supprimer de votre syst?me, ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Nous vous remercions ?galement d'en avertir imm?diatement l'exp?diteur par retour du message. Il est impossible de garantir que les communications par messagerie ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute erreur ou virus. ____________________________________________________ This message and any attachments (the 'Message') are intended solely for the addressees. The information contained in this Message is confidential. Any use of information contained in this Message not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval. If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return message. E-mail communication cannot be guaranteed to be timely secure, error or virus-free. -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Mon Nov 24 10:55:21 2014 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Mon, 24 Nov 2014 11:55:21 +0200 Subject: [erlang-questions] Try-catch-after considered harmful. In-Reply-To: References: Message-ID: <54730089.7040905@ninenines.eu> There is at least one exception to this, and it has to do with sockets. Closing the socket on crash is fine but most protocols require you to send an error response before you go away (4xx or 5xx with "close" option in HTTP1, GOAWAY in HTTP2, for example). Sending this error is almost always a good thing because it makes sure the client knows the connection gets closed and will not get stuck with a half-open scenario. There is no way to send these responses from another process using spawn_link because the socket dies with the process that died. Otherwise yes, for most use cases I would agree with you, and would also add the lonely catch to the list of constructs to avoid (perhaps even more than try-catch-after). On 11/24/2014 11:45 AM, Joe Armstrong wrote: > Is it just me, or is something wrong about try-catch-after? > > I keep reading a lot of articles, and replying to private posts > concerning the correct us of try-catch-after. > > My first response is "don't use use try-catch-otherwise at all" let > the process crash and let some other process fix up any damaging > consequences of the crash. > > The best was to write error handling code is to write no error > handling code at all. > > At a first approximation, all the libraries are designed to be used > this way. If you open a resource like a file/socket/ets table/dets > table/database table then if your program crashes the resource (which > is represented by a process) will detect the crash and is supposed to > "do the right thing" and clean up behind you. > > The try-catch syntax was deliberately chosen to be reminiscent of > Java, the idea being that if you understood this try-catch consequence > in Java you'd easily understand the Erlang code. > > The problem I see with this is that programmers with previous > experience in Java are tempted to blindly convert sequential try-catch > code in Java into sequential try-catch code in Erlang, but this is > almost always the wrong thing to do. > > Beginners should be forbidden to use try-catch - the Erlang "way" is > to spawn_link a regular process (ie a process that does not trap > exits) and just let that process die if anything goes wrong. There > should be no error trapping code in the spawned process. Call > exit(Why) in every place where the behaviour is unspecified. > > The process doing the spawn_link should trap exits and report errors > in the child processes. This is what all the supervisor and > gen_my_aunties_hat stuff does in the standard libraries. > > If you really-really-really understand what changing the flow of > execution in sequential code does then by all means try-catch-after - > but the best advice is not to use it at all. > > There should really be a compiler warning - > > ** try-catch-after used - this is probably not the right way to do things > > Cheers > > /Joe > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin http://ninenines.eu From askjuise@REDACTED Mon Nov 24 10:59:43 2014 From: askjuise@REDACTED (Alexander Petrovsky) Date: Mon, 24 Nov 2014 13:59:43 +0400 Subject: [erlang-questions] Problem with erlang:memory(processes_used) In-Reply-To: References: Message-ID: Hi! Sure! The results in attachment. 2014-11-24 9:49 GMT+00:00 Lukas Larsson : > Hello, > > Could you please provide the output of > erlang:system_info({allocator,eheap_alloc}) > and erlang:system_info({allocator,fix_alloc}). erlang:memory is based on > the total of the first element in the blocks_size tuple of each instance > for sbcs and mbcs. Something seems to be calculated or reported incorrectly > there. > > Lukas > > On Mon, Nov 24, 2014 at 10:29 AM, Alexander Petrovsky > wrote: > >> Hi! >> >> On Erlang v.17, the function erlang:memory(process_used) shows that >> summary size on used memory near 17 (exa?)byte. The erlang application >> build on the CI server as erlang release and then rolled on the cluster. >> >> CI server: >> >> > erlang:system_info(system_version). >> "Erlang/OTP 17 [erts-6.2] [source] [64-bit] [smp:4:4] [async-threads:10] >> [kernel-poll:false]\n" >> >> # uname -a >> Linux ci.our.net 3.2.0-4-amd64 #1 SMP Debian 3.2.54-2 x86_64 GNU/Linux >> >> >> Target server: >> >> > erlang:system_info(system_version). >> "Erlang/OTP 17 [erts-6.2] [source] [64-bit] [smp:4:4] [async-threads:10] >> [kernel-poll:true]\n" >> >> > erlang:memory(processes_used). >> >> 18446744073687889792 >> >> > lists:sum([V || {memory, V} <- [erlang:process_info(Pid, memory) || Pid >> <- erlang:processes()]]). >> 15523320 >> >> # uname -a >> Linux n2432.our.net 3.2.0-4-amd64 #1 SMP Debian 3.2.63-2 x86_64 GNU/Linux >> >> # free -m >> total used free shared buffers cached >> Mem: 7978 3397 4581 0 289 1511 >> -/+ buffers/cache: 1596 6382 >> Swap: 15625 0 15625 >> >> Can someone please help me with this? >> >> -- >> ?????????? ????????? / Alexander Petrovsky, >> >> Skype: askjuise >> Phone: +7 914 8 820 815 >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -- ?????????? ????????? / Alexander Petrovsky, Skype: askjuise Phone: +7 914 8 820 815 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: eheap_alloc Type: application/octet-stream Size: 8989 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: fix_alloc Type: application/octet-stream Size: 10328 bytes Desc: not available URL: From davidnwelton@REDACTED Mon Nov 24 11:32:19 2014 From: davidnwelton@REDACTED (David Welton) Date: Mon, 24 Nov 2014 11:32:19 +0100 Subject: [erlang-questions] Try-catch-after considered harmful. In-Reply-To: References: Message-ID: I'm not going to disagree with you, but take the opportunity to encourage educators such as yourself to go a few steps beyond "letting it crash" from time to time. For instance, there are a lot of situations where you do not want your application to fall over because some component, such as a database is unavailable. There are various "circuit breakers" that can help in this situation, but they are not widely known: https://github.com/jlouis/fuse - these should be in books, or maybe even a gen_*something* of their own within OTP. It's something I've struggled with in a system I'm involved with. For instance, there's a GUI, and Erlang code it talks with. The Erlang code it talks with should be about the last thing to go down in the system. Indeed, it really shouldn't ever go down even if other bits and pieces are flopping around on the floor like fish out of water, as they restart their allotted restarts and then crash stuff higher up the chain. The GUI communication code should just calmly make note of the fact that various other subsystems are not functional, and thus the system as a whole is not functional, and let the user decide what to do, which might involve running some diagnostics, calling the service department to read out a somewhat-intelligible error message (no stack traces with 'nomatch': http://joearms.github.io/2013/04/28/Fail-fast-noisely-and-politely.html ) , or in the case of an IT pro, rebooting the machine and seeing what happens. We mostly have things figured out at this point, but I think it's an area where Erlang could use some more documentation and examples. This is a good article, by the way: http://jlouisramblings.blogspot.it/2010/11/on-erlang-state-and-crashes.html Thank you -- David N. Welton http://www.welton.it/davidw/ http://www.dedasys.com/ From jon@REDACTED Mon Nov 24 11:39:55 2014 From: jon@REDACTED (Jon Schneider) Date: Mon, 24 Nov 2014 10:39:55 -0000 Subject: [erlang-questions] About a simple program In-Reply-To: References: Message-ID: > Hi, > I write a simple Erlang program,But it doesnot function like what i > think. I think maybe it does do what you think but you haven't put anything in there to make this visible. Also missing is a catch-all case. start() -> lists:foreach(fun (I) -> J = try generate_exception(I) of Res -> Res catch throw:T -> T; exit:E -> E; error: Er -> Er; X1: X2 -> {X1,X2} end, io:format("exec this line:~p ~p~n",[I, J]) end, lists:seq(1,5)). then the output is exec this line:1 a exec this line:2 throw_exception exec this line:3 error_exception exec this line:4 exit_exception exec this line:5 {'EXIT',a} Jon From lcastro@REDACTED Mon Nov 24 12:40:42 2014 From: lcastro@REDACTED (Laura M. Castro) Date: Mon, 24 Nov 2014 12:40:42 +0100 Subject: [erlang-questions] Try-catch-after considered harmful. In-Reply-To: References: Message-ID: > The try-catch syntax was deliberately chosen to be reminiscent of > Java, the idea being that if you understood this try-catch consequence > in Java you'd easily understand the Erlang code. > > The problem I see with this is that programmers with previous > experience in Java are tempted to blindly convert sequential try-catch > code in Java into sequential try-catch code in Erlang In my experience, this is indeed the case. Not as often, but same goes for 'if' vs pattern-matching on function clauses. -- Laura M. Castro Department of Computer Science Universidade da Coru?a (Spain) http://www.madsgroup.org/staff/laura From russelldb@REDACTED Mon Nov 24 13:56:23 2014 From: russelldb@REDACTED (Russell Brown) Date: Mon, 24 Nov 2014 12:56:23 +0000 Subject: [erlang-questions] How to pass the length of binary to match into a function that matches the binary? Message-ID: Hi, Is there a way to do this: my_fun(Bin, Len, <>) -> %% do stuff with Bin ok. Len is unbound is the reason I can?t, but it seems like the sort of thing that would be useful if I could. Many thanks in advance Russell From imantc@REDACTED Mon Nov 24 14:18:20 2014 From: imantc@REDACTED (Imants Cekusins) Date: Mon, 24 Nov 2014 14:18:20 +0100 Subject: [erlang-questions] How to pass the length of binary to match into a function that matches the binary? In-Reply-To: References: Message-ID: Would this work: my_fun(<>) -> ok; my_fun(<>) -> ok. ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ulf@REDACTED Mon Nov 24 14:26:26 2014 From: ulf@REDACTED (Ulf Wiger) Date: Mon, 24 Nov 2014 14:26:26 +0100 Subject: [erlang-questions] How to pass the length of binary to match into a function that matches the binary? In-Reply-To: References: Message-ID: <0F74B700-D31B-443C-9324-690EE8BF3724@feuerlabs.com> On 24 Nov 2014, at 13:56, Russell Brown wrote: > Hi, > Is there a way to do this: > > my_fun(Bin, Len, <>) -> > %% do stuff with Bin > ok. > > Len is unbound is the reason I can?t, but it seems like the sort of thing that would be useful if I could. Yes, it?s slightly unfortunate, but you can write: my_fun(B, Len, Binary) -> case Binary of <> -> %% do stuff with B ok; ? end. BR, Ulf W Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. http://feuerlabs.com -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 496 bytes Desc: Message signed with OpenPGP using GPGMail URL: From russelldb@REDACTED Mon Nov 24 15:53:45 2014 From: russelldb@REDACTED (Russell Brown) Date: Mon, 24 Nov 2014 14:53:45 +0000 Subject: [erlang-questions] How to pass the length of binary to match into a function that matches the binary? In-Reply-To: <0F74B700-D31B-443C-9324-690EE8BF3724@feuerlabs.com> References: <0F74B700-D31B-443C-9324-690EE8BF3724@feuerlabs.com> Message-ID: <34C72DD2-A84D-442E-BAEC-AFC3C2D7D1A1@basho.com> On 24 Nov 2014, at 13:26, Ulf Wiger wrote: > > On 24 Nov 2014, at 13:56, Russell Brown wrote: > >> Hi, >> Is there a way to do this: >> >> my_fun(Bin, Len, <>) -> >> %% do stuff with Bin >> ok. >> >> Len is unbound is the reason I can?t, but it seems like the sort of thing that would be useful if I could. > > Yes, it?s slightly unfortunate, but you can write: > > my_fun(B, Len, Binary) -> > case Binary of > <> -> > %% do stuff with B > ok; > ? > end. This is what I went with. Thanks Ulf Russell > > BR, > Ulf W > > Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. > http://feuerlabs.com > > > From irfah.senuoy@REDACTED Mon Nov 24 14:24:53 2014 From: irfah.senuoy@REDACTED (=?UTF-8?Q?Youn=C3=A8s_HAFRI?=) Date: Mon, 24 Nov 2014 14:24:53 +0100 Subject: [erlang-questions] How to pass the length of binary to match into a function that matches the binary? In-Reply-To: References: Message-ID: my_fun(Len, Bin) -> fun (<>) -> ... (_) -> ... end(Bin). Le lundi 24 novembre 2014, Imants Cekusins a ?crit : > Would this work: > > my_fun(<>) -> ok; > > my_fun(<>) -> ok. > > ? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nathanfiedler@REDACTED Mon Nov 24 16:08:44 2014 From: nathanfiedler@REDACTED (Nathan Fiedler) Date: Mon, 24 Nov 2014 07:08:44 -0800 Subject: [erlang-questions] How to start application before all eunit cases In-Reply-To: References: Message-ID: I don't have an explanation for the application sometimes failing to find the config file, but I can say that when I tried to use EUnit for anything slightly more complicated than the simplest of tests, it turned into a frustrating experience. Instead, I changed my tests to Common Test, which may be a bit like killing a fly with a baseball bat, but it sure does the job. And you can ramp up slowly with CT; I'm just using the _data directory to hold data files and the init_per_suite/end_per_suite hooks to set up and tear down. The best part is I didn't have to change the tests themselves, other than to receive a Config argument. Short answer: use Common Test. n On Sun, Nov 23, 2014 at 11:06 PM, linbo liao wrote: > My Erlang project managed by rebar, it divides to different module. > > -pro > |-- rel > |-- src > |--- module1 > |--- module2 > |-- test > |--- module1_tests.erl > |--- module2_tests.erl > > and for each module*_tests.erl, Use Eunit Fixtures > to setup > environment. For example, > > module1_test_() -> > {setup, > fun setup/0, > fun clean_up/1, > fun (SetupData) -> > [ > test_function(SetupData) > ] > end}. > > setup() -> > pro:start(), > ok. > > clean_up(_) -> > pro:stop(). > > And Makefile is: > > test: > ERL_AFLAGS="-config rel/pro/etc/app.config" $(REBAR) compile eunit skip_deps=true > > Here I encounter a problem, since I have many test module in test/, each > test module will start and stop application for whole executing flow. > Sometimes start application will be failed, told not find app.config > configure file, not sure why. > > So I think is there a way to start application before all test module? > > Thanks, > > Linbo > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mononcqc@REDACTED Mon Nov 24 16:23:54 2014 From: mononcqc@REDACTED (Fred Hebert) Date: Mon, 24 Nov 2014 10:23:54 -0500 Subject: [erlang-questions] Try-catch-after considered harmful. In-Reply-To: References: Message-ID: <20141124152353.GG28152@ferdair.local> On 11/24, Joe Armstrong wrote: > My first response is "don't use use try-catch-otherwise at all" let > the process crash and let some other process fix up any damaging > consequences of the crash. > > The best was to write error handling code is to write no error > handling code at all. > You can get away with not handling errors when youcan afford not to handle them, and that restarting is alright, and losing all the state is alright. There are cases where you have a subsystem that you know will fail or timeoutand you want to handle that fine without tearing down everything you have. You get on this idea of resources soon enough: > At a first approximation, all the libraries are designed to be used > this way. If you open a resource like a file/socket/ets table/dets > table/database table then if your program crashes the resource (which > is represented by a process) will detect the crash and is supposed to > "do the right thing" and clean up behind you. > Yes, but what if what you want to do is display a thing such a 'parsing event #328492 failed' rather than going silent all of a sudden? It is the kind of code you cannot just keep retrying if the message is badly formatted, and where letting the parser crash may make more sense than just having it return `{error, badparse}` or whatever all the time. Or maybe that's just not in your control. So to work around that without a try ... catch, you need to fix it through architecture: - You spawn the parser in another process, either linked or monitored - You go into a 'receive ... end' expression that listens to that single process you spawned - If you receive the right message, it worked, if you received a monitor, it failed. Moreover, if you want to know if it was an error, an exit, or a throw, you have to go parse the `Reason' part of the message. You'll notice that what we designed there is pretty much the exact same thing as a try ... catch. Except we had to do it ad-hoc with processes, links, and monitors. Now let's try this mental exercise. I'm going to write a special syntax for that operation: try catch after end. This is going to be translated to: Parent = self(), {Pid, Ref} = spawn_monitor(fun() -> Parent ! {ok, Expr()} end), receive {ok, ExprRes} -> erlang:demonitor(Ref, [flush]), ExprRes; {'DOWN', Ref, process, Pid, Info} -> Patterns(Info) end, After() This will give me roughly similar semantics to try ... catch, but it will use processes instead. Now to keep it fully working in most cases, I'll need to also make sure that: - File handlers are handed off (and back in) - ETS table descriptors are handed off (and back in) - Sockets are handed off (and back in) - Process dictionaries are sync'd - Tracing goes for more than one process - Etc. The argument on whether you *need* anything but crash-fast could be had, but with the assumption that it is *needed* today, try ... catch is necessary if at the very least to keep people from having to write the complex blob above. I won't disagree with the idea that if you can, crash and restart. But there are times when you can't, or where it's more confusing to do so. > The try-catch syntax was deliberately chosen to be reminiscent of > Java, the idea being that if you understood this try-catch consequence > in Java you'd easily understand the Erlang code. > > The problem I see with this is that programmers with previous > experience in Java are tempted to blindly convert sequential try-catch > code in Java into sequential try-catch code in Erlang, but this is > almost always the wrong thing to do. > > Beginners should be forbidden to use try-catch - the Erlang "way" is > to spawn_link a regular process (ie a process that does not trap > exits) and just let that process die if anything goes wrong. There > should be no error trapping code in the spawned process. Call > exit(Why) in every place where the behaviour is unspecified. > try ... catch ... end is used for more than just exits. - throw(X), which lets you do a non-local return if you need it to, without the need to spawn new processes, copy and synchronize the state, and then send a message back; - error(X), programmer error in code - exit(X), process can't keep going. You do gain some richer semantics when using these 3 classes of exceptions. Only when they are not caught or handled do they lead to the bubbling of the exception into an exit(X), killing the process. > The process doing the spawn_link should trap exits and report errors > in the child processes. This is what all the supervisor and > gen_my_aunties_hat stuff does in the standard libraries. > the gen_stuff also runs try ... catches for you: https://github.com/erlang/otp/blob/maint/lib/stdlib/src/gen_server.erl#L570-L631 Specifically, they handle the non-local returning, and transform the exceptions into a value in order to let the behaviour take over and call 'terminate/N' on your behalf: https://github.com/erlang/otp/blob/maint/lib/stdlib/src/gen_server.erl#L705-L720 They don't purely run links and monitors and receives. They have to take over some of the program control flow to provide the mechanisms they currently provide without needlessly messaging and carrying resource across process boundaries all the time. > If you really-really-really understand what changing the flow of > execution in sequential code does then by all means try-catch-after - > but the best advice is not to use it at all. > > There should really be a compiler warning - > > ** try-catch-after used - this is probably not the right way to do things > > Cheers > You'd get a warning in every single OTP module. I get that try ... catch distracts from let-it-crash semantics, but it doesn't mean that there is one true way to handle errors. You can, quickly, count some ideas: - Explicit checks for error values ({error, _}, {ok, _}) with code handling both cases - Implicit assertions on error values with crashes on errors ({ok, _} = Expr) - Monadic flow (Maybe/Either) for exception handling. You run the equivalent of this one in Erlang when every single branch of your code explicitly checks for error values - Exceptions (throw/error + catching) - Monitors - Links What try ... catch lets you do, for example, is change the decision someone made to go with exceptions or assertions, and turn it into whatever else you want -- You can transform an exception or a throw into an exit signal, or transform a local exit signal into a value in order to do your monadic flow or explicit checks. What try ... catch gives you is not just Java mimicking, but a mechanism to take control and unify whatever type of error handling your dependencies use into the one your program needs. Of course you should be careful to not use exceptions as your only control flow mechanism for errors, but that's because others might be the better tool. It's just not always the case. Regards, Fred. From matthieu.tourne@REDACTED Mon Nov 24 22:23:30 2014 From: matthieu.tourne@REDACTED (Matthieu Tourne) Date: Mon, 24 Nov 2014 13:23:30 -0800 Subject: [erlang-questions] Weird behavior with the visual debugger. Message-ID: I'm using Erlang/OTP 17 on Mac OS, and I'm seeing a weird behavior with the debugger. I'm pretty new to the Erlang world, so not sure what is expected or not .. So here is what I'm doing : $> ii(module_to_debug). $> im(). This will launch the visual debugger with the module I want to debug. I've been trying to do 2 different things : 1. Set up a break point in my code I'm setting a break point, and setting up the option "Auto attach - On Break". My problem is that what I want to step through is behind a gen_server call, and the whole thing times out before I can do any meaningful debugging. Is there a way to prevent the timeouts *without* having to change the code that does the call ? 2. Inspecting the call stack on exception In this case I'm just setting the option "Auto attach - On Exit". That is where the weirdest behaviors lies, upon reaching the exception I get 2 blank debugger windows, one saying "State: uninterpreted" at the bottom and the other one "Loading module..." In my Erlang console I get the following message (which only ever happen if I'm trying to attach the debugger this way) : $> 13:08:38.300 [error] gen_server dbg_iserver terminated with reason: no match of right hand value [] in dbg_iserver:handle_call/3 line 287 (I'm using lager, hence the formatting). All subsequent calls to what leads to the exception behave weirdly : ** exception exit: {noproc,{gen_server,call, [undefined,{get_meta,<0.14874.0>},infinity]}} $> 13:20:22.337 [error] CRASH REPORT Process <0.14874.0> with 2 neighbours exited with reason: no such process or port in call to gen_server:call(undefined, {get_meta,<0.14874.0>}, infinity) in gen_server:init_it/6 line 330 I'm not sure what I'm doing wrong here, or if there is a real problem. If you can eventually point me to an article that explains how to make my two possible workflows above work, that would be great too. Thank you, Matthieu ps: Yes I already know "dbg:tracer(),dbg:p(all,c),dbg:tpl(ess_tcp, x)." Which doesn't entirely solve my problem. -------------- next part -------------- An HTML attachment was scrubbed... URL: From klacke@REDACTED Tue Nov 25 00:02:39 2014 From: klacke@REDACTED (Claes Wikstrom) Date: Tue, 25 Nov 2014 00:02:39 +0100 Subject: [erlang-questions] Yaws 1.99 Message-ID: <5473B90F.9080309@hyber.org> Folks, Yaws 1.99 is out. We're like wine, maturing. Getting better !! Tested release at http://yaws.hyber.org/download/yaws-1.99.tar.gz and http://yaws.hyber.org/download/Yaws-1.99-windows-installer.exe Code at https://github.com/klacke/yaws Thanks everyone for contributing. Community effort at it's best !! /klacke/steve/christopher From dgud@REDACTED Tue Nov 25 09:19:58 2014 From: dgud@REDACTED (Dan Gudmundsson) Date: Tue, 25 Nov 2014 09:19:58 +0100 Subject: [erlang-questions] Weird behavior with the visual debugger. In-Reply-To: References: Message-ID: First one is problematic, I would make a debug macro for timeouts and set it to infinity when debug compiled. The second one looks like a bug in the interpreter. /Dan On Mon, Nov 24, 2014 at 10:23 PM, Matthieu Tourne wrote: > I'm using Erlang/OTP 17 on Mac OS, and I'm seeing a weird behavior with > the debugger. > > I'm pretty new to the Erlang world, so not sure what is expected or not .. > > So here is what I'm doing : > > $> ii(module_to_debug). > $> im(). > > This will launch the visual debugger with the module I want to debug. > I've been trying to do 2 different things : > > 1. Set up a break point in my code > I'm setting a break point, and setting up the option "Auto attach - On > Break". > > My problem is that what I want to step through is behind a gen_server > call, and the whole thing times out before I can do any meaningful > debugging. > Is there a way to prevent the timeouts *without* having to change the code > that does the call ? > > 2. Inspecting the call stack on exception > In this case I'm just setting the option "Auto attach - On Exit". > > That is where the weirdest behaviors lies, upon reaching the exception I > get 2 blank debugger windows, one saying "State: uninterpreted" at the > bottom and the other one "Loading module..." > In my Erlang console I get the following message (which only ever happen > if I'm trying to attach the debugger this way) : > > $> 13:08:38.300 [error] gen_server dbg_iserver terminated with reason: no > match of right hand value [] in dbg_iserver:handle_call/3 line 287 > (I'm using lager, hence the formatting). > > All subsequent calls to what leads to the exception behave weirdly : > > ** exception exit: {noproc,{gen_server,call, > > [undefined,{get_meta,<0.14874.0>},infinity]}} > $> 13:20:22.337 [error] CRASH REPORT Process <0.14874.0> with 2 neighbours > exited with reason: no such process or port in call to > gen_server:call(undefined, {get_meta,<0.14874.0>}, infinity) in > gen_server:init_it/6 line 330 > > > I'm not sure what I'm doing wrong here, or if there is a real problem. > If you can eventually point me to an article that explains how to make my > two possible workflows above work, that would be great too. > > > Thank you, > Matthieu > > > ps: Yes I already know "dbg:tracer(),dbg:p(all,c),dbg:tpl(ess_tcp, x)." > Which doesn't entirely solve my problem. > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From imantc@REDACTED Tue Nov 25 10:52:46 2014 From: imantc@REDACTED (Imants Cekusins) Date: Tue, 25 Nov 2014 10:52:46 +0100 Subject: [erlang-questions] Weird behavior with the visual debugger. In-Reply-To: References: Message-ID: Just in case: dbg functions allow filtering by module, function. It is possible to trace errors only. Call args & return values are available too. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tuncer.ayaz@REDACTED Tue Nov 25 13:45:37 2014 From: tuncer.ayaz@REDACTED (Tuncer Ayaz) Date: Tue, 25 Nov 2014 13:45:37 +0100 Subject: [erlang-questions] How to start application before all eunit cases In-Reply-To: References: Message-ID: On Mon, Nov 24, 2014 at 8:06 AM, linbo liao wrote: > My Erlang project managed by rebar, it divides to different module. > > -pro > |-- rel > |-- src > |--- module1 > |--- module2 > |-- test > |--- module1_tests.erl > |--- module2_tests.erl > > and for each module*_tests.erl, Use Eunit Fixtures to setup > environment. For example, > > module1_test_() -> > {setup, > fun setup/0, > fun clean_up/1, > fun (SetupData) -> > [ > test_function(SetupData) > ] > end}. > > setup() -> > pro:start(), > ok. > > clean_up(_) -> > pro:stop(). > > And Makefile is: > > test: > ERL_AFLAGS="-config rel/pro/etc/app.config" $(REBAR) compile eunit > skip_deps=true > > Here I encounter a problem, since I have many test module in test/, > each test module will start and stop application for whole executing > flow. Sometimes start application will be failed, told not find > app.config configure file, not sure why. > > So I think is there a way to start application before all test > module? Without a complete project to check out, like Nathan, I also cannot answer why the config file is sometimes not found. However, there is a rebar hook solution you can use: $ cat pro/rebar.config: {plugins, [pro_eunit_hook]}. ... $ cat pro/pro_eunit_hook.erl: -module(pro_eunit_hook). -export([pre_eunit/2]). pre_eunit(_, _) -> %% if not already started, start the application ok. From ivan@REDACTED Tue Nov 25 16:13:20 2014 From: ivan@REDACTED (Ivan Uemlianin) Date: Tue, 25 Nov 2014 15:13:20 +0000 Subject: [erlang-questions] Why does Dialyzer crash on this map? Message-ID: <54749C90.6000602@llaisdy.com> Dear All Running Dialyzer on the module below crashes Dialyzer with the error following: ** module -module(ltu). -export([get_map/0]). -spec get_map() -> map(). get_map() -> #{labels => [one, two], number => 27, [1,2,3] => wer, %% ok {4,5,6} => sdf, %% ok kvok => #{ a => qwe, 2 => asd, [1,2,3] => wer, %% bad {4,5,6} => sdf, %% bad "abc" => zxc } }. ** crash $ dialyzer ltu.erl Checking whether the PLT /home/ivan/.dialyzer_plt is up-to-date... yes Proceeding with analysis... =ERROR REPORT==== 25-Nov-2014::14:52:50 === Error in process <0.45.0> with exit value: {{case_clause,map},[{dialyzer_dataflow,find_terminals,1,[{file,"dialyzer_dataflow.erl"},{line,3451}]},{dialyzer_dataflow,find_terminals_list,3,[{file,"dialyzer_dataflow.erl"},{line,3504}]},{dialyzer_dataflow,classify_returns... dialyzer: Analysis failed with error: {{case_clause,map}, [{dialyzer_dataflow,find_terminals,1, [{file,"dialyzer_dataflow.erl"},{line,3451}]}, {dialyzer_dataflow,find_terminals_list,3, [{file,"dialyzer_dataflow.erl"},{line,3504}]}, {dialyzer_dataflow,classify_returns,1, [{file,"dialyzer_dataflow.erl"},{line,3443}]}, {dialyzer_dataflow,'-state__get_warnings/2-fun-0-',7, [{file,"dialyzer_dataflow.erl"},{line,2908}]}, {lists,foldl,3,[{file,"lists.erl"},{line,1261}]}, {dialyzer_dataflow,state__get_warnings,2, [{file,"dialyzer_dataflow.erl"},{line,2934}]}, {dialyzer_dataflow,get_warnings,5, [{file,"dialyzer_dataflow.erl"},{line,142}]}, {dialyzer_succ_typings,collect_warnings,2, [{file,"dialyzer_succ_typings.erl"},{line,182}]}]} Last messages in the log cache: Reading files and computing callgraph... done in 0.07 secs Removing edges... done in 0.00 secs If I comment out the two lines marked "%% bad", Dialyzer can check the module. nb the same key-value pairs that are bad in the nested map kvok, are ok in the top-level map. Is this supposed to happen? If so, why? With thanks and best wishes Ivan -- ============================================================ Ivan A. Uemlianin PhD Llaisdy Speech Technology Research and Development ivan@REDACTED @llaisdy llaisdy.wordpress.com github.com/llaisdy www.linkedin.com/in/ivanuemlianin festina lente ============================================================ From ivan@REDACTED Tue Nov 25 18:20:12 2014 From: ivan@REDACTED (Ivan Uemlianin) Date: Tue, 25 Nov 2014 17:20:12 +0000 Subject: [erlang-questions] Why does Dialyzer crash on this map? In-Reply-To: <54749C90.6000602@llaisdy.com> References: <54749C90.6000602@llaisdy.com> Message-ID: <5474BA4C.4030706@llaisdy.com> Dear All Quick update: Just tried this at home on my Mac, and Dialyzer Dialyzes ltu.erl with no problem. The work machine that displayed the error below is a FreeBSD machine. Both machines run the same version of Dialyzer (2.7.2); both obvs have a 17.x erlang but the BSD might be a bit older. Have any minor bugs in Dialyzer been ironed out recently? Thanks Ivan On 25/11/2014 15:13, Ivan Uemlianin wrote: > Dear All > > Running Dialyzer on the module below crashes Dialyzer with the error > following: > > ** module > > -module(ltu). > -export([get_map/0]). > > -spec get_map() -> map(). > get_map() -> > #{labels => [one, two], > number => 27, > [1,2,3] => wer, %% ok > {4,5,6} => sdf, %% ok > kvok => #{ > a => qwe, > 2 => asd, > [1,2,3] => wer, %% bad > {4,5,6} => sdf, %% bad > "abc" => zxc > } > }. > > ** crash > > $ dialyzer ltu.erl > Checking whether the PLT /home/ivan/.dialyzer_plt is > up-to-date... yes > Proceeding with analysis... > =ERROR REPORT==== 25-Nov-2014::14:52:50 === > Error in process <0.45.0> with exit value: > {{case_clause,map},[{dialyzer_dataflow,find_terminals,1,[{file,"dialyzer_dataflow.erl"},{line,3451}]},{dialyzer_dataflow,find_terminals_list,3,[{file,"dialyzer_dataflow.erl"},{line,3504}]},{dialyzer_dataflow,classify_returns... > > > > dialyzer: Analysis failed with error: > {{case_clause,map}, > [{dialyzer_dataflow,find_terminals,1, > [{file,"dialyzer_dataflow.erl"},{line,3451}]}, > {dialyzer_dataflow,find_terminals_list,3, > [{file,"dialyzer_dataflow.erl"},{line,3504}]}, > {dialyzer_dataflow,classify_returns,1, > [{file,"dialyzer_dataflow.erl"},{line,3443}]}, > {dialyzer_dataflow,'-state__get_warnings/2-fun-0-',7, > [{file,"dialyzer_dataflow.erl"},{line,2908}]}, > {lists,foldl,3,[{file,"lists.erl"},{line,1261}]}, > {dialyzer_dataflow,state__get_warnings,2, > [{file,"dialyzer_dataflow.erl"},{line,2934}]}, > {dialyzer_dataflow,get_warnings,5, > [{file,"dialyzer_dataflow.erl"},{line,142}]}, > {dialyzer_succ_typings,collect_warnings,2, > [{file,"dialyzer_succ_typings.erl"},{line,182}]}]} > Last messages in the log cache: > Reading files and computing callgraph... done in 0.07 secs > Removing edges... done in 0.00 secs > > If I comment out the two lines marked "%% bad", Dialyzer can check the > module. nb the same key-value pairs that are bad in the nested map > kvok, are ok in the top-level map. > > Is this supposed to happen? If so, why? > > With thanks and best wishes > > Ivan > > -- ============================================================ Ivan A. Uemlianin PhD Llaisdy Speech Technology Research and Development ivan@REDACTED @llaisdy llaisdy.wordpress.com github.com/llaisdy www.linkedin.com/in/ivanuemlianin festina lente ============================================================ From lostcolony@REDACTED Tue Nov 25 18:18:05 2014 From: lostcolony@REDACTED (Christopher Phillips) Date: Tue, 25 Nov 2014 12:18:05 -0500 Subject: [erlang-questions] Best practices for QAing distribution? Message-ID: Does anyone have a library or a set of best practices for testing netsplits? I.e., given a couple nodes, create a netsplit between them and see what one's distributed application ~really~ does in such a situation? I've looked around a bit online but my google-fu has failed me, as I wasn't able to find anything that seemed to do what I'm looking for, either in Erlang specifically, or just in a general application sense (there's Jepsen, but that seems geared for testing generic distribution libraries in a basic sense, rather than domain specific code, and I'm not very good with Clojure) which seems...odd. I've got some ideas, but figured I'd ask around to see if anyone has done anything similar first. -------------- next part -------------- An HTML attachment was scrubbed... URL: From felixgallo@REDACTED Tue Nov 25 19:54:13 2014 From: felixgallo@REDACTED (Felix Gallo) Date: Tue, 25 Nov 2014 10:54:13 -0800 Subject: [erlang-questions] Best practices for QAing distribution? In-Reply-To: References: Message-ID: http://www.linuxfoundation.org/collaborate/workgroups/networking/netem On Tue, Nov 25, 2014 at 9:18 AM, Christopher Phillips wrote: > > Does anyone have a library or a set of best practices for testing > netsplits? I.e., given a couple nodes, create a netsplit between them and > see what one's distributed application ~really~ does in such a situation? > I've looked around a bit online but my google-fu has failed me, as I wasn't > able to find anything that seemed to do what I'm looking for, either in > Erlang specifically, or just in a general application sense (there's > Jepsen, but that seems geared for testing generic distribution libraries in > a basic sense, rather than domain specific code, and I'm not very good with > Clojure) which seems...odd. I've got some ideas, but figured I'd ask > around to see if anyone has done anything similar first. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sergej.jurecko@REDACTED Tue Nov 25 20:13:39 2014 From: sergej.jurecko@REDACTED (Sergej Jurecko) Date: Tue, 25 Nov 2014 20:13:39 +0100 Subject: [erlang-questions] Best practices for QAing distribution? In-Reply-To: References: Message-ID: <47AC7BCC-E644-4AC3-9BEF-9C615558DB76@gmail.com> What I do in ActorDB is use eunit and start up slave nodes when running tests. On test setup create nodes, then use rpc:call to execute code. You can take down nodes at any time and start them back up to test nodes dropping. This way of testing will tell you a lot but is not the whole story. Distributed computation can often be dependent on timings and running slave nodes can mislead you. When you get around to running code on actual networked servers, timings can be very different. You also can not test behaviour of bad networking conditions. What will happen if your network is dropping connections randomly, experiencing packet loss, etc. It would be awesome if erlang distribution offered some test option of connecting nodes ?poorly?. In that mode it would do random disconnects, delays in data delivery, etc. Anything that could happen in an actual network. I guess you could do some sort of automated testing using docker. This is a half baked idea I have that I should get around to trying one of these days. Sergej On 25 Nov 2014, at 18:18, Christopher Phillips wrote: > > Does anyone have a library or a set of best practices for testing netsplits? I.e., given a couple nodes, create a netsplit between them and see what one's distributed application ~really~ does in such a situation? I've looked around a bit online but my google-fu has failed me, as I wasn't able to find anything that seemed to do what I'm looking for, either in Erlang specifically, or just in a general application sense (there's Jepsen, but that seems geared for testing generic distribution libraries in a basic sense, rather than domain specific code, and I'm not very good with Clojure) which seems...odd. I've got some ideas, but figured I'd ask around to see if anyone has done anything similar first. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From heinz@REDACTED Tue Nov 25 20:47:18 2014 From: heinz@REDACTED (Heinz Nikolaus Gies) Date: Tue, 25 Nov 2014 20:47:18 +0100 Subject: [erlang-questions] Best practices for QAing distribution? In-Reply-To: References: Message-ID: riak_test could be a good starting point, I had started to make it less riak and more test but ended with something along the lines of riak_core_test. However I think with a bit of effort it could be generalized to erlang_net_test. It handles multiple nodes and can spin them up turn them down and run tests against a interface on them. When I am not mistaken it can also simulate splits by changing the erlang cookie on the nodes and breaking distribution. --- Cheers, Heinz Nikolaus Gies heinz@REDACTED > On Nov 25, 2014, at 18:18, Christopher Phillips wrote: > > > Does anyone have a library or a set of best practices for testing netsplits? I.e., given a couple nodes, create a netsplit between them and see what one's distributed application ~really~ does in such a situation? I've looked around a bit online but my google-fu has failed me, as I wasn't able to find anything that seemed to do what I'm looking for, either in Erlang specifically, or just in a general application sense (there's Jepsen, but that seems geared for testing generic distribution libraries in a basic sense, rather than domain specific code, and I'm not very good with Clojure) which seems...odd. I've got some ideas, but figured I'd ask around to see if anyone has done anything similar first. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From lostcolony@REDACTED Tue Nov 25 20:22:51 2014 From: lostcolony@REDACTED (Christopher Phillips) Date: Tue, 25 Nov 2014 14:22:51 -0500 Subject: [erlang-questions] Best practices for QAing distribution? In-Reply-To: References: Message-ID: Thanks, I'm actually familiar with netem, and assumed it could play a piece in any solution people had. I more meant, what are your actual testing mechanisms in a distributed environment? Do you just start up a couple of nodes by hand on separate VMs and then manually fire off commands to netem, run a test suite, then set the interface back to normal? As that seems inefficient. I want something that I can write some config, some tests, and reproducibly have my code exercise the tests, with netsplits, packet delays/loss, etc, and confirm that my expectations still work. It doesn't seem like it would be that hard to do, if I assume I'm running on a *nix, as between netem, iptables, and ifconfig, I can create new interfaces, start my apps listening on them, and proceed to mess with traffic in all manner of interesting ways, creating partially disconnected clusters, partitioned clusters, partitioned nodes, delayed traffic, spotty traffic, etc, and see what happens. Maybe even a long running, random process that introduces these in different places in my cluster, a sort of chaos monkey for the network. Etc. I figure other people have probably wanted to test things in a similar manner, so I wanted to see what methods others used. On Tue, Nov 25, 2014 at 1:54 PM, Felix Gallo wrote: > http://www.linuxfoundation.org/collaborate/workgroups/networking/netem > > On Tue, Nov 25, 2014 at 9:18 AM, Christopher Phillips < > lostcolony@REDACTED> wrote: > >> >> Does anyone have a library or a set of best practices for testing >> netsplits? I.e., given a couple nodes, create a netsplit between them and >> see what one's distributed application ~really~ does in such a situation? >> I've looked around a bit online but my google-fu has failed me, as I wasn't >> able to find anything that seemed to do what I'm looking for, either in >> Erlang specifically, or just in a general application sense (there's >> Jepsen, but that seems geared for testing generic distribution libraries in >> a basic sense, rather than domain specific code, and I'm not very good with >> Clojure) which seems...odd. I've got some ideas, but figured I'd ask >> around to see if anyone has done anything similar first. >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthieu.tourne@REDACTED Tue Nov 25 20:47:08 2014 From: matthieu.tourne@REDACTED (Matthieu Tourne) Date: Tue, 25 Nov 2014 11:47:08 -0800 Subject: [erlang-questions] Weird behavior with the visual debugger. In-Reply-To: References: Message-ID: Dan, could you go a little bit into details how you would get around to do this ? Again, very new to the Erlang world. Imants, Thank you I already have this as my user_default.erl [1], but not exactly what I've been looking for. [1] http://www.snookles.com/erlang/user_default.erl On Tue, Nov 25, 2014 at 1:52 AM, Imants Cekusins wrote: > Just in case: dbg functions allow filtering by module, function. > > It is possible to trace errors only. Call args & return values are > available too. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dch@REDACTED Tue Nov 25 21:29:25 2014 From: dch@REDACTED (Dave Cottlehuber) Date: Tue, 25 Nov 2014 21:29:25 +0100 Subject: [erlang-questions] Why does Dialyzer crash on this map? In-Reply-To: <5474BA4C.4030706@llaisdy.com> References: <54749C90.6000602@llaisdy.com> <5474BA4C.4030706@llaisdy.com> Message-ID: Ivan, EWORKS=46ORME here. =46reeBSD 10.1,=C2=A0erlang-17.3=5F2,3 from pkg. What=E2=80=99s yours=3F dch =C2=A0 /tmp =E2=9D=AF=E2=9D=AF=E2=9D=AF =C2=A0 dialyzer --build=5Fplt= --apps erts kernel stdlib =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0=E2=8F=8E =C2=A0 Creating PLT /home/dch/.dialyzer=5Fplt ... Unknown functions: =C2=A0 compile:file/2 =C2=A0 compile:forms/2 =C2=A0 compile:noenv=5Fforms/2 =C2=A0 compile:output=5Fgenerated/1 =C2=A0 crypto:block=5Fdecrypt/4 =C2=A0 crypto:start/0 Unknown types: =C2=A0 compile:option/0 =C2=A0done in 1m8.53s done (passed successfully) dch =C2=A0 /tmp =E2=9D=AF=E2=9D=AF=E2=9D=AFdialyzer ltu.erl =C2=A0 Checking whether the PLT /home/dch/.dialyzer=5Fplt is up-to-date..= . yes =C2=A0 Proceeding with analysis... done in 0m0.41s done (passed successfully) dch =C2=A0 /tmp =E2=9D=AF=E2=9D=AF=E2=9D=AFerl Erlang/OTP 17 =5Berts-6.2=5D =5Bsource=5D =5B64-bit=5D =5Bsmp:8:8=5D =5Ba= sync-threads:64=5D =5Bkernel-poll:true=5D Eshell V6.2 =C2=A0(abort with =5EG) 1> BREAK: (a)bort (c)ontinue (p)roc info (i)nfo (l)oaded =C2=A0 =C2=A0 =C2=A0 =C2=A0(v)ersion (k)ill (D)b-tables (d)istribution a dch =C2=A0 /tmp =E2=9D=AF=E2=9D=AF=E2=9D=AF A+, Dave =E2=80=94 sent from my Couch -----Original Message----- =46rom:=C2=A0Ivan Uemlianin Reply:=C2=A0Ivan Uemlianin > Date:=C2=A025. November 2014 at 18:21:02 To:=C2=A0erlang-questions=40erlang.org > Subject:=C2=A0 Re: =5Berlang-questions=5D Why does Dialyzer crash on this= map=3F > Dear All > =20 > Quick update: > =20 > Just tried this at home on my Mac, and Dialyzer Dialyzes ltu.erl with n= o > problem. > =20 > The work machine that displayed the error below is a =46reeBSD machine.= > Both machines run the same version of Dialyzer (2.7.2); both obvs have = a > 17.x erlang but the BSD might be a bit older. > =20 > Have any minor bugs in Dialyzer been ironed out recently=3F > =20 > Thanks > =20 > Ivan > =20 > =20 > On 25/11/2014 15:13, Ivan Uemlianin wrote: > > Dear All > > > > Running Dialyzer on the module below crashes Dialyzer with the error > > following: > > > > ** module > > > > -module(ltu). > > -export(=5Bget=5Fmap/0=5D). > > > > -spec get=5Fmap() -> map(). > > get=5Fmap() -> > > =23=7Blabels =3D> =5Bone, two=5D, > > number =3D> 27, > > =5B1,2,3=5D =3D> wer, %% ok > > =7B4,5,6=7D =3D> sdf, %% ok > > kvok =3D> =23=7B > > a =3D> qwe, > > 2 =3D> asd, > > =5B1,2,3=5D =3D> wer, %% bad > > =7B4,5,6=7D =3D> sdf, %% bad > > =22abc=22 =3D> zxc > > =7D > > =7D. > > > > ** crash > > > > =24 dialyzer ltu.erl > > Checking whether the PLT /home/ivan/.dialyzer=5Fplt is > > up-to-date... yes > > Proceeding with analysis... > > =3DERROR REPORT=3D=3D=3D=3D 25-Nov-2014::14:52:50 =3D=3D=3D > > Error in process <0.45.0> with exit value: > > =7B=7Bcase=5Fclause,map=7D,=5B=7Bdialyzer=5Fdataflow,find=5Fterminals= ,1,=5B=7Bfile,=22dialyzer=5Fdataflow.erl=22=7D,=7Bline,3451=7D=5D=7D,=7Bd= ialyzer=5Fdataflow,find=5Fterminals=5Flist,3,=5B=7Bfile,=22dialyzer=5Fdat= aflow.erl=22=7D,=7Bline,3504=7D=5D=7D,=7Bdialyzer=5Fdataflow,classify=5Fr= eturns... =20 > > > > > > > > dialyzer: Analysis failed with error: > > =7B=7Bcase=5Fclause,map=7D, > > =5B=7Bdialyzer=5Fdataflow,find=5Fterminals,1, > > =5B=7Bfile,=22dialyzer=5Fdataflow.erl=22=7D,=7Bline,3451=7D=5D=7D, > > =7Bdialyzer=5Fdataflow,find=5Fterminals=5Flist,3, > > =5B=7Bfile,=22dialyzer=5Fdataflow.erl=22=7D,=7Bline,3504=7D=5D=7D, > > =7Bdialyzer=5Fdataflow,classify=5Freturns,1, > > =5B=7Bfile,=22dialyzer=5Fdataflow.erl=22=7D,=7Bline,3443=7D=5D=7D, > > =7Bdialyzer=5Fdataflow,'-state=5F=5Fget=5Fwarnings/2-fun-0-',7, > > =5B=7Bfile,=22dialyzer=5Fdataflow.erl=22=7D,=7Bline,2908=7D=5D=7D, > > =7Blists,foldl,3,=5B=7Bfile,=22lists.erl=22=7D,=7Bline,1261=7D=5D=7D,= > > =7Bdialyzer=5Fdataflow,state=5F=5Fget=5Fwarnings,2, > > =5B=7Bfile,=22dialyzer=5Fdataflow.erl=22=7D,=7Bline,2934=7D=5D=7D, > > =7Bdialyzer=5Fdataflow,get=5Fwarnings,5, > > =5B=7Bfile,=22dialyzer=5Fdataflow.erl=22=7D,=7Bline,142=7D=5D=7D, > > =7Bdialyzer=5Fsucc=5Ftypings,collect=5Fwarnings,2, > > =5B=7Bfile,=22dialyzer=5Fsucc=5Ftypings.erl=22=7D,=7Bline,182=7D=5D=7D= =5D=7D > > Last messages in the log cache: > > Reading files and computing callgraph... done in 0.07 secs > > Removing edges... done in 0.00 secs > > > > If I comment out the two lines marked =22%% bad=22, Dialyzer can chec= k the > > module. nb the same key-value pairs that are bad in the nested map > > kvok, are ok in the top-level map. > > > > Is this supposed to happen=3F If so, why=3F > > > > With thanks and best wishes > > > > Ivan > > > > > =20 > -- > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > Ivan A. Uemlianin PhD > Llaisdy > Speech Technology Research and Development > =20 > ivan=40llaisdy.com > =40llaisdy > llaisdy.wordpress.com > github.com/llaisdy > www.linkedin.com/in/ivanuemlianin > =20 > festina lente > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > =20 > =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F > erlang-questions mailing list > erlang-questions=40erlang.org > http://erlang.org/mailman/listinfo/erlang-questions > =20 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 101 bytes Desc: Message signed with OpenPGP using AMPGpg URL: From dch@REDACTED Tue Nov 25 21:37:55 2014 From: dch@REDACTED (Dave Cottlehuber) Date: Tue, 25 Nov 2014 21:37:55 +0100 Subject: [erlang-questions] Why does Dialyzer crash on this map? In-Reply-To: References: <54749C90.6000602@llaisdy.com> <5474BA4C.4030706@llaisdy.com> Message-ID: Sorry about that last email ? EWORKSFORME. FreeBSD 10.1 amd64, erlang-17.3_2,3 from pkg. What FreeBSD & erlang are you using? dch ? /tmp ??? ? dialyzer --build_plt --apps erts kernel stdlib ? Creating PLT /home/dch/.dialyzer_plt ... Unknown functions: ? compile:file/2 ? compile:forms/2 ? compile:noenv_forms/2 ? compile:output_generated/1 ? crypto:block_decrypt/4 ? crypto:start/0 Unknown types: ? compile:option/0 ?done in 1m8.53s done (passed successfully) dch ? /tmp ???dialyzer ltu.erl ? Checking whether the PLT /home/dch/.dialyzer_plt is up-to-date... yes ? Proceeding with analysis... done in 0m0.41s done (passed successfully) dch ? /tmp ???erl Erlang/OTP 17 [erts-6.2] [source] [64-bit] [smp:8:8] [async-threads:64] [kernel-poll:true] Eshell V6.2 ?(abort with ^G) 1> A+, Dave ? sent from my Couch From kostis@REDACTED Tue Nov 25 22:37:46 2014 From: kostis@REDACTED (Kostis Sagonas) Date: Tue, 25 Nov 2014 22:37:46 +0100 Subject: [erlang-questions] Why does Dialyzer crash on this map? In-Reply-To: <5474BA4C.4030706@llaisdy.com> References: <54749C90.6000602@llaisdy.com> <5474BA4C.4030706@llaisdy.com> Message-ID: <5474F6AA.5010900@cs.ntua.gr> On 11/25/2014 06:20 PM, Ivan Uemlianin wrote: > Just tried this at home on my Mac, and Dialyzer Dialyzes ltu.erl with no > problem. > > The work machine that displayed the error below is a FreeBSD machine. > Both machines run the same version of Dialyzer (2.7.2); both obvs have a > 17.x erlang but the BSD might be a bit older. Both your mail above and the original one below are sufficiently vague to do something about them. Please understand that it's very hard for others to investigate what's happening without proper information. For example, AFAIK there is no Erlang/OTP release named 17.x. Instead on http://www.erlang.org/download.html you can find 17.0, 17.1, and 17.3. On github (https://github.com/erlang/otp/releases) one can find 17.0-rc1, 17.0-rc2, 17.0, 17.0.1, 17.0.2, 17.1, 17.1.1, 17.1.2, 17.2, 17.2.1, 17.2.2, 17.3, 17.3.1, 17.3.2, and then there are the various maint code bases. How do you expect somebody to guess or investigate on which of all these the problem you are seeing might exist? Especially without mentioning at first that this is on a FreeBSD machine, which is not the most common of platforms and therefore cannot possibly reproduce what you are experiencing? > On 25/11/2014 15:13, Ivan Uemlianin wrote: >> Dear All >> >> Running Dialyzer on the module below crashes Dialyzer with the error >> following: .... Anyway, it may be worthwhile for you to check whether this is reproducible on your FreeBSD machine using some "official" OTP release (this means 17.0, 17.1 or 17.3) and if so whether the *same* release also shows this problem on your Mac or not. Under normal circumstances, Dialyzer should not have any dependency on the OS. If it does then this most likely exposes a bug in OTP itself on that platform and should be investigated further. Hence this mail. Kostis From comptekki@REDACTED Tue Nov 25 23:23:29 2014 From: comptekki@REDACTED (Wes James) Date: Tue, 25 Nov 2014 15:23:29 -0700 Subject: [erlang-questions] ssl error ugrading clients to erl 17.3 Message-ID: I upgraded my esysman installation to use erlang 17.3 on xubuntu. That works fine with my clients with erlang 16b, but when I update the clients to 17.3, I'm getting an ssl error. https://gist.github.com/comptekki/512cd7d2971062297a19 I googled and found a similar error here: https://gist.github.com/asabil/1dd2b44bb527de97f325 Is there a solution to fix this? Thanks, Wes -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Tue Nov 25 23:24:32 2014 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Wed, 26 Nov 2014 00:24:32 +0200 Subject: [erlang-questions] ssl error ugrading clients to erl 17.3 In-Reply-To: References: Message-ID: <547501A0.8040602@ninenines.eu> It was fixed in 17.3.2. On 11/26/2014 12:23 AM, Wes James wrote: > I upgraded my esysman installation to use erlang 17.3 on xubuntu. That > works fine with my clients with erlang 16b, but when I update the > clients to 17.3, I'm getting an ssl error. > > https://gist.github.com/comptekki/512cd7d2971062297a19 > > I googled and found a similar error here: > > https://gist.github.com/asabil/1dd2b44bb527de97f325 > > Is there a solution to fix this? > > Thanks, > > Wes > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin http://ninenines.eu From comptekki@REDACTED Tue Nov 25 23:26:10 2014 From: comptekki@REDACTED (Wes James) Date: Tue, 25 Nov 2014 15:26:10 -0700 Subject: [erlang-questions] ssl error ugrading clients to erl 17.3 In-Reply-To: <547501A0.8040602@ninenines.eu> References: <547501A0.8040602@ninenines.eu> Message-ID: OK Thanks. Do I git clone erlang and then compile the 17.3.2 branch? wes On Tue, Nov 25, 2014 at 3:24 PM, Lo?c Hoguin wrote: > It was fixed in 17.3.2. > > > On 11/26/2014 12:23 AM, Wes James wrote: > >> I upgraded my esysman installation to use erlang 17.3 on xubuntu. That >> works fine with my clients with erlang 16b, but when I update the >> clients to 17.3, I'm getting an ssl error. >> >> https://gist.github.com/comptekki/512cd7d2971062297a19 >> >> I googled and found a similar error here: >> >> https://gist.github.com/asabil/1dd2b44bb527de97f325 >> >> Is there a solution to fix this? >> >> Thanks, >> >> Wes >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -- > Lo?c Hoguin > http://ninenines.eu > -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Tue Nov 25 23:27:32 2014 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Wed, 26 Nov 2014 00:27:32 +0200 Subject: [erlang-questions] ssl error ugrading clients to erl 17.3 In-Reply-To: References: <547501A0.8040602@ninenines.eu> Message-ID: <54750254.1040100@ninenines.eu> Try with kerl: kerl build git https://github.com/erlang/otp OTP-17.3.4 17.3.4 On 11/26/2014 12:26 AM, Wes James wrote: > OK Thanks. Do I git clone erlang and then compile the 17.3.2 branch? > > wes > > On Tue, Nov 25, 2014 at 3:24 PM, Lo?c Hoguin > wrote: > > It was fixed in 17.3.2. > > > On 11/26/2014 12:23 AM, Wes James wrote: > > I upgraded my esysman installation to use erlang 17.3 on > xubuntu. That > works fine with my clients with erlang 16b, but when I update the > clients to 17.3, I'm getting an ssl error. > > https://gist.github.com/__comptekki/512cd7d2971062297a19 > > > I googled and found a similar error here: > > https://gist.github.com/__asabil/1dd2b44bb527de97f325 > > > Is there a solution to fix this? > > Thanks, > > Wes > > > _________________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/__listinfo/erlang-questions > > > > -- > Lo?c Hoguin > http://ninenines.eu > > -- Lo?c Hoguin http://ninenines.eu From comptekki@REDACTED Tue Nov 25 23:28:21 2014 From: comptekki@REDACTED (Wes James) Date: Tue, 25 Nov 2014 15:28:21 -0700 Subject: [erlang-questions] ssl error ugrading clients to erl 17.3 In-Reply-To: <54750254.1040100@ninenines.eu> References: <547501A0.8040602@ninenines.eu> <54750254.1040100@ninenines.eu> Message-ID: OK. I'll try that. Thanks, wes On Tue, Nov 25, 2014 at 3:27 PM, Lo?c Hoguin wrote: > Try with kerl: > > kerl build git https://github.com/erlang/otp OTP-17.3.4 17.3.4 > > On 11/26/2014 12:26 AM, Wes James wrote: > >> OK Thanks. Do I git clone erlang and then compile the 17.3.2 branch? >> >> wes >> >> On Tue, Nov 25, 2014 at 3:24 PM, Lo?c Hoguin > > wrote: >> >> It was fixed in 17.3.2. >> >> >> On 11/26/2014 12:23 AM, Wes James wrote: >> >> I upgraded my esysman installation to use erlang 17.3 on >> xubuntu. That >> works fine with my clients with erlang 16b, but when I update the >> clients to 17.3, I'm getting an ssl error. >> >> https://gist.github.com/__comptekki/512cd7d2971062297a19 >> >> >> I googled and found a similar error here: >> >> https://gist.github.com/__asabil/1dd2b44bb527de97f325 >> >> >> Is there a solution to fix this? >> >> Thanks, >> >> Wes >> >> >> _________________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/__listinfo/erlang-questions >> >> >> >> -- >> Lo?c Hoguin >> http://ninenines.eu >> >> >> > -- > Lo?c Hoguin > http://ninenines.eu > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ivan@REDACTED Tue Nov 25 23:31:08 2014 From: ivan@REDACTED (Ivan Uemlianin) Date: Tue, 25 Nov 2014 22:31:08 +0000 Subject: [erlang-questions] Why does Dialyzer crash on this map? In-Reply-To: References: <54749C90.6000602@llaisdy.com> <5474BA4C.4030706@llaisdy.com> Message-ID: Thanks, that's a good result. I'll get more details and report back. Ivan -- festina lente > On 25 Nov 2014, at 20:37, Dave Cottlehuber wrote: > > Sorry about that last email ? > > EWORKSFORME. > > FreeBSD 10.1 amd64, erlang-17.3_2,3 from pkg. > > What FreeBSD & erlang are you using? > > dch /tmp ??? dialyzer --build_plt --apps erts kernel stdlib > Creating PLT /home/dch/.dialyzer_plt ... > Unknown functions: > compile:file/2 > compile:forms/2 > compile:noenv_forms/2 > compile:output_generated/1 > crypto:block_decrypt/4 > crypto:start/0 > Unknown types: > compile:option/0 > done in 1m8.53s > done (passed successfully) > > dch /tmp ???dialyzer ltu.erl > Checking whether the PLT /home/dch/.dialyzer_plt is up-to-date... yes > Proceeding with analysis... done in 0m0.41s > done (passed successfully) > > dch /tmp ???erl > Erlang/OTP 17 [erts-6.2] [source] [64-bit] [smp:8:8] [async-threads:64] [kernel-poll:true] > > Eshell V6.2 (abort with ^G) > 1> > > A+, Dave > ? sent from my Couch > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From comptekki@REDACTED Wed Nov 26 00:04:17 2014 From: comptekki@REDACTED (Wes James) Date: Tue, 25 Nov 2014 16:04:17 -0700 Subject: [erlang-questions] ssl error ugrading clients to erl 17.3 In-Reply-To: <54750254.1040100@ninenines.eu> References: <547501A0.8040602@ninenines.eu> <54750254.1040100@ninenines.eu> Message-ID: Thanks for the quick response Loic! That fixed it. I updated erlang on the server to 17.3.4, but didn't have to update the client (17.3 right now) and it doesn't choke on that error now :) And the server is receiving 16b data fine also. wes On Tue, Nov 25, 2014 at 3:27 PM, Lo?c Hoguin wrote: > Try with kerl: > > kerl build git https://github.com/erlang/otp OTP-17.3.4 17.3.4 > > On 11/26/2014 12:26 AM, Wes James wrote: > >> OK Thanks. Do I git clone erlang and then compile the 17.3.2 branch? >> >> wes >> >> On Tue, Nov 25, 2014 at 3:24 PM, Lo?c Hoguin > > wrote: >> >> It was fixed in 17.3.2. >> >> >> On 11/26/2014 12:23 AM, Wes James wrote: >> >> I upgraded my esysman installation to use erlang 17.3 on >> xubuntu. That >> works fine with my clients with erlang 16b, but when I update the >> clients to 17.3, I'm getting an ssl error. >> >> https://gist.github.com/__comptekki/512cd7d2971062297a19 >> >> >> I googled and found a similar error here: >> >> https://gist.github.com/__asabil/1dd2b44bb527de97f325 >> >> >> Is there a solution to fix this? >> >> Thanks, >> >> Wes >> >> >> _________________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/__listinfo/erlang-questions >> >> >> >> -- >> Lo?c Hoguin >> http://ninenines.eu >> >> >> > -- > Lo?c Hoguin > http://ninenines.eu > -------------- next part -------------- An HTML attachment was scrubbed... URL: From comptekki@REDACTED Wed Nov 26 05:53:53 2014 From: comptekki@REDACTED (Wes James) Date: Tue, 25 Nov 2014 21:53:53 -0700 Subject: [erlang-questions] erl installer command line options Message-ID: I found the command line option /S for installing erlang 17.3 x64 for windows in silent mode, but I'd also like know know if there are command line options for no docs and no Star Menu items? Thanks, wes -------------- next part -------------- An HTML attachment was scrubbed... URL: From roger@REDACTED Wed Nov 26 10:17:46 2014 From: roger@REDACTED (Roger Lipscombe) Date: Wed, 26 Nov 2014 09:17:46 +0000 Subject: [erlang-questions] Weird behavior with the visual debugger. In-Reply-To: References: Message-ID: This question on Stackoverflow is the best I've found for examples of using dbg: http://stackoverflow.com/questions/1954894/using-trace-and-dbg-in-erlang On 25 November 2014 at 19:47, Matthieu Tourne wrote: > Dan, could you go a little bit into details how you would get around to do > this ? Again, very new to the Erlang world. > > Imants, Thank you I already have this as my user_default.erl [1], but not > exactly what I've been looking for. > > > [1] http://www.snookles.com/erlang/user_default.erl > > On Tue, Nov 25, 2014 at 1:52 AM, Imants Cekusins wrote: >> >> Just in case: dbg functions allow filtering by module, function. >> >> It is possible to trace errors only. Call args & return values are >> available too. > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From essen@REDACTED Wed Nov 26 10:32:45 2014 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Wed, 26 Nov 2014 11:32:45 +0200 Subject: [erlang-questions] erl installer command line options In-Reply-To: References: Message-ID: <54759E3D.6070404@ninenines.eu> Hijacking the topic: is the source for creating the installer somewhere? On 11/26/2014 06:53 AM, Wes James wrote: > I found the command line option /S for installing erlang 17.3 x64 for > windows in silent mode, but I'd also like know know if there are command > line options for no docs and no Star Menu items? > > Thanks, > > wes > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin http://ninenines.eu From dgud@REDACTED Wed Nov 26 10:49:18 2014 From: dgud@REDACTED (Dan Gudmundsson) Date: Wed, 26 Nov 2014 10:49:18 +0100 Subject: [erlang-questions] Weird behavior with the visual debugger. In-Reply-To: References: Message-ID: There is no way around timeouts without modifying your code, what I try to say is: write your code so that the timeouts are macros and infinity when debug compiled. Something like this (untested): -ifdef(DEBUG). -define(TIMEOUT(TIME), inifinity). -else. -define(TIMEOUT(TIME), (TIME)). -endif. fetch(Data) -> gen_server:call(server, {Fetch,Data}, ?TIMEOUT(5000)). or receive Msg -> do_something(Msg) after ?TIMEOUT(5000) -> timeout end. /Dan On Tue, Nov 25, 2014 at 8:47 PM, Matthieu Tourne wrote: > Dan, could you go a little bit into details how you would get around to do > this ? Again, very new to the Erlang world. > > Imants, Thank you I already have this as my user_default.erl [1], but not > exactly what I've been looking for. > > > [1] http://www.snookles.com/erlang/user_default.erl > > On Tue, Nov 25, 2014 at 1:52 AM, Imants Cekusins wrote: > >> Just in case: dbg functions allow filtering by module, function. >> >> It is possible to trace errors only. Call args & return values are >> available too. >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas@REDACTED Wed Nov 26 10:56:42 2014 From: lukas@REDACTED (Lukas Larsson) Date: Wed, 26 Nov 2014 10:56:42 +0100 Subject: [erlang-questions] erl installer command line options In-Reply-To: References: Message-ID: Hello, I don't think there is a way to do what you want with the current installer. The /S option is builtin into any nsis installer, which is why it is there. For a full list of installer options see http://nsis.sourceforge.net/Docs/Chapter3.html#installerusagecommon To do what you want you will most probably have to edit the nsis script, which can be found here: https://github.com/erlang/otp/blob/maint/erts/etc/win32/nsis/erlang.nsi. I've never really taken the time to understand what it does, but I think it should be possible to do what you want. Lukas On Wed, Nov 26, 2014 at 5:53 AM, Wes James wrote: > I found the command line option /S for installing erlang 17.3 x64 for > windows in silent mode, but I'd also like know know if there are command > line options for no docs and no Star Menu items? > > Thanks, > > wes > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From roger@REDACTED Wed Nov 26 14:51:29 2014 From: roger@REDACTED (Roger Lipscombe) Date: Wed, 26 Nov 2014 13:51:29 +0000 Subject: [erlang-questions] Adding now + interval? Message-ID: IntervalMs = 500, Now = os:timestamp(), Future = timer:now_add(Now, IntervalMs). There is no timer:now_add/2 function, so how do I do this? From sergej.jurecko@REDACTED Wed Nov 26 14:56:56 2014 From: sergej.jurecko@REDACTED (=?UTF-8?Q?Sergej_Jure=C4=8Dko?=) Date: Wed, 26 Nov 2014 14:56:56 +0100 Subject: [erlang-questions] Adding now + interval? In-Reply-To: References: Message-ID: Adding 500000 to third element in tuple will work just fine. Unless you really need it to be proper format without overflowing microsecs. Sergej On Nov 26, 2014 2:51 PM, "Roger Lipscombe" wrote: > IntervalMs = 500, > Now = os:timestamp(), > Future = timer:now_add(Now, IntervalMs). > > There is no timer:now_add/2 function, so how do I do this? > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From roger@REDACTED Wed Nov 26 15:37:19 2014 From: roger@REDACTED (Roger Lipscombe) Date: Wed, 26 Nov 2014 14:37:19 +0000 Subject: [erlang-questions] Adding now + interval? In-Reply-To: References: Message-ID: On 26 November 2014 at 13:56, Sergej Jure?ko wrote: > Adding 500000 to third element in tuple will work just fine. Unless you > really need it to be proper format without overflowing microsecs. Yeah, I do, otherwise FurtherFuture >= Future won't return the correct answer -- as soon as the second element of the tuple is higher (i.e. next second), it'll return true, even if 500ms have not yet elapsed. From darach@REDACTED Wed Nov 26 15:43:34 2014 From: darach@REDACTED (Darach Ennis) Date: Wed, 26 Nov 2014 14:43:34 +0000 Subject: [erlang-questions] Adding now + interval? In-Reply-To: References: Message-ID: Converting to milliseconds since the UNIX epoch and comparing would be simpler and less error prone IMHO than sticking with the tuple here. Instead of needing to account for overflow, avoid it. Cheers, Darach. On Wed, Nov 26, 2014 at 2:37 PM, Roger Lipscombe wrote: > On 26 November 2014 at 13:56, Sergej Jure?ko > wrote: > > Adding 500000 to third element in tuple will work just fine. Unless you > > really need it to be proper format without overflowing microsecs. > > Yeah, I do, otherwise FurtherFuture >= Future won't return the correct > answer -- as soon as the second element of the tuple is higher (i.e. > next second), it'll return true, even if 500ms have not yet elapsed. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From axling@REDACTED Wed Nov 26 16:59:45 2014 From: axling@REDACTED (Erik Axling) Date: Wed, 26 Nov 2014 16:59:45 +0100 Subject: [erlang-questions] What are the debugging possibilities with a C port Message-ID: Hello! Im currently writing an erlang port of a C library. I can start the port but when sending commands to it it fails. What are the debugging option here. Can i use GDB in some way even though the port isn't a shared library like a nif? Can i write to stdout or will those not be seen as the port communicates over stdin and stdout? Thanks for any help. /axling -------------- next part -------------- An HTML attachment was scrubbed... URL: From comptekki@REDACTED Wed Nov 26 17:28:06 2014 From: comptekki@REDACTED (Wes James) Date: Wed, 26 Nov 2014 09:28:06 -0700 Subject: [erlang-questions] erl installer command line options In-Reply-To: References: Message-ID: Thanks. I'll see if I can figure something out from that. wes On Wed, Nov 26, 2014 at 2:56 AM, Lukas Larsson wrote: > Hello, > > I don't think there is a way to do what you want with the current > installer. The /S option is builtin into any nsis installer, which is why > it is there. For a full list of installer options see > http://nsis.sourceforge.net/Docs/Chapter3.html#installerusagecommon > > To do what you want you will most probably have to edit the nsis script, > which can be found here: > https://github.com/erlang/otp/blob/maint/erts/etc/win32/nsis/erlang.nsi. > I've never really taken the time to understand what it does, but I think it > should be possible to do what you want. > > Lukas > > On Wed, Nov 26, 2014 at 5:53 AM, Wes James wrote: > >> I found the command line option /S for installing erlang 17.3 x64 for >> windows in silent mode, but I'd also like know know if there are command >> line options for no docs and no Star Menu items? >> >> Thanks, >> >> wes >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.santos@REDACTED Wed Nov 26 17:43:10 2014 From: michael.santos@REDACTED (Michael Santos) Date: Wed, 26 Nov 2014 11:43:10 -0500 Subject: [erlang-questions] What are the debugging possibilities with a C port In-Reply-To: References: Message-ID: <20141126164310.GA24969@brk> On Wed, Nov 26, 2014 at 04:59:45PM +0100, Erik Axling wrote: > Hello! > > Im currently writing an erlang port of a C library. I can start the port > but when sending commands to it it fails. What are the debugging option > here. Can i use GDB in some way even though the port isn't a shared library > like a nif? > > Can i write to stdout or will those not be seen as the port communicates > over stdin and stdout? > > Thanks for any help. Sure, ports are easy to debug. For gdb, just run the port normally and attach to the PID. So if the port is the priv/port and the PID is 1234: gdb priv/port 1234 If you want to add some debug printf's, you can either use stderr or if you're using the stderr_to_stdout option to open_port/2, open and write to a file. From quuoms@REDACTED Wed Nov 26 15:52:44 2014 From: quuoms@REDACTED (=?utf-8?B?5YWo5Y2T?=) Date: Wed, 26 Nov 2014 22:52:44 +0800 Subject: [erlang-questions] About Erlang Shell Message-ID: Hi: It will be wonderful If the Erlang shell support clear screen and show different type of files in color . Will these features be added in the later Erlang/OTP release? Regurds Quan -------------- next part -------------- An HTML attachment was scrubbed... URL: From ct.radu.001@REDACTED Wed Nov 26 16:36:49 2014 From: ct.radu.001@REDACTED (CT Radu) Date: Wed, 26 Nov 2014 17:36:49 +0200 Subject: [erlang-questions] Removing SSL v3 support from the ssl module In-Reply-To: References: <20141015091030.GA52663@k2r.org> <368069533.74861.1413365696097.JavaMail.zimbra@tpip.net> <16095605.Z2g7XaJzps@moltowork> Message-ID: Hi all, I am currently stuck with a R15 erlang release and I do want to disable sslv3 on listening connections. More specifically on cowboy websocket listeners. The problem I am facing is that: I passed these args to the erlang vm: -ssl protocol_version "[\'tlsv1.2\',\'tlsv1.1\',tlsv1]" I passed in cowboy ssl options (that gets passed along the way to ssl:listen options) {versions, ['tlsv1.2', 'tlsv1.1', tlsv1]} And the best I got until now is: ssl:versions(). [{ssl_app,"5.1.2"}, {supported,['tlsv1.2','tlsv1.1',tlsv1]}, {available,['tlsv1.2','tlsv1.1',tlsv1,sslv3]}] And the end result is that the application can still serve via sslv3: openssl s_client -ssl3 -connect 127.0.0.1:8936 |grep "Protocol" Protocol : SSLv3 Cipher : DHE-RSA-AES256-SHA >From what I understand on R15 the versions options in ssl:listen() are ignored. Are there any other methods of disabling SSLv3 in a R15 system ? Many thanks, Costin-Tiberiu 2014-11-06 16:40 GMT+02:00 Ingela Andin : > Hi! > > 2014-10-22 17:29 GMT+02:00 Vincent de Phily < > vincent.dephily@REDACTED>: > >> On Wednesday 15 October 2014 19:41:23 Steve Vinoski wrote: >> > On Wed, Oct 15, 2014 at 5:34 AM, Andreas Schultz >> wrote: >> > > Hi, >> > > >> > > ----- On 15 Oct, 2014, at 11:10, Kenji Rikitake kenji@REDACTED wrote: >> > > > I'd be glad if how to remove SSL v3 support from OTP ssl module is >> > > > provided by the OTP Team, to prevent getting trapped into the POODLE >> > > > bug. (I think it won't be that hard, regarding what I've found from >> the >> > > > ssl module source code. The keyword atom is "sslv3".) >> > > >> > > Add {versions, ['tlsv1.2', 'tls1.1', 'tls1']} to your SSL options to >> > > restrict >> > > the version choice. >> > >> > Slight correction: {versions, ['tlsv1.2', 'tlsv1.1', 'tlsv1']} >> >> I suggest going with >> >> proplists:get_value(available,ssl:versions()) -- [sslv3] >> >> to future-proof your code a bit. I'm not sure what the difference between >> 'supported' and 'available' is (a clarification in the docs would be >> nice), >> neither of them seem to be affected by the command-line argument to >> restrict >> versions. >> >> It was proably a quoting problem, or that you did not start the ssl > application before callinge ssl:versions(). ! > > > erl -ssl protocol_version '[tlsv1]' > Erlang/OTP 18 [DEVELOPMENT] [erts-7.0] [source-7ed6eb5] [64-bit] [smp:8:8] > [async-threads:10] [hipe] [kernel-poll:false] > > Eshell V7.0 (abort with ^G) > 1> ssl:start(). > ok > 2> ssl:versions(). > [{ssl_app,"5.3.7"}, > {supported,[tlsv1]}, > {available,['tlsv1.2','tlsv1.1',tlsv1,sslv3]}] > > > erl -ssl protocol_version "['tlsv1.2', 'tlsv1.1']" > Erlang/OTP 18 [DEVELOPMENT] [erts-7.0] [source-7ed6eb5] [64-bit] [smp:8:8] > [async-threads:10] [hipe] [kernel-poll:false] > > Eshell V7.0 (abort with ^G) > 1> ssl:start(). > ok > 2> ssl:versions(). > [{ssl_app,"5.3.7"}, > {supported,['tlsv1.2','tlsv1.1']}, > {available,['tlsv1.2','tlsv1.1',tlsv1,sslv3]}] > 3> > > > Maybe we should call them configured_default (supported) and > system_default (available) ? We will think about it. > We might exclude sslv3 from the system default and make it only available > through configuration. > > Regards Ingela Erlang/OTP team - Ericsson AB > > > >> -- >> Vincent de Phily >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From simon@REDACTED Wed Nov 26 17:51:28 2014 From: simon@REDACTED (Simon MacMullen) Date: Wed, 26 Nov 2014 16:51:28 +0000 Subject: [erlang-questions] Removing SSL v3 support from the ssl module In-Reply-To: References: <20141015091030.GA52663@k2r.org> <368069533.74861.1413365696097.JavaMail.zimbra@tpip.net> <16095605.Z2g7XaJzps@moltowork> Message-ID: <54760510.3030909@rabbitmq.com> On 26/11/14 15:36, CT Radu wrote: > I am currently stuck with a R15 erlang release and I do want to disable > sslv3 on listening connections. You can't. The bug OTP-10905 prevents setting SSL versions at all on older versions. You need at least R16B01. Sorry. Cheers, Simon From garry@REDACTED Wed Nov 26 19:36:08 2014 From: garry@REDACTED (Garry Hodgson) Date: Wed, 26 Nov 2014 13:36:08 -0500 Subject: [erlang-questions] About Erlang Shell In-Reply-To: References: Message-ID: <54761D98.9040602@research.att.com> On 11/26/14 9:52 AM, ?? wrote: > Hi: > It will be wonderful If the Erlang shell support clear screen and > show different type of files in color . > Will these features be added in the later Erlang/OTP release? > i sure hope not. -------------- next part -------------- An HTML attachment was scrubbed... URL: From felixgallo@REDACTED Wed Nov 26 21:22:30 2014 From: felixgallo@REDACTED (Felix Gallo) Date: Wed, 26 Nov 2014 12:22:30 -0800 Subject: [erlang-questions] About Erlang Shell In-Reply-To: <54761D98.9040602@research.att.com> References: <54761D98.9040602@research.att.com> Message-ID: I believe there's a few efforts to make the erlang shell more, ah, beautiful. One I recently ran across: https://github.com/karlll/kjell/tree/develop with overview: http://karlll.github.io/kjell/ F. On Wed, Nov 26, 2014 at 10:36 AM, Garry Hodgson wrote: > On 11/26/14 9:52 AM, ?? wrote: > > Hi: > It will be wonderful If the Erlang shell support clear screen and show > different type of files in color . > Will these features be added in the later Erlang/OTP release? > > i sure hope not. > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zxq9@REDACTED Wed Nov 26 21:32:28 2014 From: zxq9@REDACTED (zxq9) Date: Thu, 27 Nov 2014 05:32:28 +0900 Subject: [erlang-questions] Adding now + interval? In-Reply-To: References: Message-ID: <3291195.84G2vlLYVv@changa> Depending on what you are trying to do, it may be adequate to measure the result of a difference and compare the diff against some total amount, rather than adding to an existing timestamp and comparing. Now = now(), % ... some stuff case timer:now_diff(now(), Now) > 500 of % ... On 2014?11?26? ??? 13:51:29 Roger Lipscombe wrote: > IntervalMs = 500, > Now = os:timestamp(), > Future = timer:now_add(Now, IntervalMs). > > There is no timer:now_add/2 function, so how do I do this? From tony@REDACTED Wed Nov 26 21:40:55 2014 From: tony@REDACTED (Tony Rogvall) Date: Wed, 26 Nov 2014 21:40:55 +0100 Subject: [erlang-questions] About Erlang Shell In-Reply-To: References: <54761D98.9040602@research.att.com> Message-ID: <55340225-F850-42F9-9C23-F6BAA10BC6FA@rogvall.se> pretty awesome name ! I will give it a try, only because of that :-) It is, by the way, the name of my long-dead father in law. He was by hearsay a fantastic man. Thanks for the link /Tony > On 26 nov 2014, at 21:22, Felix Gallo wrote: > > I believe there's a few efforts to make the erlang shell more, ah, beautiful. One I recently ran across: > > https://github.com/karlll/kjell/tree/develop > > with overview: http://karlll.github.io/kjell/ > > F. > > On Wed, Nov 26, 2014 at 10:36 AM, Garry Hodgson > wrote: > On 11/26/14 9:52 AM, ?? wrote: >> Hi: >> It will be wonderful If the Erlang shell support clear screen and show different type of files in color . >> Will these features be added in the later Erlang/OTP release? >> > i sure hope not. > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ingela.andin@REDACTED Wed Nov 26 22:07:50 2014 From: ingela.andin@REDACTED (Ingela Andin) Date: Wed, 26 Nov 2014 22:07:50 +0100 Subject: [erlang-questions] Removing SSL v3 support from the ssl module In-Reply-To: <54760510.3030909@rabbitmq.com> References: <20141015091030.GA52663@k2r.org> <368069533.74861.1413365696097.JavaMail.zimbra@tpip.net> <16095605.Z2g7XaJzps@moltowork> <54760510.3030909@rabbitmq.com> Message-ID: Hi! 2014-11-26 17:51 GMT+01:00 Simon MacMullen : > On 26/11/14 15:36, CT Radu wrote: > >> I am currently stuck with a R15 erlang release and I do want to disable >> sslv3 on listening connections. >> > > You can't. The bug OTP-10905 prevents setting SSL versions at all on older > versions. You need at least R16B01. > > No you can not do it cleanly, but you could always patch the ssl header file which defines the system default and then recompile as a workaround. Regards Ingela Erlang/OTP team - Ericsson AB > Sorry. > > Cheers, Simon > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlsson.richard@REDACTED Wed Nov 26 22:29:31 2014 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Wed, 26 Nov 2014 22:29:31 +0100 Subject: [erlang-questions] [ANN] erl_unused_includes.sh - removing unnecessary include declarations Message-ID: Have a large and somewhat messy codebase? Ever wanted an automatic way of stripping out include declarations that aren't actually needed? Then this is for you! https://github.com/klarna/erl_unused_includes Might also make your teeth whiter, but I can't guarantee it. /Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From davidnwelton@REDACTED Thu Nov 27 11:50:23 2014 From: davidnwelton@REDACTED (David Welton) Date: Thu, 27 Nov 2014 11:50:23 +0100 Subject: [erlang-questions] About Erlang Shell In-Reply-To: References: Message-ID: You could implement things like clear screen using escape codes: clear() -> io:format("\033[2J"). domove(X, Y) -> io:format("\e[~B;~BH", [Y, X]). On Wed, Nov 26, 2014 at 3:52 PM, ?? wrote: > Hi: > It will be wonderful If the Erlang shell support clear screen and show > different type of files in color . > Will these features be added in the later Erlang/OTP release? > > Regurds > Quan > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- David N. Welton http://www.welton.it/davidw/ http://www.dedasys.com/ From ivan@REDACTED Thu Nov 27 13:20:08 2014 From: ivan@REDACTED (Ivan Uemlianin) Date: Thu, 27 Nov 2014 12:20:08 +0000 Subject: [erlang-questions] Why does Dialyzer crash on this map? In-Reply-To: References: <54749C90.6000602@llaisdy.com> <5474BA4C.4030706@llaisdy.com> Message-ID: <547716F8.7010208@llaisdy.com> Dear Dave Thanks for your help. Here are some more details: *** OS: FreeBSD 10.0-RELEASE $ uname -a FreeBSD simba 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789: Thu Jan 16 22:34:59 UTC 2014 root@REDACTED:/usr/obj/usr/src/sys/GENERIC amd64 Upgrading to 10.1 is on my todo list. *** Erlang version now 17.3_3,3 $ pkg info erlang erlang-17.3_1,3 ... $ sudo pkg upgrade ... $ pkg info erlang erlang-17.3_3,3 ... *** Errors persist Upgrading erlang has not changed behaviour. A couple more observations: 1. Running dialyzer on the module dt below gets the error as noted previously. However, if I uncomment the '%% ok' line, dialyzer will process the module with no problems. -module(dt). -export([get_map/0]). -spec get_map() -> map(). get_map() -> #{labels => [one, two], number => 27, %% [1,2,3] => wer, %% ok kvok => #{ a => qwe, 2 => asd, [1,2,3] => wer, %% bad {4,5,6} => sdf, %% bad "abc" => zxc } }. $ dialyzer dt.erl Checking whether the PLT /home/ivan/.dialyzer_plt is up-to-date... yes Proceeding with analysis... =ERROR REPORT==== 27-Nov-2014::12:17:02 === Error in process <0.45.0> with exit value: {{case_clause,map},[{dialyzer_dataflow,find_terminals,1,[{file,"dialyzer_dataflow.erl"},{line,3451}]},{dialyzer_dataflow,find_terminals_list,3,[{file,"dialyzer_dataflow.erl"},{line,3504}]},{dialyzer_dataflow,classify_returns... dialyzer: Analysis failed with error: {{case_clause,map}, [{dialyzer_dataflow,find_terminals,1, [{file,"dialyzer_dataflow.erl"},{line,3451}]}, {dialyzer_dataflow,find_terminals_list,3, [{file,"dialyzer_dataflow.erl"},{line,3504}]}, {dialyzer_dataflow,classify_returns,1, [{file,"dialyzer_dataflow.erl"},{line,3443}]}, {dialyzer_dataflow,'-state__get_warnings/2-fun-0-',7, [{file,"dialyzer_dataflow.erl"},{line,2908}]}, {lists,foldl,3,[{file,"lists.erl"},{line,1261}]}, {dialyzer_dataflow,state__get_warnings,2, [{file,"dialyzer_dataflow.erl"},{line,2934}]}, {dialyzer_dataflow,get_warnings,5, [{file,"dialyzer_dataflow.erl"},{line,142}]}, {dialyzer_succ_typings,collect_warnings,2, [{file,"dialyzer_succ_typings.erl"},{line,182}]}]} Last messages in the log cache: Reading files and computing callgraph... done in 0.07 secs Removing edges... done in 0.00 secs 2. Running dialyzer on the module dt2 below raises the warning following. Uncommenting the '%% ok' line has no effect. -module(dt2). -export([get_map/0]). -spec get_map() -> map(). get_map() -> X = #{labels => [one, two], number => 27, %% [1,2,3] => wer, %% ok kvok => #{ a => qwe, 2 => asd, [1,2,3] => wer, %% bad {4,5,6} => sdf, %% bad "abc" => zxc } }, X. $ dialyzer dt2.erl Checking whether the PLT /home/ivan/.dialyzer_plt is up-to-date... yes Proceeding with analysis... dt2.erl:5: Function get_map/0 has no local return done in 0m0.41s done (warnings were emitted) Best wishes Ivan On 11/25/14 20:37, Dave Cottlehuber wrote: > Sorry about that last email ? > > EWORKSFORME. > > FreeBSD 10.1 amd64, erlang-17.3_2,3 from pkg. > > What FreeBSD & erlang are you using? > > dch /tmp ??? dialyzer --build_plt --apps erts kernel stdlib > Creating PLT /home/dch/.dialyzer_plt ... > Unknown functions: > compile:file/2 > compile:forms/2 > compile:noenv_forms/2 > compile:output_generated/1 > crypto:block_decrypt/4 > crypto:start/0 > Unknown types: > compile:option/0 > done in 1m8.53s > done (passed successfully) > > dch /tmp ???dialyzer ltu.erl > Checking whether the PLT /home/dch/.dialyzer_plt is up-to-date... yes > Proceeding with analysis... done in 0m0.41s > done (passed successfully) > > dch /tmp ???erl > Erlang/OTP 17 [erts-6.2] [source] [64-bit] [smp:8:8] [async-threads:64] [kernel-poll:true] > > Eshell V6.2 (abort with ^G) > 1> > > A+, Dave > ? sent from my Couch > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- ============================================================ Ivan A. Uemlianin PhD Llaisdy Speech Technology Research and Development ivan@REDACTED @llaisdy llaisdy.wordpress.com github.com/llaisdy www.linkedin.com/in/ivanuemlianin festina lente ============================================================ From bog495@REDACTED Thu Nov 27 13:38:00 2014 From: bog495@REDACTED (Bogdan Andu) Date: Thu, 27 Nov 2014 14:38:00 +0200 Subject: [erlang-questions] ssl client issue Message-ID: Hi, I am trying to connet to a site using https protocol and I get error: $ erl -noshell -s inets -s ssl -eval 'ok= httpc:request(get, {" https://github.com/rebar/rebar/wiki/rebar", []}, [], [{stream, "./rebar"}])' -s init stop {"init terminating in do_boot",{{badmatch,{error,{failed_connect,[{to_address,{"github.com",443}},{inet,[inet],{eoptions,{{{badmatch,<<0 bytes>>},[{ssl_handshake,dec_hello_extensions,2,[{file,"ssl_handshake.erl"},{line,1737}]},{ssl_handshake,decode_handshake,3,[{file,"ssl_handshake.erl"},{line,926}]},{tls_handshake,get_tls_handshake_aux,3,[{file,"tls_handshake.erl"},{line,155}]},{tls_connection,next_state,4,[{file,"tls_connection.erl"},{line,433}]},{gen_fsm,handle_msg,7,[{file,"gen_fsm.erl"},{line,503}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,237}]}]},{gen_fsm,sync_send_all_state_event,[<0.54.0>,{start,infinity},infinity]}}}}]}}},[{erl_eval,expr,3,[]}]}} I fed the ssl versions options to ssl option in httpc: $ erl -noshell -s inets -s ssl -ssl protocol_versions '[tlsv1]' -eval 'ok= httpc:request(get, {"https://github.com/rebar/rebar/wiki/rebar", []}, [{ssl, [ {versions, [tlsv1]} ]}], [{stream, "./rebar"}])' -s init stop and I get the same error. githum.com does not serves sslv3 connections anymore, but google.com does and I get the same error (more detailed) : =ERROR REPORT==== 27-Nov-2014::14:36:42 === ** State machine <0.54.0> terminating ** Last message in was {tcp,#Port<0.1310>, <<22,3,1,0,93,2,0,0,89,3,1,84,119,26,218,49,140, 143,214,55,227,58,228,149,69,14,208,108,222,237, 222,62,130,116,69,128,135,31,62,197,66,236,180, 32,177,252,205,17,16,73,136,136,192,180,178,231, 184,31,16,165,117,167,10,94,112,148,137,123,19, 218,177,209,242,30,105,160,192,7,0,0,17,0,0,0,0, 255,1,0,1,0,0,11,0,4,3,0,1,2,22,3,1,14,78,11,0, 14,74,0,14,71,0,6,201,48,130,6,197,48,130,5,173, 160,3,2,1,2,2,8,37,174,101,117,232,1,104,64,48, 13,6,9,42,134,72,134,247,13,1,1,5,5,0,48,73,49, 11,48,9,6,3,85,4,6,19,2,85,83,49,19,48,17,6,3, 85,4,10,19,10,71,111,111,103,108,101,32,73,110, 99,49,37,48,35,6,3,85,4,3,19,28,71,111,111,103, 108,101,32,73,110,116,101,114,110,101,116,32,65, 117,116,104,111,114,105,116,121,32,71,50,48,30, 23,13,49,52,49,49,50,48,48,57,50,57,49,52,90,23, 13,49,53,48,50,49,56,48,48,48,48,48,48,90,48, 102,49,11,48,9,6,3,85,4,6,19,2,85,83,49,19,48, 17,6,3,85,4,8,12,10,67,97,108,105,102,111,114, 110,105,97,49,22,48,20,6,3,85,4,7,12,13,77,111, 117,110,116,97,105,110,32,86,105,101,119,49,19, 48,17,6,3,85,4,10,12,10,71,111,111,103,108,101, 32,73,110,99,49,21,48,19,6,3,85,4,3,12,12,42,46, 103,111,111,103,108,101,46,99,111,109,48,89,48, 19,6,7,42,134,72,206,61,2,1,6,8,42,134,72,206, 61,3,1,7,3,66,0,4,239,17,2,30,58,209,122,37,213, 224,230,99,232,174,177,189,137,205,210,61,217, 56,170,25,54,178,25,237,226,204,35,216,210,186, 50,69,44,117,245,182,108,110,144,22,136,1,44, 223,216,101,244,211,3,91,5,143,64,182,1,108,149, 202,14,212,163,130,4,93,48,130,4,89,48,29,6,3, 85,29,37,4,22,48,20,6,8,43,6,1,5,5,7,3,1,6,8,43, 6,1,5,5,7,3,2,48,130,3,38,6,3,85,29,17,4,130,3, 29,48,130,3,25,130,12,42,46,103,111,111,103,108, 101,46,99,111,109,130,13,42,46,97,110,100,114, 111,105,100,46,99,111,109,130,22,42,46,97,112, 112,101,110,103,105,110,101,46,103,111,111,103, 108,101,46,99,111,109,130,18,42,46,99,108,111, 117,100,46,103,111,111,103,108,101,46,99,111, 109,130,22,42,46,103,111,111,103,108,101,45,97, 110,97,108,121,116,105,99,115,46,99,111,109,130, 11,42,46,103,111,111,103,108,101,46,99,97,130, 11,42,46,103,111,111,103,108,101,46,99,108,130, 14,42,46,103,111,111,103,108,101,46,99,111,46, 105,110,130,14,42,46,103,111,111,103,108,101,46, 99,111,46,106,112,130,14,42,46,103,111,111,103, 108,101,46,99,111,46,117,107,130,15,42,46,103, 111,111,103,108,101,46,99,111,109,46,97,114,130, 15,42,46,103,111,111,103,108,101,46,99,111,109, 46,97,117,130,15,42,46,103,111,111,103,108,101, 46,99,111,109,46,98,114,130,15,42,46,103,111, 111,103,108,101,46,99,111,109,46,99,111,130,15, 42,46,103,111,111,103,108,101,46,99,111,109,46, 109,120,130,15,42,46,103,111,111,103,108,101,46, 99,111,109,46,116,114,130,15,42,46,103,111,111, 103,108,101,46,99,111,109,46,118,110,130,11,42, 46,103,111,111,103,108,101,46,100,101,130,11,42, 46,103,111,111,103,108,101,46,101,115,130,11,42, 46,103,111,111,103,108,101,46,102,114,130,11,42, 46,103,111,111,103,108,101,46,104,117,130,11,42, 46,103,111,111,103,108,101,46,105,116,130,11,42, 46,103,111,111,103,108,101,46,110,108,130,11,42, 46,103,111,111,103,108,101,46,112,108,130,11,42, 46,103,111,111,103,108,101,46,112,116,130,18,42, 46,103,111,111,103,108,101,97,100,97,112,105, 115,46,99,111,109,130,15,42,46,103,111,111,103, 108,101,97,112,105,115,46,99,110,130,20,42,46, 103,111,111,103,108,101,99,111,109,109,101,114, 99,101,46,99,111,109,130,17,42,46,103,111,111, 103,108,101,118,105,100,101,111,46,99,111,109, 130,12,42,46,103,115,116,97,116,105,99,46,99, 110,130,13,42,46,103,115,116,97,116,105,99,46, 99,111,109,130,10,42,46,103,118,116,49,46,99, 111,109,130,10,42,46,103,118,116,50,46,99,111, 109,130,20,42,46,109,101,116,114,105,99,46,103, 115,116,97,116,105,99,46,99,111,109,130,12,42, 46,117,114,99,104,105,110,46,99,111,109,130,16, 42,46,117,114,108,46,103,111,111,103,108,101,46, 99,111,109,130,22,42,46,121,111,117,116,117,98, 101,45,110,111,99,111,111,107,105,101,46,99,111, 109,130,13,42,46,121,111,117,116,117,98,101,46, 99,111,109,130,22,42,46,121,111,117,116,117,98, 101,101,100,117,99,97,116,105,111,110,46,99,111, 109,130,11,42,46,121,116,105,109,103,46,99,111, 109,130,11,97,110,100,114,111,105,100,46,99,111, 109,130,4,103,46,99,111,130,6,103,111,111,46, 103,108,130,20,103,111,111,103,108,101,45,97, 110,97,108,121,116,105,99,115,46,99,111,109,130, 10,103,111,111,103,108,101,46,99,111,109,130,18, 103,111,111,103,108,101,99,111,109,109,101,114, 99,101,46,99,111,109,130,10,117,114,99,104,105, 110,46,99,111,109,130,8,121,111,117,116,117,46, 98,101,130,11,121,111,117,116,117,98,101,46,99, 111,109,130,20,121,111,117,116,117,98,101,101, 100,117,99,97,116,105,111,110,46,99,111,109,48, 11,6,3,85,29,15,4,4,3,2,7,128,48,104,6,8,43,6,1, 5,5,7,1,1,4,92,48,90,48,43,6,8,43,6,1,5,5,7,48, 2,134,31,104,116,116,112,58,47,47,112,107,105, 46,103,111,111,103,108,101,46,99,111,109,47,71, 73,65,71,50,46,99,114,116,48,43,6,8,43,6,1,5,5, 7,48,1,134,31,104,116,116,112,58,47,47,99,108, 105,101,110,116,115,49,46,103,111,111,103,108, 101,46,99,111,109,47,111,99,115,112,48,29,6,3, 85,29,14,4,22,4,20,94,50,174,238,148,244,84,157, 175,208,233,225,119,147,54,144,110,147,101,106, 48,12,6,3,85,29,19,1>>} ** When State == hello ** Data == [{data, [{"StateData", {state,client, {#Ref<0.0.0.55>,<0.50.0>}, gen_tcp,tls_connection,tcp,tcp_closed,tcp_error, "google.com",443,#Port<0.1310>, {ssl_options,tls, [{3,1}], verify_none, {#Fun,[]}, #Fun,false,false,undefined,1, <<>>,"***",<<>>,"***","***","***",<<>>,"***", undefined,undefined,"***","***", [<<"?\n">>, <<192,20>>, <<0,57>>, <<0,56>>, <<192,5>>, <<192,15>>, <<0,53>>, <<"?\b">>, <<192,18>>, <<0,22>>, <<0,19>>, <<192,3>>, <<"?\r">>, <<0,10>>, <<"?\t">>, <<192,19>>, <<0,51>>, <<0,50>>, <<192,4>>, <<192,14>>, <<0,47>>, <<192,7>>, <<192,17>>, <<0,5>>, <<0,4>>, <<0,21>>, <<192,2>>, <<"?\f">>, <<0,9>>], #Fun,true,268435456,false, undefined,false,undefined,undefined,true, undefined,false}, {socket_options,binary,0,0,0,false}, "***","***","***",12306,"***",24597, ssl_session_cache, {3,1}, false,undefined, {undefined,undefined}, undefined,undefined,"***","***","***",undefined, "***","***","***",16403,#Ref<0.0.0.57>,undefined, "***", {false,first}, {<0.50.0>,#Ref<0.0.0.54>}, undefined, {[],[]}, false,true,false,false,undefined,undefined, undefined}}]}] ** Reason for termination = ** {{badmatch,<<>>}, [{ssl_handshake,dec_hello_extensions,2, [{file,"ssl_handshake.erl"},{line,1737}]}, {ssl_handshake,decode_handshake,3, [{file,"ssl_handshake.erl"},{line,926}]}, {tls_handshake,get_tls_handshake_aux,3, [{file,"tls_handshake.erl"},{line,155}]}, {tls_connection,next_state,4,[{file,"tls_connection.erl"},{line,433}]}, {gen_fsm,handle_msg,7,[{file,"gen_fsm.erl"},{line,503}]}, {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,237}]}]} why I cannot connect to a https site? Bogdan -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Thu Nov 27 13:49:51 2014 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Thu, 27 Nov 2014 14:49:51 +0200 Subject: [erlang-questions] ssl client issue In-Reply-To: References: Message-ID: <54771DEF.5040205@ninenines.eu> If you are using 17.3 you need to update to 17.3.2 or above. 17.3 shipped with a broken SSL client and the OTP team didn't deem worthwhile to issue an official patch. On 11/27/2014 02:38 PM, Bogdan Andu wrote: > Hi, > > I am trying to connet to a site using https protocol and I get error: > $ erl -noshell -s inets -s ssl -eval 'ok= httpc:request(get, > {"https://github.com/rebar/rebar/wiki/rebar", []}, [], [{stream, > "./rebar"}])' -s init stop > > > {"init terminating in > do_boot",{{badmatch,{error,{failed_connect,[{to_address,{"github.com > ",443}},{inet,[inet],{eoptions,{{{badmatch,<<0 > bytes>>},[{ssl_handshake,dec_hello_extensions,2,[{file,"ssl_handshake.erl"},{line,1737}]},{ssl_handshake,decode_handshake,3,[{file,"ssl_handshake.erl"},{line,926}]},{tls_handshake,get_tls_handshake_aux,3,[{file,"tls_handshake.erl"},{line,155}]},{tls_connection,next_state,4,[{file,"tls_connection.erl"},{line,433}]},{gen_fsm,handle_msg,7,[{file,"gen_fsm.erl"},{line,503}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,237}]}]},{gen_fsm,sync_send_all_state_event,[<0.54.0>,{start,infinity},infinity]}}}}]}}},[{erl_eval,expr,3,[]}]}} > > I fed the ssl versions options to ssl option in httpc: > > $ erl -noshell -s inets -s ssl -ssl protocol_versions '[tlsv1]' -eval > 'ok= httpc:request(get, {"https://github.com/rebar/rebar/wiki/rebar", > []}, [{ssl, [ {versions, [tlsv1]} ]}], [{stream, "./rebar"}])' -s init > stop > > and I get the same error. > > githum.com does not serves sslv3 connections > anymore, but google.com does and I get the same > error (more detailed) : > > =ERROR REPORT==== 27-Nov-2014::14:36:42 === > ** State machine <0.54.0> terminating > ** Last message in was {tcp,#Port<0.1310>, > > <<22,3,1,0,93,2,0,0,89,3,1,84,119,26,218,49,140, > > 143,214,55,227,58,228,149,69,14,208,108,222,237, > > 222,62,130,116,69,128,135,31,62,197,66,236,180, > > 32,177,252,205,17,16,73,136,136,192,180,178,231, > > 184,31,16,165,117,167,10,94,112,148,137,123,19, > > 218,177,209,242,30,105,160,192,7,0,0,17,0,0,0,0, > > 255,1,0,1,0,0,11,0,4,3,0,1,2,22,3,1,14,78,11,0, > > 14,74,0,14,71,0,6,201,48,130,6,197,48,130,5,173, > > 160,3,2,1,2,2,8,37,174,101,117,232,1,104,64,48, > > 13,6,9,42,134,72,134,247,13,1,1,5,5,0,48,73,49, > > 11,48,9,6,3,85,4,6,19,2,85,83,49,19,48,17,6,3, > > 85,4,10,19,10,71,111,111,103,108,101,32,73,110, > > 99,49,37,48,35,6,3,85,4,3,19,28,71,111,111,103, > > 108,101,32,73,110,116,101,114,110,101,116,32,65, > > 117,116,104,111,114,105,116,121,32,71,50,48,30, > > 23,13,49,52,49,49,50,48,48,57,50,57,49,52,90,23, > 13,49,53,48,50,49,56,48,48,48,48,48,48,90,48, > > 102,49,11,48,9,6,3,85,4,6,19,2,85,83,49,19,48, > > 17,6,3,85,4,8,12,10,67,97,108,105,102,111,114, > > 110,105,97,49,22,48,20,6,3,85,4,7,12,13,77,111, > > 117,110,116,97,105,110,32,86,105,101,119,49,19, > > 48,17,6,3,85,4,10,12,10,71,111,111,103,108,101, > > 32,73,110,99,49,21,48,19,6,3,85,4,3,12,12,42,46, > > 103,111,111,103,108,101,46,99,111,109,48,89,48, > > 19,6,7,42,134,72,206,61,2,1,6,8,42,134,72,206, > > 61,3,1,7,3,66,0,4,239,17,2,30,58,209,122,37,213, > > 224,230,99,232,174,177,189,137,205,210,61,217, > > 56,170,25,54,178,25,237,226,204,35,216,210,186, > 50,69,44,117,245,182,108,110,144,22,136,1,44, > > 223,216,101,244,211,3,91,5,143,64,182,1,108,149, > > 202,14,212,163,130,4,93,48,130,4,89,48,29,6,3, > > 85,29,37,4,22,48,20,6,8,43,6,1,5,5,7,3,1,6,8,43, > > 6,1,5,5,7,3,2,48,130,3,38,6,3,85,29,17,4,130,3, > > 29,48,130,3,25,130,12,42,46,103,111,111,103,108, > > 101,46,99,111,109,130,13,42,46,97,110,100,114, > > 111,105,100,46,99,111,109,130,22,42,46,97,112, > > 112,101,110,103,105,110,101,46,103,111,111,103, > > 108,101,46,99,111,109,130,18,42,46,99,108,111, > 117,100,46,103,111,111,103,108,101,46,99,111, > > 109,130,22,42,46,103,111,111,103,108,101,45,97, > > 110,97,108,121,116,105,99,115,46,99,111,109,130, > > 11,42,46,103,111,111,103,108,101,46,99,97,130, > > 11,42,46,103,111,111,103,108,101,46,99,108,130, > > 14,42,46,103,111,111,103,108,101,46,99,111,46, > > 105,110,130,14,42,46,103,111,111,103,108,101,46, > > 99,111,46,106,112,130,14,42,46,103,111,111,103, > > 108,101,46,99,111,46,117,107,130,15,42,46,103, > > 111,111,103,108,101,46,99,111,109,46,97,114,130, > > 15,42,46,103,111,111,103,108,101,46,99,111,109, > > 46,97,117,130,15,42,46,103,111,111,103,108,101, > 46,99,111,109,46,98,114,130,15,42,46,103,111, > > 111,103,108,101,46,99,111,109,46,99,111,130,15, > > 42,46,103,111,111,103,108,101,46,99,111,109,46, > > 109,120,130,15,42,46,103,111,111,103,108,101,46, > > 99,111,109,46,116,114,130,15,42,46,103,111,111, > > 103,108,101,46,99,111,109,46,118,110,130,11,42, > > 46,103,111,111,103,108,101,46,100,101,130,11,42, > > 46,103,111,111,103,108,101,46,101,115,130,11,42, > > 46,103,111,111,103,108,101,46,102,114,130,11,42, > > 46,103,111,111,103,108,101,46,104,117,130,11,42, > > 46,103,111,111,103,108,101,46,105,116,130,11,42, > > 46,103,111,111,103,108,101,46,110,108,130,11,42, > > 46,103,111,111,103,108,101,46,112,108,130,11,42, > > 46,103,111,111,103,108,101,46,112,116,130,18,42, > 46,103,111,111,103,108,101,97,100,97,112,105, > > 115,46,99,111,109,130,15,42,46,103,111,111,103, > > 108,101,97,112,105,115,46,99,110,130,20,42,46, > > 103,111,111,103,108,101,99,111,109,109,101,114, > > 99,101,46,99,111,109,130,17,42,46,103,111,111, > > 103,108,101,118,105,100,101,111,46,99,111,109, > 130,12,42,46,103,115,116,97,116,105,99,46,99, > > 110,130,13,42,46,103,115,116,97,116,105,99,46, > 99,111,109,130,10,42,46,103,118,116,49,46,99, > > 111,109,130,10,42,46,103,118,116,50,46,99,111, > > 109,130,20,42,46,109,101,116,114,105,99,46,103, > > 115,116,97,116,105,99,46,99,111,109,130,12,42, > > 46,117,114,99,104,105,110,46,99,111,109,130,16, > > 42,46,117,114,108,46,103,111,111,103,108,101,46, > > 99,111,109,130,22,42,46,121,111,117,116,117,98, > > 101,45,110,111,99,111,111,107,105,101,46,99,111, > > 109,130,13,42,46,121,111,117,116,117,98,101,46, > > 99,111,109,130,22,42,46,121,111,117,116,117,98, > > 101,101,100,117,99,97,116,105,111,110,46,99,111, > > 109,130,11,42,46,121,116,105,109,103,46,99,111, > > 109,130,11,97,110,100,114,111,105,100,46,99,111, > 109,130,4,103,46,99,111,130,6,103,111,111,46, > 103,108,130,20,103,111,111,103,108,101,45,97, > > 110,97,108,121,116,105,99,115,46,99,111,109,130, > > 10,103,111,111,103,108,101,46,99,111,109,130,18, > > 103,111,111,103,108,101,99,111,109,109,101,114, > > 99,101,46,99,111,109,130,10,117,114,99,104,105, > > 110,46,99,111,109,130,8,121,111,117,116,117,46, > > 98,101,130,11,121,111,117,116,117,98,101,46,99, > > 111,109,130,20,121,111,117,116,117,98,101,101, > > 100,117,99,97,116,105,111,110,46,99,111,109,48, > > 11,6,3,85,29,15,4,4,3,2,7,128,48,104,6,8,43,6,1, > > 5,5,7,1,1,4,92,48,90,48,43,6,8,43,6,1,5,5,7,48, > > 2,134,31,104,116,116,112,58,47,47,112,107,105, > > 46,103,111,111,103,108,101,46,99,111,109,47,71, > > 73,65,71,50,46,99,114,116,48,43,6,8,43,6,1,5,5, > > 7,48,1,134,31,104,116,116,112,58,47,47,99,108, > > 105,101,110,116,115,49,46,103,111,111,103,108, > > 101,46,99,111,109,47,111,99,115,112,48,29,6,3, > > 85,29,14,4,22,4,20,94,50,174,238,148,244,84,157, > > 175,208,233,225,119,147,54,144,110,147,101,106, > 48,12,6,3,85,29,19,1>>} > ** When State == hello > ** Data == [{data, > [{"StateData", > {state,client, > {#Ref<0.0.0.55>,<0.50.0>}, > > gen_tcp,tls_connection,tcp,tcp_closed,tcp_error, > "google.com > ",443,#Port<0.1310>, > {ssl_options,tls, > [{3,1}], > verify_none, > {#Fun,[]}, > > #Fun,false,false,undefined,1, > > <<>>,"***",<<>>,"***","***","***",<<>>,"***", > undefined,undefined,"***","***", > [<<"?\n">>, > <<192,20>>, > <<0,57>>, > <<0,56>>, > <<192,5>>, > <<192,15>>, > <<0,53>>, > <<"?\b">>, > <<192,18>>, > <<0,22>>, > <<0,19>>, > <<192,3>>, > <<"?\r">>, > <<0,10>>, > <<"?\t">>, > <<192,19>>, > <<0,51>>, > <<0,50>>, > <<192,4>>, > <<192,14>>, > <<0,47>>, > <<192,7>>, > <<192,17>>, > <<0,5>>, > <<0,4>>, > <<0,21>>, > <<192,2>>, > <<"?\f">>, > <<0,9>>], > #Fun,true,268435456,false, > undefined,false,undefined,undefined,true, > undefined,false}, > {socket_options,binary,0,0,0,false}, > "***","***","***",12306,"***",24597, > ssl_session_cache, > {3,1}, > false,undefined, > {undefined,undefined}, > > undefined,undefined,"***","***","***",undefined, > > "***","***","***",16403,#Ref<0.0.0.57>,undefined, > "***", > {false,first}, > {<0.50.0>,#Ref<0.0.0.54>}, > undefined, > {[],[]}, > false,true,false,false,undefined,undefined, > undefined}}]}] > ** Reason for termination = > ** {{badmatch,<<>>}, > [{ssl_handshake,dec_hello_extensions,2, > [{file,"ssl_handshake.erl"},{line,1737}]}, > {ssl_handshake,decode_handshake,3, > [{file,"ssl_handshake.erl"},{line,926}]}, > {tls_handshake,get_tls_handshake_aux,3, > [{file,"tls_handshake.erl"},{line,155}]}, > > {tls_connection,next_state,4,[{file,"tls_connection.erl"},{line,433}]}, > {gen_fsm,handle_msg,7,[{file,"gen_fsm.erl"},{line,503}]}, > {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,237}]}]} > > > why I cannot connect to a https site? > > Bogdan > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin http://ninenines.eu From g@REDACTED Thu Nov 27 13:54:27 2014 From: g@REDACTED (Garrett Smith) Date: Thu, 27 Nov 2014 14:54:27 +0200 Subject: [erlang-questions] ssl client issue In-Reply-To: <54771DEF.5040205@ninenines.eu> References: <54771DEF.5040205@ninenines.eu> Message-ID: I've seen this as well and I believe I got things working by installing wget. Maybe this is wrong though. On Thu, Nov 27, 2014 at 2:49 PM, Lo?c Hoguin wrote: > If you are using 17.3 you need to update to 17.3.2 or above. 17.3 shipped > with a broken SSL client and the OTP team didn't deem worthwhile to issue an > official patch. > > On 11/27/2014 02:38 PM, Bogdan Andu wrote: >> >> Hi, >> >> I am trying to connet to a site using https protocol and I get error: >> $ erl -noshell -s inets -s ssl -eval 'ok= httpc:request(get, >> {"https://github.com/rebar/rebar/wiki/rebar", []}, [], [{stream, >> "./rebar"}])' -s init stop >> >> >> {"init terminating in >> do_boot",{{badmatch,{error,{failed_connect,[{to_address,{"github.com >> ",443}},{inet,[inet],{eoptions,{{{badmatch,<<0 >> >> bytes>>},[{ssl_handshake,dec_hello_extensions,2,[{file,"ssl_handshake.erl"},{line,1737}]},{ssl_handshake,decode_handshake,3,[{file,"ssl_handshake.erl"},{line,926}]},{tls_handshake,get_tls_handshake_aux,3,[{file,"tls_handshake.erl"},{line,155}]},{tls_connection,next_state,4,[{file,"tls_connection.erl"},{line,433}]},{gen_fsm,handle_msg,7,[{file,"gen_fsm.erl"},{line,503}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,237}]}]},{gen_fsm,sync_send_all_state_event,[<0.54.0>,{start,infinity},infinity]}}}}]}}},[{erl_eval,expr,3,[]}]}} >> >> I fed the ssl versions options to ssl option in httpc: >> >> $ erl -noshell -s inets -s ssl -ssl protocol_versions '[tlsv1]' -eval >> 'ok= httpc:request(get, {"https://github.com/rebar/rebar/wiki/rebar", >> []}, [{ssl, [ {versions, [tlsv1]} ]}], [{stream, "./rebar"}])' -s init >> stop >> >> and I get the same error. >> >> githum.com does not serves sslv3 connections >> anymore, but google.com does and I get the same >> >> error (more detailed) : >> >> =ERROR REPORT==== 27-Nov-2014::14:36:42 === >> ** State machine <0.54.0> terminating >> ** Last message in was {tcp,#Port<0.1310>, >> >> <<22,3,1,0,93,2,0,0,89,3,1,84,119,26,218,49,140, >> >> 143,214,55,227,58,228,149,69,14,208,108,222,237, >> >> 222,62,130,116,69,128,135,31,62,197,66,236,180, >> >> 32,177,252,205,17,16,73,136,136,192,180,178,231, >> >> 184,31,16,165,117,167,10,94,112,148,137,123,19, >> >> 218,177,209,242,30,105,160,192,7,0,0,17,0,0,0,0, >> >> 255,1,0,1,0,0,11,0,4,3,0,1,2,22,3,1,14,78,11,0, >> >> 14,74,0,14,71,0,6,201,48,130,6,197,48,130,5,173, >> >> 160,3,2,1,2,2,8,37,174,101,117,232,1,104,64,48, >> >> 13,6,9,42,134,72,134,247,13,1,1,5,5,0,48,73,49, >> >> 11,48,9,6,3,85,4,6,19,2,85,83,49,19,48,17,6,3, >> >> 85,4,10,19,10,71,111,111,103,108,101,32,73,110, >> >> 99,49,37,48,35,6,3,85,4,3,19,28,71,111,111,103, >> >> 108,101,32,73,110,116,101,114,110,101,116,32,65, >> >> 117,116,104,111,114,105,116,121,32,71,50,48,30, >> >> 23,13,49,52,49,49,50,48,48,57,50,57,49,52,90,23, >> >> 13,49,53,48,50,49,56,48,48,48,48,48,48,90,48, >> >> 102,49,11,48,9,6,3,85,4,6,19,2,85,83,49,19,48, >> >> 17,6,3,85,4,8,12,10,67,97,108,105,102,111,114, >> >> 110,105,97,49,22,48,20,6,3,85,4,7,12,13,77,111, >> >> 117,110,116,97,105,110,32,86,105,101,119,49,19, >> >> 48,17,6,3,85,4,10,12,10,71,111,111,103,108,101, >> >> 32,73,110,99,49,21,48,19,6,3,85,4,3,12,12,42,46, >> >> 103,111,111,103,108,101,46,99,111,109,48,89,48, >> >> 19,6,7,42,134,72,206,61,2,1,6,8,42,134,72,206, >> >> 61,3,1,7,3,66,0,4,239,17,2,30,58,209,122,37,213, >> >> 224,230,99,232,174,177,189,137,205,210,61,217, >> >> 56,170,25,54,178,25,237,226,204,35,216,210,186, >> >> 50,69,44,117,245,182,108,110,144,22,136,1,44, >> >> 223,216,101,244,211,3,91,5,143,64,182,1,108,149, >> >> 202,14,212,163,130,4,93,48,130,4,89,48,29,6,3, >> >> 85,29,37,4,22,48,20,6,8,43,6,1,5,5,7,3,1,6,8,43, >> >> 6,1,5,5,7,3,2,48,130,3,38,6,3,85,29,17,4,130,3, >> >> 29,48,130,3,25,130,12,42,46,103,111,111,103,108, >> >> 101,46,99,111,109,130,13,42,46,97,110,100,114, >> >> 111,105,100,46,99,111,109,130,22,42,46,97,112, >> >> 112,101,110,103,105,110,101,46,103,111,111,103, >> >> 108,101,46,99,111,109,130,18,42,46,99,108,111, >> >> 117,100,46,103,111,111,103,108,101,46,99,111, >> >> 109,130,22,42,46,103,111,111,103,108,101,45,97, >> >> 110,97,108,121,116,105,99,115,46,99,111,109,130, >> >> 11,42,46,103,111,111,103,108,101,46,99,97,130, >> >> 11,42,46,103,111,111,103,108,101,46,99,108,130, >> >> 14,42,46,103,111,111,103,108,101,46,99,111,46, >> >> 105,110,130,14,42,46,103,111,111,103,108,101,46, >> >> 99,111,46,106,112,130,14,42,46,103,111,111,103, >> >> 108,101,46,99,111,46,117,107,130,15,42,46,103, >> >> 111,111,103,108,101,46,99,111,109,46,97,114,130, >> >> 15,42,46,103,111,111,103,108,101,46,99,111,109, >> >> 46,97,117,130,15,42,46,103,111,111,103,108,101, >> >> 46,99,111,109,46,98,114,130,15,42,46,103,111, >> >> 111,103,108,101,46,99,111,109,46,99,111,130,15, >> >> 42,46,103,111,111,103,108,101,46,99,111,109,46, >> >> 109,120,130,15,42,46,103,111,111,103,108,101,46, >> >> 99,111,109,46,116,114,130,15,42,46,103,111,111, >> >> 103,108,101,46,99,111,109,46,118,110,130,11,42, >> >> 46,103,111,111,103,108,101,46,100,101,130,11,42, >> >> 46,103,111,111,103,108,101,46,101,115,130,11,42, >> >> 46,103,111,111,103,108,101,46,102,114,130,11,42, >> >> 46,103,111,111,103,108,101,46,104,117,130,11,42, >> >> 46,103,111,111,103,108,101,46,105,116,130,11,42, >> >> 46,103,111,111,103,108,101,46,110,108,130,11,42, >> >> 46,103,111,111,103,108,101,46,112,108,130,11,42, >> >> 46,103,111,111,103,108,101,46,112,116,130,18,42, >> >> 46,103,111,111,103,108,101,97,100,97,112,105, >> >> 115,46,99,111,109,130,15,42,46,103,111,111,103, >> >> 108,101,97,112,105,115,46,99,110,130,20,42,46, >> >> 103,111,111,103,108,101,99,111,109,109,101,114, >> >> 99,101,46,99,111,109,130,17,42,46,103,111,111, >> >> 103,108,101,118,105,100,101,111,46,99,111,109, >> >> 130,12,42,46,103,115,116,97,116,105,99,46,99, >> >> 110,130,13,42,46,103,115,116,97,116,105,99,46, >> >> 99,111,109,130,10,42,46,103,118,116,49,46,99, >> >> 111,109,130,10,42,46,103,118,116,50,46,99,111, >> >> 109,130,20,42,46,109,101,116,114,105,99,46,103, >> >> 115,116,97,116,105,99,46,99,111,109,130,12,42, >> >> 46,117,114,99,104,105,110,46,99,111,109,130,16, >> >> 42,46,117,114,108,46,103,111,111,103,108,101,46, >> >> 99,111,109,130,22,42,46,121,111,117,116,117,98, >> >> 101,45,110,111,99,111,111,107,105,101,46,99,111, >> >> 109,130,13,42,46,121,111,117,116,117,98,101,46, >> >> 99,111,109,130,22,42,46,121,111,117,116,117,98, >> >> 101,101,100,117,99,97,116,105,111,110,46,99,111, >> >> 109,130,11,42,46,121,116,105,109,103,46,99,111, >> >> 109,130,11,97,110,100,114,111,105,100,46,99,111, >> >> 109,130,4,103,46,99,111,130,6,103,111,111,46, >> >> 103,108,130,20,103,111,111,103,108,101,45,97, >> >> 110,97,108,121,116,105,99,115,46,99,111,109,130, >> >> 10,103,111,111,103,108,101,46,99,111,109,130,18, >> >> 103,111,111,103,108,101,99,111,109,109,101,114, >> >> 99,101,46,99,111,109,130,10,117,114,99,104,105, >> >> 110,46,99,111,109,130,8,121,111,117,116,117,46, >> >> 98,101,130,11,121,111,117,116,117,98,101,46,99, >> >> 111,109,130,20,121,111,117,116,117,98,101,101, >> >> 100,117,99,97,116,105,111,110,46,99,111,109,48, >> >> 11,6,3,85,29,15,4,4,3,2,7,128,48,104,6,8,43,6,1, >> >> 5,5,7,1,1,4,92,48,90,48,43,6,8,43,6,1,5,5,7,48, >> >> 2,134,31,104,116,116,112,58,47,47,112,107,105, >> >> 46,103,111,111,103,108,101,46,99,111,109,47,71, >> >> 73,65,71,50,46,99,114,116,48,43,6,8,43,6,1,5,5, >> >> 7,48,1,134,31,104,116,116,112,58,47,47,99,108, >> >> 105,101,110,116,115,49,46,103,111,111,103,108, >> >> 101,46,99,111,109,47,111,99,115,112,48,29,6,3, >> >> 85,29,14,4,22,4,20,94,50,174,238,148,244,84,157, >> >> 175,208,233,225,119,147,54,144,110,147,101,106, >> 48,12,6,3,85,29,19,1>>} >> ** When State == hello >> ** Data == [{data, >> [{"StateData", >> {state,client, >> {#Ref<0.0.0.55>,<0.50.0>}, >> >> gen_tcp,tls_connection,tcp,tcp_closed,tcp_error, >> "google.com >> ",443,#Port<0.1310>, >> >> {ssl_options,tls, >> [{3,1}], >> verify_none, >> {#Fun,[]}, >> >> #Fun,false,false,undefined,1, >> >> <<>>,"***",<<>>,"***","***","***",<<>>,"***", >> undefined,undefined,"***","***", >> [<<"?\n">>, >> <<192,20>>, >> <<0,57>>, >> <<0,56>>, >> <<192,5>>, >> <<192,15>>, >> <<0,53>>, >> <<"?\b">>, >> <<192,18>>, >> <<0,22>>, >> <<0,19>>, >> <<192,3>>, >> <<"?\r">>, >> <<0,10>>, >> <<"?\t">>, >> <<192,19>>, >> <<0,51>>, >> <<0,50>>, >> <<192,4>>, >> <<192,14>>, >> <<0,47>>, >> <<192,7>>, >> <<192,17>>, >> <<0,5>>, >> <<0,4>>, >> <<0,21>>, >> <<192,2>>, >> <<"?\f">>, >> <<0,9>>], >> >> #Fun,true,268435456,false, >> undefined,false,undefined,undefined,true, >> undefined,false}, >> {socket_options,binary,0,0,0,false}, >> "***","***","***",12306,"***",24597, >> ssl_session_cache, >> {3,1}, >> false,undefined, >> {undefined,undefined}, >> >> undefined,undefined,"***","***","***",undefined, >> >> "***","***","***",16403,#Ref<0.0.0.57>,undefined, >> "***", >> {false,first}, >> {<0.50.0>,#Ref<0.0.0.54>}, >> undefined, >> {[],[]}, >> false,true,false,false,undefined,undefined, >> undefined}}]}] >> ** Reason for termination = >> ** {{badmatch,<<>>}, >> [{ssl_handshake,dec_hello_extensions,2, >> [{file,"ssl_handshake.erl"},{line,1737}]}, >> {ssl_handshake,decode_handshake,3, >> [{file,"ssl_handshake.erl"},{line,926}]}, >> {tls_handshake,get_tls_handshake_aux,3, >> [{file,"tls_handshake.erl"},{line,155}]}, >> >> {tls_connection,next_state,4,[{file,"tls_connection.erl"},{line,433}]}, >> {gen_fsm,handle_msg,7,[{file,"gen_fsm.erl"},{line,503}]}, >> {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,237}]}]} >> >> >> why I cannot connect to a https site? >> >> Bogdan >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > -- > Lo?c Hoguin > http://ninenines.eu > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From bog495@REDACTED Thu Nov 27 13:56:55 2014 From: bog495@REDACTED (Bogdan Andu) Date: Thu, 27 Nov 2014 14:56:55 +0200 Subject: [erlang-questions] ssl client issue In-Reply-To: <54771DEF.5040205@ninenines.eu> References: <54771DEF.5040205@ninenines.eu> Message-ID: Hi, I didn't know that, I'll upgrade to the latest otp. Thank you, Bogdan On Thu, Nov 27, 2014 at 2:49 PM, Lo?c Hoguin wrote: > If you are using 17.3 you need to update to 17.3.2 or above. 17.3 shipped > with a broken SSL client and the OTP team didn't deem worthwhile to issue > an official patch. > > On 11/27/2014 02:38 PM, Bogdan Andu wrote: > >> Hi, >> >> I am trying to connet to a site using https protocol and I get error: >> $ erl -noshell -s inets -s ssl -eval 'ok= httpc:request(get, >> {"https://github.com/rebar/rebar/wiki/rebar", []}, [], [{stream, >> "./rebar"}])' -s init stop >> >> >> {"init terminating in >> do_boot",{{badmatch,{error,{failed_connect,[{to_address,{"github.com >> ",443}},{inet,[inet],{eoptions,{{{badmatch,<<0 >> bytes>>},[{ssl_handshake,dec_hello_extensions,2,[{file," >> ssl_handshake.erl"},{line,1737}]},{ssl_handshake,decode_ >> handshake,3,[{file,"ssl_handshake.erl"},{line,926}]},{ >> tls_handshake,get_tls_handshake_aux,3,[{file,"tls_ >> handshake.erl"},{line,155}]},{tls_connection,next_state,4,[{ >> file,"tls_connection.erl"},{line,433}]},{gen_fsm,handle_ >> msg,7,[{file,"gen_fsm.erl"},{line,503}]},{proc_lib,init_p_ >> do_apply,3,[{file,"proc_lib.erl"},{line,237}]}]},{gen_fsm, >> sync_send_all_state_event,[<0.54.0>,{start,infinity}, >> infinity]}}}}]}}},[{erl_eval,expr,3,[]}]}} >> >> I fed the ssl versions options to ssl option in httpc: >> >> $ erl -noshell -s inets -s ssl -ssl protocol_versions '[tlsv1]' -eval >> 'ok= httpc:request(get, {"https://github.com/rebar/rebar/wiki/rebar", >> []}, [{ssl, [ {versions, [tlsv1]} ]}], [{stream, "./rebar"}])' -s init >> stop >> >> and I get the same error. >> >> githum.com does not serves sslv3 connections >> anymore, but google.com does and I get the same >> >> error (more detailed) : >> >> =ERROR REPORT==== 27-Nov-2014::14:36:42 === >> ** State machine <0.54.0> terminating >> ** Last message in was {tcp,#Port<0.1310>, >> >> <<22,3,1,0,93,2,0,0,89,3,1,84,119,26,218,49,140, >> >> 143,214,55,227,58,228,149,69,14,208,108,222,237, >> >> 222,62,130,116,69,128,135,31,62,197,66,236,180, >> >> 32,177,252,205,17,16,73,136,136,192,180,178,231, >> >> 184,31,16,165,117,167,10,94,112,148,137,123,19, >> >> 218,177,209,242,30,105,160,192,7,0,0,17,0,0,0,0, >> >> 255,1,0,1,0,0,11,0,4,3,0,1,2,22,3,1,14,78,11,0, >> >> 14,74,0,14,71,0,6,201,48,130,6,197,48,130,5,173, >> >> 160,3,2,1,2,2,8,37,174,101,117,232,1,104,64,48, >> >> 13,6,9,42,134,72,134,247,13,1,1,5,5,0,48,73,49, >> >> 11,48,9,6,3,85,4,6,19,2,85,83,49,19,48,17,6,3, >> >> 85,4,10,19,10,71,111,111,103,108,101,32,73,110, >> >> 99,49,37,48,35,6,3,85,4,3,19,28,71,111,111,103, >> >> 108,101,32,73,110,116,101,114,110,101,116,32,65, >> >> 117,116,104,111,114,105,116,121,32,71,50,48,30, >> >> 23,13,49,52,49,49,50,48,48,57,50,57,49,52,90,23, >> 13,49,53,48,50,49,56,48,48,48, >> 48,48,48,90,48, >> >> 102,49,11,48,9,6,3,85,4,6,19,2,85,83,49,19,48, >> >> 17,6,3,85,4,8,12,10,67,97,108,105,102,111,114, >> >> 110,105,97,49,22,48,20,6,3,85,4,7,12,13,77,111, >> >> 117,110,116,97,105,110,32,86,105,101,119,49,19, >> >> 48,17,6,3,85,4,10,12,10,71,111,111,103,108,101, >> >> 32,73,110,99,49,21,48,19,6,3,85,4,3,12,12,42,46, >> >> 103,111,111,103,108,101,46,99,111,109,48,89,48, >> >> 19,6,7,42,134,72,206,61,2,1,6,8,42,134,72,206, >> >> 61,3,1,7,3,66,0,4,239,17,2,30,58,209,122,37,213, >> >> 224,230,99,232,174,177,189,137,205,210,61,217, >> >> 56,170,25,54,178,25,237,226,204,35,216,210,186, >> 50,69,44,117,245,182,108,110, >> 144,22,136,1,44, >> >> 223,216,101,244,211,3,91,5,143,64,182,1,108,149, >> >> 202,14,212,163,130,4,93,48,130,4,89,48,29,6,3, >> >> 85,29,37,4,22,48,20,6,8,43,6,1,5,5,7,3,1,6,8,43, >> >> 6,1,5,5,7,3,2,48,130,3,38,6,3,85,29,17,4,130,3, >> >> 29,48,130,3,25,130,12,42,46,103,111,111,103,108, >> >> 101,46,99,111,109,130,13,42,46,97,110,100,114, >> >> 111,105,100,46,99,111,109,130,22,42,46,97,112, >> >> 112,101,110,103,105,110,101,46,103,111,111,103, >> >> 108,101,46,99,111,109,130,18,42,46,99,108,111, >> 117,100,46,103,111,111,103, >> 108,101,46,99,111, >> >> 109,130,22,42,46,103,111,111,103,108,101,45,97, >> >> 110,97,108,121,116,105,99,115,46,99,111,109,130, >> >> 11,42,46,103,111,111,103,108,101,46,99,97,130, >> >> 11,42,46,103,111,111,103,108,101,46,99,108,130, >> >> 14,42,46,103,111,111,103,108,101,46,99,111,46, >> >> 105,110,130,14,42,46,103,111,111,103,108,101,46, >> >> 99,111,46,106,112,130,14,42,46,103,111,111,103, >> >> 108,101,46,99,111,46,117,107,130,15,42,46,103, >> >> 111,111,103,108,101,46,99,111,109,46,97,114,130, >> >> 15,42,46,103,111,111,103,108,101,46,99,111,109, >> >> 46,97,117,130,15,42,46,103,111,111,103,108,101, >> 46,99,111,109,46,98,114,130, >> 15,42,46,103,111, >> >> 111,103,108,101,46,99,111,109,46,99,111,130,15, >> >> 42,46,103,111,111,103,108,101,46,99,111,109,46, >> >> 109,120,130,15,42,46,103,111,111,103,108,101,46, >> >> 99,111,109,46,116,114,130,15,42,46,103,111,111, >> >> 103,108,101,46,99,111,109,46,118,110,130,11,42, >> >> 46,103,111,111,103,108,101,46,100,101,130,11,42, >> >> 46,103,111,111,103,108,101,46,101,115,130,11,42, >> >> 46,103,111,111,103,108,101,46,102,114,130,11,42, >> >> 46,103,111,111,103,108,101,46,104,117,130,11,42, >> >> 46,103,111,111,103,108,101,46,105,116,130,11,42, >> >> 46,103,111,111,103,108,101,46,110,108,130,11,42, >> >> 46,103,111,111,103,108,101,46,112,108,130,11,42, >> >> 46,103,111,111,103,108,101,46,112,116,130,18,42, >> 46,103,111,111,103,108,101,97, >> 100,97,112,105, >> >> 115,46,99,111,109,130,15,42,46,103,111,111,103, >> >> 108,101,97,112,105,115,46,99,110,130,20,42,46, >> >> 103,111,111,103,108,101,99,111,109,109,101,114, >> >> 99,101,46,99,111,109,130,17,42,46,103,111,111, >> >> 103,108,101,118,105,100,101,111,46,99,111,109, >> 130,12,42,46,103,115,116,97, >> 116,105,99,46,99, >> >> 110,130,13,42,46,103,115,116,97,116,105,99,46, >> 99,111,109,130,10,42,46,103, >> 118,116,49,46,99, >> >> 111,109,130,10,42,46,103,118,116,50,46,99,111, >> >> 109,130,20,42,46,109,101,116,114,105,99,46,103, >> >> 115,116,97,116,105,99,46,99,111,109,130,12,42, >> >> 46,117,114,99,104,105,110,46,99,111,109,130,16, >> >> 42,46,117,114,108,46,103,111,111,103,108,101,46, >> >> 99,111,109,130,22,42,46,121,111,117,116,117,98, >> >> 101,45,110,111,99,111,111,107,105,101,46,99,111, >> >> 109,130,13,42,46,121,111,117,116,117,98,101,46, >> >> 99,111,109,130,22,42,46,121,111,117,116,117,98, >> >> 101,101,100,117,99,97,116,105,111,110,46,99,111, >> >> 109,130,11,42,46,121,116,105,109,103,46,99,111, >> >> 109,130,11,97,110,100,114,111,105,100,46,99,111, >> 109,130,4,103,46,99,111,130,6, >> 103,111,111,46, >> 103,108,130,20,103,111,111, >> 103,108,101,45,97, >> >> 110,97,108,121,116,105,99,115,46,99,111,109,130, >> >> 10,103,111,111,103,108,101,46,99,111,109,130,18, >> >> 103,111,111,103,108,101,99,111,109,109,101,114, >> >> 99,101,46,99,111,109,130,10,117,114,99,104,105, >> >> 110,46,99,111,109,130,8,121,111,117,116,117,46, >> >> 98,101,130,11,121,111,117,116,117,98,101,46,99, >> >> 111,109,130,20,121,111,117,116,117,98,101,101, >> >> 100,117,99,97,116,105,111,110,46,99,111,109,48, >> >> 11,6,3,85,29,15,4,4,3,2,7,128,48,104,6,8,43,6,1, >> >> 5,5,7,1,1,4,92,48,90,48,43,6,8,43,6,1,5,5,7,48, >> >> 2,134,31,104,116,116,112,58,47,47,112,107,105, >> >> 46,103,111,111,103,108,101,46,99,111,109,47,71, >> >> 73,65,71,50,46,99,114,116,48,43,6,8,43,6,1,5,5, >> >> 7,48,1,134,31,104,116,116,112,58,47,47,99,108, >> >> 105,101,110,116,115,49,46,103,111,111,103,108, >> >> 101,46,99,111,109,47,111,99,115,112,48,29,6,3, >> >> 85,29,14,4,22,4,20,94,50,174,238,148,244,84,157, >> >> 175,208,233,225,119,147,54,144,110,147,101,106, >> 48,12,6,3,85,29,19,1>>} >> ** When State == hello >> ** Data == [{data, >> [{"StateData", >> {state,client, >> {#Ref<0.0.0.55>,<0.50.0>}, >> >> gen_tcp,tls_connection,tcp,tcp_closed,tcp_error, >> "google.com >> ",443,#Port<0.1310>, >> >> {ssl_options,tls, >> [{3,1}], >> verify_none, >> {#Fun,[]}, >> >> #Fun,false,false,undefined,1, >> >> <<>>,"***",<<>>,"***","***","***",<<>>,"***", >> undefined,undefined,"***","***", >> [<<"?\n">>, >> <<192,20>>, >> <<0,57>>, >> <<0,56>>, >> <<192,5>>, >> <<192,15>>, >> <<0,53>>, >> <<"?\b">>, >> <<192,18>>, >> <<0,22>>, >> <<0,19>>, >> <<192,3>>, >> <<"?\r">>, >> <<0,10>>, >> <<"?\t">>, >> <<192,19>>, >> <<0,51>>, >> <<0,50>>, >> <<192,4>>, >> <<192,14>>, >> <<0,47>>, >> <<192,7>>, >> <<192,17>>, >> <<0,5>>, >> <<0,4>>, >> <<0,21>>, >> <<192,2>>, >> <<"?\f">>, >> <<0,9>>], >> #Fun,true, >> 268435456,false, >> undefined,false,undefined, >> undefined,true, >> undefined,false}, >> {socket_options,binary,0,0,0,false}, >> "***","***","***",12306,"***",24597, >> ssl_session_cache, >> {3,1}, >> false,undefined, >> {undefined,undefined}, >> >> undefined,undefined,"***","***","***",undefined, >> >> "***","***","***",16403,#Ref<0.0.0.57>,undefined, >> "***", >> {false,first}, >> {<0.50.0>,#Ref<0.0.0.54>}, >> undefined, >> {[],[]}, >> false,true,false,false,undefined,undefined, >> undefined}}]}] >> ** Reason for termination = >> ** {{badmatch,<<>>}, >> [{ssl_handshake,dec_hello_extensions,2, >> [{file,"ssl_handshake.erl"},{line,1737}]}, >> {ssl_handshake,decode_handshake,3, >> [{file,"ssl_handshake.erl"},{line,926}]}, >> {tls_handshake,get_tls_handshake_aux,3, >> [{file,"tls_handshake.erl"},{line,155}]}, >> >> {tls_connection,next_state,4,[{file,"tls_connection.erl"},{line,433}]}, >> {gen_fsm,handle_msg,7,[{file,"gen_fsm.erl"},{line,503}]}, >> {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,237}]}]} >> >> >> why I cannot connect to a https site? >> >> Bogdan >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -- > Lo?c Hoguin > http://ninenines.eu > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wallentin.dahlberg@REDACTED Thu Nov 27 21:13:21 2014 From: wallentin.dahlberg@REDACTED (=?UTF-8?Q?Bj=C3=B6rn=2DEgil_Dahlberg?=) Date: Thu, 27 Nov 2014 21:13:21 +0100 Subject: [erlang-questions] Why does Dialyzer crash on this map? In-Reply-To: <547716F8.7010208@llaisdy.com> References: <54749C90.6000602@llaisdy.com> <5474BA4C.4030706@llaisdy.com> <547716F8.7010208@llaisdy.com> Message-ID: I managed to reproduce your errors on dt.erl and dt2.erl using OTP-17.3.3 tag. Seems like I've missed something in dataflow .. looks like a clause missing for the map type. I'm not well versed in dialyzer. Hopefully I can do a fix for it for 17.4. There are a bit more stuff going on here too. When dialyzer uses the erlang compiler to compile to core it does not coalesce things to literals but keeps them abstract instead .. Compound keys such as [1,2,3] and {4,5,6} will thus become non-literals (variables) and such keys are not allowed in maps. I need to ponder that a bit before I try to solve it. // Bj?rn-Egil 2014-11-27 13:20 GMT+01:00 Ivan Uemlianin : > Dear Dave > > Thanks for your help. Here are some more details: > > > *** OS: FreeBSD 10.0-RELEASE > > $ uname -a > FreeBSD simba 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789: Thu Jan 16 > 22:34:59 UTC 2014 root@REDACTED:/usr/obj/usr/src/sys/GENERIC > amd64 > > Upgrading to 10.1 is on my todo list. > > > *** Erlang version now 17.3_3,3 > > $ pkg info erlang > erlang-17.3_1,3 > ... > $ sudo pkg upgrade > ... > > $ pkg info erlang > erlang-17.3_3,3 > ... > > > *** Errors persist > > Upgrading erlang has not changed behaviour. > > A couple more observations: > > 1. Running dialyzer on the module dt below gets the error as noted > previously. However, if I uncomment the '%% ok' line, dialyzer will > process the module with no problems. > > -module(dt). > -export([get_map/0]). > > -spec get_map() -> map(). > get_map() -> > #{labels => [one, two], > number => 27, > %% [1,2,3] => wer, %% ok > kvok => #{ > a => qwe, > 2 => asd, > [1,2,3] => wer, %% bad > {4,5,6} => sdf, %% bad > "abc" => zxc > } > }. > > $ dialyzer dt.erl > Checking whether the PLT /home/ivan/.dialyzer_plt is up-to-date... > yes > Proceeding with analysis... > =ERROR REPORT==== 27-Nov-2014::12:17:02 === > Error in process <0.45.0> with exit value: > {{case_clause,map},[{dialyzer_dataflow,find_terminals,1,[{ > file,"dialyzer_dataflow.erl"},{line,3451}]},{dialyzer_ > dataflow,find_terminals_list,3,[{file,"dialyzer_dataflow. > erl"},{line,3504}]},{dialyzer_dataflow,classify_returns... > > > dialyzer: Analysis failed with error: > {{case_clause,map}, > [{dialyzer_dataflow,find_terminals,1, > [{file,"dialyzer_dataflow.erl"},{line,3451}]}, > {dialyzer_dataflow,find_terminals_list,3, > [{file,"dialyzer_dataflow.erl"},{line,3504}]}, > {dialyzer_dataflow,classify_returns,1, > [{file,"dialyzer_dataflow.erl"},{line,3443}]}, > {dialyzer_dataflow,'-state__get_warnings/2-fun-0-',7, > [{file,"dialyzer_dataflow.erl"},{line,2908}]}, > {lists,foldl,3,[{file,"lists.erl"},{line,1261}]}, > {dialyzer_dataflow,state__get_warnings,2, > [{file,"dialyzer_dataflow.erl"},{line,2934}]}, > {dialyzer_dataflow,get_warnings,5, > [{file,"dialyzer_dataflow.erl"},{line,142}]}, > {dialyzer_succ_typings,collect_warnings,2, > [{file,"dialyzer_succ_typings.erl"},{line,182}]}]} > Last messages in the log cache: > Reading files and computing callgraph... done in 0.07 secs > Removing edges... done in 0.00 secs > > > 2. Running dialyzer on the module dt2 below raises the warning following. > Uncommenting the '%% ok' line has no effect. > > -module(dt2). > -export([get_map/0]). > > -spec get_map() -> map(). > get_map() -> > X = #{labels => [one, two], > number => 27, > %% [1,2,3] => wer, %% ok > kvok => #{ > a => qwe, > 2 => asd, > [1,2,3] => wer, %% bad > {4,5,6} => sdf, %% bad > "abc" => zxc > } > }, > X. > > $ dialyzer dt2.erl > Checking whether the PLT /home/ivan/.dialyzer_plt is up-to-date... > yes > Proceeding with analysis... > dt2.erl:5: Function get_map/0 has no local return > done in 0m0.41s > done (warnings were emitted) > > Best wishes > > Ivan > > > > > On 11/25/14 20:37, Dave Cottlehuber wrote: > >> Sorry about that last email ? >> >> EWORKSFORME. >> >> FreeBSD 10.1 amd64, erlang-17.3_2,3 from pkg. >> >> What FreeBSD & erlang are you using? >> >> dch /tmp ??? dialyzer --build_plt --apps erts kernel stdlib >> Creating PLT /home/dch/.dialyzer_plt ... >> Unknown functions: >> compile:file/2 >> compile:forms/2 >> compile:noenv_forms/2 >> compile:output_generated/1 >> crypto:block_decrypt/4 >> crypto:start/0 >> Unknown types: >> compile:option/0 >> done in 1m8.53s >> done (passed successfully) >> >> dch /tmp ???dialyzer ltu.erl >> Checking whether the PLT /home/dch/.dialyzer_plt is up-to-date... yes >> Proceeding with analysis... done in 0m0.41s >> done (passed successfully) >> >> dch /tmp ???erl >> Erlang/OTP 17 [erts-6.2] [source] [64-bit] [smp:8:8] [async-threads:64] >> [kernel-poll:true] >> >> Eshell V6.2 (abort with ^G) >> 1> >> >> A+, Dave >> ? sent from my Couch >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > -- > ============================================================ > Ivan A. Uemlianin PhD > Llaisdy > Speech Technology Research and Development > > ivan@REDACTED > @llaisdy > llaisdy.wordpress.com > github.com/llaisdy > www.linkedin.com/in/ivanuemlianin > > festina lente > ============================================================ > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wallentin.dahlberg@REDACTED Thu Nov 27 21:52:36 2014 From: wallentin.dahlberg@REDACTED (=?UTF-8?Q?Bj=C3=B6rn=2DEgil_Dahlberg?=) Date: Thu, 27 Nov 2014 21:52:36 +0100 Subject: [erlang-questions] Why does Dialyzer crash on this map? In-Reply-To: References: <54749C90.6000602@llaisdy.com> <5474BA4C.4030706@llaisdy.com> <547716F8.7010208@llaisdy.com> Message-ID: Looks like the regression was introduced in 'hb/dialyzer/fix_recognition_of_records/OTP-11935' (805f9c89fc01220bc1bb0f27e1b68fd4eca688ba) included from OTP 17.1 This becomes a none-issue in the master branch (for 18) since variables are allowed as keys there but it needs to be solved for the 17 track. 2014-11-27 21:13 GMT+01:00 Bj?rn-Egil Dahlberg : > I managed to reproduce your errors on dt.erl and dt2.erl using OTP-17.3.3 > tag. > > Seems like I've missed something in dataflow .. looks like a clause > missing for the map type. > I'm not well versed in dialyzer. Hopefully I can do a fix for it for 17.4. > > There are a bit more stuff going on here too. When dialyzer uses the > erlang compiler to compile to core > it does not coalesce things to literals but keeps them abstract instead .. > Compound keys such as [1,2,3] and {4,5,6} will thus become non-literals > (variables) and such keys are not allowed in maps. > > I need to ponder that a bit before I try to solve it. > > // Bj?rn-Egil > > 2014-11-27 13:20 GMT+01:00 Ivan Uemlianin : > >> Dear Dave >> >> Thanks for your help. Here are some more details: >> >> >> *** OS: FreeBSD 10.0-RELEASE >> >> $ uname -a >> FreeBSD simba 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789: Thu Jan 16 >> 22:34:59 UTC 2014 root@REDACTED:/usr/obj/usr/src/sys/GENERIC >> amd64 >> >> Upgrading to 10.1 is on my todo list. >> >> >> *** Erlang version now 17.3_3,3 >> >> $ pkg info erlang >> erlang-17.3_1,3 >> ... >> $ sudo pkg upgrade >> ... >> >> $ pkg info erlang >> erlang-17.3_3,3 >> ... >> >> >> *** Errors persist >> >> Upgrading erlang has not changed behaviour. >> >> A couple more observations: >> >> 1. Running dialyzer on the module dt below gets the error as noted >> previously. However, if I uncomment the '%% ok' line, dialyzer will >> process the module with no problems. >> >> -module(dt). >> -export([get_map/0]). >> >> -spec get_map() -> map(). >> get_map() -> >> #{labels => [one, two], >> number => 27, >> %% [1,2,3] => wer, %% ok >> kvok => #{ >> a => qwe, >> 2 => asd, >> [1,2,3] => wer, %% bad >> {4,5,6} => sdf, %% bad >> "abc" => zxc >> } >> }. >> >> $ dialyzer dt.erl >> Checking whether the PLT /home/ivan/.dialyzer_plt is up-to-date... >> yes >> Proceeding with analysis... >> =ERROR REPORT==== 27-Nov-2014::12:17:02 === >> Error in process <0.45.0> with exit value: >> {{case_clause,map},[{dialyzer_dataflow,find_terminals,1,[{ >> file,"dialyzer_dataflow.erl"},{line,3451}]},{dialyzer_ >> dataflow,find_terminals_list,3,[{file,"dialyzer_dataflow. >> erl"},{line,3504}]},{dialyzer_dataflow,classify_returns... >> >> >> dialyzer: Analysis failed with error: >> {{case_clause,map}, >> [{dialyzer_dataflow,find_terminals,1, >> [{file,"dialyzer_dataflow.erl"},{line,3451}]}, >> {dialyzer_dataflow,find_terminals_list,3, >> [{file,"dialyzer_dataflow.erl"},{line,3504}]}, >> {dialyzer_dataflow,classify_returns,1, >> [{file,"dialyzer_dataflow.erl"},{line,3443}]}, >> {dialyzer_dataflow,'-state__get_warnings/2-fun-0-',7, >> [{file,"dialyzer_dataflow.erl"},{line,2908}]}, >> {lists,foldl,3,[{file,"lists.erl"},{line,1261}]}, >> {dialyzer_dataflow,state__get_warnings,2, >> [{file,"dialyzer_dataflow.erl"},{line,2934}]}, >> {dialyzer_dataflow,get_warnings,5, >> [{file,"dialyzer_dataflow.erl"},{line,142}]}, >> {dialyzer_succ_typings,collect_warnings,2, >> [{file,"dialyzer_succ_typings.erl"},{line,182}]}]} >> Last messages in the log cache: >> Reading files and computing callgraph... done in 0.07 secs >> Removing edges... done in 0.00 secs >> >> >> 2. Running dialyzer on the module dt2 below raises the warning >> following. Uncommenting the '%% ok' line has no effect. >> >> -module(dt2). >> -export([get_map/0]). >> >> -spec get_map() -> map(). >> get_map() -> >> X = #{labels => [one, two], >> number => 27, >> %% [1,2,3] => wer, %% ok >> kvok => #{ >> a => qwe, >> 2 => asd, >> [1,2,3] => wer, %% bad >> {4,5,6} => sdf, %% bad >> "abc" => zxc >> } >> }, >> X. >> >> $ dialyzer dt2.erl >> Checking whether the PLT /home/ivan/.dialyzer_plt is up-to-date... >> yes >> Proceeding with analysis... >> dt2.erl:5: Function get_map/0 has no local return >> done in 0m0.41s >> done (warnings were emitted) >> >> Best wishes >> >> Ivan >> >> >> >> >> On 11/25/14 20:37, Dave Cottlehuber wrote: >> >>> Sorry about that last email ? >>> >>> EWORKSFORME. >>> >>> FreeBSD 10.1 amd64, erlang-17.3_2,3 from pkg. >>> >>> What FreeBSD & erlang are you using? >>> >>> dch /tmp ??? dialyzer --build_plt --apps erts kernel stdlib >>> Creating PLT /home/dch/.dialyzer_plt ... >>> Unknown functions: >>> compile:file/2 >>> compile:forms/2 >>> compile:noenv_forms/2 >>> compile:output_generated/1 >>> crypto:block_decrypt/4 >>> crypto:start/0 >>> Unknown types: >>> compile:option/0 >>> done in 1m8.53s >>> done (passed successfully) >>> >>> dch /tmp ???dialyzer ltu.erl >>> Checking whether the PLT /home/dch/.dialyzer_plt is up-to-date... yes >>> Proceeding with analysis... done in 0m0.41s >>> done (passed successfully) >>> >>> dch /tmp ???erl >>> Erlang/OTP 17 [erts-6.2] [source] [64-bit] [smp:8:8] [async-threads:64] >>> [kernel-poll:true] >>> >>> Eshell V6.2 (abort with ^G) >>> 1> >>> >>> A+, Dave >>> ? sent from my Couch >>> >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >> >> -- >> ============================================================ >> Ivan A. Uemlianin PhD >> Llaisdy >> Speech Technology Research and Development >> >> ivan@REDACTED >> @llaisdy >> llaisdy.wordpress.com >> github.com/llaisdy >> www.linkedin.com/in/ivanuemlianin >> >> festina lente >> ============================================================ >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ivan@REDACTED Thu Nov 27 22:05:19 2014 From: ivan@REDACTED (Ivan Uemlianin) Date: Thu, 27 Nov 2014 21:05:19 +0000 Subject: [erlang-questions] Why does Dialyzer crash on this map? In-Reply-To: References: <54749C90.6000602@llaisdy.com> <5474BA4C.4030706@llaisdy.com> <547716F8.7010208@llaisdy.com> Message-ID: <4B9AEB77-A709-4D97-8EF2-8FE1AF5A4071@llaisdy.com> Dear Bj?rn-Egil Thanks very much for your work and for reporting back. It looks like good news (for me and for 18 at least). Best wishes Ivan -- festina lente > On 27 Nov 2014, at 20:52, Bj?rn-Egil Dahlberg wrote: > > Looks like the regression was introduced in 'hb/dialyzer/fix_recognition_of_records/OTP-11935' (805f9c89fc01220bc1bb0f27e1b68fd4eca688ba) included from OTP 17.1 > > This becomes a none-issue in the master branch (for 18) since variables are allowed as keys there but it needs to be solved for the 17 track. > > > 2014-11-27 21:13 GMT+01:00 Bj?rn-Egil Dahlberg : >> I managed to reproduce your errors on dt.erl and dt2.erl using OTP-17.3.3 tag. >> >> Seems like I've missed something in dataflow .. looks like a clause missing for the map type. >> I'm not well versed in dialyzer. Hopefully I can do a fix for it for 17.4. >> >> There are a bit more stuff going on here too. When dialyzer uses the erlang compiler to compile to core >> it does not coalesce things to literals but keeps them abstract instead .. >> Compound keys such as [1,2,3] and {4,5,6} will thus become non-literals (variables) and such keys are not allowed in maps. >> >> I need to ponder that a bit before I try to solve it. >> >> // Bj?rn-Egil >> >> 2014-11-27 13:20 GMT+01:00 Ivan Uemlianin : >>> Dear Dave >>> >>> Thanks for your help. Here are some more details: >>> >>> >>> *** OS: FreeBSD 10.0-RELEASE >>> >>> $ uname -a >>> FreeBSD simba 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789: Thu Jan 16 22:34:59 UTC 2014 root@REDACTED:/usr/obj/usr/src/sys/GENERIC amd64 >>> >>> Upgrading to 10.1 is on my todo list. >>> >>> >>> *** Erlang version now 17.3_3,3 >>> >>> $ pkg info erlang >>> erlang-17.3_1,3 >>> ... >>> $ sudo pkg upgrade >>> ... >>> >>> $ pkg info erlang >>> erlang-17.3_3,3 >>> ... >>> >>> >>> *** Errors persist >>> >>> Upgrading erlang has not changed behaviour. >>> >>> A couple more observations: >>> >>> 1. Running dialyzer on the module dt below gets the error as noted previously. However, if I uncomment the '%% ok' line, dialyzer will process the module with no problems. >>> >>> -module(dt). >>> -export([get_map/0]). >>> >>> -spec get_map() -> map(). >>> get_map() -> >>> #{labels => [one, two], >>> number => 27, >>> %% [1,2,3] => wer, %% ok >>> kvok => #{ >>> a => qwe, >>> 2 => asd, >>> [1,2,3] => wer, %% bad >>> {4,5,6} => sdf, %% bad >>> "abc" => zxc >>> } >>> }. >>> >>> $ dialyzer dt.erl >>> Checking whether the PLT /home/ivan/.dialyzer_plt is up-to-date... yes >>> Proceeding with analysis... >>> =ERROR REPORT==== 27-Nov-2014::12:17:02 === >>> Error in process <0.45.0> with exit value: {{case_clause,map},[{dialyzer_dataflow,find_terminals,1,[{file,"dialyzer_dataflow.erl"},{line,3451}]},{dialyzer_dataflow,find_terminals_list,3,[{file,"dialyzer_dataflow.erl"},{line,3504}]},{dialyzer_dataflow,classify_returns... >>> >>> >>> dialyzer: Analysis failed with error: >>> {{case_clause,map}, >>> [{dialyzer_dataflow,find_terminals,1, >>> [{file,"dialyzer_dataflow.erl"},{line,3451}]}, >>> {dialyzer_dataflow,find_terminals_list,3, >>> [{file,"dialyzer_dataflow.erl"},{line,3504}]}, >>> {dialyzer_dataflow,classify_returns,1, >>> [{file,"dialyzer_dataflow.erl"},{line,3443}]}, >>> {dialyzer_dataflow,'-state__get_warnings/2-fun-0-',7, >>> [{file,"dialyzer_dataflow.erl"},{line,2908}]}, >>> {lists,foldl,3,[{file,"lists.erl"},{line,1261}]}, >>> {dialyzer_dataflow,state__get_warnings,2, >>> [{file,"dialyzer_dataflow.erl"},{line,2934}]}, >>> {dialyzer_dataflow,get_warnings,5, >>> [{file,"dialyzer_dataflow.erl"},{line,142}]}, >>> {dialyzer_succ_typings,collect_warnings,2, >>> [{file,"dialyzer_succ_typings.erl"},{line,182}]}]} >>> Last messages in the log cache: >>> Reading files and computing callgraph... done in 0.07 secs >>> Removing edges... done in 0.00 secs >>> >>> >>> 2. Running dialyzer on the module dt2 below raises the warning following. Uncommenting the '%% ok' line has no effect. >>> >>> -module(dt2). >>> -export([get_map/0]). >>> >>> -spec get_map() -> map(). >>> get_map() -> >>> X = #{labels => [one, two], >>> number => 27, >>> %% [1,2,3] => wer, %% ok >>> kvok => #{ >>> a => qwe, >>> 2 => asd, >>> [1,2,3] => wer, %% bad >>> {4,5,6} => sdf, %% bad >>> "abc" => zxc >>> } >>> }, >>> X. >>> >>> $ dialyzer dt2.erl >>> Checking whether the PLT /home/ivan/.dialyzer_plt is up-to-date... yes >>> Proceeding with analysis... >>> dt2.erl:5: Function get_map/0 has no local return >>> done in 0m0.41s >>> done (warnings were emitted) >>> >>> Best wishes >>> >>> Ivan >>> >>> >>> >>> >>>> On 11/25/14 20:37, Dave Cottlehuber wrote: >>>> Sorry about that last email ? >>>> >>>> EWORKSFORME. >>>> >>>> FreeBSD 10.1 amd64, erlang-17.3_2,3 from pkg. >>>> >>>> What FreeBSD & erlang are you using? >>>> >>>> dch /tmp ??? dialyzer --build_plt --apps erts kernel stdlib >>>> Creating PLT /home/dch/.dialyzer_plt ... >>>> Unknown functions: >>>> compile:file/2 >>>> compile:forms/2 >>>> compile:noenv_forms/2 >>>> compile:output_generated/1 >>>> crypto:block_decrypt/4 >>>> crypto:start/0 >>>> Unknown types: >>>> compile:option/0 >>>> done in 1m8.53s >>>> done (passed successfully) >>>> >>>> dch /tmp ???dialyzer ltu.erl >>>> Checking whether the PLT /home/dch/.dialyzer_plt is up-to-date... yes >>>> Proceeding with analysis... done in 0m0.41s >>>> done (passed successfully) >>>> >>>> dch /tmp ???erl >>>> Erlang/OTP 17 [erts-6.2] [source] [64-bit] [smp:8:8] [async-threads:64] [kernel-poll:true] >>>> >>>> Eshell V6.2 (abort with ^G) >>>> 1> >>>> >>>> A+, Dave >>>> ? sent from my Couch >>>> >>>> >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> -- >>> ============================================================ >>> Ivan A. Uemlianin PhD >>> Llaisdy >>> Speech Technology Research and Development >>> >>> ivan@REDACTED >>> @llaisdy >>> llaisdy.wordpress.com >>> github.com/llaisdy >>> www.linkedin.com/in/ivanuemlianin >>> >>> festina lente >>> ============================================================ >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bog495@REDACTED Fri Nov 28 10:29:05 2014 From: bog495@REDACTED (Bogdan Andu) Date: Fri, 28 Nov 2014 11:29:05 +0200 Subject: [erlang-questions] ssl client issue In-Reply-To: References: <54771DEF.5040205@ninenines.eu> Message-ID: but curiously enough, with self-signed-certificate servers is working regardless protocol versions - just works. Perhaps 'SNI-extension that must be empty' is not involved in this case? Bogdan On Thu, Nov 27, 2014 at 10:49 PM, Dave Cottlehuber wrote: > Looks like > http://erlang.org/pipermail/erlang-questions/2014-September/081176.html > > On OSX I applied > https://github.com/erlang/otp/commit/b196730a325cfe74312c3a5f4b1273ba7c705ed6.diff to > fix this, and for FreeBSD I just switched to a newer > https://github.com/erlang/otp/archive/OTP-17.3.4.tar.gz > > A+ > Dave > > -----Original Message----- > From: Bogdan Andu > Reply: Bogdan Andu > > Date: 27. November 2014 at 13:57:10 > To: Erlang > > Subject: Re: [erlang-questions] ssl client issue > > > Hi, > > > > I didn't know that, I'll upgrade to the latest otp. > > > > Thank you, > > > > Bogdan > > > > On Thu, Nov 27, 2014 at 2:49 PM, Lo?c Hoguin wrote: > > > > > If you are using 17.3 you need to update to 17.3.2 or above. 17.3 > shipped > > > with a broken SSL client and the OTP team didn't deem worthwhile to > issue > > > an official patch. > > > > > > On 11/27/2014 02:38 PM, Bogdan Andu wrote: > > > > > >> Hi, > > >> > > >> I am trying to connet to a site using https protocol and I get error: > > >> $ erl -noshell -s inets -s ssl -eval 'ok= httpc:request(get, > > >> {"https://github.com/rebar/rebar/wiki/rebar", []}, [], [{stream, > > >> "./rebar"}])' -s init stop > > >> > > >> > > >> {"init terminating in > > >> do_boot",{{badmatch,{error,{failed_connect,[{to_address,{"github.com > > >> ",443}},{inet,[inet],{eoptions,{{{badmatch,<<0 > > >> bytes>>},[{ssl_handshake,dec_hello_extensions,2,[{file," > > >> ssl_handshake.erl"},{line,1737}]},{ssl_handshake,decode_ > > >> handshake,3,[{file,"ssl_handshake.erl"},{line,926}]},{ > > >> tls_handshake,get_tls_handshake_aux,3,[{file,"tls_ > > >> handshake.erl"},{line,155}]},{tls_connection,next_state,4,[{ > > >> file,"tls_connection.erl"},{line,433}]},{gen_fsm,handle_ > > >> msg,7,[{file,"gen_fsm.erl"},{line,503}]},{proc_lib,init_p_ > > >> do_apply,3,[{file,"proc_lib.erl"},{line,237}]}]},{gen_fsm, > > >> sync_send_all_state_event,[<0.54.0>,{start,infinity}, > > >> infinity]}}}}]}}},[{erl_eval,expr,3,[]}]}} > > >> > > >> I fed the ssl versions options to ssl option in httpc: > > >> > > >> $ erl -noshell -s inets -s ssl -ssl protocol_versions '[tlsv1]' -eval > > >> 'ok= httpc:request(get, {"https://github.com/rebar/rebar/wiki/rebar", > > >> []}, [{ssl, [ {versions, [tlsv1]} ]}], [{stream, "./rebar"}])' -s init > > >> stop > > >> > > >> and I get the same error. > > >> > > >> githum.com does not serves sslv3 connections > > >> anymore, but google.com does and I get the same > > >> > > >> error (more detailed) : > > >> > > >> =ERROR REPORT==== 27-Nov-2014::14:36:42 === > > >> ** State machine <0.54.0> terminating > > >> ** Last message in was {tcp,#Port<0.1310>, > > >> > > >> <<22,3,1,0,93,2,0,0,89,3,1,84,119,26,218,49,140, > > >> > > >> 143,214,55,227,58,228,149,69,14,208,108,222,237, > > >> > > >> 222,62,130,116,69,128,135,31,62,197,66,236,180, > > >> > > >> 32,177,252,205,17,16,73,136,136,192,180,178,231, > > >> > > >> 184,31,16,165,117,167,10,94,112,148,137,123,19, > > >> > > >> 218,177,209,242,30,105,160,192,7,0,0,17,0,0,0,0, > > >> > > >> 255,1,0,1,0,0,11,0,4,3,0,1,2,22,3,1,14,78,11,0, > > >> > > >> 14,74,0,14,71,0,6,201,48,130,6,197,48,130,5,173, > > >> > > >> 160,3,2,1,2,2,8,37,174,101,117,232,1,104,64,48, > > >> > > >> 13,6,9,42,134,72,134,247,13,1,1,5,5,0,48,73,49, > > >> > > >> 11,48,9,6,3,85,4,6,19,2,85,83,49,19,48,17,6,3, > > >> > > >> 85,4,10,19,10,71,111,111,103,108,101,32,73,110, > > >> > > >> 99,49,37,48,35,6,3,85,4,3,19,28,71,111,111,103, > > >> > > >> 108,101,32,73,110,116,101,114,110,101,116,32,65, > > >> > > >> 117,116,104,111,114,105,116,121,32,71,50,48,30, > > >> > > >> 23,13,49,52,49,49,50,48,48,57,50,57,49,52,90,23, > > >> 13,49,53,48,50,49,56,48,48,48, > > >> 48,48,48,90,48, > > >> > > >> 102,49,11,48,9,6,3,85,4,6,19,2,85,83,49,19,48, > > >> > > >> 17,6,3,85,4,8,12,10,67,97,108,105,102,111,114, > > >> > > >> 110,105,97,49,22,48,20,6,3,85,4,7,12,13,77,111, > > >> > > >> 117,110,116,97,105,110,32,86,105,101,119,49,19, > > >> > > >> 48,17,6,3,85,4,10,12,10,71,111,111,103,108,101, > > >> > > >> 32,73,110,99,49,21,48,19,6,3,85,4,3,12,12,42,46, > > >> > > >> 103,111,111,103,108,101,46,99,111,109,48,89,48, > > >> > > >> 19,6,7,42,134,72,206,61,2,1,6,8,42,134,72,206, > > >> > > >> 61,3,1,7,3,66,0,4,239,17,2,30,58,209,122,37,213, > > >> > > >> 224,230,99,232,174,177,189,137,205,210,61,217, > > >> > > >> 56,170,25,54,178,25,237,226,204,35,216,210,186, > > >> 50,69,44,117,245,182,108,110, > > >> 144,22,136,1,44, > > >> > > >> 223,216,101,244,211,3,91,5,143,64,182,1,108,149, > > >> > > >> 202,14,212,163,130,4,93,48,130,4,89,48,29,6,3, > > >> > > >> 85,29,37,4,22,48,20,6,8,43,6,1,5,5,7,3,1,6,8,43, > > >> > > >> 6,1,5,5,7,3,2,48,130,3,38,6,3,85,29,17,4,130,3, > > >> > > >> 29,48,130,3,25,130,12,42,46,103,111,111,103,108, > > >> > > >> 101,46,99,111,109,130,13,42,46,97,110,100,114, > > >> > > >> 111,105,100,46,99,111,109,130,22,42,46,97,112, > > >> > > >> 112,101,110,103,105,110,101,46,103,111,111,103, > > >> > > >> 108,101,46,99,111,109,130,18,42,46,99,108,111, > > >> 117,100,46,103,111,111,103, > > >> 108,101,46,99,111, > > >> > > >> 109,130,22,42,46,103,111,111,103,108,101,45,97, > > >> > > >> 110,97,108,121,116,105,99,115,46,99,111,109,130, > > >> > > >> 11,42,46,103,111,111,103,108,101,46,99,97,130, > > >> > > >> 11,42,46,103,111,111,103,108,101,46,99,108,130, > > >> > > >> 14,42,46,103,111,111,103,108,101,46,99,111,46, > > >> > > >> 105,110,130,14,42,46,103,111,111,103,108,101,46, > > >> > > >> 99,111,46,106,112,130,14,42,46,103,111,111,103, > > >> > > >> 108,101,46,99,111,46,117,107,130,15,42,46,103, > > >> > > >> 111,111,103,108,101,46,99,111,109,46,97,114,130, > > >> > > >> 15,42,46,103,111,111,103,108,101,46,99,111,109, > > >> > > >> 46,97,117,130,15,42,46,103,111,111,103,108,101, > > >> 46,99,111,109,46,98,114,130, > > >> 15,42,46,103,111, > > >> > > >> 111,103,108,101,46,99,111,109,46,99,111,130,15, > > >> > > >> 42,46,103,111,111,103,108,101,46,99,111,109,46, > > >> > > >> 109,120,130,15,42,46,103,111,111,103,108,101,46, > > >> > > >> 99,111,109,46,116,114,130,15,42,46,103,111,111, > > >> > > >> 103,108,101,46,99,111,109,46,118,110,130,11,42, > > >> > > >> 46,103,111,111,103,108,101,46,100,101,130,11,42, > > >> > > >> 46,103,111,111,103,108,101,46,101,115,130,11,42, > > >> > > >> 46,103,111,111,103,108,101,46,102,114,130,11,42, > > >> > > >> 46,103,111,111,103,108,101,46,104,117,130,11,42, > > >> > > >> 46,103,111,111,103,108,101,46,105,116,130,11,42, > > >> > > >> 46,103,111,111,103,108,101,46,110,108,130,11,42, > > >> > > >> 46,103,111,111,103,108,101,46,112,108,130,11,42, > > >> > > >> 46,103,111,111,103,108,101,46,112,116,130,18,42, > > >> 46,103,111,111,103,108,101,97, > > >> 100,97,112,105, > > >> > > >> 115,46,99,111,109,130,15,42,46,103,111,111,103, > > >> > > >> 108,101,97,112,105,115,46,99,110,130,20,42,46, > > >> > > >> 103,111,111,103,108,101,99,111,109,109,101,114, > > >> > > >> 99,101,46,99,111,109,130,17,42,46,103,111,111, > > >> > > >> 103,108,101,118,105,100,101,111,46,99,111,109, > > >> 130,12,42,46,103,115,116,97, > > >> 116,105,99,46,99, > > >> > > >> 110,130,13,42,46,103,115,116,97,116,105,99,46, > > >> 99,111,109,130,10,42,46,103, > > >> 118,116,49,46,99, > > >> > > >> 111,109,130,10,42,46,103,118,116,50,46,99,111, > > >> > > >> 109,130,20,42,46,109,101,116,114,105,99,46,103, > > >> > > >> 115,116,97,116,105,99,46,99,111,109,130,12,42, > > >> > > >> 46,117,114,99,104,105,110,46,99,111,109,130,16, > > >> > > >> 42,46,117,114,108,46,103,111,111,103,108,101,46, > > >> > > >> 99,111,109,130,22,42,46,121,111,117,116,117,98, > > >> > > >> 101,45,110,111,99,111,111,107,105,101,46,99,111, > > >> > > >> 109,130,13,42,46,121,111,117,116,117,98,101,46, > > >> > > >> 99,111,109,130,22,42,46,121,111,117,116,117,98, > > >> > > >> 101,101,100,117,99,97,116,105,111,110,46,99,111, > > >> > > >> 109,130,11,42,46,121,116,105,109,103,46,99,111, > > >> > > >> 109,130,11,97,110,100,114,111,105,100,46,99,111, > > >> 109,130,4,103,46,99,111,130,6, > > >> 103,111,111,46, > > >> 103,108,130,20,103,111,111, > > >> 103,108,101,45,97, > > >> > > >> 110,97,108,121,116,105,99,115,46,99,111,109,130, > > >> > > >> 10,103,111,111,103,108,101,46,99,111,109,130,18, > > >> > > >> 103,111,111,103,108,101,99,111,109,109,101,114, > > >> > > >> 99,101,46,99,111,109,130,10,117,114,99,104,105, > > >> > > >> 110,46,99,111,109,130,8,121,111,117,116,117,46, > > >> > > >> 98,101,130,11,121,111,117,116,117,98,101,46,99, > > >> > > >> 111,109,130,20,121,111,117,116,117,98,101,101, > > >> > > >> 100,117,99,97,116,105,111,110,46,99,111,109,48, > > >> > > >> 11,6,3,85,29,15,4,4,3,2,7,128,48,104,6,8,43,6,1, > > >> > > >> 5,5,7,1,1,4,92,48,90,48,43,6,8,43,6,1,5,5,7,48, > > >> > > >> 2,134,31,104,116,116,112,58,47,47,112,107,105, > > >> > > >> 46,103,111,111,103,108,101,46,99,111,109,47,71, > > >> > > >> 73,65,71,50,46,99,114,116,48,43,6,8,43,6,1,5,5, > > >> > > >> 7,48,1,134,31,104,116,116,112,58,47,47,99,108, > > >> > > >> 105,101,110,116,115,49,46,103,111,111,103,108, > > >> > > >> 101,46,99,111,109,47,111,99,115,112,48,29,6,3, > > >> > > >> 85,29,14,4,22,4,20,94,50,174,238,148,244,84,157, > > >> > > >> 175,208,233,225,119,147,54,144,110,147,101,106, > > >> 48,12,6,3,85,29,19,1>>} > > >> ** When State == hello > > >> ** Data == [{data, > > >> [{"StateData", > > >> {state,client, > > >> {#Ref<0.0.0.55>,<0.50.0>}, > > >> > > >> gen_tcp,tls_connection,tcp,tcp_closed,tcp_error, > > >> "google.com > > >> ",443,#Port<0.1310>, > > >> > > >> {ssl_options,tls, > > >> [{3,1}], > > >> verify_none, > > >> {#Fun,[]}, > > >> > > >> #Fun,false,false,undefined,1, > > >> > > >> <<>>,"***",<<>>,"***","***","***",<<>>,"***", > > >> undefined,undefined,"***","***", > > >> [<<"?\n">>, > > >> <<192,20>>, > > >> <<0,57>>, > > >> <<0,56>>, > > >> <<192,5>>, > > >> <<192,15>>, > > >> <<0,53>>, > > >> <<"?\b">>, > > >> <<192,18>>, > > >> <<0,22>>, > > >> <<0,19>>, > > >> <<192,3>>, > > >> <<"?\r">>, > > >> <<0,10>>, > > >> <<"?\t">>, > > >> <<192,19>>, > > >> <<0,51>>, > > >> <<0,50>>, > > >> <<192,4>>, > > >> <<192,14>>, > > >> <<0,47>>, > > >> <<192,7>>, > > >> <<192,17>>, > > >> <<0,5>>, > > >> <<0,4>>, > > >> <<0,21>>, > > >> <<192,2>>, > > >> <<"?\f">>, > > >> <<0,9>>], > > >> #Fun,true, > > >> 268435456,false, > > >> undefined,false,undefined, > > >> undefined,true, > > >> undefined,false}, > > >> {socket_options,binary,0,0,0,false}, > > >> "***","***","***",12306,"***",24597, > > >> ssl_session_cache, > > >> {3,1}, > > >> false,undefined, > > >> {undefined,undefined}, > > >> > > >> undefined,undefined,"***","***","***",undefined, > > >> > > >> "***","***","***",16403,#Ref<0.0.0.57>,undefined, > > >> "***", > > >> {false,first}, > > >> {<0.50.0>,#Ref<0.0.0.54>}, > > >> undefined, > > >> {[],[]}, > > >> false,true,false,false,undefined,undefined, > > >> undefined}}]}] > > >> ** Reason for termination = > > >> ** {{badmatch,<<>>}, > > >> [{ssl_handshake,dec_hello_extensions,2, > > >> [{file,"ssl_handshake.erl"},{line,1737}]}, > > >> {ssl_handshake,decode_handshake,3, > > >> [{file,"ssl_handshake.erl"},{line,926}]}, > > >> {tls_handshake,get_tls_handshake_aux,3, > > >> [{file,"tls_handshake.erl"},{line,155}]}, > > >> > > >> > {tls_connection,next_state,4,[{file,"tls_connection.erl"},{line,433}]}, > > >> {gen_fsm,handle_msg,7,[{file,"gen_fsm.erl"},{line,503}]}, > > >> {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,237}]}]} > > >> > > >> > > >> why I cannot connect to a https site? > > >> > > >> Bogdan > > >> > > >> > > >> > > >> _______________________________________________ > > >> erlang-questions mailing list > > >> erlang-questions@REDACTED > > >> http://erlang.org/mailman/listinfo/erlang-questions > > >> > > >> > > > -- > > > Lo?c Hoguin > > > http://ninenines.eu > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > > ? > Dave Cottlehuber > dch@REDACTED > +43 688 60 56 21 44 > Sent from the Cloud > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Sat Nov 29 22:30:36 2014 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Sat, 29 Nov 2014 22:30:36 +0100 Subject: [erlang-questions] [ANN] enacl v0.9.0 - NaCl/libsodium encryption for Erlang Message-ID: Hi Erlangers, NaCl is a cryptographic library by Dan J. Bernstein, Tanja Lange and Peter Schwabe. Libsodium is a portable implementation of NaCl by Frank Denis. Cryptographic systems are hard to get right. You have to beware of timing attacks in your low level primitives, so you can't use timing to attack the code. Also, even if you manage to get the primitives correct, you can still miss by using the primitives incorrectly. NaCl/libsodium avoids the first by: * Having no flow from the secret to load addresses. In particular, it avoids large lookup tables which can be attacked by cache timings. * Having no flow secret to branch conditions. This avoids timing attacks where one measures what branches were taken in the code. * Having protection against padding oracle side channel attacks. * Using a standardized RNG from the operting system. * Avoiding primitives that require randomness as much as possible and picking primitives that can get away with pseudorandomness. Furthermore, the library tries to make it hard to use cryptographic primitives incorrectly, by supplying higher-level APIs where the underlying primitives and their compositions have been pre-selected by cryptographers who knows what they are doing. While no guarantee, it does remove some typical mistakes. Finally NaCl/libsodium is very fast, even for a large security margin. In other words, it is possible to encrypt high-bandwidth traffic for protection. I've decided to implement yet another set of bindings for NaCl for Erlang as a set of C NIFs. These bindings differ from the earlier bindings in some areas: * QuickCheck is used all over the place for positive/negative testing of the APIs correctness and lack of memory leaks. * The bindings require Erlang 17.3.x and uses the experimental dirty scheduler API to avoid long-running NIFs. When called on small messages, the bindings will run the operation directly on the Erlang scheduler for speed however, and it will scale its reduction cost depending on the size of the message. * Includes a complete timing module which can be used to measure if the reduction strategy used will fit on the target machine. All in all, these bindings will be fast, and have low latency even on systems which will encrypt lots of data. They can be used to build cryptographic libraries for Erlang on top. For instance "gen_tcp_nacl". Thanks: Tony Garnock-Jones whose bindings were an inspiration. Steve Vinoski, Rickard Green and Sverker Eriksson for Dirty Schedulers https://github.com/jlouis/enacl Also see, http://nacl.cr.yp.to/ https://github.com/jedisct1/libsodium Comments and patches welcome as always. Things are written with an MIT license for maximal use. -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From achowdhury918@REDACTED Sun Nov 30 05:42:21 2014 From: achowdhury918@REDACTED (Akash Chowdhury) Date: Sat, 29 Nov 2014 23:42:21 -0500 Subject: [erlang-questions] Powermedia: pros and cons Message-ID: Hi, Has anyone used powermedia for industrial purpose? I would like to know its pros and cons for large scale system. Does it scale good? Any information will be highly appreciated. (Here is the link of powermedia : https://github.com/powermedia/pm_websphere_mq) Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ulf@REDACTED Sun Nov 30 09:23:55 2014 From: ulf@REDACTED (Ulf Wiger) Date: Sun, 30 Nov 2014 09:23:55 +0100 Subject: [erlang-questions] ANN: toker - erlang parser fun Message-ID: <67702387-C32E-4C24-B029-D37365079115@feuerlabs.com> https://github.com/uwiger/toker Pluggable parsers for the Erlang compiler This project started as an experiment with runtime code injection using the parse_trans library. (Note, 'Toker' is the Swedish name for Dopey the dwarf, but also obviously a word play refering to token transformation.) This library installs itself into the Erlang compiler, allowing modules to switch parser modules as well as install token transformers. Example: the toker_test module uses a custom syntax: -module(toker_test). -export([double/1, i2l/1]). -toker_parser(toker_erl_parse). double(L) -> lists:map(`(X) -> X*2`, L). i2l(L) -> lists:map(`integer_to_list/1, L). This module will not compile with the standard Erlang parser, but toker's own build chain bootstraps itself and installs a hook in the erl_parse module, which then detects the instruction -toker_parser(toker_erl_parse). Demonstration: Eshell V5.10.3 (abort with ^G) 1> toker_test:double([1,2,3]). [2,4,6] 2> toker_test:i2l([1,2,3]). ["1","2","3?] Contributions are welcome, although the library is mostly for fun (it?s been gathering dust for a while, and I thought I?d at least publish it). BR, Ulf W Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. http://feuerlabs.com -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 496 bytes Desc: Message signed with OpenPGP using GPGMail URL: From bchesneau@REDACTED Sun Nov 30 23:23:45 2014 From: bchesneau@REDACTED (Benoit Chesneau) Date: Sun, 30 Nov 2014 23:23:45 +0100 Subject: [erlang-questions] [ANN] hackney 1.0.0 released Message-ID: Hi all, I'm pleased to announce the first stable and long support release of Hackney, an HTTP client library for Erlang. Hackney is out since a long time and is actually used in production in many projects. This releases highlights the following features: - no message passing (except for asynchronous responses): response is directly streamed to the socket and state is kept in the requesting process. - binary streams - SSL support - Keepalive handling - fetch a response asynchronously - multipart support (streamed or not) - chunked encoding support - Can send files using the sendfile API - REST syntax: hackney:Method(URL) (where a method can be get, post, put, delete, ...) - Metrics using folsom or exometer More details about the release can be found here: https://github.com/benoitc/hackney/releases/tag/1.0.0 Documentation is available on Github: https://github.com/benoitc/hackney Hope you find it useful. Enjoy! - benoit -------------- next part -------------- An HTML attachment was scrubbed... URL: