From eric.pailleau@REDACTED Tue Jan 1 11:43:41 2019 From: eric.pailleau@REDACTED (PAILLEAU Eric) Date: Tue, 1 Jan 2019 11:43:41 +0100 Subject: [erlang-questions] display floats without -e? In-Reply-To: <2a399915-91a6-bb67-b45b-ffee0d72a14a@potatochowder.com> References: <1045210247.12216694.1546274258569.ref@mail.yahoo.com> <1045210247.12216694.1546274258569@mail.yahoo.com> <2a399915-91a6-bb67-b45b-ffee0d72a14a@potatochowder.com> Message-ID: <88f0ca3f-f0ae-545d-b535-c7904e0e9a6d@wanadoo.fr> Hi, 1> c(floaty). {ok,floaty} 2> floaty:auto(12.345555). "12.345555" 3> floaty:auto(12.3455553333333). "12.3455553333333" 4> list_to_float(floaty:auto(12.3455553333333)). 12.3455553333333 Le 31/12/2018 ? 20:31, Dan Sommers a ?crit?: > On 12/31/18 10:37 AM, Vans S wrote: > > > > 1> 0.0001. > > 0.0001 > > > 2> 0.00001. > > 1.0e-5 > > > Once floats get to the 4th/5th decimal place they start being > > displayed with the e-x notation.? Is there a way to make them always > > display "normally", for example 0.00001 in this case ... > > At some point, it's shorter and easier to read with the e-x notation. > > Consider a number like 1e-44.? The last thing I want is to have to count > all those zeros.? ;-) > > But you probably knew that. > > > ... The problem is when you turn the float into a string, its also > > displayed as 1.0e-5, adding a format function works yes, but gets > > annoying to maintain when you need to do arithmetic on the floats. > > LIke sorting by the value. > > I'm confused:? if you're doing arithmetic on the floats, why are you > concerned with the strings? > > If you're sorting the strings, then you'll probably either (a) have to > ensure a consistent format, perhaps with something like a ~24.18.0e > specifier, or (b) convert the strings back to floats and suffer various > rounding errors. > > > Also debugging becomes annoying when you see the e-5, e-6, etc > > notation everywhere and have to match it up. > > On that we agree.? :-) > > > Is there a way to compile / add arg to the VM to not do this? > > I'm pretty new at Erlang, so I'm probably missing a lot, but how are you > displaying the values now if not with a format function?? (I guess > there's io:write, but at that point, you're awfully close to calling > io:fwrite anyway.) > > Are you just using the shell to do immediate calculations?? I'm sure > that the shell has its own default formatting, or maybe it just calls > io:write. > > Dan > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- A non-text attachment was scrubbed... Name: floaty.erl Type: text/x-erlang Size: 950 bytes Desc: not available URL: From eric.pailleau@REDACTED Tue Jan 1 12:10:58 2019 From: eric.pailleau@REDACTED (PAILLEAU Eric) Date: Tue, 1 Jan 2019 12:10:58 +0100 Subject: [erlang-questions] display floats without -e? In-Reply-To: <88f0ca3f-f0ae-545d-b535-c7904e0e9a6d@wanadoo.fr> References: <1045210247.12216694.1546274258569.ref@mail.yahoo.com> <1045210247.12216694.1546274258569@mail.yahoo.com> <2a399915-91a6-bb67-b45b-ffee0d72a14a@potatochowder.com> <88f0ca3f-f0ae-545d-b535-c7904e0e9a6d@wanadoo.fr> Message-ID: Hi again, you may add also 'compact' option for unuseful zeros. 4> floaty:auto(3.4e-89). "0.000000000000000000000000000000000000000000000000000000000000000000000000000000000000000034" Warning : documentation says for float_to_list/2 option 'decimals': " If the number does not fit in the internal static buffer of 256 bytes, the function throws badarg." Le 01/01/2019 ? 11:43, PAILLEAU Eric a ?crit?: > Hi, > > > 1> c(floaty). > {ok,floaty} > 2> floaty:auto(12.345555). > "12.345555" > 3> floaty:auto(12.3455553333333). > "12.3455553333333" > 4> list_to_float(floaty:auto(12.3455553333333)). > 12.3455553333333 > > > > > Le 31/12/2018 ? 20:31, Dan Sommers a ?crit?: >> On 12/31/18 10:37 AM, Vans S wrote: >> >> ?> > 1> 0.0001. >> ?> 0.0001 >> ?> > 2> 0.00001. >> ?> 1.0e-5 >> >> ?> Once floats get to the 4th/5th decimal place they start being >> ?> displayed with the e-x notation.? Is there a way to make them always >> ?> display "normally", for example 0.00001 in this case ... >> >> At some point, it's shorter and easier to read with the e-x notation. >> >> Consider a number like 1e-44.? The last thing I want is to have to count >> all those zeros.? ;-) >> >> But you probably knew that. >> >> ?> ... The problem is when you turn the float into a string, its also >> ?> displayed as 1.0e-5, adding a format function works yes, but gets >> ?> annoying to maintain when you need to do arithmetic on the floats. >> ?> LIke sorting by the value. >> >> I'm confused:? if you're doing arithmetic on the floats, why are you >> concerned with the strings? >> >> If you're sorting the strings, then you'll probably either (a) have to >> ensure a consistent format, perhaps with something like a ~24.18.0e >> specifier, or (b) convert the strings back to floats and suffer various >> rounding errors. >> >> ?> Also debugging becomes annoying when you see the e-5, e-6, etc >> ?> notation everywhere and have to match it up. >> >> On that we agree.? :-) >> >> ?> Is there a way to compile / add arg to the VM to not do this? >> >> I'm pretty new at Erlang, so I'm probably missing a lot, but how are you >> displaying the values now if not with a format function?? (I guess >> there's io:write, but at that point, you're awfully close to calling >> io:fwrite anyway.) >> >> Are you just using the shell to do immediate calculations?? I'm sure >> that the shell has its own default formatting, or maybe it just calls >> io:write. >> >> Dan >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From raoknz@REDACTED Tue Jan 1 12:11:43 2019 From: raoknz@REDACTED (Richard O'Keefe) Date: Wed, 2 Jan 2019 00:11:43 +1300 Subject: [erlang-questions] display floats without -e? In-Reply-To: <88f0ca3f-f0ae-545d-b535-c7904e0e9a6d@wanadoo.fr> References: <1045210247.12216694.1546274258569.ref@mail.yahoo.com> <1045210247.12216694.1546274258569@mail.yahoo.com> <2a399915-91a6-bb67-b45b-ffee0d72a14a@potatochowder.com> <88f0ca3f-f0ae-545d-b535-c7904e0e9a6d@wanadoo.fr> Message-ID: You might want to do some more testing. 11> floaty:auto(12.3455555533). "12.345555553300000" It is not clear that this is something you would be happy with. I discovered this by simplifying the code to auto(F) when is_float(F) -> P = precision(abs(F), 0), erlang:float_to_list(F, [{decimals,P}]). -spec precision(float(), integer()) -> integer(). precision(A, P) -> if A == trunc(A) -> P ; true -> precision(A*10.0, P+1) end. and doing some testing of my own. I was rather relieved to discover that the original code behaved the same way. It's really not clear what the aim is here. I suspect that it is something that should be done with numbers rather than lists. On Tue, 1 Jan 2019 at 23:43, PAILLEAU Eric wrote: > Hi, > > > 1> c(floaty). > {ok,floaty} > 2> floaty:auto(12.345555). > "12.345555" > 3> floaty:auto(12.3455553333333). > "12.3455553333333" > 4> list_to_float(floaty:auto(12.3455553333333)). > 12.3455553333333 > > > > > Le 31/12/2018 ? 20:31, Dan Sommers a ?crit : > > On 12/31/18 10:37 AM, Vans S wrote: > > > > > > 1> 0.0001. > > > 0.0001 > > > > 2> 0.00001. > > > 1.0e-5 > > > > > Once floats get to the 4th/5th decimal place they start being > > > displayed with the e-x notation. Is there a way to make them always > > > display "normally", for example 0.00001 in this case ... > > > > At some point, it's shorter and easier to read with the e-x notation. > > > > Consider a number like 1e-44. The last thing I want is to have to count > > all those zeros. ;-) > > > > But you probably knew that. > > > > > ... The problem is when you turn the float into a string, its also > > > displayed as 1.0e-5, adding a format function works yes, but gets > > > annoying to maintain when you need to do arithmetic on the floats. > > > LIke sorting by the value. > > > > I'm confused: if you're doing arithmetic on the floats, why are you > > concerned with the strings? > > > > If you're sorting the strings, then you'll probably either (a) have to > > ensure a consistent format, perhaps with something like a ~24.18.0e > > specifier, or (b) convert the strings back to floats and suffer various > > rounding errors. > > > > > Also debugging becomes annoying when you see the e-5, e-6, etc > > > notation everywhere and have to match it up. > > > > On that we agree. :-) > > > > > Is there a way to compile / add arg to the VM to not do this? > > > > I'm pretty new at Erlang, so I'm probably missing a lot, but how are you > > displaying the values now if not with a format function? (I guess > > there's io:write, but at that point, you're awfully close to calling > > io:fwrite anyway.) > > > > Are you just using the shell to do immediate calculations? I'm sure > > that the shell has its own default formatting, or maybe it just calls > > io:write. > > > > Dan > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric.pailleau@REDACTED Tue Jan 1 12:17:55 2019 From: eric.pailleau@REDACTED (PAILLEAU Eric) Date: Tue, 1 Jan 2019 12:17:55 +0100 Subject: [erlang-questions] display floats without -e? In-Reply-To: References: <1045210247.12216694.1546274258569.ref@mail.yahoo.com> <1045210247.12216694.1546274258569@mail.yahoo.com> <2a399915-91a6-bb67-b45b-ffee0d72a14a@potatochowder.com> <88f0ca3f-f0ae-545d-b535-c7904e0e9a6d@wanadoo.fr> Message-ID: Hi Richard, yes by using 'compact' option of float_to_list/2 you get 1> floaty:auto(12.3455555533). "12.3455555533" cheers Le 01/01/2019 ? 12:11, Richard O'Keefe a ?crit?: > You might want to do some more testing. > > 11> floaty:auto(12.3455555533). > "12.345555553300000" > > It is not clear that this is something you would be happy with. > I discovered this by simplifying the code to > > auto(F) when is_float(F) -> > ? ?P = precision(abs(F), 0), > ? ?erlang:float_to_list(F, [{decimals,P}]). > > -spec precision(float(), integer()) -> integer(). > precision(A, P) -> > ? ? if A == trunc(A) -> P > ? ? ?; true? ? ? ? ? -> precision(A*10.0, P+1) > ? ? end. > > and doing some testing of my own.? I was rather relieved to discover > that the original code behaved the same way. > > It's really not clear what the aim is here. > I suspect that it is something that should be > done with numbers rather than lists. > > > On Tue, 1 Jan 2019 at 23:43, PAILLEAU Eric > wrote: > > Hi, > > > 1> c(floaty). > {ok,floaty} > 2> floaty:auto(12.345555). > "12.345555" > 3> floaty:auto(12.3455553333333). > "12.3455553333333" > 4> list_to_float(floaty:auto(12.3455553333333)). > 12.3455553333333 > > > > > Le 31/12/2018 ? 20:31, Dan Sommers a ?crit?: > > On 12/31/18 10:37 AM, Vans S wrote: > > > >? > > 1> 0.0001. > >? > 0.0001 > >? > > 2> 0.00001. > >? > 1.0e-5 > > > >? > Once floats get to the 4th/5th decimal place they start being > >? > displayed with the e-x notation.? Is there a way to make them > always > >? > display "normally", for example 0.00001 in this case ... > > > > At some point, it's shorter and easier to read with the e-x notation. > > > > Consider a number like 1e-44.? The last thing I want is to have > to count > > all those zeros.? ;-) > > > > But you probably knew that. > > > >? > ... The problem is when you turn the float into a string, its also > >? > displayed as 1.0e-5, adding a format function works yes, but gets > >? > annoying to maintain when you need to do arithmetic on the floats. > >? > LIke sorting by the value. > > > > I'm confused:? if you're doing arithmetic on the floats, why are you > > concerned with the strings? > > > > If you're sorting the strings, then you'll probably either (a) > have to > > ensure a consistent format, perhaps with something like a ~24.18.0e > > specifier, or (b) convert the strings back to floats and suffer > various > > rounding errors. > > > >? > Also debugging becomes annoying when you see the e-5, e-6, etc > >? > notation everywhere and have to match it up. > > > > On that we agree.? :-) > > > >? > Is there a way to compile / add arg to the VM to not do this? > > > > I'm pretty new at Erlang, so I'm probably missing a lot, but how > are you > > displaying the values now if not with a format function?? (I guess > > there's io:write, but at that point, you're awfully close to calling > > io:fwrite anyway.) > > > > Are you just using the shell to do immediate calculations?? I'm sure > > that the shell has its own default formatting, or maybe it just calls > > io:write. > > > > Dan > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From arkadiusz@REDACTED Tue Jan 1 17:51:31 2019 From: arkadiusz@REDACTED (arkadiusz@REDACTED) Date: Tue, 1 Jan 2019 17:51:31 +0100 Subject: [erlang-questions] Telemetry v0.3.0 Message-ID: Hello, I?m happy to announce that Telemetry v0.3.0 is out! This release marks the conversion from Elixir to Erlang so that all the libraries and projects on the BEAM can expose useful instrumentation data via Telemetry events. If you wish to learn more about Telemetry, see the README on GitHub (https://github.com/beam-telemetry/telemetry) or documentation on hexdocs (https://hexdocs.pm/telemetry/index.html), or just ask in this thread. Many thanks to Tristan Sloughter for his work on the rewrite. Arkadiusz -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay@REDACTED Tue Jan 1 19:58:02 2019 From: jay@REDACTED (Jay Nelson) Date: Tue, 1 Jan 2019 10:58:02 -0800 Subject: [erlang-questions] Telemetry v0.3.0 Message-ID: <1634A83B-BF8A-435A-AF6F-15D83EFABE07@duomark.com> I?m curious how this is different from gen_event (http://erlang.org/doc/man/gen_event.html ) and how you deal with the need to supervise restart of failed handlers. I guess one big difference is that you crash the caller on the first handler error, skipping the remaining ones, whereas gen_event shields the caller and removes the callback on first failure. gen_event allows both asynchronous and synchronous notifications, and also the benefit of using the sys tools and debug that gen_event brings, plus the ability of handlers to invoke hibernate to save memory under specific circumstances. I?m not sure at what point using ets over an in memory gen state is worthwhile in terms of number of callbacks. I assume that is the main point, but I?m guessing it would take 10K to 100K before it was noticeable during garbage collection. Adding handlers is bottlenecked on the gen_server the same as gen_event, so maybe there is a speedup when looking up a handler (although duplicate_bag may not be super fast depending on the key distribution). Under what circumstances did gen_event fall short, and what benefits does telemetry offer over it? Is this for many, many notifications for each event, or for many, many different event types? I can see how gen_event can get clogged up with many pending callbacks, whereas telemetry relies on the concurrency of the existing processes. So maybe backpressure is the biggest benefit overall. jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From 0x6e6562@REDACTED Wed Jan 2 11:37:47 2019 From: 0x6e6562@REDACTED (Ben Hood) Date: Wed, 2 Jan 2019 10:37:47 +0000 Subject: [erlang-questions] remsh failing silently In-Reply-To: References: Message-ID: On Sun, Dec 30, 2018 at 9:10 AM T Ty wrote: > > How are you accessing the console ? Via telnet ? Serial cable ? Possibly your terminal type is not set correctly. Some apps like putty / iterm interpret certain keystrokes for you. Real pain for emacs programmers :D I'm connecting using Putty and Mobaxterm from a Windows 7 Citrix instance - I suspect the terminal emulation settings are not the greatest and I have little control over it. However, I do have an 17.4 OTP which was compiled natively for ppc - on the same machine using the same shell this version is able to interpret Ctrl-G and bring up the user switch command. So this is why I believe there is an issue is the way that I cross compiled and statically linked ncurses on the x86 build agent. From ameretat.reith@REDACTED Wed Jan 2 11:42:07 2019 From: ameretat.reith@REDACTED (Ameretat Reith) Date: Wed, 2 Jan 2019 14:12:07 +0330 Subject: [erlang-questions] remsh failing silently In-Reply-To: References: Message-ID: Can you check $TERM and whether terminfo is known to ncurses? Can connect with TERM=xterm? -------------- next part -------------- An HTML attachment was scrubbed... URL: From 0x6e6562@REDACTED Wed Jan 2 11:43:54 2019 From: 0x6e6562@REDACTED (Ben Hood) Date: Wed, 2 Jan 2019 10:43:54 +0000 Subject: [erlang-questions] remsh failing silently In-Reply-To: References: Message-ID: On Sun, Dec 30, 2018 at 2:34 PM Mikael Pettersson wrote: > For the record, I just built OTP 17.5.6.10 natively on > powerpc64-unknown-linux-gnu, and ^G works just fine there in the > Erlang shell. > > I'd have to suspect your cross compilation setup. Could be some > ./configure test not doing the right thing in the cross compile case. That sounds quite plausible. This is the way that I configured the cross compile for ppc (I'm trying to link openssl and ncurses statically, so that these versions are independent of the package manager on the target machine). There seems to be configure options to link openssl statically, but fewer options for ncurses - does this look vaguely sane? CONFIGURE_ARGS := --host=$(TARGET_ARCH) \ --disable-hipe \ --enable-smp-support \ --enable-threads \ --enable-kernel-poll \ --with-termcap \ --with-ssl=$(OPENSSL_DIST_PATH) \ --disable-dynamic-ssl-lib \ --build=$(BUILD_ARCH) \ --enable-native-libs \ --enable-static-nifs LDFLAGS := "-L$(NCURSES_DIST_PATH)/lib -L$(OPENSSL_DIST_PATH)/lib -L/opt/at12.0/$(TARGET_ARCH)/lib64" LIBS := $(OPENSSL_STATIC_LIB) CFLAGS := "-D_GNU_SOURCE -I$(NCURSES_DIST_PATH)/include -I$(OPENSSL_DIST_PATH)/include -mcpu=$(TARGET_CPU) -mtune=$(TARGET_CPU) -O3 -pthread" $(OTP_SRC_DIR)/make/output.mk: $(SKIP_LIBS) $(NCURSES_STATIC_LIB) $(OPENSSL_STATIC_LIB) $(OTP_SRC_DIR)/$(BOOTSTRAP_BEAM) cd $(OTP_SRC_DIR) && erl_xcomp_sysroot=$(ERL_TOP) \ CC=$(CC) \ CXX=$(CXX) \ LD=$(LD) \ LDFLAGS=$(LDFLAGS) \ CFLAGS=$(CFLAGS) \ LIBS=$(LIBS) \ ./configure $(CONFIGURE_ARGS) --prefix=$(OTP_INSTALL_DIR) From 0x6e6562@REDACTED Wed Jan 2 11:50:50 2019 From: 0x6e6562@REDACTED (Ben Hood) Date: Wed, 2 Jan 2019 10:50:50 +0000 Subject: [erlang-questions] remsh failing silently In-Reply-To: References: Message-ID: On Wed, Jan 2, 2019 at 10:42 AM Ameretat Reith wrote: > > Can you check $TERM and whether terminfo is known to ncurses? Can connect with TERM=xterm? Many thanks for the tip. $TERM is set to xterm. Unfortunately running the remote shell like this TERM=xterm erl -sname b -setcookie 1 -remsh a@REDACTED Has the same behavior as without the TERM variable set. So I'm thinking this is related to the way ncurses is linked during cross compilation. From kuna.prime@REDACTED Wed Jan 2 13:34:14 2019 From: kuna.prime@REDACTED (Karlo Kuna) Date: Wed, 2 Jan 2019 13:34:14 +0100 Subject: [erlang-questions] erlang QR code generation library Message-ID: Hello, it is 2019, can anyone recommend good open source QR code generation library for erlang (to PNG)? thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikpelinux@REDACTED Wed Jan 2 14:07:14 2019 From: mikpelinux@REDACTED (Mikael Pettersson) Date: Wed, 2 Jan 2019 14:07:14 +0100 Subject: [erlang-questions] remsh failing silently In-Reply-To: References: Message-ID: On Wed, Jan 2, 2019 at 11:44 AM Ben Hood <0x6e6562@REDACTED> wrote: > > On Sun, Dec 30, 2018 at 2:34 PM Mikael Pettersson wrote: > > For the record, I just built OTP 17.5.6.10 natively on > > powerpc64-unknown-linux-gnu, and ^G works just fine there in the > > Erlang shell. > > > > I'd have to suspect your cross compilation setup. Could be some > > ./configure test not doing the right thing in the cross compile case. > > That sounds quite plausible. > > This is the way that I configured the cross compile for ppc (I'm > trying to link openssl and ncurses statically, so that these versions > are independent of the package manager on the target machine). > > There seems to be configure options to link openssl statically, but3. If you want ncurses and openssl statically linked I'd build them with --disable-shared in the first place, alternatively (temporarily) remove their .so files from the cross compiler's libs. > fewer options for ncurses - does this look vaguely sane? > > CONFIGURE_ARGS := --host=$(TARGET_ARCH) \ > --disable-hipe \ > --enable-smp-support \ > --enable-threads \ > --enable-kernel-poll \ > --with-termcap \ > --with-ssl=$(OPENSSL_DIST_PATH) \ > --disable-dynamic-ssl-lib \ > --build=$(BUILD_ARCH) \ > --enable-native-libs \ > --enable-static-nifs > LDFLAGS := "-L$(NCURSES_DIST_PATH)/lib -L$(OPENSSL_DIST_PATH)/lib > -L/opt/at12.0/$(TARGET_ARCH)/lib64" > LIBS := $(OPENSSL_STATIC_LIB)3. If you want ncurses and openssl statically linked I'd build them with --disable-shared in the first place, alternatively (temporarily) remove their .so files from the cross compiler's libs. > CFLAGS := "-D_GNU_SOURCE -I$(NCURSES_DIST_PATH)/include > -I$(OPENSSL_DIST_PATH)/include -mcpu=$(TARGET_CPU) > -mtune=$(TARGET_CPU) -O3 -pthread" > > $(OTP_SRC_DIR)/make/output.mk: $(SKIP_LIBS) $(NCURSES_STATIC_LIB) > $(OPENSSL_STATIC_LIB) $(OTP_SRC_DIR)/$(BOOTSTRAP_BEAM) > cd $(OTP_SRC_DIR) && erl_xcomp_sysroot=$(ERL_TOP) \ > CC=$(CC) \ > CXX=$(CXX) \ > LD=$(LD) \ > LDFLAGS=$(LDFLAGS) \ > CFLAGS=$(CFLAGS) \ > LIBS=$(LIBS) \ > ./configure $(CONFIGURE_ARGS) --prefix=$(OTP_INSTALL_DIR) This doesn't say how ncurses was build, but I note a few things: 1. --enable-native-libs is meaningless with --disable-hipe 2. All those overrides of LDFLAGS, LIBS, CFLAGS etc look unnecessary. If I was doing this, I'd build a self-contained cross-compiler with binutils, gcc, glibc, ncurses, and openssl in $CROSS (e.g. /opt/cross-ppc64-linux/), put $CROSS/bin first in PATH, then build OTP following the procedure in HOWTO/INSTALL-CROSS.md section "Building With configure/make Directly". 3. You might want to build ncurses and openssl with --disable-shared to ensure that no dynamic linking is attempted. 4. If you `ldd beam.smp` on the target, does it list any unexpected shared libraries? And even if say libc is dynamically linked, is it the same version as used in the cross compilation environment? From zxq9@REDACTED Wed Jan 2 14:08:21 2019 From: zxq9@REDACTED (zxq9@REDACTED) Date: Wed, 02 Jan 2019 22:08:21 +0900 Subject: [erlang-questions] erlang QR code generation library In-Reply-To: References: Message-ID: <3382732.UTLQ7gZzq9@takoyaki> On 2019?1?2???? 13?34?14? JST Karlo Kuna wrote: > Hello, > > it is 2019, can anyone recommend good open source QR code generation > library for erlang (to PNG)? I don't know of any, but I totally want to write one now. :-) -Craig From sean.hinde@REDACTED Wed Jan 2 14:20:01 2019 From: sean.hinde@REDACTED (Sean Hinde) Date: Wed, 2 Jan 2019 14:20:01 +0100 Subject: [erlang-questions] erlang QR code generation library In-Reply-To: <3382732.UTLQ7gZzq9@takoyaki> References: <3382732.UTLQ7gZzq9@takoyaki> Message-ID: <7B366D9D-E47A-4216-A807-04AA92B1A640@mac.com> > On 2 Jan 2019, at 14:08, zxq9@REDACTED wrote: > > On 2019?1?2???? 13?34?14? JST Karlo Kuna wrote: >> Hello, >> >> it is 2019, can anyone recommend good open source QR code generation >> library for erlang (to PNG)? > > I don't know of any, but I totally want to write one now. > :-) This seems like a good place to start: https://github.com/komone/qrcode A whole bunch of forks added rebar support over the years. qrcode_demo.erl includes a png encoder that seems to work perfectly well despite the disclaimer about how basic it is. I'm using it for the current day job Sean From dmkolesnikov@REDACTED Wed Jan 2 14:22:09 2019 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Wed, 2 Jan 2019 15:22:09 +0200 Subject: [erlang-questions] remsh failing silently In-Reply-To: References: Message-ID: <3E186C81-0BED-43BB-95DA-AD8B140FBB45@gmail.com> Hello, > On 2 Jan 2019, at 15.07, Mikael Pettersson wrote: > > 3. You might want to build ncurses with --disable-shared > to ensure that no dynamic linking is attempted. Recently, I?ve been fighting with same issue of cross compiling VM to alpine/musl. Unfortunately, statically linked libncurses would not help (or I hold it wrong). I?ve made a cross-compile of ncurses + openssl and VM with --disable-shared. As result, VM is failed to start due to missing terminal info database. I suspect that you need to compile ncurses with some magic? Do you happened to know it? Best Regards, Dmitry -------------- next part -------------- An HTML attachment was scrubbed... URL: From zxq9@REDACTED Wed Jan 2 14:24:38 2019 From: zxq9@REDACTED (zxq9@REDACTED) Date: Wed, 02 Jan 2019 22:24:38 +0900 Subject: [erlang-questions] erlang QR code generation library In-Reply-To: <7B366D9D-E47A-4216-A807-04AA92B1A640@mac.com> References: <3382732.UTLQ7gZzq9@takoyaki> <7B366D9D-E47A-4216-A807-04AA92B1A640@mac.com> Message-ID: <1812847.xtdgC1adrc@takoyaki> On 2019?1?2???? 14?20?01? JST you wrote: > > > On 2 Jan 2019, at 14:08, zxq9@REDACTED wrote: > > > > On 2019?1?2???? 13?34?14? JST Karlo Kuna wrote: > >> Hello, > >> > >> it is 2019, can anyone recommend good open source QR code generation > >> library for erlang (to PNG)? > > > > I don't know of any, but I totally want to write one now. > > :-) > > This seems like a good place to start: https://github.com/komone/qrcode > > A whole bunch of forks added rebar support over the years. > > qrcode_demo.erl includes a png encoder that seems to work perfectly well despite the disclaimer about how basic it is. > > I'm using it for the current day job Thank you for the reference. I'll give this a look. -Craig From 0x6e6562@REDACTED Wed Jan 2 15:51:28 2019 From: 0x6e6562@REDACTED (Ben Hood) Date: Wed, 2 Jan 2019 14:51:28 +0000 Subject: [erlang-questions] remsh failing silently In-Reply-To: References: Message-ID: Hi Mikael, Many thanks for taking the time to look into this, very much appreciated. To make it easier to see what is going on, I've pushed my Makefile here: https://github.com/0x6e6562/otp-ppc64le/blob/master/otp/Makefile This set of Makefiles first builds openssl and ncurses statically, and then attempts to cross compile OTP and link those two libraries. The cross compiler is provided by the IBM Advanced Toolchain: https://developer.ibm.com/linuxonpower/advance-toolchain/ The glibc runtime for the target machine is provided by the IBM Advanced Toolchain as well. On Wed, Jan 2, 2019 at 1:07 PM Mikael Pettersson wrote: > > There seems to be configure options to link openssl statically, but3. If you want ncurses and openssl statically linked I'd build them with --disable-shared in the first place, alternatively (temporarily) remove their .so files from the cross compiler's libs. It doesn't look like the ncurses/openssl builds are producing any *.so files. But I'll look into the disable-shared flag for good measure. ATM the ncurses build produces what seems to be a static lib: $ readelf -h ncurses/dist/lib/libncurses.a File: ncurses/dist/lib/libncurses.a(version.o) ELF Header: Magic: 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 Class: ELF64 Data: 2's complement, little endian Version: 1 (current) OS/ABI: UNIX - System V ABI Version: 0 Type: REL (Relocatable file) Machine: PowerPC64 Version: 0x1 Entry point address: 0x0 Start of program headers: 0 (bytes into file) Start of section headers: 752 (bytes into file) Flags: 0x2, abiv2 Size of this header: 64 (bytes) Size of program headers: 0 (bytes) Number of program headers: 0 Size of section headers: 64 (bytes) Number of section headers: 13 Section header string table index: 12 > This doesn't say how ncurses was build, but I note a few things: Fair point. I was trying to be too minimalistic - for reference I've pushed the Makefile(s) here: https://github.com/0x6e6562/otp-ppc64le > 1. --enable-native-libs is meaningless with --disable-hipe OK, that's good to know. I think this is left over from me trying to get HiPE compilation working with the cross compiler, but I abandoned this - walk before you can run. > 2. All those overrides of LDFLAGS, LIBS, CFLAGS etc look unnecessary. By trial and error, this was the only way I found to specify the IBM Advance Toolchain version of gcc, ld, etc. > If I was doing this, I'd build a self-contained cross-compiler with > binutils, gcc, glibc, ncurses, and openssl in $CROSS (e.g. > /opt/cross-ppc64-linux/), put $CROSS/bin first in PATH, then build OTP > following the procedure in HOWTO/INSTALL-CROSS.md section "Building > With configure/make Directly". Ah, so you build a special directory of all of the cross compiled dependencies, put the bin directory first on the PATH and then this will make sure that the subsequent OTP build picks up those specific versions of gcc etc? > 3. You might want to build ncurses and openssl with --disable-shared > to ensure that no dynamic linking is attempted. OK - I'll make this explicit in the ncurses Makefile. > 4. If you `ldd beam.smp` on the target, does it list any unexpected > shared libraries? And even if say libc is dynamically linked, is it > the same version as used in the cross compilation environment? This is the ldd output for beam.smp - not sure whether it shows anything surprising. linux-vdso64.so.1 => (0x00003fff84c30000) libutil.so.1 => /opt/at12.0/lib64/power8/libutil.so.1 (0x00003fff84c00000) libdl.so.2 => /opt/at12.0/lib64/power8/libdl.so.2 (0x00003fff84bd0000) libm.so.6 => /opt/at12.0/lib64/power8/libm.so.6 (0x00003fff84a70000) libpthread.so.0 => /opt/at12.0/lib64/power8/libpthread.so.0 (0x00003fff84a20000) libc.so.6 => /opt/at12.0/lib64/power8/libc.so.6 (0x00003fff847d0000) /opt/at12.0/lib64/ld64.so.2 => /lib64/ld64.so.2 (0x0000000051060000) I wonder if there is an executable test utility in the ncurses build to make sure the library itself is compiled properly. Ben From t@REDACTED Wed Jan 2 16:59:31 2019 From: t@REDACTED (Tristan Sloughter) Date: Wed, 02 Jan 2019 10:59:31 -0500 Subject: [erlang-questions] Telemetry v0.3.0 In-Reply-To: <1634A83B-BF8A-435A-AF6F-15D83EFABE07@duomark.com> References: <1634A83B-BF8A-435A-AF6F-15D83EFABE07@duomark.com> Message-ID: <6c17e2cf-906a-4df0-8ede-6a01b6ffcaae@www.fastmail.com> The plan for detaching on failure is to trigger an event that can be handled. It is for many different event types. Maybe some of those will be handled by something that makes use of gen_event to manage the event, but for the general case of essentially hooking into many dynamically defined events it makes more sense to not require them all to be going through another process. The individual handlers themselves can then pass off to a process to serialize through if that handler requires it. Tristan On Tue, Jan 1, 2019, at 11:58, Jay Nelson wrote: > I?m curious how this is different from gen_event (http://erlang.org/doc/man/gen_event.html) and how you deal with the need to supervise restart of failed handlers. I guess one big difference is that you crash the caller on the first handler error, skipping the remaining ones, whereas gen_event shields the caller and removes the callback on first failure. gen_event allows both asynchronous and synchronous notifications, and also the benefit of using the sys tools and debug that gen_event brings, plus the ability of handlers to invoke hibernate to save memory under specific circumstances. > > I?m not sure at what point using ets over an in memory gen state is worthwhile in terms of number of callbacks. I assume that is the main point, but I?m guessing it would take 10K to 100K before it was noticeable during garbage collection. Adding handlers is bottlenecked on the gen_server the same as gen_event, so maybe there is a speedup when looking up a handler (although duplicate_bag may not be super fast depending on the key distribution). > > Under what circumstances did gen_event fall short, and what benefits does telemetry offer over it? Is this for many, many notifications for each event, or for many, many different event types? I can see how gen_event can get clogged up with many pending callbacks, whereas telemetry relies on the concurrency of the existing processes. So maybe backpressure is the biggest benefit overall. > > jay > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From 0x6e6562@REDACTED Wed Jan 2 17:07:55 2019 From: 0x6e6562@REDACTED (Ben Hood) Date: Wed, 2 Jan 2019 16:07:55 +0000 Subject: [erlang-questions] remsh failing silently In-Reply-To: References: Message-ID: On Wed, Jan 2, 2019 at 10:42 AM Ameretat Reith wrote: > > Can you check $TERM and whether terminfo is known to ncurses? Can connect with TERM=xterm? I've managed to make some progress with terminfo. By copying the terminfo directory from the ncurses build directory to the target machine, and pointing the TERMINFO variable at this database, I was able to able to get Ctrl-G working. This also solved the remsh issue. I'm not sure what is wrong with the standard /usr/share/terminfo database from RHEL, but using this custom database makes Erlang a lot happier. I'm also not sure what the best way is to distribute the terminfo DB - I imagine most OTP installs assume the existence of the OS standard libncurses? From 0x6e6562@REDACTED Wed Jan 2 18:22:39 2019 From: 0x6e6562@REDACTED (Ben Hood) Date: Wed, 2 Jan 2019 17:22:39 +0000 Subject: [erlang-questions] remsh failing silently In-Reply-To: <3E186C81-0BED-43BB-95DA-AD8B140FBB45@gmail.com> References: <3E186C81-0BED-43BB-95DA-AD8B140FBB45@gmail.com> Message-ID: On Wed, Jan 2, 2019 at 1:22 PM Dmitry Kolesnikov wrote: > Recently, I?ve been fighting with same issue of cross compiling VM to alpine/musl. Unfortunately, statically linked libncurses would not help (or I hold it wrong). I?ve made a cross-compile of ncurses + openssl and VM with --disable-shared. As result, VM is failed to start due to missing terminal info database. I suspect that you need to compile ncurses with some magic? Do you happened to know it? What seems to work for me is to upload the terminfo database that is generated as part of the ncurses build and point the TERMINFO env variable to that database directory. This is on RHEL which seems to have some quite aged stuff installed by default. From lloyd@REDACTED Wed Jan 2 18:39:56 2019 From: lloyd@REDACTED (lloyd@REDACTED) Date: Wed, 2 Jan 2019 12:39:56 -0500 (EST) Subject: [erlang-questions] erlang QR code generation library In-Reply-To: <3382732.UTLQ7gZzq9@takoyaki> References: <3382732.UTLQ7gZzq9@takoyaki> Message-ID: <1546450796.47492763@apps.rackspace.com> Hey Craig, Haven't heard from you for awhile. Hope you and your family are well and prospering. I too would love to have QR code generation. But it's beyond me to write it. I did run across an algorithm a few months ago, but memory fails me where. Quick web search gives me these: https://www.thonky.com/qr-code-tutorial -- thorough but math heavy https://www.nayuki.io/page/qr-code-generator-library I'm still working hard on Writersgeln--- but it goes slowly. I spent many months working on the erlpress PDF library that I released some weeks ago. I've done extensive work on it since, which tightens up the code and adds several important features. It's nearly ready to release, but other obligations have pulled me away. I've also spent considerable time trying to learn the many deployment options for Writersglen. I do hope to have some kind of viewable release of WG within the next six months. All the best, Lloyd -----Original Message----- From: zxq9@REDACTED Sent: Wednesday, January 2, 2019 8:08am To: erlang-questions@REDACTED Subject: Re: [erlang-questions] erlang QR code generation library On 2019?1?2???? 13?34?14? JST Karlo Kuna wrote: > Hello, > > it is 2019, can anyone recommend good open source QR code generation > library for erlang (to PNG)? I don't know of any, but I totally want to write one now. :-) -Craig _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions From t6sn7gt@REDACTED Thu Jan 3 00:06:43 2019 From: t6sn7gt@REDACTED (Donald Steven) Date: Wed, 2 Jan 2019 18:06:43 -0500 Subject: [erlang-questions] Thanks to all re avoiding process dictionary Message-ID: An HTML attachment was scrubbed... URL: From bchesneau@REDACTED Thu Jan 3 14:52:33 2019 From: bchesneau@REDACTED (Benoit Chesneau) Date: Thu, 3 Jan 2019 14:52:33 +0100 Subject: [erlang-questions] monitoring algorithm Message-ID: Hi all, I would like to add the capacity to monitor a remote pid or node over a TCP connection and wondering if there is a description of the algorithm/protocol that allows Erlang to monitor a pid or a node. I can see the following messages to start a monitor/demonitor but then not sur what is exchanged. Is there any doc/link about it? Beno?t -------------- next part -------------- An HTML attachment was scrubbed... URL: From mjollnir.ray@REDACTED Thu Jan 3 14:52:49 2019 From: mjollnir.ray@REDACTED (Wu Ray) Date: Thu, 3 Jan 2019 21:52:49 +0800 Subject: [erlang-questions] Is there anyway to share NIF resource type among NIF modules? Message-ID: Hi, guys I have a C struct, as a NIF resource type, and want it to be shared between several NIF modules, is it possible? Best Regards -------------- next part -------------- An HTML attachment was scrubbed... URL: From otp@REDACTED Thu Jan 3 16:50:59 2019 From: otp@REDACTED (Erlang/OTP) Date: Thu, 3 Jan 2019 16:50:59 +0100 Subject: [erlang-questions] Patch Package OTP 20.3.8.16 Released Message-ID: <20190103155059.GA21305@erix.ericsson.se> Patch Package: OTP 20.3.8.16 Git Tag: OTP-20.3.8.16 Date: 2019-01-03 Trouble Report Id: OTP-15397, OTP-15487 Seq num: ERL-801, ERL-804 System: OTP Release: 20 Application: erts-9.3.3.7, ssh-4.6.9.3 Predecessor: OTP 20.3.8.15 Check out the git tag OTP-20.3.8.16, and build a full OTP system including documentation. Apply one or more applications from this build as patches to your installation using the 'otp_patch_apply' tool. For information on install requirements, see descriptions for each application version below. --------------------------------------------------------------------- --- erts-9.3.3.7 ---------------------------------------------------- --------------------------------------------------------------------- The erts-9.3.3.7 application can be applied independently of other applications on a full OTP 20 installation. --- Fixed Bugs and Malfunctions --- OTP-15487 Application(s): erts Related Id(s): ERL-804 Fixed bug in operator band of two negative operands causing erroneous result if the absolute value of one of the operands have the lowest N*W bits as zero and the other absolute value is not larger than N*W bits. N is an integer of 1 or larger and W is 32 or 64 depending on word size. Full runtime dependencies of erts-9.3.3.7: kernel-5.0, sasl-3.0.1, stdlib-3.0 --------------------------------------------------------------------- --- ssh-4.6.9.3 ----------------------------------------------------- --------------------------------------------------------------------- Note! The ssh-4.6.9.3 application can *not* be applied independently of other applications on an arbitrary OTP 20 installation. On a full OTP 20 installation, also the following runtime dependencies have to be satisfied: -- crypto-4.2 (first satisfied in OTP 20.2) -- public_key-1.5.2 (first satisfied in OTP 20.2) --- Fixed Bugs and Malfunctions --- OTP-15397 Application(s): ssh Related Id(s): ERL-801 Fixed port leakage if a ssh:daemon call failed. Full runtime dependencies of ssh-4.6.9.3: crypto-4.2, erts-6.0, kernel-3.0, public_key-1.5.2, stdlib-3.3 --------------------------------------------------------------------- --------------------------------------------------------------------- --------------------------------------------------------------------- From g@REDACTED Thu Jan 3 17:24:40 2019 From: g@REDACTED (Guilherme Andrade) Date: Thu, 3 Jan 2019 16:24:40 +0000 Subject: [erlang-questions] monitoring algorithm In-Reply-To: References: Message-ID: As to the protocol, I believe these are the messages you mention: http://erlang.org/doc/apps/erts/erl_dist_protocol.html#new-ctrlmessages-for-distrvsn-=-4--erlang-otp-r6- On Thu, 3 Jan 2019 at 13:52, Benoit Chesneau wrote: > Hi all, > > I would like to add the capacity to monitor a remote pid or node over a > TCP connection and wondering if there is a description of the > algorithm/protocol that allows Erlang to monitor a pid or a node. I can see > the following messages to start a monitor/demonitor but then not sur what > is exchanged. > > Is there any doc/link about it? > > Beno?t > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Guilherme -------------- next part -------------- An HTML attachment was scrubbed... URL: From nalundgaard@REDACTED Thu Jan 3 18:53:20 2019 From: nalundgaard@REDACTED (Nicholas Lundgaard) Date: Thu, 03 Jan 2019 12:53:20 -0500 Subject: [erlang-questions] =?utf-8?q?ERL-823=3A_SSL_cipher=5Fsuites_too_l?= =?utf-8?q?imited_when_compiling_with_OPENSSL=5FNO=5FEC=3D1?= Message-ID: <09da934e-11d0-4e82-bb80-1b327fe653e7@www.fastmail.com> Hi, I wanted to call ERL-823 (https://bugs.erlang.org/browse/ERL-823) to this list's attention. My company operates Erlang microservices in AWS on a kerl-built OTP installation on Amazon Linux (RedHat/CentOS-based), and we've encountered a serious challenge to upgrading to OTP 21: When you disable OpenSSL EC ciphers during an OTP build, which is necessary to build an OTP installation that doesn't erroneously think it has a bunch of EC ciphers that aren't built into the underlying OpenSSL, you're no longer able to connect to google.com via https (not to mention many, many other web properties, like much of AWS infrastructure). It confuses me that there is not a simpler way to align the Erlang crypto/ssl cipher support with the underlying openssl installation it's linked to, but that notwithstanding, It would be really helpful to have a flag to build OTP with support for RedHat/Fedora's EC cipher subset, or something similar to this. Thanks, ?Nicholas Lundgaard From ingela.andin@REDACTED Thu Jan 3 22:24:37 2019 From: ingela.andin@REDACTED (Ingela Andin) Date: Thu, 3 Jan 2019 22:24:37 +0100 Subject: [erlang-questions] ERL-823: SSL cipher_suites too limited when compiling with OPENSSL_NO_EC=1 In-Reply-To: <09da934e-11d0-4e82-bb80-1b327fe653e7@www.fastmail.com> References: <09da934e-11d0-4e82-bb80-1b327fe653e7@www.fastmail.com> Message-ID: Hi! This is a configuration problem I suggest solutions in the ERL-823. Regards Ingela Erlang/OTP team Den tors 3 jan. 2019 kl 21:18 skrev Nicholas Lundgaard < nalundgaard@REDACTED>: > Hi, > > I wanted to call ERL-823 (https://bugs.erlang.org/browse/ERL-823) to this > list's attention. My company operates Erlang microservices in AWS on a > kerl-built OTP installation on Amazon Linux (RedHat/CentOS-based), and > we've encountered a serious challenge to upgrading to OTP 21: When you > disable OpenSSL EC ciphers during an OTP build, which is necessary to build > an OTP installation that doesn't erroneously think it has a bunch of EC > ciphers that aren't built into the underlying OpenSSL, you're no longer > able to connect to google.com via https (not to mention many, many other > web properties, like much of AWS infrastructure). > > It confuses me that there is not a simpler way to align the Erlang > crypto/ssl cipher support with the underlying openssl installation it's > linked to, but that notwithstanding, It would be really helpful to have a > flag to build OTP with support for RedHat/Fedora's EC cipher subset, or > something similar to this. > > Thanks, > ?Nicholas Lundgaard > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From g@REDACTED Thu Jan 3 22:29:23 2019 From: g@REDACTED (Guilherme Andrade) Date: Thu, 3 Jan 2019 21:29:23 +0000 Subject: [erlang-questions] ERL-823: SSL cipher_suites too limited when compiling with OPENSSL_NO_EC=1 In-Reply-To: <09da934e-11d0-4e82-bb80-1b327fe653e7@www.fastmail.com> References: <09da934e-11d0-4e82-bb80-1b327fe653e7@www.fastmail.com> Message-ID: Hello, Some people have worked around the issue by building OpenSSL separately and statically linking it against ERTS. This does have the disadvantage of not benefiting from distro package upgrades, though. There's a tutorial that lists the appropriate steps[1]. (I know this doesn't solve your particular problem, but it might work out as an alternative in case you haven't considered it already - depending on your particular requirements.) [1]: https://github.com/lrascao/erlang-ec2-build On Thu, 3 Jan 2019 at 20:18, Nicholas Lundgaard wrote: > Hi, > > I wanted to call ERL-823 (https://bugs.erlang.org/browse/ERL-823) to this > list's attention. My company operates Erlang microservices in AWS on a > kerl-built OTP installation on Amazon Linux (RedHat/CentOS-based), and > we've encountered a serious challenge to upgrading to OTP 21: When you > disable OpenSSL EC ciphers during an OTP build, which is necessary to build > an OTP installation that doesn't erroneously think it has a bunch of EC > ciphers that aren't built into the underlying OpenSSL, you're no longer > able to connect to google.com via https (not to mention many, many other > web properties, like much of AWS infrastructure). > > It confuses me that there is not a simpler way to align the Erlang > crypto/ssl cipher support with the underlying openssl installation it's > linked to, but that notwithstanding, It would be really helpful to have a > flag to build OTP with support for RedHat/Fedora's EC cipher subset, or > something similar to this. > > Thanks, > ?Nicholas Lundgaard > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Guilherme -------------- next part -------------- An HTML attachment was scrubbed... URL: From ingela.andin@REDACTED Thu Jan 3 23:11:09 2019 From: ingela.andin@REDACTED (Ingela Andin) Date: Thu, 3 Jan 2019 23:11:09 +0100 Subject: [erlang-questions] ERL-823: SSL cipher_suites too limited when compiling with OPENSSL_NO_EC=1 In-Reply-To: References: <09da934e-11d0-4e82-bb80-1b327fe653e7@www.fastmail.com> Message-ID: I say it would be a lot easier to configure the erlang cipher suites the way you like and skip trying to tweak OpenSSL. Please see ERL382. Regards Ingela Erlang/OTP team Den tors 3 jan. 2019 kl 22:29 skrev Guilherme Andrade : > Hello, > > Some people have worked around the issue by building OpenSSL separately > and statically linking it against ERTS. This does have the disadvantage of > not benefiting from distro package upgrades, though. > > There's a tutorial that lists the appropriate steps[1]. > > (I know this doesn't solve your particular problem, but it might work out > as an alternative in case you haven't considered it already - depending on > your particular requirements.) > > [1]: https://github.com/lrascao/erlang-ec2-build > > On Thu, 3 Jan 2019 at 20:18, Nicholas Lundgaard > wrote: > >> Hi, >> >> I wanted to call ERL-823 (https://bugs.erlang.org/browse/ERL-823) to >> this list's attention. My company operates Erlang microservices in AWS on a >> kerl-built OTP installation on Amazon Linux (RedHat/CentOS-based), and >> we've encountered a serious challenge to upgrading to OTP 21: When you >> disable OpenSSL EC ciphers during an OTP build, which is necessary to build >> an OTP installation that doesn't erroneously think it has a bunch of EC >> ciphers that aren't built into the underlying OpenSSL, you're no longer >> able to connect to google.com via https (not to mention many, many other >> web properties, like much of AWS infrastructure). >> >> It confuses me that there is not a simpler way to align the Erlang >> crypto/ssl cipher support with the underlying openssl installation it's >> linked to, but that notwithstanding, It would be really helpful to have a >> flag to build OTP with support for RedHat/Fedora's EC cipher subset, or >> something similar to this. >> >> Thanks, >> ?Nicholas Lundgaard >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > > -- > Guilherme > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From g@REDACTED Fri Jan 4 00:08:22 2019 From: g@REDACTED (Guilherme Andrade) Date: Thu, 3 Jan 2019 23:08:22 +0000 Subject: [erlang-questions] ERL-823: SSL cipher_suites too limited when compiling with OPENSSL_NO_EC=1 In-Reply-To: References: <09da934e-11d0-4e82-bb80-1b327fe653e7@www.fastmail.com> Message-ID: Fair enough :-) The intention behind my suggestion to Nicholas was not to misguide, but rather to present an alternative in case everything else fails - one that has worked for me in the past, albeit for different reasons (using EC ciphers in Amazon Linux.) However, this being cryptography and me a layman in it, I'll humbly atone for my heresy in case I've given terrible advice (no sarcasm, truly.) On Thu, 3 Jan 2019 at 22:11, Ingela Andin wrote: > I say it would be a lot easier to configure the erlang cipher suites the > way you like and skip trying to tweak OpenSSL. Please see ERL382. > > Regards Ingela Erlang/OTP team > > Den tors 3 jan. 2019 kl 22:29 skrev Guilherme Andrade : > >> Hello, >> >> Some people have worked around the issue by building OpenSSL separately >> and statically linking it against ERTS. This does have the disadvantage of >> not benefiting from distro package upgrades, though. >> >> There's a tutorial that lists the appropriate steps[1]. >> >> (I know this doesn't solve your particular problem, but it might work out >> as an alternative in case you haven't considered it already - depending on >> your particular requirements.) >> >> [1]: https://github.com/lrascao/erlang-ec2-build >> >> On Thu, 3 Jan 2019 at 20:18, Nicholas Lundgaard >> wrote: >> >>> Hi, >>> >>> I wanted to call ERL-823 (https://bugs.erlang.org/browse/ERL-823) to >>> this list's attention. My company operates Erlang microservices in AWS on a >>> kerl-built OTP installation on Amazon Linux (RedHat/CentOS-based), and >>> we've encountered a serious challenge to upgrading to OTP 21: When you >>> disable OpenSSL EC ciphers during an OTP build, which is necessary to build >>> an OTP installation that doesn't erroneously think it has a bunch of EC >>> ciphers that aren't built into the underlying OpenSSL, you're no longer >>> able to connect to google.com via https (not to mention many, many >>> other web properties, like much of AWS infrastructure). >>> >>> It confuses me that there is not a simpler way to align the Erlang >>> crypto/ssl cipher support with the underlying openssl installation it's >>> linked to, but that notwithstanding, It would be really helpful to have a >>> flag to build OTP with support for RedHat/Fedora's EC cipher subset, or >>> something similar to this. >>> >>> Thanks, >>> ?Nicholas Lundgaard >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >> >> >> -- >> Guilherme >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > -- Guilherme -------------- next part -------------- An HTML attachment was scrubbed... URL: From g@REDACTED Fri Jan 4 00:15:52 2019 From: g@REDACTED (Guilherme Andrade) Date: Thu, 3 Jan 2019 23:15:52 +0000 Subject: [erlang-questions] ERL-823: SSL cipher_suites too limited when compiling with OPENSSL_NO_EC=1 In-Reply-To: References: <09da934e-11d0-4e82-bb80-1b327fe653e7@www.fastmail.com> Message-ID: I've in the mean time noticed Nicholas mentioned custom OpenSSL builds in the ticket, which I obviously didn't read through well enough. Long week, I apologize for the noise. On Thu, 3 Jan 2019 at 23:08, Guilherme Andrade wrote: > Fair enough :-) > > The intention behind my suggestion to Nicholas was not to misguide, but > rather to present an alternative in case everything else fails - one that > has worked for me in the past, albeit for different reasons (using EC > ciphers in Amazon Linux.) > > However, this being cryptography and me a layman in it, I'll humbly atone > for my heresy in case I've given terrible advice (no sarcasm, truly.) > > On Thu, 3 Jan 2019 at 22:11, Ingela Andin wrote: > >> I say it would be a lot easier to configure the erlang cipher suites the >> way you like and skip trying to tweak OpenSSL. Please see ERL382. >> >> Regards Ingela Erlang/OTP team >> >> Den tors 3 jan. 2019 kl 22:29 skrev Guilherme Andrade : >> >>> Hello, >>> >>> Some people have worked around the issue by building OpenSSL separately >>> and statically linking it against ERTS. This does have the disadvantage of >>> not benefiting from distro package upgrades, though. >>> >>> There's a tutorial that lists the appropriate steps[1]. >>> >>> (I know this doesn't solve your particular problem, but it might work >>> out as an alternative in case you haven't considered it already - depending >>> on your particular requirements.) >>> >>> [1]: https://github.com/lrascao/erlang-ec2-build >>> >>> On Thu, 3 Jan 2019 at 20:18, Nicholas Lundgaard >>> wrote: >>> >>>> Hi, >>>> >>>> I wanted to call ERL-823 (https://bugs.erlang.org/browse/ERL-823) to >>>> this list's attention. My company operates Erlang microservices in AWS on a >>>> kerl-built OTP installation on Amazon Linux (RedHat/CentOS-based), and >>>> we've encountered a serious challenge to upgrading to OTP 21: When you >>>> disable OpenSSL EC ciphers during an OTP build, which is necessary to build >>>> an OTP installation that doesn't erroneously think it has a bunch of EC >>>> ciphers that aren't built into the underlying OpenSSL, you're no longer >>>> able to connect to google.com via https (not to mention many, many >>>> other web properties, like much of AWS infrastructure). >>>> >>>> It confuses me that there is not a simpler way to align the Erlang >>>> crypto/ssl cipher support with the underlying openssl installation it's >>>> linked to, but that notwithstanding, It would be really helpful to have a >>>> flag to build OTP with support for RedHat/Fedora's EC cipher subset, or >>>> something similar to this. >>>> >>>> Thanks, >>>> ?Nicholas Lundgaard >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> >>> >>> >>> -- >>> Guilherme >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >> > > -- > Guilherme > -- Guilherme -------------- next part -------------- An HTML attachment was scrubbed... URL: From mononcqc@REDACTED Fri Jan 4 03:41:49 2019 From: mononcqc@REDACTED (Fred Hebert) Date: Thu, 3 Jan 2019 21:41:49 -0500 Subject: [erlang-questions] ERL-823: SSL cipher_suites too limited when compiling with OPENSSL_NO_EC=1 In-Reply-To: References: <09da934e-11d0-4e82-bb80-1b327fe653e7@www.fastmail.com> Message-ID: <20190104024148.GA74587@ferdmbp.local> On 01/03, Ingela Andin wrote: >I say it would be a lot easier to configure the erlang cipher suites the >way you like and skip trying to tweak OpenSSL. Please see ERL382. > >Regards Ingela Erlang/OTP team > Additionally, Heroku has an open source library to handle TLS servers with SNI for dispatch, which comes with some default configurations and ways to easily translate from a known SSL list of ciphers such as what you'd find at: - https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-security-policy-table.html - https://wiki.mozilla.org/Security/Server_Side_TLS#Intermediate_compatibility_.28default.29 In all of these lists, the format chosen is something like ECDHE-ECDSA-AES128-GCM-SHA256 rather than the Erlang-specific maps like #{cipher => aes_128_cbc,key_exchange => ecdhe_rsa, mac => sha256,prf => sha256} or {ecdhe_ecdsa,aes_128_gcm,null,sha256} (in OTP 20.2 and earlier) To facilitate with this, I've written and updated their lib (PR pending) to include procedures to make the conversion between both formats easy: https://github.com/heroku/snit/pull/36/files#diff-1ea45e0c99dd72cbe37b5b3f4f561c70R38 It goes a bit like this: %% Declare the list of suites you want in the order you need them OpenSSLSuites = [ "ECDHE-ECDSA-AES128-GCM-SHA256", "ECDHE-ECDSA-AES256-GCM-SHA384", "ECDHE-RSA-AES128-GCM-SHA256", ... ], %% Declare the supported TLS versions you want (you may want to drop %% tlsv1 and possibly 'tlsv1.1' as well, depending on compatibility SupportedVersions = [tlsv1, 'tlsv1.1', 'tlsv1.2'], %% Then run the following (see the source file for equivalents in %% older versions of Erlang and OTP) Suites = lists:usort(lists:append( [ssl:cipher_suites(all, Vsn) || Vsn <- SupportedVersions] )), Table = [{Str, [S]} || S <- Suites, Raw <- [ssl_cipher_format:suite(S)], Str <- [ssl_cipher_format:openssl_suite_name(Raw)], not is_map(Str)], [lists:keyfind(Suite, 1, Table) || Suite <- OpenSSLSuites]. This gives a full table of all currently supported suites with both the OpenSSL and the Erlang format, such as: [{"ECDHE-ECDSA-AES128-GCM-SHA256", [#{cipher => aes_128_gcm,key_exchange => ecdhe_ecdsa, mac => aead,prf => sha256}]}, {"ECDHE-ECDSA-AES256-GCM-SHA384", [#{cipher => aes_256_gcm,key_exchange => ecdhe_ecdsa, mac => aead,prf => sha384}]}, {"ECDHE-RSA-AES128-GCM-SHA256", [#{cipher => aes_128_gcm,key_exchange => ecdhe_rsa,mac => aead, prf => sha256}]}, ... ] Which in my opinion, makes it a lot easier to manage configurations if you don't want to carefully groom them all -- just copy a list you trust from some source online and get it applied directly. Snit uses that list and re-validates it at start time, but it would be easy to retransform it by just using lists:flatten([Map || {_Name, Map} <- Result]) for the format the Erlang ssl lib uses natively. This also has the benefit that if you have any infosec department, they'll love you for providing them a list and format they're familiar with rather than the unique internal erlang format. Regards, Fred. From ingela.andin@REDACTED Fri Jan 4 08:47:54 2019 From: ingela.andin@REDACTED (Ingela Andin) Date: Fri, 4 Jan 2019 08:47:54 +0100 Subject: [erlang-questions] ERL-823: SSL cipher_suites too limited when compiling with OPENSSL_NO_EC=1 In-Reply-To: <20190104024148.GA74587@ferdmbp.local> References: <09da934e-11d0-4e82-bb80-1b327fe653e7@www.fastmail.com> <20190104024148.GA74587@ferdmbp.local> Message-ID: Hi! Well I would not say that explicitly writing a long lists of maps is the best way. For instance if you want to add the removed rsa -key exchange suite it would be done like. add_rsa_suites(Version) -> All = ssl:cipher_suites(all, Version), Default = ssl:cipher_suites(default, Version), RSASuites = ssl:filter_cipher_suites(All,[{key_exchange, fun(rsa) -> true;(_) -> false end}]), ssl:append_cipher_suites(RSASuites, Default). >From a security perspective that would not be my recommendation though. For backward compatibility reasons just supplying the OpenSSL names will probably still work. However we do not guarantee forward compatibility for that. We plan on having some support for RFC names which are similar to OpenSSL names but more consistent. We do already have suite_to_str function but we plan on having a str_to_suite too . Regards Ingela Erlang/OTP team Den fre 4 jan. 2019 kl 03:41 skrev Fred Hebert : > On 01/03, Ingela Andin wrote: > >I say it would be a lot easier to configure the erlang cipher suites the > >way you like and skip trying to tweak OpenSSL. Please see ERL382. > > > >Regards Ingela Erlang/OTP team > > > > Additionally, Heroku has an open source library to handle TLS servers > with SNI for dispatch, which comes with some default configurations and > ways to easily translate from a known SSL list of ciphers such as what > you'd find at: > > - > https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-security-policy-table.html > - > https://wiki.mozilla.org/Security/Server_Side_TLS#Intermediate_compatibility_.28default.29 > > In all of these lists, the format chosen is something like > ECDHE-ECDSA-AES128-GCM-SHA256 rather than the Erlang-specific maps like > #{cipher => aes_128_cbc,key_exchange => ecdhe_rsa, mac => sha256,prf => > sha256} or {ecdhe_ecdsa,aes_128_gcm,null,sha256} (in OTP 20.2 and > earlier) > > To facilitate with this, I've written and updated their lib (PR pending) > to include procedures to make the conversion between both formats easy: > > https://github.com/heroku/snit/pull/36/files#diff-1ea45e0c99dd72cbe37b5b3f4f561c70R38 > > It goes a bit like this: > > %% Declare the list of suites you want in the order you need them > OpenSSLSuites = [ > "ECDHE-ECDSA-AES128-GCM-SHA256", > "ECDHE-ECDSA-AES256-GCM-SHA384", > "ECDHE-RSA-AES128-GCM-SHA256", > ... > ], > > %% Declare the supported TLS versions you want (you may want to drop > %% tlsv1 and possibly 'tlsv1.1' as well, depending on compatibility > SupportedVersions = [tlsv1, 'tlsv1.1', 'tlsv1.2'], > > %% Then run the following (see the source file for equivalents in > %% older versions of Erlang and OTP) > Suites = lists:usort(lists:append( > [ssl:cipher_suites(all, Vsn) || Vsn <- SupportedVersions] > )), > Table = [{Str, [S]} || S <- Suites, > Raw <- [ssl_cipher_format:suite(S)], > Str <- > [ssl_cipher_format:openssl_suite_name(Raw)], > not is_map(Str)], > [lists:keyfind(Suite, 1, Table) || Suite <- OpenSSLSuites]. > > This gives a full table of all currently supported suites with both the > OpenSSL and the Erlang format, such as: > > [{"ECDHE-ECDSA-AES128-GCM-SHA256", > [#{cipher => aes_128_gcm,key_exchange => ecdhe_ecdsa, > mac => aead,prf => sha256}]}, > {"ECDHE-ECDSA-AES256-GCM-SHA384", > [#{cipher => aes_256_gcm,key_exchange => ecdhe_ecdsa, > mac => aead,prf => sha384}]}, > {"ECDHE-RSA-AES128-GCM-SHA256", > [#{cipher => aes_128_gcm,key_exchange => ecdhe_rsa,mac => aead, > prf => sha256}]}, > ... > ] > > Which in my opinion, makes it a lot easier to manage configurations if > you don't want to carefully groom them all -- just copy a list you trust > from some source online and get it applied directly. Snit uses that list > and re-validates it at start time, but it would be easy to retransform > it by just using lists:flatten([Map || {_Name, Map} <- Result]) for the > format the Erlang ssl lib uses natively. > > This also has the benefit that if you have any infosec department, > they'll love you for providing them a list and format they're familiar > with rather than the unique internal erlang format. > > Regards, > Fred. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ingela.andin@REDACTED Fri Jan 4 09:23:42 2019 From: ingela.andin@REDACTED (Ingela Andin) Date: Fri, 4 Jan 2019 09:23:42 +0100 Subject: [erlang-questions] ERL-823: SSL cipher_suites too limited when compiling with OPENSSL_NO_EC=1 In-Reply-To: References: <09da934e-11d0-4e82-bb80-1b327fe653e7@www.fastmail.com> <20190104024148.GA74587@ferdmbp.local> Message-ID: Hi again! Maybe I should add that using filters where you can access each logical part of the cipher suite is a more powerful way to customize cipher suites than regular expressions over complex strings. Also see ssl User Guide http://erlang.org/doc/search/?q=ssl&x=0&y=0 section 3.2 Regards Ingela Erlang/OTP team - Ericsson AB Den fre 4 jan. 2019 kl 08:47 skrev Ingela Andin : > Hi! > > Well I would not say that explicitly writing a long lists of maps is the > best way. For instance if you want to add the removed rsa -key exchange > suite it would be done like. > > add_rsa_suites(Version) -> > > All = ssl:cipher_suites(all, Version), > > Default = ssl:cipher_suites(default, Version), > > RSASuites = ssl:filter_cipher_suites(All,[{key_exchange, fun(rsa) -> true;(_) -> false end}]), > > ssl:append_cipher_suites(RSASuites, Default). > > > From a security perspective that would not be my recommendation though. > > For backward compatibility reasons just supplying the OpenSSL names will probably still work. However we do not guarantee forward compatibility for that. > We plan on having some support for RFC names which are similar to OpenSSL names but more consistent. We do already have suite_to_str function but we plan on having a str_to_suite too . > > Regards Ingela Erlang/OTP team > > > Den fre 4 jan. 2019 kl 03:41 skrev Fred Hebert : > >> On 01/03, Ingela Andin wrote: >> >I say it would be a lot easier to configure the erlang cipher suites the >> >way you like and skip trying to tweak OpenSSL. Please see ERL382. >> > >> >Regards Ingela Erlang/OTP team >> > >> >> Additionally, Heroku has an open source library to handle TLS servers >> with SNI for dispatch, which comes with some default configurations and >> ways to easily translate from a known SSL list of ciphers such as what >> you'd find at: >> >> - >> https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-security-policy-table.html >> - >> https://wiki.mozilla.org/Security/Server_Side_TLS#Intermediate_compatibility_.28default.29 >> >> In all of these lists, the format chosen is something like >> ECDHE-ECDSA-AES128-GCM-SHA256 rather than the Erlang-specific maps like >> #{cipher => aes_128_cbc,key_exchange => ecdhe_rsa, mac => sha256,prf => >> sha256} or {ecdhe_ecdsa,aes_128_gcm,null,sha256} (in OTP 20.2 and >> earlier) >> >> To facilitate with this, I've written and updated their lib (PR pending) >> to include procedures to make the conversion between both formats easy: >> >> https://github.com/heroku/snit/pull/36/files#diff-1ea45e0c99dd72cbe37b5b3f4f561c70R38 >> >> It goes a bit like this: >> >> %% Declare the list of suites you want in the order you need them >> OpenSSLSuites = [ >> "ECDHE-ECDSA-AES128-GCM-SHA256", >> "ECDHE-ECDSA-AES256-GCM-SHA384", >> "ECDHE-RSA-AES128-GCM-SHA256", >> ... >> ], >> >> %% Declare the supported TLS versions you want (you may want to drop >> %% tlsv1 and possibly 'tlsv1.1' as well, depending on compatibility >> SupportedVersions = [tlsv1, 'tlsv1.1', 'tlsv1.2'], >> >> %% Then run the following (see the source file for equivalents in >> %% older versions of Erlang and OTP) >> Suites = lists:usort(lists:append( >> [ssl:cipher_suites(all, Vsn) || Vsn <- SupportedVersions] >> )), >> Table = [{Str, [S]} || S <- Suites, >> Raw <- [ssl_cipher_format:suite(S)], >> Str <- >> [ssl_cipher_format:openssl_suite_name(Raw)], >> not is_map(Str)], >> [lists:keyfind(Suite, 1, Table) || Suite <- OpenSSLSuites]. >> >> This gives a full table of all currently supported suites with both the >> OpenSSL and the Erlang format, such as: >> >> [{"ECDHE-ECDSA-AES128-GCM-SHA256", >> [#{cipher => aes_128_gcm,key_exchange => ecdhe_ecdsa, >> mac => aead,prf => sha256}]}, >> {"ECDHE-ECDSA-AES256-GCM-SHA384", >> [#{cipher => aes_256_gcm,key_exchange => ecdhe_ecdsa, >> mac => aead,prf => sha384}]}, >> {"ECDHE-RSA-AES128-GCM-SHA256", >> [#{cipher => aes_128_gcm,key_exchange => ecdhe_rsa,mac => aead, >> prf => sha256}]}, >> ... >> ] >> >> Which in my opinion, makes it a lot easier to manage configurations if >> you don't want to carefully groom them all -- just copy a list you trust >> from some source online and get it applied directly. Snit uses that list >> and re-validates it at start time, but it would be easy to retransform >> it by just using lists:flatten([Map || {_Name, Map} <- Result]) for the >> format the Erlang ssl lib uses natively. >> >> This also has the benefit that if you have any infosec department, >> they'll love you for providing them a list and format they're familiar >> with rather than the unique internal erlang format. >> >> Regards, >> Fred. >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From andreas.schultz@REDACTED Fri Jan 4 10:38:18 2019 From: andreas.schultz@REDACTED (Andreas Schultz) Date: Fri, 4 Jan 2019 10:38:18 +0100 Subject: [erlang-questions] ERL-823: SSL cipher_suites too limited when compiling with OPENSSL_NO_EC=1 In-Reply-To: <09da934e-11d0-4e82-bb80-1b327fe653e7@www.fastmail.com> References: <09da934e-11d0-4e82-bb80-1b327fe653e7@www.fastmail.com> Message-ID: Nicholas Lundgaard schrieb am Do., 3. Jan. 2019 um 21:18 Uhr: > Hi, > > I wanted to call ERL-823 (https://bugs.erlang.org/browse/ERL-823) to this > list's attention. My company operates Erlang microservices in AWS on a > kerl-built OTP installation on Amazon Linux (RedHat/CentOS-based), and > we've encountered a serious challenge to upgrading to OTP 21: ECC Ciphers are supported since RHEL 7.6 (also newer CentOS versions), Amazon Linux 2018.03 also has support (`openssl ciphers -v` does contain ECDH ciphers). Maybe it is time to upgrade your systems to something more current? Regards, Andreas -- -- Dipl.-Inform. Andreas Schultz ----------------------- enabling your networks ---------------------- Travelping GmbH Phone: +49-391-81 90 99 0 Roentgenstr. 13 Fax: +49-391-81 90 99 299 39108 Magdeburg Email: info@REDACTED GERMANY Web: http://www.travelping.com Company Registration: Amtsgericht Stendal Reg No.: HRB 10578 Geschaeftsfuehrer: Holger Winkelmann VAT ID No.: DE236673780 --------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From per@REDACTED Fri Jan 4 13:07:17 2019 From: per@REDACTED (Per Hedeland) Date: Fri, 4 Jan 2019 13:07:17 +0100 Subject: [erlang-questions] ERL-823: SSL cipher_suites too limited when compiling with OPENSSL_NO_EC=1 In-Reply-To: References: <09da934e-11d0-4e82-bb80-1b327fe653e7@www.fastmail.com> Message-ID: <951eee52-6859-05c3-6713-1097b1387347@hedeland.org> On 2019-01-04 10:38, Andreas Schultz wrote: > Nicholas Lundgaard > schrieb am Do., 3. Jan. 2019 um 21:18 Uhr: > > Hi, > > I wanted to call ERL-823 (https://bugs.erlang.org/browse/ERL-823) to this list's attention. My company operates Erlang microservices in AWS on a kerl-built OTP installation on Amazon Linux > (RedHat/CentOS-based), and we've encountered a serious challenge to upgrading to OTP 21: > > > ECC Ciphers are supported since RHEL 7.6 (also newer CentOS versions), Amazon Linux 2018.03 also has support (`openssl ciphers -v` does contain ECDH ciphers). Actually I think there is some confusion here - also older versions of RHEL/CentOS(/Fedora?) support ECC cipher suites (I checked CentOS 7.2 just now) - it is only a specific *set of curves* they don't support (and probably newer versions don't either), due to concerns about potential patents. And there was actually a time when you *had* to use the brutal OPENSSL_NO_EC=1 workaround/hack to get OTP crypto to work on these systems, since crypto.c couldn't handle this "partial" exclusion of curves - depending on how you built it (excluding static linking* of libcrypto), you would get a failure either at build or run time. But that was long ago, crypto.c was fixed in this respect already for OTP 17, in https://github.com/erlang/otp/commit/8837c1be2ba, by - Andreas Schultz! Thanks for that!:-) So, if you use OTP 17 or newer: - If you build with the libcrypto shipped with RHEL/CentOS, it will "just work", without the OPENSSL_NO_EC hack. - If you for some reason build with a libcrypto that *includes* the curves that are excluded on those systems, you can still do a build that works there, *and* makes ECC suites in general available, by using OPENSSL_NO_EC2M=1 *instead* of OPENSSL_NO_EC=1. I guess the moral is that if you are given a recipe for some problem, make sure that you understand why it is needed, so you can stop using it when the actual problem is fixed...:-) *) I Am Not A Lawyer, but I will still advise at least those shipping commercial products based on Erlang/OTP to look carefully into the general legal issues with statically linking libcrypto into those products. And after having done that, to think again about the wisdom of doing the static linking specifically to be able to include algorithms that may be patented. --Per Hedeland From t6sn7gt@REDACTED Fri Jan 4 13:59:04 2019 From: t6sn7gt@REDACTED (Donald Steven) Date: Fri, 4 Jan 2019 07:59:04 -0500 Subject: [erlang-questions] Understanding error messages Message-ID: Is there a good explanation -- suitable for beginners -- of the run time error messages?? Undoubtedly they provide a wealth of useful information, but they seem so cryptic for newbies. Thanks. Don From eric.pailleau@REDACTED Fri Jan 4 15:22:41 2019 From: eric.pailleau@REDACTED (PAILLEAU Eric) Date: Fri, 4 Jan 2019 15:22:41 +0100 Subject: [erlang-questions] Understanding error messages In-Reply-To: References: Message-ID: https://learnyousomeerlang.com/errors-and-exceptions#run-time-errors cheers Le 04/01/2019 ? 13:59, Donald Steven a ?crit?: > Is there a good explanation -- suitable for beginners -- of the run time > error messages?? Undoubtedly they provide a wealth of useful > information, but they seem so cryptic for newbies. > > > Thanks. > > > Don > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From t6sn7gt@REDACTED Fri Jan 4 16:14:03 2019 From: t6sn7gt@REDACTED (Donald Steven) Date: Fri, 4 Jan 2019 10:14:03 -0500 Subject: [erlang-questions] Understanding error messages In-Reply-To: References: Message-ID: <44059133-dfc8-e018-86e5-8b4005374506@aim.com> Thanks Eric! On 1/4/2019 9.22 AM, PAILLEAU Eric wrote: > https://learnyousomeerlang.com/errors-and-exceptions#run-time-errors > > cheers > > Le 04/01/2019 ? 13:59, Donald Steven a ?crit?: >> Is there a good explanation -- suitable for beginners -- of the run >> time error messages?? Undoubtedly they provide a wealth of useful >> information, but they seem so cryptic for newbies. >> >> >> Thanks. >> >> >> Don >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From mononcqc@REDACTED Fri Jan 4 17:32:02 2019 From: mononcqc@REDACTED (Fred Hebert) Date: Fri, 4 Jan 2019 11:32:02 -0500 Subject: [erlang-questions] ERL-823: SSL cipher_suites too limited when compiling with OPENSSL_NO_EC=1 In-Reply-To: References: <09da934e-11d0-4e82-bb80-1b327fe653e7@www.fastmail.com> <20190104024148.GA74587@ferdmbp.local> Message-ID: <20190104163200.GB74587@ferdmbp.local> On 01/04, Ingela Andin wrote: >Hi again! > >Maybe I should add that using filters where you can access each logical >part of the cipher suite is a more powerful way to customize cipher suites >than regular expressions over complex strings. >Also see ssl User Guide http://erlang.org/doc/search/?q=ssl&x=0&y=0 section >3.2 > Agreed, it's more powerful. But when working with established teams and policies, having a unique format just for Erlang tends to be problematic as non-standard. In some places where I've been, if you can't get the security team to approve the list, you are not greenlit to go to prod. It's much, much simpler to work with non-erlang folks when we have a way to more easily communicate and review the lists -- mostly there may just be a list that will be adopted by all stacks, whether they're Erlang, Go, C#, ruby, or servers like nginx, and so on. At least getting the direct mapping between both can be very useful to validate filtering rules and everything else :) From ingela.andin@REDACTED Fri Jan 4 18:23:11 2019 From: ingela.andin@REDACTED (Ingela Andin) Date: Fri, 4 Jan 2019 18:23:11 +0100 Subject: [erlang-questions] ERL-823: SSL cipher_suites too limited when compiling with OPENSSL_NO_EC=1 In-Reply-To: <20190104163200.GB74587@ferdmbp.local> References: <09da934e-11d0-4e82-bb80-1b327fe653e7@www.fastmail.com> <20190104024148.GA74587@ferdmbp.local> <20190104163200.GB74587@ferdmbp.local> Message-ID: Hi Fred! Agree that a mapping functions are good. That is sort of why we like to accomplish with suite_to_str and we are working on the inverse. They will use the RFC-names that are more general and better documented. But most of the code to create the OpenSSL names exist so maybe we will expose that too as a utility function. Although we do not want to be dependent on OpenSSL configuration, we only want to have dependencies to its cryptolib. Regards Ingela Erlang/OTP team - Ericsson AB Den fre 4 jan. 2019 kl 17:32 skrev Fred Hebert : > On 01/04, Ingela Andin wrote: > >Hi again! > > > >Maybe I should add that using filters where you can access each logical > >part of the cipher suite is a more powerful way to customize cipher suites > >than regular expressions over complex strings. > >Also see ssl User Guide http://erlang.org/doc/search/?q=ssl&x=0&y=0 > section > >3.2 > > > > Agreed, it's more powerful. > > But when working with established teams and policies, having a unique > format just for Erlang tends to be problematic as non-standard. In some > places where I've been, if you can't get the security team to approve > the list, you are not greenlit to go to prod. > > It's much, much simpler to work with non-erlang folks when we have a way > to more easily communicate and review the lists -- mostly there may just > be a list that will be adopted by all stacks, whether they're Erlang, > Go, C#, ruby, or servers like nginx, and so on. > > At least getting the direct mapping between both can be very useful to > validate filtering rules and everything else :) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nalundgaard@REDACTED Fri Jan 4 18:41:16 2019 From: nalundgaard@REDACTED (Nicholas Lundgaard) Date: Fri, 04 Jan 2019 12:41:16 -0500 Subject: [erlang-questions] =?utf-8?q?ERL-823=3A_SSL_cipher=5Fsuites_too_l?= =?utf-8?q?imited_when_compiling_with_OPENSSL=5FNO=5FEC=3D1?= In-Reply-To: <951eee52-6859-05c3-6713-1097b1387347@hedeland.org> References: <09da934e-11d0-4e82-bb80-1b327fe653e7@www.fastmail.com> <951eee52-6859-05c3-6713-1097b1387347@hedeland.org> Message-ID: Thanks for your insights, all. I've looked at Erlang/OTP 21 builds from our CentOS box that were built with no OPENSSL_NO_EC* set, and also just tried one built with OPENSSL_NO_EC2M set, as advised below. It looks like it's true that these options are ideal for my company's use, and in both cases we properly get an Erlang/OTP build that has a fine set of cipher suites that work with the underlying OpenSSL library on the OS. It seems like there is still some potential problem with this setup, though: both the ssl and crypto modules claim to support EC curves that the underlying SSL library does not, in fact, support. A simple function that wraps a key-generate/sign/verify process in a try-catch can be used to verify this: VerifyCurve = fun(Curve) -> Msg = <<"hello world!">>, try {PublicKey, PrivKeyOut} = crypto:generate_key(ecdh,Curve), Signature = crypto:sign(ecdsa, sha512, Msg, [PrivKeyOut, Curve]), true = crypto:verify(ecdsa, sha512, <<"hello world!">>, Signature, [PublicKey, Curve]), ok catch Err:Reason -> {Err, Reason} end end. When I run this on ssl:eccs/0 or crypto:ec_curves/0, I get this result: 2> [{C, VerifyCurve(C)} || C <- ssl:eccs()]. [{secp521r1,ok}, {brainpoolP512r1,ok}, {brainpoolP384r1,ok}, {secp384r1,ok}, {brainpoolP256r1,ok}, {secp256k1,ok}, {secp256r1,ok}, {secp224k1,{error,badarg}}, {secp224r1,{error,badarg}}, {secp192k1,{error,badarg}}, {secp192r1,{error,badarg}}, {secp160k1,{error,badarg}}, {secp160r1,{error,badarg}}, {secp160r2,{error,badarg}}] 3> rp([{C, VerifyCurve(C)} || C <- crypto:ec_curves()]). [{secp160k1,{error,badarg}}, {secp160r1,{error,badarg}}, {secp160r2,{error,badarg}}, {secp192r1,{error,badarg}}, {secp192k1,{error,badarg}}, {secp224k1,{error,badarg}}, {secp224r1,{error,badarg}}, {secp256k1,ok}, {secp256r1,ok}, {secp384r1,ok}, {secp521r1,ok}, {prime192v1,{error,badarg}}, {prime192v2,{error,badarg}}, {prime192v3,{error,badarg}}, {prime239v1,{error,badarg}}, {prime239v2,{error,badarg}}, {prime239v3,{error,badarg}}, {prime256v1,ok}, {wtls7,{error,badarg}}, {wtls9,{error,badarg}}, {wtls12,{error,badarg}}, {brainpoolP160r1,{error,badarg}}, {brainpoolP160t1,{error,badarg}}, {brainpoolP192r1,{error,badarg}}, {brainpoolP192t1,{error,badarg}}, {brainpoolP224r1,{error,badarg}}, {brainpoolP224t1,{error,badarg}}, {brainpoolP256r1,ok}, {brainpoolP256t1,ok}, {brainpoolP320r1,ok}, {brainpoolP320t1,ok}, {brainpoolP384r1,ok}, {brainpoolP384t1,ok}, {brainpoolP512r1,ok}, {brainpoolP512t1,ok}, {secp112r1,{error,badarg}}, {secp112r2,{error,badarg}}, {secp128r1,{error,badarg}}, {secp128r2,{error,badarg}}, {wtls6,{error,badarg}}, {wtls8,{error,badarg}}] So.. this is what actually is troubling me about an OTP that is installed this way: these two modules definitely report named curves that cause runtime errors. The underlying OpenSSL library definitely reports which ones it supports, so I guess my question is why doesn't ssl/crypto leverage this? I think in point of fact, this concern isn't significant. It appears that Erlang/OTP 21 (built on RHEL/CentOS) has an SSL that works out of the box for most SSL connections, certainly to the hosts we care about (AWS services and ELBs/gateways in front of our infrastructure) and I'm guessing that's because the first 7 curves from "ssl:eccs/0" are, in fact, supported by the underlying OpenSSL library (and the list is in preference order). If I understand correctly, it would only be a problem to negotiate a connection with a peer who does not support any of the first 7 preferred curves. So, I understand that a workaround that I could implement in my HTTPS clients in Erlang is to cull the list of eccs, which can be passed explicitly when connecting (e.g., [{eccs, [secp521r1, brainpoolP512r1, brainpoolP384r1, secp384r1, brainpoolP256r1, secp256k1, secp256r1]}]). However, I feel that this begs the question, what is the reason for the ssl/crypto modules not reporting a lack of support for unsupported curves? Thanks again! I will copy-paste this comment into the ERL-823 issue as well, for the record. Regards, ?Nicholas Lundgaard On Fri, Jan 4, 2019, at 6:07 AM, Per Hedeland wrote: > On 2019-01-04 10:38, Andreas Schultz wrote: > > Nicholas Lundgaard > schrieb am Do., 3. Jan. 2019 um 21:18 Uhr: > > > > Hi, > > > > I wanted to call ERL-823 (https://bugs.erlang.org/browse/ERL-823) to this list's attention. My company operates Erlang microservices in AWS on a kerl-built OTP installation on Amazon Linux > > (RedHat/CentOS-based), and we've encountered a serious challenge to upgrading to OTP 21: > > > > > > ECC Ciphers are supported since RHEL 7.6 (also newer CentOS versions), Amazon Linux 2018.03 also has support (`openssl ciphers -v` does contain ECDH ciphers). > > Actually I think there is some confusion here - also older versions of > RHEL/CentOS(/Fedora?) support ECC cipher suites (I checked CentOS 7.2 > just now) - it is only a specific *set of curves* they don't support > (and probably newer versions don't either), due to concerns about > potential patents. > > And there was actually a time when you *had* to use the brutal > OPENSSL_NO_EC=1 workaround/hack to get OTP crypto to work on these > systems, since crypto.c couldn't handle this "partial" exclusion of > curves - depending on how you built it (excluding static linking* of > libcrypto), you would get a failure either at build or run time. > > But that was long ago, crypto.c was fixed in this respect already for > OTP 17, in https://github.com/erlang/otp/commit/8837c1be2ba, by - > Andreas Schultz! Thanks for that!:-) > > So, if you use OTP 17 or newer: > > - If you build with the libcrypto shipped with RHEL/CentOS, it will > "just work", without the OPENSSL_NO_EC hack. > > - If you for some reason build with a libcrypto that *includes* the > curves that are excluded on those systems, you can still do a build > that works there, *and* makes ECC suites in general available, by > using OPENSSL_NO_EC2M=1 *instead* of OPENSSL_NO_EC=1. > > I guess the moral is that if you are given a recipe for some problem, > make sure that you understand why it is needed, so you can stop using > it when the actual problem is fixed...:-) > > *) I Am Not A Lawyer, but I will still advise at least those shipping > commercial products based on Erlang/OTP to look carefully into the > general legal issues with statically linking libcrypto into those > products. And after having done that, to think again about the wisdom > of doing the static linking specifically to be able to include > algorithms that may be patented. > > --Per Hedeland > From per@REDACTED Sat Jan 5 00:34:24 2019 From: per@REDACTED (Per Hedeland) Date: Sat, 5 Jan 2019 00:34:24 +0100 Subject: [erlang-questions] ERL-823: SSL cipher_suites too limited when compiling with OPENSSL_NO_EC=1 In-Reply-To: References: <09da934e-11d0-4e82-bb80-1b327fe653e7@www.fastmail.com> <951eee52-6859-05c3-6713-1097b1387347@hedeland.org> Message-ID: On 2019-01-04 18:41, Nicholas Lundgaard wrote: > Thanks for your insights, all. > > I've looked at Erlang/OTP 21 builds from our CentOS box that were built with no OPENSSL_NO_EC* set, and also just tried one built with OPENSSL_NO_EC2M set, as advised below. It looks like it's true that these options are ideal for my company's use, and in both cases we properly get an Erlang/OTP build that has a fine set of cipher suites that work with the underlying OpenSSL library on the OS. > > It seems like there is still some potential problem with this setup, though: both the ssl and crypto modules claim to support EC curves that the underlying SSL library does not, in fact, support. A simple function that wraps a key-generate/sign/verify process in a try-catch can be used to verify this: Well, I'm not really an expert on this stuff, but as far as I understand Andreas' fix, the OPENSSL_NO_EC2M (which is set in the relevant OpenSSL header file on the RHEL/CentOS systems) will only cause the sect* curves to be excluded (there is also a note to this effect added in lib/crypto/doc/src/crypto.xml in the commit). And in your tests, the sect* curves are actually completely absent, which is not the case if you do the same thing in an OTP built with a "normal" OpenSSL libcrypto. So it seems the OPENSSL_NO_EC2M is working as expected for you, and that the failures you see are unrelated to this. Perhaps there is some other limitation in the OpenSSL version shipped with RHEL/CentOS that prevents them from working? No idea what that might be though... --Per From nalundgaard@REDACTED Sat Jan 5 16:03:53 2019 From: nalundgaard@REDACTED (Nicholas Lundgaard) Date: Sat, 05 Jan 2019 10:03:53 -0500 Subject: [erlang-questions] =?utf-8?q?ERL-823=3A_SSL_cipher=5Fsuites_too_l?= =?utf-8?q?imited_when_compiling_with_OPENSSL=5FNO=5FEC=3D1?= In-Reply-To: References: <09da934e-11d0-4e82-bb80-1b327fe653e7@www.fastmail.com> <951eee52-6859-05c3-6713-1097b1387347@hedeland.org> Message-ID: <6ad2ad0f-a068-49b2-908d-a5c5f54c5083@www.fastmail.com> "Well, I'm not really an expert on this stuff" That makes two of us, at least... :) You're exactly right, when I compile OTP 21 on CentOS with or without OPENSSL_NO_EC2M, in either case, the built OTP correctly leaves ec_gf2m out of the public keys and filters out the GF2m named EC curves that aren't supported in the CentOS/RHEL OpenSSL. The remaining problem on RHEL/CentOS systems is that (for whatever reason) these systems lack support for a bunch of non-GF2m curves that Erlang/OTP's crypto/ssl modules claim to support, but emit runtime exceptions when you try to use them. It seems that is because they are explicitly hard-coded into the C headers in the OTP applications, when that information could be extracted or extrapolated from the linked SSL library itself (either dynamically at runtime or during compilation). For clarity, I think this situation with these missing EC curves on RHEL/CentOS has been like this for many years with almost no change, so upgrading to a newer version of RHEL/CentOS/Amazon Linux isn't going to help. Here is the output from 'openssl ecparam -list_curves' on the latest (2018.03) Amazon Linux AMI running "OpenSSL 1.0.2k-fips 26 Jan 2017": openssl ecparam -list_curves secp256k1 : SECG curve over a 256 bit prime field secp384r1 : NIST/SECG curve over a 384 bit prime field secp521r1 : NIST/SECG curve over a 521 bit prime field prime256v1: X9.62/SECG curve over a 256 bit prime field And here is the output from 'openssl ecparam -list_curves' on a CentOS 6.9 box running "OpenSSL 1.0.1e-fips 11 Feb 2013". The older OpenSSL library omits the secp256k1 curve, but again, pretty similar output: openssl ecparam -list_curves secp384r1 : NIST/SECG curve over a 384 bit prime field secp521r1 : NIST/SECG curve over a 521 bit prime field prime256v1: X9.62/SECG curve over a 256 bit prime field Checking the supported curves in the OTP ssl/crypto applications seems like the natural way to reliably get this list correct, and IMO that is a better solution than expecting end-users in this situation to find this issue and resolve it by hand-filtering on every SSL connection. Regards, ?Nicholas Lundgaard On Fri, Jan 4, 2019, at 5:34 PM, Per Hedeland wrote: > Well, I'm not really an expert on this stuff, but as far as I > understand Andreas' fix, the OPENSSL_NO_EC2M (which is set in the > relevant OpenSSL header file on the RHEL/CentOS systems) will only > cause the sect* curves to be excluded (there is also a note to this > effect added in lib/crypto/doc/src/crypto.xml in the commit). > > And in your tests, the sect* curves are actually completely absent, > which is not the case if you do the same thing in an OTP built with a > "normal" OpenSSL libcrypto. So it seems the OPENSSL_NO_EC2M is working > as expected for you, and that the failures you see are unrelated to > this. Perhaps there is some other limitation in the OpenSSL version > shipped with RHEL/CentOS that prevents them from working? No idea what > that might be though... > > --Per > From per@REDACTED Sat Jan 5 22:39:00 2019 From: per@REDACTED (Per Hedeland) Date: Sat, 5 Jan 2019 22:39:00 +0100 Subject: [erlang-questions] ERL-823: SSL cipher_suites too limited when compiling with OPENSSL_NO_EC=1 In-Reply-To: <6ad2ad0f-a068-49b2-908d-a5c5f54c5083@www.fastmail.com> References: <09da934e-11d0-4e82-bb80-1b327fe653e7@www.fastmail.com> <951eee52-6859-05c3-6713-1097b1387347@hedeland.org> <6ad2ad0f-a068-49b2-908d-a5c5f54c5083@www.fastmail.com> Message-ID: <3abc08ae-8e1a-2b11-897d-3b8da7893897@hedeland.org> On 2019-01-05 16:03, Nicholas Lundgaard wrote: > "Well, I'm not really an expert on this stuff" > > That makes two of us, at least... :) I was very impressed by your VerifyCurve function though.:-) > You're exactly right, when I compile OTP 21 on CentOS with or without OPENSSL_NO_EC2M, in either case, the built OTP correctly leaves ec_gf2m out of the public keys and filters out the GF2m named EC curves that aren't supported in the CentOS/RHEL OpenSSL. > > The remaining problem on RHEL/CentOS systems is that (for whatever reason) these systems lack support for a bunch of non-GF2m curves that Erlang/OTP's crypto/ssl modules claim to support, but emit runtime exceptions when you try to use them. It seems that is because they are explicitly hard-coded into the C headers in the OTP applications, when that information could be extracted or extrapolated from the linked SSL library itself (either dynamically at runtime or during compilation). Traditionally the build of the crypto.c NIF module has been based on the #defines in the *OpenSSL* headers, in particular openssl/opensslconf.h, which is supposed to describe how the OpenSSL libraries were built - this is where both OPENSSL_NO_EC and OPENSSL_NO_EC2M originates (i.e. from using the 'no-ec' and 'no-ec2m' options, respectively, in the OpenSSL configure/build). I don't think there are any more specific #defines for the individual ECC curves though - and in fact there doesn't seem to be any options for that in the OpenSSL build either. But in any case, using only the OpenSSL headers at build time, while linking dynamically with libcrypto, means that the libcrypto found at run time may not match the build, and runtime checks are motivated. And I believe both OTP crypto and OTP ssl do some such checks, but maybe they aren't 100%. > For clarity, I think this situation with these missing EC curves on RHEL/CentOS has been like this for many years with almost no change, so upgrading to a newer version of RHEL/CentOS/Amazon Linux isn't going to help. Here is the output from 'openssl ecparam -list_curves' on the latest (2018.03) Amazon Linux AMI running "OpenSSL 1.0.2k-fips 26 Jan 2017": > > openssl ecparam -list_curves > secp256k1 : SECG curve over a 256 bit prime field > secp384r1 : NIST/SECG curve over a 384 bit prime field > secp521r1 : NIST/SECG curve over a 521 bit prime field > prime256v1: X9.62/SECG curve over a 256 bit prime field > > And here is the output from 'openssl ecparam -list_curves' on a CentOS 6.9 box running "OpenSSL 1.0.1e-fips 11 Feb 2013". The older OpenSSL library omits the secp256k1 curve, but again, pretty similar output: > > openssl ecparam -list_curves > secp384r1 : NIST/SECG curve over a 384 bit prime field > secp521r1 : NIST/SECG curve over a 521 bit prime field > prime256v1: X9.62/SECG curve over a 256 bit prime field I also tried the 'ecparam -list_curves' on various systems, and was pretty surprised by the result... As you can see, the list you get on CentOS is way shorter than the list of successful invocations of your VerifyCurve funcion - and with a "normal" OpenSSL-1.0.1 build, I get this: secp112r1 : SECG/WTLS curve over a 112 bit prime field secp112r2 : SECG curve over a 112 bit prime field secp128r1 : SECG curve over a 128 bit prime field secp128r2 : SECG curve over a 128 bit prime field secp160k1 : SECG curve over a 160 bit prime field secp160r1 : SECG curve over a 160 bit prime field secp160r2 : SECG/WTLS curve over a 160 bit prime field secp192k1 : SECG curve over a 192 bit prime field secp224k1 : SECG curve over a 224 bit prime field secp224r1 : NIST/SECG curve over a 224 bit prime field secp256k1 : SECG curve over a 256 bit prime field secp384r1 : NIST/SECG curve over a 384 bit prime field secp521r1 : NIST/SECG curve over a 521 bit prime field prime192v1: NIST/X9.62/SECG curve over a 192 bit prime field prime192v2: X9.62 curve over a 192 bit prime field prime192v3: X9.62 curve over a 192 bit prime field prime239v1: X9.62 curve over a 239 bit prime field prime239v2: X9.62 curve over a 239 bit prime field prime239v3: X9.62 curve over a 239 bit prime field prime256v1: X9.62/SECG curve over a 256 bit prime field sect113r1 : SECG curve over a 113 bit binary field sect113r2 : SECG curve over a 113 bit binary field sect131r1 : SECG/WTLS curve over a 131 bit binary field sect131r2 : SECG curve over a 131 bit binary field sect163k1 : NIST/SECG/WTLS curve over a 163 bit binary field sect163r1 : SECG curve over a 163 bit binary field sect163r2 : NIST/SECG curve over a 163 bit binary field sect193r1 : SECG curve over a 193 bit binary field sect193r2 : SECG curve over a 193 bit binary field sect233k1 : NIST/SECG/WTLS curve over a 233 bit binary field sect233r1 : NIST/SECG/WTLS curve over a 233 bit binary field sect239k1 : SECG curve over a 239 bit binary field sect283k1 : NIST/SECG curve over a 283 bit binary field sect283r1 : NIST/SECG curve over a 283 bit binary field sect409k1 : NIST/SECG curve over a 409 bit binary field sect409r1 : NIST/SECG curve over a 409 bit binary field sect571k1 : NIST/SECG curve over a 571 bit binary field sect571r1 : NIST/SECG curve over a 571 bit binary field c2pnb163v1: X9.62 curve over a 163 bit binary field c2pnb163v2: X9.62 curve over a 163 bit binary field c2pnb163v3: X9.62 curve over a 163 bit binary field c2pnb176v1: X9.62 curve over a 176 bit binary field c2tnb191v1: X9.62 curve over a 191 bit binary field c2tnb191v2: X9.62 curve over a 191 bit binary field c2tnb191v3: X9.62 curve over a 191 bit binary field c2pnb208w1: X9.62 curve over a 208 bit binary field c2tnb239v1: X9.62 curve over a 239 bit binary field c2tnb239v2: X9.62 curve over a 239 bit binary field c2tnb239v3: X9.62 curve over a 239 bit binary field c2pnb272w1: X9.62 curve over a 272 bit binary field c2pnb304w1: X9.62 curve over a 304 bit binary field c2tnb359v1: X9.62 curve over a 359 bit binary field c2pnb368w1: X9.62 curve over a 368 bit binary field c2tnb431r1: X9.62 curve over a 431 bit binary field wap-wsg-idm-ecid-wtls1: WTLS curve over a 113 bit binary field wap-wsg-idm-ecid-wtls3: NIST/SECG/WTLS curve over a 163 bit binary field wap-wsg-idm-ecid-wtls4: SECG curve over a 113 bit binary field wap-wsg-idm-ecid-wtls5: X9.62 curve over a 163 bit binary field wap-wsg-idm-ecid-wtls6: SECG/WTLS curve over a 112 bit prime field wap-wsg-idm-ecid-wtls7: SECG/WTLS curve over a 160 bit prime field wap-wsg-idm-ecid-wtls8: WTLS curve over a 112 bit prime field wap-wsg-idm-ecid-wtls9: WTLS curve over a 160 bit prime field wap-wsg-idm-ecid-wtls10: NIST/SECG/WTLS curve over a 233 bit binary field wap-wsg-idm-ecid-wtls11: NIST/SECG/WTLS curve over a 233 bit binary field wap-wsg-idm-ecid-wtls12: WTLS curvs over a 224 bit prime field Oakley-EC2N-3: IPSec/IKE/Oakley curve #3 over a 155 bit binary field. Not suitable for ECDSA. Questionable extension field! Oakley-EC2N-4: IPSec/IKE/Oakley curve #4 over a 185 bit binary field. Not suitable for ECDSA. Questionable extension field! I haven't done a strict comparison of this output with the results of running your VerifyCurve function on the same system, but my impression is that all curves that were successful in the latter are also included in the former. I really wonder how the CentOS libcrypto was built. The "-fips" in the version might be a clue, but I believe this only means that it was built in such a way that "FIPS mode" is *available*, and thus it shouldn't affect something like the set of "supported" ECC curves. (If "FIPS mode" was *enforced*, it wouldn't work at all before OTP 20 I believe, and this is definitely not the case.) Anyway, 'ecparam -list_curves' uses the OpenSSL function EC_get_builtin_curves(), which is documented in the OpenSSL man page with, among other info: Whilst the library can be used to create any curve using the functions described above, there are also a number of predefined curves that are available. In order to obtain a list of all of the predefined curves, call the function EC_get_builtin_curves. So I guess this isn't really the set of *supported* curves, and the OpenSSL ecparam(1) man page does say -list_curves If this options is specified ecparam will print out a list of all currently implemented EC parameters names and exit. ^^^^^^^^^^^ - and there may already be some magic going on in the OTP 'crypto' app to create additional curves "on demand". But by now I'm way out of whatever expertise I may have... --Per From tty.erlang@REDACTED Sun Jan 6 00:27:33 2019 From: tty.erlang@REDACTED (T Ty) Date: Sat, 5 Jan 2019 23:27:33 +0000 Subject: [erlang-questions] Understanding error messages In-Reply-To: References: Message-ID: Just look at the first line and the last line. One gives you the exact file and line that is causing the error, the other the reason for the error aka function clause, pattern match etc. If non of the file names on these two lines look recognizable its likely you triggered something in your gen_server. A very rough guide. :) On Fri, Jan 4, 2019 at 2:22 PM PAILLEAU Eric wrote: > https://learnyousomeerlang.com/errors-and-exceptions#run-time-errors > > cheers > > Le 04/01/2019 ? 13:59, Donald Steven a ?crit : > > Is there a good explanation -- suitable for beginners -- of the run time > > error messages? Undoubtedly they provide a wealth of useful > > information, but they seem so cryptic for newbies. > > > > > > Thanks. > > > > > > Don > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From t6sn7gt@REDACTED Sun Jan 6 00:39:19 2019 From: t6sn7gt@REDACTED (Donald Steven) Date: Sat, 5 Jan 2019 18:39:19 -0500 Subject: [erlang-questions] Understanding error messages In-Reply-To: References: Message-ID: <2133b826-a619-04a3-a54c-1b742b77c448@aim.com> An HTML attachment was scrubbed... URL: From tty.erlang@REDACTED Sun Jan 6 09:48:55 2019 From: tty.erlang@REDACTED (T Ty) Date: Sun, 6 Jan 2019 08:48:55 +0000 Subject: [erlang-questions] Understanding error messages In-Reply-To: <2133b826-a619-04a3-a54c-1b742b77c448@aim.com> References: <2133b826-a619-04a3-a54c-1b742b77c448@aim.com> Message-ID: In this particular instance the last line is a red herring. It was a rough guide after-all :) Basically *{init, do_boot,3 ....}* is indicating it failed while trying to initialize the system. The next line up shows the problem is in line 24 of *readscoretest:main/1* and that *diolib:openINPUTfile* is undefined. Usually I would ignore the non-recognizable stuff and focus on the modules and functions in my code to home in on the problem area. On Sat, Jan 5, 2019 at 11:39 PM Donald Steven wrote: > Thanks! > > Just to try a very simple run, I changed one letter (on line 24) in an > otherwise ok block of code so that a function call to an external module > would fail. Here's the error message: {"init terminating in > do_boot",{undef,[{diolib,openINPUTfile,['score.dat'],[]},{readscoretest,main,1,[{file,"readscoretest.erl"},{line,24}]},{init,start_em,1,[]},{init,do_boot,3,[]}]}} > > If I wrote a simple error message along these lines, it might say: *Line > 24, function diolib:openINPUTfile is undefined.* > > But what is all the other stuff (for example, > *{init,start_em,1,[]},{init,do_boot,3,[]}]}}*) trying to say? > > Don > > On 1/5/2019 6.27 PM, T Ty wrote: > > Just look at the first line and the last line. One gives you the exact > file and line that is causing the error, the other the reason for the error > aka function clause, pattern match etc. > > If non of the file names on these two lines look recognizable its likely > you triggered something in your gen_server. > > A very rough guide. :) > > On Fri, Jan 4, 2019 at 2:22 PM PAILLEAU Eric > wrote: > >> https://learnyousomeerlang.com/errors-and-exceptions#run-time-errors >> >> cheers >> >> Le 04/01/2019 ? 13:59, Donald Steven a ?crit : >> > Is there a good explanation -- suitable for beginners -- of the run >> time >> > error messages? Undoubtedly they provide a wealth of useful >> > information, but they seem so cryptic for newbies. >> > >> > >> > Thanks. >> > >> > >> > Don >> > >> > _______________________________________________ >> > erlang-questions mailing list >> > erlang-questions@REDACTED >> > http://erlang.org/mailman/listinfo/erlang-questions >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > _______________________________________________ > erlang-questions mailing listerlang-questions@REDACTED://erlang.org/mailman/listinfo/erlang-questions > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kuna.prime@REDACTED Sun Jan 6 15:38:06 2019 From: kuna.prime@REDACTED (Karlo Kuna) Date: Sun, 6 Jan 2019 15:38:06 +0100 Subject: [erlang-questions] erlang QR code generation library In-Reply-To: <1546450796.47492763@apps.rackspace.com> References: <3382732.UTLQ7gZzq9@takoyaki> <1546450796.47492763@apps.rackspace.com> Message-ID: thank you all, Sean, https://github.com/komone/qrcode really looks like starting point if i find anything intresting in it i will report back On Wed, Jan 2, 2019 at 6:40 PM wrote: > Hey Craig, > > Haven't heard from you for awhile. Hope you and your family are well and > prospering. > > I too would love to have QR code generation. But it's beyond me to write > it. I did run across an algorithm a few months ago, but memory fails me > where. Quick web search gives me these: > > https://www.thonky.com/qr-code-tutorial -- thorough but math heavy > https://www.nayuki.io/page/qr-code-generator-library > > I'm still working hard on Writersgeln--- but it goes slowly. I spent many > months working on the erlpress PDF library that I released some weeks ago. > I've done extensive work on it since, which tightens up the code and adds > several important features. It's nearly ready to release, but other > obligations have pulled me away. I've also spent considerable time trying > to learn the many deployment options for Writersglen. I do hope to have > some kind of viewable release of WG within the next six months. > > All the best, > > Lloyd > > > > -----Original Message----- > From: zxq9@REDACTED > Sent: Wednesday, January 2, 2019 8:08am > To: erlang-questions@REDACTED > Subject: Re: [erlang-questions] erlang QR code generation library > > On 2019?1?2???? 13?34?14? JST Karlo Kuna wrote: > > Hello, > > > > it is 2019, can anyone recommend good open source QR code generation > > library for erlang (to PNG)? > > I don't know of any, but I totally want to write one now. > :-) > > -Craig > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmkolesnikov@REDACTED Mon Jan 7 16:56:03 2019 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Mon, 7 Jan 2019 17:56:03 +0200 Subject: [erlang-questions] Infix function and user-defined operators Message-ID: <01B6187C-9731-4740-A770-532B8472AB97@gmail.com> Hello Community, Now and then, I am missing an infix notation in Erlang. There was a great discussion about this subject decade and half ago but I do not think anything is changed [1]. I am testing and stretching limits of functional abstracts in Erlang as part of my pure functional and generic programming exercises [2]. I?ve decided to experiment with Infix function via parse-transform. The parse_transform feature implements a syntax sugar for infix, which is compiled into valid Erlang code using corresponding function at compile-time. To apply a function using infix notation, we enclose its name in slash characters (/). This allows us to use any arity 2 functions as infix operators. ``` -compile({parse_transform, infix}). 1 /plus/ 1. F /lists:map/ [1, 2, 3]. ``` Right now, I am working on this feature here https://github.com/fogfish/datum/pull/64 , you can find more detailed description there, code, etc. I would appreciate for any feedback, suggestion from the community. Does it sounds any usable, still desired or total braindead? What other use-case/applications you might imagine for infix notation? Best Regards, Dmitry References 1. http://erlang.org/pipermail/erlang-questions/2004-March/011929.html 2. https://github.com/fogfish/datum -------------- next part -------------- An HTML attachment was scrubbed... URL: From hans.r.nilsson@REDACTED Mon Jan 7 18:36:22 2019 From: hans.r.nilsson@REDACTED (Hans Nilsson R) Date: Mon, 7 Jan 2019 17:36:22 +0000 Subject: [erlang-questions] ERL-823: SSL cipher_suites too limited when compiling with OPENSSL_NO_EC=1 In-Reply-To: <09da934e-11d0-4e82-bb80-1b327fe653e7@www.fastmail.com> References: <09da934e-11d0-4e82-bb80-1b327fe653e7@www.fastmail.com> Message-ID: Hi, I've made a PR with #ifdefs for the EC names in crypto.c: https://github.com/erlang/otp/pull/2085 The C-function algorithms() uses this for the 'curves' value in crypto:supports/0 which both SSL and SSH is supposed to use. Comments? /Hans On 1/3/19 6:53 PM, Nicholas Lundgaard wrote: > Hi, > > I wanted to call ERL-823 (https://bugs.erlang.org/browse/ERL-823) to this list's attention. My company operates Erlang microservices in AWS on a kerl-built OTP installation on Amazon Linux (RedHat/CentOS-based), and we've encountered a serious challenge to upgrading to OTP 21: When you disable OpenSSL EC ciphers during an OTP build, which is necessary to build an OTP installation that doesn't erroneously think it has a bunch of EC ciphers that aren't built into the underlying OpenSSL, you're no longer able to connect to google.com via https (not to mention many, many other web properties, like much of AWS infrastructure). > > It confuses me that there is not a simpler way to align the Erlang crypto/ssl cipher support with the underlying openssl installation it's linked to, but that notwithstanding, It would be really helpful to have a flag to build OTP with support for RedHat/Fedora's EC cipher subset, or something similar to this. > > Thanks, > ?Nicholas Lundgaard > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4161 bytes Desc: S/MIME Cryptographic Signature URL: From per@REDACTED Mon Jan 7 23:49:38 2019 From: per@REDACTED (Per Hedeland) Date: Mon, 7 Jan 2019 23:49:38 +0100 Subject: [erlang-questions] ERL-823: SSL cipher_suites too limited when compiling with OPENSSL_NO_EC=1 In-Reply-To: References: <09da934e-11d0-4e82-bb80-1b327fe653e7@www.fastmail.com> Message-ID: <1ffd5a62-173a-2f08-96c5-24f725f0f5ea@hedeland.org> On 2019-01-07 18:36, Hans Nilsson R wrote: > Hi, > > I've made a PR with #ifdefs for the EC names in crypto.c: > https://github.com/erlang/otp/pull/2085 > > The C-function algorithms() uses this for the 'curves' value in crypto:supports/0 which both SSL and SSH is supposed to use. > > Comments? Hm, is the idea that those NID_xxx #defines in the OpenSSL headers will reflect what the library supports? I don't think that is the case, but I guess Nicholas can check if his CentOS systems have a different set of #defines (in /usr/include/openssl/obj_mac.h) from a what a "normal" build of the same OpenSSL version has. (I can atm only check a CentOS "live-cd", which doesn't have the headers - for some reason my VirtualBox install always hangs...) --Per > /Hans > > On 1/3/19 6:53 PM, Nicholas Lundgaard wrote: >> Hi, >> >> I wanted to call ERL-823 (https://bugs.erlang.org/browse/ERL-823) to this list's attention. My company operates Erlang microservices in AWS on a kerl-built OTP installation on Amazon Linux (RedHat/CentOS-based), and we've encountered a serious challenge to upgrading to OTP 21: When you disable OpenSSL EC ciphers during an OTP build, which is necessary to build an OTP installation that doesn't erroneously think it has a bunch of EC ciphers that aren't built into the underlying OpenSSL, you're no longer able to connect to google.com via https (not to mention many, many other web properties, like much of AWS infrastructure). >> >> It confuses me that there is not a simpler way to align the Erlang crypto/ssl cipher support with the underlying openssl installation it's linked to, but that notwithstanding, It would be really helpful to have a flag to build OTP with support for RedHat/Fedora's EC cipher subset, or something similar to this. >> >> Thanks, >> Nicholas Lundgaard >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From per@REDACTED Tue Jan 8 13:16:03 2019 From: per@REDACTED (Per Hedeland) Date: Tue, 8 Jan 2019 13:16:03 +0100 Subject: [erlang-questions] ERL-823: SSL cipher_suites too limited when compiling with OPENSSL_NO_EC=1 In-Reply-To: <1ffd5a62-173a-2f08-96c5-24f725f0f5ea@hedeland.org> References: <09da934e-11d0-4e82-bb80-1b327fe653e7@www.fastmail.com> <1ffd5a62-173a-2f08-96c5-24f725f0f5ea@hedeland.org> Message-ID: <6b94f616-c215-c63b-7295-aadac019d5a4@hedeland.org> I fixed the VirtualBox problem (again:-(), and could thus do some testing on CentOS 7.6 with headers installed. Using OTP 21.2 and Nicholas' VerifyCurve function - first a "normal" OpenSSL build, i.e. straight from openssl.org source, no modifications or algorithm-related config options: $ openssl version OpenSSL 1.0.2k 26 Jan 2017 $ openssl ecparam -list_curves | grep ':' | wc -l 81 2> length(ssl:eccs()). 28 3> length([C || C <- ssl:eccs(), VerifyCurve(C) == ok]). 28 4> length(crypto:ec_curves()). 83 5> length([C || C <- crypto:ec_curves(), VerifyCurve(C) == ok]). 81 (I believe the diff here, due to {ipsec3,{error,badkey}}, {ipsec4,{error,badkey}}] may be caused by improper usage rather than lack of support. Those curves are also not listed by 'ecparam -list_curves'.) - and then CentOS: $ openssl version OpenSSL 1.0.2k-fips 26 Jan 2017 $ openssl ecparam -list_curves | grep ':' | wc -l 4 2> length(ssl:eccs()). 14 3> length([C || C <- ssl:eccs(), VerifyCurve(C) == ok]). 7 4> length(crypto:ec_curves()). 41 5> length([C || C <- crypto:ec_curves(), VerifyCurve(C) == ok]). 13 And, the installed openssl/obj_mac.h header file is identical for the two, and has all the NID_xxx #defines that your PR adds #ifdefs for - thus I'm afraid it makes no difference at all for *this* problem (yes, I did test it:-). --Per On 2019-01-07 23:49, Per Hedeland wrote: > On 2019-01-07 18:36, Hans Nilsson R wrote: >> Hi, >> >> I've made a PR with #ifdefs for the EC names in crypto.c: >> https://github.com/erlang/otp/pull/2085 >> >> The C-function algorithms() uses this for the 'curves' value in crypto:supports/0 which both SSL and SSH is supposed to use. >> >> Comments? > > Hm, is the idea that those NID_xxx #defines in the OpenSSL headers > will reflect what the library supports? I don't think that is the > case, but I guess Nicholas can check if his CentOS systems have a > different set of #defines (in /usr/include/openssl/obj_mac.h) from a > what a "normal" build of the same OpenSSL version has. (I can atm only > check a CentOS "live-cd", which doesn't have the headers - for some > reason my VirtualBox install always hangs...) > > --Per > >> /Hans >> >> On 1/3/19 6:53 PM, Nicholas Lundgaard wrote: >>> Hi, >>> >>> I wanted to call ERL-823 (https://bugs.erlang.org/browse/ERL-823) to this list's attention. My company operates Erlang microservices in AWS on a kerl-built OTP installation on Amazon Linux >>> (RedHat/CentOS-based), and we've encountered a serious challenge to upgrading to OTP 21: When you disable OpenSSL EC ciphers during an OTP build, which is necessary to build an OTP installation >>> that doesn't erroneously think it has a bunch of EC ciphers that aren't built into the underlying OpenSSL, you're no longer able to connect to google.com via https (not to mention many, many other >>> web properties, like much of AWS infrastructure). >>> >>> It confuses me that there is not a simpler way to align the Erlang crypto/ssl cipher support with the underlying openssl installation it's linked to, but that notwithstanding, It would be really >>> helpful to have a flag to build OTP with support for RedHat/Fedora's EC cipher subset, or something similar to this. >>> >>> Thanks, >>> Nicholas Lundgaard >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From hans.r.nilsson@REDACTED Tue Jan 8 13:51:10 2019 From: hans.r.nilsson@REDACTED (Hans Nilsson R) Date: Tue, 8 Jan 2019 12:51:10 +0000 Subject: [erlang-questions] ERL-823: SSL cipher_suites too limited when compiling with OPENSSL_NO_EC=1 In-Reply-To: <6b94f616-c215-c63b-7295-aadac019d5a4@hedeland.org> References: <09da934e-11d0-4e82-bb80-1b327fe653e7@www.fastmail.com> <1ffd5a62-173a-2f08-96c5-24f725f0f5ea@hedeland.org> <6b94f616-c215-c63b-7295-aadac019d5a4@hedeland.org> Message-ID: <9e56ba8a-a52e-623d-7a17-b5519c4e4c00@ericsson.com> Thanks Per! I'll cancel the PR. /Hans On 1/8/19 1:16 PM, Per Hedeland wrote: > I fixed the VirtualBox problem (again:-(), and could thus do some > testing on CentOS 7.6 with headers installed. Using OTP 21.2 and > Nicholas' VerifyCurve function - first a "normal" OpenSSL build, > i.e. straight from openssl.org source, no modifications or > algorithm-related config options: > > $ openssl version > OpenSSL 1.0.2k? 26 Jan 2017 > $ openssl ecparam -list_curves | grep ':' | wc -l > ????? 81 > > 2> length(ssl:eccs()). > 28 > 3> length([C || C <- ssl:eccs(), VerifyCurve(C) == ok]). > 28 > 4> length(crypto:ec_curves()). > 83 > 5> length([C || C <- crypto:ec_curves(), VerifyCurve(C) == ok]). > 81 > > (I believe the diff here, due to > ?{ipsec3,{error,badkey}}, > ?{ipsec4,{error,badkey}}] > may be caused by improper usage rather than lack of support. Those > curves are also not listed by 'ecparam -list_curves'.) > > - and then CentOS: > > $ openssl version > OpenSSL 1.0.2k-fips? 26 Jan 2017 > $ openssl ecparam -list_curves | grep ':' | wc -l > 4 > > 2> length(ssl:eccs()). > 14 > 3> length([C || C <- ssl:eccs(), VerifyCurve(C) == ok]). > 7 > 4> length(crypto:ec_curves()). > 41 > 5> length([C || C <- crypto:ec_curves(), VerifyCurve(C) == ok]). > 13 > > And, the installed openssl/obj_mac.h header file is identical for the > two, and has all the NID_xxx #defines that your PR adds #ifdefs for > - thus I'm afraid it makes no difference at all for *this* problem > (yes, I did test it:-). > > --Per > > On 2019-01-07 23:49, Per Hedeland wrote: >> On 2019-01-07 18:36, Hans Nilsson R wrote: >>> Hi, >>> >>> I've made a PR with #ifdefs for the EC names in crypto.c: >>> ????? https://github.com/erlang/otp/pull/2085 >>> >>> The C-function algorithms() uses this for the 'curves' value in crypto:supports/0 which both SSL and SSH is supposed to use. >>> >>> Comments? >> >> Hm, is the idea that those NID_xxx #defines in the OpenSSL headers >> will reflect what the library supports? I don't think that is the >> case, but I guess Nicholas can check if his CentOS systems have a >> different set of #defines (in /usr/include/openssl/obj_mac.h) from a >> what a "normal" build of the same OpenSSL version has. (I can atm only >> check a CentOS "live-cd", which doesn't have the headers - for some >> reason my VirtualBox install always hangs...) >> >> --Per >> >>> /Hans >>> >>> On 1/3/19 6:53 PM, Nicholas Lundgaard wrote: >>>> Hi, >>>> >>>> I wanted to call ERL-823 (https://bugs.erlang.org/browse/ERL-823) to this list's attention. My company operates Erlang microservices in AWS on a kerl-built OTP installation on Amazon Linux >>>> (RedHat/CentOS-based), and we've encountered a serious challenge to upgrading to OTP 21: When you disable OpenSSL EC ciphers during an OTP build, which is necessary to build an OTP installation >>>> that doesn't erroneously think it has a bunch of EC ciphers that aren't built into the underlying OpenSSL, you're no longer able to connect to google.com via https (not to mention many, many other >>>> web properties, like much of AWS infrastructure). >>>> >>>> It confuses me that there is not a simpler way to align the Erlang crypto/ssl cipher support with the underlying openssl installation it's linked to, but that notwithstanding, It would be really >>>> helpful to have a flag to build OTP with support for RedHat/Fedora's EC cipher subset, or something similar to this. >>>> >>>> Thanks, >>>> Nicholas Lundgaard >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> >>> >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > From lukas@REDACTED Tue Jan 8 14:11:00 2019 From: lukas@REDACTED (Lukas Larsson) Date: Tue, 8 Jan 2019 14:11:00 +0100 Subject: [erlang-questions] Is there anyway to share NIF resource type among NIF modules? In-Reply-To: References: Message-ID: Hello, On Thu, Jan 3, 2019 at 2:53 PM Wu Ray wrote: > I have a C struct, as a NIF resource type, and want it to be shared > between several NIF modules, is it possible? > As far as I know it is not possible. Lukas -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmytro.lytovchenko@REDACTED Tue Jan 8 14:20:10 2019 From: dmytro.lytovchenko@REDACTED (Dmytro Lytovchenko) Date: Tue, 8 Jan 2019 14:20:10 +0100 Subject: [erlang-questions] Is there anyway to share NIF resource type among NIF modules? In-Reply-To: References: Message-ID: The reasonable way would be to use a ref-counted pointer to struct which will be shared by both NIF modules. I believe it must also be allocated somewhere in VM memory in case both NIF modules get unloaded. As C is not particularly famous for its smart refc pointers, you will have to find a library or do it yourself, and hook the refc check onto both NIF resource free functions. On Tue, 8 Jan 2019 at 14:11, Lukas Larsson wrote: > Hello, > > On Thu, Jan 3, 2019 at 2:53 PM Wu Ray wrote: > >> I have a C struct, as a NIF resource type, and want it to be shared >> between several NIF modules, is it possible? >> > > As far as I know it is not possible. > > Lukas > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From otp@REDACTED Tue Jan 8 14:36:37 2019 From: otp@REDACTED (Erlang/OTP) Date: Tue, 8 Jan 2019 14:36:37 +0100 Subject: [erlang-questions] Patch Package OTP 20.3.8.17 Released Message-ID: <20190108133637.GA40762@erix.ericsson.se> Patch Package: OTP 20.3.8.17 Git Tag: OTP-20.3.8.17 Date: 2019-01-08 Trouble Report Id: OTP-15492 Seq num: ERIERL-283 System: OTP Release: 20 Application: xmerl-1.3.16.1 Predecessor: OTP 20.3.8.16 Check out the git tag OTP-20.3.8.17, and build a full OTP system including documentation. Apply one or more applications from this build as patches to your installation using the 'otp_patch_apply' tool. For information on install requirements, see descriptions for each application version below. --------------------------------------------------------------------- --- xmerl-1.3.16.1 -------------------------------------------------- --------------------------------------------------------------------- The xmerl-1.3.16.1 application can be applied independently of other applications on a full OTP 20 installation. --- Fixed Bugs and Malfunctions --- OTP-15492 Application(s): xmerl Related Id(s): ERIERL-283 The charset detection parsing crash in some cases when the XML directive is not syntactic correct. Full runtime dependencies of xmerl-1.3.16.1: erts-6.0, kernel-3.0, stdlib-2.5 --------------------------------------------------------------------- --------------------------------------------------------------------- --------------------------------------------------------------------- From raoknz@REDACTED Tue Jan 8 14:53:01 2019 From: raoknz@REDACTED (Richard O'Keefe) Date: Wed, 9 Jan 2019 02:53:01 +1300 Subject: [erlang-questions] Infix function and user-defined operators In-Reply-To: <01B6187C-9731-4740-A770-532B8472AB97@gmail.com> References: <01B6187C-9731-4740-A770-532B8472AB97@gmail.com> Message-ID: The Haskell convention for user-defined operators is `name` . I don't see any point in inventing a new one. I note that (a) http://mlton.org/InfixingOperators shows a relatively simple scheme for ML that would fit Erlang without too much strain. It starts with a warning that operator properties are not part of module interfaces. This was a problem in Prolog too, and I surmise that it may have been the reason for not having user- declared operators in Erlang. (b) In F#, the type and precedence of a user-defined operator are determined by its first character. See https://docs.microsoft.com/en-us/dotnet/fsharp/language-reference/symbol-and-operator-reference/index (c) ECMA Eiffel 8.28.5 puts all unary user-defined operators at the same precedence level and all binary user-defined operators at another precedence level. Some other Eiffel dialects followed the same rule as F#. The (b) and (c) rules appear to also be related to not having operator properties in signatures/classes/modules/whatever. More years ago than I care to admit, I wrote an extended Prolog parser that allowed user-defined "operators" like "N of L are V" or "at least N of L are V". It was based on Vaughan Pratt's CGOL; see https://en.wikipedia.org/wiki/CGOL On Tue, 8 Jan 2019 at 04:56, Dmitry Kolesnikov wrote: > Hello Community, > > Now and then, I am missing an infix notation in Erlang. There was a great > discussion about this subject decade and half ago but I do not think > anything is changed [1]. > > I am testing and stretching limits of functional abstracts in Erlang as > part of my pure functional and generic programming exercises [2]. I?ve > decided to experiment with Infix function via parse-transform. > > The parse_transform feature implements a syntax sugar for infix, which is > compiled into valid Erlang code using corresponding function at > compile-time. To apply a function using infix notation, we enclose its name > in slash characters (/). This allows us to use any arity 2 functions as > infix operators. > > ``` > -compile({parse_transform, infix}). > > 1 /plus/ 1. > > F /lists:map/ [1, 2, 3]. > ``` > > Right now, I am working on this feature here > https://github.com/fogfish/datum/pull/64, you can find more detailed > description there, code, etc. > > I would appreciate for any feedback, suggestion from the community. > Does it sounds any usable, still desired or total braindead? > What other use-case/applications you might imagine for infix notation? > > Best Regards, > Dmitry > > References > > 1. http://erlang.org/pipermail/erlang-questions/2004-March/011929.html > 2. https://github.com/fogfish/datum > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sverker@REDACTED Tue Jan 8 15:06:46 2019 From: sverker@REDACTED (Sverker Eriksson) Date: Tue, 8 Jan 2019 15:06:46 +0100 Subject: [erlang-questions] Is there anyway to share NIF resource type among NIF modules? In-Reply-To: References: Message-ID: <1546956406.12727.33.camel@erlang.org> On tor, 2019-01-03 at 21:52 +0800, Wu Ray wrote: > Hi, guys > > I have a C struct, as a NIF resource type, and want it to be shared between > several NIF modules, is it possible? > > Best Regards > There is no offical documented support to use the same resource type from more than one NIF module. But if you want to experiment, there are two problems I can think of. 1. How to get the other NIF modules access to the resource type pointer. One way to solve this is to put the resource type in a separate dynamic shared library, dynamically linked by all NIF modules using the resource type. 2. Prevent usage of an unloaded resource type.? You must make sure yourself that other modules do not try to use the pointer to an unloaded resource type ?(as argument to enif_get_resource or enif_make_resource for example). That will probably lead to VM crash or other unpleasantries. A resource type will exists as long as the NIF module that called enif_open_resource_type is loaded. If the resource type has a destructor function then the NIF module will not be unloaded until the last resource object is garbage collected. /Sverker -------------- next part -------------- An HTML attachment was scrubbed... URL: From jstimpson@REDACTED Tue Jan 8 17:13:22 2019 From: jstimpson@REDACTED (Jesse Stimpson) Date: Tue, 8 Jan 2019 11:13:22 -0500 Subject: [erlang-questions] Dirty CPU schedulers stuck at zero utilization Message-ID: Hello, We run a large scale WebRTC-based VOIP application, and currently we are evaluating whether or not dirty CPU schedulers can help us get more consistent performance with some critical NIFs. We've created a test that mimics our production load and have run it on AWS EC2 instances of various sizes. When dirty CPU schedulers are working as expected, we do see measurable improvement in performance, but unfortunately we have seen that they can drop to 0 utilization. They do not recover from this state. The timings from 2 of our 12 NIFs are in the vicinity of the 1 msec rule of thumb, so we have experimented with them both on and off dirty CPU schedulers. We've left the other NIFs on the traditional schedulers. More detailed NIF timings can be found below. At the end of this message is the concerning data (from recon:scheduler_usage/1).We have schedulers 1-36 as the traditional schedulers bound to the first hardware thread of each core, and schedulers 37-72 are dirty CPU schedulers. Based on mpstat it appears the dirty CPU schedulers are not bound to threads. After about 10 minutes of running the load test, schedulers 42, 43, 44, 61, 63, 64, 66, 68, 72 went to at 0 utilization and stayed there indefinitely. As time goes on, more dirty CPU schedulers enter this 0 utilization state. Schedulers 1-36 do not exhibit this behaviour. We have reproduced this consistently on other EC2 c5 instances of various sizes. We have tried various beam adjustments, but we have seen the same behaviour each time. Some examples of things we have tried: +sbwt very_long +sbwtdcpu very_long +swt very_low +swtdcpu very_low +sfwi 500 Is there anything else that we can try to avoid this 0-utilization issue? We are happy to provide more data if needed. NIF: webrtc_acm_add_data_10ms median: 606 usec 90 percentile: 1230 usec 99 percentile: 2285 usec 99.9 percentile: 3223 usec NIF: webrtc_acm_playout_data_10ms median: 98 usec 90 percentile: 371 usec 99 percentile: 410 usec 99.9 percentile: 449 usec EC2 instance: c5.18xlarge OS: Linux 4.4.0-1073-aws #83-Ubuntu SMP Sat Nov 17 00:26:27 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux beam flags: +P 4194304 +sbt db +SP 50:50 +sub true +A 128 +K true Erlang banner: Erlang/OTP 21 [erts-10.2] [source] [64-bit] [smp:36:36] [ds:36:36:10] [async-threads:128] [hipe] cpu_topology: [{node,[{processor,[{core,[{thread,{logical,0}}, {thread,{logical,36}}]}, {core,[{thread,{logical,1}},{thread,{logical,37}}]}, {core,[{thread,{logical,2}},{thread,{logical,38}}]}, {core,[{thread,{logical,3}},{thread,{logical,39}}]}, {core,[{thread,{logical,4}},{thread,{logical,40}}]}, {core,[{thread,{logical,5}},{thread,{logical,41}}]}, {core,[{thread,{logical,6}},{thread,{logical,42}}]}, {core,[{thread,{logical,7}},{thread,{logical,43}}]}, {core,[{thread,{logical,8}},{thread,{logical,44}}]}, {core,[{thread,{logical,9}},{thread,{logical,45}}]}, {core,[{thread,{logical,10}},{thread,{logical,46}}]}, {core,[{thread,{logical,11}},{thread,{logical,47}}]}, {core,[{thread,{logical,12}},{thread,{logical,48}}]}, {core,[{thread,{logical,13}},{thread,{logical,49}}]}, {core,[{thread,{logical,14}},{thread,{logical,50}}]}, {core,[{thread,{logical,15}},{thread,{logical,51}}]}, {core,[{thread,{logical,16}},{thread,{logical,52}}]}, {core,[{thread,{logical,17}},{thread,{logical,53}}]}]}]}, {node,[{processor,[{core,[{thread,{logical,18}}, {thread,{logical,54}}]}, {core,[{thread,{logical,19}},{thread,{logical,55}}]}, {core,[{thread,{logical,20}},{thread,{logical,56}}]}, {core,[{thread,{logical,21}},{thread,{logical,57}}]}, {core,[{thread,{logical,22}},{thread,{logical,58}}]}, {core,[{thread,{logical,23}},{thread,{logical,59}}]}, {core,[{thread,{logical,24}},{thread,{logical,60}}]}, {core,[{thread,{logical,25}},{thread,{logical,61}}]}, {core,[{thread,{logical,26}},{thread,{logical,62}}]}, {core,[{thread,{logical,27}},{thread,{logical,63}}]}, {core,[{thread,{logical,28}},{thread,{logical,64}}]}, {core,[{thread,{logical,29}},{thread,{logical,65}}]}, {core,[{thread,{logical,30}},{thread,{logical,66}}]}, {core,[{thread,{logical,31}},{thread,{logical,67}}]}, {core,[{thread,{logical,32}},{thread,{logical,68}}]}, {core,[{thread,{logical,33}},{thread,{logical,69}}]}, {core,[{thread,{logical,34}},{thread,{logical,70}}]}, {core,[{thread,{logical,35}},{thread,{logical,71}}]}]}]}] scheduler_bindings: {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22, 23,24,25,26,27,28,29,30,31,32,33,34,35} CPU utilization (from mpstat) during our load test: Linux 4.4.0-1073-aws (isw-core-x-qa-awsoh02) 01/04/2019 _x86_64_ (72 CPU) 08:55:15 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle 08:55:15 PM all 51.58 0.00 1.64 0.01 0.00 0.42 0.00 0.00 0.00 46.36 08:55:15 PM 0 62.34 0.00 2.69 0.06 0.00 0.00 0.00 0.00 0.00 34.90 08:55:15 PM 1 62.84 0.00 2.58 0.02 0.00 0.00 0.00 0.00 0.00 34.55 08:55:15 PM 2 62.04 0.00 2.52 0.00 0.00 0.00 0.00 0.00 0.00 35.44 08:55:15 PM 3 62.17 0.00 2.62 0.00 0.00 0.00 0.00 0.00 0.00 35.21 08:55:15 PM 4 62.14 0.00 2.54 0.02 0.00 0.00 0.00 0.00 0.00 35.30 08:55:15 PM 5 62.06 0.00 2.51 0.00 0.00 0.00 0.00 0.00 0.00 35.43 08:55:15 PM 6 61.93 0.00 2.58 0.01 0.00 0.00 0.02 0.00 0.00 35.46 08:55:15 PM 7 61.99 0.00 2.54 0.00 0.00 0.00 0.00 0.00 0.00 35.48 08:55:15 PM 8 62.01 0.00 2.52 0.00 0.00 0.00 0.00 0.00 0.00 35.47 08:55:15 PM 9 61.88 0.00 2.49 0.00 0.00 0.00 0.00 0.00 0.00 35.63 08:55:15 PM 10 62.18 0.00 2.48 0.00 0.00 0.00 0.00 0.00 0.00 35.34 08:55:15 PM 11 61.89 0.00 2.47 0.00 0.00 0.00 0.00 0.00 0.00 35.64 08:55:15 PM 12 61.98 0.00 2.50 0.00 0.00 0.00 0.00 0.00 0.00 35.52 08:55:15 PM 13 61.80 0.00 2.54 0.00 0.00 0.00 0.00 0.00 0.00 35.66 08:55:15 PM 14 61.96 0.00 2.56 0.00 0.00 0.00 0.00 0.00 0.00 35.48 08:55:15 PM 15 62.11 0.00 2.57 0.00 0.00 0.00 0.00 0.00 0.00 35.31 08:55:15 PM 16 62.03 0.00 2.55 0.00 0.00 0.00 0.00 0.00 0.00 35.42 08:55:15 PM 17 61.90 0.00 2.53 0.00 0.00 0.00 0.00 0.00 0.00 35.57 08:55:15 PM 18 58.36 0.00 2.41 0.09 0.00 3.72 0.00 0.00 0.00 35.43 08:55:15 PM 19 58.29 0.00 2.45 0.02 0.00 4.00 0.00 0.00 0.00 35.24 08:55:15 PM 20 58.45 0.00 2.36 0.02 0.00 3.73 0.00 0.00 0.00 35.44 08:55:15 PM 21 58.56 0.00 2.34 0.04 0.00 3.54 0.00 0.00 0.00 35.51 08:55:15 PM 22 58.12 0.00 2.46 0.00 0.00 3.84 0.00 0.00 0.00 35.58 08:55:15 PM 23 58.24 0.00 2.36 0.01 0.00 3.82 0.00 0.00 0.00 35.57 08:55:15 PM 24 58.84 0.00 2.42 0.00 0.00 3.60 0.00 0.00 0.00 35.14 08:55:15 PM 25 58.35 0.00 2.43 0.00 0.00 3.72 0.00 0.00 0.00 35.51 08:55:15 PM 26 62.09 0.00 2.47 0.00 0.00 0.00 0.00 0.00 0.00 35.45 08:55:15 PM 27 62.09 0.00 2.46 0.00 0.00 0.00 0.00 0.00 0.00 35.45 08:55:15 PM 28 62.15 0.00 2.48 0.00 0.00 0.00 0.00 0.00 0.00 35.37 08:55:15 PM 29 62.09 0.00 2.46 0.00 0.00 0.00 0.00 0.00 0.00 35.45 08:55:15 PM 30 62.74 0.00 2.45 0.00 0.00 0.00 0.00 0.00 0.00 34.81 08:55:15 PM 31 62.14 0.00 2.47 0.00 0.00 0.00 0.00 0.00 0.00 35.39 08:55:15 PM 32 62.35 0.00 2.52 0.00 0.00 0.00 0.00 0.00 0.00 35.13 08:55:15 PM 33 62.13 0.00 2.42 0.00 0.00 0.00 0.00 0.00 0.00 35.46 08:55:15 PM 34 62.08 0.00 2.47 0.00 0.00 0.00 0.00 0.00 0.00 35.45 08:55:15 PM 35 62.15 0.00 2.49 0.00 0.00 0.00 0.00 0.00 0.00 35.35 08:55:15 PM 36 43.19 0.00 0.80 0.00 0.00 0.00 0.00 0.00 0.00 56.01 08:55:15 PM 37 43.09 0.00 0.83 0.01 0.00 0.00 0.00 0.00 0.00 56.06 08:55:15 PM 38 42.59 0.00 0.78 0.01 0.00 0.00 0.00 0.00 0.00 56.63 08:55:15 PM 39 42.60 0.00 0.82 0.00 0.00 0.00 0.00 0.00 0.00 56.58 08:55:15 PM 40 42.94 0.00 0.76 0.02 0.00 0.00 0.00 0.00 0.00 56.28 08:55:15 PM 41 42.11 0.00 0.76 0.01 0.00 0.00 0.00 0.00 0.00 57.13 08:55:15 PM 42 43.26 0.00 0.85 0.01 0.00 0.00 0.00 0.00 0.00 55.88 08:55:15 PM 43 42.23 0.00 0.77 0.01 0.00 0.00 0.00 0.00 0.00 56.99 08:55:15 PM 44 42.24 0.00 0.80 0.00 0.00 0.00 0.00 0.00 0.00 56.95 08:55:15 PM 45 41.65 0.00 0.79 0.01 0.00 0.00 0.00 0.00 0.00 57.55 08:55:15 PM 46 41.76 0.00 0.77 0.01 0.00 0.00 0.00 0.00 0.00 57.46 08:55:15 PM 47 42.66 0.00 0.82 0.00 0.00 0.00 0.00 0.00 0.00 56.52 08:55:15 PM 48 41.83 0.00 0.76 0.01 0.00 0.00 0.00 0.00 0.00 57.40 08:55:15 PM 49 42.04 0.00 0.82 0.00 0.00 0.00 0.00 0.00 0.00 57.13 08:55:15 PM 50 41.55 0.00 0.78 0.01 0.00 0.00 0.00 0.00 0.00 57.67 08:55:15 PM 51 42.37 0.00 0.78 0.00 0.00 0.00 0.00 0.00 0.00 56.84 08:55:15 PM 52 41.66 0.00 0.77 0.00 0.00 0.00 0.00 0.00 0.00 57.57 08:55:15 PM 53 41.77 0.00 0.79 0.00 0.00 0.00 0.00 0.00 0.00 57.45 08:55:15 PM 54 40.33 0.00 0.88 0.01 0.00 0.00 0.00 0.00 0.00 58.79 08:55:15 PM 55 41.62 0.00 0.78 0.01 0.00 0.00 0.00 0.00 0.00 57.59 08:55:15 PM 56 41.51 0.00 0.82 0.00 0.00 0.00 0.00 0.00 0.00 57.67 08:55:15 PM 57 40.35 0.00 0.76 0.00 0.00 0.00 0.00 0.00 0.00 58.89 08:55:15 PM 58 41.92 0.00 0.79 0.01 0.00 0.00 0.00 0.00 0.00 57.28 08:55:15 PM 59 41.48 0.00 0.80 0.00 0.00 0.00 0.00 0.00 0.00 57.72 08:55:15 PM 60 40.83 0.00 0.75 0.01 0.00 0.00 0.00 0.00 0.00 58.41 08:55:15 PM 61 40.95 0.00 0.78 0.01 0.00 0.00 0.00 0.00 0.00 58.26 08:55:15 PM 62 40.85 0.00 0.80 0.01 0.00 0.00 0.00 0.00 0.00 58.35 08:55:15 PM 63 42.11 0.00 0.77 0.01 0.00 0.00 0.00 0.00 0.00 57.12 08:55:15 PM 64 42.22 0.00 0.78 0.01 0.00 0.00 0.00 0.00 0.00 56.99 08:55:15 PM 65 41.55 0.00 0.78 0.00 0.00 0.00 0.00 0.00 0.00 57.67 08:55:15 PM 66 41.68 0.00 0.81 0.00 0.00 0.00 0.00 0.00 0.00 57.51 08:55:15 PM 67 41.06 0.00 0.87 0.01 0.00 0.00 0.00 0.00 0.00 58.07 08:55:15 PM 68 41.80 0.00 0.79 0.00 0.00 0.00 0.00 0.00 0.00 57.41 08:55:15 PM 69 41.38 0.00 0.75 0.00 0.00 0.00 0.00 0.00 0.00 57.87 08:55:15 PM 70 42.17 0.00 0.75 0.00 0.00 0.00 0.00 0.00 0.00 57.08 08:55:15 PM 71 41.26 0.00 0.81 0.00 0.00 0.00 0.01 0.00 0.00 57.92 1> rp(recon:scheduler_usage(1000)). [{1,0.8581586716045433}, {2,0.8607597847736659}, {3,0.8501543449289042}, {4,0.8619718426658023}, {5,0.8660226578705742}, {6,0.8555763933364806}, {7,0.8557195972097537}, {8,0.8668284625654983}, {9,0.858588675506532}, {10,0.8650501369640686}, {11,0.8615557214275474}, {12,0.8523392789683996}, {13,0.8469329348407398}, {14,0.8615637534991641}, {15,0.8591930739986557}, {16,0.8578596164201188}, {17,0.8549696086049403}, {18,0.8677098230685564}, {19,0.8556328340054443}, {20,0.8642542459479063}, {21,0.8670655179064826}, {22,0.8562638308551557}, {23,0.8511202058055239}, {24,0.8697339264798696}, {25,0.8594823462601853}, {26,0.8634550008001265}, {27,0.8568590130172198}, {28,0.864025757952233}, {29,0.8575614770622326}, {30,0.8611760239143281}, {31,0.8589789125669829}, {32,0.8537754468447621}, {33,0.8675538912040871}, {34,0.8607522803519515}, {35,0.8639389099445753}, {36,0.8743759133171144}, {37,0.9526507413773687}, {38,0.948925611917577}, {39,0.9511904612810083}, {40,0.9581015421004746}, {41,0.9534977201467516}, {42,0.0}, {43,0.0}, {44,0.0}, {45,0.9522518499948341}, {46,0.9453306755503674}, {47,0.9586374601140047}, {48,0.9574611845671873}, {49,0.950914304057791}, {50,0.9582124294398642}, {51,0.9448074463900029}, {52,0.9474771999511407}, {53,0.9450778098401229}, {54,0.0}, {55,0.9478167911711937}, {56,0.9612587860764551}, {57,0.943468696358506}, {58,0.9509498642900455}, {59,0.9603595134495414}, {60,0.9517397912369205}, {61,0.0}, {62,0.9587029756970658}, {63,0.0}, {64,0.0}, {65,0.9526167902065421}, {66,0.0}, {67,0.9475668680715805}, {68,0.0}, {69,0.9630301804839606}, {70,0.9475359592879946}, {71,0.9541718747681374}, {72,0.0}] Thanks for your time, Jesse Stimpson -------------- next part -------------- An HTML attachment was scrubbed... URL: From nalundgaard@REDACTED Wed Jan 9 06:09:29 2019 From: nalundgaard@REDACTED (Nicholas Lundgaard) Date: Wed, 09 Jan 2019 00:09:29 -0500 Subject: [erlang-questions] =?utf-8?q?ERL-823=3A_SSL_cipher=5Fsuites_too_l?= =?utf-8?q?imited_when_compiling_with_OPENSSL=5FNO=5FEC=3D1?= In-Reply-To: <3abc08ae-8e1a-2b11-897d-3b8da7893897@hedeland.org> References: <09da934e-11d0-4e82-bb80-1b327fe653e7@www.fastmail.com> <951eee52-6859-05c3-6713-1097b1387347@hedeland.org> <6ad2ad0f-a068-49b2-908d-a5c5f54c5083@www.fastmail.com> <3abc08ae-8e1a-2b11-897d-3b8da7893897@hedeland.org> Message-ID: <5130b416-2115-45f9-b8e3-1313640ff992@www.fastmail.com> It seems that the remaining issue here, after removing the unneeded OPENSSL_NO_EC=1 in my Erlang/OTP build, has turned out to be a quirk of either OpenSSL with FIPS support in general, or RedHat/CentOS/Fedora's OpenSSL with FIPS support (even without FIPS enabled). It seems that the library categorically enforces a minimum of 256 bits for EC curves, and it rejects the definition of curves smaller than this, resulting in the run-time errors I have observed. I have opened up ERL-825 to cover this very distinct issue, and I wanted to post a final note on this thread to anyone interested in following along: https://bugs.erlang.org/browse/ERL-825 I'm not sure this behavior is really worth fixing, but hopefully that issue will track whatever final resolution is reached. I've closed https://bugs.erlang.org/browse/ERL-823. Thanks & Regards, ?Nicholas Lundgaard On Sat, Jan 5, 2019, at 3:39 PM, Per Hedeland wrote: > On 2019-01-05 16:03, Nicholas Lundgaard wrote: > > "Well, I'm not really an expert on this stuff" > > > > That makes two of us, at least... :) > > I was very impressed by your VerifyCurve function though.:-) > > > You're exactly right, when I compile OTP 21 on CentOS with or without OPENSSL_NO_EC2M, in either case, the built OTP correctly leaves ec_gf2m out of the public keys and filters out the GF2m named EC curves that aren't supported in the CentOS/RHEL OpenSSL. > > > > The remaining problem on RHEL/CentOS systems is that (for whatever reason) these systems lack support for a bunch of non-GF2m curves that Erlang/OTP's crypto/ssl modules claim to support, but emit runtime exceptions when you try to use them. It seems that is because they are explicitly hard-coded into the C headers in the OTP applications, when that information could be extracted or extrapolated from the linked SSL library itself (either dynamically at runtime or during compilation). > > Traditionally the build of the crypto.c NIF module has been based on > the #defines in the *OpenSSL* headers, in particular > openssl/opensslconf.h, which is supposed to describe how the OpenSSL > libraries were built - this is where both OPENSSL_NO_EC and > OPENSSL_NO_EC2M originates (i.e. from using the 'no-ec' and 'no-ec2m' > options, respectively, in the OpenSSL configure/build). > > I don't think there are any more specific #defines for the individual > ECC curves though - and in fact there doesn't seem to be any options > for that in the OpenSSL build either. But in any case, using only the > OpenSSL headers at build time, while linking dynamically with > libcrypto, means that the libcrypto found at run time may not match > the build, and runtime checks are motivated. And I believe both OTP > crypto and OTP ssl do some such checks, but maybe they aren't 100%. > > > For clarity, I think this situation with these missing EC curves on RHEL/CentOS has been like this for many years with almost no change, so upgrading to a newer version of RHEL/CentOS/Amazon Linux isn't going to help. Here is the output from 'openssl ecparam -list_curves' on the latest (2018.03) Amazon Linux AMI running "OpenSSL 1.0.2k-fips 26 Jan 2017": > > > > openssl ecparam -list_curves > > secp256k1 : SECG curve over a 256 bit prime field > > secp384r1 : NIST/SECG curve over a 384 bit prime field > > secp521r1 : NIST/SECG curve over a 521 bit prime field > > prime256v1: X9.62/SECG curve over a 256 bit prime field > > > > And here is the output from 'openssl ecparam -list_curves' on a CentOS 6.9 box running "OpenSSL 1.0.1e-fips 11 Feb 2013". The older OpenSSL library omits the secp256k1 curve, but again, pretty similar output: > > > > openssl ecparam -list_curves > > secp384r1 : NIST/SECG curve over a 384 bit prime field > > secp521r1 : NIST/SECG curve over a 521 bit prime field > > prime256v1: X9.62/SECG curve over a 256 bit prime field > > I also tried the 'ecparam -list_curves' on various systems, and was > pretty surprised by the result... As you can see, the list you get on > CentOS is way shorter than the list of successful invocations of your > VerifyCurve funcion - and with a "normal" OpenSSL-1.0.1 build, I get > this: > > secp112r1 : SECG/WTLS curve over a 112 bit prime field > secp112r2 : SECG curve over a 112 bit prime field > secp128r1 : SECG curve over a 128 bit prime field > secp128r2 : SECG curve over a 128 bit prime field > secp160k1 : SECG curve over a 160 bit prime field > secp160r1 : SECG curve over a 160 bit prime field > secp160r2 : SECG/WTLS curve over a 160 bit prime field > secp192k1 : SECG curve over a 192 bit prime field > secp224k1 : SECG curve over a 224 bit prime field > secp224r1 : NIST/SECG curve over a 224 bit prime field > secp256k1 : SECG curve over a 256 bit prime field > secp384r1 : NIST/SECG curve over a 384 bit prime field > secp521r1 : NIST/SECG curve over a 521 bit prime field > prime192v1: NIST/X9.62/SECG curve over a 192 bit prime field > prime192v2: X9.62 curve over a 192 bit prime field > prime192v3: X9.62 curve over a 192 bit prime field > prime239v1: X9.62 curve over a 239 bit prime field > prime239v2: X9.62 curve over a 239 bit prime field > prime239v3: X9.62 curve over a 239 bit prime field > prime256v1: X9.62/SECG curve over a 256 bit prime field > sect113r1 : SECG curve over a 113 bit binary field > sect113r2 : SECG curve over a 113 bit binary field > sect131r1 : SECG/WTLS curve over a 131 bit binary field > sect131r2 : SECG curve over a 131 bit binary field > sect163k1 : NIST/SECG/WTLS curve over a 163 bit binary field > sect163r1 : SECG curve over a 163 bit binary field > sect163r2 : NIST/SECG curve over a 163 bit binary field > sect193r1 : SECG curve over a 193 bit binary field > sect193r2 : SECG curve over a 193 bit binary field > sect233k1 : NIST/SECG/WTLS curve over a 233 bit binary field > sect233r1 : NIST/SECG/WTLS curve over a 233 bit binary field > sect239k1 : SECG curve over a 239 bit binary field > sect283k1 : NIST/SECG curve over a 283 bit binary field > sect283r1 : NIST/SECG curve over a 283 bit binary field > sect409k1 : NIST/SECG curve over a 409 bit binary field > sect409r1 : NIST/SECG curve over a 409 bit binary field > sect571k1 : NIST/SECG curve over a 571 bit binary field > sect571r1 : NIST/SECG curve over a 571 bit binary field > c2pnb163v1: X9.62 curve over a 163 bit binary field > c2pnb163v2: X9.62 curve over a 163 bit binary field > c2pnb163v3: X9.62 curve over a 163 bit binary field > c2pnb176v1: X9.62 curve over a 176 bit binary field > c2tnb191v1: X9.62 curve over a 191 bit binary field > c2tnb191v2: X9.62 curve over a 191 bit binary field > c2tnb191v3: X9.62 curve over a 191 bit binary field > c2pnb208w1: X9.62 curve over a 208 bit binary field > c2tnb239v1: X9.62 curve over a 239 bit binary field > c2tnb239v2: X9.62 curve over a 239 bit binary field > c2tnb239v3: X9.62 curve over a 239 bit binary field > c2pnb272w1: X9.62 curve over a 272 bit binary field > c2pnb304w1: X9.62 curve over a 304 bit binary field > c2tnb359v1: X9.62 curve over a 359 bit binary field > c2pnb368w1: X9.62 curve over a 368 bit binary field > c2tnb431r1: X9.62 curve over a 431 bit binary field > wap-wsg-idm-ecid-wtls1: WTLS curve over a 113 bit binary field > wap-wsg-idm-ecid-wtls3: NIST/SECG/WTLS curve over a 163 bit binary field > wap-wsg-idm-ecid-wtls4: SECG curve over a 113 bit binary field > wap-wsg-idm-ecid-wtls5: X9.62 curve over a 163 bit binary field > wap-wsg-idm-ecid-wtls6: SECG/WTLS curve over a 112 bit prime field > wap-wsg-idm-ecid-wtls7: SECG/WTLS curve over a 160 bit prime field > wap-wsg-idm-ecid-wtls8: WTLS curve over a 112 bit prime field > wap-wsg-idm-ecid-wtls9: WTLS curve over a 160 bit prime field > wap-wsg-idm-ecid-wtls10: NIST/SECG/WTLS curve over a 233 bit binary field > wap-wsg-idm-ecid-wtls11: NIST/SECG/WTLS curve over a 233 bit binary field > wap-wsg-idm-ecid-wtls12: WTLS curvs over a 224 bit prime field > Oakley-EC2N-3: > IPSec/IKE/Oakley curve #3 over a 155 bit binary field. > Not suitable for ECDSA. > Questionable extension field! > Oakley-EC2N-4: > IPSec/IKE/Oakley curve #4 over a 185 bit binary field. > Not suitable for ECDSA. > Questionable extension field! > > I haven't done a strict comparison of this output with the results of > running your VerifyCurve function on the same system, but my > impression is that all curves that were successful in the latter are > also included in the former. > > I really wonder how the CentOS libcrypto was built. The "-fips" in the > version might be a clue, but I believe this only means that it was > built in such a way that "FIPS mode" is *available*, and thus it > shouldn't affect something like the set of "supported" ECC curves. > (If "FIPS mode" was *enforced*, it wouldn't work at all before OTP 20 > I believe, and this is definitely not the case.) > > Anyway, 'ecparam -list_curves' uses the OpenSSL function > EC_get_builtin_curves(), which is documented in the OpenSSL man page > with, among other info: > > Whilst the library can be used to create any curve using the functions > described above, there are also a number of predefined curves that are > available. In order to obtain a list of all of the predefined curves, > call the function EC_get_builtin_curves. > > So I guess this isn't really the set of *supported* curves, and the > OpenSSL ecparam(1) man page does say > > -list_curves > If this options is specified ecparam will print out a list of all > currently implemented EC parameters names and exit. > ^^^^^^^^^^^ > > - and there may already be some magic going on in the OTP 'crypto' app > to create additional curves "on demand". But by now I'm way out of > whatever expertise I may have... > > --Per > From jonas.falkevik@REDACTED Wed Jan 9 09:33:03 2019 From: jonas.falkevik@REDACTED (Jonas Falkevik) Date: Wed, 9 Jan 2019 09:33:03 +0100 Subject: [erlang-questions] Dirty CPU schedulers stuck at zero utilization In-Reply-To: References: Message-ID: Hi, This sounds like the issue I bumped into. After some time, many of the dirty schedulers are not performing any work anymore. Which was related to a corrupted list of sleeping dirty schedulers. I haven?t found what triggered it in my case (maybe a GC?). For me it took approx. 1 day before it happened. Doing Multi Scheduler Block can trigger the problem with the list of sleeping dirty schedulers which gets corrupted. The sleepers list can contain only one or some schedulers. That list is later used to wake up schedulers when there is work to do. For each enqueued dirty-cpu or dirty-io job, it wakes the first scheduler in the sleepers list. But if this list only point to a few schedulers only those are woken up to pick jobs from the queue. Since fixing that specific problem I have not seen the collapse of dirty schedulers anymore. It would be great if you can test and see if it fixes your problem as well, https://github.com/erlang/otp/pull/2027 Any feedback on the bug fix is appreciated. /Jonas > On 8 Jan 2019, at 17:13, Jesse Stimpson wrote: > > Hello, > > We run a large scale WebRTC-based VOIP application, and currently we are evaluating whether or not dirty CPU schedulers can help us get more consistent performance with some critical NIFs. We've created a test that mimics our production load and have run it on AWS EC2 instances of various sizes. When dirty CPU schedulers are working as expected, we do see measurable improvement in performance, but unfortunately we have seen that they can drop to 0 utilization. They do not recover from this state. > > The timings from 2 of our 12 NIFs are in the vicinity of the 1 msec rule of thumb, so we have experimented with them both on and off dirty CPU schedulers. We've left the other NIFs on the traditional schedulers. More detailed NIF timings can be found below. > > At the end of this message is the concerning data (from recon:scheduler_usage/1).We have schedulers 1-36 as the traditional schedulers bound to the first hardware thread of each core, and schedulers 37-72 are dirty CPU schedulers. Based on mpstat it appears the dirty CPU schedulers are not bound to threads. After about 10 minutes of running the load test, schedulers 42, 43, 44, 61, 63, 64, 66, 68, 72 went to at 0 utilization and stayed there indefinitely. As time goes on, more dirty CPU schedulers enter this 0 utilization state. Schedulers 1-36 do not exhibit this behaviour. We have reproduced this consistently on other EC2 c5 instances of various sizes. > > We have tried various beam adjustments, but we have seen the same behaviour each time. Some examples of things we have tried: > +sbwt very_long +sbwtdcpu very_long +swt very_low +swtdcpu very_low > +sfwi 500 > > Is there anything else that we can try to avoid this 0-utilization issue? We are happy to provide more data if needed. > > NIF: webrtc_acm_add_data_10ms > median: 606 usec > 90 percentile: 1230 usec > 99 percentile: 2285 usec > 99.9 percentile: 3223 usec > > NIF: webrtc_acm_playout_data_10ms > median: 98 usec > 90 percentile: 371 usec > 99 percentile: 410 usec > 99.9 percentile: 449 usec > > EC2 instance: c5.18xlarge > > OS: Linux 4.4.0-1073-aws #83-Ubuntu SMP Sat Nov 17 00:26:27 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux > > beam flags: +P 4194304 +sbt db +SP 50:50 +sub true +A 128 +K true > > Erlang banner: Erlang/OTP 21 [erts-10.2] [source] [64-bit] [smp:36:36] [ds:36:36:10] [async-threads:128] [hipe] > > cpu_topology: > [{node,[{processor,[{core,[{thread,{logical,0}}, > {thread,{logical,36}}]}, > {core,[{thread,{logical,1}},{thread,{logical,37}}]}, > {core,[{thread,{logical,2}},{thread,{logical,38}}]}, > {core,[{thread,{logical,3}},{thread,{logical,39}}]}, > {core,[{thread,{logical,4}},{thread,{logical,40}}]}, > {core,[{thread,{logical,5}},{thread,{logical,41}}]}, > {core,[{thread,{logical,6}},{thread,{logical,42}}]}, > {core,[{thread,{logical,7}},{thread,{logical,43}}]}, > {core,[{thread,{logical,8}},{thread,{logical,44}}]}, > {core,[{thread,{logical,9}},{thread,{logical,45}}]}, > {core,[{thread,{logical,10}},{thread,{logical,46}}]}, > {core,[{thread,{logical,11}},{thread,{logical,47}}]}, > {core,[{thread,{logical,12}},{thread,{logical,48}}]}, > {core,[{thread,{logical,13}},{thread,{logical,49}}]}, > {core,[{thread,{logical,14}},{thread,{logical,50}}]}, > {core,[{thread,{logical,15}},{thread,{logical,51}}]}, > {core,[{thread,{logical,16}},{thread,{logical,52}}]}, > {core,[{thread,{logical,17}},{thread,{logical,53}}]}]}]}, > {node,[{processor,[{core,[{thread,{logical,18}}, > {thread,{logical,54}}]}, > {core,[{thread,{logical,19}},{thread,{logical,55}}]}, > {core,[{thread,{logical,20}},{thread,{logical,56}}]}, > {core,[{thread,{logical,21}},{thread,{logical,57}}]}, > {core,[{thread,{logical,22}},{thread,{logical,58}}]}, > {core,[{thread,{logical,23}},{thread,{logical,59}}]}, > {core,[{thread,{logical,24}},{thread,{logical,60}}]}, > {core,[{thread,{logical,25}},{thread,{logical,61}}]}, > {core,[{thread,{logical,26}},{thread,{logical,62}}]}, > {core,[{thread,{logical,27}},{thread,{logical,63}}]}, > {core,[{thread,{logical,28}},{thread,{logical,64}}]}, > {core,[{thread,{logical,29}},{thread,{logical,65}}]}, > {core,[{thread,{logical,30}},{thread,{logical,66}}]}, > {core,[{thread,{logical,31}},{thread,{logical,67}}]}, > {core,[{thread,{logical,32}},{thread,{logical,68}}]}, > {core,[{thread,{logical,33}},{thread,{logical,69}}]}, > {core,[{thread,{logical,34}},{thread,{logical,70}}]}, > {core,[{thread,{logical,35}},{thread,{logical,71}}]}]}]}] > > scheduler_bindings: {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22, > 23,24,25,26,27,28,29,30,31,32,33,34,35} > > CPU utilization (from mpstat) during our load test: > Linux 4.4.0-1073-aws (isw-core-x-qa-awsoh02) 01/04/2019 _x86_64_ (72 CPU) > > 08:55:15 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle > 08:55:15 PM all 51.58 0.00 1.64 0.01 0.00 0.42 0.00 0.00 0.00 46.36 > 08:55:15 PM 0 62.34 0.00 2.69 0.06 0.00 0.00 0.00 0.00 0.00 34.90 > 08:55:15 PM 1 62.84 0.00 2.58 0.02 0.00 0.00 0.00 0.00 0.00 34.55 > 08:55:15 PM 2 62.04 0.00 2.52 0.00 0.00 0.00 0.00 0.00 0.00 35.44 > 08:55:15 PM 3 62.17 0.00 2.62 0.00 0.00 0.00 0.00 0.00 0.00 35.21 > 08:55:15 PM 4 62.14 0.00 2.54 0.02 0.00 0.00 0.00 0.00 0.00 35.30 > 08:55:15 PM 5 62.06 0.00 2.51 0.00 0.00 0.00 0.00 0.00 0.00 35.43 > 08:55:15 PM 6 61.93 0.00 2.58 0.01 0.00 0.00 0.02 0.00 0.00 35.46 > 08:55:15 PM 7 61.99 0.00 2.54 0.00 0.00 0.00 0.00 0.00 0.00 35.48 > 08:55:15 PM 8 62.01 0.00 2.52 0.00 0.00 0.00 0.00 0.00 0.00 35.47 > 08:55:15 PM 9 61.88 0.00 2.49 0.00 0.00 0.00 0.00 0.00 0.00 35.63 > 08:55:15 PM 10 62.18 0.00 2.48 0.00 0.00 0.00 0.00 0.00 0.00 35.34 > 08:55:15 PM 11 61.89 0.00 2.47 0.00 0.00 0.00 0.00 0.00 0.00 35.64 > 08:55:15 PM 12 61.98 0.00 2.50 0.00 0.00 0.00 0.00 0.00 0.00 35.52 > 08:55:15 PM 13 61.80 0.00 2.54 0.00 0.00 0.00 0.00 0.00 0.00 35.66 > 08:55:15 PM 14 61.96 0.00 2.56 0.00 0.00 0.00 0.00 0.00 0.00 35.48 > 08:55:15 PM 15 62.11 0.00 2.57 0.00 0.00 0.00 0.00 0.00 0.00 35.31 > 08:55:15 PM 16 62.03 0.00 2.55 0.00 0.00 0.00 0.00 0.00 0.00 35.42 > 08:55:15 PM 17 61.90 0.00 2.53 0.00 0.00 0.00 0.00 0.00 0.00 35.57 > 08:55:15 PM 18 58.36 0.00 2.41 0.09 0.00 3.72 0.00 0.00 0.00 35.43 > 08:55:15 PM 19 58.29 0.00 2.45 0.02 0.00 4.00 0.00 0.00 0.00 35.24 > 08:55:15 PM 20 58.45 0.00 2.36 0.02 0.00 3.73 0.00 0.00 0.00 35.44 > 08:55:15 PM 21 58.56 0.00 2.34 0.04 0.00 3.54 0.00 0.00 0.00 35.51 > 08:55:15 PM 22 58.12 0.00 2.46 0.00 0.00 3.84 0.00 0.00 0.00 35.58 > 08:55:15 PM 23 58.24 0.00 2.36 0.01 0.00 3.82 0.00 0.00 0.00 35.57 > 08:55:15 PM 24 58.84 0.00 2.42 0.00 0.00 3.60 0.00 0.00 0.00 35.14 > 08:55:15 PM 25 58.35 0.00 2.43 0.00 0.00 3.72 0.00 0.00 0.00 35.51 > 08:55:15 PM 26 62.09 0.00 2.47 0.00 0.00 0.00 0.00 0.00 0.00 35.45 > 08:55:15 PM 27 62.09 0.00 2.46 0.00 0.00 0.00 0.00 0.00 0.00 35.45 > 08:55:15 PM 28 62.15 0.00 2.48 0.00 0.00 0.00 0.00 0.00 0.00 35.37 > 08:55:15 PM 29 62.09 0.00 2.46 0.00 0.00 0.00 0.00 0.00 0.00 35.45 > 08:55:15 PM 30 62.74 0.00 2.45 0.00 0.00 0.00 0.00 0.00 0.00 34.81 > 08:55:15 PM 31 62.14 0.00 2.47 0.00 0.00 0.00 0.00 0.00 0.00 35.39 > 08:55:15 PM 32 62.35 0.00 2.52 0.00 0.00 0.00 0.00 0.00 0.00 35.13 > 08:55:15 PM 33 62.13 0.00 2.42 0.00 0.00 0.00 0.00 0.00 0.00 35.46 > 08:55:15 PM 34 62.08 0.00 2.47 0.00 0.00 0.00 0.00 0.00 0.00 35.45 > 08:55:15 PM 35 62.15 0.00 2.49 0.00 0.00 0.00 0.00 0.00 0.00 35.35 > 08:55:15 PM 36 43.19 0.00 0.80 0.00 0.00 0.00 0.00 0.00 0.00 56.01 > 08:55:15 PM 37 43.09 0.00 0.83 0.01 0.00 0.00 0.00 0.00 0.00 56.06 > 08:55:15 PM 38 42.59 0.00 0.78 0.01 0.00 0.00 0.00 0.00 0.00 56.63 > 08:55:15 PM 39 42.60 0.00 0.82 0.00 0.00 0.00 0.00 0.00 0.00 56.58 > 08:55:15 PM 40 42.94 0.00 0.76 0.02 0.00 0.00 0.00 0.00 0.00 56.28 > 08:55:15 PM 41 42.11 0.00 0.76 0.01 0.00 0.00 0.00 0.00 0.00 57.13 > 08:55:15 PM 42 43.26 0.00 0.85 0.01 0.00 0.00 0.00 0.00 0.00 55.88 > 08:55:15 PM 43 42.23 0.00 0.77 0.01 0.00 0.00 0.00 0.00 0.00 56.99 > 08:55:15 PM 44 42.24 0.00 0.80 0.00 0.00 0.00 0.00 0.00 0.00 56.95 > 08:55:15 PM 45 41.65 0.00 0.79 0.01 0.00 0.00 0.00 0.00 0.00 57.55 > 08:55:15 PM 46 41.76 0.00 0.77 0.01 0.00 0.00 0.00 0.00 0.00 57.46 > 08:55:15 PM 47 42.66 0.00 0.82 0.00 0.00 0.00 0.00 0.00 0.00 56.52 > 08:55:15 PM 48 41.83 0.00 0.76 0.01 0.00 0.00 0.00 0.00 0.00 57.40 > 08:55:15 PM 49 42.04 0.00 0.82 0.00 0.00 0.00 0.00 0.00 0.00 57.13 > 08:55:15 PM 50 41.55 0.00 0.78 0.01 0.00 0.00 0.00 0.00 0.00 57.67 > 08:55:15 PM 51 42.37 0.00 0.78 0.00 0.00 0.00 0.00 0.00 0.00 56.84 > 08:55:15 PM 52 41.66 0.00 0.77 0.00 0.00 0.00 0.00 0.00 0.00 57.57 > 08:55:15 PM 53 41.77 0.00 0.79 0.00 0.00 0.00 0.00 0.00 0.00 57.45 > 08:55:15 PM 54 40.33 0.00 0.88 0.01 0.00 0.00 0.00 0.00 0.00 58.79 > 08:55:15 PM 55 41.62 0.00 0.78 0.01 0.00 0.00 0.00 0.00 0.00 57.59 > 08:55:15 PM 56 41.51 0.00 0.82 0.00 0.00 0.00 0.00 0.00 0.00 57.67 > 08:55:15 PM 57 40.35 0.00 0.76 0.00 0.00 0.00 0.00 0.00 0.00 58.89 > 08:55:15 PM 58 41.92 0.00 0.79 0.01 0.00 0.00 0.00 0.00 0.00 57.28 > 08:55:15 PM 59 41.48 0.00 0.80 0.00 0.00 0.00 0.00 0.00 0.00 57.72 > 08:55:15 PM 60 40.83 0.00 0.75 0.01 0.00 0.00 0.00 0.00 0.00 58.41 > 08:55:15 PM 61 40.95 0.00 0.78 0.01 0.00 0.00 0.00 0.00 0.00 58.26 > 08:55:15 PM 62 40.85 0.00 0.80 0.01 0.00 0.00 0.00 0.00 0.00 58.35 > 08:55:15 PM 63 42.11 0.00 0.77 0.01 0.00 0.00 0.00 0.00 0.00 57.12 > 08:55:15 PM 64 42.22 0.00 0.78 0.01 0.00 0.00 0.00 0.00 0.00 56.99 > 08:55:15 PM 65 41.55 0.00 0.78 0.00 0.00 0.00 0.00 0.00 0.00 57.67 > 08:55:15 PM 66 41.68 0.00 0.81 0.00 0.00 0.00 0.00 0.00 0.00 57.51 > 08:55:15 PM 67 41.06 0.00 0.87 0.01 0.00 0.00 0.00 0.00 0.00 58.07 > 08:55:15 PM 68 41.80 0.00 0.79 0.00 0.00 0.00 0.00 0.00 0.00 57.41 > 08:55:15 PM 69 41.38 0.00 0.75 0.00 0.00 0.00 0.00 0.00 0.00 57.87 > 08:55:15 PM 70 42.17 0.00 0.75 0.00 0.00 0.00 0.00 0.00 0.00 57.08 > 08:55:15 PM 71 41.26 0.00 0.81 0.00 0.00 0.00 0.01 0.00 0.00 57.92 > > > 1> rp(recon:scheduler_usage(1000)). > [{1,0.8581586716045433}, > {2,0.8607597847736659}, > {3,0.8501543449289042}, > {4,0.8619718426658023}, > {5,0.8660226578705742}, > {6,0.8555763933364806}, > {7,0.8557195972097537}, > {8,0.8668284625654983}, > {9,0.858588675506532}, > {10,0.8650501369640686}, > {11,0.8615557214275474}, > {12,0.8523392789683996}, > {13,0.8469329348407398}, > {14,0.8615637534991641}, > {15,0.8591930739986557}, > {16,0.8578596164201188}, > {17,0.8549696086049403}, > {18,0.8677098230685564}, > {19,0.8556328340054443}, > {20,0.8642542459479063}, > {21,0.8670655179064826}, > {22,0.8562638308551557}, > {23,0.8511202058055239}, > {24,0.8697339264798696}, > {25,0.8594823462601853}, > {26,0.8634550008001265}, > {27,0.8568590130172198}, > {28,0.864025757952233}, > {29,0.8575614770622326}, > {30,0.8611760239143281}, > {31,0.8589789125669829}, > {32,0.8537754468447621}, > {33,0.8675538912040871}, > {34,0.8607522803519515}, > {35,0.8639389099445753}, > {36,0.8743759133171144}, > {37,0.9526507413773687}, > {38,0.948925611917577}, > {39,0.9511904612810083}, > {40,0.9581015421004746}, > {41,0.9534977201467516}, > {42,0.0}, > {43,0.0}, > {44,0.0}, > {45,0.9522518499948341}, > {46,0.9453306755503674}, > {47,0.9586374601140047}, > {48,0.9574611845671873}, > {49,0.950914304057791}, > {50,0.9582124294398642}, > {51,0.9448074463900029}, > {52,0.9474771999511407}, > {53,0.9450778098401229}, > {54,0.0}, > {55,0.9478167911711937}, > {56,0.9612587860764551}, > {57,0.943468696358506}, > {58,0.9509498642900455}, > {59,0.9603595134495414}, > {60,0.9517397912369205}, > {61,0.0}, > {62,0.9587029756970658}, > {63,0.0}, > {64,0.0}, > {65,0.9526167902065421}, > {66,0.0}, > {67,0.9475668680715805}, > {68,0.0}, > {69,0.9630301804839606}, > {70,0.9475359592879946}, > {71,0.9541718747681374}, > {72,0.0}] > > > Thanks for your time, > > Jesse Stimpson > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From mononcqc@REDACTED Wed Jan 9 17:11:24 2019 From: mononcqc@REDACTED (Fred Hebert) Date: Wed, 9 Jan 2019 11:11:24 -0500 Subject: [erlang-questions] [ANN] Property-Based Testing with PropEr, Erlang, and Elixir Message-ID: Hi everyone, today I'm happy to announce that my latest book is now out! [image: proper.png] The book is *Property-Based Testing with PropEr, Erlang, and Elixir*, and is probably the friendliest introduction you can get to Property-Based Testing in this community, and also one of the best resource to bring you to a reasonably advanced level of skill. If you're interested in more details, please see any of: - My blog post on the book: https://ferd.ca/property-based-testing-with-proper-erlang-and-elixir.html - The Pragmatic Programmers newsletter: https://media.pragprog.com/newsletters/2019-01-09.html And here's also a twitter link in case any of you feels like making some noise on my behalf :) https://twitter.com/mononcqc/status/1083027123777036291 Thanks, and enjoy the read! - Fred. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: proper.png Type: image/png Size: 36994 bytes Desc: not available URL: From gerhard@REDACTED Wed Jan 9 18:08:38 2019 From: gerhard@REDACTED (Gerhard Lazu) Date: Wed, 9 Jan 2019 17:08:38 +0000 Subject: [erlang-questions] Missing checksums for github.com/erlang/otp/releases Message-ID: I think it would be great to have checksums publicly available when a new Erlang/OTP patch is tagged on GitHub. Something as simple as this will do: sha256sum OTP-21.2.2.tar.gz > OTP-21.2.2.tar.gz.sha256 curl --request POST --data-binary "@OTP-21.2.2.tar.gz.sha256" --header "Content-Type: text/plain" https://uploads.github.com/repos/erlang/otp/releases/OTP-21.2.2/assets?name=OTP-21.2.2.tar.gz.sha256 Is this something that others are missing? If not, how do you answer "*I know that this Erlang/OTP build is legit*" in your production environments? Thank you, Gerhard. -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric.des.courtis@REDACTED Wed Jan 9 18:24:32 2019 From: eric.des.courtis@REDACTED (Eric des Courtis) Date: Wed, 9 Jan 2019 12:24:32 -0500 Subject: [erlang-questions] [ANN] Property-Based Testing with PropEr, Erlang, and Elixir In-Reply-To: References: Message-ID: Congratulations! This type of content is always appreciated and well received by the Erlang and Elixir community. On Wed, Jan 9, 2019 at 11:11 AM Fred Hebert wrote: > Hi everyone, today I'm happy to announce that my latest book is now out! > > [image: proper.png] > > The book is *Property-Based Testing with PropEr, Erlang, and Elixir*, and > is probably the friendliest introduction you can get to Property-Based > Testing in this community, and also one of the best resource to bring you > to a reasonably advanced level of skill. > > If you're interested in more details, please see any of: > > - My blog post on the book: > https://ferd.ca/property-based-testing-with-proper-erlang-and-elixir.html > - The Pragmatic Programmers newsletter: > https://media.pragprog.com/newsletters/2019-01-09.html > > And here's also a twitter link in case any of you feels like making some > noise on my behalf :) > https://twitter.com/mononcqc/status/1083027123777036291 > > Thanks, and enjoy the read! > - Fred. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lloyd@REDACTED Wed Jan 9 19:28:19 2019 From: lloyd@REDACTED (Lloyd R. Prentice) Date: Wed, 9 Jan 2019 13:28:19 -0500 Subject: [erlang-questions] [ANN] Property-Based Testing with PropEr, Erlang, and Elixir In-Reply-To: References: Message-ID: <27F28D73-64F8-4D87-85A4-6005E3610423@writersglen.com> Hi Fred, Will order and look forward to reading and learning. Congratulations, Lloyd Sent from my iPad > On Jan 9, 2019, at 11:11 AM, Fred Hebert wrote: > > Hi everyone, today I'm happy to announce that my latest book is now out! > > > > The book is Property-Based Testing with PropEr, Erlang, and Elixir, and is probably the friendliest introduction you can get to Property-Based Testing in this community, and also one of the best resource to bring you to a reasonably advanced level of skill. > > If you're interested in more details, please see any of: > My blog post on the book: https://ferd.ca/property-based-testing-with-proper-erlang-and-elixir.html > The Pragmatic Programmers newsletter: https://media.pragprog.com/newsletters/2019-01-09.html > And here's also a twitter link in case any of you feels like making some noise on my behalf :) https://twitter.com/mononcqc/status/1083027123777036291 > > Thanks, and enjoy the read! > - Fred. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonas.falkevik@REDACTED Thu Jan 10 00:06:07 2019 From: jonas.falkevik@REDACTED (Jonas Falkevik) Date: Thu, 10 Jan 2019 00:06:07 +0100 Subject: [erlang-questions] Dirty CPU schedulers stuck at zero utilization In-Reply-To: References: Message-ID: Hi Jesse, we have not observe the behaviour you see. Can you share microstate accounting stats for the different cases? Maybe that could shed some light on what the schedulers are doing? 1> msacc:start(1000), msacc:print(). And maybe also running perf top as root on one of the scheduler threads that are fully utilised can be interesting. # perf top --tid /Jonas On Wed, Jan 9, 2019 at 10:52 PM Jesse Stimpson < jstimpson@REDACTED> wrote: > Rikard and Jonas, > > Thank you for your response. With the patch at OTP PR-2027, we do observe > that the dirty CPU schedulers do not drop to zero utilization, but in > addition we see that the performance of the traditional schedulers > suffers. Specifically, when we apply our load test, scheduler usage > toggles in between the following two states. I've snipped the output of > similar values for readability. > > [{1,0.8933020805456553}, > > ... > > {36,0.8882068476700309}, > > {37,0.4932569692588096}, > > ... > > {72,0.49842881346116297}] > > > and > > > [{1,1.0}, > > ... > > {36,1.0}, > > {37,0.42909590032946493}, > > ... > > {72,0.4641339106204525}] > > When applying this same load to the un-patched Erlang 21.2, the > traditional schedulers are utilized at about 75%. So it appears that the > 2027 patch may put undesirable pressure on the schedulers as a trade-off > for consistent performance from the dirty CPU schedulers. Are you observing > a similar response on your system? > > Thanks, > Jesse > > On Wed, Jan 9, 2019 at 3:33 AM Jonas Falkevik < > jonas.falkevik@REDACTED> wrote: > >> Hi, >> This sounds like the issue I bumped into. >> After some time, many of the dirty schedulers are not performing any work >> anymore. >> Which was related to a corrupted list of sleeping dirty schedulers. >> >> I haven?t found what triggered it in my case (maybe a GC?). For me it >> took approx. 1 day before it happened. >> >> Doing Multi Scheduler Block can trigger the problem with the list of >> sleeping dirty schedulers which gets corrupted. >> The sleepers list can contain only one or some schedulers. That list is >> later used to wake up schedulers when there is work to do. >> For each enqueued dirty-cpu or dirty-io job, it wakes the first scheduler >> in the sleepers list. >> But if this list only point to a few schedulers only those are woken up >> to pick jobs from the queue. >> >> Since fixing that specific problem I have not seen the collapse of dirty >> schedulers anymore. >> >> It would be great if you can test and see if it fixes your problem as >> well, https://github.com/erlang/otp/pull/2027 >> Any feedback on the bug fix is appreciated. >> >> /Jonas >> >> > On 8 Jan 2019, at 17:13, Jesse Stimpson >> wrote: >> > >> > Hello, >> > >> > We run a large scale WebRTC-based VOIP application, and currently we >> are evaluating whether or not dirty CPU schedulers can help us get more >> consistent performance with some critical NIFs. We've created a test that >> mimics our production load and have run it on AWS EC2 instances of various >> sizes. When dirty CPU schedulers are working as expected, we do see >> measurable improvement in performance, but unfortunately we have seen that >> they can drop to 0 utilization. They do not recover from this state. >> > >> > The timings from 2 of our 12 NIFs are in the vicinity of the 1 msec >> rule of thumb, so we have experimented with them both on and off dirty CPU >> schedulers. We've left the other NIFs on the traditional schedulers. More >> detailed NIF timings can be found below. >> > >> > At the end of this message is the concerning data (from >> recon:scheduler_usage/1).We have schedulers 1-36 as the traditional >> schedulers bound to the first hardware thread of each core, and schedulers >> 37-72 are dirty CPU schedulers. Based on mpstat it appears the dirty CPU >> schedulers are not bound to threads. After about 10 minutes of running the >> load test, schedulers 42, 43, 44, 61, 63, 64, 66, 68, 72 went to at 0 >> utilization and stayed there indefinitely. As time goes on, more dirty CPU >> schedulers enter this 0 utilization state. Schedulers 1-36 do not exhibit >> this behaviour. We have reproduced this consistently on other EC2 c5 >> instances of various sizes. >> > >> > We have tried various beam adjustments, but we have seen the same >> behaviour each time. Some examples of things we have tried: >> > +sbwt very_long +sbwtdcpu very_long +swt very_low +swtdcpu very_low >> > +sfwi 500 >> > >> > Is there anything else that we can try to avoid this 0-utilization >> issue? We are happy to provide more data if needed. >> > >> > NIF: webrtc_acm_add_data_10ms >> > median: 606 usec >> > 90 percentile: 1230 usec >> > 99 percentile: 2285 usec >> > 99.9 percentile: 3223 usec >> > >> > NIF: webrtc_acm_playout_data_10ms >> > median: 98 usec >> > 90 percentile: 371 usec >> > 99 percentile: 410 usec >> > 99.9 percentile: 449 usec >> > >> > EC2 instance: c5.18xlarge >> > >> > OS: Linux 4.4.0-1073-aws #83-Ubuntu SMP Sat Nov 17 00:26:27 UTC 2018 >> x86_64 x86_64 x86_64 GNU/Linux >> > >> > beam flags: +P 4194304 +sbt db +SP 50:50 +sub true +A 128 +K true >> > >> > Erlang banner: Erlang/OTP 21 [erts-10.2] [source] [64-bit] [smp:36:36] >> [ds:36:36:10] [async-threads:128] [hipe] >> > >> > cpu_topology: >> > [{node,[{processor,[{core,[{thread,{logical,0}}, >> > {thread,{logical,36}}]}, >> > {core,[{thread,{logical,1}},{thread,{logical,37}}]}, >> > {core,[{thread,{logical,2}},{thread,{logical,38}}]}, >> > {core,[{thread,{logical,3}},{thread,{logical,39}}]}, >> > {core,[{thread,{logical,4}},{thread,{logical,40}}]}, >> > {core,[{thread,{logical,5}},{thread,{logical,41}}]}, >> > {core,[{thread,{logical,6}},{thread,{logical,42}}]}, >> > {core,[{thread,{logical,7}},{thread,{logical,43}}]}, >> > {core,[{thread,{logical,8}},{thread,{logical,44}}]}, >> > {core,[{thread,{logical,9}},{thread,{logical,45}}]}, >> > >> {core,[{thread,{logical,10}},{thread,{logical,46}}]}, >> > >> {core,[{thread,{logical,11}},{thread,{logical,47}}]}, >> > >> {core,[{thread,{logical,12}},{thread,{logical,48}}]}, >> > >> {core,[{thread,{logical,13}},{thread,{logical,49}}]}, >> > >> {core,[{thread,{logical,14}},{thread,{logical,50}}]}, >> > >> {core,[{thread,{logical,15}},{thread,{logical,51}}]}, >> > >> {core,[{thread,{logical,16}},{thread,{logical,52}}]}, >> > >> {core,[{thread,{logical,17}},{thread,{logical,53}}]}]}]}, >> > {node,[{processor,[{core,[{thread,{logical,18}}, >> > {thread,{logical,54}}]}, >> > >> {core,[{thread,{logical,19}},{thread,{logical,55}}]}, >> > >> {core,[{thread,{logical,20}},{thread,{logical,56}}]}, >> > >> {core,[{thread,{logical,21}},{thread,{logical,57}}]}, >> > >> {core,[{thread,{logical,22}},{thread,{logical,58}}]}, >> > >> {core,[{thread,{logical,23}},{thread,{logical,59}}]}, >> > >> {core,[{thread,{logical,24}},{thread,{logical,60}}]}, >> > >> {core,[{thread,{logical,25}},{thread,{logical,61}}]}, >> > >> {core,[{thread,{logical,26}},{thread,{logical,62}}]}, >> > >> {core,[{thread,{logical,27}},{thread,{logical,63}}]}, >> > >> {core,[{thread,{logical,28}},{thread,{logical,64}}]}, >> > >> {core,[{thread,{logical,29}},{thread,{logical,65}}]}, >> > >> {core,[{thread,{logical,30}},{thread,{logical,66}}]}, >> > >> {core,[{thread,{logical,31}},{thread,{logical,67}}]}, >> > >> {core,[{thread,{logical,32}},{thread,{logical,68}}]}, >> > >> {core,[{thread,{logical,33}},{thread,{logical,69}}]}, >> > >> {core,[{thread,{logical,34}},{thread,{logical,70}}]}, >> > >> {core,[{thread,{logical,35}},{thread,{logical,71}}]}]}]}] >> > >> > scheduler_bindings: >> {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22, >> > 23,24,25,26,27,28,29,30,31,32,33,34,35} >> > >> > CPU utilization (from mpstat) during our load test: >> > Linux 4.4.0-1073-aws (isw-core-x-qa-awsoh02) 01/04/2019 >> _x86_64_ (72 CPU) >> > >> > 08:55:15 PM CPU %usr %nice %sys %iowait %irq %soft >> %steal %guest %gnice %idle >> > 08:55:15 PM all 51.58 0.00 1.64 0.01 0.00 0.42 >> 0.00 0.00 0.00 46.36 >> > 08:55:15 PM 0 62.34 0.00 2.69 0.06 0.00 0.00 >> 0.00 0.00 0.00 34.90 >> > 08:55:15 PM 1 62.84 0.00 2.58 0.02 0.00 0.00 >> 0.00 0.00 0.00 34.55 >> > 08:55:15 PM 2 62.04 0.00 2.52 0.00 0.00 0.00 >> 0.00 0.00 0.00 35.44 >> > 08:55:15 PM 3 62.17 0.00 2.62 0.00 0.00 0.00 >> 0.00 0.00 0.00 35.21 >> > 08:55:15 PM 4 62.14 0.00 2.54 0.02 0.00 0.00 >> 0.00 0.00 0.00 35.30 >> > 08:55:15 PM 5 62.06 0.00 2.51 0.00 0.00 0.00 >> 0.00 0.00 0.00 35.43 >> > 08:55:15 PM 6 61.93 0.00 2.58 0.01 0.00 0.00 >> 0.02 0.00 0.00 35.46 >> > 08:55:15 PM 7 61.99 0.00 2.54 0.00 0.00 0.00 >> 0.00 0.00 0.00 35.48 >> > 08:55:15 PM 8 62.01 0.00 2.52 0.00 0.00 0.00 >> 0.00 0.00 0.00 35.47 >> > 08:55:15 PM 9 61.88 0.00 2.49 0.00 0.00 0.00 >> 0.00 0.00 0.00 35.63 >> > 08:55:15 PM 10 62.18 0.00 2.48 0.00 0.00 0.00 >> 0.00 0.00 0.00 35.34 >> > 08:55:15 PM 11 61.89 0.00 2.47 0.00 0.00 0.00 >> 0.00 0.00 0.00 35.64 >> > 08:55:15 PM 12 61.98 0.00 2.50 0.00 0.00 0.00 >> 0.00 0.00 0.00 35.52 >> > 08:55:15 PM 13 61.80 0.00 2.54 0.00 0.00 0.00 >> 0.00 0.00 0.00 35.66 >> > 08:55:15 PM 14 61.96 0.00 2.56 0.00 0.00 0.00 >> 0.00 0.00 0.00 35.48 >> > 08:55:15 PM 15 62.11 0.00 2.57 0.00 0.00 0.00 >> 0.00 0.00 0.00 35.31 >> > 08:55:15 PM 16 62.03 0.00 2.55 0.00 0.00 0.00 >> 0.00 0.00 0.00 35.42 >> > 08:55:15 PM 17 61.90 0.00 2.53 0.00 0.00 0.00 >> 0.00 0.00 0.00 35.57 >> > 08:55:15 PM 18 58.36 0.00 2.41 0.09 0.00 3.72 >> 0.00 0.00 0.00 35.43 >> > 08:55:15 PM 19 58.29 0.00 2.45 0.02 0.00 4.00 >> 0.00 0.00 0.00 35.24 >> > 08:55:15 PM 20 58.45 0.00 2.36 0.02 0.00 3.73 >> 0.00 0.00 0.00 35.44 >> > 08:55:15 PM 21 58.56 0.00 2.34 0.04 0.00 3.54 >> 0.00 0.00 0.00 35.51 >> > 08:55:15 PM 22 58.12 0.00 2.46 0.00 0.00 3.84 >> 0.00 0.00 0.00 35.58 >> > 08:55:15 PM 23 58.24 0.00 2.36 0.01 0.00 3.82 >> 0.00 0.00 0.00 35.57 >> > 08:55:15 PM 24 58.84 0.00 2.42 0.00 0.00 3.60 >> 0.00 0.00 0.00 35.14 >> > 08:55:15 PM 25 58.35 0.00 2.43 0.00 0.00 3.72 >> 0.00 0.00 0.00 35.51 >> > 08:55:15 PM 26 62.09 0.00 2.47 0.00 0.00 0.00 >> 0.00 0.00 0.00 35.45 >> > 08:55:15 PM 27 62.09 0.00 2.46 0.00 0.00 0.00 >> 0.00 0.00 0.00 35.45 >> > 08:55:15 PM 28 62.15 0.00 2.48 0.00 0.00 0.00 >> 0.00 0.00 0.00 35.37 >> > 08:55:15 PM 29 62.09 0.00 2.46 0.00 0.00 0.00 >> 0.00 0.00 0.00 35.45 >> > 08:55:15 PM 30 62.74 0.00 2.45 0.00 0.00 0.00 >> 0.00 0.00 0.00 34.81 >> > 08:55:15 PM 31 62.14 0.00 2.47 0.00 0.00 0.00 >> 0.00 0.00 0.00 35.39 >> > 08:55:15 PM 32 62.35 0.00 2.52 0.00 0.00 0.00 >> 0.00 0.00 0.00 35.13 >> > 08:55:15 PM 33 62.13 0.00 2.42 0.00 0.00 0.00 >> 0.00 0.00 0.00 35.46 >> > 08:55:15 PM 34 62.08 0.00 2.47 0.00 0.00 0.00 >> 0.00 0.00 0.00 35.45 >> > 08:55:15 PM 35 62.15 0.00 2.49 0.00 0.00 0.00 >> 0.00 0.00 0.00 35.35 >> > 08:55:15 PM 36 43.19 0.00 0.80 0.00 0.00 0.00 >> 0.00 0.00 0.00 56.01 >> > 08:55:15 PM 37 43.09 0.00 0.83 0.01 0.00 0.00 >> 0.00 0.00 0.00 56.06 >> > 08:55:15 PM 38 42.59 0.00 0.78 0.01 0.00 0.00 >> 0.00 0.00 0.00 56.63 >> > 08:55:15 PM 39 42.60 0.00 0.82 0.00 0.00 0.00 >> 0.00 0.00 0.00 56.58 >> > 08:55:15 PM 40 42.94 0.00 0.76 0.02 0.00 0.00 >> 0.00 0.00 0.00 56.28 >> > 08:55:15 PM 41 42.11 0.00 0.76 0.01 0.00 0.00 >> 0.00 0.00 0.00 57.13 >> > 08:55:15 PM 42 43.26 0.00 0.85 0.01 0.00 0.00 >> 0.00 0.00 0.00 55.88 >> > 08:55:15 PM 43 42.23 0.00 0.77 0.01 0.00 0.00 >> 0.00 0.00 0.00 56.99 >> > 08:55:15 PM 44 42.24 0.00 0.80 0.00 0.00 0.00 >> 0.00 0.00 0.00 56.95 >> > 08:55:15 PM 45 41.65 0.00 0.79 0.01 0.00 0.00 >> 0.00 0.00 0.00 57.55 >> > 08:55:15 PM 46 41.76 0.00 0.77 0.01 0.00 0.00 >> 0.00 0.00 0.00 57.46 >> > 08:55:15 PM 47 42.66 0.00 0.82 0.00 0.00 0.00 >> 0.00 0.00 0.00 56.52 >> > 08:55:15 PM 48 41.83 0.00 0.76 0.01 0.00 0.00 >> 0.00 0.00 0.00 57.40 >> > 08:55:15 PM 49 42.04 0.00 0.82 0.00 0.00 0.00 >> 0.00 0.00 0.00 57.13 >> > 08:55:15 PM 50 41.55 0.00 0.78 0.01 0.00 0.00 >> 0.00 0.00 0.00 57.67 >> > 08:55:15 PM 51 42.37 0.00 0.78 0.00 0.00 0.00 >> 0.00 0.00 0.00 56.84 >> > 08:55:15 PM 52 41.66 0.00 0.77 0.00 0.00 0.00 >> 0.00 0.00 0.00 57.57 >> > 08:55:15 PM 53 41.77 0.00 0.79 0.00 0.00 0.00 >> 0.00 0.00 0.00 57.45 >> > 08:55:15 PM 54 40.33 0.00 0.88 0.01 0.00 0.00 >> 0.00 0.00 0.00 58.79 >> > 08:55:15 PM 55 41.62 0.00 0.78 0.01 0.00 0.00 >> 0.00 0.00 0.00 57.59 >> > 08:55:15 PM 56 41.51 0.00 0.82 0.00 0.00 0.00 >> 0.00 0.00 0.00 57.67 >> > 08:55:15 PM 57 40.35 0.00 0.76 0.00 0.00 0.00 >> 0.00 0.00 0.00 58.89 >> > 08:55:15 PM 58 41.92 0.00 0.79 0.01 0.00 0.00 >> 0.00 0.00 0.00 57.28 >> > 08:55:15 PM 59 41.48 0.00 0.80 0.00 0.00 0.00 >> 0.00 0.00 0.00 57.72 >> > 08:55:15 PM 60 40.83 0.00 0.75 0.01 0.00 0.00 >> 0.00 0.00 0.00 58.41 >> > 08:55:15 PM 61 40.95 0.00 0.78 0.01 0.00 0.00 >> 0.00 0.00 0.00 58.26 >> > 08:55:15 PM 62 40.85 0.00 0.80 0.01 0.00 0.00 >> 0.00 0.00 0.00 58.35 >> > 08:55:15 PM 63 42.11 0.00 0.77 0.01 0.00 0.00 >> 0.00 0.00 0.00 57.12 >> > 08:55:15 PM 64 42.22 0.00 0.78 0.01 0.00 0.00 >> 0.00 0.00 0.00 56.99 >> > 08:55:15 PM 65 41.55 0.00 0.78 0.00 0.00 0.00 >> 0.00 0.00 0.00 57.67 >> > 08:55:15 PM 66 41.68 0.00 0.81 0.00 0.00 0.00 >> 0.00 0.00 0.00 57.51 >> > 08:55:15 PM 67 41.06 0.00 0.87 0.01 0.00 0.00 >> 0.00 0.00 0.00 58.07 >> > 08:55:15 PM 68 41.80 0.00 0.79 0.00 0.00 0.00 >> 0.00 0.00 0.00 57.41 >> > 08:55:15 PM 69 41.38 0.00 0.75 0.00 0.00 0.00 >> 0.00 0.00 0.00 57.87 >> > 08:55:15 PM 70 42.17 0.00 0.75 0.00 0.00 0.00 >> 0.00 0.00 0.00 57.08 >> > 08:55:15 PM 71 41.26 0.00 0.81 0.00 0.00 0.00 >> 0.01 0.00 0.00 57.92 >> > >> > >> > 1> rp(recon:scheduler_usage(1000)). >> > [{1,0.8581586716045433}, >> > {2,0.8607597847736659}, >> > {3,0.8501543449289042}, >> > {4,0.8619718426658023}, >> > {5,0.8660226578705742}, >> > {6,0.8555763933364806}, >> > {7,0.8557195972097537}, >> > {8,0.8668284625654983}, >> > {9,0.858588675506532}, >> > {10,0.8650501369640686}, >> > {11,0.8615557214275474}, >> > {12,0.8523392789683996}, >> > {13,0.8469329348407398}, >> > {14,0.8615637534991641}, >> > {15,0.8591930739986557}, >> > {16,0.8578596164201188}, >> > {17,0.8549696086049403}, >> > {18,0.8677098230685564}, >> > {19,0.8556328340054443}, >> > {20,0.8642542459479063}, >> > {21,0.8670655179064826}, >> > {22,0.8562638308551557}, >> > {23,0.8511202058055239}, >> > {24,0.8697339264798696}, >> > {25,0.8594823462601853}, >> > {26,0.8634550008001265}, >> > {27,0.8568590130172198}, >> > {28,0.864025757952233}, >> > {29,0.8575614770622326}, >> > {30,0.8611760239143281}, >> > {31,0.8589789125669829}, >> > {32,0.8537754468447621}, >> > {33,0.8675538912040871}, >> > {34,0.8607522803519515}, >> > {35,0.8639389099445753}, >> > {36,0.8743759133171144}, >> > {37,0.9526507413773687}, >> > {38,0.948925611917577}, >> > {39,0.9511904612810083}, >> > {40,0.9581015421004746}, >> > {41,0.9534977201467516}, >> > {42,0.0}, >> > {43,0.0}, >> > {44,0.0}, >> > {45,0.9522518499948341}, >> > {46,0.9453306755503674}, >> > {47,0.9586374601140047}, >> > {48,0.9574611845671873}, >> > {49,0.950914304057791}, >> > {50,0.9582124294398642}, >> > {51,0.9448074463900029}, >> > {52,0.9474771999511407}, >> > {53,0.9450778098401229}, >> > {54,0.0}, >> > {55,0.9478167911711937}, >> > {56,0.9612587860764551}, >> > {57,0.943468696358506}, >> > {58,0.9509498642900455}, >> > {59,0.9603595134495414}, >> > {60,0.9517397912369205}, >> > {61,0.0}, >> > {62,0.9587029756970658}, >> > {63,0.0}, >> > {64,0.0}, >> > {65,0.9526167902065421}, >> > {66,0.0}, >> > {67,0.9475668680715805}, >> > {68,0.0}, >> > {69,0.9630301804839606}, >> > {70,0.9475359592879946}, >> > {71,0.9541718747681374}, >> > {72,0.0}] >> > >> > >> > Thanks for your time, >> > >> > Jesse Stimpson >> > _______________________________________________ >> > erlang-questions mailing list >> > erlang-questions@REDACTED >> > http://erlang.org/mailman/listinfo/erlang-questions >> >> > > -- > > > > Jesse Stimpson > > Site Reliability Engineering > > m: 9199950424 <(919)%20995-0424> > RepublicWireless.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From icfp.publicity@REDACTED Thu Jan 10 02:50:32 2019 From: icfp.publicity@REDACTED (Sam Tobin-Hochstadt) Date: Wed, 09 Jan 2019 20:50:32 -0500 Subject: [erlang-questions] Second Call for Papers: PACMPL issue ICFP 2019 Message-ID: <5c36a4e8ba81f_64534650dc1859c@hermes.mail> PACMPL Volume 3, Issue ICFP 2019 Call for Papers accepted papers to be invited for presentation at The 24th ACM SIGPLAN International Conference on Functional Programming Berlin, Germany http://icfp19.sigplan.org/ ### Important dates Submissions due: 1 March 2019 (Friday) Anywhere on Earth https://icfp19.hotcrp.com Author response: 16 April (Tuesday) - 18 Apri (Friday) 14:00 UTC Notification: 3 May (Friday) Final copy due: 22 June (Saturday) Conference: 18 August (Sunday) - 23 August (Friday) ### About PACMPL Proceedings of the ACM on Programming Languages (PACMPL ) is a Gold Open Access journal publishing research on all aspects of programming languages, from design to implementation and from mathematical formalisms to empirical studies. Each issue of the journal is devoted to a particular subject area within programming languages and will be announced through publicized Calls for Papers, like this one. ### Scope [PACMPL](https://pacmpl.acm.org/) issue ICFP 2019 seeks original papers on the art and science of functional programming. Submissions are invited on all topics from principles to practice, from foundations to features, and from abstraction to application. The scope includes all languages that encourage functional programming, including both purely applicative and imperative languages, as well as languages with objects, concurrency, or parallelism. Topics of interest include (but are not limited to): * *Language Design*: concurrency, parallelism, and distribution; modules; components and composition; metaprogramming; type systems; interoperability; domain-specific languages; and relations to imperative, object-oriented, or logic programming. * *Implementation*: abstract machines; virtual machines; interpretation; compilation; compile-time and run-time optimization; garbage collection and memory management; multi-threading; exploiting parallel hardware; interfaces to foreign functions, services, components, or low-level machine resources. * *Software-Development Techniques*: algorithms and data structures; design patterns; specification; verification; validation; proof assistants; debugging; testing; tracing; profiling. * *Foundations*: formal semantics; lambda calculus; rewriting; type theory; monads; continuations; control; state; effects; program verification; dependent types. * *Analysis and Transformation*: control-flow; data-flow; abstract interpretation; partial evaluation; program calculation. * *Applications*: symbolic computing; formal-methods tools; artificial intelligence; systems programming; distributed-systems and web programming; hardware design; databases; XML processing; scientific and numerical computing; graphical user interfaces; multimedia and 3D graphics programming; scripting; system administration; security. * *Education*: teaching introductory programming; parallel programming; mathematical proof; algebra. Submissions will be evaluated according to their relevance, correctness, significance, originality, and clarity. Each submission should explain its contributions in both general and technical terms, clearly identifying what has been accomplished, explaining why it is significant, and comparing it with previous work. The technical content should be accessible to a broad audience. PACMPL issue ICFP 2019 also welcomes submissions in two separate categories — Functional Pearls and Experience Reports — that must be marked as such at the time of submission and that need not report original research results. Detailed guidelines on both categories are given at the end of this call. Please contact the principal editor if you have questions or are concerned about the appropriateness of a topic. ### Preparation of submissions **Deadline**: The deadline for submissions is **Friday, March 1, 2019**, Anywhere on Earth (). This deadline will be strictly enforced. **Formatting**: Submissions must be in PDF format, printable in black and white on US Letter sized paper, and interpretable by common PDF tools. All submissions must adhere to the "ACM Small" template that is available (in both LaTeX and Word formats) from . For authors using LaTeX, a lighter-weight package, including only the essential files, is available from . There is a limit of **25 pages for a full paper or Functional Pearl** and **12 pages for an Experience Report**; in either case, the bibliography will not be counted against these limits. Submissions that exceed the page limits or, for other reasons, do not meet the requirements for formatting, will be summarily rejected. Supplementary material can and should be **separately** submitted (see below). See also PACMPL's Information and Guidelines for Authors at . **Submission**: Submissions will be accepted at Improved versions of a paper may be submitted at any point before the submission deadline using the same web interface. **Author Response Period**: Authors will have a 72-hour period, starting at 14:00 UTC on **Tuesday, April 16, 2019**, to read reviews and respond to them. **Supplementary Material**: Authors have the option to attach supplementary material to a submission, on the understanding that reviewers may choose not to look at it. This supplementary material should **not** be submitted as part of the main document; instead, it should be uploaded as a **separate** PDF document or tarball. Supplementary material should be uploaded **at submission time**, not by providing a URL in the paper that points to an external repository. Authors are free to upload both anonymized and non-anonymized supplementary material. Anonymized supplementary material will be visible to reviewers immediately; non-anonymized supplementary material will be revealed to reviewers only after they have submitted their review of the paper and learned the identity of the author(s). **Authorship Policies**: All submissions are expected to comply with the ACM Policies for Authorship that are detailed at . **Republication Policies**: Each submission must adhere to SIGPLAN's republication policy, as explained on the web at . **Resubmitted Papers**: Authors who submit a revised version of a paper that has previously been rejected by another conference have the option to attach an annotated copy of the reviews of their previous submission(s), explaining how they have addressed these previous reviews in the present submission. If a reviewer identifies him/herself as a reviewer of this previous submission and wishes to see how his/her comments have been addressed, the principal editor will communicate to this reviewer the annotated copy of his/her previous review. Otherwise, no reviewer will read the annotated copies of the previous reviews. ### Review Process This section outlines the two-stage process with lightweight double-blind reviewing that will be used to select papers for PACMPL issue ICFP 2019. We anticipate that there will be a need to clarify and expand on this process, and we will maintain a list of frequently asked questions and answers on the conference website to address common concerns. **PACMPL issue ICFP 2019 will employ a two-stage review process.** The first stage in the review process will assess submitted papers using the criteria stated above and will allow for feedback and input on initial reviews through the author response period mentioned previously. At the review meeting, a set of papers will be conditionally accepted and all other papers will be rejected. Authors will be notified of these decisions on **May 3, 2019**. Authors of conditionally accepted papers will be provided with committee reviews (just as in previous conferences) along with a set of mandatory revisions. After four weeks (May 31, 2019), the authors will provide a second submission. The second and final reviewing phase assesses whether the mandatory revisions have been adequately addressed by the authors and thereby determines the final accept/reject status of the paper. The intent and expectation is that the mandatory revisions can be addressed within four weeks and hence that conditionally accepted papers will in general be accepted in the second phase. The second submission should clearly identify how the mandatory revisions were addressed. To that end, the second submission must be accompanied by a cover letter mapping each mandatory revision request to specific parts of the paper. The cover letter will facilitate a quick second review, allowing for confirmation of final acceptance within two weeks. Conversely, the absence of a cover letter will be grounds for the paper?s rejection. **PACMPL issue ICFP 2019 will employ a lightweight double-blind reviewing process.** To facilitate this, submitted papers must adhere to two rules: 1. **author names and institutions must be omitted**, and 2. **references to authors' own related work should be in the third person** (e.g., not "We build on our previous work ..." but rather "We build on the work of ..."). The purpose of this process is to help the reviewers come to an initial judgement about the paper without bias, not to make it impossible for them to discover the authors if they were to try. Nothing should be done in the name of anonymity that weakens the submission or makes the job of reviewing the paper more difficult (e.g., important background references should not be omitted or anonymized). In addition, authors should feel free to disseminate their ideas or draft versions of their paper as they normally would. For instance, authors may post drafts of their papers on the web or give talks on their research ideas. ### Information for Authors of Accepted Papers * As a condition of acceptance, final versions of all papers must adhere to the new ACM Small format. The page limit for the final versions of papers will be increased by two pages to help authors respond to reviewer comments and mandatory revisions: **27 pages plus bibliography for a regular paper or Functional Pearl, 14 pages plus bibliography for an Experience Report**. * Authors of accepted submissions will be required to agree to one of the three ACM licensing options: open access on payment of a fee (**recommended**, and SIGPLAN can cover the cost as described next); copyright transfer to ACM; or retaining copyright but granting ACM exclusive publication rights. Further information about ACM author rights is available from . * PACMPL is a Gold Open Access journal. It will be archived in ACM?s Digital Library, but no membership or fee is required for access. Gold Open Access has been made possible by generous funding through ACM SIGPLAN, which will cover all open access costs in the event authors cannot. Authors who can cover the costs may do so by paying an Article Processing Charge (APC). PACMPL, SIGPLAN, and ACM Headquarters are committed to exploring routes to making Gold Open Access publication both affordable and sustainable. * ACM offers authors a range of copyright options, one of which is Creative Commons CC-BY publication; this is the option recommended by the PACMPL editorial board. A reasoned argument in favour of this option can be found in the article [Why CC-BY?](https://oaspa.org/why-cc-by/) published by OASPA, the Open Access Scholarly Publishers Association. * We intend that the papers will be freely available for download from the ACM Digital Library in perpetuity via the OpenTOC mechanism. * ACM Author-Izer is a unique service that enables ACM authors to generate and post links on either their home page or institutional repository for visitors to download the definitive version of their articles from the ACM Digital Library at no charge. Downloads through Author-Izer links are captured in official ACM statistics, improving the accuracy of usage and impact measurements. Consistently linking to the definitive version of an ACM article should reduce user confusion over article versioning. After an article has been published and assigned to the appropriate ACM Author Profile pages, authors should visit to learn how to create links for free downloads from the ACM DL. * At least one author of each accepted submissions will be expected to attend and present their paper at the conference. The schedule for presentations will be determined and shared with authors after the full program has been selected. Presentations will be videotaped and released online if the presenter consents. * The official publication date is the date the papers are made available in the ACM Digital Library. This date may be up to *two weeks prior* to the first day of the conference. The official publication date affects the deadline for any patent filings related to published work. ### Artifact Evaluation Authors of papers that are conditionally accepted in the first phase of the review process will be encouraged (but not required) to submit supporting materials for Artifact Evaluation. These items will then be reviewed by an Artifact Evaluation Committee, separate from the paper Review Committee, whose task is to assess how the artifacts support the work described in the associated paper. Papers that go through the Artifact Evaluation process successfully will receive a seal of approval printed on the papers themselves. Authors of accepted papers will be encouraged to make the supporting materials publicly available upon publication of the papers, for example, by including them as "source materials" in the ACM Digital Library. An additional seal will mark papers whose artifacts are made available, as outlined in the ACM guidelines for artifact badging. Participation in Artifact Evaluation is voluntary and will not influence the final decision regarding paper acceptance. ### Special categories of papers In addition to research papers, PACMPL issue ICFP solicits two kinds of papers that do not require original research contributions: Functional Pearls, which are full papers, and Experience Reports, which are limited to half the length of a full paper. Authors submitting such papers should consider the following guidelines. #### Functional Pearls A Functional Pearl is an elegant essay about something related to functional programming. Examples include, but are not limited to: * a new and thought-provoking way of looking at an old idea * an instructive example of program calculation or proof * a nifty presentation of an old or new data structure * an interesting application of functional programming techniques * a novel use or exposition of functional programming in the classroom While pearls often demonstrate an idea through the development of a short program, there is no requirement or expectation that they do so. Thus, they encompass the notions of theoretical and educational pearls. Functional Pearls are valued as highly and judged as rigorously as ordinary papers, but using somewhat different criteria. In particular, a pearl is not required to report original research, but, it should be concise, instructive, and entertaining. A pearl is likely to be rejected if its readers get bored, if the material gets too complicated, if too much specialized knowledge is needed, or if the writing is inelegant. The key to writing a good pearl is polishing. A submission that is intended to be treated as a pearl must be marked as such on the submission web page, and should contain the words "Functional Pearl" somewhere in its title or subtitle. These steps will alert reviewers to use the appropriate evaluation criteria. Pearls will be combined with ordinary papers, however, for the purpose of computing the conference's acceptance rate. #### Experience Reports The purpose of an Experience Report is to help create a body of published, refereed, citable evidence that functional programming really works — or to describe what obstacles prevent it from working. Possible topics for an Experience Report include, but are not limited to: * insights gained from real-world projects using functional programming * comparison of functional programming with conventional programming in the context of an industrial project or a university curriculum * project-management, business, or legal issues encountered when using functional programming in a real-world project * curricular issues encountered when using functional programming in education * real-world constraints that created special challenges for an implementation of a functional language or for functional programming in general An Experience Report is distinguished from a normal PACMPL issue ICFP paper by its title, by its length, and by the criteria used to evaluate it. * Both in the papers and in any citations, the title of each accepted Experience Report must end with the words "(Experience Report)" in parentheses. The acceptance rate for Experience Reports will be computed and reported separately from the rate for ordinary papers. * Experience Report submissions can be at most 12 pages long, excluding bibliography. * Each accepted Experience Report will be presented at the conference, but depending on the number of Experience Reports and regular papers accepted, authors of Experience reports may be asked to give shorter talks. * Because the purpose of Experience Reports is to enable our community to accumulate a body of evidence about the efficacy of functional programming, an acceptable Experience Report need not add to the body of knowledge of the functional-programming community by presenting novel results or conclusions. It is sufficient if the Report states a clear thesis and provides supporting evidence. The thesis must be relevant to ICFP, but it need not be novel. The review committee will accept or reject Experience Reports based on whether they judge the evidence to be convincing. Anecdotal evidence will be acceptable provided it is well argued and the author explains what efforts were made to gather as much evidence as possible. Typically, more convincing evidence is obtained from papers which show how functional programming was used than from papers which only say that functional programming was used. The most convincing evidence often includes comparisons of situations before and after the introduction or discontinuation of functional programming. Evidence drawn from a single person's experience may be sufficient, but more weight will be given to evidence drawn from the experience of groups of people. An Experience Report should be short and to the point: it should make a claim about how well functional programming worked on a particular project and why, and produce evidence to substantiate this claim. If functional programming worked in this case in the same ways it has worked for others, the paper need only summarize the results — the main part of the paper should discuss how well it worked and in what context. Most readers will not want to know all the details of the project and its implementation, but the paper should characterize the project and its context well enough so that readers can judge to what degree this experience is relevant to their own projects. The paper should take care to highlight any unusual aspects of the project. Specifics about the project are more valuable than generalities about functional programming; for example, it is more valuable to say that the team delivered its software a month ahead of schedule than it is to say that functional programming made the team more productive. If the paper not only describes experience but also presents new technical results, or if the experience refutes cherished beliefs of the functional-programming community, it may be better to submit it as a full paper, which will be judged by the usual criteria of novelty, originality, and relevance. The principal editor will be happy to advise on any concerns about which category to submit to. ### ICFP Organizers General Chair: Derek Dreyer (MPI-SWS, Germany) Artifact Evaluation Co-Chairs: Simon Marlow (Facebook, UK) Industrial Relations Chair: Alan Jeffrey (Mozilla Research, USA) Programming Contest Organiser: Ilya Sergey (Yale-NUS College, Singapore) Publicity and Web Chair: Sam Tobin-Hochstadt (Indiana University, USA) Student Research Competition Chair: William J. Bowman (University of British Columbia, Canada) Workshops Co-Chair: Christophe Scholliers (Universiteit Gent, Belgium) Jennifer Hackett (University of Nottingham, UK) Conference Manager: Annabel Satin (P.C.K.) ### PACMPL Volume 3, Issue ICFP 2019 Principal Editor: Fran?ois Pottier (Inria, France) Review Committee: Lennart Beringer (Princeton University, United States) Joachim Breitner (DFINITY Foundation, Germany) Laura M. Castro (University of A Coru?a, Spain) Ezgi ?i?ek (Facebook London, United Kingdom) Pierre-Evariste Dagand (LIP6/CNRS, France) Christos Dimoulas (Northwestern University, United States) Jacques-Henri Jourdan (CNRS, LRI, Universit? Paris-Sud, France) Andrew Kennedy (Facebook London, United Kingdom) Daan Leijen (Microsoft Research, United States) Kazutaka Matsuda (Tohoku University, Japan) Bruno C. d. S. Oliveira (University of Hong Kong, China) Klaus Ostermann (University of T?bingen, Germany) Jennifer Paykin (Galois, United States) Frank Pfenning (Carnegie Mellon University, USA) Mike Rainey (Indiana University, USA) Chung-chieh Shan (Indiana University, USA) Sam Staton (University of Oxford, UK) Pierre-Yves Strub (Ecole Polytechnique, France) German Vidal (Universitat Politecnica de Valencia, Spain) External Review Committee: Michael D. Adams (University of Utah, USA) Robert Atkey (University of Strathclyde, IK) Sheng Chen (University of Louisiana at Lafayette, USA) James Cheney (University of Edinburgh, UK) Adam Chlipala (Massachusetts Institute of Technology, USA) Evelyne Contejean (LRI, Universit? Paris-Sud, France) Germ?n Andr?s Delbianco (IRIF, Universit? Paris Diderot, France) Dominique Devriese (Vrije Universiteit Brussel, Belgium) Richard A. Eisenberg (Bryn Mawr College, USA) Conal Elliott (Target, USA) Sebastian Erdweg (Delft University of Technology, Netherlands) Michael Greenberg (Pomona College, USA) Adrien Guatto (IRIF, Universit? Paris Diderot, France) Jennifer Hackett (University of Nottingham, UK) Troels Henriksen (University of Copenhagen, Denmark) Chung-Kil Hur (Seoul National University, Republic of Korea) Roberto Ierusalimschy (PUC-Rio, Brazil) Ranjit Jhala (University of California, San Diego, USA) Ralf Jung (MPI-SWS, Germany) Ohad Kammar (University of Oxford, UK) Oleg Kiselyov (Tohoku University, Japan) Hsiang-Shang ?Josh? Ko (National Institute of Informatics, Japan) Ond?ej Lhot?k (University of Waterloo, Canada) Dan Licata (Wesleyan University, USA) Geoffrey Mainland (Drexel University, USA) Simon Marlow (Facebook, UK) Akimasa Morihata (University of Tokyo, Japan) Shin-Cheng Mu (Academia Sinica, Taiwan) Guillaume Munch-Maccagnoni (Inria, France) Kim Nguy?n (University of Paris-Sud, France) Ulf Norell (Gothenburg University, Sweden) Atsushi Ohori (Tohoku University, Japan) Rex Page (University of Oklahoma, USA) Zoe Paraskevopoulou (Princeton University, USA) Nadia Polikarpova (University of California, San Diego, USA) Jonathan Protzenko (Microsoft Research, USA) Tiark Rompf (Purdue University, USA) Andreas Rossberg (Dfinity, Germany) KC Sivaramakrishnan (University of Cambridge, UI) Nicholas Smallbone (Chalmers University of Technology, Sweden) Matthieu Sozeau (Inria, France) Sandro Stucki (Chalmers | University of Gothenburg, Sweden) Don Syme (Microsoft, UK) Zachary Tatlock (University of Washington, USA) Sam Tobin-Hochstadt (Indiana University, USA) Takeshi Tsukada (University of Tokyo, Japan) Tarmo Uustalu (Reykjavik University, Iceland) Benoit Valiron (LRI, CentraleSupelec, Univ. Paris Saclay, France) Daniel Winograd-Cort (University of Pennsylvania, USA) Nicolas Wu (University of Bristol, UK) From jstimpson@REDACTED Wed Jan 9 22:51:57 2019 From: jstimpson@REDACTED (Jesse Stimpson) Date: Wed, 9 Jan 2019 16:51:57 -0500 Subject: [erlang-questions] Dirty CPU schedulers stuck at zero utilization In-Reply-To: References: Message-ID: Rikard and Jonas, Thank you for your response. With the patch at OTP PR-2027, we do observe that the dirty CPU schedulers do not drop to zero utilization, but in addition we see that the performance of the traditional schedulers suffers. Specifically, when we apply our load test, scheduler usage toggles in between the following two states. I've snipped the output of similar values for readability. [{1,0.8933020805456553}, ... {36,0.8882068476700309}, {37,0.4932569692588096}, ... {72,0.49842881346116297}] and [{1,1.0}, ... {36,1.0}, {37,0.42909590032946493}, ... {72,0.4641339106204525}] When applying this same load to the un-patched Erlang 21.2, the traditional schedulers are utilized at about 75%. So it appears that the 2027 patch may put undesirable pressure on the schedulers as a trade-off for consistent performance from the dirty CPU schedulers. Are you observing a similar response on your system? Thanks, Jesse On Wed, Jan 9, 2019 at 3:33 AM Jonas Falkevik wrote: > Hi, > This sounds like the issue I bumped into. > After some time, many of the dirty schedulers are not performing any work > anymore. > Which was related to a corrupted list of sleeping dirty schedulers. > > I haven?t found what triggered it in my case (maybe a GC?). For me it took > approx. 1 day before it happened. > > Doing Multi Scheduler Block can trigger the problem with the list of > sleeping dirty schedulers which gets corrupted. > The sleepers list can contain only one or some schedulers. That list is > later used to wake up schedulers when there is work to do. > For each enqueued dirty-cpu or dirty-io job, it wakes the first scheduler > in the sleepers list. > But if this list only point to a few schedulers only those are woken up to > pick jobs from the queue. > > Since fixing that specific problem I have not seen the collapse of dirty > schedulers anymore. > > It would be great if you can test and see if it fixes your problem as > well, https://github.com/erlang/otp/pull/2027 > Any feedback on the bug fix is appreciated. > > /Jonas > > > On 8 Jan 2019, at 17:13, Jesse Stimpson > wrote: > > > > Hello, > > > > We run a large scale WebRTC-based VOIP application, and currently we are > evaluating whether or not dirty CPU schedulers can help us get more > consistent performance with some critical NIFs. We've created a test that > mimics our production load and have run it on AWS EC2 instances of various > sizes. When dirty CPU schedulers are working as expected, we do see > measurable improvement in performance, but unfortunately we have seen that > they can drop to 0 utilization. They do not recover from this state. > > > > The timings from 2 of our 12 NIFs are in the vicinity of the 1 msec rule > of thumb, so we have experimented with them both on and off dirty CPU > schedulers. We've left the other NIFs on the traditional schedulers. More > detailed NIF timings can be found below. > > > > At the end of this message is the concerning data (from > recon:scheduler_usage/1).We have schedulers 1-36 as the traditional > schedulers bound to the first hardware thread of each core, and schedulers > 37-72 are dirty CPU schedulers. Based on mpstat it appears the dirty CPU > schedulers are not bound to threads. After about 10 minutes of running the > load test, schedulers 42, 43, 44, 61, 63, 64, 66, 68, 72 went to at 0 > utilization and stayed there indefinitely. As time goes on, more dirty CPU > schedulers enter this 0 utilization state. Schedulers 1-36 do not exhibit > this behaviour. We have reproduced this consistently on other EC2 c5 > instances of various sizes. > > > > We have tried various beam adjustments, but we have seen the same > behaviour each time. Some examples of things we have tried: > > +sbwt very_long +sbwtdcpu very_long +swt very_low +swtdcpu very_low > > +sfwi 500 > > > > Is there anything else that we can try to avoid this 0-utilization > issue? We are happy to provide more data if needed. > > > > NIF: webrtc_acm_add_data_10ms > > median: 606 usec > > 90 percentile: 1230 usec > > 99 percentile: 2285 usec > > 99.9 percentile: 3223 usec > > > > NIF: webrtc_acm_playout_data_10ms > > median: 98 usec > > 90 percentile: 371 usec > > 99 percentile: 410 usec > > 99.9 percentile: 449 usec > > > > EC2 instance: c5.18xlarge > > > > OS: Linux 4.4.0-1073-aws #83-Ubuntu SMP Sat Nov 17 00:26:27 UTC 2018 > x86_64 x86_64 x86_64 GNU/Linux > > > > beam flags: +P 4194304 +sbt db +SP 50:50 +sub true +A 128 +K true > > > > Erlang banner: Erlang/OTP 21 [erts-10.2] [source] [64-bit] [smp:36:36] > [ds:36:36:10] [async-threads:128] [hipe] > > > > cpu_topology: > > [{node,[{processor,[{core,[{thread,{logical,0}}, > > {thread,{logical,36}}]}, > > {core,[{thread,{logical,1}},{thread,{logical,37}}]}, > > {core,[{thread,{logical,2}},{thread,{logical,38}}]}, > > {core,[{thread,{logical,3}},{thread,{logical,39}}]}, > > {core,[{thread,{logical,4}},{thread,{logical,40}}]}, > > {core,[{thread,{logical,5}},{thread,{logical,41}}]}, > > {core,[{thread,{logical,6}},{thread,{logical,42}}]}, > > {core,[{thread,{logical,7}},{thread,{logical,43}}]}, > > {core,[{thread,{logical,8}},{thread,{logical,44}}]}, > > {core,[{thread,{logical,9}},{thread,{logical,45}}]}, > > {core,[{thread,{logical,10}},{thread,{logical,46}}]}, > > {core,[{thread,{logical,11}},{thread,{logical,47}}]}, > > {core,[{thread,{logical,12}},{thread,{logical,48}}]}, > > {core,[{thread,{logical,13}},{thread,{logical,49}}]}, > > {core,[{thread,{logical,14}},{thread,{logical,50}}]}, > > {core,[{thread,{logical,15}},{thread,{logical,51}}]}, > > {core,[{thread,{logical,16}},{thread,{logical,52}}]}, > > > {core,[{thread,{logical,17}},{thread,{logical,53}}]}]}]}, > > {node,[{processor,[{core,[{thread,{logical,18}}, > > {thread,{logical,54}}]}, > > {core,[{thread,{logical,19}},{thread,{logical,55}}]}, > > {core,[{thread,{logical,20}},{thread,{logical,56}}]}, > > {core,[{thread,{logical,21}},{thread,{logical,57}}]}, > > {core,[{thread,{logical,22}},{thread,{logical,58}}]}, > > {core,[{thread,{logical,23}},{thread,{logical,59}}]}, > > {core,[{thread,{logical,24}},{thread,{logical,60}}]}, > > {core,[{thread,{logical,25}},{thread,{logical,61}}]}, > > {core,[{thread,{logical,26}},{thread,{logical,62}}]}, > > {core,[{thread,{logical,27}},{thread,{logical,63}}]}, > > {core,[{thread,{logical,28}},{thread,{logical,64}}]}, > > {core,[{thread,{logical,29}},{thread,{logical,65}}]}, > > {core,[{thread,{logical,30}},{thread,{logical,66}}]}, > > {core,[{thread,{logical,31}},{thread,{logical,67}}]}, > > {core,[{thread,{logical,32}},{thread,{logical,68}}]}, > > {core,[{thread,{logical,33}},{thread,{logical,69}}]}, > > {core,[{thread,{logical,34}},{thread,{logical,70}}]}, > > > {core,[{thread,{logical,35}},{thread,{logical,71}}]}]}]}] > > > > scheduler_bindings: > {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22, > > 23,24,25,26,27,28,29,30,31,32,33,34,35} > > > > CPU utilization (from mpstat) during our load test: > > Linux 4.4.0-1073-aws (isw-core-x-qa-awsoh02) 01/04/2019 > _x86_64_ (72 CPU) > > > > 08:55:15 PM CPU %usr %nice %sys %iowait %irq %soft > %steal %guest %gnice %idle > > 08:55:15 PM all 51.58 0.00 1.64 0.01 0.00 0.42 > 0.00 0.00 0.00 46.36 > > 08:55:15 PM 0 62.34 0.00 2.69 0.06 0.00 0.00 > 0.00 0.00 0.00 34.90 > > 08:55:15 PM 1 62.84 0.00 2.58 0.02 0.00 0.00 > 0.00 0.00 0.00 34.55 > > 08:55:15 PM 2 62.04 0.00 2.52 0.00 0.00 0.00 > 0.00 0.00 0.00 35.44 > > 08:55:15 PM 3 62.17 0.00 2.62 0.00 0.00 0.00 > 0.00 0.00 0.00 35.21 > > 08:55:15 PM 4 62.14 0.00 2.54 0.02 0.00 0.00 > 0.00 0.00 0.00 35.30 > > 08:55:15 PM 5 62.06 0.00 2.51 0.00 0.00 0.00 > 0.00 0.00 0.00 35.43 > > 08:55:15 PM 6 61.93 0.00 2.58 0.01 0.00 0.00 > 0.02 0.00 0.00 35.46 > > 08:55:15 PM 7 61.99 0.00 2.54 0.00 0.00 0.00 > 0.00 0.00 0.00 35.48 > > 08:55:15 PM 8 62.01 0.00 2.52 0.00 0.00 0.00 > 0.00 0.00 0.00 35.47 > > 08:55:15 PM 9 61.88 0.00 2.49 0.00 0.00 0.00 > 0.00 0.00 0.00 35.63 > > 08:55:15 PM 10 62.18 0.00 2.48 0.00 0.00 0.00 > 0.00 0.00 0.00 35.34 > > 08:55:15 PM 11 61.89 0.00 2.47 0.00 0.00 0.00 > 0.00 0.00 0.00 35.64 > > 08:55:15 PM 12 61.98 0.00 2.50 0.00 0.00 0.00 > 0.00 0.00 0.00 35.52 > > 08:55:15 PM 13 61.80 0.00 2.54 0.00 0.00 0.00 > 0.00 0.00 0.00 35.66 > > 08:55:15 PM 14 61.96 0.00 2.56 0.00 0.00 0.00 > 0.00 0.00 0.00 35.48 > > 08:55:15 PM 15 62.11 0.00 2.57 0.00 0.00 0.00 > 0.00 0.00 0.00 35.31 > > 08:55:15 PM 16 62.03 0.00 2.55 0.00 0.00 0.00 > 0.00 0.00 0.00 35.42 > > 08:55:15 PM 17 61.90 0.00 2.53 0.00 0.00 0.00 > 0.00 0.00 0.00 35.57 > > 08:55:15 PM 18 58.36 0.00 2.41 0.09 0.00 3.72 > 0.00 0.00 0.00 35.43 > > 08:55:15 PM 19 58.29 0.00 2.45 0.02 0.00 4.00 > 0.00 0.00 0.00 35.24 > > 08:55:15 PM 20 58.45 0.00 2.36 0.02 0.00 3.73 > 0.00 0.00 0.00 35.44 > > 08:55:15 PM 21 58.56 0.00 2.34 0.04 0.00 3.54 > 0.00 0.00 0.00 35.51 > > 08:55:15 PM 22 58.12 0.00 2.46 0.00 0.00 3.84 > 0.00 0.00 0.00 35.58 > > 08:55:15 PM 23 58.24 0.00 2.36 0.01 0.00 3.82 > 0.00 0.00 0.00 35.57 > > 08:55:15 PM 24 58.84 0.00 2.42 0.00 0.00 3.60 > 0.00 0.00 0.00 35.14 > > 08:55:15 PM 25 58.35 0.00 2.43 0.00 0.00 3.72 > 0.00 0.00 0.00 35.51 > > 08:55:15 PM 26 62.09 0.00 2.47 0.00 0.00 0.00 > 0.00 0.00 0.00 35.45 > > 08:55:15 PM 27 62.09 0.00 2.46 0.00 0.00 0.00 > 0.00 0.00 0.00 35.45 > > 08:55:15 PM 28 62.15 0.00 2.48 0.00 0.00 0.00 > 0.00 0.00 0.00 35.37 > > 08:55:15 PM 29 62.09 0.00 2.46 0.00 0.00 0.00 > 0.00 0.00 0.00 35.45 > > 08:55:15 PM 30 62.74 0.00 2.45 0.00 0.00 0.00 > 0.00 0.00 0.00 34.81 > > 08:55:15 PM 31 62.14 0.00 2.47 0.00 0.00 0.00 > 0.00 0.00 0.00 35.39 > > 08:55:15 PM 32 62.35 0.00 2.52 0.00 0.00 0.00 > 0.00 0.00 0.00 35.13 > > 08:55:15 PM 33 62.13 0.00 2.42 0.00 0.00 0.00 > 0.00 0.00 0.00 35.46 > > 08:55:15 PM 34 62.08 0.00 2.47 0.00 0.00 0.00 > 0.00 0.00 0.00 35.45 > > 08:55:15 PM 35 62.15 0.00 2.49 0.00 0.00 0.00 > 0.00 0.00 0.00 35.35 > > 08:55:15 PM 36 43.19 0.00 0.80 0.00 0.00 0.00 > 0.00 0.00 0.00 56.01 > > 08:55:15 PM 37 43.09 0.00 0.83 0.01 0.00 0.00 > 0.00 0.00 0.00 56.06 > > 08:55:15 PM 38 42.59 0.00 0.78 0.01 0.00 0.00 > 0.00 0.00 0.00 56.63 > > 08:55:15 PM 39 42.60 0.00 0.82 0.00 0.00 0.00 > 0.00 0.00 0.00 56.58 > > 08:55:15 PM 40 42.94 0.00 0.76 0.02 0.00 0.00 > 0.00 0.00 0.00 56.28 > > 08:55:15 PM 41 42.11 0.00 0.76 0.01 0.00 0.00 > 0.00 0.00 0.00 57.13 > > 08:55:15 PM 42 43.26 0.00 0.85 0.01 0.00 0.00 > 0.00 0.00 0.00 55.88 > > 08:55:15 PM 43 42.23 0.00 0.77 0.01 0.00 0.00 > 0.00 0.00 0.00 56.99 > > 08:55:15 PM 44 42.24 0.00 0.80 0.00 0.00 0.00 > 0.00 0.00 0.00 56.95 > > 08:55:15 PM 45 41.65 0.00 0.79 0.01 0.00 0.00 > 0.00 0.00 0.00 57.55 > > 08:55:15 PM 46 41.76 0.00 0.77 0.01 0.00 0.00 > 0.00 0.00 0.00 57.46 > > 08:55:15 PM 47 42.66 0.00 0.82 0.00 0.00 0.00 > 0.00 0.00 0.00 56.52 > > 08:55:15 PM 48 41.83 0.00 0.76 0.01 0.00 0.00 > 0.00 0.00 0.00 57.40 > > 08:55:15 PM 49 42.04 0.00 0.82 0.00 0.00 0.00 > 0.00 0.00 0.00 57.13 > > 08:55:15 PM 50 41.55 0.00 0.78 0.01 0.00 0.00 > 0.00 0.00 0.00 57.67 > > 08:55:15 PM 51 42.37 0.00 0.78 0.00 0.00 0.00 > 0.00 0.00 0.00 56.84 > > 08:55:15 PM 52 41.66 0.00 0.77 0.00 0.00 0.00 > 0.00 0.00 0.00 57.57 > > 08:55:15 PM 53 41.77 0.00 0.79 0.00 0.00 0.00 > 0.00 0.00 0.00 57.45 > > 08:55:15 PM 54 40.33 0.00 0.88 0.01 0.00 0.00 > 0.00 0.00 0.00 58.79 > > 08:55:15 PM 55 41.62 0.00 0.78 0.01 0.00 0.00 > 0.00 0.00 0.00 57.59 > > 08:55:15 PM 56 41.51 0.00 0.82 0.00 0.00 0.00 > 0.00 0.00 0.00 57.67 > > 08:55:15 PM 57 40.35 0.00 0.76 0.00 0.00 0.00 > 0.00 0.00 0.00 58.89 > > 08:55:15 PM 58 41.92 0.00 0.79 0.01 0.00 0.00 > 0.00 0.00 0.00 57.28 > > 08:55:15 PM 59 41.48 0.00 0.80 0.00 0.00 0.00 > 0.00 0.00 0.00 57.72 > > 08:55:15 PM 60 40.83 0.00 0.75 0.01 0.00 0.00 > 0.00 0.00 0.00 58.41 > > 08:55:15 PM 61 40.95 0.00 0.78 0.01 0.00 0.00 > 0.00 0.00 0.00 58.26 > > 08:55:15 PM 62 40.85 0.00 0.80 0.01 0.00 0.00 > 0.00 0.00 0.00 58.35 > > 08:55:15 PM 63 42.11 0.00 0.77 0.01 0.00 0.00 > 0.00 0.00 0.00 57.12 > > 08:55:15 PM 64 42.22 0.00 0.78 0.01 0.00 0.00 > 0.00 0.00 0.00 56.99 > > 08:55:15 PM 65 41.55 0.00 0.78 0.00 0.00 0.00 > 0.00 0.00 0.00 57.67 > > 08:55:15 PM 66 41.68 0.00 0.81 0.00 0.00 0.00 > 0.00 0.00 0.00 57.51 > > 08:55:15 PM 67 41.06 0.00 0.87 0.01 0.00 0.00 > 0.00 0.00 0.00 58.07 > > 08:55:15 PM 68 41.80 0.00 0.79 0.00 0.00 0.00 > 0.00 0.00 0.00 57.41 > > 08:55:15 PM 69 41.38 0.00 0.75 0.00 0.00 0.00 > 0.00 0.00 0.00 57.87 > > 08:55:15 PM 70 42.17 0.00 0.75 0.00 0.00 0.00 > 0.00 0.00 0.00 57.08 > > 08:55:15 PM 71 41.26 0.00 0.81 0.00 0.00 0.00 > 0.01 0.00 0.00 57.92 > > > > > > 1> rp(recon:scheduler_usage(1000)). > > [{1,0.8581586716045433}, > > {2,0.8607597847736659}, > > {3,0.8501543449289042}, > > {4,0.8619718426658023}, > > {5,0.8660226578705742}, > > {6,0.8555763933364806}, > > {7,0.8557195972097537}, > > {8,0.8668284625654983}, > > {9,0.858588675506532}, > > {10,0.8650501369640686}, > > {11,0.8615557214275474}, > > {12,0.8523392789683996}, > > {13,0.8469329348407398}, > > {14,0.8615637534991641}, > > {15,0.8591930739986557}, > > {16,0.8578596164201188}, > > {17,0.8549696086049403}, > > {18,0.8677098230685564}, > > {19,0.8556328340054443}, > > {20,0.8642542459479063}, > > {21,0.8670655179064826}, > > {22,0.8562638308551557}, > > {23,0.8511202058055239}, > > {24,0.8697339264798696}, > > {25,0.8594823462601853}, > > {26,0.8634550008001265}, > > {27,0.8568590130172198}, > > {28,0.864025757952233}, > > {29,0.8575614770622326}, > > {30,0.8611760239143281}, > > {31,0.8589789125669829}, > > {32,0.8537754468447621}, > > {33,0.8675538912040871}, > > {34,0.8607522803519515}, > > {35,0.8639389099445753}, > > {36,0.8743759133171144}, > > {37,0.9526507413773687}, > > {38,0.948925611917577}, > > {39,0.9511904612810083}, > > {40,0.9581015421004746}, > > {41,0.9534977201467516}, > > {42,0.0}, > > {43,0.0}, > > {44,0.0}, > > {45,0.9522518499948341}, > > {46,0.9453306755503674}, > > {47,0.9586374601140047}, > > {48,0.9574611845671873}, > > {49,0.950914304057791}, > > {50,0.9582124294398642}, > > {51,0.9448074463900029}, > > {52,0.9474771999511407}, > > {53,0.9450778098401229}, > > {54,0.0}, > > {55,0.9478167911711937}, > > {56,0.9612587860764551}, > > {57,0.943468696358506}, > > {58,0.9509498642900455}, > > {59,0.9603595134495414}, > > {60,0.9517397912369205}, > > {61,0.0}, > > {62,0.9587029756970658}, > > {63,0.0}, > > {64,0.0}, > > {65,0.9526167902065421}, > > {66,0.0}, > > {67,0.9475668680715805}, > > {68,0.0}, > > {69,0.9630301804839606}, > > {70,0.9475359592879946}, > > {71,0.9541718747681374}, > > {72,0.0}] > > > > > > Thanks for your time, > > > > Jesse Stimpson > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > -- Jesse Stimpson Site Reliability Engineering m: 9199950424 <(919)%20995-0424> RepublicWireless.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From roger@REDACTED Thu Jan 10 15:44:42 2019 From: roger@REDACTED (Roger Lipscombe) Date: Thu, 10 Jan 2019 14:44:42 +0000 Subject: [erlang-questions] SIGUSR1 isn't writing a crash dump? Message-ID: I've got a release, using OTP-21.0, which I'm starting by using a relx-generated script (relx 3.26.0, managed by erlang.mk). When I send it SIGUSR1, it quits, but without writing erl_crash.dump I'm using ERL_CRASH_DUMP to set the location to somewhere writable. Using erlang:halt("whoops") from a remote shell successfully writes erl_crash.dump, but SIGUSR1 doesn't. The daemon simply quits. Searching the internets got me to OTP-13425, which was apparently fixed in OTP-19. Did it somehow become un-fixed? Or am I missing something else? Cheers, Roger. From mark.geib.44@REDACTED Thu Jan 10 22:48:44 2019 From: mark.geib.44@REDACTED (Mark Geib) Date: Thu, 10 Jan 2019 14:48:44 -0700 Subject: [erlang-questions] Include ERTS in an escript package Message-ID: <7557A90B-34F8-4230-910C-A9CA4E27197E@gmail.com> I normally build erlang releases with rebar3 for deployment in our production environments. I do not require erlang to be installed on these systems since the release includes erlang, etc. I have a need to provide a escript for some metrics gathering and housekeeping operations, but with no erlang on these production servers I am wondering what the best approach is to provide these escripts for production. My question, is it possible, with rebar3 to package an escript in such a way as to include erlang, eliminating the dependency of install erlang on these servers. This would also allow me to continue providing releases based on different versions of erlang that run on the same production server. Thanks, Mark. From dmkolesnikov@REDACTED Fri Jan 11 08:40:54 2019 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Fri, 11 Jan 2019 09:40:54 +0200 Subject: [erlang-questions] Include ERTS in an escript package In-Reply-To: <7557A90B-34F8-4230-910C-A9CA4E27197E@gmail.com> References: <7557A90B-34F8-4230-910C-A9CA4E27197E@gmail.com> Message-ID: Hello Mark, As far I know, It is not possible to package runtime to escript with rebar3. Easies, faster and reliable solution to package escript along with ERTS to docker... Another options to package entire application and its deps to escript with rebar3, like you are doing it now. Then prepare tarball with ERTS. You can either use of-the-shelf kerl builds, use Erlang Solution packages [1] or make a lightweight release that will weight 20 - 30MB. And ship both deliverables. If you?d like to ship a single file then you can make a self extracting tarball. I?ve shared a link to my project where I did similar [2]. However, I?ve stopped doing it now and switched 100% to dockers. - Dmitry References [1] https://www.erlang-solutions.com/resources/download.html [2] https://github.com/fogfish/hyperion/tree/0.5.0/rel > On 10 Jan 2019, at 23.48, Mark Geib wrote: > > I normally build erlang releases with rebar3 for deployment in our production environments. I do not require erlang to be installed on these systems since the release includes erlang, etc. > > I have a need to provide a escript for some metrics gathering and housekeeping operations, but with no erlang on these production > servers I am wondering what the best approach is to provide these escripts for production. > > My question, is it possible, with rebar3 to package an escript in such a way as to include erlang, eliminating the dependency of > install erlang on these servers. This would also allow me to continue providing releases based on different versions of erlang that run > on the same production server. > > Thanks, > Mark. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From dmkolesnikov@REDACTED Fri Jan 11 09:18:08 2019 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Fri, 11 Jan 2019 10:18:08 +0200 Subject: [erlang-questions] Infix function and user-defined operators In-Reply-To: References: <01B6187C-9731-4740-A770-532B8472AB97@gmail.com> Message-ID: <6B5DCA31-DDA2-4B04-A0CD-28AB9B8ED718@gmail.com> Hello Richard, Thank you for valuable feedback. Indeed `name` is the best option. However, it requires changes to the compiler. Unfortunately, it do not have any visibility why this feature has been rejected 14 years ago. I found that /name/ is the closed approximation of Haskell?s `name` achievable with core Erlang syntax thus implementable via parse transforms. Current proposal of user defined operator is left associative with same precedence as multiplication. I also thought about /f> and On 8 Jan 2019, at 15.53, Richard O'Keefe wrote: > > The Haskell convention for user-defined operators is `name` . > I don't see any point in inventing a new one. > I note that > (a) http://mlton.org/InfixingOperators shows a relatively simple > scheme for ML that would fit Erlang without too much strain. It > starts with a warning that operator properties are not part of > module interfaces. This was a problem in Prolog too, and I > surmise that it may have been the reason for not having user- > declared operators in Erlang. > (b) In F#, the type and precedence of a user-defined operator > are determined by its first character. See > https://docs.microsoft.com/en-us/dotnet/fsharp/language-reference/symbol-and-operator-reference/index > (c) ECMA Eiffel 8.28.5 puts all unary user-defined operators at the > same precedence level and all binary user-defined operators at another > precedence level. Some other Eiffel dialects followed the same rule as > F#. The (b) and (c) rules appear to also be related to not having > operator properties in signatures/classes/modules/whatever. > > More years ago than I care to admit, I wrote an extended > Prolog parser that allowed user-defined "operators" like > "N of L are V" or "at least N of L are V". It was based > on Vaughan Pratt's CGOL; see > https://en.wikipedia.org/wiki/CGOL > > > > On Tue, 8 Jan 2019 at 04:56, Dmitry Kolesnikov > wrote: > Hello Community, > > Now and then, I am missing an infix notation in Erlang. There was a great discussion about this subject decade and half ago but I do not think anything is changed [1]. > > I am testing and stretching limits of functional abstracts in Erlang as part of my pure functional and generic programming exercises [2]. I?ve decided to experiment with Infix function via parse-transform. > > The parse_transform feature implements a syntax sugar for infix, which is compiled into valid Erlang code using corresponding function at compile-time. To apply a function using infix notation, we enclose its name in slash characters (/). This allows us to use any arity 2 functions as infix operators. > > ``` > -compile({parse_transform, infix}). > > 1 /plus/ 1. > > F /lists:map/ [1, 2, 3]. > ``` > > Right now, I am working on this feature here https://github.com/fogfish/datum/pull/64 , you can find more detailed description there, code, etc. > > I would appreciate for any feedback, suggestion from the community. > Does it sounds any usable, still desired or total braindead? > What other use-case/applications you might imagine for infix notation? > > Best Regards, > Dmitry > > References > > 1. http://erlang.org/pipermail/erlang-questions/2004-March/011929.html > 2. https://github.com/fogfish/datum > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Fri Jan 11 09:27:53 2019 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Fri, 11 Jan 2019 09:27:53 +0100 Subject: [erlang-questions] Include ERTS in an escript package In-Reply-To: <7557A90B-34F8-4230-910C-A9CA4E27197E@gmail.com> References: <7557A90B-34F8-4230-910C-A9CA4E27197E@gmail.com> Message-ID: <3cf1d511-d99a-3902-b8dd-351b9381e527@ninenines.eu> Erlang.mk provides self-extracting releases for these situations.[1] Maybe you can adapt the code[2] for your own use case. Basically it's a script with the relx tar file appended at the end. [1] https://erlang.mk/guide/sfx.html [2] https://github.com/ninenines/erlang.mk/blob/master/plugins/sfx.mk On 10/01/2019 22:48, Mark Geib wrote: > I normally build erlang releases with rebar3 for deployment in our production environments. I do not require erlang to be installed on these systems since the release includes erlang, etc. > > I have a need to provide a escript for some metrics gathering and housekeeping operations, but with no erlang on these production > servers I am wondering what the best approach is to provide these escripts for production. > > My question, is it possible, with rebar3 to package an escript in such a way as to include erlang, eliminating the dependency of > install erlang on these servers. This would also allow me to continue providing releases based on different versions of erlang that run > on the same production server. > > Thanks, > Mark. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin https://ninenines.eu From otp@REDACTED Fri Jan 11 09:54:07 2019 From: otp@REDACTED (Erlang/OTP) Date: Fri, 11 Jan 2019 09:54:07 +0100 Subject: [erlang-questions] Patch Package OTP 19.3.6.13 Released Message-ID: <20190111085406.GA13063@erix.ericsson.se> Patch Package: OTP 19.3.6.13 Git Tag: OTP-19.3.6.13 Date: 2019-01-11 Trouble Report Id: OTP-15430, OTP-15487 Seq num: ERIERL-237, ERL-804 System: OTP Release: 19 Application: erts-8.3.5.7 Predecessor: OTP 19.3.6.12 Check out the git tag OTP-19.3.6.13, and build a full OTP system including documentation. Apply one or more applications from this build as patches to your installation using the 'otp_patch_apply' tool. For information on install requirements, see descriptions for each application version below. --------------------------------------------------------------------- --- erts-8.3.5.7 ---------------------------------------------------- --------------------------------------------------------------------- Note! The erts-8.3.5.7 application can *not* be applied independently of other applications on an arbitrary OTP 19 installation. On a full OTP 19 installation, also the following runtime dependency has to be satisfied: -- sasl-3.0.1 (first satisfied in OTP 19.1) --- Fixed Bugs and Malfunctions --- OTP-15487 Application(s): erts Related Id(s): ERL-804 Fixed bug in operator band of two negative operands causing erroneous result if the absolute value of one of the operands have the lowest N*W bits as zero and the other absolute value is not larger than N*W bits. N is an integer of 1 or larger and W is 32 or 64 depending on word size. --- Improvements and New Features --- OTP-15430 Application(s): erts Related Id(s): ERIERL-237 Added an optional ./configure flag to compile the emulator with spectre mitigation: --with-spectre-mitigation Note that this requires a recent version of GCC with support for spectre mitigation and the --mindirect-branch=thunk flag, such as 8.1. Full runtime dependencies of erts-8.3.5.7: kernel-5.0, sasl-3.0.1, stdlib-3.0 --------------------------------------------------------------------- --------------------------------------------------------------------- --------------------------------------------------------------------- From jstimpson@REDACTED Thu Jan 10 23:23:10 2019 From: jstimpson@REDACTED (Jesse Stimpson) Date: Thu, 10 Jan 2019 17:23:10 -0500 Subject: [erlang-questions] Dirty CPU schedulers stuck at zero utilization In-Reply-To: References: Message-ID: Jonas, Below is an excerpt from msacc for one scheduler while the system is under moderate load (not saturated as in my previous data). On a periodic basis, the schedulers under the 2027 patch have a significantly higher utilization in the emulator column. At times I saw this spike occur every 5 minutes on the nose. Other times it was less predictable. But it typically occurs on the order of minutes (3 - 7 minutes). The full output is included as well. scheduler(36) 0.68% 0.22% 35.16% 0.55% 55.38% 1.35% 6.66% scheduler(36) 0.39% 0.08% 76.21% 0.48% 19.52% 1.03% 2.29% I've not been able to get the thread id for a scheduler yet to run perf. There is a past mailing list thread suggesting use of "ps -Leo pid,tid,comm | grep beam", but this is not returning the thread ids on my current system. Average thread real-time : 1000868 us Accumulated system run-time : 52140693 us Average scheduler run-time : 929256 us Thread aux check_io emulator gc other port sleep Stats per thread: async( 0) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async( 1) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async( 2) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async( 3) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async( 4) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async( 5) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async( 6) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async( 7) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async( 8) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async( 9) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(10) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(11) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(12) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(13) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(14) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(15) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(16) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(17) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(18) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(19) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(20) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(21) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(22) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(23) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(24) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(25) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(26) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(27) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(28) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(29) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(30) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(31) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(32) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(33) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(34) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(35) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(36) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(37) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(38) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(39) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(40) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(41) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(42) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(43) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(44) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(45) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(46) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(47) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(48) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(49) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(50) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(51) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(52) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(53) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(54) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(55) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(56) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(57) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(58) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(59) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(60) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(61) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(62) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(63) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(64) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(65) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(66) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(67) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(68) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(69) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(70) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(71) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(72) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(73) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(74) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(75) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(76) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(77) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(78) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(79) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(80) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(81) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(82) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(83) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(84) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(85) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(86) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(87) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(88) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(89) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(90) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(91) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(92) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(93) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(94) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(95) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(96) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(97) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(98) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(99) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% aux( 1) 3.61% 1.11% 0.00% 0.00% 0.03% 0.00% 95.24% dirty_cpu_( 1) 0.00% 0.00% 35.00% 0.00% 19.49% 0.00% 45.51% dirty_cpu_( 2) 0.00% 0.00% 34.84% 0.00% 18.28% 0.00% 46.89% dirty_cpu_( 3) 0.00% 0.00% 35.72% 0.00% 19.16% 0.00% 45.11% dirty_cpu_( 4) 0.00% 0.00% 30.97% 0.00% 16.72% 0.00% 52.30% dirty_cpu_( 5) 0.00% 0.00% 30.20% 0.00% 18.21% 0.00% 51.58% dirty_cpu_( 6) 0.00% 0.00% 33.56% 0.00% 17.62% 0.00% 48.82% dirty_cpu_( 7) 0.00% 0.00% 33.94% 0.00% 17.94% 0.00% 48.12% dirty_cpu_( 8) 0.00% 0.00% 32.99% 0.00% 18.24% 0.00% 48.77% dirty_cpu_( 9) 0.00% 0.00% 31.68% 0.00% 17.47% 0.00% 50.86% dirty_cpu_(10) 0.00% 0.00% 34.06% 0.00% 19.18% 0.00% 46.75% dirty_cpu_(11) 0.00% 0.00% 33.91% 0.00% 17.87% 0.00% 48.23% dirty_cpu_(12) 0.00% 0.00% 30.02% 0.00% 19.52% 0.00% 50.46% dirty_cpu_(13) 0.00% 0.00% 32.33% 0.00% 18.73% 0.00% 48.94% dirty_cpu_(14) 0.00% 0.00% 36.54% 0.00% 19.58% 0.00% 43.88% dirty_cpu_(15) 0.00% 0.00% 37.19% 0.00% 18.62% 0.00% 44.19% dirty_cpu_(16) 0.00% 0.00% 30.57% 0.00% 17.45% 0.00% 51.99% dirty_cpu_(17) 0.00% 0.00% 30.65% 0.00% 18.08% 0.00% 51.28% dirty_cpu_(18) 0.00% 0.00% 36.95% 0.00% 19.01% 0.00% 44.04% dirty_cpu_(19) 0.00% 0.00% 34.25% 0.00% 18.26% 0.00% 47.49% dirty_cpu_(20) 0.00% 0.00% 34.76% 0.00% 17.07% 0.00% 48.17% dirty_cpu_(21) 0.00% 0.00% 34.37% 0.00% 18.78% 0.00% 46.85% dirty_cpu_(22) 0.00% 0.00% 29.20% 0.00% 17.98% 0.00% 52.82% dirty_cpu_(23) 0.00% 0.00% 32.41% 0.00% 18.01% 0.00% 49.58% dirty_cpu_(24) 0.00% 0.00% 32.83% 0.00% 17.95% 0.00% 49.23% dirty_cpu_(25) 0.00% 0.00% 34.42% 0.00% 21.18% 0.00% 44.40% dirty_cpu_(26) 0.00% 0.00% 34.99% 0.00% 18.62% 0.00% 46.39% dirty_cpu_(27) 0.00% 0.00% 31.30% 0.00% 17.76% 0.00% 50.94% dirty_cpu_(28) 0.00% 0.00% 32.23% 0.00% 17.42% 0.00% 50.35% dirty_cpu_(29) 0.00% 0.00% 33.25% 0.00% 18.83% 0.00% 47.92% dirty_cpu_(30) 0.00% 0.00% 34.15% 0.00% 18.93% 0.00% 46.93% dirty_cpu_(31) 0.00% 0.00% 33.88% 0.00% 17.94% 0.00% 48.17% dirty_cpu_(32) 0.00% 0.00% 35.30% 0.00% 17.34% 0.00% 47.36% dirty_cpu_(33) 0.00% 0.00% 34.17% 0.00% 17.96% 0.00% 47.87% dirty_cpu_(34) 0.00% 0.00% 32.21% 0.00% 18.94% 0.00% 48.85% dirty_cpu_(35) 0.00% 0.00% 35.37% 0.00% 19.08% 0.00% 45.56% dirty_cpu_(36) 0.00% 0.00% 31.21% 0.00% 17.71% 0.00% 51.07% dirty_io_s( 1) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_io_s( 2) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_io_s( 3) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_io_s( 4) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_io_s( 5) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_io_s( 6) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_io_s( 7) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_io_s( 8) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_io_s( 9) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_io_s(10) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% poll( 0) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% scheduler( 1) 0.72% 0.18% 36.27% 0.51% 55.29% 1.59% 5.44% scheduler( 2) 0.62% 0.19% 33.97% 0.55% 54.90% 1.49% 8.27% scheduler( 3) 0.64% 0.22% 34.90% 0.53% 55.50% 1.43% 6.78% scheduler( 4) 0.56% 0.16% 31.66% 0.49% 55.55% 1.33% 10.25% scheduler( 5) 0.71% 0.20% 35.66% 0.58% 54.59% 1.49% 6.77% scheduler( 6) 0.59% 0.23% 31.19% 0.48% 57.81% 1.28% 8.42% scheduler( 7) 0.65% 0.23% 34.66% 0.56% 56.09% 1.60% 6.22% scheduler( 8) 0.78% 0.23% 35.04% 0.59% 55.32% 1.58% 6.47% scheduler( 9) 0.64% 0.21% 35.12% 0.47% 56.39% 1.50% 5.66% scheduler(10) 0.80% 0.21% 37.18% 0.53% 54.64% 1.59% 5.04% scheduler(11) 0.63% 0.20% 31.56% 0.53% 57.86% 1.48% 7.74% scheduler(12) 0.69% 0.22% 37.50% 0.57% 54.38% 1.75% 4.89% scheduler(13) 0.78% 0.21% 35.19% 0.55% 55.96% 1.54% 5.76% scheduler(14) 0.70% 0.22% 31.07% 0.48% 54.35% 1.30% 11.88% scheduler(15) 0.64% 0.23% 36.66% 0.51% 54.74% 1.66% 5.56% scheduler(16) 0.68% 0.18% 33.76% 0.50% 55.47% 1.28% 8.12% scheduler(17) 0.55% 0.28% 27.34% 0.43% 57.13% 1.24% 13.03% scheduler(18) 0.64% 0.18% 32.85% 0.47% 56.12% 1.32% 8.42% scheduler(19) 0.76% 0.24% 35.84% 0.53% 55.13% 1.56% 5.94% scheduler(20) 0.57% 0.19% 27.58% 0.42% 57.17% 1.12% 12.96% scheduler(21) 0.67% 0.20% 33.02% 0.44% 56.36% 1.36% 7.95% scheduler(22) 0.81% 0.23% 34.44% 0.54% 56.18% 1.35% 6.46% scheduler(23) 0.72% 0.22% 36.49% 0.53% 55.47% 1.53% 5.04% scheduler(24) 0.65% 0.24% 34.68% 0.45% 55.89% 1.44% 6.65% scheduler(25) 0.71% 0.21% 35.66% 0.46% 55.12% 1.65% 6.18% scheduler(26) 0.73% 0.20% 33.23% 0.46% 56.45% 1.40% 7.54% scheduler(27) 0.71% 0.20% 35.40% 0.60% 54.82% 1.47% 6.79% scheduler(28) 0.61% 0.18% 31.44% 0.48% 58.58% 1.37% 7.34% scheduler(29) 0.73% 0.22% 36.53% 0.52% 55.65% 1.61% 4.75% scheduler(30) 0.71% 0.17% 36.66% 0.48% 55.07% 1.49% 5.44% scheduler(31) 0.76% 0.21% 37.13% 0.50% 54.88% 1.51% 5.02% scheduler(32) 0.75% 0.17% 32.85% 0.49% 56.76% 1.35% 7.62% scheduler(33) 0.72% 0.23% 34.21% 0.54% 54.55% 1.42% 8.34% scheduler(34) 0.73% 0.19% 35.86% 0.58% 55.21% 1.41% 6.03% scheduler(35) 0.62% 0.19% 36.14% 0.51% 55.15% 1.57% 5.82% scheduler(36) 0.68% 0.22% 35.16% 0.55% 55.38% 1.35% 6.66% Stats per type: async 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% aux 3.61% 1.11% 0.00% 0.00% 0.03% 0.00% 95.24% dirty_cpu_sche 0.00% 0.00% 33.37% 0.00% 18.36% 0.00% 48.27% dirty_io_sched 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% poll 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% scheduler 0.69% 0.21% 34.27% 0.51% 55.72% 1.46% 7.15% Average thread real-time : 1001205 us Accumulated system run-time : 53032153 us Average scheduler run-time : 976005 us Thread aux check_io emulator gc other port sleep Stats per thread: async( 0) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async( 1) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async( 2) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async( 3) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async( 4) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async( 5) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async( 6) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async( 7) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async( 8) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async( 9) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(10) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(11) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(12) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(13) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(14) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(15) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(16) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(17) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(18) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(19) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(20) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(21) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(22) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(23) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(24) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(25) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(26) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(27) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(28) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(29) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(30) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(31) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(32) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(33) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(34) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(35) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(36) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(37) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(38) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(39) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(40) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(41) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(42) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(43) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(44) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(45) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(46) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(47) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(48) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(49) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(50) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(51) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(52) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(53) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(54) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(55) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(56) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(57) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(58) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(59) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(60) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(61) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(62) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(63) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(64) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(65) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(66) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(67) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(68) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(69) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(70) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(71) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(72) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(73) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(74) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(75) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(76) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(77) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(78) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(79) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(80) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(81) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(82) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(83) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(84) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(85) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(86) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(87) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(88) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(89) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(90) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(91) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(92) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(93) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(94) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(95) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(96) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(97) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(98) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(99) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% async(**) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% aux( 1) 4.15% 0.43% 0.00% 0.00% 0.21% 0.00% 95.21% dirty_cpu_( 1) 0.00% 0.00% 27.65% 0.00% 19.43% 0.00% 52.92% dirty_cpu_( 2) 0.00% 0.00% 33.04% 0.00% 21.14% 0.00% 45.82% dirty_cpu_( 3) 0.00% 0.00% 27.82% 0.00% 18.95% 0.00% 53.23% dirty_cpu_( 4) 0.00% 0.00% 28.83% 0.00% 18.41% 0.00% 52.76% dirty_cpu_( 5) 0.00% 0.00% 31.85% 0.00% 19.13% 0.00% 49.02% dirty_cpu_( 6) 0.00% 0.00% 37.99% 0.00% 22.95% 0.00% 39.06% dirty_cpu_( 7) 0.00% 0.00% 30.23% 0.00% 19.39% 0.00% 50.38% dirty_cpu_( 8) 0.00% 0.00% 27.41% 0.00% 19.20% 0.00% 53.39% dirty_cpu_( 9) 0.00% 0.00% 25.97% 0.00% 17.06% 0.00% 56.97% dirty_cpu_(10) 0.00% 0.00% 33.72% 0.00% 22.95% 0.00% 43.33% dirty_cpu_(11) 0.00% 0.00% 29.36% 0.00% 17.68% 0.00% 52.96% dirty_cpu_(12) 0.00% 0.00% 31.76% 0.00% 21.85% 0.00% 46.39% dirty_cpu_(13) 0.00% 0.00% 29.66% 0.00% 16.06% 0.00% 54.28% dirty_cpu_(14) 0.00% 0.00% 28.40% 0.00% 19.99% 0.00% 51.61% dirty_cpu_(15) 0.00% 0.00% 30.92% 0.00% 17.43% 0.00% 51.65% dirty_cpu_(16) 0.00% 0.00% 26.79% 0.00% 16.10% 0.00% 57.11% dirty_cpu_(17) 0.00% 0.00% 27.51% 0.00% 17.93% 0.00% 54.56% dirty_cpu_(18) 0.00% 0.00% 29.95% 0.00% 17.58% 0.00% 52.47% dirty_cpu_(19) 0.00% 0.00% 32.60% 0.00% 20.00% 0.00% 47.40% dirty_cpu_(20) 0.00% 0.00% 28.34% 0.00% 16.17% 0.00% 55.49% dirty_cpu_(21) 0.00% 0.00% 30.26% 0.00% 16.87% 0.00% 52.87% dirty_cpu_(22) 0.00% 0.00% 32.03% 0.00% 19.08% 0.00% 48.88% dirty_cpu_(23) 0.00% 0.00% 28.59% 0.00% 19.82% 0.00% 51.59% dirty_cpu_(24) 0.00% 0.00% 29.55% 0.00% 17.38% 0.00% 53.07% dirty_cpu_(25) 0.00% 0.00% 30.51% 0.00% 18.91% 0.00% 50.58% dirty_cpu_(26) 0.00% 0.00% 29.72% 0.00% 17.58% 0.00% 52.70% dirty_cpu_(27) 0.00% 0.00% 28.25% 0.00% 18.94% 0.00% 52.81% dirty_cpu_(28) 0.00% 0.00% 33.06% 0.00% 19.75% 0.00% 47.19% dirty_cpu_(29) 0.00% 0.00% 27.28% 0.00% 19.40% 0.00% 53.32% dirty_cpu_(30) 0.00% 0.00% 34.10% 0.00% 22.20% 0.00% 43.70% dirty_cpu_(31) 0.00% 0.00% 32.76% 0.00% 21.75% 0.00% 45.49% dirty_cpu_(32) 0.00% 0.00% 30.09% 0.00% 22.01% 0.00% 47.90% dirty_cpu_(33) 0.00% 0.00% 31.58% 0.00% 20.70% 0.00% 47.72% dirty_cpu_(34) 0.00% 0.00% 29.87% 0.00% 20.26% 0.00% 49.86% dirty_cpu_(35) 0.00% 0.00% 31.85% 0.00% 21.93% 0.00% 46.21% dirty_cpu_(36) 0.00% 0.00% 28.77% 0.00% 18.50% 0.00% 52.73% dirty_io_s( 1) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_io_s( 2) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_io_s( 3) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_io_s( 4) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_io_s( 5) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_io_s( 6) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_io_s( 7) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_io_s( 8) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_io_s( 9) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_io_s(10) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% poll( 0) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% scheduler( 1) 0.40% 0.09% 76.76% 0.50% 18.81% 1.15% 2.30% scheduler( 2) 0.36% 0.14% 77.57% 0.49% 18.74% 1.08% 1.61% scheduler( 3) 0.35% 0.08% 76.81% 0.46% 18.96% 1.03% 2.32% scheduler( 4) 0.36% 0.08% 78.06% 0.40% 18.20% 1.11% 1.78% scheduler( 5) 0.33% 0.07% 76.77% 0.40% 19.36% 0.88% 2.19% scheduler( 6) 0.31% 0.06% 74.23% 0.38% 19.66% 0.81% 4.54% scheduler( 7) 0.40% 0.08% 76.56% 0.48% 18.86% 1.25% 2.36% scheduler( 8) 0.48% 0.07% 75.85% 0.50% 19.36% 1.40% 2.34% scheduler( 9) 0.36% 0.07% 76.28% 0.52% 19.21% 1.39% 2.17% scheduler(10) 0.32% 0.07% 75.76% 0.43% 19.01% 1.19% 3.23% scheduler(11) 0.35% 0.07% 76.43% 0.52% 19.00% 1.19% 2.43% scheduler(12) 0.31% 0.06% 75.31% 0.41% 19.55% 1.02% 3.35% scheduler(13) 0.38% 0.07% 77.07% 0.59% 18.84% 1.33% 1.73% scheduler(14) 0.41% 0.18% 77.68% 0.48% 18.25% 1.34% 1.66% scheduler(15) 0.34% 0.08% 76.80% 0.44% 19.12% 1.05% 2.16% scheduler(16) 0.38% 0.06% 77.68% 0.44% 18.22% 1.13% 2.08% scheduler(17) 0.36% 0.06% 76.70% 0.49% 18.93% 1.19% 2.27% scheduler(18) 0.40% 0.08% 77.05% 0.44% 18.97% 1.25% 1.82% scheduler(19) 0.35% 0.07% 75.97% 0.45% 19.40% 1.07% 2.69% scheduler(20) 0.37% 0.07% 76.11% 0.46% 19.42% 1.14% 2.43% scheduler(21) 0.37% 0.08% 76.33% 0.47% 19.34% 1.01% 2.40% scheduler(22) 0.46% 0.08% 77.41% 0.50% 18.75% 0.97% 1.83% scheduler(23) 0.43% 0.08% 75.50% 0.51% 19.85% 1.13% 2.51% scheduler(24) 0.39% 0.07% 74.49% 0.49% 20.29% 1.07% 3.20% scheduler(25) 0.36% 0.07% 74.69% 0.48% 19.32% 0.98% 4.10% scheduler(26) 0.35% 0.07% 75.96% 0.41% 18.79% 0.98% 3.44% scheduler(27) 0.36% 0.06% 76.86% 0.43% 19.18% 0.96% 2.15% scheduler(28) 0.32% 0.08% 75.08% 0.46% 19.71% 1.02% 3.34% scheduler(29) 0.43% 0.14% 77.10% 0.43% 18.80% 1.22% 1.88% scheduler(30) 0.35% 0.06% 76.45% 0.47% 18.64% 1.11% 2.92% scheduler(31) 0.38% 0.08% 78.35% 0.52% 18.21% 1.17% 1.29% scheduler(32) 0.37% 0.06% 74.47% 0.50% 19.71% 1.34% 3.56% scheduler(33) 0.38% 0.06% 77.12% 0.44% 18.59% 1.12% 2.29% scheduler(34) 0.46% 0.07% 75.58% 0.51% 19.87% 0.93% 2.58% scheduler(35) 0.38% 0.13% 75.24% 0.46% 19.68% 1.03% 3.07% scheduler(36) 0.39% 0.08% 76.21% 0.48% 19.52% 1.03% 2.29% Stats per type: async 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% aux 4.15% 0.43% 0.00% 0.00% 0.21% 0.00% 95.21% dirty_cpu_sche 0.00% 0.00% 30.22% 0.00% 19.29% 0.00% 50.48% dirty_io_sched 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% poll 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% scheduler 0.38% 0.08% 76.34% 0.47% 19.11% 1.11% 2.51% On Wed, Jan 9, 2019 at 6:06 PM Jonas Falkevik wrote: > Hi Jesse, > we have not observe the behaviour you see. > Can you share microstate accounting stats for the different cases? > Maybe that could shed some light on what the schedulers are doing? > 1> msacc:start(1000), msacc:print(). > > And maybe also running perf top as root on one of the scheduler threads > that are fully utilised can be interesting. > # perf top --tid > > /Jonas > > On Wed, Jan 9, 2019 at 10:52 PM Jesse Stimpson < > jstimpson@REDACTED> wrote: > >> Rikard and Jonas, >> >> Thank you for your response. With the patch at OTP PR-2027, we do observe >> that the dirty CPU schedulers do not drop to zero utilization, but in >> addition we see that the performance of the traditional schedulers >> suffers. Specifically, when we apply our load test, scheduler usage >> toggles in between the following two states. I've snipped the output of >> similar values for readability. >> >> [{1,0.8933020805456553}, >> >> ... >> >> {36,0.8882068476700309}, >> >> {37,0.4932569692588096}, >> >> ... >> >> {72,0.49842881346116297}] >> >> >> and >> >> >> [{1,1.0}, >> >> ... >> >> {36,1.0}, >> >> {37,0.42909590032946493}, >> >> ... >> >> {72,0.4641339106204525}] >> >> When applying this same load to the un-patched Erlang 21.2, the >> traditional schedulers are utilized at about 75%. So it appears that the >> 2027 patch may put undesirable pressure on the schedulers as a trade-off >> for consistent performance from the dirty CPU schedulers. Are you observing >> a similar response on your system? >> >> Thanks, >> Jesse >> >> On Wed, Jan 9, 2019 at 3:33 AM Jonas Falkevik < >> jonas.falkevik@REDACTED> wrote: >> >>> Hi, >>> This sounds like the issue I bumped into. >>> After some time, many of the dirty schedulers are not performing any >>> work anymore. >>> Which was related to a corrupted list of sleeping dirty schedulers. >>> >>> I haven?t found what triggered it in my case (maybe a GC?). For me it >>> took approx. 1 day before it happened. >>> >>> Doing Multi Scheduler Block can trigger the problem with the list of >>> sleeping dirty schedulers which gets corrupted. >>> The sleepers list can contain only one or some schedulers. That list is >>> later used to wake up schedulers when there is work to do. >>> For each enqueued dirty-cpu or dirty-io job, it wakes the first >>> scheduler in the sleepers list. >>> But if this list only point to a few schedulers only those are woken up >>> to pick jobs from the queue. >>> >>> Since fixing that specific problem I have not seen the collapse of dirty >>> schedulers anymore. >>> >>> It would be great if you can test and see if it fixes your problem as >>> well, https://github.com/erlang/otp/pull/2027 >>> Any feedback on the bug fix is appreciated. >>> >>> /Jonas >>> >>> > On 8 Jan 2019, at 17:13, Jesse Stimpson < >>> jstimpson@REDACTED> wrote: >>> > >>> > Hello, >>> > >>> > We run a large scale WebRTC-based VOIP application, and currently we >>> are evaluating whether or not dirty CPU schedulers can help us get more >>> consistent performance with some critical NIFs. We've created a test that >>> mimics our production load and have run it on AWS EC2 instances of various >>> sizes. When dirty CPU schedulers are working as expected, we do see >>> measurable improvement in performance, but unfortunately we have seen that >>> they can drop to 0 utilization. They do not recover from this state. >>> > >>> > The timings from 2 of our 12 NIFs are in the vicinity of the 1 msec >>> rule of thumb, so we have experimented with them both on and off dirty CPU >>> schedulers. We've left the other NIFs on the traditional schedulers. More >>> detailed NIF timings can be found below. >>> > >>> > At the end of this message is the concerning data (from >>> recon:scheduler_usage/1).We have schedulers 1-36 as the traditional >>> schedulers bound to the first hardware thread of each core, and schedulers >>> 37-72 are dirty CPU schedulers. Based on mpstat it appears the dirty CPU >>> schedulers are not bound to threads. After about 10 minutes of running the >>> load test, schedulers 42, 43, 44, 61, 63, 64, 66, 68, 72 went to at 0 >>> utilization and stayed there indefinitely. As time goes on, more dirty CPU >>> schedulers enter this 0 utilization state. Schedulers 1-36 do not exhibit >>> this behaviour. We have reproduced this consistently on other EC2 c5 >>> instances of various sizes. >>> > >>> > We have tried various beam adjustments, but we have seen the same >>> behaviour each time. Some examples of things we have tried: >>> > +sbwt very_long +sbwtdcpu very_long +swt very_low +swtdcpu very_low >>> > +sfwi 500 >>> > >>> > Is there anything else that we can try to avoid this 0-utilization >>> issue? We are happy to provide more data if needed. >>> > >>> > NIF: webrtc_acm_add_data_10ms >>> > median: 606 usec >>> > 90 percentile: 1230 usec >>> > 99 percentile: 2285 usec >>> > 99.9 percentile: 3223 usec >>> > >>> > NIF: webrtc_acm_playout_data_10ms >>> > median: 98 usec >>> > 90 percentile: 371 usec >>> > 99 percentile: 410 usec >>> > 99.9 percentile: 449 usec >>> > >>> > EC2 instance: c5.18xlarge >>> > >>> > OS: Linux 4.4.0-1073-aws #83-Ubuntu SMP Sat Nov 17 00:26:27 UTC 2018 >>> x86_64 x86_64 x86_64 GNU/Linux >>> > >>> > beam flags: +P 4194304 +sbt db +SP 50:50 +sub true +A 128 +K true >>> > >>> > Erlang banner: Erlang/OTP 21 [erts-10.2] [source] [64-bit] [smp:36:36] >>> [ds:36:36:10] [async-threads:128] [hipe] >>> > >>> > cpu_topology: >>> > [{node,[{processor,[{core,[{thread,{logical,0}}, >>> > {thread,{logical,36}}]}, >>> > >>> {core,[{thread,{logical,1}},{thread,{logical,37}}]}, >>> > >>> {core,[{thread,{logical,2}},{thread,{logical,38}}]}, >>> > >>> {core,[{thread,{logical,3}},{thread,{logical,39}}]}, >>> > >>> {core,[{thread,{logical,4}},{thread,{logical,40}}]}, >>> > >>> {core,[{thread,{logical,5}},{thread,{logical,41}}]}, >>> > >>> {core,[{thread,{logical,6}},{thread,{logical,42}}]}, >>> > >>> {core,[{thread,{logical,7}},{thread,{logical,43}}]}, >>> > >>> {core,[{thread,{logical,8}},{thread,{logical,44}}]}, >>> > >>> {core,[{thread,{logical,9}},{thread,{logical,45}}]}, >>> > >>> {core,[{thread,{logical,10}},{thread,{logical,46}}]}, >>> > >>> {core,[{thread,{logical,11}},{thread,{logical,47}}]}, >>> > >>> {core,[{thread,{logical,12}},{thread,{logical,48}}]}, >>> > >>> {core,[{thread,{logical,13}},{thread,{logical,49}}]}, >>> > >>> {core,[{thread,{logical,14}},{thread,{logical,50}}]}, >>> > >>> {core,[{thread,{logical,15}},{thread,{logical,51}}]}, >>> > >>> {core,[{thread,{logical,16}},{thread,{logical,52}}]}, >>> > >>> {core,[{thread,{logical,17}},{thread,{logical,53}}]}]}]}, >>> > {node,[{processor,[{core,[{thread,{logical,18}}, >>> > {thread,{logical,54}}]}, >>> > >>> {core,[{thread,{logical,19}},{thread,{logical,55}}]}, >>> > >>> {core,[{thread,{logical,20}},{thread,{logical,56}}]}, >>> > >>> {core,[{thread,{logical,21}},{thread,{logical,57}}]}, >>> > >>> {core,[{thread,{logical,22}},{thread,{logical,58}}]}, >>> > >>> {core,[{thread,{logical,23}},{thread,{logical,59}}]}, >>> > >>> {core,[{thread,{logical,24}},{thread,{logical,60}}]}, >>> > >>> {core,[{thread,{logical,25}},{thread,{logical,61}}]}, >>> > >>> {core,[{thread,{logical,26}},{thread,{logical,62}}]}, >>> > >>> {core,[{thread,{logical,27}},{thread,{logical,63}}]}, >>> > >>> {core,[{thread,{logical,28}},{thread,{logical,64}}]}, >>> > >>> {core,[{thread,{logical,29}},{thread,{logical,65}}]}, >>> > >>> {core,[{thread,{logical,30}},{thread,{logical,66}}]}, >>> > >>> {core,[{thread,{logical,31}},{thread,{logical,67}}]}, >>> > >>> {core,[{thread,{logical,32}},{thread,{logical,68}}]}, >>> > >>> {core,[{thread,{logical,33}},{thread,{logical,69}}]}, >>> > >>> {core,[{thread,{logical,34}},{thread,{logical,70}}]}, >>> > >>> {core,[{thread,{logical,35}},{thread,{logical,71}}]}]}]}] >>> > >>> > scheduler_bindings: >>> {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22, >>> > 23,24,25,26,27,28,29,30,31,32,33,34,35} >>> > >>> > CPU utilization (from mpstat) during our load test: >>> > Linux 4.4.0-1073-aws (isw-core-x-qa-awsoh02) 01/04/2019 >>> _x86_64_ (72 CPU) >>> > >>> > 08:55:15 PM CPU %usr %nice %sys %iowait %irq %soft >>> %steal %guest %gnice %idle >>> > 08:55:15 PM all 51.58 0.00 1.64 0.01 0.00 0.42 >>> 0.00 0.00 0.00 46.36 >>> > 08:55:15 PM 0 62.34 0.00 2.69 0.06 0.00 0.00 >>> 0.00 0.00 0.00 34.90 >>> > 08:55:15 PM 1 62.84 0.00 2.58 0.02 0.00 0.00 >>> 0.00 0.00 0.00 34.55 >>> > 08:55:15 PM 2 62.04 0.00 2.52 0.00 0.00 0.00 >>> 0.00 0.00 0.00 35.44 >>> > 08:55:15 PM 3 62.17 0.00 2.62 0.00 0.00 0.00 >>> 0.00 0.00 0.00 35.21 >>> > 08:55:15 PM 4 62.14 0.00 2.54 0.02 0.00 0.00 >>> 0.00 0.00 0.00 35.30 >>> > 08:55:15 PM 5 62.06 0.00 2.51 0.00 0.00 0.00 >>> 0.00 0.00 0.00 35.43 >>> > 08:55:15 PM 6 61.93 0.00 2.58 0.01 0.00 0.00 >>> 0.02 0.00 0.00 35.46 >>> > 08:55:15 PM 7 61.99 0.00 2.54 0.00 0.00 0.00 >>> 0.00 0.00 0.00 35.48 >>> > 08:55:15 PM 8 62.01 0.00 2.52 0.00 0.00 0.00 >>> 0.00 0.00 0.00 35.47 >>> > 08:55:15 PM 9 61.88 0.00 2.49 0.00 0.00 0.00 >>> 0.00 0.00 0.00 35.63 >>> > 08:55:15 PM 10 62.18 0.00 2.48 0.00 0.00 0.00 >>> 0.00 0.00 0.00 35.34 >>> > 08:55:15 PM 11 61.89 0.00 2.47 0.00 0.00 0.00 >>> 0.00 0.00 0.00 35.64 >>> > 08:55:15 PM 12 61.98 0.00 2.50 0.00 0.00 0.00 >>> 0.00 0.00 0.00 35.52 >>> > 08:55:15 PM 13 61.80 0.00 2.54 0.00 0.00 0.00 >>> 0.00 0.00 0.00 35.66 >>> > 08:55:15 PM 14 61.96 0.00 2.56 0.00 0.00 0.00 >>> 0.00 0.00 0.00 35.48 >>> > 08:55:15 PM 15 62.11 0.00 2.57 0.00 0.00 0.00 >>> 0.00 0.00 0.00 35.31 >>> > 08:55:15 PM 16 62.03 0.00 2.55 0.00 0.00 0.00 >>> 0.00 0.00 0.00 35.42 >>> > 08:55:15 PM 17 61.90 0.00 2.53 0.00 0.00 0.00 >>> 0.00 0.00 0.00 35.57 >>> > 08:55:15 PM 18 58.36 0.00 2.41 0.09 0.00 3.72 >>> 0.00 0.00 0.00 35.43 >>> > 08:55:15 PM 19 58.29 0.00 2.45 0.02 0.00 4.00 >>> 0.00 0.00 0.00 35.24 >>> > 08:55:15 PM 20 58.45 0.00 2.36 0.02 0.00 3.73 >>> 0.00 0.00 0.00 35.44 >>> > 08:55:15 PM 21 58.56 0.00 2.34 0.04 0.00 3.54 >>> 0.00 0.00 0.00 35.51 >>> > 08:55:15 PM 22 58.12 0.00 2.46 0.00 0.00 3.84 >>> 0.00 0.00 0.00 35.58 >>> > 08:55:15 PM 23 58.24 0.00 2.36 0.01 0.00 3.82 >>> 0.00 0.00 0.00 35.57 >>> > 08:55:15 PM 24 58.84 0.00 2.42 0.00 0.00 3.60 >>> 0.00 0.00 0.00 35.14 >>> > 08:55:15 PM 25 58.35 0.00 2.43 0.00 0.00 3.72 >>> 0.00 0.00 0.00 35.51 >>> > 08:55:15 PM 26 62.09 0.00 2.47 0.00 0.00 0.00 >>> 0.00 0.00 0.00 35.45 >>> > 08:55:15 PM 27 62.09 0.00 2.46 0.00 0.00 0.00 >>> 0.00 0.00 0.00 35.45 >>> > 08:55:15 PM 28 62.15 0.00 2.48 0.00 0.00 0.00 >>> 0.00 0.00 0.00 35.37 >>> > 08:55:15 PM 29 62.09 0.00 2.46 0.00 0.00 0.00 >>> 0.00 0.00 0.00 35.45 >>> > 08:55:15 PM 30 62.74 0.00 2.45 0.00 0.00 0.00 >>> 0.00 0.00 0.00 34.81 >>> > 08:55:15 PM 31 62.14 0.00 2.47 0.00 0.00 0.00 >>> 0.00 0.00 0.00 35.39 >>> > 08:55:15 PM 32 62.35 0.00 2.52 0.00 0.00 0.00 >>> 0.00 0.00 0.00 35.13 >>> > 08:55:15 PM 33 62.13 0.00 2.42 0.00 0.00 0.00 >>> 0.00 0.00 0.00 35.46 >>> > 08:55:15 PM 34 62.08 0.00 2.47 0.00 0.00 0.00 >>> 0.00 0.00 0.00 35.45 >>> > 08:55:15 PM 35 62.15 0.00 2.49 0.00 0.00 0.00 >>> 0.00 0.00 0.00 35.35 >>> > 08:55:15 PM 36 43.19 0.00 0.80 0.00 0.00 0.00 >>> 0.00 0.00 0.00 56.01 >>> > 08:55:15 PM 37 43.09 0.00 0.83 0.01 0.00 0.00 >>> 0.00 0.00 0.00 56.06 >>> > 08:55:15 PM 38 42.59 0.00 0.78 0.01 0.00 0.00 >>> 0.00 0.00 0.00 56.63 >>> > 08:55:15 PM 39 42.60 0.00 0.82 0.00 0.00 0.00 >>> 0.00 0.00 0.00 56.58 >>> > 08:55:15 PM 40 42.94 0.00 0.76 0.02 0.00 0.00 >>> 0.00 0.00 0.00 56.28 >>> > 08:55:15 PM 41 42.11 0.00 0.76 0.01 0.00 0.00 >>> 0.00 0.00 0.00 57.13 >>> > 08:55:15 PM 42 43.26 0.00 0.85 0.01 0.00 0.00 >>> 0.00 0.00 0.00 55.88 >>> > 08:55:15 PM 43 42.23 0.00 0.77 0.01 0.00 0.00 >>> 0.00 0.00 0.00 56.99 >>> > 08:55:15 PM 44 42.24 0.00 0.80 0.00 0.00 0.00 >>> 0.00 0.00 0.00 56.95 >>> > 08:55:15 PM 45 41.65 0.00 0.79 0.01 0.00 0.00 >>> 0.00 0.00 0.00 57.55 >>> > 08:55:15 PM 46 41.76 0.00 0.77 0.01 0.00 0.00 >>> 0.00 0.00 0.00 57.46 >>> > 08:55:15 PM 47 42.66 0.00 0.82 0.00 0.00 0.00 >>> 0.00 0.00 0.00 56.52 >>> > 08:55:15 PM 48 41.83 0.00 0.76 0.01 0.00 0.00 >>> 0.00 0.00 0.00 57.40 >>> > 08:55:15 PM 49 42.04 0.00 0.82 0.00 0.00 0.00 >>> 0.00 0.00 0.00 57.13 >>> > 08:55:15 PM 50 41.55 0.00 0.78 0.01 0.00 0.00 >>> 0.00 0.00 0.00 57.67 >>> > 08:55:15 PM 51 42.37 0.00 0.78 0.00 0.00 0.00 >>> 0.00 0.00 0.00 56.84 >>> > 08:55:15 PM 52 41.66 0.00 0.77 0.00 0.00 0.00 >>> 0.00 0.00 0.00 57.57 >>> > 08:55:15 PM 53 41.77 0.00 0.79 0.00 0.00 0.00 >>> 0.00 0.00 0.00 57.45 >>> > 08:55:15 PM 54 40.33 0.00 0.88 0.01 0.00 0.00 >>> 0.00 0.00 0.00 58.79 >>> > 08:55:15 PM 55 41.62 0.00 0.78 0.01 0.00 0.00 >>> 0.00 0.00 0.00 57.59 >>> > 08:55:15 PM 56 41.51 0.00 0.82 0.00 0.00 0.00 >>> 0.00 0.00 0.00 57.67 >>> > 08:55:15 PM 57 40.35 0.00 0.76 0.00 0.00 0.00 >>> 0.00 0.00 0.00 58.89 >>> > 08:55:15 PM 58 41.92 0.00 0.79 0.01 0.00 0.00 >>> 0.00 0.00 0.00 57.28 >>> > 08:55:15 PM 59 41.48 0.00 0.80 0.00 0.00 0.00 >>> 0.00 0.00 0.00 57.72 >>> > 08:55:15 PM 60 40.83 0.00 0.75 0.01 0.00 0.00 >>> 0.00 0.00 0.00 58.41 >>> > 08:55:15 PM 61 40.95 0.00 0.78 0.01 0.00 0.00 >>> 0.00 0.00 0.00 58.26 >>> > 08:55:15 PM 62 40.85 0.00 0.80 0.01 0.00 0.00 >>> 0.00 0.00 0.00 58.35 >>> > 08:55:15 PM 63 42.11 0.00 0.77 0.01 0.00 0.00 >>> 0.00 0.00 0.00 57.12 >>> > 08:55:15 PM 64 42.22 0.00 0.78 0.01 0.00 0.00 >>> 0.00 0.00 0.00 56.99 >>> > 08:55:15 PM 65 41.55 0.00 0.78 0.00 0.00 0.00 >>> 0.00 0.00 0.00 57.67 >>> > 08:55:15 PM 66 41.68 0.00 0.81 0.00 0.00 0.00 >>> 0.00 0.00 0.00 57.51 >>> > 08:55:15 PM 67 41.06 0.00 0.87 0.01 0.00 0.00 >>> 0.00 0.00 0.00 58.07 >>> > 08:55:15 PM 68 41.80 0.00 0.79 0.00 0.00 0.00 >>> 0.00 0.00 0.00 57.41 >>> > 08:55:15 PM 69 41.38 0.00 0.75 0.00 0.00 0.00 >>> 0.00 0.00 0.00 57.87 >>> > 08:55:15 PM 70 42.17 0.00 0.75 0.00 0.00 0.00 >>> 0.00 0.00 0.00 57.08 >>> > 08:55:15 PM 71 41.26 0.00 0.81 0.00 0.00 0.00 >>> 0.01 0.00 0.00 57.92 >>> > >>> > >>> > 1> rp(recon:scheduler_usage(1000)). >>> > [{1,0.8581586716045433}, >>> > {2,0.8607597847736659}, >>> > {3,0.8501543449289042}, >>> > {4,0.8619718426658023}, >>> > {5,0.8660226578705742}, >>> > {6,0.8555763933364806}, >>> > {7,0.8557195972097537}, >>> > {8,0.8668284625654983}, >>> > {9,0.858588675506532}, >>> > {10,0.8650501369640686}, >>> > {11,0.8615557214275474}, >>> > {12,0.8523392789683996}, >>> > {13,0.8469329348407398}, >>> > {14,0.8615637534991641}, >>> > {15,0.8591930739986557}, >>> > {16,0.8578596164201188}, >>> > {17,0.8549696086049403}, >>> > {18,0.8677098230685564}, >>> > {19,0.8556328340054443}, >>> > {20,0.8642542459479063}, >>> > {21,0.8670655179064826}, >>> > {22,0.8562638308551557}, >>> > {23,0.8511202058055239}, >>> > {24,0.8697339264798696}, >>> > {25,0.8594823462601853}, >>> > {26,0.8634550008001265}, >>> > {27,0.8568590130172198}, >>> > {28,0.864025757952233}, >>> > {29,0.8575614770622326}, >>> > {30,0.8611760239143281}, >>> > {31,0.8589789125669829}, >>> > {32,0.8537754468447621}, >>> > {33,0.8675538912040871}, >>> > {34,0.8607522803519515}, >>> > {35,0.8639389099445753}, >>> > {36,0.8743759133171144}, >>> > {37,0.9526507413773687}, >>> > {38,0.948925611917577}, >>> > {39,0.9511904612810083}, >>> > {40,0.9581015421004746}, >>> > {41,0.9534977201467516}, >>> > {42,0.0}, >>> > {43,0.0}, >>> > {44,0.0}, >>> > {45,0.9522518499948341}, >>> > {46,0.9453306755503674}, >>> > {47,0.9586374601140047}, >>> > {48,0.9574611845671873}, >>> > {49,0.950914304057791}, >>> > {50,0.9582124294398642}, >>> > {51,0.9448074463900029}, >>> > {52,0.9474771999511407}, >>> > {53,0.9450778098401229}, >>> > {54,0.0}, >>> > {55,0.9478167911711937}, >>> > {56,0.9612587860764551}, >>> > {57,0.943468696358506}, >>> > {58,0.9509498642900455}, >>> > {59,0.9603595134495414}, >>> > {60,0.9517397912369205}, >>> > {61,0.0}, >>> > {62,0.9587029756970658}, >>> > {63,0.0}, >>> > {64,0.0}, >>> > {65,0.9526167902065421}, >>> > {66,0.0}, >>> > {67,0.9475668680715805}, >>> > {68,0.0}, >>> > {69,0.9630301804839606}, >>> > {70,0.9475359592879946}, >>> > {71,0.9541718747681374}, >>> > {72,0.0}] >>> > >>> > >>> > Thanks for your time, >>> > >>> > Jesse Stimpson >>> > _______________________________________________ >>> > erlang-questions mailing list >>> > erlang-questions@REDACTED >>> > http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >> >> -- >> >> >> >> Jesse Stimpson >> >> Site Reliability Engineering >> >> m: 9199950424 <(919)%20995-0424> >> RepublicWireless.com >> > -- Jesse Stimpson Site Reliability Engineering m: 9199950424 <(919)%20995-0424> RepublicWireless.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From silver.surfertab@REDACTED Fri Jan 11 14:33:28 2019 From: silver.surfertab@REDACTED (Greg) Date: Fri, 11 Jan 2019 19:03:28 +0530 Subject: [erlang-questions] SD Erlang In-Reply-To: References: Message-ID: Hi Natalia, I would like to discuss it off-line. On Thu, Nov 15, 2018 at 8:32 PM Natalia Chechina wrote: > Hello George, > > Yes, that would be nice! I'm not sure what you mean by "other options", > but will be happy to discuss that off-line :). > > Best wishes, > Natalia. > > > On Mon, 12 Nov 2018 at 06:03, Greg wrote: > >> I was really hoping that it would be part of the standard release. >> >> What are the options now? >> >> >> On Sun, Nov 11, 2018 at 4:00 PM Natalia Chechina < >> natalia.chechina@REDACTED> wrote: >> >>> Hello, >>> >>> No they are not... and unfortunately at the moment there are no plans to >>> add them... >>> >>> Best wishes, >>> Natalia. >>> >>> >>> >>> On Fri, 9 Nov 2018 at 09:18, Greg wrote: >>> >>>> Hi, >>>> Is SD Erlang (s_groups) part of the standard release yet? >>>> >>>> Greg >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> >>> >>> >>> -- >>> Natalia Chechina >>> >> > > -- > Natalia Chechina > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gerhard@REDACTED Fri Jan 11 15:29:24 2019 From: gerhard@REDACTED (Gerhard Lazu) Date: Fri, 11 Jan 2019 14:29:24 +0000 Subject: [erlang-questions] Missing checksums for github.com/erlang/otp/releases In-Reply-To: References: Message-ID: I've noticed that the bundle-otp script in github.com/erlang/otp is used when minor releases are produced, such as 21.2. I've also noticed that this script is responsible for creating the bundle.txt which contains the HEAD git sha at the time of bundling. Lukas, I can see that you have released 21.2, as well as 21.1. Would you be willing to sign OTP releases and upload the signature when creating a release on GitHub? On team RabbitMQ, this is an automated process for all public artefacts, I would be happy to help. We can use TravisCI and adapt bundle-otp for all releases, not only minor ones, as well as add GPG signing. What do you think? Thank you, Gerhard. On Wed, Jan 9, 2019 at 5:08 PM Gerhard Lazu wrote: > I think it would be great to have checksums publicly available when a new > Erlang/OTP patch is tagged on GitHub. Something as simple as this will do: > > sha256sum OTP-21.2.2.tar.gz > OTP-21.2.2.tar.gz.sha256 > curl --request POST --data-binary "@OTP-21.2.2.tar.gz.sha256" --header > "Content-Type: text/plain" > https://uploads.github.com/repos/erlang/otp/releases/OTP-21.2.2/assets?name=OTP-21.2.2.tar.gz.sha256 > > Is this something that others are missing? If not, how do you answer "*I > know that this Erlang/OTP build is legit*" in your production > environments? > > Thank you, Gerhard. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas@REDACTED Fri Jan 11 15:46:19 2019 From: lukas@REDACTED (Lukas Larsson) Date: Fri, 11 Jan 2019 15:46:19 +0100 Subject: [erlang-questions] Missing checksums for github.com/erlang/otp/releases In-Reply-To: References: Message-ID: On Fri, Jan 11, 2019 at 3:30 PM Gerhard Lazu wrote: > I've noticed that the bundle-otp script in github.com/erlang/otp is used > when minor releases are produced, such as 21.2. I've also noticed that this > script is responsible for creating the bundle.txt which contains the HEAD > git sha at the time of bundling. > > Lukas, I can see that you have released 21.2, as well as 21.1. Would you > be willing to sign OTP releases and upload the signature when creating a > release on GitHub? On team RabbitMQ, this is an automated process for all > public artefacts, I would be happy to help. We can use TravisCI and adapt > bundle-otp for all releases, not only minor ones, as well as add GPG > signing. What do you think? > The bundling script is already done by travis, it just happens to be my user that is used to authenticate with github when updating the artifacts. https://github.com/erlang/otp/blob/master/.travis.yml#L92-L111 The bundler was mainly something I did because Ericsson needed it, but if it can be extended to be usefull to the open source community as well that would be great :) Keep in mind though that one of the things that bundle-otp does it associate a corba version with an Erlang/OTP version. This is only possible to automate for major and minor release, not for patches. So the otp-bundle.tar.gz should not be created for patches, but any GPG signing etc could be done for all tags. > > Thank you, Gerhard. > > On Wed, Jan 9, 2019 at 5:08 PM Gerhard Lazu wrote: > >> I think it would be great to have checksums publicly available when a new >> Erlang/OTP patch is tagged on GitHub. Something as simple as this will do: >> >> sha256sum OTP-21.2.2.tar.gz > OTP-21.2.2.tar.gz.sha256 >> curl --request POST --data-binary "@OTP-21.2.2.tar.gz.sha256" --header >> "Content-Type: text/plain" >> https://uploads.github.com/repos/erlang/otp/releases/OTP-21.2.2/assets?name=OTP-21.2.2.tar.gz.sha256 >> >> Is this something that others are missing? If not, how do you answer "*I >> know that this Erlang/OTP build is legit*" in your production >> environments? >> >> Thank you, Gerhard. >> > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jkrukoff@REDACTED Fri Jan 11 18:20:06 2019 From: jkrukoff@REDACTED (John Krukoff) Date: Fri, 11 Jan 2019 17:20:06 +0000 Subject: [erlang-questions] Include ERTS in an escript package In-Reply-To: <3cf1d511-d99a-3902-b8dd-351b9381e527@ninenines.eu> References: <7557A90B-34F8-4230-910C-A9CA4E27197E@gmail.com> <3cf1d511-d99a-3902-b8dd-351b9381e527@ninenines.eu> Message-ID: <93fea284e0a047bcab2062f6b17e3dc2@EX2013MBX2.bit9.local> Another possible option that recently came up on the Erlang slack channel: https://github.com/dgiagio/warp/tree/master/examples/erlang -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6290 bytes Desc: not available URL: From rickard@REDACTED Fri Jan 11 20:58:35 2019 From: rickard@REDACTED (Rickard Green) Date: Fri, 11 Jan 2019 20:58:35 +0100 Subject: [erlang-questions] Dirty CPU schedulers stuck at zero utilization In-Reply-To: References: Message-ID: Please try the fix in PR-2093 Regards, Rickadr -- Rickard Green, Erlang/OTP, Ericsson AB -------------- next part -------------- An HTML attachment was scrubbed... URL: From gerhard@REDACTED Sat Jan 12 20:28:05 2019 From: gerhard@REDACTED (Gerhard Lazu) Date: Sat, 12 Jan 2019 19:28:05 +0000 Subject: [erlang-questions] Missing checksums for github.com/erlang/otp/releases In-Reply-To: References: Message-ID: Hi, > The bundler was mainly something I did because Ericsson needed it, but if > it can be extended to be usefull to the open source community as well that > would be great :) > > Keep in mind though that one of the things that bundle-otp does it > associate a corba version with an Erlang/OTP version. This is only possible > to automate for major and minor release, not for patches. So the > otp-bundle.tar.gz should not be created for patches, but any GPG signing > etc could be done for all tags. > I will have a go at modifying the .travis.yml to build & GPG sign Erlang/OTP bundles when it detects a tag. I will leave bundle-otp unmodified since it seems to server a different purpose. I figured out how the resulting artefacts get published to github.com/erlang/otp/releases, thank you! I couldn't find your GPG key on sks-keyservers.net. Do you have one Lukas? -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas@REDACTED Mon Jan 14 09:34:41 2019 From: lukas@REDACTED (Lukas Larsson) Date: Mon, 14 Jan 2019 09:34:41 +0100 Subject: [erlang-questions] Missing checksums for github.com/erlang/otp/releases In-Reply-To: References: Message-ID: On Sat, Jan 12, 2019 at 8:28 PM Gerhard Lazu wrote: > Hi, > > >> The bundler was mainly something I did because Ericsson needed it, but if >> it can be extended to be usefull to the open source community as well that >> would be great :) >> >> Keep in mind though that one of the things that bundle-otp does it >> associate a corba version with an Erlang/OTP version. This is only possible >> to automate for major and minor release, not for patches. So the >> otp-bundle.tar.gz should not be created for patches, but any GPG signing >> etc could be done for all tags. >> > > I will have a go at modifying the .travis.yml to build & GPG sign > Erlang/OTP bundles when it detects a tag. I will leave bundle-otp > unmodified since it seems to server a different purpose. I figured out how > the resulting artefacts get published to github.com/erlang/otp/releases, > thank you! > Great! > > I couldn't find your GPG key on sks-keyservers.net. Do you have one Lukas? > Nope, I do not. I can get one though if you just point me to a guide on how to get one :) I've never done anything with GPG so this is all new to me. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From roger@REDACTED Mon Jan 14 10:13:27 2019 From: roger@REDACTED (Roger Lipscombe) Date: Mon, 14 Jan 2019 09:13:27 +0000 Subject: [erlang-questions] SIGUSR1 isn't writing a crash dump? In-Reply-To: References: Message-ID: On Thu, 10 Jan 2019 at 14:44, Roger Lipscombe wrote: > I've got a release, using OTP-21.0, which I'm starting by using a > relx-generated script (relx 3.26.0, managed by erlang.mk). > > When I send it SIGUSR1, it quits, but without writing erl_crash.dump I should point out that sending SIGUSR1 to a "normal" erl works fine. There seems to be something about the way the release is running that breaks it. Any ideas? I'd really like to be able to get crash dumps out of it when I can't get a remote shell. From lloyd@REDACTED Mon Jan 14 17:40:07 2019 From: lloyd@REDACTED (lloyd@REDACTED) Date: Mon, 14 Jan 2019 11:40:07 -0500 (EST) Subject: [erlang-questions] [Zotonic-Dev] zotonic CI deployment best practice for 2019 In-Reply-To: References: Message-ID: <1547484007.511431217@apps.rackspace.com> Hi Eddie, I'm in a similar situation--- planning soon to release a Nitrogen web application but feeling profound apprehension since I don't have a clear understanding of risks and how to minimize them. No doubt the flip answer is, "It depends..." Are we talking cloud hosting or on-premises? What are projected traffic patterns? Etc. Etc. But "It depends..." doesn't help folks like us who lack experience or organizational support. One can find considerable information on the web covering firewalls, proxy servers, load balancers, site hardening, etc. But, like you, I've found little in the Erlang corpus that provides sufficiently clear patterns and guidelines to assuage my Erlang release/production apprehensions. It may well be a book-length topic or more, but even a thoughtful, thorough tutorial, or even a checklist, would be helpful. Please do let me know what you come up with. All the best, Lloyd -----Original Message----- From: layeddie@REDACTED Sent: Monday, January 14, 2019 9:28am To: "Zotonic developers" Subject: [Zotonic-Dev] zotonic CI deployment best practice for 2019 Hi, I have followed zotonic for the last couple of years and am now interested in setting up a production server for zotonic. I have just spent some time searching zotonic users and zotonic developers for an up to date guide on how you would go about developing and deploying zotonic to a production environment. most of the links I found range from between 2010 - 2012. I have looked through the zotonic documentation as well. I have also installed locally using the zotonic full docker container and the manual install. I am looking for a guide for Git / continuous integration / continuous deployment guide and any best practice tips. Is something like this available? Many thanks Eddie L (Uk based)-- --- You received this message because you are subscribed to the Google Groups "Zotonic developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [ zotonic-developers+unsubscribe@REDACTED ]( mailto:zotonic-developers+unsubscribe@REDACTED ). For more options, visit [ https://groups.google.com/d/optout ]( https://groups.google.com/d/optout ). -------------- next part -------------- An HTML attachment was scrubbed... URL: From lloyd@REDACTED Mon Jan 14 19:47:34 2019 From: lloyd@REDACTED (lloyd@REDACTED) Date: Mon, 14 Jan 2019 13:47:34 -0500 (EST) Subject: [erlang-questions] [Zotonic-Dev] zotonic CI deployment best practice for 2019 In-Reply-To: <36BD202E-7907-4499-AA88-ACE61E858877@me.com> References: <1547484007.511431217@apps.rackspace.com> <36BD202E-7907-4499-AA88-ACE61E858877@me.com> Message-ID: <1547491654.501916739@apps.rackspace.com> Hi Marc, Many thanks for your prompt response. This addresses one part of the puzzle. I can't speak for Eddie, but as a total DevOps noob, a big-picture perspective would give me greater comfort as I move into my own deployments. I'm guessing that there are different issues if we're considering on-premises vs. cloud deployment, so considering the two hosting scenarios... What are the specific security, scaling, or other issues we need to attend to? And how do we implement them? For instance, if on-premises: Do we need a dmz to protect our LAN or is port-forwarding sufficient? Should we put a proxy server such as nginx or HAProxy in front of our app? How can we most effectively harden our servers and applications against exploits? I understand much of this goes beyond the realm of Zotonic, Erlang, and Nitrogen. But when I step out into the web for answers I'm totally overwhelmed with confusing and oft contradictory info. I'd love to create a living-document tutorial and checklist for, at least, the simplest deployments, but wouldn't know where to start. Thanks again, Lloyd -----Original Message----- From: "'Marc Worrell' via Zotonic developers" Sent: Monday, January 14, 2019 11:53am To: "'Marc Worrell' via Zotonic developers" Cc: "erlang-questions@REDACTED" Subject: Re: [Zotonic-Dev] zotonic CI deployment best practice for 2019 Hi Eddie and Lloyd, ?It depends? is indeed the correct answer... Most Zotonic installations fall in two categories: 1. Deployment via a Docker container, which is updated before deployment 2. Deployment via git (manual or automatic) Some follow (1), I follow (2). Nice thing of (2) is that it allows for hot code upgrades and quick turnaround of small patches. When we are i a development cycle for new features we might have 10 to 20 deployments per day. This goes very smooth, we even deploy minor updates to Zotonic and dependencies using hot code upgrades. Zotonic itself can watch changes in the file system and dynamically load new files. Most of the updates we deploy have only minor changes. Think of changes to templates, css, or translations. And minor Erlang changes (ie. not completely new apps or other dependencies). That is why we can have updates-via-git without service interruption. The orgs that deploy via Docker (option 1) - have typically less frequent updates and perform periodic major updates. As the whole container is updated this also restarts the Zotonic server. Of course it is also possible to deploy using the OTP release mechanism. This is something that will be possible with the 1.0 (really soon now) of Zotonic, as that version is OTP compliant. I personally never had the need to use OTP releases as the version control via git is good enough for us. (Especially with rebar3 managing versions of dependencies.) Cheers, Marc On 14 Jan 2019, at 17:40, [ lloyd@REDACTED ]( mailto:lloyd@REDACTED ) wrote: Hi Eddie, I'm in a similar situation--- planning soon to release a Nitrogen web application but feeling profound apprehension since I don't have a clear understanding of risks and how to minimize them. No doubt the flip answer is, "It depends..." Are we talking cloud hosting or on-premises? What are projected traffic patterns? Etc. Etc. But "It depends..." doesn't help folks like us who lack experience or organizational support. One can find considerable information on the web covering firewalls, proxy servers, load balancers, site hardening, etc. But, like you, I've found little in the Erlang corpus that provides sufficiently clear patterns and guidelines to assuage my Erlang release/production apprehensions. It may well be a book-length topic or more, but even a thoughtful, thorough tutorial, or even a checklist, would be helpful. Please do let me know what you come up with. All the best, Lloyd -----Original Message-----From: [ layeddie@REDACTED ]( mailto:layeddie@REDACTED )Sent: Monday, January 14, 2019 9:28amTo: "Zotonic developers" <[ zotonic-developers@REDACTED ]( mailto:zotonic-developers@REDACTED )>Subject: [Zotonic-Dev] zotonic CI deployment best practice for 2019 Hi, I have followed zotonic for the last couple of years and am now interested in setting up a production server for zotonic. I have just spent some time searching zotonic users and zotonic developers for an up to date guide on how you would go about developing and deploying zotonic to a production environment. most of the links I found range from between 2010 - 2012. I have looked through the zotonic documentation as well. I have also installed locally using the zotonic full docker container and the manual install. I am looking for a guide for Git / continuous integration / continuous deployment guide and any best practice tips. Is something like this available? Many thanks Eddie L (Uk based)-- --- You received this message because you are subscribed to the Google Groups "Zotonic developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [ zotonic-developers+unsubscribe@REDACTED ]( mailto:zotonic-developers+unsubscribe@REDACTED ). For more options, visit [ https://groups.google.com/d/optout ]( https://groups.google.com/d/optout ).-- --- You received this message because you are subscribed to the Google Groups "Zotonic developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [ zotonic-developers+unsubscribe@REDACTED ]( mailto:zotonic-developers+unsubscribe@REDACTED ). For more options, visit [ https://groups.google.com/d/optout ]( https://groups.google.com/d/optout ). -- --- You received this message because you are subscribed to the Google Groups "Zotonic developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [ zotonic-developers+unsubscribe@REDACTED ]( mailto:zotonic-developers+unsubscribe@REDACTED ). For more options, visit [ https://groups.google.com/d/optout ]( https://groups.google.com/d/optout ). -------------- next part -------------- An HTML attachment was scrubbed... URL: From frank.muller.erl@REDACTED Tue Jan 15 18:18:24 2019 From: frank.muller.erl@REDACTED (Frank Muller) Date: Tue, 15 Jan 2019 18:18:24 +0100 Subject: [erlang-questions] Dirty CPU schedulers stuck at zero utilization In-Reply-To: References: Message-ID: Hello Jesse Did Rickard?s patch (PR-2093) fixed your Dirty scheduler issue? Best /Frank Please try the fix in PR-2093 > > Regards, > Rickadr > -- > Rickard Green, Erlang/OTP, Ericsson AB > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lloyd@REDACTED Tue Jan 15 19:22:05 2019 From: lloyd@REDACTED (lloyd@REDACTED) Date: Tue, 15 Jan 2019 13:22:05 -0500 (EST) Subject: [erlang-questions] [Zotonic-Dev] zotonic CI deployment best practice for 2019 In-Reply-To: References: <1547484007.511431217@apps.rackspace.com> <36BD202E-7907-4499-AA88-ACE61E858877@me.com> <1547491654.501916739@apps.rackspace.com> Message-ID: <1547576525.421728639@apps.rackspace.com> Hi Marc, Really helpful! You're the best. Lloyd -----Original Message----- From: "'Marc Worrell' via Zotonic developers" Sent: Tuesday, January 15, 2019 3:39am To: "'Marc Worrell' via Zotonic developers" Cc: "erlang-questions@REDACTED" Subject: Re: [Zotonic-Dev] zotonic CI deployment best practice for 2019 Hi Lloyd (and rest of list), For Zotonic I have seen two setups: 1. Only Zotonic (Erlang) (single machine) With this setup external port 80 is redirected (using iptables prerouting rules) to 127.0.0.1:8000. And port 443 to 8433. Erlang is directly handling all (SSL) traffic. Same is done with the SMTP port 25, which is usually mapped to 2525. 2. Using a proxy (one or more machines) In this haproxy or nginx are used to terminate the SSL connections and proxy requests to Zotonic/Erlang. The Zotonic node is either running locally or on some other host in local network. In this setup it is also common to have a mail server running to forward incoming email to the Zotonic/Erlang node. Setup 1 (direct) is the easiest for small single-server operations. Setup 2 (proxy) is when you want to have multiple machines and a local network. We have setup 1 running on a ?50/month VPS, handling monthly traffic of 1.5+ TB (~2M monthly visitors). Another company is running setup 1 on dedicated hardware with more than 1M hourly requests. (That server is mostly idle?) So for most (98%?) sites the simple setup is actually very realistic. You just need to be sure that you have a good backup scheme, as it is a single machine. I know of some people that are working on a ?lukewarm? failover setup. With the single VPS server solution we didn?t have any significant down time in the last years and usually have 100% monthly uptime (according to pingdom). So, for me, the single server solution works best. Especially with a VPS where the hosting company moves the VPS in case of any hardware problems. Cheers, Marc On 14 Jan 2019, at 19:47, [ lloyd@REDACTED ]( mailto:lloyd@REDACTED ) wrote: Hi Marc, Many thanks for your prompt response. This addresses one part of the puzzle. I can't speak for Eddie, but as a total DevOps noob, a big-picture perspective would give me greater comfort as I move into my own deployments. I'm guessing that there are different issues if we're considering on-premises vs. cloud deployment, so considering the two hosting scenarios... What are the specific security, scaling, or other issues we need to attend to? And how do we implement them? For instance, if on-premises: Do we need a dmz to protect our LAN or is port-forwarding sufficient? Should we put a proxy server such as nginx or HAProxy in front of our app? How can we most effectively harden our servers and applications against exploits? I understand much of this goes beyond the realm of Zotonic, Erlang, and Nitrogen. But when I step out into the web for answers I'm totally overwhelmed with confusing and oft contradictory info. I'd love to create a living-document tutorial and checklist for, at least, the simplest deployments, but wouldn't know where to start. Thanks again, Lloyd -----Original Message-----From: "'Marc Worrell' via Zotonic developers" <[ zotonic-developers@REDACTED ]( mailto:zotonic-developers@REDACTED )>Sent: Monday, January 14, 2019 11:53amTo: "'Marc Worrell' via Zotonic developers" <[ zotonic-developers@REDACTED ]( mailto:zotonic-developers@REDACTED )>Cc: "[ erlang-questions@REDACTED ]( mailto:erlang-questions@REDACTED )" <[ erlang-questions@REDACTED ]( mailto:erlang-questions@REDACTED )>Subject: Re: [Zotonic-Dev] zotonic CI deployment best practice for 2019 Hi Eddie and Lloyd, ?It depends? is indeed the correct answer... Most Zotonic installations fall in two categories: 1. Deployment via a Docker container, which is updated before deployment 2. Deployment via git (manual or automatic) Some follow (1), I follow (2). Nice thing of (2) is that it allows for hot code upgrades and quick turnaround of small patches. When we are i a development cycle for new features we might have 10 to 20 deployments per day. This goes very smooth, we even deploy minor updates to Zotonic and dependencies using hot code upgrades. Zotonic itself can watch changes in the file system and dynamically load new files. Most of the updates we deploy have only minor changes. Think of changes to templates, css, or translations. And minor Erlang changes (ie. not completely new apps or other dependencies). That is why we can have updates-via-git without service interruption. The orgs that deploy via Docker (option 1) - have typically less frequent updates and perform periodic major updates. As the whole container is updated this also restarts the Zotonic server. Of course it is also possible to deploy using the OTP release mechanism. This is something that will be possible with the 1.0 (really soon now) of Zotonic, as that version is OTP compliant. I personally never had the need to use OTP releases as the version control via git is good enough for us. (Especially with rebar3 managing versions of dependencies.) Cheers, Marc On 14 Jan 2019, at 17:40, [ lloyd@REDACTED ]( mailto:lloyd@REDACTED ) wrote: Hi Eddie, I'm in a similar situation--- planning soon to release a Nitrogen web application but feeling profound apprehension since I don't have a clear understanding of risks and how to minimize them. No doubt the flip answer is, "It depends..." Are we talking cloud hosting or on-premises? What are projected traffic patterns? Etc. Etc. But "It depends..." doesn't help folks like us who lack experience or organizational support. One can find considerable information on the web covering firewalls, proxy servers, load balancers, site hardening, etc. But, like you, I've found little in the Erlang corpus that provides sufficiently clear patterns and guidelines to assuage my Erlang release/production apprehensions. It may well be a book-length topic or more, but even a thoughtful, thorough tutorial, or even a checklist, would be helpful. Please do let me know what you come up with. All the best, Lloyd -----Original Message-----From: [ layeddie@REDACTED ]( mailto:layeddie@REDACTED )Sent: Monday, January 14, 2019 9:28amTo: "Zotonic developers" <[ zotonic-developers@REDACTED ]( mailto:zotonic-developers@REDACTED )>Subject: [Zotonic-Dev] zotonic CI deployment best practice for 2019 Hi, I have followed zotonic for the last couple of years and am now interested in setting up a production server for zotonic. I have just spent some time searching zotonic users and zotonic developers for an up to date guide on how you would go about developing and deploying zotonic to a production environment. most of the links I found range from between 2010 - 2012. I have looked through the zotonic documentation as well. I have also installed locally using the zotonic full docker container and the manual install. I am looking for a guide for Git / continuous integration / continuous deployment guide and any best practice tips. Is something like this available? Many thanks Eddie L (Uk based)-- --- You received this message because you are subscribed to the Google Groups "Zotonic developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [ zotonic-developers+unsubscribe@REDACTED ]( mailto:zotonic-developers+unsubscribe@REDACTED ). For more options, visit [ https://groups.google.com/d/optout ]( https://groups.google.com/d/optout ).-- --- You received this message because you are subscribed to the Google Groups "Zotonic developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [ zotonic-developers+unsubscribe@REDACTED ]( mailto:zotonic-developers+unsubscribe@REDACTED ). For more options, visit [ https://groups.google.com/d/optout ]( https://groups.google.com/d/optout ). -- --- You received this message because you are subscribed to the Google Groups "Zotonic developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [ zotonic-developers+unsubscribe@REDACTED ]( mailto:zotonic-developers+unsubscribe@REDACTED ). For more options, visit [ https://groups.google.com/d/optout ]( https://groups.google.com/d/optout ).-- --- You received this message because you are subscribed to the Google Groups "Zotonic developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [ zotonic-developers+unsubscribe@REDACTED ]( mailto:zotonic-developers+unsubscribe@REDACTED ). For more options, visit [ https://groups.google.com/d/optout ]( https://groups.google.com/d/optout ). -- --- You received this message because you are subscribed to the Google Groups "Zotonic developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [ zotonic-developers+unsubscribe@REDACTED ]( mailto:zotonic-developers+unsubscribe@REDACTED ). For more options, visit [ https://groups.google.com/d/optout ]( https://groups.google.com/d/optout ). -------------- next part -------------- An HTML attachment was scrubbed... URL: From jstimpson@REDACTED Tue Jan 15 22:14:08 2019 From: jstimpson@REDACTED (Jesse Stimpson) Date: Tue, 15 Jan 2019 16:14:08 -0500 Subject: [erlang-questions] Dirty CPU schedulers stuck at zero utilization In-Reply-To: References: Message-ID: My apologies as I've been distracted by other matters recently. I hope to be able to report back within the next few days. Thanks On Tue, Jan 15, 2019 at 12:18 PM Frank Muller wrote: > Hello Jesse > > Did Rickard?s patch (PR-2093) fixed your Dirty scheduler issue? > > Best > /Frank > > Please try the fix in PR-2093 >> >> Regards, >> Rickadr >> -- >> Rickard Green, Erlang/OTP, Ericsson AB >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > -- Jesse Stimpson Site Reliability Engineering m: 9199950424 <(919)%20995-0424> RepublicWireless.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From jstimpson@REDACTED Tue Jan 15 23:11:38 2019 From: jstimpson@REDACTED (Jesse Stimpson) Date: Tue, 15 Jan 2019 17:11:38 -0500 Subject: [erlang-questions] Dirty CPU schedulers stuck at zero utilization In-Reply-To: References: Message-ID: Behavior of the schedulers appears to have the same issue with 2093 patch. But I did notice something new in the msacc output. There is a very brief period, at approx the same time as the normal schedulers usage spikes, where all the dirty cpu schedulers have a significant sleep time. I've included timestamped excerpts below, starting with the increase in dirty cpu sleep, and ending with a "steady state" utilization. Thread aux check_io emulator gc other port sleep {{2019,1,15},{21,57,34}} dirty_cpu_( 1) 0.00% 0.00% 26.80% 0.00% 22.75% 0.00% 50.45% scheduler( 1) 0.17% 0.00% 98.22% 0.50% 0.18% 0.77% 0.16% {{2019,1,15},{21,57,36}} dirty_cpu_( 1) 0.00% 0.00% 76.92% 0.00% 17.73% 0.00% 5.35% scheduler( 1) 0.61% 0.00% 94.20% 1.23% 0.61% 3.35% 0.00% {{2019,1,15},{21,57,38}} dirty_cpu_( 1) 0.00% 0.00% 87.16% 0.00% 6.52% 0.00% 6.31% scheduler( 1) 0.94% 0.14% 93.72% 1.51% 0.63% 2.84% 0.22% {{2019,1,15},{21,57,39}} dirty_cpu_( 1) 0.00% 0.00% 87.92% 0.00% 6.85% 0.00% 5.23% scheduler( 1) 0.98% 0.14% 90.88% 1.32% 3.71% 2.71% 0.26% {{2019,1,15},{21,57,40}} dirty_cpu_( 1) 0.00% 0.00% 67.47% 0.00% 19.51% 0.00% 13.02% scheduler( 1) 1.09% 0.17% 81.95% 1.07% 12.01% 3.08% 0.64% {{2019,1,15},{21,57,41}} dirty_cpu_( 1) 0.00% 0.00% 64.25% 0.00% 18.86% 0.00% 16.88% scheduler( 1) 0.99% 0.28% 74.67% 0.98% 19.56% 2.80% 0.71% Thank you for the continuing support! Jesse On Tue, Jan 15, 2019 at 4:14 PM Jesse Stimpson < jstimpson@REDACTED> wrote: > My apologies as I've been distracted by other matters recently. I hope to > be able to report back within the next few days. > > Thanks > > On Tue, Jan 15, 2019 at 12:18 PM Frank Muller > wrote: > >> Hello Jesse >> >> Did Rickard?s patch (PR-2093) fixed your Dirty scheduler issue? >> >> Best >> /Frank >> >> Please try the fix in PR-2093 >>> >>> Regards, >>> Rickadr >>> -- >>> Rickard Green, Erlang/OTP, Ericsson AB >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >> > > -- > > > > Jesse Stimpson > > Site Reliability Engineering > > m: 9199950424 <(919)%20995-0424> > RepublicWireless.com > -- Jesse Stimpson Site Reliability Engineering m: 9199950424 <(919)%20995-0424> RepublicWireless.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From dch@REDACTED Wed Jan 16 00:44:00 2019 From: dch@REDACTED (Dave Cottlehuber) Date: Wed, 16 Jan 2019 00:44:00 +0100 Subject: [erlang-questions] SIGUSR1 isn't writing a crash dump? In-Reply-To: References: Message-ID: <1547595840.2814265.1635657224.391B787C@webmail.messagingengine.com> On Thu, 10 Jan 2019, at 15:44, Roger Lipscombe wrote: > I've got a release, using OTP-21.0, which I'm starting by using a > relx-generated script (relx 3.26.0, managed by erlang.mk). > > When I send it SIGUSR1, it quits, but without writing erl_crash.dump fwiw I?ve just observed this today as well you?re not alone. Dave -------------- next part -------------- An HTML attachment was scrubbed... URL: From otp@REDACTED Wed Jan 16 12:00:32 2019 From: otp@REDACTED (Erlang/OTP) Date: Wed, 16 Jan 2019 12:00:32 +0100 Subject: [erlang-questions] Patch Package OTP 21.2.3 Released Message-ID: <20190116110032.GA25252@erix.ericsson.se> Patch Package: OTP 21.2.3 Git Tag: OTP-21.2.3 Date: 2019-01-16 Trouble Report Id: OTP-15477, OTP-15492, OTP-15495, OTP-15501, OTP-15504, OTP-15505, OTP-15509 Seq num: ERIERL-283, ERL-371, ERL-514, ERL-790, ERL-807, ERL-821 System: OTP Release: 21 Application: compiler-7.3.1, erts-10.2.2, ssl-9.1.2, xmerl-1.3.19 Predecessor: OTP 21.2.2 Check out the git tag OTP-21.2.3, and build a full OTP system including documentation. Apply one or more applications from this build as patches to your installation using the 'otp_patch_apply' tool. For information on install requirements, see descriptions for each application version below. --------------------------------------------------------------------- --- compiler-7.3.1 -------------------------------------------------- --------------------------------------------------------------------- The compiler-7.3.1 application can be applied independently of other applications on a full OTP 21 installation. --- Fixed Bugs and Malfunctions --- OTP-15501 Application(s): compiler Related Id(s): ERL-514, ERL-807, OTP-14808 An optimization that avoided allocation of a stack frame for some case expressions was introduced in OTP 21. (ERL-504/OTP-14808) It turns out that in rare circumstances, this optimization is not safe. Therefore, this optimization has been disabled. A similar optimization will be included in OTP 22 in a safe way. Full runtime dependencies of compiler-7.3.1: crypto-3.6, erts-9.0, hipe-3.12, kernel-4.0, stdlib-2.5 --------------------------------------------------------------------- --- erts-10.2.2 ----------------------------------------------------- --------------------------------------------------------------------- Note! The erts-10.2.2 application can *not* be applied independently of other applications on an arbitrary OTP 21 installation. On a full OTP 21 installation, also the following runtime dependencies have to be satisfied: -- kernel-6.1 (first satisfied in OTP 21.1) -- sasl-3.3 (first satisfied in OTP 21.2) --- Fixed Bugs and Malfunctions --- OTP-15495 Application(s): erts Related Id(s): ERL-821 Fixed a crash when dangling files were closed after init:restart/0. OTP-15509 Application(s): erts Related Id(s): PR-2027, PR-2093 A bug that could cause dirty schedulers to become unresponsive has been fixed. Full runtime dependencies of erts-10.2.2: kernel-6.1, sasl-3.3, stdlib-3.5 --------------------------------------------------------------------- --- ssl-9.1.2 ------------------------------------------------------- --------------------------------------------------------------------- The ssl-9.1.2 application can be applied independently of other applications on a full OTP 21 installation. --- Fixed Bugs and Malfunctions --- OTP-15477 Application(s): ssl Related Id(s): ERL-790 Fix encoding of the SRP extension length field in ssl. The old encoding of the SRP extension length could cause interoperability problems with third party SSL implementations when SRP was used. OTP-15504 Application(s): ssl Related Id(s): ERL-371 Guarantee active once data delivery, handling TCP stream properly. OTP-15505 Application(s): ssl Correct gen_statem returns for some error cases Full runtime dependencies of ssl-9.1.2: crypto-4.2, erts-10.0, inets-5.10.7, kernel-6.0, public_key-1.5, stdlib-3.5 --------------------------------------------------------------------- --- xmerl-1.3.19 ---------------------------------------------------- --------------------------------------------------------------------- The xmerl-1.3.19 application can be applied independently of other applications on a full OTP 21 installation. --- Fixed Bugs and Malfunctions --- OTP-15492 Application(s): xmerl Related Id(s): ERIERL-283 The charset detection parsing crash in some cases when the XML directive is not syntactic correct. Full runtime dependencies of xmerl-1.3.19: erts-6.0, kernel-3.0, stdlib-2.5 --------------------------------------------------------------------- --------------------------------------------------------------------- --------------------------------------------------------------------- From rickard@REDACTED Wed Jan 16 12:35:25 2019 From: rickard@REDACTED (Rickard Green) Date: Wed, 16 Jan 2019 12:35:25 +0100 Subject: [erlang-questions] Dirty CPU schedulers stuck at zero utilization In-Reply-To: References: Message-ID: <7f7287b8-f6d9-da4b-59ed-52dc2d9c76f7@erlang.org> On 2019-01-15 23:11, Jesse Stimpson wrote: > Behavior of the schedulers appears to have the same issue with 2093 patch. > > But I did notice something new in the msacc output. There is a very > brief period, at approx the same time as the normal schedulers usage > spikes, where all the dirty cpu schedulers have a significant sleep > time. I've included timestamped excerpts below, starting with the > increase in dirty cpu sleep, and ending with a "steady state" utilization. > We just released OTP-21.2.3 containing PR-2093. I don't think PR-2093 cause the spikes. This change does not affect how work is moved between normal and dirty schedulers, only prevents the "loss" of dirty schedulers. If a process is scheduled on a dirty scheduler it wont make progress until it has executed on a dirty scheduler and vice versa (for normal schedulers). This is the same both before and after PR-2093. Since dirty schedulers aren't "lost" after PR-2093 progress of such processes will happen earlier which of course change the behavior, but that is due to the work load. Regards, Rickard From frank.muller.erl@REDACTED Wed Jan 16 13:48:15 2019 From: frank.muller.erl@REDACTED (Frank Muller) Date: Wed, 16 Jan 2019 13:48:15 +0100 Subject: [erlang-questions] Dirty CPU schedulers stuck at zero utilization In-Reply-To: <7f7287b8-f6d9-da4b-59ed-52dc2d9c76f7@erlang.org> References: <7f7287b8-f6d9-da4b-59ed-52dc2d9c76f7@erlang.org> Message-ID: Thanks Rickard. @Jonas: did the PR2093 help in your case? /Frank On 2019-01-15 23:11, Jesse Stimpson wrote: > > Behavior of the schedulers appears to have the same issue with 2093 > patch. > > > > But I did notice something new in the msacc output. There is a very > > brief period, at approx the same time as the normal schedulers usage > > spikes, where all the dirty cpu schedulers have a significant sleep > > time. I've included timestamped excerpts below, starting with the > > increase in dirty cpu sleep, and ending with a "steady state" > utilization. > > > > We just released OTP-21.2.3 containing PR-2093. > > I don't think PR-2093 cause the spikes. This change does not affect how > work is moved between normal and dirty schedulers, only prevents the > "loss" of dirty schedulers. > > If a process is scheduled on a dirty scheduler it wont make progress > until it has executed on a dirty scheduler and vice versa (for normal > schedulers). This is the same both before and after PR-2093. Since dirty > schedulers aren't "lost" after PR-2093 progress of such processes will > happen earlier which of course change the behavior, but that is due to > the work load. > > Regards, > Rickard > -------------- next part -------------- An HTML attachment was scrubbed... URL: From raoknz@REDACTED Wed Jan 16 15:15:52 2019 From: raoknz@REDACTED (Richard O'Keefe) Date: Thu, 17 Jan 2019 03:15:52 +1300 Subject: [erlang-questions] Infix function and user-defined operators In-Reply-To: <6B5DCA31-DDA2-4B04-A0CD-28AB9B8ED718@gmail.com> References: <01B6187C-9731-4740-A770-532B8472AB97@gmail.com> <6B5DCA31-DDA2-4B04-A0CD-28AB9B8ED718@gmail.com> Message-ID: For what it's worth, Fortran also has user-defined operators, of the form /[.][[:alpha:]]+[.]/. I fear that back in 2004 too many people confused *user-defined* *alphabetic* operators with *overloaded* or *symbolic* operators, or even user-defined operators with user-selected precedence. (And some forgot that '#!$@' is already a perfectly good function name...) I recall Bill Clocksin (of "Clocksin & Mellish" Prolog textbook fame) saying that if he ever had a chance to redesign Prolog the one thing he would improve would be to delete user-defined operators... In a functional language I expect there to be fewer opportunities to use prefix or infix operators than in Fortran, because you tend to need more arguments. Speaking of Fortran, that may be why I dislike /foo/ (looks like a COMMON block name), and of course in most languages 1/foo/2 would be an arithmetic expression with two divisions, which would confuse me. On Fri, 11 Jan 2019 at 21:18, Dmitry Kolesnikov wrote: > Hello Richard, > > Thank you for valuable feedback. > > Indeed `name` is the best option. However, it requires changes to the > compiler. Unfortunately, it do not have any visibility why this feature has > been rejected 14 years ago. I found that /name/ is the closed approximation > of Haskell?s `name` achievable with core Erlang syntax thus implementable > via parse transforms. > > Current proposal of user defined operator is left associative with same > precedence as multiplication. > > I also thought about /f> and final conclusion yet. The ?single? character operators can be achieved via > /$_/ with some limitation. One of the problem here is name of module to > conduct its implementation. > > > Best Regards, > Dmitry > > > On 8 Jan 2019, at 15.53, Richard O'Keefe wrote: > > The Haskell convention for user-defined operators is `name` . > I don't see any point in inventing a new one. > I note that > (a) http://mlton.org/InfixingOperators shows a relatively simple > scheme for ML that would fit Erlang without too much strain. It > starts with a warning that operator properties are not part of > module interfaces. This was a problem in Prolog too, and I > surmise that it may have been the reason for not having user- > declared operators in Erlang. > (b) In F#, the type and precedence of a user-defined operator > are determined by its first character. See > > https://docs.microsoft.com/en-us/dotnet/fsharp/language-reference/symbol-and-operator-reference/index > (c) ECMA Eiffel 8.28.5 puts all unary user-defined operators at the > same precedence level and all binary user-defined operators at another > precedence level. Some other Eiffel dialects followed the same rule as > F#. The (b) and (c) rules appear to also be related to not having > operator properties in signatures/classes/modules/whatever. > > More years ago than I care to admit, I wrote an extended > Prolog parser that allowed user-defined "operators" like > "N of L are V" or "at least N of L are V". It was based > on Vaughan Pratt's CGOL; see > https://en.wikipedia.org/wiki/CGOL > > > > On Tue, 8 Jan 2019 at 04:56, Dmitry Kolesnikov > wrote: > >> Hello Community, >> >> Now and then, I am missing an infix notation in Erlang. There was a great >> discussion about this subject decade and half ago but I do not think >> anything is changed [1]. >> >> I am testing and stretching limits of functional abstracts in Erlang as >> part of my pure functional and generic programming exercises [2]. I?ve >> decided to experiment with Infix function via parse-transform. >> >> The parse_transform feature implements a syntax sugar for infix, which is >> compiled into valid Erlang code using corresponding function at >> compile-time. To apply a function using infix notation, we enclose its name >> in slash characters (/). This allows us to use any arity 2 functions as >> infix operators. >> >> ``` >> -compile({parse_transform, infix}). >> >> 1 /plus/ 1. >> >> F /lists:map/ [1, 2, 3]. >> ``` >> >> Right now, I am working on this feature here >> https://github.com/fogfish/datum/pull/64, you can find more detailed >> description there, code, etc. >> >> I would appreciate for any feedback, suggestion from the community. >> Does it sounds any usable, still desired or total braindead? >> What other use-case/applications you might imagine for infix notation? >> >> Best Regards, >> Dmitry >> >> References >> >> 1. http://erlang.org/pipermail/erlang-questions/2004-March/011929.html >> 2. https://github.com/fogfish/datum >> >> >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jstimpson@REDACTED Wed Jan 16 16:00:17 2019 From: jstimpson@REDACTED (Jesse Stimpson) Date: Wed, 16 Jan 2019 10:00:17 -0500 Subject: [erlang-questions] Dirty CPU schedulers stuck at zero utilization In-Reply-To: <7f7287b8-f6d9-da4b-59ed-52dc2d9c76f7@erlang.org> References: <7f7287b8-f6d9-da4b-59ed-52dc2d9c76f7@erlang.org> Message-ID: It's possible that during our tests the utilization spike was masked by the collapse issue fixed in the recent PRs. Is there any other analysis I can provide on the utilization spike/sleep behavior we're seeing, or any other debugging or code reading you recommend? As far as I can tell, there's nothing about our workload that would cause periodic behavior like this. The application is slinging RTP audio via udp to remote endpoints at a 20 msec ptime. Each function call for the NIF in question adds 10 msec of audio to the WebRTC buffer. As point of corroboration, this user on stackoverflow appears to be having the same or a similar issue: https://stackoverflow.com/questions/49563067/erlang-schedulers-just-sleep-why As always, the level of support from the Erlang community is second to none. Thanks to all for your time! Jesse On Wed, Jan 16, 2019 at 6:35 AM Rickard Green wrote: > On 2019-01-15 23:11, Jesse Stimpson wrote: > > Behavior of the schedulers appears to have the same issue with 2093 > patch. > > > > But I did notice something new in the msacc output. There is a very > > brief period, at approx the same time as the normal schedulers usage > > spikes, where all the dirty cpu schedulers have a significant sleep > > time. I've included timestamped excerpts below, starting with the > > increase in dirty cpu sleep, and ending with a "steady state" > utilization. > > > > We just released OTP-21.2.3 containing PR-2093. > > I don't think PR-2093 cause the spikes. This change does not affect how > work is moved between normal and dirty schedulers, only prevents the > "loss" of dirty schedulers. > > If a process is scheduled on a dirty scheduler it wont make progress > until it has executed on a dirty scheduler and vice versa (for normal > schedulers). This is the same both before and after PR-2093. Since dirty > schedulers aren't "lost" after PR-2093 progress of such processes will > happen earlier which of course change the behavior, but that is due to > the work load. > > Regards, > Rickard > -- Jesse Stimpson Site Reliability Engineering m: 9199950424 <(919)%20995-0424> RepublicWireless.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From empro2@REDACTED Wed Jan 16 20:45:55 2019 From: empro2@REDACTED (empro2@REDACTED) Date: Wed, 16 Jan 2019 20:45:55 +0100 Subject: [erlang-questions] Infix function and user-defined operators In-Reply-To: References: <01B6187C-9731-4740-A770-532B8472AB97@gmail.com> <6B5DCA31-DDA2-4B04-A0CD-28AB9B8ED718@gmail.com> Message-ID: <20190116204555.75da33b9@raspy> > >> I would appreciate for any feedback, suggestion from > >> the community. Does it sounds any usable, still > >> desired or total braindead? I distantly recall C++-people fear the inventiveness of programmers in (ab-?)using 'operator' ... > >> What other > >> use-case/applications you might imagine for infix > >> notation? Does infix-notation carry any inherent superiority apart from people being used to infix-expressions? And: is that set of expressions not more or less small and limited to what they have learned in school, at college, ...? I am (so far) missing only infix "B^E". Would unary-prefix "ln A" and "e^A" mostly be used with a bracketed argument anyway? would these be some real improvement compared to log(A) and exp(A) ...? Perhaps some 'in_module M' could clean away "math:" clutter in calculations ... but I am drifting off topic ... Michael -- That which was said, is not that which was spoken, but that which was understood; and none of these comes necessarily close to that which was meant. From frank.muller.erl@REDACTED Thu Jan 17 07:03:22 2019 From: frank.muller.erl@REDACTED (Frank Muller) Date: Thu, 17 Jan 2019 07:03:22 +0100 Subject: [erlang-questions] Dirty CPU schedulers stuck at zero utilization In-Reply-To: References: <7f7287b8-f6d9-da4b-59ed-52dc2d9c76f7@erlang.org> Message-ID: Hey Jesse Glad to hear your Dirty schedulers collapse issue was solved by Rickard?s PR. The 21.2.3 release fixed a similar problem we had with Dirty Schedulers. Jesse: can?t you rewrite your Dirty NIF as a Yielding one? There?s a nice slide deck from ?Andrew Bennett? describing this technique (page 58): https://cdn.rawgit.com/potatosalad/elixirconf2017/master/presentation.pdf Best /Frank It's possible that during our tests the utilization spike was masked by the > collapse issue fixed in the recent PRs. Is there any other analysis I can > provide on the utilization spike/sleep behavior we're seeing, or any other > debugging or code reading you recommend? As far as I can tell, there's > nothing about our workload that would cause periodic behavior like this. > The application is slinging RTP audio via udp to remote endpoints at a 20 > msec ptime. Each function call for the NIF in question adds 10 msec of > audio to the WebRTC buffer. > > As point of corroboration, this user on stackoverflow appears to be having > the same or a similar issue: > https://stackoverflow.com/questions/49563067/erlang-schedulers-just-sleep-why > > As always, the level of support from the Erlang community is second to > none. Thanks to all for your time! > > Jesse > > > > On Wed, Jan 16, 2019 at 6:35 AM Rickard Green wrote: > >> On 2019-01-15 23:11, Jesse Stimpson wrote: >> > Behavior of the schedulers appears to have the same issue with 2093 >> patch. >> > >> > But I did notice something new in the msacc output. There is a very >> > brief period, at approx the same time as the normal schedulers usage >> > spikes, where all the dirty cpu schedulers have a significant sleep >> > time. I've included timestamped excerpts below, starting with the >> > increase in dirty cpu sleep, and ending with a "steady state" >> utilization. >> > >> >> We just released OTP-21.2.3 containing PR-2093. >> >> I don't think PR-2093 cause the spikes. This change does not affect how >> work is moved between normal and dirty schedulers, only prevents the >> "loss" of dirty schedulers. >> >> If a process is scheduled on a dirty scheduler it wont make progress >> until it has executed on a dirty scheduler and vice versa (for normal >> schedulers). This is the same both before and after PR-2093. Since dirty >> schedulers aren't "lost" after PR-2093 progress of such processes will >> happen earlier which of course change the behavior, but that is due to >> the work load. >> >> Regards, >> Rickard >> > > > -- > > > > Jesse Stimpson > > Site Reliability Engineering > > m: 9199950424 <(919)%20995-0424> > RepublicWireless.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikpelinux@REDACTED Thu Jan 17 10:22:31 2019 From: mikpelinux@REDACTED (Mikael Pettersson) Date: Thu, 17 Jan 2019 10:22:31 +0100 Subject: [erlang-questions] Infix function and user-defined operators In-Reply-To: <20190116204555.75da33b9@raspy> References: <01B6187C-9731-4740-A770-532B8472AB97@gmail.com> <6B5DCA31-DDA2-4B04-A0CD-28AB9B8ED718@gmail.com> <20190116204555.75da33b9@raspy> Message-ID: On Wed, Jan 16, 2019 at 8:46 PM wrote: > > >> What other > > >> use-case/applications you might imagine for infix > > >> notation? > > Does infix-notation carry any inherent superiority apart > from people being used to infix-expressions? Ah, this is the rub. As an old-time LISP and Scheme aficionado, I find this obsession with infix notation weird. The generic function call syntax, whether you prefer f(X, Y) or (f X Y), is unambiguous and more flexible since it allows different arities. Infix operators are tolerable when they're few and fixed by the language. Add user-defined ones, overloading, user-defined precedence and associativity, and you get a recipe for unreadable code. No thanks. About the only use for non-prefix notation I can find is for compound literals, where it's useful to be able to say e.g. [X, Y | Z] in _both_ pattern-matching and expression evaluation contexts. (One thing I think LISP got wrong is that the syntax for a compound literal wasn't also the syntax for an expression evaluating to that literal. But I digress.) From otp@REDACTED Thu Jan 17 16:27:02 2019 From: otp@REDACTED (Erlang/OTP) Date: Thu, 17 Jan 2019 16:27:02 +0100 Subject: [erlang-questions] Patch Package OTP 20.3.8.18 Released Message-ID: <20190117152702.GA61743@erix.ericsson.se> Patch Package: OTP 20.3.8.18 Git Tag: OTP-20.3.8.18 Date: 2019-01-17 Trouble Report Id: OTP-15509 Seq num: System: OTP Release: 20 Application: erts-9.3.3.8 Predecessor: OTP 20.3.8.17 Check out the git tag OTP-20.3.8.18, and build a full OTP system including documentation. Apply one or more applications from this build as patches to your installation using the 'otp_patch_apply' tool. For information on install requirements, see descriptions for each application version below. --------------------------------------------------------------------- --- erts-9.3.3.8 ---------------------------------------------------- --------------------------------------------------------------------- The erts-9.3.3.8 application can be applied independently of other applications on a full OTP 20 installation. --- Fixed Bugs and Malfunctions --- OTP-15509 Application(s): erts Related Id(s): PR-2027, PR-2093 A bug that could cause dirty schedulers to become unresponsive has been fixed. Full runtime dependencies of erts-9.3.3.8: kernel-5.0, sasl-3.0.1, stdlib-3.0 --------------------------------------------------------------------- --------------------------------------------------------------------- --------------------------------------------------------------------- From areege.alzubaidi@REDACTED Thu Jan 17 17:48:02 2019 From: areege.alzubaidi@REDACTED (Areege Al Zubaidi) Date: Thu, 17 Jan 2019 17:48:02 +0100 Subject: [erlang-questions] Last chance to get early bird tickets to Code BEAM SF 2019! Message-ID: Hi all, It?s Code BEAM SF time! I wanted to drop you a line to make you aware that this is your last chance to secure early bird tickets to Code BEAM SF! Early bird ends at 23:59 PST tonight. For the uninitiated, Code BEAM SF is the reincarnation of the much loved Erlang and Elixir Factory San Francisco, the biggest and best Erlang and Elixir conference in North America. It?s back this February with a host of brilliant speakers and trainers to share, learn, inspire. At Code BEAM SF this year, OTP, Elixir, Phoenix, RabbitMQ, and Kazoo training courses , as well as certification exams and a host of talks from leaders and innovators in the community, including: - Jose Valim , author of Elixir, who will be keynoting! - Renee Orser , VP of Engineering at NS1, will also join us as a keynote speaker! - Miriam Pena , voted one of the women to watch in tech by Women 2.0 - Fred Hebert , Erlang and distributed systems enthusiast, author of Learn you some Erlang - Ayanda Dube , RabbitMQ lead - Anna Neyzberg , co-founder of ElixirBridge - Hannah Howard , programmer and life long iconoclast - Andrea Leopardi , Elixir core team member, developer and advocate - Lukas Larsson , Erlang VM core committer - Jacqui Manzi will join us to share her experience using OTP: presence to power real-time analytics - ? and more! Learn more about the conference, and the 2019 speakers: https://codesync.global/conferences/code-beam-sf-2019/#Speakers What: Code BEAM SF 2019 When: 28 February - 01 March 2019 Where: Hyatt Centric Fisherman?s Wharf, San Francisco How to attend: - Attend the conference Secure an early bird rate ticket to the conference by 23:59 PST on Thursday 17 January. You can find out more and register here . - Diversity ticket We are pleased to announce that Code BEAM SF 2019 is implementing a diversity scheme . The goal of the programme is to increase diversity of attendees and offer support to those in the tech community who would not otherwise be able to attend the conference. The deadline for applications is 3 February 2019. Recipients will be notified on a rolling basis, no later than 8 February 2019. - Student tickets Code BEAM SF offers a 50% discount to students. To learn more, email info@REDACTED from your academic email address to get a unique discount code. - Group tickets Bring the whole team along! Contact info@REDACTED for group discounts. - Sponsor If you want your company to join WhatsApp, Brex, and The RealReal and support the BEAM community, please do not hesitate to contact us about community support and sponsorship. We hope to see you at Code BEAM SF! Warm regards, Areege -- *Areege Al Zubaidi* *Conferences Marketing Executive* T: +44 (0) 2076550332 W: www.codesync.global A: Erlang Solutions Ltd, The Loom,14 Gower's Walk, London, E1 8PY *Code Sync & Erlang Solutions Conferences* Lambda Days - Krak?w - 21-22 February 2019 Code BEAM SF - San Francisco - 28 February - 1 March 2019 Code BEAM Lite - Bologna - 22 March 2019 ElixirConf EU - Prague - 8-9 April 2019 Code BEAM STO - Stockholm - 16-17 May 2019 Code Elixir LDN - London - 18 July 2019 Code Mesh LDN - London - 7-8 November 2019 *Erlang Solutions and Code Sync care about your data and privacy; please find all details about the basis for communicating with you and the way we process your data in our Privacy Policy . You can update your email preferences or opt-out from receiving marketing emails here .* -------------- next part -------------- An HTML attachment was scrubbed... URL: From erdemaksu@REDACTED Thu Jan 17 21:25:19 2019 From: erdemaksu@REDACTED (Erdem Aksu) Date: Thu, 17 Jan 2019 21:25:19 +0100 Subject: [erlang-questions] Internal compiler atoms In-Reply-To: References: Message-ID: Hello H?kan, If you compile your module from core erlang (cerl) forms and pass 'from_core' option to compile:forms/2 call, compiler will skip many passes including core pass, which calls v3_call:module/2 and will only parse the core forms and run core_passes. Thus you can leap list_to_atom/1 calls that are invoked by v3_call:module/2 call. You can see the link below for the compile call and how cerl is used: https://github.com/pundunlabs/gb_reg/blob/36f3c32ef18386a10e7a3007e2b25d380e9c2e74/src/gb_reg_worker.erl#L236 Br, Erdem Aksu On Thu, Jul 27, 2017 at 4:19 PM H?kan Mattsson wrote: > > I am trying to compile a > ?(? > big > ?)? > module from a list of forms > ?? > , but it seems > ? like the compiler internally generates lots of new atoms:? > > > no more index entries in atom_tab (max=1048576) > > Crash dump is being written to: erl_crash.dump... > > ?In this case the compiler itself generated ?more than 300K atoms while > compiling my forms. > ? > ?Why is the atoms generated?? > > ?Is this anything that can be disabled?? > > /H?kan > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From petergi@REDACTED Fri Jan 18 06:02:45 2019 From: petergi@REDACTED (Peter J Etheridge) Date: Fri, 18 Jan 2019 16:02:45 +1100 Subject: [erlang-questions] exported variables Message-ID: <83c4d2241e1051df521a216391d0cb6d9f51dea3@webmail.iinet.net.au> Dear Erlangers, in regard to exporting variables, is this still the case; http://erlang.org/pipermail/erlang-questions/2004-April/012104.html thank you in advance,Peter -------------- next part -------------- An HTML attachment was scrubbed... URL: From koops.j@REDACTED Fri Jan 18 15:04:21 2019 From: koops.j@REDACTED (Jeroen Koops) Date: Fri, 18 Jan 2019 15:04:21 +0100 Subject: [erlang-questions] Why is this expression illegal? Message-ID: Hi List! If this expressions is legal: case A of foo -> 1; bar -> 2 end bor case A of foo -> 8; bar -> 16 end. ... and this expression is legal: << case A of foo -> 1; bar -> 2 end >>. ... then why is this expression illegal: << case A of foo -> 1; bar -> 2 end bor case A of foo -> 8; bar -> 16 end >>. (shell responds with "* 1: syntax error before: 'bor'") Thanks, -- Jeroen Koops M: koops.j@REDACTED T: +31-6-55590300 -------------- next part -------------- An HTML attachment was scrubbed... URL: From elbrujohalcon@REDACTED Fri Jan 18 15:08:19 2019 From: elbrujohalcon@REDACTED (Brujo Benavides) Date: Fri, 18 Jan 2019 11:08:19 -0300 Subject: [erlang-questions] Why is this expression illegal? In-Reply-To: References: Message-ID: <75E66A53-861D-4185-A7FF-DD259C9CDBF3@gmail.com> I believe that inline operators (like bor) are disallowed in binary constructions. See? 1> << 1 bor 16 >>. * 1: syntax error before: 'bor' 1> << 1 band 16 >>. * 1: syntax error before: 'band' 1> << (1 band 16) >>. <<0>> Brujo Benavides > On 18 Jan 2019, at 11:04, Jeroen Koops wrote: > > Hi List! > > If this expressions is legal: > > case A of foo -> 1; bar -> 2 end bor case A of foo -> 8; bar -> 16 end. > > ... and this expression is legal: > > << case A of foo -> 1; bar -> 2 end >>. > > ... then why is this expression illegal: > > << case A of foo -> 1; bar -> 2 end bor case A of foo -> 8; bar -> 16 end >>. > > (shell responds with "* 1: syntax error before: 'bor'") > > Thanks, > > -- > Jeroen Koops > > M: koops.j@REDACTED > T: +31-6-55590300 > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From freza@REDACTED Fri Jan 18 15:19:07 2019 From: freza@REDACTED (=?utf-8?Q?J=C3=A1chym=20Hole=C4=8Dek?=) Date: Fri, 18 Jan 2019 15:19:07 +0100 Subject: [erlang-questions] Why is this expression illegal? In-Reply-To: References: Message-ID: <1547821147.3249804.1638016912.0AD20CE2@webmail.messagingengine.com> Hi Joeren, My guess: allowing entirely arbitrary expression within binary constructorsyntax could lead to ambiguities with respect to `/` and `-` and `:` tokenswhich have special meaning in this context? Therefore only select expressionforms are permitted, which are free of ambiguities, one of them being `( E )`which in turn opens access to arbitrary expressions if one is a bit more explicit. So this will work: 4> << (case foo of foo -> 1; bar -> 2 end bor case foo of foo -> 8; bar -> 16 end) >>. <<"\t">> The file `stdlib-3.4.4/src/erl_parse.yrl` has the mechanics of it (search for `binary`,`bin_elements`, `bin_element`) though not the underlying reasoning. HTH, -- Jachym On Fri, Jan 18, 2019, at 15:04, Jeroen Koops wrote: > Hi List! > > If this expressions is legal: > > case A of foo -> 1; bar -> 2 end bor case A of foo -> 8; bar -> 16 > end.> > ... and this expression is legal: > > > << case A of foo -> 1; bar -> 2 end >>. > > ... then why is this expression illegal: > > > << case A of foo -> 1; bar -> 2 end bor case A of foo -> 8; bar -> > 16 end >>.> > (shell responds with "* 1: syntax error before: 'bor'") > > Thanks, > > -- > Jeroen Koops > > M: koops.j@REDACTED > T: +31-6-55590300 > _________________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From koops.j@REDACTED Fri Jan 18 15:25:04 2019 From: koops.j@REDACTED (Jeroen Koops) Date: Fri, 18 Jan 2019 15:25:04 +0100 Subject: [erlang-questions] Why is this expression illegal? In-Reply-To: <1547821147.3249804.1638016912.0AD20CE2@webmail.messagingengine.com> References: <1547821147.3249804.1638016912.0AD20CE2@webmail.messagingengine.com> Message-ID: Thanks Jachym and Brujo, I did try << (case foo of foo -> 1; bar -> 2 end) bor (case foo of foo -> 8; bar -> 16 end) >>. which didn't work, but << (case foo of foo -> 1; bar -> 2 end bor case foo of foo -> 8; bar -> 16 end) >>. indeed does work. On Fri, Jan 18, 2019 at 3:19 PM J?chym Hole?ek wrote: > Hi Joeren, > > My guess: allowing entirely arbitrary expression within binary constructor > syntax could lead to ambiguities with respect to `/` and `-` and `:` tokens > which have special meaning in this context? Therefore only select > expression > forms are permitted, which are free of ambiguities, one of them being `( E > )` > which in turn opens access to arbitrary expressions if one is a bit more > explicit. > > So this will work: > > 4> << (case foo of foo -> 1; bar -> 2 end bor case foo of foo -> 8; > bar -> 16 end) >>. > <<"\t">> > > The file `stdlib-3.4.4/src/erl_parse.yrl` has the mechanics of it (search > for `binary`, > `bin_elements`, `bin_element`) though not the underlying reasoning. > > HTH, > -- Jachym > > On Fri, Jan 18, 2019, at 15:04, Jeroen Koops wrote: > > Hi List! > > If this expressions is legal: > > case A of foo -> 1; bar -> 2 end bor case A of foo -> 8; bar -> 16 end. > > > ... and this expression is legal: > > > > << case A of foo -> 1; bar -> 2 end >>. > > > ... then why is this expression illegal: > > > > << case A of foo -> 1; bar -> 2 end bor case A of foo -> 8; bar -> 16 end > >>. > > > (shell responds with "* 1: syntax error before: 'bor'") > > > Thanks, > > -- > Jeroen Koops > > M: koops.j@REDACTED > T: +31-6-55590300 > *_______________________________________________* > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Jeroen Koops M: koops.j@REDACTED T: +31-6-55590300 -------------- next part -------------- An HTML attachment was scrubbed... URL: From dieter@REDACTED Fri Jan 18 18:37:12 2019 From: dieter@REDACTED (=?UTF-8?Q?Dieter_Sch=c3=b6n?=) Date: Fri, 18 Jan 2019 18:37:12 +0100 Subject: [erlang-questions] Why is this expression illegal? In-Reply-To: References: <1547821147.3249804.1638016912.0AD20CE2@webmail.messagingengine.com> Message-ID: <795995e1-616b-8988-de65-d9115b3e4cf1@schoen.or.at> Hi, in http://erlang.org/doc/programming_examples/bit_syntax.html#constructing-binaries-and-bitstrings I found: As mentioned earlier, segments have the following general syntax: Value:Size/TypeSpecifierList When constructing binaries, Value and Size can be any Erlang expression. However, for syntactical reasons, both Value and Size must be enclosed in parenthesis if the expression consists of anything more than a single literal or a variable. The following gives a compiler syntax error: <> This expression must be rewritten into the following, to be accepted by the compiler: <<(X+1):8>> Seems to me that this condition makes live for the parser a bit easier. kind regards, Dieter On 18.01.19 15:25, Jeroen Koops wrote: > Thanks Jachym and Brujo, > > I did try > > << (case foo of foo -> 1; bar -> 2 end) bor (case foo of foo -> 8; bar > -> 16 end) >>. > > which didn't work, but > > << (case foo of foo -> 1; bar -> 2 end bor case foo of foo -> 8; bar > -> 16 end) >>. > > indeed does work. > > > > On Fri, Jan 18, 2019 at 3:19 PM J?chym Hole?ek > wrote: > > Hi Joeren, > > My guess: allowing entirely arbitrary expression within binary > constructor > syntax could lead to ambiguities with respect to `/` and `-` and > `:` tokens > which have special meaning in this context? Therefore only select > expression > forms are permitted, which are free of ambiguities, one of them > being `( E )` > which in turn opens access to arbitrary expressions if one is a > bit more explicit. > > So this will work: > > ? ??4> << (case foo of foo -> 1; bar -> 2 end bor case foo of foo > -> 8; bar -> 16 end) >>. > ? ? <<"\t">> > > The file `stdlib-3.4.4/src/erl_parse.yrl` has the mechanics of it > (search for `binary`, > `bin_elements`, `bin_element`) though not the underlying reasoning. > > HTH, > ? ? -- Jachym > > On Fri, Jan 18, 2019, at 15:04, Jeroen Koops wrote: >> Hi List! >> >> If this expressions is legal: >> >> case A of foo -> 1; bar -> 2 end bor case A of foo -> 8; bar -> >> 16 end. >> >> >> ... and this expression is legal: >> >> >> >> << case A of foo -> 1; bar -> 2 end >>. >> >> >> ... then why is this expression illegal: >> >> >> >> << case A of foo -> 1; bar -> 2 end bor case A of foo -> 8; bar >> -> 16 end >>. >> >> >> (shell responds with "* 1: syntax error before: 'bor'") >> >> >> Thanks, >> >> >> -- >> Jeroen Koops >> >> M: koops.j@REDACTED >> T: +31-6-55590300 >> _________________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > -- > Jeroen Koops > > M: koops.j@REDACTED > T: +31-6-55590300 > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlsson.richard@REDACTED Sat Jan 19 11:43:45 2019 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Sat, 19 Jan 2019 11:43:45 +0100 Subject: [erlang-questions] exported variables In-Reply-To: <83c4d2241e1051df521a216391d0cb6d9f51dea3@webmail.iinet.net.au> References: <83c4d2241e1051df521a216391d0cb6d9f51dea3@webmail.iinet.net.au> Message-ID: Some time after that discussion, the compiler was changed to warn only for those variables that occur in a pattern and are exported from an above case/if/receive/try. So you can export a variable from the clauses of a case and then use it in a computation without complaints from the compiler, but if you use it in a match, you get a warning because it's likely that you thought it was actually free. /Richard Den fre 18 jan. 2019 kl 06:03 skrev Peter J Etheridge : > Dear Erlangers, > > in regard to exporting variables, is this still the case; > > http://erlang.org/pipermail/erlang-questions/2004-April/012104.html > > thank you in advance, > Peter > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From frank.muller.erl@REDACTED Sat Jan 19 12:04:33 2019 From: frank.muller.erl@REDACTED (Frank Muller) Date: Sat, 19 Jan 2019 12:04:33 +0100 Subject: [erlang-questions] Supervisor restart backoff Message-ID: Hi guys What?s the status if this PR: https://github.com/erlang/otp/pull/1287 Is it stable enough and why it?s not merged? Can I use it PROD? /Frank -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlsson.richard@REDACTED Sat Jan 19 12:12:07 2019 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Sat, 19 Jan 2019 12:12:07 +0100 Subject: [erlang-questions] Supervisor restart backoff In-Reply-To: References: Message-ID: Just too many things to do. I had a discussion with OTP about it and we agreed that it should be a feature of the supervisor, but at the same time they had rewritten some things which caused a big merge conflict and I had to rethink some details, and it's been in limbo ever since. I'd like to try to get it done though. Probably not for recommended for production in the current state. /Richard Den l?r 19 jan. 2019 kl 12:04 skrev Frank Muller : > Hi guys > > What?s the status if this PR: > https://github.com/erlang/otp/pull/1287 > > Is it stable enough and why it?s not merged? Can I use it PROD? > > > /Frank > -------------- next part -------------- An HTML attachment was scrubbed... URL: From frank.muller.erl@REDACTED Sat Jan 19 12:33:02 2019 From: frank.muller.erl@REDACTED (Frank Muller) Date: Sat, 19 Jan 2019 12:33:02 +0100 Subject: [erlang-questions] Supervisor restart backoff In-Reply-To: References: Message-ID: Ok, thanks for the update. Just too many things to do. I had a discussion with OTP about it and we > agreed that it should be a feature of the supervisor, but at the same time > they had rewritten some things which caused a big merge conflict and I had > to rethink some details, and it's been in limbo ever since. I'd like to try > to get it done though. Probably not for recommended for production in the > current state. > > /Richard > > > Den l?r 19 jan. 2019 kl 12:04 skrev Frank Muller < > frank.muller.erl@REDACTED>: > >> Hi guys >> >> What?s the status if this PR: >> https://github.com/erlang/otp/pull/1287 >> >> Is it stable enough and why it?s not merged? Can I use it PROD? >> >> >> /Frank >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From petergi@REDACTED Sun Jan 20 04:55:09 2019 From: petergi@REDACTED (Peter J Etheridge) Date: Sun, 20 Jan 2019 14:55:09 +1100 Subject: [erlang-questions] exported variables Message-ID: <7f309f3e6a6384dd2653a4c3d2302d1be74a9777@webmail.iinet.net.au> Message-ID: Thank you for the update, Richard. -------------- next part -------------- An HTML attachment was scrubbed... URL: From raoknz@REDACTED Sun Jan 20 13:25:56 2019 From: raoknz@REDACTED (Richard O'Keefe) Date: Mon, 21 Jan 2019 01:25:56 +1300 Subject: [erlang-questions] Infix function and user-defined operators In-Reply-To: <20190116204555.75da33b9@raspy> References: <01B6187C-9731-4740-A770-532B8472AB97@gmail.com> <6B5DCA31-DDA2-4B04-A0CD-28AB9B8ED718@gmail.com> <20190116204555.75da33b9@raspy> Message-ID: It's ironic that you mention C++, because C++ is *the* example of operator abuse. I mean, *really*, using "<<" for both left shift and output? That's almost as evil as using "+" for string concatenation. The problem with C++ is that it allows completely ad hoc overloading BUT does not allow operator *introduction* so that there is enormous pressure on the same few operators that happen to be built in. The example of X^Y is an interesting one, because you don't, you really don't, want to go around writing exp(Y*log(X)) all the time. Not only does it involve three operations, when only one is intended, but in many cases they are the *wrong* operations. (Notably, but not only, when Y is an integer.) The other thing is the notation itself. There is no such operator in high-school maths. And the reason ^ is used is that in ASCII-63 (as implemented on the Model 33 teletype, for one popular example), that codepoint was used for ? so people were actually writing X?Y. ASCII-67 replaced ? with ^. And there were two operators ASCII meant to give us that we seldom get. The \ character was added to ASCII for the specific purpose of letting you write /\ (and, min, meet) and \/ (or, max, join). How many programming languages let you use _those_ very well established operators? There are many things that might reasonably be taken as infix operators: List includes Element Long begins_with Short X <=> Y -- increasingly popular 3-way comparison (these examples were chosen to be at the same level as >=) Mat1 %*% Mat2 -- matrix multiplication in R Set1 union: Set2 -- set operation in Smalltalk and of course assorted parser combinators and other combinators. I count 28 operators in SML. (Proper handling of IEEE comparison needs more than six "relational operators", and SML has some but not all of the extras.) Here's an interesting question. The designers of Ada thought that letting people mix up 'and' and 'or' without parentheses was dangerous. So you can have e0 and e1 and ... and en and you can have e0 or e1 or ... or en but you cannot have e0 and e1 or e2 So are 'and' and 'or' operators in Ada, or should they be thought of as special multi-operator forms? If the latter, how would you declare such things? My own impression, and it is no more than that, is that infix operators help a lot when you are treating something as a quasi-algebraic expression that you are interested in rearranging and simplifying in a mathematical way. I don't think it is a coincidence that the Bird-Meertens approach does that, and that specification languages often have lots of operators. In particular, when you are transforming expressions by hand, you can tolerate a fair bit of precedence fuzziness and even outright ambiguity. On the other hand, bulkier, clunkier, but more obvious syntax seems to work better for code that you want other people to read. Just an impression, for what it is worth. > > > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fchschneider@REDACTED Sun Jan 20 17:10:13 2019 From: fchschneider@REDACTED (Frans Schneider) Date: Sun, 20 Jan 2019 17:10:13 +0100 Subject: [erlang-questions] Dialyzer Message-ID: <6d1a70d5-9324-6eb7-6a1b-0247b201939f@gmail.com> Dear lsit, I have the following function plus specification: -spec new(Seq_nums, DDS_filter) -> Cfrs when Seq_nums :: nonempty_list(rtps:sequence_number()), DDS_filter :: fun((rtps:sequence_number()) -> boolean()), Cfrs :: nonempty_list(cfr()). new(Seq_nums, DDS_filter) -> [#cfr{sequence_number = N, status = unsent, is_relevant = not DDS_filter(N)} || N <- Seq_nums]. Dialyzer complains with: Invalid type specification for function rtps_cfr:new/2. The success typing is ([],_) -> [] The filter is an external function, What is wrong with the spec? TIA Frans From mononcqc@REDACTED Sun Jan 20 17:34:01 2019 From: mononcqc@REDACTED (Fred Hebert) Date: Sun, 20 Jan 2019 11:34:01 -0500 Subject: [erlang-questions] Dialyzer In-Reply-To: <6d1a70d5-9324-6eb7-6a1b-0247b201939f@gmail.com> References: <6d1a70d5-9324-6eb7-6a1b-0247b201939f@gmail.com> Message-ID: <20190120163400.GA1915@ferdmbp.local> On 01/20, Frans Schneider wrote: >Dear lsit, > >I have the following function plus specification: > >-spec new(Seq_nums, DDS_filter) -> Cfrs when > Seq_nums :: nonempty_list(rtps:sequence_number()), > DDS_filter :: fun((rtps:sequence_number()) -> boolean()), > Cfrs :: nonempty_list(cfr()). > >new(Seq_nums, DDS_filter) -> > [#cfr{sequence_number = N, status = unsent, > is_relevant = not DDS_filter(N)} > || N <- Seq_nums]. > >Dialyzer complains with: > >Invalid type specification for function rtps_cfr:new/2. The success >typing is ([],_) -> [] > >The filter is an external function, >What is wrong with the spec? > You should look at the call-site. There's a possibility that the only places that call the function use arguments that are lists of terms that don't match the sequence number type. In doing so, dialyzer "removes" the types with these as crashing (i.e. DDS_filter would raise an exception). What is left is the set of empty lists (once all the filters are done) which only gives ([],_) -> [] That's at least my guess without seeing the calling code. From fchschneider@REDACTED Sun Jan 20 18:31:01 2019 From: fchschneider@REDACTED (Frans Schneider) Date: Sun, 20 Jan 2019 18:31:01 +0100 Subject: [erlang-questions] Dialyzer In-Reply-To: <20190120163400.GA1915@ferdmbp.local> References: <6d1a70d5-9324-6eb7-6a1b-0247b201939f@gmail.com> <20190120163400.GA1915@ferdmbp.local> Message-ID: <64700a1c-4a22-ce0d-8e2c-588669c06df0@gmail.com> Yep, it were the nonempty lists which caused the problem. I used nonempty because _I_ know that the function never is called with an empty list, but that is not relevant for the specs of the function. On 1/20/19 5:34 PM, Fred Hebert wrote: > On 01/20, Frans Schneider wrote: >> Dear lsit, >> >> I have the following function plus specification: >> >> -spec new(Seq_nums, DDS_filter) -> Cfrs when >> ???? Seq_nums :: nonempty_list(rtps:sequence_number()), >> ???? DDS_filter :: fun((rtps:sequence_number()) -> boolean()), >> ???? Cfrs :: nonempty_list(cfr()). >> >> new(Seq_nums, DDS_filter) -> >> ?? [#cfr{sequence_number = N, status = unsent, >> ???????? is_relevant = not DDS_filter(N)} >> ??? || N <- Seq_nums]. >> >> Dialyzer complains with: >> >> Invalid type specification for function rtps_cfr:new/2. The success >> typing is ([],_) -> [] >> >> The filter is an external function, >> What is wrong with the spec? >> > > You should look at the call-site. There's a possibility that the only > places that call the function use arguments that are lists of terms that > don't match the sequence number type. In doing so, dialyzer "removes" > the types with these as crashing (i.e. DDS_filter would raise an > exception). > > What is left is the set of empty lists (once all the filters are done) > which only gives ([],_) -> [] > > That's at least my guess without seeing the calling code. From empro2@REDACTED Sun Jan 20 19:59:56 2019 From: empro2@REDACTED (empro2@REDACTED) Date: Sun, 20 Jan 2019 19:59:56 +0100 Subject: [erlang-questions] Infix function and user-defined operators In-Reply-To: References: <01B6187C-9731-4740-A770-532B8472AB97@gmail.com> <6B5DCA31-DDA2-4B04-A0CD-28AB9B8ED718@gmail.com> <20190116204555.75da33b9@raspy> Message-ID: <20190120195956.363e9698@raspy> > It's ironic that you mention C++, because C++ is *the* > example of operator abuse. Ironic? That is the reason why I mentioned it. I put the "ab-" in brackets because I am too stupid to get anywhere near a licence to mention naked mole rats. > The example of X^Y is an interesting one, because you > don't, you really don't, want to go around writing > exp(Y*log(X)) all the time. math:pow(X, Y)? (Sorry, if I am being stupid.) Speaking of it: I do not understand why there is no math:log/2 ... but, lest there be any misunderstanding, I do not want it as an infix fun! (or should I?) > My own impression, and it is no more than that, > is that infix operators help a lot when you are treating > something as a quasi-algebraic expression that you are > interested in rearranging and simplifying in a > mathematical way. But is that the operators in themselves or the being used to them? Are there Lispers who do not have to "translate" between "common" algebraic and Lisp phrasings (like I have to)? or who even rearrange and simplify s-exps more easily than ...? To reverse the poles (how punny): some Forth people claim that understanding Forth is an enlightening experience ... (then again, sawing off ones foot might also be ;-) When this thread came up, one of my first thoughts was: why not stop _teaching_ any infix notation at all and forget all these lexically invisible precedence and associativeness rules? (I mitigated this heresy to that question about "inherent superiority".) > and even outright ambiguity. On the other hand, bulkier, > clunkier, but more obvious syntax seems to work better > for code that you want other people to read. That could explain why coding style guides often suggest or demand use of brackets instead of relying on operator rules. -- ?Even after a thousand explanations a fool is no wiser, whereas someone intelligent requires only one fourth of these.? ? from the Mah?bh?rata (???????) From soverdor@REDACTED Mon Jan 21 02:01:29 2019 From: soverdor@REDACTED (Sam Overdorf) Date: Sun, 20 Jan 2019 17:01:29 -0800 Subject: [erlang-questions] what is point of having a .erlang file? Message-ID: What is the point of having a".erlang" file if I have to run "c:erlangrc()" in all of my programs. I like to add a search a path when all of my programs run and not just some of them. It shouldn't make any difference if I run them with "erl" or "escript". Thanks, Sam Overdorf soverdor@REDACTED From bombadil@REDACTED Mon Jan 21 14:58:42 2019 From: bombadil@REDACTED (Manuel A. Rubio) Date: Mon, 21 Jan 2019 14:58:42 +0100 Subject: [erlang-questions] [ANN] Erlang and Elixir books translation project from Spanish Message-ID: Hello, I wrote a couple of books in Erlang/OTP in Spanish and a friend of mine told me he wanted to read it but because they were only in Spanish he couldn't. I realize the material I use inside of the book (examples mainly) are a bit different from other books and maybe it could help to people to understand better who Erlang works if they didn't before with other existent material. In the same way, I think it could be worth to have more material available to learn Erlang so, I started a crowdfunding project to support the translation of these books to English: https://www.kickstarter.com/projects/altenwald/erlang-otp-and-elixir-books-to-english/ I hope you think like me and support the initiative. Thanks! Manuel Rubio. From jesper.louis.andersen@REDACTED Mon Jan 21 16:34:46 2019 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Mon, 21 Jan 2019 16:34:46 +0100 Subject: [erlang-questions] what is point of having a .erlang file? In-Reply-To: References: Message-ID: On Mon, Jan 21, 2019 at 1:53 AM Sam Overdorf wrote: > What is the point of having a".erlang" file if I have to run > "c:erlangrc()" in all of my programs. > > Usually, it is to have a set of development changes and configurations to the Erlang system. There might be some helper functionality you wrote for yourself, or tend to use in projects you want to check out. Rather than having to add them to each project, it is somewhat easier to configure them from a central point. This is also, roughly, its historical usage to my knowledge. However, it is a doubly edged sword, insofar it also means that the software you run has local configuration and setup. If more than one developer works on the same set of software, the local configuration can mean the software doesn't run without the .erlangrc specific configuration. To a large extend, the focus on standard tooling such as Continuous Integration/Continuous Deployment seek to alleviate this. An even more modern approach is to always build everything the same way scratch such that the software is built in a reproducible way with idempotence as well. My experience is that the more these (admittedly functional programming-like) ways of handling software builds are adapted, the easier it gets to work on software. -------------- next part -------------- An HTML attachment was scrubbed... URL: From soverdor@REDACTED Mon Jan 21 22:54:03 2019 From: soverdor@REDACTED (Sam Overdorf) Date: Mon, 21 Jan 2019 13:54:03 -0800 Subject: [erlang-questions] what is point of having a .erlang file? In-Reply-To: References: Message-ID: I have customers and I want the same configuration for all of them so ".erlang" sounds like a good option for me. The problems is "erl" will run the ".erlang" file but "escript" does not run the ".erlang" file. I prefer to use "escript" because the consistent handling of arguments passed in. What is the solution? Thanks, Sam Overdorf soverdor@REDACTED On Mon, Jan 21, 2019 at 7:35 AM Jesper Louis Andersen wrote: > > On Mon, Jan 21, 2019 at 1:53 AM Sam Overdorf wrote: >> >> What is the point of having a".erlang" file if I have to run >> "c:erlangrc()" in all of my programs. >> > > Usually, it is to have a set of development changes and configurations to the Erlang system. There might be some helper functionality you wrote for yourself, or tend to use in projects you want to check out. Rather than having to add them to each project, it is somewhat easier to configure them from a central point. This is also, roughly, its historical usage to my knowledge. > > However, it is a doubly edged sword, insofar it also means that the software you run has local configuration and setup. If more than one developer works on the same set of software, the local configuration can mean the software doesn't run without the .erlangrc specific configuration. To a large extend, the focus on standard tooling such as Continuous Integration/Continuous Deployment seek to alleviate this. An even more modern approach is to always build everything the same way scratch such that the software is built in a reproducible way with idempotence as well. > > My experience is that the more these (admittedly functional programming-like) ways of handling software builds are adapted, the easier it gets to work on software. > From hugo@REDACTED Mon Jan 21 23:17:29 2019 From: hugo@REDACTED (Hugo Mills) Date: Mon, 21 Jan 2019 22:17:29 +0000 Subject: [erlang-questions] what is point of having a .erlang file? In-Reply-To: References: Message-ID: <20190121221729.GM5134@carfax.org.uk> On Mon, Jan 21, 2019 at 01:54:03PM -0800, Sam Overdorf wrote: > I have customers and I want the same configuration for all of them so > ".erlang" sounds like a good option for me. > > The problems is "erl" will run the ".erlang" file but "escript" does > not run the ".erlang" file. > > I prefer to use "escript" because the consistent handling of arguments > passed in. > > What is the solution? I would suggest to use releases for your non-escript packages, and use either escript's ability to package a full runtime system in with the escript, or use it's ability to pass command-line options in the second or third line of the script. To quote the man page on that: On the third line (or second line depending on the presence of the Emacs directive), arguments can be specified to the emula? tor, for example: %%! -smp enable -sname factorial -mnesia debug verbose Hugo. > Thanks, > Sam Overdorf > soverdor@REDACTED > > > On Mon, Jan 21, 2019 at 7:35 AM Jesper Louis Andersen > wrote: > > > > On Mon, Jan 21, 2019 at 1:53 AM Sam Overdorf wrote: > >> > >> What is the point of having a".erlang" file if I have to run > >> "c:erlangrc()" in all of my programs. > >> > > > > Usually, it is to have a set of development changes and configurations to the Erlang system. There might be some helper functionality you wrote for yourself, or tend to use in projects you want to check out. Rather than having to add them to each project, it is somewhat easier to configure them from a central point. This is also, roughly, its historical usage to my knowledge. > > > > However, it is a doubly edged sword, insofar it also means that the software you run has local configuration and setup. If more than one developer works on the same set of software, the local configuration can mean the software doesn't run without the .erlangrc specific configuration. To a large extend, the focus on standard tooling such as Continuous Integration/Continuous Deployment seek to alleviate this. An even more modern approach is to always build everything the same way scratch such that the software is built in a reproducible way with idempotence as well. > > > > My experience is that the more these (admittedly functional programming-like) ways of handling software builds are adapted, the easier it gets to work on software. > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- Hugo Mills | Gort! Klaatu barada nikto! hugo@REDACTED carfax.org.uk | http://carfax.org.uk/ | PGP: E2AB1DE4 | The Day the Earth Stood Still -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: Digital signature URL: From aronisstav@REDACTED Tue Jan 22 09:57:07 2019 From: aronisstav@REDACTED (Stavros Aronis) Date: Tue, 22 Jan 2019 09:57:07 +0100 Subject: [erlang-questions] Dialyzer In-Reply-To: <64700a1c-4a22-ce0d-8e2c-588669c06df0@gmail.com> References: <6d1a70d5-9324-6eb7-6a1b-0247b201939f@gmail.com> <20190120163400.GA1915@ferdmbp.local> <64700a1c-4a22-ce0d-8e2c-588669c06df0@gmail.com> Message-ID: On Sun, Jan 20, 2019 at 6:31 PM Frans Schneider wrote: > > Yep, it were the nonempty lists which caused the problem. I used > nonempty because _I_ know that the function never is called with an > empty list, but that is not relevant for the specs of the function. Good, but this is not what Fred pointed out! It is likely that this function is *never* called with values that satisfy the specified type for `Seq_nums`, either because the calls themselves are bad (not likely if the code is working) or because the type definition for `Seq_nums` is wrong. By removing the 'nonempty', you're **silencing** the warning that the only way for the function to succeed as specified, is if passed an empty list. > On 1/20/19 5:34 PM, Fred Hebert wrote: > > On 01/20, Frans Schneider wrote: > >> Dear lsit, > >> > >> I have the following function plus specification: > >> > >> -spec new(Seq_nums, DDS_filter) -> Cfrs when > >> Seq_nums :: nonempty_list(rtps:sequence_number()), > >> DDS_filter :: fun((rtps:sequence_number()) -> boolean()), > >> Cfrs :: nonempty_list(cfr()). > >> > >> new(Seq_nums, DDS_filter) -> > >> [#cfr{sequence_number = N, status = unsent, > >> is_relevant = not DDS_filter(N)} > >> || N <- Seq_nums]. > >> > >> Dialyzer complains with: > >> > >> Invalid type specification for function rtps_cfr:new/2. The success > >> typing is ([],_) -> [] > >> > >> The filter is an external function, > >> What is wrong with the spec? > >> > > > > You should look at the call-site. There's a possibility that the only > > places that call the function use arguments that are lists of terms that > > don't match the sequence number type. In doing so, dialyzer "removes" > > the types with these as crashing (i.e. DDS_filter would raise an > > exception). > > > > What is left is the set of empty lists (once all the filters are done) > > which only gives ([],_) -> [] > > > > That's at least my guess without seeing the calling code. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From ml-erlang@REDACTED Tue Jan 22 11:32:06 2019 From: ml-erlang@REDACTED (Matthias Rieber) Date: Tue, 22 Jan 2019 11:32:06 +0100 (CET) Subject: [erlang-questions] mnesia_leveldb + prefix keys Message-ID: Hello, I've a question regarding the prefix keys in mnesia_eleveldb. I have a record like this: -record(key, { jid :: { binary(), binary() }, timestamp :: erlang:timestamp() }). -record(table, { key = key{}, payload :: term() }). As far as I understand these queries are fast: mnesia:match_object( #table{key=#key{jid={<<"user1">>, <<"localhost">>}, timestamp = '_'}, payload='_'}) Since the keys are ordered I expected that query to be fast: mnesia:select(table, ets:fun2ms( fun(#table{key=#key{jid={<<"user1">>, <<"localhost">>}, timestamp = Timstamp}, payload=P}) when Timstamp > {1,2,3} -> P end) But performance tests show that this will read all records (probably only the ones with the matching jid). Is it possible to select this in an efficient manner? Best regards, Matthias From mikpelinux@REDACTED Tue Jan 22 13:41:10 2019 From: mikpelinux@REDACTED (Mikael Pettersson) Date: Tue, 22 Jan 2019 13:41:10 +0100 Subject: [erlang-questions] mnesia_leveldb + prefix keys In-Reply-To: References: Message-ID: On Tue, Jan 22, 2019 at 11:32 AM Matthias Rieber wrote: > > Hello, > > I've a question regarding the prefix keys in mnesia_eleveldb. I have a > record like this: > > -record(key, { > jid :: { binary(), binary() }, > timestamp :: erlang:timestamp() > }). > > -record(table, { > key = key{}, > payload :: term() > }). > > > As far as I understand these queries are fast: > > mnesia:match_object( > #table{key=#key{jid={<<"user1">>, <<"localhost">>}, > timestamp = '_'}, > payload='_'}) > > Since the keys are ordered I expected that query to be fast: > > mnesia:select(table, ets:fun2ms( > fun(#table{key=#key{jid={<<"user1">>, <<"localhost">>}, > timestamp = Timstamp}, > payload=P}) when Timstamp > {1,2,3} -> P > end) > > But performance tests show that this will read all records (probably only > the ones with the matching jid). Is it possible to select this in an > efficient manner? Your observations are correct. The second case is a "range select" which LevelDB can support efficiently, but mnesia_eleveldb doesn't implement that optimization yet. I've had a ticket to implement this for a while now, but the initial prototype didn't work and I haven't had time to up-prioritize it yet. Meanwhile we've implemented another optimization in mnesia_eleveldb: we no longer store the tags of records in mnesia_eleveldb tables as those tags are invariant and redundant. This reduces I/O and CPU usage. /Mikael From roger@REDACTED Tue Jan 22 15:15:27 2019 From: roger@REDACTED (Roger Lipscombe) Date: Tue, 22 Jan 2019 14:15:27 +0000 Subject: [erlang-questions] what is point of having a .erlang file? In-Reply-To: References: Message-ID: On Mon, 21 Jan 2019 at 21:45, Sam Overdorf wrote: > > I have customers and I want the same configuration for all of them so > ".erlang" sounds like a good option for me. ~/.erlang is for when you want *different* configuration. From dszoboszlay@REDACTED Tue Jan 22 15:19:48 2019 From: dszoboszlay@REDACTED (=?UTF-8?Q?D=C3=A1niel_Szoboszlay?=) Date: Tue, 22 Jan 2019 15:19:48 +0100 Subject: [erlang-questions] mnesia_leveldb + prefix keys In-Reply-To: References: Message-ID: Until the optimisations mentioned by Mikael are implemented, you can reach good performance by writing the loop manually: loop(mnesia:next(table, #key{jid={<<"user1">>, <<"localhost">>}, timestamp = {1,2,3}}), []). loop(K = #key{jid={<<"user1">>, <<"localhost">>}}, Acc) -> loop(mnesia:next(table, K), mnesia:read(table, K) ++ Acc); loop(_, Acc) -> Acc. On Tue, 22 Jan 2019 at 13:41 Mikael Pettersson wrote: > On Tue, Jan 22, 2019 at 11:32 AM Matthias Rieber > wrote: > > > > Hello, > > > > I've a question regarding the prefix keys in mnesia_eleveldb. I have a > > record like this: > > > > -record(key, { > > jid :: { binary(), binary() }, > > timestamp :: erlang:timestamp() > > }). > > > > -record(table, { > > key = key{}, > > payload :: term() > > }). > > > > > > As far as I understand these queries are fast: > > > > mnesia:match_object( > > #table{key=#key{jid={<<"user1">>, <<"localhost">>}, > > timestamp = '_'}, > > payload='_'}) > > > > Since the keys are ordered I expected that query to be fast: > > > > mnesia:select(table, ets:fun2ms( > > fun(#table{key=#key{jid={<<"user1">>, <<"localhost">>}, > > timestamp = Timstamp}, > > payload=P}) when Timstamp > {1,2,3} -> P > > end) > > > > But performance tests show that this will read all records (probably only > > the ones with the matching jid). Is it possible to select this in an > > efficient manner? > > Your observations are correct. The second case is a "range select" > which LevelDB can support efficiently, but mnesia_eleveldb doesn't > implement that optimization yet. I've had a ticket to implement this > for a while now, but the initial prototype didn't work and I haven't > had time to up-prioritize it yet. > > Meanwhile we've implemented another optimization in mnesia_eleveldb: > we no longer store the tags of records in mnesia_eleveldb tables as > those tags are invariant and redundant. This reduces I/O and CPU > usage. > > /Mikael > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinoski@REDACTED Tue Jan 22 16:27:06 2019 From: vinoski@REDACTED (Steve Vinoski) Date: Tue, 22 Jan 2019 10:27:06 -0500 Subject: [erlang-questions] what is point of having a .erlang file? In-Reply-To: References: Message-ID: On Tue, Jan 22, 2019 at 9:15 AM Roger Lipscombe wrote: > On Mon, 21 Jan 2019 at 21:45, Sam Overdorf wrote: > > > > I have customers and I want the same configuration for all of them so > > ".erlang" sounds like a good option for me. > > ~/.erlang is for when you want *different* configuration. > Roger is correct, and also, you've asked this same question in two other threads in the recent past: http://erlang.org/pipermail/erlang-questions/2018-September/096428.html http://erlang.org/pipermail/erlang-questions/2018-December/096858.html The answers you've already gotten clearly aren't what you want to hear, but they've consistently indicated you're misusing the ~/.erlang feature. --steve -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Tue Jan 22 17:43:54 2019 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Tue, 22 Jan 2019 17:43:54 +0100 Subject: [erlang-questions] what is point of having a .erlang file? In-Reply-To: References: Message-ID: On Mon, Jan 21, 2019 at 10:45 PM Sam Overdorf wrote: > I have customers and I want the same configuration for all of them so > ".erlang" sounds like a good option for me. > > The problems is "erl" will run the ".erlang" file but "escript" does > not run the ".erlang" file. > > I prefer to use "escript" because the consistent handling of arguments > passed in. > > What is the solution? > > Usually, Erlang systems are run as releases. They specify configuration in a file, usually sys.config but that file can read further configuration from another file, or you can override configuration on the command line. However, I am curious about your process of running your software. Maybe, if you explained it, we would be able to describe an alternative way of handling the distribution and running of said software. -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Tue Jan 22 20:56:19 2019 From: max.lapshin@REDACTED (Max Lapshin) Date: Tue, 22 Jan 2019 22:56:19 +0300 Subject: [erlang-questions] Dirty CPU schedulers stuck at zero utilization In-Reply-To: References: <7f7287b8-f6d9-da4b-59ed-52dc2d9c76f7@erlang.org> Message-ID: why do you do it via nif? On Wed, Jan 16, 2019 at 6:00 PM Jesse Stimpson < jstimpson@REDACTED> wrote: > It's possible that during our tests the utilization spike was masked by > the collapse issue fixed in the recent PRs. Is there any other analysis I > can provide on the utilization spike/sleep behavior we're seeing, or any > other debugging or code reading you recommend? As far as I can tell, > there's nothing about our workload that would cause periodic behavior like > this. The application is slinging RTP audio via udp to remote endpoints at > a 20 msec ptime. Each function call for the NIF in question adds 10 msec of > audio to the WebRTC buffer. > > As point of corroboration, this user on stackoverflow appears to be having > the same or a similar issue: > https://stackoverflow.com/questions/49563067/erlang-schedulers-just-sleep-why > > As always, the level of support from the Erlang community is second to > none. Thanks to all for your time! > > Jesse > > > > On Wed, Jan 16, 2019 at 6:35 AM Rickard Green wrote: > >> On 2019-01-15 23:11, Jesse Stimpson wrote: >> > Behavior of the schedulers appears to have the same issue with 2093 >> patch. >> > >> > But I did notice something new in the msacc output. There is a very >> > brief period, at approx the same time as the normal schedulers usage >> > spikes, where all the dirty cpu schedulers have a significant sleep >> > time. I've included timestamped excerpts below, starting with the >> > increase in dirty cpu sleep, and ending with a "steady state" >> utilization. >> > >> >> We just released OTP-21.2.3 containing PR-2093. >> >> I don't think PR-2093 cause the spikes. This change does not affect how >> work is moved between normal and dirty schedulers, only prevents the >> "loss" of dirty schedulers. >> >> If a process is scheduled on a dirty scheduler it wont make progress >> until it has executed on a dirty scheduler and vice versa (for normal >> schedulers). This is the same both before and after PR-2093. Since dirty >> schedulers aren't "lost" after PR-2093 progress of such processes will >> happen earlier which of course change the behavior, but that is due to >> the work load. >> >> Regards, >> Rickard >> > > > -- > > > > Jesse Stimpson > > Site Reliability Engineering > > m: 9199950424 <(919)%20995-0424> > RepublicWireless.com > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From t6sn7gt@REDACTED Tue Jan 22 21:10:47 2019 From: t6sn7gt@REDACTED (Donald Steven) Date: Tue, 22 Jan 2019 15:10:47 -0500 Subject: [erlang-questions] Updating elements of a record Message-ID: <08771336-4746-49c2-26f9-f2e48e14cae1@aim.com> I would like to pass a list of some fields of a record to a function to update those fields, leaving the other fields as they were.? The list will be different for each function call.? For example, one call might want to change fields 1, 2 and 7, another call only field 8, another call fields 3 and 4.? Of course, each field will have a unique name.? Each call to updateStatus (let's call it) would look like? updateStatus(Status, List_of_Fields_to_Update) -> code. -------------------------------- I can't figure put how to do this economically, without a long succession of tests like: extract fieldnames from list... case field1name of ??????? true -> update field1; ??????? false -> ok % do nothing end, repeat for each field... -------------------------------- Any suggestions? Many thanks. Don From nathaniel@REDACTED Tue Jan 22 21:23:30 2019 From: nathaniel@REDACTED (Nathaniel Waisbrot) Date: Tue, 22 Jan 2019 15:23:30 -0500 Subject: [erlang-questions] Updating elements of a record In-Reply-To: <08771336-4746-49c2-26f9-f2e48e14cae1@aim.com> References: <08771336-4746-49c2-26f9-f2e48e14cae1@aim.com> Message-ID: <7a22bde8-08ca-4e99-a8d4-f4b1585c8b57@www.fastmail.com> On Tue, Jan 22, 2019, at 3:10 PM, Donald Steven wrote: > I would like to pass a list of some fields of a record to a function to > update those fields, leaving the other fields as they were. The list > will be different for each function call. For example, one call might > want to change fields 1, 2 and 7, another call only field 8, another > call fields 3 and 4. Of course, each field will have a unique name. > Each call to updateStatus (let's call it) would look like > updateStatus(Status, List_of_Fields_to_Update) -> code. > > -------------------------------- > > I can't figure put how to do this economically, without a long > succession of tests like: > > extract fieldnames from list... > > case field1name of > true -> update field1; > false -> ok % do nothing > end, > > repeat for each field... > > > -------------------------------- > > Any suggestions? Do you need it to be a record? This sounds like a use-case for maps: update_fields(Data, []) -> Data; update_fields(Data, [Field|Rest]) -> Value = maps:get(Field, Data), NewValue = update_value(Value), NewData = maps:put(Field, NewValue, Data), update_fields(NewData, Rest). -------------- next part -------------- An HTML attachment was scrubbed... URL: From roger@REDACTED Tue Jan 22 21:57:03 2019 From: roger@REDACTED (Roger Lipscombe) Date: Tue, 22 Jan 2019 20:57:03 +0000 Subject: [erlang-questions] Updating elements of a record In-Reply-To: <7a22bde8-08ca-4e99-a8d4-f4b1585c8b57@www.fastmail.com> References: <08771336-4746-49c2-26f9-f2e48e14cae1@aim.com> <7a22bde8-08ca-4e99-a8d4-f4b1585c8b57@www.fastmail.com> Message-ID: On Tue, Jan 22, 2019, at 3:10 PM, Donald Steven wrote: > I would like to pass a list of some fields of a record to a function to > update those fields, leaving the other fields as they were. The list > will be different for each function call. For example, one call might > want to change fields 1, 2 and 7, another call only field 8, another > call fields 3 and 4. Of course, each field will have a unique name. > Each call to updateStatus (let's call it) would look like > updateStatus(Status, List_of_Fields_to_Update) -> code. > Records are just tuples. Use setelement with #rec.field: -module(rec). -export([new/0, update/3, test/0]). -record(rec, {a, b, c}). new() -> #rec{}. update(R, Val, [F | Fields]) -> R2 = setelement(F, R, Val), update(R2, Val, Fields); update(R, _Val, []) -> R. test() -> R = rec:new(), R2 = rec:update(R, 2, [#rec.a, #rec.c]), R3 = rec:update(R2, 1, [#rec.b]), R3. From igor.clark@REDACTED Tue Jan 22 23:11:15 2019 From: igor.clark@REDACTED (Igor Clark) Date: Tue, 22 Jan 2019 22:11:15 +0000 Subject: [erlang-questions] Attaching to running release in docker container Message-ID: Hi folks, I'm running an R21 release inside a docker container, basically doing what https://github.com/erlang/docker-erlang-example does. (It works great for my app, so thanks for that!) When I do this: docker exec -ti /app/bin/app attach I get this: failed to attach, user '' does not have sufficient privileges on > '/tmp/erl_pipes/mwc@REDACTED/', please run node as a different user I can run 'remote_console' just fine, and everything else is working as expected too. I've tried hacking it to work by setting ${USER} via the environment (in which case it just says "user 'XX" does not have privileges..."), and even setting all the vars I retrieve via os:getenv() when running under remote_console, but no dice. When I 'docker exec' with /bin/sh and have a look in /tmp, there's nothing in there at all, as expected, and certainly no 'erl_pipes'. Which I guess might be something to do with the build steps. Any clues on how to get this working? Thanks! Igor -------------- next part -------------- An HTML attachment was scrubbed... URL: From t6sn7gt@REDACTED Tue Jan 22 23:48:46 2019 From: t6sn7gt@REDACTED (Donald Steven) Date: Tue, 22 Jan 2019 17:48:46 -0500 Subject: [erlang-questions] Updating elements of a record In-Reply-To: References: <08771336-4746-49c2-26f9-f2e48e14cae1@aim.com> <7a22bde8-08ca-4e99-a8d4-f4b1585c8b57@www.fastmail.com> Message-ID: Thanks Roger.? I see how this works, but the issue is as much how to get it to do: test([{a, Val},{c, Val}]), leaving, for example, field b untouched. On 1/22/2019 3.57 PM, Roger Lipscombe wrote: > On Tue, Jan 22, 2019, at 3:10 PM, Donald Steven wrote: >> I would like to pass a list of some fields of a record to a function to >> update those fields, leaving the other fields as they were. The list >> will be different for each function call. For example, one call might >> want to change fields 1, 2 and 7, another call only field 8, another >> call fields 3 and 4. Of course, each field will have a unique name. >> Each call to updateStatus (let's call it) would look like >> updateStatus(Status, List_of_Fields_to_Update) -> code. >> > Records are just tuples. Use setelement with #rec.field: > > -module(rec). > -export([new/0, update/3, test/0]). > > -record(rec, {a, b, c}). > > new() -> > #rec{}. > > update(R, Val, [F | Fields]) -> > R2 = setelement(F, R, Val), > update(R2, Val, Fields); > update(R, _Val, []) -> > R. > > test() -> > R = rec:new(), > R2 = rec:update(R, 2, [#rec.a, #rec.c]), > R3 = rec:update(R2, 1, [#rec.b]), > R3. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From t@REDACTED Tue Jan 22 23:59:48 2019 From: t@REDACTED (Tristan Sloughter) Date: Tue, 22 Jan 2019 17:59:48 -0500 Subject: [erlang-questions] =?utf-8?q?Attaching_to_running_release_in_dock?= =?utf-8?q?er_container?= In-Reply-To: References: Message-ID: `attach` is only for if you use the `start` (the Dockerfile example you link to uses `foreground`) which runs the release with `run_erl` (see http://erlang.org/doc/man/run_erl.html). I would advise against using `start` and thus not using `attach`. I don't think it would even be possible to make work with a docker container and would not be useful even if it could. I wish I had removed `start` and `attach` from relx years ago because they cause confusion often and I think are rarely, if ever, needed. Maybe it will be done for the next major release of relx and rebar3 :) Tristan On Tue, Jan 22, 2019, at 15:11, Igor Clark wrote: > Hi folks, > > I'm running an R21 release inside a docker container, basically doing what https://github.com/erlang/docker-erlang-example does. (It works great for my app, so thanks for that!) > > When I do this: > >> docker exec -ti /app/bin/app attach > > I get this: > >> failed to attach, user '' does not have sufficient privileges on '/tmp/erl_pipes/mwc@REDACTED/', please run node as a different user > > I can run 'remote_console' just fine, and everything else is working as expected too. I've tried hacking it to work by setting ${USER} via the environment (in which case it just says "user 'XX" does not have privileges..."), and even setting all the vars I retrieve via os:getenv() when running under remote_console, but no dice. > > When I 'docker exec' with /bin/sh and have a look in /tmp, there's nothing in there at all, as expected, and certainly no 'erl_pipes'. Which I guess might be something to do with the build steps. > > Any clues on how to get this working? > > Thanks! > Igor > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From t6sn7gt@REDACTED Wed Jan 23 00:04:13 2019 From: t6sn7gt@REDACTED (Donald Steven) Date: Tue, 22 Jan 2019 18:04:13 -0500 Subject: [erlang-questions] Updating elements of a record In-Reply-To: References: <08771336-4746-49c2-26f9-f2e48e14cae1@aim.com> <7a22bde8-08ca-4e99-a8d4-f4b1585c8b57@www.fastmail.com> Message-ID: I should have said: test(Rec, [{a, Val},{c, Val}]) On 1/22/2019 5.48 PM, Donald Steven wrote: > Thanks Roger.? I see how this works, but the issue is as much how to > get it to do: > > test([{a, Val},{c, Val}]), leaving, for example, field b untouched. > > On 1/22/2019 3.57 PM, Roger Lipscombe wrote: >> On Tue, Jan 22, 2019, at 3:10 PM, Donald Steven wrote: >>> I would like to pass a list of some fields of a record to a function to >>> update those fields, leaving the other fields as they were. The list >>> will be different for each function call.? For example, one call might >>> want to change fields 1, 2 and 7, another call only field 8, another >>> call fields 3 and 4.? Of course, each field will have a unique name. >>> Each call to updateStatus (let's call it) would look like >>> updateStatus(Status, List_of_Fields_to_Update) -> code. >>> >> Records are just tuples. Use setelement with #rec.field: >> >> -module(rec). >> -export([new/0, update/3, test/0]). >> >> -record(rec, {a, b, c}). >> >> new() -> >> ???? #rec{}. >> >> update(R, Val, [F | Fields]) -> >> ???? R2 = setelement(F, R, Val), >> ???? update(R2, Val, Fields); >> update(R, _Val, []) -> >> ???? R. >> >> test() -> >> ???? R = rec:new(), >> ???? R2 = rec:update(R, 2, [#rec.a, #rec.c]), >> ???? R3 = rec:update(R2, 1, [#rec.b]), >> ???? R3. >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > From igor.clark@REDACTED Wed Jan 23 00:24:52 2019 From: igor.clark@REDACTED (Igor Clark) Date: Tue, 22 Jan 2019 23:24:52 +0000 Subject: [erlang-questions] Attaching to running release in docker container In-Reply-To: References: Message-ID: <2CA163E2-5640-41CD-A230-11793BB607E3@gmail.com> Ah ok! Got it. Thanks Tristan. > On 22 Jan 2019, at 22:59, Tristan Sloughter wrote: > > `attach` is only for if you use the `start` (the Dockerfile example you link to uses `foreground`) which runs the release with `run_erl` (see http://erlang.org/doc/man/run_erl.html). > > I would advise against using `start` and thus not using `attach`. I don't think it would even be possible to make work with a docker container and would not be useful even if it could. > > I wish I had removed `start` and `attach` from relx years ago because they cause confusion often and I think are rarely, if ever, needed. > > Maybe it will be done for the next major release of relx and rebar3 :) > > Tristan > >> On Tue, Jan 22, 2019, at 15:11, Igor Clark wrote: >> Hi folks, >> >> I'm running an R21 release inside a docker container, basically doing what https://github.com/erlang/docker-erlang-example does. (It works great for my app, so thanks for that!) >> >> When I do this: >> >> docker exec -ti /app/bin/app attach >> >> I get this: >> >> failed to attach, user '' does not have sufficient privileges on '/tmp/erl_pipes/mwc@REDACTED/', please run node as a different user >> >> I can run 'remote_console' just fine, and everything else is working as expected too. I've tried hacking it to work by setting ${USER} via the environment (in which case it just says "user 'XX" does not have privileges..."), and even setting all the vars I retrieve via os:getenv() when running under remote_console, but no dice. >> >> When I 'docker exec' with /bin/sh and have a look in /tmp, there's nothing in there at all, as expected, and certainly no 'erl_pipes'. Which I guess might be something to do with the build steps. >> >> Any clues on how to get this working? >> >> Thanks! >> Igor >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From gergo.nyiro@REDACTED Tue Jan 22 23:19:27 2019 From: gergo.nyiro@REDACTED (=?UTF-8?B?TnlpcsWRIEdlcmfFkQ==?=) Date: Tue, 22 Jan 2019 23:19:27 +0100 Subject: [erlang-questions] How to mimic Erlang nodes in PropEr statem test Message-ID: Dear Erlangers, I have a simple project [1] to learn distributed Erlang where I wrote a proper_statem test [2] to simulate the communication between Erlang nodes. Everything fine and nice until I have a single process to communicate with, but I want to cut my program into multiple processes, but I don't know how should I register the processes to avoid pid passing and run multiple process group on the same VM. Which process registry is used in this situation? Best regard, Gerg? [1]: https://github.com/nyirog/crdt [2]: https://github.com/nyirog/crdt/blob/master/test/prop_crdt_cluster.erl From ml-erlang@REDACTED Wed Jan 23 08:22:34 2019 From: ml-erlang@REDACTED (Matthias Rieber) Date: Wed, 23 Jan 2019 08:22:34 +0100 (CET) Subject: [erlang-questions] mnesia_leveldb + prefix keys In-Reply-To: References: Message-ID: Hi Mikael, On Tue, 22 Jan 2019, Mikael Pettersson wrote: > On Tue, Jan 22, 2019 at 11:32 AM Matthias Rieber wrote: [...] > > Since the keys are ordered I expected that query to be fast: > > > > mnesia:select(table, ets:fun2ms( > > fun(#table{key=#key{jid={<<"user1">>, <<"localhost">>}, > > timestamp = Timstamp}, > > payload=P}) when Timstamp > {1,2,3} -> P > > end) > > > > But performance tests show that this will read all records (probably only > > the ones with the matching jid). Is it possible to select this in an > > efficient manner? > > Your observations are correct. The second case is a "range select" > which LevelDB can support efficiently, but mnesia_eleveldb doesn't > implement that optimization yet. I've had a ticket to implement this > for a while now, but the initial prototype didn't work and I haven't > had time to up-prioritize it yet. > > Meanwhile we've implemented another optimization in mnesia_eleveldb: > we no longer store the tags of records in mnesia_eleveldb tables as > those tags are invariant and redundant. This reduces I/O and CPU > usage. thanks for your answer. I'll try that! Matthias From ml-erlang@REDACTED Wed Jan 23 08:27:44 2019 From: ml-erlang@REDACTED (Matthias Rieber) Date: Wed, 23 Jan 2019 08:27:44 +0100 (CET) Subject: [erlang-questions] mnesia_leveldb + prefix keys In-Reply-To: References: Message-ID: Hi D?niel, On Tue, 22 Jan 2019, D?niel Szoboszlay wrote: > Until the optimisations mentioned by Mikael are implemented, you can reach > good performance by writing the loop manually: > > loop(mnesia:next(table, #key{jid={<<"user1">>, <<"localhost">>}, timestamp > = {1,2,3}}), []). > > loop(K = #key{jid={<<"user1">>, <<"localhost">>}}, Acc) -> > loop(mnesia:next(table, K), mnesia:read(table, K) ++ Acc); > loop(_, Acc) -> Acc. Thanks! This works. I was not aware that the key does not have to exist in mnesia:next. Matthias From raimo+erlang-questions@REDACTED Wed Jan 23 17:07:58 2019 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Wed, 23 Jan 2019 17:07:58 +0100 Subject: [erlang-questions] Updating elements of a record In-Reply-To: References: <08771336-4746-49c2-26f9-f2e48e14cae1@aim.com> <7a22bde8-08ca-4e99-a8d4-f4b1585c8b57@www.fastmail.com> Message-ID: <20190123160758.GA88201@erix.ericsson.se> On Tue, Jan 22, 2019 at 06:04:13PM -0500, Donald Steven wrote: > I should have said: > > test(Rec, [{a, Val},{c, Val}]) > > On 1/22/2019 5.48 PM, Donald Steven wrote: > > Thanks Roger.? I see how this works, but the issue is as much how to > > get it to do: > > > > test([{a, Val},{c, Val}]), leaving, for example, field b untouched. setelement/3 does exactly that; copies the other fields > > > > On 1/22/2019 3.57 PM, Roger Lipscombe wrote: > >> On Tue, Jan 22, 2019, at 3:10 PM, Donald Steven wrote: > >>> I would like to pass a list of some fields of a record to a function to > >>> update those fields, leaving the other fields as they were. The list > >>> will be different for each function call.? For example, one call might > >>> want to change fields 1, 2 and 7, another call only field 8, another > >>> call fields 3 and 4.? Of course, each field will have a unique name. > >>> Each call to updateStatus (let's call it) would look like > >>> updateStatus(Status, List_of_Fields_to_Update) -> code. > >>> > >> Records are just tuples. Use setelement with #rec.field: > >> > >> -module(rec). > >> -export([new/0, update/3, test/0]). > >> > >> -record(rec, {a, b, c}). > >> > >> new() -> > >> ???? #rec{}. > >> > >> update(R, Val, [F | Fields]) -> > >> ???? R2 = setelement(F, R, Val), > >> ???? update(R2, Val, Fields); > >> update(R, _Val, []) -> > >> ???? R. > >> > >> test() -> > >> ???? R = rec:new(), > >> ???? R2 = rec:update(R, 2, [#rec.a, #rec.c]), > >> ???? R3 = rec:update(R2, 1, [#rec.b]), > >> ???? R3. > >> _______________________________________________ > >> erlang-questions mailing list > >> erlang-questions@REDACTED > >> http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From mononcqc@REDACTED Wed Jan 23 18:52:42 2019 From: mononcqc@REDACTED (Fred Hebert) Date: Wed, 23 Jan 2019 12:52:42 -0500 Subject: [erlang-questions] How to mimic Erlang nodes in PropEr statem test In-Reply-To: References: Message-ID: <20190123175241.GC1915@ferdmbp.local> On 01/22, Nyir? Gerg? wrote: >nice until I have a single process to communicate with, but I want to >cut my program >into multiple processes, but I don't know how should I register the >processes to avoid >pid passing and run multiple process group on the same VM. You already name all three processes according to if they were node names. Can you be a bit more specific about what you intend to do? Is giving names not good enough? From jstimpson@REDACTED Wed Jan 23 19:14:26 2019 From: jstimpson@REDACTED (Jesse Stimpson) Date: Wed, 23 Jan 2019 13:14:26 -0500 Subject: [erlang-questions] Dirty CPU schedulers stuck at zero utilization In-Reply-To: References: <7f7287b8-f6d9-da4b-59ed-52dc2d9c76f7@erlang.org> Message-ID: To clarify about our workload, the NIF execution itself is around 1 msec, the data on which is operates represents 10 msec of audio. Apologies if my last message was unclear. Out of convenience, we're using the open source WebRTC project to take advantage of their built in PLC, FEC, Opus, etc. The project is written in C++, so we have integrated with it via NIF. Unfortunately, re-writing using a yielding NIF, or re-writing to Erlang is not as straightforward as we would like. Although I admit it would alleviate our scheduling issues. We'll continue our testing with 21.2.3 and report back if there appear to be any other leads. Thanks, Jesse On Tue, Jan 22, 2019 at 2:56 PM Max Lapshin wrote: > why do you do it via nif? > > On Wed, Jan 16, 2019 at 6:00 PM Jesse Stimpson < > jstimpson@REDACTED> wrote: > >> It's possible that during our tests the utilization spike was masked by >> the collapse issue fixed in the recent PRs. Is there any other analysis I >> can provide on the utilization spike/sleep behavior we're seeing, or any >> other debugging or code reading you recommend? As far as I can tell, >> there's nothing about our workload that would cause periodic behavior like >> this. The application is slinging RTP audio via udp to remote endpoints at >> a 20 msec ptime. Each function call for the NIF in question adds 10 msec of >> audio to the WebRTC buffer. >> >> As point of corroboration, this user on stackoverflow appears to be >> having the same or a similar issue: >> https://stackoverflow.com/questions/49563067/erlang-schedulers-just-sleep-why >> >> As always, the level of support from the Erlang community is second to >> none. Thanks to all for your time! >> >> Jesse >> >> >> >> On Wed, Jan 16, 2019 at 6:35 AM Rickard Green wrote: >> >>> On 2019-01-15 23:11, Jesse Stimpson wrote: >>> > Behavior of the schedulers appears to have the same issue with 2093 >>> patch. >>> > >>> > But I did notice something new in the msacc output. There is a very >>> > brief period, at approx the same time as the normal schedulers usage >>> > spikes, where all the dirty cpu schedulers have a significant sleep >>> > time. I've included timestamped excerpts below, starting with the >>> > increase in dirty cpu sleep, and ending with a "steady state" >>> utilization. >>> > >>> >>> We just released OTP-21.2.3 containing PR-2093. >>> >>> I don't think PR-2093 cause the spikes. This change does not affect how >>> work is moved between normal and dirty schedulers, only prevents the >>> "loss" of dirty schedulers. >>> >>> If a process is scheduled on a dirty scheduler it wont make progress >>> until it has executed on a dirty scheduler and vice versa (for normal >>> schedulers). This is the same both before and after PR-2093. Since dirty >>> schedulers aren't "lost" after PR-2093 progress of such processes will >>> happen earlier which of course change the behavior, but that is due to >>> the work load. >>> >>> Regards, >>> Rickard >>> >> >> >> -- >> >> >> >> Jesse Stimpson >> >> Site Reliability Engineering >> >> m: 9199950424 <(919)%20995-0424> >> RepublicWireless.com >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > -- Jesse Stimpson Site Reliability Engineering m: 9199950424 <(919)%20995-0424> RepublicWireless.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From t6sn7gt@REDACTED Wed Jan 23 19:51:44 2019 From: t6sn7gt@REDACTED (Donald Steven) Date: Wed, 23 Jan 2019 13:51:44 -0500 Subject: [erlang-questions] Updating elements of a record In-Reply-To: <20190123160758.GA88201@erix.ericsson.se> References: <08771336-4746-49c2-26f9-f2e48e14cae1@aim.com> <7a22bde8-08ca-4e99-a8d4-f4b1585c8b57@www.fastmail.com> <20190123160758.GA88201@erix.ericsson.se> Message-ID: Thanks everyone! On 1/23/2019 11.07 AM, Raimo Niskanen wrote: > On Tue, Jan 22, 2019 at 06:04:13PM -0500, Donald Steven wrote: >> I should have said: >> >> test(Rec, [{a, Val},{c, Val}]) >> >> On 1/22/2019 5.48 PM, Donald Steven wrote: >>> Thanks Roger.? I see how this works, but the issue is as much how to >>> get it to do: >>> >>> test([{a, Val},{c, Val}]), leaving, for example, field b untouched. > setelement/3 does exactly that; copies the other fields > >>> On 1/22/2019 3.57 PM, Roger Lipscombe wrote: >>>> On Tue, Jan 22, 2019, at 3:10 PM, Donald Steven wrote: >>>>> I would like to pass a list of some fields of a record to a function to >>>>> update those fields, leaving the other fields as they were. The list >>>>> will be different for each function call.? For example, one call might >>>>> want to change fields 1, 2 and 7, another call only field 8, another >>>>> call fields 3 and 4.? Of course, each field will have a unique name. >>>>> Each call to updateStatus (let's call it) would look like >>>>> updateStatus(Status, List_of_Fields_to_Update) -> code. >>>>> >>>> Records are just tuples. Use setelement with #rec.field: >>>> >>>> -module(rec). >>>> -export([new/0, update/3, test/0]). >>>> >>>> -record(rec, {a, b, c}). >>>> >>>> new() -> >>>> ???? #rec{}. >>>> >>>> update(R, Val, [F | Fields]) -> >>>> ???? R2 = setelement(F, R, Val), >>>> ???? update(R2, Val, Fields); >>>> update(R, _Val, []) -> >>>> ???? R. >>>> >>>> test() -> >>>> ???? R = rec:new(), >>>> ???? R2 = rec:update(R, 2, [#rec.a, #rec.c]), >>>> ???? R3 = rec:update(R2, 1, [#rec.b]), >>>> ???? R3. >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions From igor.clark@REDACTED Wed Jan 23 20:35:50 2019 From: igor.clark@REDACTED (Igor Clark) Date: Wed, 23 Jan 2019 19:35:50 +0000 Subject: [erlang-questions] Building C code in dep with rebar3: c_src not found Message-ID: <0AD45A52-6A3B-4376-96F4-E4A6B86C67D4@gmail.com> Hi everyone, I have an simple app which has a C port in it. I have a rebar.config from which it builds correctly when I run 'rebar3 compile' inside the app directory: > { erl_opts, [debug_info] }. > { deps, [] }. > > { plugins, [ > { pc, { git, "https://github.com/blt/port_compiler.git", { tag, "v1.10.1" } } } > ] }. > > { provider_hooks, [ > { pre, [ > { compile, { pc, compile }}, > { clean, { pc, clean }} > ] } > ] }. > > { port_specs, [ > { "priv/bin/my_port_binary", [ > "c_src/my_c_file.1.c", > "c_src/subdir/my_c_file_2.c" > ] } > ] }. > > { port_env, [ > { "CFLAGS", "$CFLAGS -Ic_src/ -g -Wall -Werror -O2" } > ] }. Compiling this works fine. The artifact is created in priv/bin, and running myapp:module_info() inside the shell shows the right details. However, when I try to include this app as a dependency, I get this error: > ===> Found no source files for: > "c_src/my_c_file.1.c" > ===> Found no source files for: > "c_src/subdir/my_c_file_2.c" I've included src, c_src and rebar.config in the 'files' entry in src/my_app.app.src. I've also included a jiffy dep as a test to make sure C code gets built correctly, and that works fine, but I can't see what jiffy's config has that I don't. Anyone know what I'm missing to make this work? Thanks very much! Igor From igor.clark@REDACTED Thu Jan 24 01:58:42 2019 From: igor.clark@REDACTED (Igor Clark) Date: Thu, 24 Jan 2019 00:58:42 +0000 Subject: [erlang-questions] Building C code in dep with rebar3: c_src not found In-Reply-To: <0AD45A52-6A3B-4376-96F4-E4A6B86C67D4@gmail.com> References: <0AD45A52-6A3B-4376-96F4-E4A6B86C67D4@gmail.com> Message-ID: Reading it back, I'm not sure if I made this very clear: - project A is an OTP app containing C port code in c_src, and the above rebar.config with port compiler info - project B is another OTP app referencing project A as a source dependency in project B's rebar config - when I build project A in its own dir with 'rebar3 compile', it compiles the C code as expected - when I build project B in its own dir with 'rebar3 compile', it recognises the port compiler dependency in project A's rebar.config, downloads and compiles it, but then fails to compile project A as a dep, saying "found no source files" So I need some way for the port compiler, when running in the context of the parent app A, to know the proper path for 'c_src' inside the source dependency B. Can't see how it could be any problem with the port compiler, not least because jiffy uses it fine, so I must be missing something! Incidentally I added CXXFLAGS to the port_env as well, but no difference, as expected. Clutching at straws somewhat there. ;-) I also tried using a Makefile, which I guess is the intended rebar3 way, and that seems to work - but I'd love to know what's happening here. Cheers, Igor On Wed, Jan 23, 2019 at 7:35 PM Igor Clark wrote: > Hi everyone, > > I have an simple app which has a C port in it. I have a rebar.config from > which it builds correctly when I run 'rebar3 compile' inside the app > directory: > > > { erl_opts, [debug_info] }. > > { deps, [] }. > > > > { plugins, [ > > { pc, { git, "https://github.com/blt/port_compiler.git", { tag, > "v1.10.1" } } } > > ] }. > > > > { provider_hooks, [ > > { pre, [ > > { compile, { pc, compile }}, > > { clean, { pc, clean }} > > ] } > > ] }. > > > > { port_specs, [ > > { "priv/bin/my_port_binary", [ > > "c_src/my_c_file.1.c", > > "c_src/subdir/my_c_file_2.c" > > ] } > > ] }. > > > > { port_env, [ > > { "CFLAGS", "$CFLAGS -Ic_src/ -g -Wall -Werror -O2" } > > ] }. > > > Compiling this works fine. The artifact is created in priv/bin, and > running myapp:module_info() inside the shell shows the right details. > > However, when I try to include this app as a dependency, I get this error: > > > ===> Found no source files for: > > "c_src/my_c_file.1.c" > > ===> Found no source files for: > > "c_src/subdir/my_c_file_2.c" > > > I've included src, c_src and rebar.config in the 'files' entry in > src/my_app.app.src. > > I've also included a jiffy dep as a test to make sure C code gets built > correctly, and that works fine, but I can't see what jiffy's config has > that I don't. > > Anyone know what I'm missing to make this work? > > Thanks very much! > Igor -------------- next part -------------- An HTML attachment was scrubbed... URL: From t6sn7gt@REDACTED Thu Jan 24 03:52:28 2019 From: t6sn7gt@REDACTED (Donald Steven) Date: Wed, 23 Jan 2019 21:52:28 -0500 Subject: [erlang-questions] Best choice for complex data structure Message-ID: <23fc155e-93ec-690f-eeec-58232dfdf517@aim.com> What, in your experience, is the best choice (for? set, get, update, etc.) for a complex data structure that's nested three or four levels deep? From gergo.nyiro@REDACTED Wed Jan 23 22:08:05 2019 From: gergo.nyiro@REDACTED (=?UTF-8?B?TnlpcsWRIEdlcmfFkQ==?=) Date: Wed, 23 Jan 2019 22:08:05 +0100 Subject: [erlang-questions] How to mimic Erlang nodes in PropEr statem test In-Reply-To: <20190123175241.GC1915@ferdmbp.local> References: <20190123175241.GC1915@ferdmbp.local> Message-ID: On Wed, Jan 23, 2019 at 6:52 PM Fred Hebert wrote: > > On 01/22, Nyir? Gerg? wrote: > >nice until I have a single process to communicate with, but I want to > >cut my program > >into multiple processes, but I don't know how should I register the > >processes to avoid > >pid passing and run multiple process group on the same VM. > > You already name all three processes according to if they were node > names. Can you be a bit more specific about what you intend to do? Right now I have only one gen_server (crdt_server) per node, but I would like to add an event handler (crdt_event) which could persist the content of the crdt set or log the executed commands. Since I just started to learn Erlang I have very limited ideas in design choices, that's why I'm looking for some advice. The following ideas came to my mind: crdt_event could be registered with a fix name then all the crdt_server processes in the proper statem test will send the events to the same event handler, so the event has to be extended with a reference to the sender (e.g: the name or pid of the crdt_server). It seems too complicated for my situation. Other solution would be to pass the NameOrPid of the crdt_event processes to crdt_server processes, but a process registry could be a more elegant solution. Should I use ct_property_test [1] instead? p.s: Thanks for the fast feedback. [1]: http://erlang.org/doc/man/ct_property_test.html From starbelly@REDACTED Thu Jan 24 06:18:08 2019 From: starbelly@REDACTED (Bryan Paxton) Date: Wed, 23 Jan 2019 23:18:08 -0600 Subject: [erlang-questions] Building C code in dep with rebar3: c_src not found In-Reply-To: References: <0AD45A52-6A3B-4376-96F4-E4A6B86C67D4@gmail.com> Message-ID: ?Hi, ?Is this a directory with a set of apps in it or is this an umbrella? (i.e., you have a top level rebar.config). AFAIK there's only two ways you can have local dependencies like this... in an umbrella or using checkout dependencies. Either way could you share the rebar config for project B? In some loose form that is. ?I just made an umbrella using rebar3 and could not replicate the issue. Note that in an umbrella I must be top level when running rebar3 compile, etc. -- Bryan On 1/23/19 6:58 PM, Igor Clark wrote: > Reading it back, I'm not sure if I made this very clear: > > - project A is an OTP app containing C port code in c_src, and the > above rebar.config with port compiler info > - project B is another OTP app referencing project A as a source > dependency in project B's rebar config > - when I build project A in its own dir with 'rebar3 compile', it > compiles the C code as expected > - when I build project B in its own dir with 'rebar3 compile', it > recognises the port compiler dependency in project A's rebar.config, > downloads and compiles it, but then fails to compile project A as a > dep, saying "found no source files" > > So I need some way for the port compiler, when running in the context > of the parent app A, to know the proper path for 'c_src' inside the > source dependency B. Can't see how it could be any problem with the > port compiler, not least because jiffy uses it fine, so I must be > missing something! > > Incidentally I added CXXFLAGS to the port_env as well, but no > difference, as expected. Clutching at straws somewhat there. ;-) > > I also tried using a Makefile, which I guess is the intended rebar3 > way, and that seems to work - but I'd love to know what's happening here. > > Cheers, > Igor > > > > On Wed, Jan 23, 2019 at 7:35 PM Igor Clark > wrote: > > Hi everyone, > > I have an simple app which has a C port in it. I have a > rebar.config from which it builds correctly when I run 'rebar3 > compile' inside the app directory: > > > { erl_opts, [debug_info] }. > > { deps, [] }. > > > > { plugins, [ > >? ? ? ?{ pc, { git, "https://github.com/blt/port_compiler.git", { > tag, "v1.10.1" } } } > > ] }. > > > > { provider_hooks, [ > >? ? ? ?{ pre, [ > >? ? ? ? ? ? ? ?{ compile, { pc, compile }}, > >? ? ? ? ? ? ? ?{ clean, { pc, clean }} > >? ? ? ?] } > > ] }. > > > > { port_specs, [ > >? ? ? ?{ "priv/bin/my_port_binary", [ > >? ? ? ? ? ? ? ?"c_src/my_c_file.1.c", > >? ? ? ? ? ? ? ?"c_src/subdir/my_c_file_2.c" > >? ? ? ?] } > > ] }. > > > > { port_env, [ > >? ? ? ?{ "CFLAGS", "$CFLAGS -Ic_src/ -g -Wall -Werror -O2" } > > ] }. > > > Compiling this works fine. The artifact is created in priv/bin, > and running myapp:module_info() inside the shell shows the right > details. > > However, when I try to include this app as a dependency, I get > this error: > > > ===> Found no source files for: > > "c_src/my_c_file.1.c" > > ===> Found no source files for: > > "c_src/subdir/my_c_file_2.c" > > > I've included src, c_src and rebar.config in the 'files' entry in > src/my_app.app.src. > > I've also included a jiffy dep as a test to make sure C code gets > built correctly, and that works fine, but I can't see what jiffy's > config has that I don't. > > Anyone know what I'm missing to make this work? > > Thanks very much! > Igor > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From frank.muller.erl@REDACTED Thu Jan 24 07:16:24 2019 From: frank.muller.erl@REDACTED (Frank Muller) Date: Thu, 24 Jan 2019 07:16:24 +0100 Subject: [erlang-questions] Cross-compilation to ARM Message-ID: Hi guys I have simple Linux Erlang release which I want to cross-compile for Raspberry Pi 2 (Ubuntu). 1. is there any simple way to achieve that? Links, examples ... are very welcome 2. what should I pay attention to during this process? Feedback, experience... Thank you. /Frank -------------- next part -------------- An HTML attachment was scrubbed... URL: From fchschneider@REDACTED Thu Jan 24 08:00:18 2019 From: fchschneider@REDACTED (Schneider) Date: Thu, 24 Jan 2019 08:00:18 +0100 Subject: [erlang-questions] Cross-compilation to ARM In-Reply-To: References: Message-ID: Why not just download from https://www.erlang-solutions.com/resources/download.html > Op 24 jan. 2019 om 07:16 heeft Frank Muller het volgende geschreven: > > Hi guys > > I have simple Linux Erlang release which I want to cross-compile for Raspberry Pi 2 (Ubuntu). > > 1. is there any simple way to achieve that? > Links, examples ... are very welcome > > 2. what should I pay attention to during this process? Feedback, experience... > > Thank you. > /Frank > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmkolesnikov@REDACTED Thu Jan 24 10:02:42 2019 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Thu, 24 Jan 2019 11:02:42 +0200 Subject: [erlang-questions] Cross-compilation to ARM In-Reply-To: References: Message-ID: Hello, I would used xcomp and ready made schemes for it. Please take a look into following README. https://github.com/erlang/otp/blob/master/HOWTO/INSTALL-ANDROID.md https://github.com/erlang/otp/blob/master/HOWTO/INSTALL-RASPBERRYPI3.md Best Regards, Dmitry > On 24 Jan 2019, at 9.00, Schneider wrote: > > Why not just download from https://www.erlang-solutions.com/resources/download.html > > > > Op 24 jan. 2019 om 07:16 heeft Frank Muller > het volgende geschreven: > >> Hi guys >> >> I have simple Linux Erlang release which I want to cross-compile for Raspberry Pi 2 (Ubuntu). >> >> 1. is there any simple way to achieve that? >> Links, examples ... are very welcome >> >> 2. what should I pay attention to during this process? Feedback, experience... >> >> Thank you. >> /Frank >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From zxq9@REDACTED Thu Jan 24 11:19:41 2019 From: zxq9@REDACTED (zxq9@REDACTED) Date: Thu, 24 Jan 2019 19:19:41 +0900 Subject: [erlang-questions] Best choice for complex data structure In-Reply-To: <23fc155e-93ec-690f-eeec-58232dfdf517@aim.com> References: <23fc155e-93ec-690f-eeec-58232dfdf517@aim.com> Message-ID: <2166343.CDc0fOby7I@takoyaki> On 2019?1?23???? 21?52?28? JST Donald Steven wrote: > What, in your experience, is the best choice (for set, get, update, > etc.) for a complex data structure that's nested three or four levels deep? Hi, Donald. It depends entirely on: - What kind of data it is - What the program is doing with the data Could you provide a little more context? Explain your problem. Describe the context. Then people can give good answers. -Craig From dmkolesnikov@REDACTED Thu Jan 24 11:43:52 2019 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Thu, 24 Jan 2019 12:43:52 +0200 Subject: [erlang-questions] Best choice for complex data structure In-Reply-To: <2166343.CDc0fOby7I@takoyaki> References: <23fc155e-93ec-690f-eeec-58232dfdf517@aim.com> <2166343.CDc0fOby7I@takoyaki> Message-ID: <82A986E6-A08D-4BAA-AA18-0BA1E3BA99AB@gmail.com> I found that lenses gives reasonable good abstraction to deal with deeply nested structures. Best Regards, Dmitry > On 24 Jan 2019, at 12.19, zxq9@REDACTED wrote: > > On 2019?1?23???? 21?52?28? JST Donald Steven wrote: >> What, in your experience, is the best choice (for set, get, update, >> etc.) for a complex data structure that's nested three or four levels deep? > > Hi, Donald. > > It depends entirely on: > - What kind of data it is > - What the program is doing with the data > > Could you provide a little more context? > Explain your problem. Describe the context. Then people can give good answers. > > -Craig > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From zxq9@REDACTED Thu Jan 24 12:11:55 2019 From: zxq9@REDACTED (zxq9@REDACTED) Date: Thu, 24 Jan 2019 20:11:55 +0900 Subject: [erlang-questions] Best choice for complex data structure In-Reply-To: <82A986E6-A08D-4BAA-AA18-0BA1E3BA99AB@gmail.com> References: <23fc155e-93ec-690f-eeec-58232dfdf517@aim.com> <2166343.CDc0fOby7I@takoyaki> <82A986E6-A08D-4BAA-AA18-0BA1E3BA99AB@gmail.com> Message-ID: <1953348.hysP6hV7Y5@takoyaki> On 2019?1?24???? 12?43?52? JST you wrote: > I found that lenses gives reasonable good abstraction to deal with deeply nested structures. That said, I find that very often things like lenses are "chindougu": Unuseless inventions. They solve problems that would never have existed had more thought been given to the actual shape of data within the program in question. That's not always true, but quite often it is. Come to think of it, much of the most heavily hyped efforts in software are chindougu. -Craig From t6sn7gt@REDACTED Thu Jan 24 12:37:03 2019 From: t6sn7gt@REDACTED (Donald Steven) Date: Thu, 24 Jan 2019 06:37:03 -0500 Subject: [erlang-questions] Best choice for complex data structure In-Reply-To: <1953348.hysP6hV7Y5@takoyaki> References: <23fc155e-93ec-690f-eeec-58232dfdf517@aim.com> <2166343.CDc0fOby7I@takoyaki> <82A986E6-A08D-4BAA-AA18-0BA1E3BA99AB@gmail.com> <1953348.hysP6hV7Y5@takoyaki> Message-ID: I'm trying to figure out how to shape the (musical) data.? In everyday language, a musical idea has an architecture of formal nested units and gestures made up of one or more events which have sounds and silences.? In Rust, the latter part of the above would yield a statement like: scoreIdea.gesture[iGesture-1].event[iEvent-1].sounds +=1 which is manageable.? But when I think about accessing data nested many levels deep in Erlang, I'm kinda flummoxed. Instinctively, I don't think transliterating something like this to records (or even maps?) is the way to go? All suggestions greatly appreciated. On 1/24/2019 6:11 AM, zxq9@REDACTED wrote: > On 2019?1?24???? 12?43?52? JST you wrote: >> I found that lenses gives reasonable good abstraction to deal with deeply nested structures. > That said, I find that very often things like lenses are "chindougu": Unuseless inventions. > They solve problems that would never have existed had more thought been given to the actual shape of data within the program in question. > > That's not always true, but quite often it is. > > Come to think of it, much of the most heavily hyped efforts in software are chindougu. > > -Craig > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From igor.clark@REDACTED Thu Jan 24 12:42:59 2019 From: igor.clark@REDACTED (Igor Clark) Date: Thu, 24 Jan 2019 11:42:59 +0000 Subject: [erlang-questions] Building C code in dep with rebar3: c_src not found In-Reply-To: References: <0AD45A52-6A3B-4376-96F4-E4A6B86C67D4@gmail.com> Message-ID: Hi Bryan! Thanks for getting back to me. Turns out this only happens with the latest version 1.10.1 of the port compiler - I tried older versions and everything works out as expected. Seems I'm not the only one who's seen this either. I'm going to pin pc to 1.9.1 in my project A, as everything works as I thought it should with that version. Just to tidy up, to answer your question, yes it's an umbrella app - the 'parent' app B is one that I made with "rebar3 new release", and then added project A as a remote source dependency, i.e. from a git repo, into 'deps' in project B's rebar.config. I'm running rebar3 commands from top level in project B. Appreciate your time! best, i > On 24 Jan 2019, at 05:18, Bryan Paxton wrote: > > Hi, > > Is this a directory with a set of apps in it or is this an umbrella? (i.e., you have a top level rebar.config). AFAIK there's only two ways you can have local dependencies like this... in an umbrella or using checkout dependencies. > Either way could you share the rebar config for project B? In some loose form that is. > I just made an umbrella using rebar3 and could not replicate the issue. Note that in an umbrella I must be top level when running rebar3 compile, etc. > -- > > Bryan > > On 1/23/19 6:58 PM, Igor Clark wrote: >> Reading it back, I'm not sure if I made this very clear: >> >> - project A is an OTP app containing C port code in c_src, and the above rebar.config with port compiler info >> - project B is another OTP app referencing project A as a source dependency in project B's rebar config >> - when I build project A in its own dir with 'rebar3 compile', it compiles the C code as expected >> - when I build project B in its own dir with 'rebar3 compile', it recognises the port compiler dependency in project A's rebar.config, downloads and compiles it, but then fails to compile project A as a dep, saying "found no source files" >> >> So I need some way for the port compiler, when running in the context of the parent app A, to know the proper path for 'c_src' inside the source dependency B. Can't see how it could be any problem with the port compiler, not least because jiffy uses it fine, so I must be missing something! >> >> Incidentally I added CXXFLAGS to the port_env as well, but no difference, as expected. Clutching at straws somewhat there. ;-) >> >> I also tried using a Makefile, which I guess is the intended rebar3 way, and that seems to work - but I'd love to know what's happening here. >> >> Cheers, >> Igor >> >> >> >> On Wed, Jan 23, 2019 at 7:35 PM Igor Clark > wrote: >> Hi everyone, >> >> I have an simple app which has a C port in it. I have a rebar.config from which it builds correctly when I run 'rebar3 compile' inside the app directory: >> >> > { erl_opts, [debug_info] }. >> > { deps, [] }. >> > >> > { plugins, [ >> > { pc, { git, "https://github.com/blt/port_compiler.git ", { tag, "v1.10.1" } } } >> > ] }. >> > >> > { provider_hooks, [ >> > { pre, [ >> > { compile, { pc, compile }}, >> > { clean, { pc, clean }} >> > ] } >> > ] }. >> > >> > { port_specs, [ >> > { "priv/bin/my_port_binary", [ >> > "c_src/my_c_file.1.c", >> > "c_src/subdir/my_c_file_2.c" >> > ] } >> > ] }. >> > >> > { port_env, [ >> > { "CFLAGS", "$CFLAGS -Ic_src/ -g -Wall -Werror -O2" } >> > ] }. >> >> >> Compiling this works fine. The artifact is created in priv/bin, and running myapp:module_info() inside the shell shows the right details. >> >> However, when I try to include this app as a dependency, I get this error: >> >> > ===> Found no source files for: >> > "c_src/my_c_file.1.c" >> > ===> Found no source files for: >> > "c_src/subdir/my_c_file_2.c" >> >> >> I've included src, c_src and rebar.config in the 'files' entry in src/my_app.app.src. >> >> I've also included a jiffy dep as a test to make sure C code gets built correctly, and that works fine, but I can't see what jiffy's config has that I don't. >> >> Anyone know what I'm missing to make this work? >> >> Thanks very much! >> Igor >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From zxq9@REDACTED Thu Jan 24 13:50:40 2019 From: zxq9@REDACTED (zxq9@REDACTED) Date: Thu, 24 Jan 2019 21:50:40 +0900 Subject: [erlang-questions] Best choice for complex data structure In-Reply-To: References: <23fc155e-93ec-690f-eeec-58232dfdf517@aim.com> <1953348.hysP6hV7Y5@takoyaki> Message-ID: <31599511.N10xD8Cjzr@takoyaki> On 2019?1?24???? 6?37?03? JST you wrote: > I'm trying to figure out how to shape the (musical) data. In everyday > language, a musical idea has an architecture of formal nested units and > gestures made up of one or more events which have sounds and silences. > In Rust, the latter part of the above would yield a statement like: > > scoreIdea.gesture[iGesture-1].event[iEvent-1].sounds +=1 > > which is manageable. But when I think about accessing data nested many > levels deep in Erlang, I'm kinda flummoxed. Instinctively, I don't think > transliterating something like this to records (or even maps?) is the > way to go? > > All suggestions greatly appreciated. This kind of context is extremely useful to know. We have some canonical examples of musical data and quite a lot of it is not nested, but rather heavily attributed (includes a lot of meta, but the data itself is flat). But that is how notations work for musical players (MIDI being a common example). If you want to capture musical architecture, though, that is something a bit different... I don't think we've done very well coming up with notations for musical structure in writing, much less computer data. Overlays and segments for mixes is the best I can think of -- where you store a MIDI sequence along with some meta that identifies it by a useful set of categories (sort of like "tags" on blog posts) and whatever other sort of data might be useful, and then pick from a library of MIDI overlays or segments to insert into an ongoing composition ("flourish X overlays the melody here, segment Y provides a bridge there", etc.). For maintaining a library of selectable inserts and references you could use simple lists of tuples as the canonical store, and then build reference indexes on them as maps. CanonicalData :: [{segment, Title :: string(), Authors :: [string()], Date :: date(), Tags :: [string()], MIDI_Data :: binary()}] TagIndex :: #{Tag := [Title]} AuthorIndex :: #{Author := [Title]} etc. Keeping in mind, of course, that some data element (or combination) will have to constitute a unique key for the data. I suppose the key question here would be whether you expect this data to be useful for rendering the music in a player, displaying visually in a composition explorer (which could possibly export combined snippets in a flatter, playable format), or interfacing with an existing music tool (Casio/Yamaha/Roland/etc. MIDI interfaces, sheet music PDF renderers, etc.). Any idea on that? Scores can themselves be compositions of these elements, actually: Score :: [{InsertTime, Title, SegmentStart, SegmentStop, MIDI_Data, RepeatMeta}] So that you can move the pieces around in chunks and use the InsertTime as the offset from 0 (in partial beats, milliseconds, whatever meter you want to use) to determine when it first occurs in the piece, the Title to reference which MIDI data should be pulled from the library to include, the SegmentStart as the start of the segment within the MIDI you want to use, and the SegmentStop as the final part of the MIDI you want to use, and RepeatMeta as an optional value (usually something like RepeatMeta :: none | {forever | Count, BreakDelay | BeatIndex}) that can indicate whether a particular segment repeats, how much the delay between occurrences should be (or what beat index within the next opening measure after prior completion it should recur, etc.). Blah blah... Anyway, this all depends on what you want the tool to do. Musical scoring in software can get pretty complicated (or sometimes be really simple) depending on what you hope to get out of it. Hopefully the wandering blather above gives you some ideas. -Craig From zxq9@REDACTED Thu Jan 24 19:00:48 2019 From: zxq9@REDACTED (zxq9@REDACTED) Date: Fri, 25 Jan 2019 03:00:48 +0900 Subject: [erlang-questions] Best choice for complex data structure In-Reply-To: References: <23fc155e-93ec-690f-eeec-58232dfdf517@aim.com> <31599511.N10xD8Cjzr@takoyaki> Message-ID: <1779429.TEnnqv59f6@takoyaki> On 2019?1?24???? 11?08?09? JST you wrote: > Hi Craig, > > Thanks so much for sharing this. You're very generous. > > I have a fully functional MIDI module with a tested and effective set of > low level IO. I'm working now at the meta level (as a composer), > looking to translate some impressionistic higher level ideas through > intermediary functions to MIDI output. > > I'm also just learning Erlang, so I'm sure some of my questions are very > naive. > > What is: > > Score :: [{InsertTime, Title, SegmentStart, SegmentStop, MIDI_Data, > RepeatMeta}] > > Is Score a variable -- or a type that is a list of tuples? Nice! Welcome to Erlang! Once your eyes adjust I think you'll thoroughly enjoy it around here. :-D And yes, you are spot on, `Score` above is a type specification. You'll find this helpful: http://erlang.org/doc/reference_manual/typespec.html It is the required notation when talking to Dialyzer (obviously) but also useful when communicating amongst Erlangers. I gave quite a few type specification examples in the original message, so that link will help you decipher them. Everything capitalized is a possible type (a type variable that I left undefined -- use your imagination about what type of values might be useful in those places) and everything that is lower-case is an actual definite type. Parens after a lower-case string are a defined type, an atom with no parens is just the atom. A pipe (like `|`) is an "or". To get more specific... Score :: [{InsertTime, Title, SegmentStart, SegmentStop, MIDI_Data, RepeatMeta}] where InsertTime :: time_unit(), Title :: string(), SegmentStart :: time_unit(), SegmentStop :: time_unit(), MIDI_Data :: binary(), RepeatMeta :: none | repeat_meta(). -type time_unit() :: non_neg_integer() % an integer value in milliseconds representing absolute time | {time_signature(), Beats :: non_neg_integer()}. The above isn't 100% proper type notation (the way Score is defined looks like a -spec, not a -type, but it is a useful notation amongst erlangers, anyway), and I never defined what `repeat_meta()` should be, but should convey some general idea what I intended. The point in expressing things this way is that a list of MIDI segments (which already carry their own musical data, relative to themselves) can be included as data into a higher-level program with the addition of metadata that is meaningful to that higher-level program, and the resulting list of MIDI segment points (some composed locally, some referenced from a library -- the origin not being important) can be synthesized by a program to produce a new MIDI output readable by a musical notation output program that reads MIDI, or by a player. That may or may not be what you're looking for (maybe you want to write a musical synthesis program that focuses specifically on musical architecture, in which case your real focus is the segment meta and arrangements among the segments, not the actual MIDI data), but hopefully this gives you some ideas for thinking of data as more than just nested data or OOP-style "objects". -Craig From jstimpson@REDACTED Thu Jan 24 22:21:12 2019 From: jstimpson@REDACTED (Jesse Stimpson) Date: Thu, 24 Jan 2019 16:21:12 -0500 Subject: [erlang-questions] Dirty CPU schedulers stuck at zero utilization In-Reply-To: References: <7f7287b8-f6d9-da4b-59ed-52dc2d9c76f7@erlang.org> Message-ID: After more testing with 21.2.3, it appears that the behavior we are seeing with respect to dirty schedulers going to sleep occurs only when the dirty scheduler runs on a cpu hyperthread. On our test instance, I disabled hyperthreading and used +SP 50:50, and the problem has gone away. This allows our NIF workload and non-NIF workload an even attribution of CPU resources, which works well for our purposes. It's unclear to us why hyperthreading causes this behavior, but we do wonder if it may have something to do with AWS virtualization. In any case, thanks to everyone for the assistance! Best, Jesse On Wed, Jan 23, 2019 at 1:14 PM Jesse Stimpson < jstimpson@REDACTED> wrote: > To clarify about our workload, the NIF execution itself is around 1 msec, > the data on which is operates represents 10 msec of audio. Apologies if my > last message was unclear. > > Out of convenience, we're using the open source WebRTC project to take > advantage of their built in PLC, FEC, Opus, etc. The project is written in > C++, so we have integrated with it via NIF. Unfortunately, re-writing using > a yielding NIF, or re-writing to Erlang is not as straightforward as we > would like. Although I admit it would alleviate our scheduling issues. > > We'll continue our testing with 21.2.3 and report back if there appear to > be any other leads. > > Thanks, > Jesse > > On Tue, Jan 22, 2019 at 2:56 PM Max Lapshin wrote: > >> why do you do it via nif? >> >> On Wed, Jan 16, 2019 at 6:00 PM Jesse Stimpson < >> jstimpson@REDACTED> wrote: >> >>> It's possible that during our tests the utilization spike was masked by >>> the collapse issue fixed in the recent PRs. Is there any other analysis I >>> can provide on the utilization spike/sleep behavior we're seeing, or any >>> other debugging or code reading you recommend? As far as I can tell, >>> there's nothing about our workload that would cause periodic behavior like >>> this. The application is slinging RTP audio via udp to remote endpoints at >>> a 20 msec ptime. Each function call for the NIF in question adds 10 msec of >>> audio to the WebRTC buffer. >>> >>> As point of corroboration, this user on stackoverflow appears to be >>> having the same or a similar issue: >>> https://stackoverflow.com/questions/49563067/erlang-schedulers-just-sleep-why >>> >>> As always, the level of support from the Erlang community is second to >>> none. Thanks to all for your time! >>> >>> Jesse >>> >>> >>> >>> On Wed, Jan 16, 2019 at 6:35 AM Rickard Green >>> wrote: >>> >>>> On 2019-01-15 23:11, Jesse Stimpson wrote: >>>> > Behavior of the schedulers appears to have the same issue with 2093 >>>> patch. >>>> > >>>> > But I did notice something new in the msacc output. There is a very >>>> > brief period, at approx the same time as the normal schedulers usage >>>> > spikes, where all the dirty cpu schedulers have a significant sleep >>>> > time. I've included timestamped excerpts below, starting with the >>>> > increase in dirty cpu sleep, and ending with a "steady state" >>>> utilization. >>>> > >>>> >>>> We just released OTP-21.2.3 containing PR-2093. >>>> >>>> I don't think PR-2093 cause the spikes. This change does not affect how >>>> work is moved between normal and dirty schedulers, only prevents the >>>> "loss" of dirty schedulers. >>>> >>>> If a process is scheduled on a dirty scheduler it wont make progress >>>> until it has executed on a dirty scheduler and vice versa (for normal >>>> schedulers). This is the same both before and after PR-2093. Since >>>> dirty >>>> schedulers aren't "lost" after PR-2093 progress of such processes will >>>> happen earlier which of course change the behavior, but that is due to >>>> the work load. >>>> >>>> Regards, >>>> Rickard >>>> >>> >>> >>> -- >>> >>> >>> >>> Jesse Stimpson >>> >>> Site Reliability Engineering >>> >>> m: 9199950424 <(919)%20995-0424> >>> RepublicWireless.com >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >> > > -- > > > > Jesse Stimpson > > Site Reliability Engineering > > m: 9199950424 <(919)%20995-0424> > RepublicWireless.com > -- Jesse Stimpson Site Reliability Engineering m: 9199950424 <(919)%20995-0424> RepublicWireless.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From frank.muller.erl@REDACTED Fri Jan 25 03:49:31 2019 From: frank.muller.erl@REDACTED (Frank Muller) Date: Fri, 25 Jan 2019 03:49:31 +0100 Subject: [erlang-questions] Cross-compilation to ARM In-Reply-To: References: Message-ID: Exactly what I was looking for. Thanks Dmitry!!! Best /Frank Hello, > > I would used xcomp and ready made schemes for it. > Please take a look into following README. > > https://github.com/erlang/otp/blob/master/HOWTO/INSTALL-ANDROID.md > https://github.com/erlang/otp/blob/master/HOWTO/INSTALL-RASPBERRYPI3.md > > Best Regards, > Dmitry > > > On 24 Jan 2019, at 9.00, Schneider wrote: > > Why not just download from > https://www.erlang-solutions.com/resources/download.html > > > > Op 24 jan. 2019 om 07:16 heeft Frank Muller > het volgende geschreven: > > Hi guys > > I have simple Linux Erlang release which I want to cross-compile for > Raspberry Pi 2 (Ubuntu). > > 1. is there any simple way to achieve that? > Links, examples ... are very welcome > > 2. what should I pay attention to during this process? Feedback, > experience... > > Thank you. > /Frank > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mononcqc@REDACTED Fri Jan 25 14:03:06 2019 From: mononcqc@REDACTED (Fred Hebert) Date: Fri, 25 Jan 2019 08:03:06 -0500 Subject: [erlang-questions] How to mimic Erlang nodes in PropEr statem test In-Reply-To: References: <20190123175241.GC1915@ferdmbp.local> Message-ID: <20190125130305.GD1915@ferdmbp.local> On 01/23, Nyir? Gerg? wrote: > >Right now I have only one gen_server (crdt_server) per node, but I would >like to add an event handler (crdt_event) which could persist the content >of the crdt set or log the executed commands. > Okay, so that's essentially duplicating the messages: send them to the server, but you want to asynchronously log with another process. Do I understand this right? >crdt_event could be registered with a fix name then all the >crdt_server processes >in the proper statem test will send the events to the same event handler, so >the event has to be extended with a reference to the sender (e.g: the name >or pid of the crdt_server). It seems too complicated for my situation. Is the event handler only required for the tests? If so, that can make sense. If you're working with the same design for your production system, it would generally be a better idea to have one process that handles the logging per CRDT server, assuming what you end up with is one file per server, in order. Using the same handler for multiple servers is a possibility, but you have a lot more risk and complexity?as you noted, you'd need to start identifying each message, rather than just giving the event manager a destination where to output traffic?while coupling the runtime of unrelated files. > >Other solution would be to pass the NameOrPid of the crdt_event processes to >crdt_server processes, but a process registry could be a more elegant solution. > This is a possibility. There are other options such as: - Each crdt_server is spawned under a crdt_sup. The crdt_sup works by booting two processes: crdt_log (first) and crdt_server (second). Each is passed a name that identifies the server ("crdt_1", `node_a`, or whatever) - By using a dynamic process registry like gproc (see https://hex.pm/packages/gproc) and replacing the name registration from {local, Name} to {via, gproc, {n, l, {crdt_log, Name}}} and {via, gproc, {n, l, {crdt_server, Name}}} you can give them unique names that they can "know" - On every call the crdt_server wants to log, it can call a function from the crdt_log module, such as crdt_log:do_log(Name, Contents) If you implement the latter function as: -module(crdt_log). ... start_link(Name) -> %% {n, l, Name} stands for "{name, local, ActualName}" but gproc %% uses shorthand gen_server:start_link({via, gproc, {n, l, {crdt_log, Name}}}, ?MODULE, [], []). ... do_log(Name, Event) -> gen_server:cast({via, gproc, {n, l, {crdt_log, Name}}}, Event). ... Then you'll be able to call the log process from within the other one and send asynchronous events this way. You can then consider both processes as one unit that fits under one supervisor. For your test, you now want to boot 3 of these supervisors instead of only 3 servers, and you should get a rather transparent layer added otherwise. >Should I use ct_property_test [1] instead? > This is an experimental call or module to integrate the results of your properties to the common test flow. You're free to use it if you want to integrate everything into CT. I personally maintain the rebar3 plugin for PropEr and I tend to prefer it for some of the additional features it adds: - meta functions https://github.com/ferd/rebar3_proper#per-properties-meta-functions (which I might need to contribute upstream to PropEr instead) - The ability to run a `rebar3 proper --retry` to get the last failing case run again - The `rebar3 proper --store` call, which can be called after a failure to store the failing cases in a file that can be used to test regressions - The `rebar3 proper --regressions` switch, which replays the previously stored couterexamples. You can, of course, get all of this by doing more manual work in CT or by transcribing counterexamples by hand into regression tests. I personally just use the proper and CT commands distinctly, and instead add a rebar3 alias for both: {alias, [{check, [ct, proper]}]}. Which then lets me call 'rebar3 check' and runs both the CT and the PropEr test suites. Regards, Fred. From otp@REDACTED Fri Jan 25 14:10:27 2019 From: otp@REDACTED (Erlang/OTP) Date: Fri, 25 Jan 2019 14:10:27 +0100 Subject: [erlang-questions] Patch Package OTP 21.2.4 Released Message-ID: <20190125131027.GA26732@erix.ericsson.se> Patch Package: OTP 21.2.4 Git Tag: OTP-21.2.4 Date: 2019-01-25 Trouble Report Id: OTP-15536, OTP-15537, OTP-15538, OTP-15544 Seq num: ERIERL-229, ERIERL-289, ERL-827 System: OTP Release: 21 Application: erts-10.2.3, inets-7.0.4 Predecessor: OTP 21.2.3 Check out the git tag OTP-21.2.4, and build a full OTP system including documentation. Apply one or more applications from this build as patches to your installation using the 'otp_patch_apply' tool. For information on install requirements, see descriptions for each application version below. --------------------------------------------------------------------- --- erts-10.2.3 ----------------------------------------------------- --------------------------------------------------------------------- Note! The erts-10.2.3 application can *not* be applied independently of other applications on an arbitrary OTP 21 installation. On a full OTP 21 installation, also the following runtime dependencies have to be satisfied: -- kernel-6.1 (first satisfied in OTP 21.1) -- sasl-3.3 (first satisfied in OTP 21.2) --- Fixed Bugs and Malfunctions --- OTP-15536 Application(s): erts Related Id(s): ERL-827 Fix bug where doing a gen_tcp:send on a socket with delay_send set to true could cause a segfault if the other side closes the connection. Bug was introduced in erts-10.2 (OTP-21.2). OTP-15537 Application(s): erts Fix a race condition when a port program closes that could result in the next started port to hang during startup. When this fault happens the following error is normally (but not always) logged: =ERROR REPORT==== 14-Jan-2019::10:45:52.868246 === Bad input fd in erts_poll()! fd=11, port=#Port<0.505>, driver=spawn, name=/bin/sh -s unix:cmd Bug was introduced in erts-10.0 (OTP-21.0). OTP-15538 Application(s): erts Related Id(s): ERIERL-229 Fix a bug where polling for external events could be delayed for a very long time if all active schedulers were 100% loaded. Bug was introduced in erts-10.2 (OTP-21.2). Full runtime dependencies of erts-10.2.3: kernel-6.1, sasl-3.3, stdlib-3.5 --------------------------------------------------------------------- --- inets-7.0.4 ----------------------------------------------------- --------------------------------------------------------------------- The inets-7.0.4 application can be applied independently of other applications on a full OTP 21 installation. --- Fixed Bugs and Malfunctions --- OTP-15544 Application(s): inets Related Id(s): ERIERL-289 Make sure ipv6 addresses with brackets in URIs are converted correctly before passing to lower level functions like gen_tcp and ssl functions. Could cause connection to fail. Full runtime dependencies of inets-7.0.4: erts-6.0, kernel-3.0, mnesia-4.12, runtime_tools-1.8.14, ssl-5.3.4, stdlib-3.5 --------------------------------------------------------------------- --------------------------------------------------------------------- --------------------------------------------------------------------- From aronisstav@REDACTED Fri Jan 25 14:46:17 2019 From: aronisstav@REDACTED (Stavros Aronis) Date: Fri, 25 Jan 2019 14:46:17 +0100 Subject: [erlang-questions] How to mimic Erlang nodes in PropEr statem test In-Reply-To: <20190125130305.GD1915@ferdmbp.local> References: <20190123175241.GC1915@ferdmbp.local> <20190125130305.GD1915@ferdmbp.local> Message-ID: Hi Gerg? (and list), before summer I was involved in writing a library for simulating distributed Erlang systems on a single node: https://github.com/klarna/vnet Perhaps you may find some ideas from there useful. You can read a paper about it (published in Erlang '18) by following the "Modelling Distributed Erlang within a Single Node" link in https://concuerror.com/publications (indirect link given to avoid ACMs access restrictions). An (admittedly poorly documented) example of its use (also discussed in the paper) is available here: https://github.com/klarna/vnet/tree/master/test/counter_server_example Regards, Stavros On Fri, Jan 25, 2019 at 2:03 PM Fred Hebert wrote: > > On 01/23, Nyir? Gerg? wrote: > > > >Right now I have only one gen_server (crdt_server) per node, but I would > >like to add an event handler (crdt_event) which could persist the content > >of the crdt set or log the executed commands. > > > > Okay, so that's essentially duplicating the messages: send them to the > server, but you want to asynchronously log with another process. Do I > understand this right? > > >crdt_event could be registered with a fix name then all the > >crdt_server processes > >in the proper statem test will send the events to the same event handler, so > >the event has to be extended with a reference to the sender (e.g: the name > >or pid of the crdt_server). It seems too complicated for my situation. > > Is the event handler only required for the tests? If so, that can make > sense. If you're working with the same design for your production > system, it would generally be a better idea to have one process that > handles the logging per CRDT server, assuming what you end up with is > one file per server, in order. > > Using the same handler for multiple servers is a possibility, but you > have a lot more risk and complexity?as you noted, you'd need to start > identifying each message, rather than just giving the event manager a > destination where to output traffic?while coupling the runtime of > unrelated files. > > > > >Other solution would be to pass the NameOrPid of the crdt_event processes to > >crdt_server processes, but a process registry could be a more elegant solution. > > > > This is a possibility. There are other options such as: > > - Each crdt_server is spawned under a crdt_sup. The crdt_sup works by > booting two processes: crdt_log (first) and crdt_server (second). Each > is passed a name that identifies the server ("crdt_1", `node_a`, or > whatever) > - By using a dynamic process registry like gproc (see > https://hex.pm/packages/gproc) and replacing the name registration > from {local, Name} to {via, gproc, {n, l, {crdt_log, Name}}} and {via, > gproc, {n, l, {crdt_server, Name}}} you can give them unique names > that they can "know" > - On every call the crdt_server wants to log, it can call a function > from the crdt_log module, such as crdt_log:do_log(Name, Contents) > > If you implement the latter function as: > > -module(crdt_log). > > ... > > start_link(Name) -> > %% {n, l, Name} stands for "{name, local, ActualName}" but gproc > %% uses shorthand > gen_server:start_link({via, gproc, {n, l, {crdt_log, Name}}}, > ?MODULE, [], []). > > ... > > do_log(Name, Event) -> > gen_server:cast({via, gproc, {n, l, {crdt_log, Name}}}, Event). > > ... > > Then you'll be able to call the log process from within the other one > and send asynchronous events this way. You can then consider both > processes as one unit that fits under one supervisor. For your test, you > now want to boot 3 of these supervisors instead of only 3 servers, and > you should get a rather transparent layer added otherwise. > > > >Should I use ct_property_test [1] instead? > > > > This is an experimental call or module to integrate the results of your > properties to the common test flow. You're free to use it if you want to > integrate everything into CT. > > I personally maintain the rebar3 plugin for PropEr and I tend to prefer > it for some of the additional features it adds: > > - meta functions > https://github.com/ferd/rebar3_proper#per-properties-meta-functions > (which I might need to contribute upstream to PropEr instead) > - The ability to run a `rebar3 proper --retry` to get the last failing > case run again > - The `rebar3 proper --store` call, which can be called after a failure > to store the failing cases in a file that can be used to test > regressions > - The `rebar3 proper --regressions` switch, which replays the previously > stored couterexamples. > > You can, of course, get all of this by doing more manual work in CT or > by transcribing counterexamples by hand into regression tests. I > personally just use the proper and CT commands distinctly, and instead > add a rebar3 alias for both: > > {alias, [{check, [ct, proper]}]}. > > Which then lets me call 'rebar3 check' and runs both the CT and the > PropEr test suites. > > Regards, > Fred. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From frank.muller.erl@REDACTED Fri Jan 25 16:49:43 2019 From: frank.muller.erl@REDACTED (Frank Muller) Date: Fri, 25 Jan 2019 16:49:43 +0100 Subject: [erlang-questions] Sending messages to non-existing process Message-ID: Hey guys Is there any side effect of sending an infinite stream of messages to a non-existing process registered by name? Ex. 1> Term = ?. 2> register(foo, self()). 3> 1 / 0. 4> [ foo ! Term || _ <- lists:eq(1. 10000000000000) ]. I?m curious about how the VM handles this scenario internally: 1. messages are simply discarded? And if this the case, will this hurt the VM?s performances? 2. something else??? Finally, what will happen if during the sending, a process get registered with name 'foo' ? Thanks in advance /Frank -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmytro.lytovchenko@REDACTED Fri Jan 25 16:52:42 2019 From: dmytro.lytovchenko@REDACTED (Dmytro Lytovchenko) Date: Fri, 25 Jan 2019 15:52:42 +0000 Subject: [erlang-questions] Sending messages to non-existing process In-Reply-To: References: Message-ID: 1. A lookup of the process is performed in the dictionary of all pids. 2. Having null result, the message is dropped (and remains in sender's heap until the next garbage collection) There should be no side effect other than the cost of locking/unlocking processes dict for that lookup. Sudden process registration should from that moment have all messages copied into the receiving heap, one copy per send, creating 100000000 copies on the receiver heap. On Fri, 25 Jan 2019 at 15:50, Frank Muller wrote: > Hey guys > > Is there any side effect of sending an infinite stream of messages to a > non-existing process registered by name? > > Ex. > 1> Term = ?. > 2> register(foo, self()). > 3> 1 / 0. > 4> [ foo ! Term || _ <- lists:eq(1. 10000000000000) ]. > > I?m curious about how the VM handles this scenario internally: > > 1. messages are simply discarded? And if this the case, will this hurt the > VM?s performances? > 2. something else??? > > Finally, what will happen if during the sending, a process get registered > with name 'foo' ? > > Thanks in advance > /Frank > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From frank.muller.erl@REDACTED Fri Jan 25 17:19:08 2019 From: frank.muller.erl@REDACTED (Frank Muller) Date: Fri, 25 Jan 2019 17:19:08 +0100 Subject: [erlang-questions] Sending messages to non-existing process In-Reply-To: References: Message-ID: Hey Dmytro So no matter if receiving process exists or not, those 100000000 messages have to be copied on the sender?s heap until next garbage collection is performed? Did I get it correctly? /Frank 1. A lookup of the process is performed in the dictionary of all pids. > 2. Having null result, the message is dropped (and remains in sender's > heap until the next garbage collection) > > There should be no side effect other than the cost of locking/unlocking > processes dict for that lookup. > Sudden process registration should from that moment have all messages > copied into the receiving heap, one copy per send, creating 100000000 > copies on the receiver heap. > > > On Fri, 25 Jan 2019 at 15:50, Frank Muller > wrote: > >> Hey guys >> >> Is there any side effect of sending an infinite stream of messages to a >> non-existing process registered by name? >> >> Ex. >> 1> Term = ?. >> 2> register(foo, self()). >> 3> 1 / 0. >> 4> [ foo ! Term || _ <- lists:eq(1. 10000000000000) ]. >> >> I?m curious about how the VM handles this scenario internally: >> >> 1. messages are simply discarded? And if this the case, will this hurt >> the VM?s performances? >> 2. something else??? >> >> Finally, what will happen if during the sending, a process get registered >> with name 'foo' ? >> >> Thanks in advance >> /Frank >> > _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmytro.lytovchenko@REDACTED Fri Jan 25 17:30:16 2019 From: dmytro.lytovchenko@REDACTED (Dmytro Lytovchenko) Date: Fri, 25 Jan 2019 16:30:16 +0000 Subject: [erlang-questions] Sending messages to non-existing process In-Reply-To: References: Message-ID: You already have it on the senders heap created in the line above. If receiver exists at any point, there will be a copy to receivers heap. If receiver never existed, then no new copies will be made. On Fri, 25 Jan 2019 at 16:19, Frank Muller wrote: > Hey Dmytro > > So no matter if receiving process exists or not, those 100000000 messages > have to be copied on the sender?s heap until next garbage collection is > performed? Did I get it correctly? > > /Frank > > 1. A lookup of the process is performed in the dictionary of all pids. >> 2. Having null result, the message is dropped (and remains in sender's >> heap until the next garbage collection) >> >> There should be no side effect other than the cost of locking/unlocking >> processes dict for that lookup. >> Sudden process registration should from that moment have all messages >> copied into the receiving heap, one copy per send, creating 100000000 >> copies on the receiver heap. >> >> >> On Fri, 25 Jan 2019 at 15:50, Frank Muller >> wrote: >> >>> Hey guys >>> >>> Is there any side effect of sending an infinite stream of messages to a >>> non-existing process registered by name? >>> >>> Ex. >>> 1> Term = ?. >>> 2> register(foo, self()). >>> 3> 1 / 0. >>> 4> [ foo ! Term || _ <- lists:eq(1. 10000000000000) ]. >>> >>> I?m curious about how the VM handles this scenario internally: >>> >>> 1. messages are simply discarded? And if this the case, will this hurt >>> the VM?s performances? >>> 2. something else??? >>> >>> Finally, what will happen if during the sending, a process get >>> registered with name 'foo' ? >>> >>> Thanks in advance >>> /Frank >>> >> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From frank.muller.erl@REDACTED Fri Jan 25 17:59:25 2019 From: frank.muller.erl@REDACTED (Frank Muller) Date: Fri, 25 Jan 2019 17:59:25 +0100 Subject: [erlang-questions] Sending messages to non-existing process In-Reply-To: References: Message-ID: Thank you for the clarification. /Frank You already have it on the senders heap created in the line above. > If receiver exists at any point, there will be a copy to receivers heap. > If receiver never existed, then no new copies will be made. > > On Fri, 25 Jan 2019 at 16:19, Frank Muller > wrote: > >> Hey Dmytro >> >> So no matter if receiving process exists or not, those 100000000 messages >> have to be copied on the sender?s heap until next garbage collection is >> performed? Did I get it correctly? >> >> /Frank >> >> 1. A lookup of the process is performed in the dictionary of all pids. >>> 2. Having null result, the message is dropped (and remains in sender's >>> heap until the next garbage collection) >>> >>> There should be no side effect other than the cost of locking/unlocking >>> processes dict for that lookup. >>> Sudden process registration should from that moment have all messages >>> copied into the receiving heap, one copy per send, creating 100000000 >>> copies on the receiver heap. >>> >>> >>> On Fri, 25 Jan 2019 at 15:50, Frank Muller >>> wrote: >>> >>>> Hey guys >>>> >>>> Is there any side effect of sending an infinite stream of messages to a >>>> non-existing process registered by name? >>>> >>>> Ex. >>>> 1> Term = ?. >>>> 2> register(foo, self()). >>>> 3> 1 / 0. >>>> 4> [ foo ! Term || _ <- lists:eq(1. 10000000000000) ]. >>>> >>>> I?m curious about how the VM handles this scenario internally: >>>> >>>> 1. messages are simply discarded? And if this the case, will this hurt >>>> the VM?s performances? >>>> 2. something else??? >>>> >>>> Finally, what will happen if during the sending, a process get >>>> registered with name 'foo' ? >>>> >>>> Thanks in advance >>>> /Frank >>>> >>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From gergo.nyiro@REDACTED Fri Jan 25 21:18:01 2019 From: gergo.nyiro@REDACTED (=?UTF-8?B?TnlpcsWRIEdlcmfFkQ==?=) Date: Fri, 25 Jan 2019 21:18:01 +0100 Subject: [erlang-questions] How to mimic Erlang nodes in PropEr statem test In-Reply-To: <20190125130305.GD1915@ferdmbp.local> References: <20190123175241.GC1915@ferdmbp.local> <20190125130305.GD1915@ferdmbp.local> Message-ID: On Fri, Jan 25, 2019 at 2:03 PM Fred Hebert wrote: > > On 01/23, Nyir? Gerg? wrote: > > > >Right now I have only one gen_server (crdt_server) per node, but I would > >like to add an event handler (crdt_event) which could persist the content > >of the crdt set or log the executed commands. > > > > Okay, so that's essentially duplicating the messages: send them to the > server, but you want to asynchronously log with another process. Do I > understand this right? Yes, and I can also write the members of the CRDT set (which will be a map in the final version) to disk with an extra event handler which would be registered only in the production code. > >Other solution would be to pass the NameOrPid of the crdt_event processes to > >crdt_server processes, but a process registry could be a more elegant solution. > > > > This is a possibility. There are other options such as: > > - Each crdt_server is spawned under a crdt_sup. The crdt_sup works by > booting two processes: crdt_log (first) and crdt_server (second). Each > is passed a name that identifies the server ("crdt_1", `node_a`, or > whatever) > - By using a dynamic process registry like gproc (see > https://hex.pm/packages/gproc) and replacing the name registration > from {local, Name} to {via, gproc, {n, l, {crdt_log, Name}}} and {via, > gproc, {n, l, {crdt_server, Name}}} you can give them unique names > that they can "know" > - On every call the crdt_server wants to log, it can call a function > from the crdt_log module, such as crdt_log:do_log(Name, Contents) I think the dynamic process registry was that I was looking for. (Good example, easy to understand). > >Should I use ct_property_test [1] instead? > > > > This is an experimental call or module to integrate the results of your > properties to the common test flow. You're free to use it if you want to > integrate everything into CT. I misunderstood the intention of the ct_property_test. I thought the test suite parameterization and dynamic node creation of ct can be passed to PropEr tests. But the warning is scary enough to avoid this solution:) > I personally maintain the rebar3 plugin for PropEr and I tend to prefer > it for some of the additional features it adds: Nice plugin (I found it through hex), makes my work easier. Thanks for the advice. Regards, Gerg? From gergo.nyiro@REDACTED Fri Jan 25 21:24:15 2019 From: gergo.nyiro@REDACTED (=?UTF-8?B?TnlpcsWRIEdlcmfFkQ==?=) Date: Fri, 25 Jan 2019 21:24:15 +0100 Subject: [erlang-questions] How to mimic Erlang nodes in PropEr statem test In-Reply-To: References: <20190123175241.GC1915@ferdmbp.local> <20190125130305.GD1915@ferdmbp.local> Message-ID: Hi Stavros, thanks for the references looks interesting. ConcuError was already on my learning plan, I will try with it. Regards, Gerg? On Fri, Jan 25, 2019 at 2:46 PM Stavros Aronis wrote: > before summer I was involved in writing a library for simulating > distributed Erlang systems on a single node: > https://github.com/klarna/vnet > > Perhaps you may find some ideas from there useful. You can read a > paper about it (published in Erlang '18) by following the "Modelling > Distributed Erlang within a Single Node" link in > https://concuerror.com/publications (indirect link given to avoid ACMs > access restrictions). > > An (admittedly poorly documented) example of its use (also discussed > in the paper) is available here: > https://github.com/klarna/vnet/tree/master/test/counter_server_example From t6sn7gt@REDACTED Sat Jan 26 17:10:52 2019 From: t6sn7gt@REDACTED (Donald Steven) Date: Sat, 26 Jan 2019 11:10:52 -0500 Subject: [erlang-questions] Spurious '100' in array:new Message-ID: <63bed5c3-cf7e-6df5-3f70-ca93ee2d93f9@aim.com> When I declare a new fixed-size array (of 19 values) within a map, with the statement: Sound = #{ ?? ?contourTag =>? SndContour?? = array:new(19, {default,{0, unspecified, 0}}), ?? ?... }, and print it out using io:format and ~p, I get: Sound: #{contourTag => {array,19,0,{0,unspecified,0},100}, --- Where is the '100' coming from? Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From fly@REDACTED Sat Jan 26 18:29:45 2019 From: fly@REDACTED (Fred Youhanaie) Date: Sat, 26 Jan 2019 17:29:45 +0000 Subject: [erlang-questions] Spurious '100' in array:new In-Reply-To: <63bed5c3-cf7e-6df5-3f70-ca93ee2d93f9@aim.com> References: <63bed5c3-cf7e-6df5-3f70-ca93ee2d93f9@aim.com> Message-ID: <5eea2443-4c11-0b71-2c72-03a11f173df4@anydata.co.uk> That's part of the internal structure of the array module, the data structure is opaque and should not be relied upon. If you need to access/print the data contents of the array use array:to_list(Array). If you're curious about the internal structure of the array, you're best option is to read the source code itself. You can also experiment with creating arrays of various sizes and assigning values to some elements. Try sizes of 9, 19, 119, etc. Cheers, Fred On 26/01/2019 16:10, Donald Steven wrote: > When I declare a new fixed-size array (of 19 values) within a map, with the statement: > > Sound = #{ > > ?? ?contourTag =>? SndContour?? = array:new(19, {default,{0, unspecified, 0}}), > ?? ?... > > }, > > and print it out using io:format and ~p, I get: > > Sound: #{contourTag => {array,19,0,{0,unspecified,0},100}, > > --- > > Where is the '100' coming from? > > Thanks. > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From t6sn7gt@REDACTED Sat Jan 26 21:04:14 2019 From: t6sn7gt@REDACTED (Donald Steven) Date: Sat, 26 Jan 2019 15:04:14 -0500 Subject: [erlang-questions] Syntax for accessing a single value of a map Message-ID: <5051b388-1cdf-4606-2211-2c614dcb8d42@aim.com> The book "Learn You Some..." notes that the syntax M#{K} may not work and indeed it results in an error when I tried to use it.? What is the correct syntax? From starbelly@REDACTED Sat Jan 26 22:20:45 2019 From: starbelly@REDACTED (Bryan Paxton) Date: Sat, 26 Jan 2019 15:20:45 -0600 Subject: [erlang-questions] Syntax for accessing a single value of a map In-Reply-To: <5051b388-1cdf-4606-2211-2c614dcb8d42@aim.com> References: <5051b388-1cdf-4606-2211-2c614dcb8d42@aim.com> Message-ID: <1bd6f9f1-b791-33af-53ee-98dc00d90ed0@pobox.com> ?Hi Donald, This part of the syntax has not been implemented yet. Currently you need to use maps:get/2 or maps:get/3 (http://erlang.org/doc/man/maps.html#get-2) Example: Eshell V10.2.1? (abort with ^G) 1> M = #{foo => bar}. #{foo => bar} 2> maps:get(foo, M). bar 3> maps:get(bar, M, undefined). undefined 4> Cheers -- Bryan On 1/26/19 2:04 PM, Donald Steven wrote: > The book "Learn You Some..." notes that the syntax M#{K} may not work > and indeed it results in an error when I tried to use it.? What is the > correct syntax? > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From t6sn7gt@REDACTED Sat Jan 26 22:26:29 2019 From: t6sn7gt@REDACTED (Donald Steven) Date: Sat, 26 Jan 2019 16:26:29 -0500 Subject: [erlang-questions] Syntax for accessing a single value of a map In-Reply-To: <1bd6f9f1-b791-33af-53ee-98dc00d90ed0@pobox.com> References: <5051b388-1cdf-4606-2211-2c614dcb8d42@aim.com> <1bd6f9f1-b791-33af-53ee-98dc00d90ed0@pobox.com> Message-ID: <13becad4-b86d-e60f-29b9-b9d770826189@aim.com> Thanks Brian, that's very helpful. Best, Don On 1/26/2019 4.20 PM, Bryan Paxton wrote: > ?Hi Donald, > > This part of the syntax has not been implemented yet. Currently you need > to use maps:get/2 or maps:get/3 (http://erlang.org/doc/man/maps.html#get-2) > > > Example: > > Eshell V10.2.1? (abort with ^G) > 1> M = #{foo => bar}. > #{foo => bar} > 2> maps:get(foo, M). > bar > 3> maps:get(bar, M, undefined). > undefined > 4> > > > Cheers > > -- > > Bryan > > > On 1/26/19 2:04 PM, Donald Steven wrote: >> The book "Learn You Some..." notes that the syntax M#{K} may not work >> and indeed it results in an error when I tried to use it.? What is the >> correct syntax? >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From empro2@REDACTED Sun Jan 27 15:22:01 2019 From: empro2@REDACTED (empro2@REDACTED) Date: Sun, 27 Jan 2019 15:22:01 +0100 Subject: [erlang-questions] Syntax for accessing a single value of a map In-Reply-To: <13becad4-b86d-e60f-29b9-b9d770826189@aim.com> References: <5051b388-1cdf-4606-2211-2c614dcb8d42@aim.com> <1bd6f9f1-b791-33af-53ee-98dc00d90ed0@pobox.com> <13becad4-b86d-e60f-29b9-b9d770826189@aim.com> Message-ID: <20190127152201.4b372522@raspy> What about matching? M = #{1 => 2}. #{1 => 2} 143> #{1 := V} = M. #{1 => 2} 144> V. 2 What about reading the manual? :-) Similarly: %% Instead of Val = Record#rec_name.some_field. #rec_name{some_field = Val} = Record. Lest Erlang turns into Java ;-) Michael -- You do not live in your environment, you are part of it. From g@REDACTED Sun Jan 27 21:56:00 2019 From: g@REDACTED (Guilherme Andrade) Date: Sun, 27 Jan 2019 20:56:00 +0000 Subject: [erlang-questions] [ANN] locus: Geolocation and ASN lookup of IP addresses In-Reply-To: References: Message-ID: Hello list, Locus 1.6.0 was released today[1]. Added: - new API method for validating loaded databases (locus:analyze/1) - new command line tool supporting database validation Changed: - safety of database HTTPS downloads was substantially improved by now rejecting expired certificates, mismatched hostnames, self-signed certificates or unknown certificate authorities - test coverage using MaxMind's test data was greatly extended - database decoder was thoroughly optimized - documentation was mildly improved Fixed: - misguided rejection of UTF-8 strings with non-printable (but valid) codepoints - unnecessarily strict refusal to load 2.x database formats succeeding 2.0 - infinite recursion in maliciously crafted databases due to circular paths * Hex package: https://hex.pm/packages/locus/1.6.0 * Documentation: https://hexdocs.pm/locus/1.6.0/ * Source code (GitHub): https://github.com/g-andrade/locus/tree/1.6.0 * Source code (GitLab): https://gitlab.com/g-andrade/locus/tree/1.6.0 [1] For those who don't have the existing thread handy, locus is an Erlang/OTP library for geolocation and ASN lookup of IP addresses, using MaxMind GeoLite2 databases. On Wed, 28 Mar 2018 at 21:34, Guilherme Andrade wrote: > Hi list, > > Locus 1.3.0 was released today. For those who don't have the existing > thread handy, it's library for looking up geolocation / ASN of IP > addresses, using MaxMind GeoLite2. > > Added: > - ability of loading databases from local file system > - type spec of database entries > > Fixed: > - wrong handling of timezones on cached tarballs > - wrong handling of daylight saving time on conditional HTTP requests > > The timezone / DST fixes mentioned above were also backported to earlier > versions and tagged under: > - 1.0.1 > - 1.1.4 > - 1.2.2 > > * Overview: https://github.com/g-andrade/locus > * Documentation: https://hexdocs.pm/locus/ > > > -- > Guilherme > -- Guilherme -------------- next part -------------- An HTML attachment was scrubbed... URL: From igor.clark@REDACTED Mon Jan 28 09:45:30 2019 From: igor.clark@REDACTED (Igor Clark) Date: Mon, 28 Jan 2019 08:45:30 +0000 Subject: [erlang-questions] Using SNI with cowboy:start_tls/3 Message-ID: <6E66D9F1-8791-4B3F-AD69-2AB2552BA99A@gmail.com> Hi there, Does anyone have any examples of how to use sni_hosts correctly in TransportOpts in cowboy:start_tls/3? I want to use different certs for respective diferent hostnames, and I'm having trouble deciphering the docs as the definition seems spread across cowboy, ranch and erlang:ssl. From what I can tell the underlying erlang:ssl wants this: { sni_hosts, [ { hostname(), [ ssl_option() ] } ] } which seems like it should work like e.g. { sni_hosts, [ { "hostname1", [ { certfile, Certfile1 }, { keyfile, Keyfile1 }, { cacertfile, CaCertFile1 } ] } ] } but when I add that directly to the TransportOpts in cowboy:start_tls/3 alongside { port, Port } etc I just get bad_return. There are other options that seem relevant e.g. server_name_indication but I'm finding it tricky to piece it all together to make the right shape of TransportOpts - I'm not clear whether the ssl_opts listed under the ranch_ssl module docs are simply passed through from erlang:ssl or whether they're a ranch-specific subset, and either way how to structure the proplist to configure Cowboy to serve multiple TLS names. Would very much appreciate any tips or examples! Thanks, Igor -------------- next part -------------- An HTML attachment was scrubbed... URL: From luke@REDACTED Mon Jan 28 16:36:51 2019 From: luke@REDACTED (Luke Bakken) Date: Mon, 28 Jan 2019 07:36:51 -0800 Subject: [erlang-questions] Spurious '100' in array:new In-Reply-To: <63bed5c3-cf7e-6df5-3f70-ca93ee2d93f9@aim.com> References: <63bed5c3-cf7e-6df5-3f70-ca93ee2d93f9@aim.com> Message-ID: Hi Donald - I have never used the array data type, but I assume that the 100 is an internal implementation detail. https://github.com/erlang/otp/blob/OTP-21.2.4/lib/stdlib/src/array.erl#L151-L160 Notice that the last element in the array record is for "elements", which are defined immediately above the record in the source code. non_neg_integer() can be the sole element in your array, and is determined by this code: https://github.com/erlang/otp/blob/OTP-21.2.4/lib/stdlib/src/array.erl#L286-L298 In your case, the ?extend macro is evaluated and the result of 10 * (NODESIZE * LEAFSIZE) is stored in the elements field of the record. At least, I think this is the correct reading of this code :-) Thanks, Luke On Sat, Jan 26, 2019 at 8:11 AM Donald Steven wrote: > > When I declare a new fixed-size array (of 19 values) within a map, with the statement: > > Sound = #{ > > contourTag => SndContour = array:new(19, {default,{0, unspecified, 0}}), > ... > > }, > > and print it out using io:format and ~p, I get: > > Sound: #{contourTag => {array,19,0,{0,unspecified,0},100}, > > --- > > Where is the '100' coming from? > > Thanks. > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From luke@REDACTED Mon Jan 28 16:40:50 2019 From: luke@REDACTED (Luke Bakken) Date: Mon, 28 Jan 2019 07:40:50 -0800 Subject: [erlang-questions] Spurious '100' in array:new In-Reply-To: References: <63bed5c3-cf7e-6df5-3f70-ca93ee2d93f9@aim.com> Message-ID: I left an extra 10 in my explanation, it should have read: In your case, the ?extend macro is evaluated and the result of (NODESIZE * LEAFSIZE) is stored in the elements field of the record. On Mon, Jan 28, 2019 at 7:36 AM Luke Bakken wrote: > > Hi Donald - > > I have never used the array data type, but I assume that the 100 is an > internal implementation detail. > > https://github.com/erlang/otp/blob/OTP-21.2.4/lib/stdlib/src/array.erl#L151-L160 > > Notice that the last element in the array record is for "elements", > which are defined immediately above the record in the source code. > non_neg_integer() can be the sole element in your array, and is > determined by this code: > > https://github.com/erlang/otp/blob/OTP-21.2.4/lib/stdlib/src/array.erl#L286-L298 > > In your case, the ?extend macro is evaluated and the result of 10 * > (NODESIZE * LEAFSIZE) is stored in the elements field of the record. > > At least, I think this is the correct reading of this code :-) > Thanks, > Luke From jesper.louis.andersen@REDACTED Mon Jan 28 17:01:53 2019 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Mon, 28 Jan 2019 17:01:53 +0100 Subject: [erlang-questions] Spurious '100' in array:new In-Reply-To: <5eea2443-4c11-0b71-2c72-03a11f173df4@anydata.co.uk> References: <63bed5c3-cf7e-6df5-3f70-ca93ee2d93f9@aim.com> <5eea2443-4c11-0b71-2c72-03a11f173df4@anydata.co.uk> Message-ID: On Sat, Jan 26, 2019 at 6:29 PM Fred Youhanaie wrote: > That's part of the internal structure of the array module, the data > structure is opaque and should not be relied upon. > > For full reference, you can glean on the internal structure: Erlang/OTP 21 [erts-10.2.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe] [dtrace] Eshell V10.2.1 (abort with ^G) 1> rr(array). [array] 2> array:new(19, {default, {0, unspecified, 0}}). #array{size = 19,max = 0, default = {0,unspecified,0}, elements = 100} 3> array:new(9, {default, {0, unspecified, 0}}). #array{size = 9,max = 0, default = {0,unspecified,0}, elements = 10} Here is what is happening: In Erlang, an Array is a very flat tree with a branching factor of 10. It is stored in `elements`. But instead of storing a tree, you may store an integer encoding how much more space is allowed in this part of the tree when it is expanded. You have created a tree of size 19, which means it needs two expansions of size 10, or 100---though not all of those expansions will be used, as it is extra padding. If you had created an array of size 101, it would have been 1000, and so on. It is always the next log10 ceiling value. In the example with size 9, only one expansion is needed in which case we have a `10` in the elements. You might ask why it isn't represented as a traditional array. This is because of persistence in data structures: older versions of the tree can be kept alive, so you cannot overwrite an older array. It is useful in a number of situations, the most important one being that you can know the state of a crashing process before it crashed, and what event led to said crash. -------------- next part -------------- An HTML attachment was scrubbed... URL: From t6sn7gt@REDACTED Mon Jan 28 21:26:25 2019 From: t6sn7gt@REDACTED (Donald Steven) Date: Mon, 28 Jan 2019 15:26:25 -0500 Subject: [erlang-questions] Spurious '100' in array:new In-Reply-To: References: <63bed5c3-cf7e-6df5-3f70-ca93ee2d93f9@aim.com> <5eea2443-4c11-0b71-2c72-03a11f173df4@anydata.co.uk> Message-ID: <2b04dac6-1b27-03dd-54d2-9b97621429a0@aim.com> Thanks Jesper, that's most helpful to know.? Is there a particular way to print it in io:format that will *not* show the 'extra stuff'. (I used ~p) Best, Don On 1/28/2019 11.01 AM, Jesper Louis Andersen wrote: > On Sat, Jan 26, 2019 at 6:29 PM Fred Youhanaie > wrote: > > That's part of the internal structure of the array module, the > data structure is opaque and should not be relied upon. > > > For full reference, you can glean on the internal structure: > > Erlang/OTP 21 [erts-10.2.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] > [async-threads:1] [hipe] [dtrace] > > Eshell V10.2.1? (abort with ^G) > 1> rr(array). > [array] > 2> array:new(19, {default, {0, unspecified, 0}}). > #array{size = 19,max = 0, > ?????? default = {0,unspecified,0}, > ?????? elements = 100} > 3> array:new(9, {default, {0, unspecified, 0}}). > #array{size = 9,max = 0, > ?????? default = {0,unspecified,0}, > ?????? elements = 10} > > Here is what is happening: In Erlang, an Array is a very flat tree > with a branching factor of 10. It is stored in `elements`. But instead > of storing a tree, you may store an integer encoding how much more > space is allowed in this part of the tree when it is expanded. You > have created a tree of size 19, which means it needs two expansions of > size 10, or 100---though not all of those expansions will be used, as > it is extra padding. If you had created an array of size 101, it would > have been 1000, and so on. It is always the next log10 ceiling value. > In the example with size 9, only one expansion is needed in which case > we have a `10` in the elements. > > You might ask why it isn't represented as a traditional array. This is > because of persistence in data structures: older versions of the tree > can be kept alive, so you cannot overwrite an older array. It is > useful in a number of situations, the most important one being that > you can know the state of a crashing process before it crashed, and > what event led to said crash. > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From fly@REDACTED Tue Jan 29 11:42:18 2019 From: fly@REDACTED (Fred Youhanaie) Date: Tue, 29 Jan 2019 10:42:18 +0000 Subject: [erlang-questions] test - reply to email never arrived Message-ID: <2fc9e328-49e0-86e0-a4f9-80d688426ec0@anydata.co.uk> sorry for the noise! checking if the erlang mail server is functioning. I sent a reply yesterday, but it never arrived, and it has not appeared in the mail archives either. Cheers Fred From fly@REDACTED Tue Jan 29 01:52:57 2019 From: fly@REDACTED (Fred Youhanaie) Date: Tue, 29 Jan 2019 00:52:57 +0000 Subject: [erlang-questions] Spurious '100' in array:new In-Reply-To: <2b04dac6-1b27-03dd-54d2-9b97621429a0@aim.com> References: <63bed5c3-cf7e-6df5-3f70-ca93ee2d93f9@aim.com> <5eea2443-4c11-0b71-2c72-03a11f173df4@anydata.co.uk> <2b04dac6-1b27-03dd-54d2-9b97621429a0@aim.com> Message-ID: <51815a43-a1df-85d1-aebf-6ed33ca1a557@anydata.co.uk> Don If you mean using a single ~p to print the whole map with the array inside it, then you'll end up with the extra stuff too. io:format cannot extract the data part of the array automatically. However, you can use the array:to_list/1 function to explicitly extract the data part of the array, and then create a "printable" map using maps:update/3, i.e. Map_with_array = ... % extract the array data as a list Array_list = array:to_list( maps:get(arrayKey, Map_with_array) ). % create the printable map, by replacing the array structure with the list equivalent Map_printable = maps:update( arrayKey, Array_list, Map_with_array ). % print the whole map with a single ~p io:format("The map: ~p.~n", [Map_printable]). Do bear in mind that array:to_list/1 will give you a long list with 19 x {0,unspecified,0} elements for your initial array! Cheers, Fred On 28/01/2019 20:26, Donald Steven wrote: > Thanks Jesper, that's most helpful to know.? Is there a particular way to print it in io:format that will *not* show the 'extra stuff'. (I used ~p) > > Best, > > Don > > On 1/28/2019 11.01 AM, Jesper Louis Andersen wrote: >> On Sat, Jan 26, 2019 at 6:29 PM Fred Youhanaie > wrote: >> >> That's part of the internal structure of the array module, the data structure is opaque and should not be relied upon. >> >> >> For full reference, you can glean on the internal structure: >> >> Erlang/OTP 21 [erts-10.2.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe] [dtrace] >> >> Eshell V10.2.1? (abort with ^G) >> 1> rr(array). >> [array] >> 2> array:new(19, {default, {0, unspecified, 0}}). >> #array{size = 19,max = 0, >> ?????? default = {0,unspecified,0}, >> ?????? elements = 100} >> 3> array:new(9, {default, {0, unspecified, 0}}). >> #array{size = 9,max = 0, >> ?????? default = {0,unspecified,0}, >> ?????? elements = 10} >> >> Here is what is happening: In Erlang, an Array is a very flat tree with a branching factor of 10. It is stored in `elements`. But instead of storing a tree, you may store an integer encoding how >> much more space is allowed in this part of the tree when it is expanded. You have created a tree of size 19, which means it needs two expansions of size 10, or 100---though not all of those >> expansions will be used, as it is extra padding. If you had created an array of size 101, it would have been 1000, and so on. It is always the next log10 ceiling value. In the example with size 9, >> only one expansion is needed in which case we have a `10` in the elements. >> >> You might ask why it isn't represented as a traditional array. This is because of persistence in data structures: older versions of the tree can be kept alive, so you cannot overwrite an older >> array. It is useful in a number of situations, the most important one being that you can know the state of a crashing process before it crashed, and what event led to said crash. >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From joagre@REDACTED Tue Jan 29 08:25:56 2019 From: joagre@REDACTED (Joa Gre) Date: Tue, 29 Jan 2019 08:25:56 +0100 Subject: [erlang-questions] Hierarchical module namespaces in Erlang Message-ID: Hi, Does anyone know the history about why Richard Carlsson's "Hierarchical module namespaces in Erlang" never made it into Erlang? http://www.erlang.se/workshop/2002/Carlsson.pdf /Joakim -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang.org@REDACTED Tue Jan 29 13:18:26 2019 From: erlang.org@REDACTED (Stanislaw Klekot) Date: Tue, 29 Jan 2019 13:18:26 +0100 Subject: [erlang-questions] Hierarchical module namespaces in Erlang In-Reply-To: References: Message-ID: <20190129121826.GA24172@jarowit.net> On Tue, Jan 29, 2019 at 08:25:56AM +0100, Joa Gre wrote: > Hi, > Does anyone know the history about why Richard Carlsson's "Hierarchical > module namespaces in Erlang" never made it into Erlang? > > http://www.erlang.se/workshop/2002/Carlsson.pdf They did, but they were scarcely used and got deprecated and removed somewhere between R15 and 19. -- Stanislaw Klekot From igor.clark@REDACTED Tue Jan 29 13:34:45 2019 From: igor.clark@REDACTED (Igor Clark) Date: Tue, 29 Jan 2019 12:34:45 +0000 Subject: [erlang-questions] test - reply to email never arrived In-Reply-To: <2fc9e328-49e0-86e0-a4f9-80d688426ec0@anydata.co.uk> References: <2fc9e328-49e0-86e0-a4f9-80d688426ec0@anydata.co.uk> Message-ID: There are definitely some issues with the list?s outgoing delivery. Messages from aim.com and yahoo.com which the list server resends don?t get delivered to my gmail, I think because Gmail thinks there?s a DMARC problem with the way the list server forwards them. So I get whole threads with missing messages and sometimes only ever see replies. And then I get bounce-probes from the list server presumably because Gmail bounces those messages. I?m guessing it?s maybe because the list server predates a lot of the dmarc/dkim stuff Gmail and other big provides do. > On 29 Jan 2019, at 10:42, Fred Youhanaie wrote: > > sorry for the noise! checking if the erlang mail server is functioning. > > I sent a reply yesterday, but it never arrived, and it has not appeared in the mail archives either. > > Cheers > Fred > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From raimo+erlang-questions@REDACTED Tue Jan 29 15:56:24 2019 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Tue, 29 Jan 2019 15:56:24 +0100 Subject: [erlang-questions] test - reply to email never arrived In-Reply-To: References: <2fc9e328-49e0-86e0-a4f9-80d688426ec0@anydata.co.uk> Message-ID: <20190129145624.GA85508@erix.ericsson.se> On Tue, Jan 29, 2019 at 12:34:45PM +0000, Igor Clark wrote: > There are definitely some issues with the list?s outgoing delivery. Messages from aim.com and yahoo.com which the list server resends don?t get delivered to my gmail, I think because Gmail thinks there?s a DMARC problem with the way the list server forwards them. So I get whole threads with missing messages and sometimes only ever see replies. And then I get bounce-probes from the list server presumably because Gmail bounces those messages. I?m guessing it?s maybe because the list server predates a lot of the dmarc/dkim stuff Gmail and other big provides do. Quite right! We are trying to find some time to work on this... > > > On 29 Jan 2019, at 10:42, Fred Youhanaie wrote: > > > > sorry for the noise! checking if the erlang mail server is functioning. > > > > I sent a reply yesterday, but it never arrived, and it has not appeared in the mail archives either. > > > > Cheers > > Fred -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From raimo+erlang-questions@REDACTED Tue Jan 29 16:14:29 2019 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Tue, 29 Jan 2019 16:14:29 +0100 Subject: [erlang-questions] test - reply to email never arrived In-Reply-To: <2fc9e328-49e0-86e0-a4f9-80d688426ec0@anydata.co.uk> References: <2fc9e328-49e0-86e0-a4f9-80d688426ec0@anydata.co.uk> Message-ID: <20190129151429.GA86267@erix.ericsson.se> On Tue, Jan 29, 2019 at 10:42:18AM +0000, Fred Youhanaie wrote: > sorry for the noise! checking if the erlang mail server is functioning. > > I sent a reply yesterday, but it never arrived, and it has not appeared in the mail archives either. I can not find a mail from you from that date, that is not even as any incoming mail from your address. I find a resend of a reply from the 26:th that got back again at the 29:th 01:52:57 CET, and both of these are in the archive. > > Cheers > Fred Best regards -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From fly@REDACTED Tue Jan 29 16:29:30 2019 From: fly@REDACTED (Fred Youhanaie) Date: Tue, 29 Jan 2019 15:29:30 +0000 Subject: [erlang-questions] test - reply to email never arrived In-Reply-To: <20190129151429.GA86267@erix.ericsson.se> References: <2fc9e328-49e0-86e0-a4f9-80d688426ec0@anydata.co.uk> <20190129151429.GA86267@erix.ericsson.se> Message-ID: Hi Raimo On 29/01/2019 15:14, Raimo Niskanen wrote: > On Tue, Jan 29, 2019 at 10:42:18AM +0000, Fred Youhanaie wrote: >> I sent a reply yesterday, but it never arrived, and it has not appeared in the mail archives either. > > I can not find a mail from you from that date, that is not even as any > incoming mail from your address. The email mentioned above is from 29th at 00:50-ish GMT. So, s/yesterday/today/ :-() That email did arrive this morning/afternoon, shortly after mentioning it on slack. Many thanks, Fred From essen@REDACTED Wed Jan 30 00:26:17 2019 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Wed, 30 Jan 2019 00:26:17 +0100 Subject: [erlang-questions] Using SNI with cowboy:start_tls/3 In-Reply-To: <6E66D9F1-8791-4B3F-AD69-2AB2552BA99A@gmail.com> References: <6E66D9F1-8791-4B3F-AD69-2AB2552BA99A@gmail.com> Message-ID: Hey, As far as Ranch is concerned, the options are given directly to the ssl application. What Ranch does is tweak some default values (unrelated to ssl), remove some options like active or packet (same), and pass them forward. The types in the docs are there merely for documentation purposes. Afraid that's all the help I have. Cheers, On 28/01/2019 09:45, Igor Clark wrote: > Hi there, > > Does anyone have any examples of how to use sni_hosts correctly in > TransportOpts in cowboy:start_tls/3? > > I want to use different certs for respective diferent hostnames,?and I'm > having trouble deciphering the docs as the definition seems spread > across cowboy, ranch and erlang:ssl. From what I can tell the underlying > erlang:ssl wants?this: > > { sni_hosts, [ { hostname(), [ ssl_option() ] } ] } > > which seems like it should work like e.g. > > { sni_hosts, [ { "hostname1", [ { certfile, Certfile1 }, { keyfile, > Keyfile1 }, { cacertfile, CaCertFile1 } ] } ] } > > but when I add that directly to the TransportOpts in cowboy:start_tls/3 > alongside { port, Port } etc I just get bad_return. There are other > options that seem relevant e.g. server_name_indication but I'm finding > it tricky to piece it all together to make the right shape of > TransportOpts - I'm not clear whether the ssl_opts listed under the > ranch_ssl module docs > ?are simply > passed through from erlang:ssl ?or > whether they're a ranch-specific subset, and either way how to structure > the proplist to configure Cowboy to serve multiple TLS names. > > Would very much appreciate any tips or examples! > > Thanks, > Igor > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin https://ninenines.eu From roger@REDACTED Wed Jan 30 12:18:52 2019 From: roger@REDACTED (Roger Lipscombe) Date: Wed, 30 Jan 2019 11:18:52 +0000 Subject: [erlang-questions] Can I build Erlang with make -j? Message-ID: I'm building Erlang/OTP (using kerl, but I don't think that's relevant) and it's taking forever. Can I use MAKEFLAGS=-j8 to make it quicker? Is this supported? I mean, sure, it *appears* to be working (though something's using more than 8 cores, somehow, and my music playback's getting choppy). But is it *guaranteed* to work? From jesper.eskilson@REDACTED Wed Jan 30 12:20:56 2019 From: jesper.eskilson@REDACTED (Jesper Eskilson) Date: Wed, 30 Jan 2019 12:20:56 +0100 Subject: [erlang-questions] Can I build Erlang with make -j? In-Reply-To: References: Message-ID: As for it being "supported" I'll let the OTP team answer, but I'm using parallell build all the time and have not seen any issues related to it. /Jesper On Wed, Jan 30, 2019 at 12:19 PM Roger Lipscombe wrote: > I'm building Erlang/OTP (using kerl, but I don't think that's > relevant) and it's taking forever. Can I use MAKEFLAGS=-j8 to make it > quicker? Is this supported? > > I mean, sure, it *appears* to be working (though something's using > more than 8 cores, somehow, and my music playback's getting choppy). > But is it *guaranteed* to work? > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- *Jesper Eskilson* Senior Software Engineer Kred Core +46-72-855-8421 Klarna Bank AB (publ) Sveav?gen 46, 111 34 Stockholm Tel: +46 8 120 120 00 <+46812012000> Reg no: 556737-0431 klarna.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ingela.andin@REDACTED Wed Jan 30 12:32:23 2019 From: ingela.andin@REDACTED (Ingela Andin) Date: Wed, 30 Jan 2019 12:32:23 +0100 Subject: [erlang-questions] Using SNI with cowboy:start_tls/3 In-Reply-To: References: <6E66D9F1-8791-4B3F-AD69-2AB2552BA99A@gmail.com> Message-ID: Hi! Your exampel looks correct. Here is another one {sni_hosts, [{"a.server", [{certfile, "a_cert.pem"}, {keyfile, "a_key.pem"}, {keyfile, "a_key.pem"} ]}, {"b.server", [{certfile, "b_cert.pem"}, {keyfile, "b_key.pem"}, {keyfile, "b_key.pem"} ]} ]} If you want to know if the problem is with cowboy or ssl try calling ssl API without cowboy Regards Ingela Den ons 30 jan. 2019 kl 00:26 skrev Lo?c Hoguin : > Hey, > > As far as Ranch is concerned, the options are given directly to the ssl > application. What Ranch does is tweak some default values (unrelated to > ssl), remove some options like active or packet (same), and pass them > forward. The types in the docs are there merely for documentation purposes. > > Afraid that's all the help I have. > > Cheers, > > On 28/01/2019 09:45, Igor Clark wrote: > > Hi there, > > > > Does anyone have any examples of how to use sni_hosts correctly in > > TransportOpts in cowboy:start_tls/3? > > > > I want to use different certs for respective diferent hostnames, and I'm > > having trouble deciphering the docs as the definition seems spread > > across cowboy, ranch and erlang:ssl. From what I can tell the underlying > > erlang:ssl wants this: > > > > { sni_hosts, [ { hostname(), [ ssl_option() ] } ] } > > > > which seems like it should work like e.g. > > > > { sni_hosts, [ { "hostname1", [ { certfile, Certfile1 }, { keyfile, > > Keyfile1 }, { cacertfile, CaCertFile1 } ] } ] } > > > > but when I add that directly to the TransportOpts in cowboy:start_tls/3 > > alongside { port, Port } etc I just get bad_return. There are other > > options that seem relevant e.g. server_name_indication but I'm finding > > it tricky to piece it all together to make the right shape of > > TransportOpts - I'm not clear whether the ssl_opts listed under the > > ranch_ssl module docs > > are simply > > passed through from erlang:ssl or > > whether they're a ranch-specific subset, and either way how to structure > > the proplist to configure Cowboy to serve multiple TLS names. > > > > Would very much appreciate any tips or examples! > > > > Thanks, > > Igor > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > > -- > Lo?c Hoguin > https://ninenines.eu > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas@REDACTED Wed Jan 30 13:15:27 2019 From: lukas@REDACTED (Lukas Larsson) Date: Wed, 30 Jan 2019 13:15:27 +0100 Subject: [erlang-questions] Can I build Erlang with make -j? In-Reply-To: References: Message-ID: On Wed, Jan 30, 2019 at 12:19 PM Roger Lipscombe wrote: > I'm building Erlang/OTP (using kerl, but I don't think that's > relevant) and it's taking forever. Can I use MAKEFLAGS=-j8 to make it > quicker? Is this supported? > yes. Search for "-j" in http://erlang.org/doc/installation_guide/INSTALL.html Lukas -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmytro.lytovchenko@REDACTED Wed Jan 30 13:17:04 2019 From: dmytro.lytovchenko@REDACTED (Dmytro Lytovchenko) Date: Wed, 30 Jan 2019 13:17:04 +0100 Subject: [erlang-questions] Can I build Erlang with make -j? In-Reply-To: References: Message-ID: Make will guarantee the clean result if the makefile targets are lined up properly. So far I have observed it works well, no matter if I set MAKEFLAGS before I build or install with kerl or asdf, or if I generate automake/autoconf on a single core and then do make -j# On Wed, 30 Jan 2019 at 12:19, Roger Lipscombe wrote: > I'm building Erlang/OTP (using kerl, but I don't think that's > relevant) and it's taking forever. Can I use MAKEFLAGS=-j8 to make it > quicker? Is this supported? > > I mean, sure, it *appears* to be working (though something's using > more than 8 cores, somehow, and my music playback's getting choppy). > But is it *guaranteed* to work? > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From t6sn7gt@REDACTED Wed Jan 30 13:27:37 2019 From: t6sn7gt@REDACTED (Donald Steven) Date: Wed, 30 Jan 2019 07:27:37 -0500 Subject: [erlang-questions] Updating a map Message-ID: <83145c4d-6900-5cd6-221a-bacd6c1a79e6@aim.com> The documentation for maps:update states:? update(Key, Value, Map1) -> Map2, but since update has an arity of 3, this doesn't work:? update(Key1, Value1, Key2, Value2, Map1) -> Map2.? Is there an other-than one-by-one-sequential-update to update more than one key-value in one shot? Thanks. Don From gerhard@REDACTED Wed Jan 30 13:34:05 2019 From: gerhard@REDACTED (Gerhard Lazu) Date: Wed, 30 Jan 2019 12:34:05 +0000 Subject: [erlang-questions] Can I build Erlang with make -j? In-Reply-To: References: Message-ID: Yes, the -j flag is guaranteed to work as expected when building Erlang from source. If still in doubt, can you think of a series of checks that you could run post-build to ensure that the resulting artefact behaves correctly? We are using -j when compiling Erlang in the RabbitMQ Docker image: https://github.com/docker-library/rabbitmq/blob/8a623bb9420b8bfb1cbad2467abb48db26e7ddbc/3.7/ubuntu/Dockerfile#L138 As for checks, we verify that Erlang/OTP crypto & ssl were compiled against OpenSSL correctly: https://github.com/docker-library/rabbitmq/blob/8a623bb9420b8bfb1cbad2467abb48db26e7ddbc/3.7/ubuntu/Dockerfile#L159 The same -j flag is used in the Erlang Docker image: https://github.com/erlang/docker-erlang-otp/blob/a35e84764af1391f174905c7d18db3c1b7c629e6/21/Dockerfile#L29 Hope this helps, Gerhard. On Wed, Jan 30, 2019 at 11:19 AM Roger Lipscombe wrote: > I'm building Erlang/OTP (using kerl, but I don't think that's > relevant) and it's taking forever. Can I use MAKEFLAGS=-j8 to make it > quicker? Is this supported? > > I mean, sure, it *appears* to be working (though something's using > more than 8 cores, somehow, and my music playback's getting choppy). > But is it *guaranteed* to work? > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmytro.lytovchenko@REDACTED Wed Jan 30 13:37:35 2019 From: dmytro.lytovchenko@REDACTED (Dmytro Lytovchenko) Date: Wed, 30 Jan 2019 13:37:35 +0100 Subject: [erlang-questions] Can I build Erlang with make -j? In-Reply-To: References: Message-ID: Binary compare of all artifacts, without those parts where timestamp or temporary pathname is embedded into the output. On Wed, 30 Jan 2019 at 13:35, Gerhard Lazu wrote: > Yes, the -j flag is guaranteed to work as expected when building Erlang > from source. If still in doubt, can you think of a series of checks that > you could run post-build to ensure that the resulting artefact behaves > correctly? > > We are using -j when compiling Erlang in the RabbitMQ Docker image: > https://github.com/docker-library/rabbitmq/blob/8a623bb9420b8bfb1cbad2467abb48db26e7ddbc/3.7/ubuntu/Dockerfile#L138 > > As for checks, we verify that Erlang/OTP crypto & ssl were compiled > against OpenSSL correctly: > https://github.com/docker-library/rabbitmq/blob/8a623bb9420b8bfb1cbad2467abb48db26e7ddbc/3.7/ubuntu/Dockerfile#L159 > > The same -j flag is used in the Erlang Docker image: > https://github.com/erlang/docker-erlang-otp/blob/a35e84764af1391f174905c7d18db3c1b7c629e6/21/Dockerfile#L29 > > Hope this helps, Gerhard. > > On Wed, Jan 30, 2019 at 11:19 AM Roger Lipscombe > wrote: > >> I'm building Erlang/OTP (using kerl, but I don't think that's >> relevant) and it's taking forever. Can I use MAKEFLAGS=-j8 to make it >> quicker? Is this supported? >> >> I mean, sure, it *appears* to be working (though something's using >> more than 8 cores, somehow, and my music playback's getting choppy). >> But is it *guaranteed* to work? >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Oliver.Korpilla@REDACTED Wed Jan 30 13:47:21 2019 From: Oliver.Korpilla@REDACTED (Oliver Korpilla) Date: Wed, 30 Jan 2019 13:47:21 +0100 Subject: [erlang-questions] Updating a map In-Reply-To: <83145c4d-6900-5cd6-221a-bacd6c1a79e6@aim.com> References: <83145c4d-6900-5cd6-221a-bacd6c1a79e6@aim.com> Message-ID: Hello, Donald. You could use lists:foldr/3 to insert a list of key values into a map. You would use the map itself as a the accumulator. Best regards, Oliver? Gesendet:?Mittwoch, 30. Januar 2019 um 13:27 Uhr Von:?"Donald Steven" An:?"Erlang Questions" Betreff:?[erlang-questions] Updating a map The documentation for maps:update states:? update(Key, Value, Map1) -> Map2, but since update has an arity of 3, this doesn't work:? update(Key1, Value1, Key2, Value2, Map1) -> Map2.? Is there an other-than one-by-one-sequential-update to update more than one key-value in one shot? Thanks. Don _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions From Oliver.Korpilla@REDACTED Wed Jan 30 13:48:28 2019 From: Oliver.Korpilla@REDACTED (Oliver Korpilla) Date: Wed, 30 Jan 2019 13:48:28 +0100 Subject: [erlang-questions] Updating a map In-Reply-To: References: <83145c4d-6900-5cd6-221a-bacd6c1a79e6@aim.com> Message-ID: PS -?maps:from_list/1 might be also what you are looking for.? Gesendet:?Mittwoch, 30. Januar 2019 um 13:47 Uhr Von:?"Oliver Korpilla" An:?"Donald Steven" Cc:?"Erlang Questions" Betreff:?Re: [erlang-questions] Updating a map Hello, Donald. You could use lists:foldr/3 to insert a list of key values into a map. You would use the map itself as a the accumulator. Best regards, Oliver? Gesendet:?Mittwoch, 30. Januar 2019 um 13:27 Uhr Von:?"Donald Steven" An:?"Erlang Questions" Betreff:?[erlang-questions] Updating a map The documentation for maps:update states:? update(Key, Value, Map1) -> Map2, but since update has an arity of 3, this doesn't work:? update(Key1, Value1, Key2, Value2, Map1) -> Map2.? Is there an other-than one-by-one-sequential-update to update more than one key-value in one shot? Thanks. Don _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions[http://erlang.org/mailman/listinfo/erlang-questions] From g@REDACTED Wed Jan 30 14:13:09 2019 From: g@REDACTED (Guilherme Andrade) Date: Wed, 30 Jan 2019 13:13:09 +0000 Subject: [erlang-questions] Updating a map In-Reply-To: <83145c4d-6900-5cd6-221a-bacd6c1a79e6@aim.com> References: <83145c4d-6900-5cd6-221a-bacd6c1a79e6@aim.com> Message-ID: Hello Donald, Ever since Erlang/OTP 18 (or 19? I don't recall which), you can use the shorter map update syntax like this: Map = #{foo => bar, hello => world}, UpdatedMap = Map#{foo := barbar, hello := moon}. More information here: http://erlang.org/doc/reference_manual/expressions.html#map-expressions On Wed, 30 Jan 2019 at 12:27, Donald Steven wrote: > The documentation for maps:update states: update(Key, Value, Map1) -> > Map2, but since update has an arity of 3, this doesn't work: > update(Key1, Value1, Key2, Value2, Map1) -> Map2. Is there an > other-than one-by-one-sequential-update to update more than one > key-value in one shot? > > Thanks. > > > Don > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Guilherme -------------- next part -------------- An HTML attachment was scrubbed... URL: From fly@REDACTED Wed Jan 30 14:13:38 2019 From: fly@REDACTED (Fred Youhanaie) Date: Wed, 30 Jan 2019 13:13:38 +0000 Subject: [erlang-questions] Updating a map In-Reply-To: <83145c4d-6900-5cd6-221a-bacd6c1a79e6@aim.com> References: <83145c4d-6900-5cd6-221a-bacd6c1a79e6@aim.com> Message-ID: You can use the maps:fold/3, e.g. M = #{one => 1, two => 2, three => 3}. maps:fold(fun maps:update/3, M, #{one=>11, two=>22}). #{one => 11,three => 3,two => 22} Cheers, Fred On 30/01/2019 12:27, Donald Steven wrote: > The documentation for maps:update states:? update(Key, Value, Map1) -> Map2, but since update has an arity of 3, this doesn't work: update(Key1, Value1, Key2, Value2, Map1) -> Map2.? Is there an > other-than one-by-one-sequential-update to update more than one key-value in one shot? > > Thanks. > > > Don > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From roger@REDACTED Wed Jan 30 14:33:01 2019 From: roger@REDACTED (Roger Lipscombe) Date: Wed, 30 Jan 2019 13:33:01 +0000 Subject: [erlang-questions] Can I build Erlang with make -j? In-Reply-To: References: Message-ID: On Wed, 30 Jan 2019 at 12:15, Lukas Larsson wrote: > yes. Search for "-j" in http://erlang.org/doc/installation_guide/INSTALL.html So... I did that, but whatever search I'd done to find that page had dropped me at http://erlang.org/documentation/doc-5.9.1/doc/installation_guide/INSTALL.html, which doesn't mention it. I wish Google didn't do that. Thanks for the clarification. From t6sn7gt@REDACTED Wed Jan 30 15:37:16 2019 From: t6sn7gt@REDACTED (Donald Steven) Date: Wed, 30 Jan 2019 09:37:16 -0500 Subject: [erlang-questions] Updating a map In-Reply-To: References: <83145c4d-6900-5cd6-221a-bacd6c1a79e6@aim.com> Message-ID: Wow!? Thanks. On 1/30/2019 8.13 AM, Guilherme Andrade wrote: > Hello Donald, > > Ever since Erlang/OTP 18 (or 19? I don't recall which), you can use > the shorter map update syntax like this: > > Map = #{foo => bar, hello => world}, > UpdatedMap = Map#{foo := barbar, hello := moon}. > > More information here: > http://erlang.org/doc/reference_manual/expressions.html#map-expressions > > > On Wed, 30 Jan 2019 at 12:27, Donald Steven > wrote: > > The documentation for maps:update states:? update(Key, Value, > Map1) -> > Map2, but since update has an arity of 3, this doesn't work: > update(Key1, Value1, Key2, Value2, Map1) -> Map2.? Is there an > other-than one-by-one-sequential-update to update more than one > key-value in one shot? > > Thanks. > > > Don > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > -- > Guilherme -------------- next part -------------- An HTML attachment was scrubbed... URL: From igor.clark@REDACTED Wed Jan 30 16:47:04 2019 From: igor.clark@REDACTED (Igor Clark) Date: Wed, 30 Jan 2019 15:47:04 +0000 Subject: [erlang-questions] Using SNI with cowboy:start_tls/3 In-Reply-To: References: <6E66D9F1-8791-4B3F-AD69-2AB2552BA99A@gmail.com> Message-ID: Thank you both. I?ll see if I can try getting under the hood with just ssl. A bit surprised this hasn?t come up more often - I imagined people would want to host multiple names using cowboy pretty frequently! Not so, it seems? If I get anywhere with it I?ll post back. Thanks again, Igor > On 30 Jan 2019, at 11:32, Ingela Andin wrote: > > Hi! > Your exampel looks correct. > > Here is another one > > {sni_hosts, > [{"a.server", [{certfile, "a_cert.pem"}, > {keyfile, "a_key.pem"}, > {keyfile, "a_key.pem"} > ]}, > {"b.server", [{certfile, "b_cert.pem"}, > {keyfile, "b_key.pem"}, > {keyfile, "b_key.pem"} > ]} > ]} > > If you want to know if the problem is with cowboy or ssl try calling ssl API without cowboy > > Regards Ingela > > > Den ons 30 jan. 2019 kl 00:26 skrev Lo?c Hoguin : >> Hey, >> >> As far as Ranch is concerned, the options are given directly to the ssl >> application. What Ranch does is tweak some default values (unrelated to >> ssl), remove some options like active or packet (same), and pass them >> forward. The types in the docs are there merely for documentation purposes. >> >> Afraid that's all the help I have. >> >> Cheers, >> >> On 28/01/2019 09:45, Igor Clark wrote: >> > Hi there, >> > >> > Does anyone have any examples of how to use sni_hosts correctly in >> > TransportOpts in cowboy:start_tls/3? >> > >> > I want to use different certs for respective diferent hostnames, and I'm >> > having trouble deciphering the docs as the definition seems spread >> > across cowboy, ranch and erlang:ssl. From what I can tell the underlying >> > erlang:ssl wants this: >> > >> > { sni_hosts, [ { hostname(), [ ssl_option() ] } ] } >> > >> > which seems like it should work like e.g. >> > >> > { sni_hosts, [ { "hostname1", [ { certfile, Certfile1 }, { keyfile, >> > Keyfile1 }, { cacertfile, CaCertFile1 } ] } ] } >> > >> > but when I add that directly to the TransportOpts in cowboy:start_tls/3 >> > alongside { port, Port } etc I just get bad_return. There are other >> > options that seem relevant e.g. server_name_indication but I'm finding >> > it tricky to piece it all together to make the right shape of >> > TransportOpts - I'm not clear whether the ssl_opts listed under the >> > ranch_ssl module docs >> > are simply >> > passed through from erlang:ssl or >> > whether they're a ranch-specific subset, and either way how to structure >> > the proplist to configure Cowboy to serve multiple TLS names. >> > >> > Would very much appreciate any tips or examples! >> > >> > Thanks, >> > Igor >> > >> > _______________________________________________ >> > erlang-questions mailing list >> > erlang-questions@REDACTED >> > http://erlang.org/mailman/listinfo/erlang-questions >> > >> >> -- >> Lo?c Hoguin >> https://ninenines.eu >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From ingela.andin@REDACTED Wed Jan 30 18:18:34 2019 From: ingela.andin@REDACTED (Ingela Andin) Date: Wed, 30 Jan 2019 18:18:34 +0100 Subject: [erlang-questions] Using SNI with cowboy:start_tls/3 In-Reply-To: References: <6E66D9F1-8791-4B3F-AD69-2AB2552BA99A@gmail.com> Message-ID: Actually the easiest way to handle the new options based on sni extention nowdays is to use the handshske_continue functionallity. I do not know if that works with cowboy. Regards Ingela onsdag 30 januari 2019 skrev Igor Clark : > Thank you both. I?ll see if I can try getting under the hood with just ssl. > > A bit surprised this hasn?t come up more often - I imagined people would > want to host multiple names using cowboy pretty frequently! Not so, it > seems? > > If I get anywhere with it I?ll post back. > > Thanks again, > Igor > > On 30 Jan 2019, at 11:32, Ingela Andin wrote: > > Hi! > Your exampel looks correct. > > Here is another one > > {sni_hosts, > [{"a.server", [{certfile, "a_cert.pem"}, > {keyfile, "a_key.pem"}, > {keyfile, "a_key.pem"} > ]}, > {"b.server", [{certfile, "b_cert.pem"}, > {keyfile, "b_key.pem"}, > {keyfile, "b_key.pem"} > ]} > ]} > > If you want to know if the problem is with cowboy or ssl try calling ssl > API without cowboy > > Regards Ingela > > > Den ons 30 jan. 2019 kl 00:26 skrev Lo?c Hoguin : > >> Hey, >> >> As far as Ranch is concerned, the options are given directly to the ssl >> application. What Ranch does is tweak some default values (unrelated to >> ssl), remove some options like active or packet (same), and pass them >> forward. The types in the docs are there merely for documentation >> purposes. >> >> Afraid that's all the help I have. >> >> Cheers, >> >> On 28/01/2019 09:45, Igor Clark wrote: >> > Hi there, >> > >> > Does anyone have any examples of how to use sni_hosts correctly in >> > TransportOpts in cowboy:start_tls/3? >> > >> > I want to use different certs for respective diferent hostnames, and >> I'm >> > having trouble deciphering the docs as the definition seems spread >> > across cowboy, ranch and erlang:ssl. From what I can tell the >> underlying >> > erlang:ssl wants this: >> > >> > { sni_hosts, [ { hostname(), [ ssl_option() ] } ] } >> > >> > which seems like it should work like e.g. >> > >> > { sni_hosts, [ { "hostname1", [ { certfile, Certfile1 }, { keyfile, >> > Keyfile1 }, { cacertfile, CaCertFile1 } ] } ] } >> > >> > but when I add that directly to the TransportOpts in cowboy:start_tls/3 >> > alongside { port, Port } etc I just get bad_return. There are other >> > options that seem relevant e.g. server_name_indication but I'm finding >> > it tricky to piece it all together to make the right shape of >> > TransportOpts - I'm not clear whether the ssl_opts listed under the >> > ranch_ssl module docs >> > are simply >> > passed through from erlang:ssl or >> > whether they're a ranch-specific subset, and either way how to >> structure >> > the proplist to configure Cowboy to serve multiple TLS names. >> > >> > Would very much appreciate any tips or examples! >> > >> > Thanks, >> > Igor >> > >> > _______________________________________________ >> > erlang-questions mailing list >> > erlang-questions@REDACTED >> > http://erlang.org/mailman/listinfo/erlang-questions >> > >> >> -- >> Lo?c Hoguin >> https://ninenines.eu >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Wed Jan 30 18:29:06 2019 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Wed, 30 Jan 2019 18:29:06 +0100 Subject: [erlang-questions] Using SNI with cowboy:start_tls/3 In-Reply-To: References: <6E66D9F1-8791-4B3F-AD69-2AB2552BA99A@gmail.com> Message-ID: <45084c4f-2c5c-09c9-a991-5feaf55ca75e@ninenines.eu> A patch might be needed. Igor, please open a ticket in Cowboy or Ranch if you think it's necessary once you've done some experimentation. Cheers, On 30/01/2019 18:18, Ingela Andin wrote: > ?Actually the easiest way to handle the new options based on sni > extention nowdays is to use the handshske_continue functionallity. I do > not know if that works with cowboy. > > Regards Ingela > > onsdag 30 januari 2019 skrev Igor Clark >: > > Thank you both. I?ll see if I can try getting under the hood with > just ssl. > > A bit surprised this hasn?t come up more often - I imagined people > would want to host multiple names using cowboy pretty frequently! > Not so, it seems? > > If I get anywhere with it I?ll post back. > > Thanks again, > Igor > > On 30 Jan 2019, at 11:32, Ingela Andin > wrote: > >> Hi! >> Your exampel looks correct. >> >> Here is another one >> >> {sni_hosts, >> ?[{"a.server", [{certfile, "a_cert.pem"}, >> ??????????????? {keyfile, "a_key.pem"}, >> ??????????????? {keyfile, "a_key.pem"} >> ?????????????? ]}, >> ? {"b.server", [{certfile, "b_cert.pem"}, >> ??????????????? {keyfile, "b_key.pem"}, >> ??????????????? {keyfile, "b_key.pem"} >> ?????????????? ]} >> ?]} >> >> If you want to know if the problem is with cowboy or ssl try >> calling ssl API without cowboy >> >> Regards Ingela >> >> >> Den ons 30 jan. 2019 kl 00:26 skrev Lo?c Hoguin >> >: >> >> Hey, >> >> As far as Ranch is concerned, the options are given directly >> to the ssl >> application. What Ranch does is tweak some default values >> (unrelated to >> ssl), remove some options like active or packet (same), and >> pass them >> forward. The types in the docs are there merely for >> documentation purposes. >> >> Afraid that's all the help I have. >> >> Cheers, >> >> On 28/01/2019 09:45, Igor Clark wrote: >> > Hi there, >> > >> > Does anyone have any examples of how to use sni_hosts >> correctly in >> > TransportOpts in cowboy:start_tls/3? >> > >> > I want to use different certs for respective diferent >> hostnames,?and I'm >> > having trouble deciphering the docs as the definition seems >> spread >> > across cowboy, ranch and erlang:ssl. From what I can tell >> the underlying >> > erlang:ssl wants?this: >> > >> > { sni_hosts, [ { hostname(), [ ssl_option() ] } ] } >> > >> > which seems like it should work like e.g. >> > >> > { sni_hosts, [ { "hostname1", [ { certfile, Certfile1 }, { >> keyfile, >> > Keyfile1 }, { cacertfile, CaCertFile1 } ] } ] } >> > >> > but when I add that directly to the TransportOpts in >> cowboy:start_tls/3 >> > alongside { port, Port } etc I just get bad_return. There >> are other >> > options that seem relevant e.g. server_name_indication but >> I'm finding >> > it tricky to piece it all together to make the right shape of >> > TransportOpts - I'm not clear whether the ssl_opts listed >> under the >> > ranch_ssl module docs >> > > > >> are simply >> > passed through from erlang:ssl >> > >?or >> > whether they're a ranch-specific subset, and either way how >> to structure >> > the proplist to configure Cowboy to serve multiple TLS names. >> > >> > Would very much appreciate any tips or examples! >> > >> > Thanks, >> > Igor >> > >> > _______________________________________________ >> > erlang-questions mailing list >> > erlang-questions@REDACTED >> > http://erlang.org/mailman/listinfo/erlang-questions >> >> > >> >> -- >> Lo?c Hoguin >> https://ninenines.eu >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> -- Lo?c Hoguin https://ninenines.eu From jenrique85@REDACTED Thu Jan 31 06:22:58 2019 From: jenrique85@REDACTED (jenrique85) Date: Thu, 31 Jan 2019 00:22:58 -0500 Subject: [erlang-questions] Freelance with Erlang Message-ID: <34hx1rt72rcqkb4hva2s8sx8.1548912178270@email.android.com> Hello friends,Anyone know a web site where I can find freelance erlang tasks?I mean a web site just for that...Thanks Enviado desde mi dispositivo Samsung -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex@REDACTED Thu Jan 31 07:45:27 2019 From: alex@REDACTED (alex@REDACTED) Date: Thu, 31 Jan 2019 01:45:27 -0500 Subject: [erlang-questions] Can I build Erlang with make -j? In-Reply-To: References: Message-ID: Never had any issue using the -j flag as... CFLAGS="-O3 -fPIC -march=native -pipe" ./configure --prefix=/usr && make -j{number of cores + 1} Cheers, Alex On 1/30/19 7:34 AM, Gerhard Lazu wrote: > Yes, the -j flag is guaranteed to work as expected when building > Erlang from source. If still in doubt, can you think of a series of > checks that you could run post-build to ensure that the resulting > artefact behaves correctly? > > We are using -j when compiling Erlang in the RabbitMQ Docker image: > https://github.com/docker-library/rabbitmq/blob/8a623bb9420b8bfb1cbad2467abb48db26e7ddbc/3.7/ubuntu/Dockerfile#L138 > > As for checks, we verify that?Erlang/OTP crypto & ssl were compiled > against OpenSSL correctly: > https://github.com/docker-library/rabbitmq/blob/8a623bb9420b8bfb1cbad2467abb48db26e7ddbc/3.7/ubuntu/Dockerfile#L159 > > The same -j flag is used in the Erlang Docker image: > https://github.com/erlang/docker-erlang-otp/blob/a35e84764af1391f174905c7d18db3c1b7c629e6/21/Dockerfile#L29 > > Hope this?helps, Gerhard. > > On Wed, Jan 30, 2019 at 11:19 AM Roger Lipscombe > > wrote: > > I'm building Erlang/OTP (using kerl, but I don't think that's > relevant) and it's taking forever. Can I use MAKEFLAGS=-j8 to make it > quicker? Is this supported? > > I mean, sure, it *appears* to be working (though something's using > more than 8 cores, somehow, and my music playback's getting choppy). > But is it *guaranteed* to work? > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From devon.c.estes@REDACTED Thu Jan 31 09:58:15 2019 From: devon.c.estes@REDACTED (Devon Estes) Date: Thu, 31 Jan 2019 09:58:15 +0100 Subject: [erlang-questions] start_epmd flag not respected on shutdown Message-ID: Hello! So one of the operations folks at my current client is reporting some strange behavior with epmd. They're starting epmd separately from the application and then passing `-start_epmd false -name 192.168.0.1@REDACTED` when starting the application. They then noticed this behavior: So starting epmd, then starting the application works as expected, however when the application shuts down it still thinks it's "managing" epmd and so kills it. Is this expected behavior? We have a workaround in place at the moment, but I'm not sure if this is expected or not, or if there's some further configuration that we're missing to make epmd live on after the application is shut down. Cheers, Devon -- _________________ Devon Estes 203.559.0323 www.devonestes.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From zkessin@REDACTED Thu Jan 31 11:35:10 2019 From: zkessin@REDACTED (Zachary Kessin) Date: Thu, 31 Jan 2019 12:35:10 +0200 Subject: [erlang-questions] Freelance with Erlang In-Reply-To: <34hx1rt72rcqkb4hva2s8sx8.1548912178270@email.android.com> References: <34hx1rt72rcqkb4hva2s8sx8.1548912178270@email.android.com> Message-ID: there really isn't such a thing. There is freelance work out there, but you will have to go out and find it. Zach Kessin zach@REDACTED Helping Teams get started with Elixir +972 54 234 3956 / +44 203 734 9790 / +1 617 778 7213 WeWork Gav Yam Negev Beer Sheva 84709 Israel https://youtu.be/Z3Yk7ipAAyk ? On Thu, Jan 31, 2019 at 8:14 AM jenrique85 wrote: > > > Hello friends, > Anyone know a web site where I can find freelance erlang tasks? > I mean a web site just for that... > Thanks > > Enviado desde mi dispositivo Samsung > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From t6sn7gt@REDACTED Thu Jan 31 15:20:43 2019 From: t6sn7gt@REDACTED (Donald Steven) Date: Thu, 31 Jan 2019 09:20:43 -0500 Subject: [erlang-questions] Cleaning up code Message-ID: I need several large maps whose data will need to be accessed and updated in a number of functions.? I've defined them in main() just to get started, but they're so extensive that they swamp the readability of main().? I haven't found a way to "include" them as 1) erlang won't accept simple file inclusion anywhere (like m4) and 2) my attempt to define a map in an .hrl file (like a -record definition), as in: keywords = #( ...}.? or Keywords = #( ...}. just produces error messages.? I could use m4, but ... Any suggestions?? Thanks. Don From hugo@REDACTED Thu Jan 31 15:30:41 2019 From: hugo@REDACTED (Hugo Mills) Date: Thu, 31 Jan 2019 14:30:41 +0000 Subject: [erlang-questions] Cleaning up code In-Reply-To: References: Message-ID: <20190131143041.GB4461@carfax.org.uk> On Thu, Jan 31, 2019 at 09:20:43AM -0500, Donald Steven wrote: > I need several large maps whose data will need to be accessed and > updated in a number of functions.? I've defined them in main() just > to get started, but they're so extensive that they swamp the > readability of main().? I haven't found a way to "include" them as > 1) erlang won't accept simple file inclusion anywhere (like m4) and > 2) my attempt to define a map in an .hrl file (like a -record > definition), as in: > > keywords = #( ...}.? or Keywords = #( ...}. > > just produces error messages.? I could use m4, but ... > > Any suggestions?? Thanks. It's not clear how you're using the data, but that sounds like either a ETS table, or a separate server process to manage the data in each map. If you're only concerned about the size of the initialisation for each map, then you could just define them in a function and call that in your main(): main() -> Map1 = init_map1(), ... init_map1() -> #{a => 3, b => 4, ...}. Hugo. -- Hugo Mills | The last man on Earth sat in a room. Suddenly, there hugo@REDACTED carfax.org.uk | was a knock at the door. http://carfax.org.uk/ | PGP: E2AB1DE4 | Frederic Brown -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: Digital signature URL: From bchesneau@REDACTED Thu Jan 31 17:51:55 2019 From: bchesneau@REDACTED (Benoit Chesneau) Date: Thu, 31 Jan 2019 17:51:55 +0100 Subject: [erlang-questions] Freelance with Erlang In-Reply-To: <34hx1rt72rcqkb4hva2s8sx8.1548912178270@email.android.com> References: <34hx1rt72rcqkb4hva2s8sx8.1548912178270@email.android.com> Message-ID: you can join the slack and ask for such thing imo :) On Thu, Jan 31, 2019 at 7:14 AM jenrique85 wrote: > > > Hello friends, > Anyone know a web site where I can find freelance erlang tasks? > I mean a web site just for that... > Thanks > > Enviado desde mi dispositivo Samsung > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bchesneau@REDACTED Thu Jan 31 18:00:34 2019 From: bchesneau@REDACTED (Benoit Chesneau) Date: Thu, 31 Jan 2019 18:00:34 +0100 Subject: [erlang-questions] [ann] erlang-rocksdb 1.0.0 has been released Message-ID: Hi all, I just released rocksdb 1.0.0 a binding of rocksdb for Erlang based applications. Enjoy the stable API :) RocksDB is a library that provides an embeddable, persistent key-value store for fast storage, you can find more information about it on http://rocksdb.org. erlang-rocksdb 1.0.0 includes the following changes: - first stable version - miscellaneous build improvements with new build options and optimisations: https://gitlab.com/barrel-db/erlang-rocksdb/blob/master/doc/customize_rocksdb_build.md - stable API Most people will want to start with the Getting Started section of our wiki: https://gitlab.com/barrel-db/erlang-rocksdb/wikis/Getting-started. Full API reference: https://gitlab.com/barrel-db/erlang-rocksdb/blob/master/doc/rocksdb.md Sources are available on gitlab: https://gitlab.com/barrel-db/erlang-rocksdb/tree/master Or via hex: https://hex.pm/packages/rocksdb Enjoy! Beno?t. -------------- next part -------------- An HTML attachment was scrubbed... URL: From t6sn7gt@REDACTED Thu Jan 31 21:59:12 2019 From: t6sn7gt@REDACTED (Donald Steven) Date: Thu, 31 Jan 2019 15:59:12 -0500 Subject: [erlang-questions] Cleaning up code In-Reply-To: <20190131143041.GB4461@carfax.org.uk> References: <20190131143041.GB4461@carfax.org.uk> Message-ID: <8105ccdf-345f-b537-b041-556f66570347@aim.com> Thanks everyone.? I put the data in a separate module and that works (so far) well. But I'll look into ETS tables, other processes, and other ideas.? Much appreciated. Don On 1/31/2019 9.30 AM, Hugo Mills wrote: > On Thu, Jan 31, 2019 at 09:20:43AM -0500, Donald Steven wrote: >> I need several large maps whose data will need to be accessed and >> updated in a number of functions.? I've defined them in main() just >> to get started, but they're so extensive that they swamp the >> readability of main().? I haven't found a way to "include" them as >> 1) erlang won't accept simple file inclusion anywhere (like m4) and >> 2) my attempt to define a map in an .hrl file (like a -record >> definition), as in: >> >> keywords = #( ...}.? or Keywords = #( ...}. >> >> just produces error messages.? I could use m4, but ... >> >> Any suggestions?? Thanks. > It's not clear how you're using the data, but that sounds like > either a ETS table, or a separate server process to manage the data in > each map. > > If you're only concerned about the size of the initialisation for > each map, then you could just define them in a function and call that > in your main(): > > main() -> > Map1 = init_map1(), > ... > > > init_map1() -> > #{a => 3, b => 4, ...}. > > Hugo. > From frank.muller.erl@REDACTED Thu Jan 31 22:21:17 2019 From: frank.muller.erl@REDACTED (Frank Muller) Date: Thu, 31 Jan 2019 22:21:17 +0100 Subject: [erlang-questions] VM leaking memory Message-ID: Hello guys After adding a new feature to my app (running non-stop for 5 years), it started leaking memory in staging. Obviously, I?m suspecting this new feature. Command top shows RES going from 410m (during startup) to 6.2g in less than 12h. For stupid security reasons, it will take me weeks to be allowed to share collected statistics (from recon, entop) here, but I can share them in private if someone is willing to help. My config: Erlang 21.2.4 on CentOS 7 Thanks you. /Frank -------------- next part -------------- An HTML attachment was scrubbed... URL: From bchesneau@REDACTED Thu Jan 31 22:42:10 2019 From: bchesneau@REDACTED (Benoit Chesneau) Date: Thu, 31 Jan 2019 22:42:10 +0100 Subject: [erlang-questions] VM leaking memory In-Reply-To: References: Message-ID: is there any nif or something used by your new feature? On Thu, Jan 31, 2019 at 10:21 PM Frank Muller wrote: > Hello guys > > After adding a new feature to my app (running non-stop for 5 years), it > started leaking memory in staging. > > Obviously, I?m suspecting this new feature. Command top shows RES going > from 410m (during startup) to 6.2g in less than 12h. > > For stupid security reasons, it will take me weeks to be allowed to share > collected statistics (from recon, entop) here, but I can share them in > private if someone is willing to help. > > > My config: Erlang 21.2.4 on CentOS 7 > > Thanks you. > /Frank > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From frank.muller.erl@REDACTED Thu Jan 31 23:03:08 2019 From: frank.muller.erl@REDACTED (Frank Muller) Date: Thu, 31 Jan 2019 23:03:08 +0100 Subject: [erlang-questions] VM leaking memory In-Reply-To: References: Message-ID: Hi Benoit No NIF. Just pure Erlang /Frank is there any nif or something used by your new feature? > > On Thu, Jan 31, 2019 at 10:21 PM Frank Muller > wrote: > >> Hello guys >> >> After adding a new feature to my app (running non-stop for 5 years), it >> started leaking memory in staging. >> >> Obviously, I?m suspecting this new feature. Command top shows RES going >> from 410m (during startup) to 6.2g in less than 12h. >> >> For stupid security reasons, it will take me weeks to be allowed to share >> collected statistics (from recon, entop) here, but I can share them in >> private if someone is willing to help. >> >> >> My config: Erlang 21.2.4 on CentOS 7 >> >> Thanks you. >> /Frank >> > _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From frank.muller.erl@REDACTED Thu Jan 31 23:09:32 2019 From: frank.muller.erl@REDACTED (Frank Muller) Date: Thu, 31 Jan 2019 23:09:32 +0100 Subject: [erlang-questions] VM leaking memory In-Reply-To: References: Message-ID: Hi John I?ve a process which does that. Every 5mn it checks it the total memory went above a threshold and I it triggers a garbage collection. But this time it didn?t help. My app is a proxy server which forward packets from left to right by applying a transformation to them on the fly. /Frank That?s not a lot to go on. FWIW, most of my memory issues have been caused > by failing to understand this section of the manual: > http://erlang.org/doc/efficiency_guide/binaryhandling.html > > > > http://erlang.org/doc/man/erlang.html#garbage_collect-1 run manually can > be an interesting experiment to see what kind of memory problem you?ve got. > > > > *From:* erlang-questions-bounces@REDACTED < > erlang-questions-bounces@REDACTED> *On Behalf Of *Frank Muller > *Sent:* Thursday, January 31, 2019 14:21 > *To:* Erlang-Questions Questions > *Subject:* [erlang-questions] VM leaking memory > > > > Hello guys > > After adding a new feature to my app (running non-stop for 5 years), it > started leaking memory in staging. > > > Obviously, I?m suspecting this new feature. Command top shows RES going > from 410m (during startup) to 6.2g in less than 12h. > > For stupid security reasons, it will take me weeks to be allowed to share > collected statistics (from recon, entop) here, but I can share them in > private if someone is willing to help. > > > My config: Erlang 21.2.4 on CentOS 7 > > Thanks you. > /Frank > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mononcqc@REDACTED Thu Jan 31 23:14:15 2019 From: mononcqc@REDACTED (Fred Hebert) Date: Thu, 31 Jan 2019 17:14:15 -0500 Subject: [erlang-questions] VM leaking memory In-Reply-To: References: Message-ID: <20190131221414.GF1915@ferdmbp.local> On 01/31, Frank Muller wrote: >After adding a new feature to my app (running non-stop for 5 years), it >started leaking memory in staging. > >Obviously, I?m suspecting this new feature. Command top shows RES going >from 410m (during startup) to 6.2g in less than 12h. > >For stupid security reasons, it will take me weeks to be allowed to share >collected statistics (from recon, entop) here, but I can share them in >private if someone is willing to help. > I'd recommend checking things like: - recon_alloc:memory(usage) and see if the ratio is high or very low; this can point towards memory fragmentation if the ratio is low. - in case there is fragmentation (or something that looks like it) recon_alloc:fragmentation(current) will return lists of all the various allocators and types, which should help point towards which type of memory is causing issues - if usage seems high, see recon_alloc:memory(allocated_types) to see if there's any allocator that's higher than others; ETS, binary, or eheap will tend to point towards an ETS table, a refc binary leak, or some process gathering lots of memory Based on this it might be possible to then orient towards other avenues without you having to share any numbers. Quick checks if it's binary memory is to call recon:bin_leak(10), which will probe all processes for their binary memory usage, run a GC on all of them, then run a probe again, and give you those that have the largest gap. This can point to processes that had the most dead memory. There's an undocumented 'binary_memory' option that recon:info, recon:proc_count, and recon:proc_window all support -- it's undocumented because it might be expensive and not always safe to run -- that you can use to find which processes are holding the most binary memory; after a call to bin_leak, this can let you know about biggest users. You can also use proc_count with: - message_queue_len for large mailboxes - memory for eheap usage You can use the same values with proc_window to see who is currently allocating the most. If ETS is taking a lot of place, calling ets:i() can show a bunch of tables with content; you might have a runaway cache table or something like that. Regards, Fred. From mononcqc@REDACTED Thu Jan 31 23:16:37 2019 From: mononcqc@REDACTED (Fred Hebert) Date: Thu, 31 Jan 2019 17:16:37 -0500 Subject: [erlang-questions] [ann] erlang-rocksdb 1.0.0 has been released In-Reply-To: References: Message-ID: <20190131221636.GG1915@ferdmbp.local> Congrats on finally reaching 1.0.0! I have not yet had a project where rocksdb would make sense but the project always looked super interesting. Keep up the good work :) From frank.muller.erl@REDACTED Thu Jan 31 23:20:36 2019 From: frank.muller.erl@REDACTED (Frank Muller) Date: Thu, 31 Jan 2019 23:20:36 +0100 Subject: [erlang-questions] VM leaking memory In-Reply-To: <20190131221414.GF1915@ferdmbp.local> References: <20190131221414.GF1915@ferdmbp.local> Message-ID: Thanks Fred. Your recon library helped me a lot with additional tools like observer_cli (which wraps most of recon calls in a nice console GUI): https://github.com/zhongwencool/observer_cli/blob/master/README.md I?ve most of these information in my possession as I said to share in private. I will try the other ideas and report back. /Frank On 01/31, Frank Muller wrote: > >After adding a new feature to my app (running non-stop for 5 years), it > >started leaking memory in staging. > > > >Obviously, I?m suspecting this new feature. Command top shows RES going > >from 410m (during startup) to 6.2g in less than 12h. > > > >For stupid security reasons, it will take me weeks to be allowed to share > >collected statistics (from recon, entop) here, but I can share them in > >private if someone is willing to help. > > > > I'd recommend checking things like: > > - recon_alloc:memory(usage) and see if the ratio is high or very low; > this can point towards memory fragmentation if the ratio is low. > - in case there is fragmentation (or something that looks like it) > recon_alloc:fragmentation(current) will return lists of all the > various allocators and types, which should help point towards which > type of memory is causing issues > - if usage seems high, see recon_alloc:memory(allocated_types) to see if > there's any allocator that's higher than others; ETS, binary, or eheap > will tend to point towards an ETS table, a refc binary leak, or some > process gathering lots of memory > > Based on this it might be possible to then orient towards other avenues > without you having to share any numbers. > > Quick checks if it's binary memory is to call recon:bin_leak(10), which > will probe all processes for their binary memory usage, run a GC on all > of them, then run a probe again, and give you those that have the > largest gap. This can point to processes that had the most dead memory. > > There's an undocumented 'binary_memory' option that recon:info, > recon:proc_count, and recon:proc_window all support -- it's undocumented > because it might be expensive and not always safe to run -- that you can > use to find which processes are holding the most binary memory; after a > call to bin_leak, this can let you know about biggest users. > > You can also use proc_count with: > - message_queue_len for large mailboxes > - memory for eheap usage > > You can use the same values with proc_window to see who is currently > allocating the most. > > If ETS is taking a lot of place, calling ets:i() can show a bunch of > tables with content; you might have a runaway cache table or something > like that. > > Regards, > Fred. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From frank.muller.erl@REDACTED Thu Jan 31 23:26:22 2019 From: frank.muller.erl@REDACTED (Frank Muller) Date: Thu, 31 Jan 2019 23:26:22 +0100 Subject: [erlang-questions] VM leaking memory In-Reply-To: References: <20190131221414.GF1915@ferdmbp.local> Message-ID: The refc binary is the one using most of VM?s memory (in GB) compared to other allocators (few MB). /Frank Thanks Fred. Your recon library helped me a lot with additional tools like > observer_cli (which wraps most of recon calls in a nice console GUI): > https://github.com/zhongwencool/observer_cli/blob/master/README.md > > I?ve most of these information in my possession as I said to share in > private. I will try the other ideas and report back. > > /Frank > > On 01/31, Frank Muller wrote: >> >After adding a new feature to my app (running non-stop for 5 years), it >> >started leaking memory in staging. >> > >> >Obviously, I?m suspecting this new feature. Command top shows RES going >> >from 410m (during startup) to 6.2g in less than 12h. >> > >> >For stupid security reasons, it will take me weeks to be allowed to share >> >collected statistics (from recon, entop) here, but I can share them in >> >private if someone is willing to help. >> > >> >> I'd recommend checking things like: >> >> - recon_alloc:memory(usage) and see if the ratio is high or very low; >> this can point towards memory fragmentation if the ratio is low. >> - in case there is fragmentation (or something that looks like it) >> recon_alloc:fragmentation(current) will return lists of all the >> various allocators and types, which should help point towards which >> type of memory is causing issues >> - if usage seems high, see recon_alloc:memory(allocated_types) to see if >> there's any allocator that's higher than others; ETS, binary, or eheap >> will tend to point towards an ETS table, a refc binary leak, or some >> process gathering lots of memory >> >> Based on this it might be possible to then orient towards other avenues >> without you having to share any numbers. >> >> Quick checks if it's binary memory is to call recon:bin_leak(10), which >> will probe all processes for their binary memory usage, run a GC on all >> of them, then run a probe again, and give you those that have the >> largest gap. This can point to processes that had the most dead memory. >> >> There's an undocumented 'binary_memory' option that recon:info, >> recon:proc_count, and recon:proc_window all support -- it's undocumented >> because it might be expensive and not always safe to run -- that you can >> use to find which processes are holding the most binary memory; after a >> call to bin_leak, this can let you know about biggest users. >> >> You can also use proc_count with: >> - message_queue_len for large mailboxes >> - memory for eheap usage >> >> You can use the same values with proc_window to see who is currently >> allocating the most. >> >> If ETS is taking a lot of place, calling ets:i() can show a bunch of >> tables with content; you might have a runaway cache table or something >> like that. >> >> Regards, >> Fred. >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From scott_ribe@REDACTED Thu Jan 31 23:38:06 2019 From: scott_ribe@REDACTED (Scott Ribe) Date: Thu, 31 Jan 2019 15:38:06 -0700 Subject: [erlang-questions] VM leaking memory In-Reply-To: References: <20190131221414.GF1915@ferdmbp.local> Message-ID: > On Jan 31, 2019, at 3:26 PM, Frank Muller wrote: > > The refc binary is the one using most of VM?s memory (in GB) compared to other allocators (few MB). So, could be the classic leak where you match on a binary and pass a sub-binary to another process? That creates a ref to the original binary, and if the first process is a long-lived one that doesn't gc often, even though the second one finishes, the original binary hangs around... From mononcqc@REDACTED Thu Jan 31 23:52:48 2019 From: mononcqc@REDACTED (Fred Hebert) Date: Thu, 31 Jan 2019 17:52:48 -0500 Subject: [erlang-questions] VM leaking memory In-Reply-To: References: <20190131221414.GF1915@ferdmbp.local> Message-ID: <20190131225247.GH1915@ferdmbp.local> On 01/31, Scott Ribe wrote: >> On Jan 31, 2019, at 3:26 PM, Frank Muller wrote: >> >> The refc binary is the one using most of VM?s memory (in GB) compared to other allocators (few MB). > >So, could be the classic leak where you match on a binary and pass a sub-binary to another process? That creates a ref to the original binary, and if the first process is a long-lived one that doesn't gc often, even though the second one finishes, the original binary hangs around... Yeah, that's a good bet. I'd check in the new feature where a sub-binary of a large binary is taken, and would call `NewBin = binary:copy(Bin)` on it to free the ref as early as possible. From jkrukoff@REDACTED Thu Jan 31 22:53:39 2019 From: jkrukoff@REDACTED (John Krukoff) Date: Thu, 31 Jan 2019 21:53:39 +0000 Subject: [erlang-questions] VM leaking memory In-Reply-To: References: Message-ID: A non-text attachment was scrubbed... Name: smime.p7m Type: application/pkcs7-mime Size: 12441 bytes Desc: not available URL: