seeing core in erlang (erl.exe) executable while shutting down rabbitMQ(3.8.14) on windows

Pooja Desai pooja.desai10@REDACTED
Thu May 27 07:43:50 CEST 2021


Hi,



Investigated this issue further by using multiple versions of Erlang and
realised that crash dump is seen when *Erlang version >= 23.2* and built
with* –enable-fips option. *Crash is not generated if –enable-fips option
is not used while building erlang.



After scanning code found below commits added into OTP 23.2 which are
causing coredumps on windows:



https://github.com/erlang/otp/commit/392e80cd2aad6dfd4e503986f66cd10f15f9cbbb

https://github.com/erlang/otp/commit/2a954fe593e513774632abfe04aab303bff4e5f9



Also confirmed that coredump is not generated for OPT 23.2 after reverting
above commits.



Note that no issue is seen on LinuxR platform for the same erlang OTP build
with FIPS option. This issue is seen only for the Windows platform while
shutting down RabbitMQ(3.8.14).



*Environment and other details :*

Erlang version :  23.2 or greater.



Referred steps to build erlang:
https://erlang.org/doc/installation_guide/INSTALL-WIN32.html



Erlang is built locally from the src package with FIPS mode enabled.



*Build details: *

*Package                          Version used while building*

Windows OS                  Microsoft Windows Server 2016 Standard

                                         Microsoft Visual Studio  2019
(14.27.29110)

                                         Microsoft Windows SDK SDK 10
(10.0.18362.0)

msys (64 bit)                  msys2-x86_64-20180531

NSIS                                 3.06.1

OpenSSL                         1.0.2u  - fips capable, loaded dynamically

JDK & wxWidget               not installed



Steps referred to build erlang is given below =>

./otp_build configure --enable-fips --enable-dynamic-ssl-lib
--with-ssl=${ERL_TOP}/openssl_libs --without-javac --without-odbc

./otp_build boot -a

./otp_build release -a

./otp_build installer_win32





Stack trace from Coredump



=====================

CONTEXT:  (.ecxr)

rax=0000000000000000 rbx=000000820ec4bd20 rcx=0000000000000000

rdx=0000000000000000 rsi=0000008209facd30 rdi=000000000001330b

rip=00007ffe120a4520 rsp=000000820b99f570 rbp=00000082099801c0

r8=0000008209facdc8  r9=0000000000000000 r10=0000000000000000

r11=00007ffe123abad0 r12=00007ffe120aa8aa r13=00007ffe120a9e5c

r14=00007ffe120a9e0b r15=0000008209facd50

iopl=0         nv up ei pl zr na po nc

cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b
efl=00010244

beam_smp!build_stacktrace+0xc40:

00007ffe`120a4520 488b11          mov     rdx,qword ptr [rcx]
ds:00000000`00000000=????????????????

Resetting default scope



FAULTING_IP:

beam_smp!build_stacktrace+c40

00007ffe`120a4520 488b11          mov     rdx,qword ptr [rcx]



EXCEPTION_RECORD:  (.exr -1)

ExceptionAddress: 00007ffe120a4520
(beam_smp!build_stacktrace+0x0000000000000c40)

   ExceptionCode: c0000005 (Access violation)

  ExceptionFlags: 00000000

NumberParameters: 2

   Parameter[0]: 0000000000000000

   Parameter[1]: 0000000000000000

Attempt to read from address 0000000000000000



DEFAULT_BUCKET_ID:  NULL_POINTER_READ

PROCESS_NAME:  erl.exe

FOLLOWUP_IP:

beam_smp!build_stacktrace+c40

00007ffe`120a4520 488b11          mov     rdx,qword ptr [rcx]



READ_ADDRESS:  0000000000000000

ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced
memory at 0x%08lx. The memory could not be %s.

EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx
referenced memory at 0x%08lx. The memory could not be %s.

EXCEPTION_CODE_STR:  c0000005

EXCEPTION_PARAMETER1:  0000000000000000

EXCEPTION_PARAMETER2:  0000000000000000

WATSON_BKT_PROCSTAMP:  60a268ce

WATSON_BKT_MODULE:  beam.smp.dll

WATSON_BKT_MODSTAMP:  60a268a0

WATSON_BKT_MODOFFSET:  4520

BUILD_VERSION_STRING:  6.3.9600.17415 (winblue_r4.141028-1500)

MODLIST_WITH_TSCHKSUM_HASH:  d0cb9d39af83f52d748e989a570501f95cbf1958

MODLIST_SHA1_HASH:  003194b5185591c6e4bed4b4f5bf621b23cda03b

NTGLOBALFLAG:  0

PROCESS_BAM_CURRENT_THROTTLED: 0

PROCESS_BAM_PREVIOUS_THROTTLED: 0

APPLICATION_VERIFIER_FLAGS:  0

PRODUCT_TYPE:  3

SUITE_MASK:  400

DUMP_FLAGS:  8000c07

DUMP_TYPE:  3

ANALYSIS_SESSION_HOST:  IGNITERS-14

ANALYSIS_SESSION_TIME:  05-21-2021 13:21:37.0090

ANALYSIS_VERSION: 10.0.18362.1 amd64fre

THREAD_ATTRIBUTES:

OS_LOCALE:  ENU



BUGCHECK_STR:  APPLICATION_FAULT_NULL_POINTER_READ_INVALID_POINTER_READ

PRIMARY_PROBLEM_CLASS:  APPLICATION_FAULT

PROBLEM_CLASSES:



    ID:     [0n313]

    Type:   [@ACCESS_VIOLATION]

    Class:  Addendum

    Scope:  BUCKET_ID

    Name:   Omit

    Data:   Omit

    PID:    [Unspecified]

    TID:    [0x19e4]

    Frame:  [0] : beam_smp!build_stacktrace



    ID:     [0n285]

    Type:   [INVALID_POINTER_READ]

    Class:  Primary

    Scope:  BUCKET_ID

    Name:   Add

    Data:   Omit

    PID:    [Unspecified]

    TID:    [0x19e4]

    Frame:  [0] : beam_smp!build_stacktrace



    ID:     [0n300]

    Type:   [NULL_POINTER_READ]

    Class:  Primary

    Scope:  DEFAULT_BUCKET_ID (Failure Bucket ID prefix)

            BUCKET_ID

    Name:   Add

    Data:   Omit

    PID:    [0x1a7c]

    TID:    [0x19e4]

    Frame:  [0] : beam_smp!build_stacktrace



LAST_CONTROL_TRANSFER:  from 00007ffe120a91d9 to 00007ffe120a4520



STACK_TEXT:

00 00000082`0b99f570 00007ffe`120a91d9 beam_smp!build_stacktrace+0xc40

01 00000082`0b99f610 00007ffe`120ce1e7 beam_smp!process_main+0x47e9

02 00000082`0b99f810 00007ffe`122c0cbd beam_smp!sched_thread_func+0x127
[C:\msys64\XXX\otp-OTP-23.2\erts\emulator\beam\erl_process.c @ 8489]

03 00000082`0b99f870 00007ffe`15c8cd70 beam_smp!thr_wrapper+0xbd
[C:\msys64\XXX\otp-OTP-23.2\erts\lib_src\win\ethread.c @ 131]

04 00000082`0b99f8c0 00007ffe`282c13d2 ucrtbase!o__realloc_base+0x60

05 00000082`0b99f8f0 00007ffe`2a9e54f4 kernel32!BaseThreadInitThunk+0x22

06 00000082`0b99f920 00000000`00000000 ntdll!RtlUserThreadStart+0x34



THREAD_SHA1_HASH_MOD_FUNC:  44df48f9269e0103c617b0eb5ee60a69f3e53b8f

THREAD_SHA1_HASH_MOD_FUNC_OFFSET:  3ab49c0509697401705cdf18f8261c0895f8bbea

THREAD_SHA1_HASH_MOD:  6d5efadb52d634f51bb798273f8ae741ccf8d747

FAULT_INSTR_CODE:  49118b48

SYMBOL_STACK_INDEX:  0

SYMBOL_NAME:  beam_smp!build_stacktrace+c40

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: beam_smp

IMAGE_NAME:  beam.smp.dll

DEBUG_FLR_IMAGE_TIMESTAMP:  60a268a0

STACK_COMMAND:  ~4s ; .ecxr ; kb

FAILURE_BUCKET_ID:  NULL_POINTER_READ_c0000005_beam.smp.dll!build_stacktrace

BUCKET_ID:
APPLICATION_FAULT_NULL_POINTER_READ_INVALID_POINTER_READ_beam_smp!build_stacktrace+c40

FAILURE_EXCEPTION_CODE:  c0000005

FAILURE_IMAGE_NAME:  beam.smp.dll



=====================


Please look into this issue and provide your suggestions as I am not able
to move forward.


Thanks,

Pooja



On Fri, Apr 23, 2021 at 9:07 PM Pooja Desai <pooja.desai10@REDACTED> wrote:

> Hi,
>
> I am seeing core in erlang (erl.exe) executable while shutting down
> rabbitMQ(3.8.14) on windows
>
> Information from crash dump (including Stack trace) as below:
>
> =====================STACK TEXT=====================================
>
> EXCEPTION_RECORD:  ffffffffffffffff -- (.exr 0xffffffffffffffff)
> .exr 0xffffffffffffffff
> ExceptionAddress: 00007fff0ab64520 (beam_smp+0x0000000000004520)
>    ExceptionCode: c0000005 (Access violation)
>   ExceptionFlags: 00000000
> NumberParameters: 2
>    Parameter[0]: 0000000000000000
>    Parameter[1]: 0000000000000000
> Attempt to read from address 0000000000000000
> FAULTING_THREAD:  00000000000002dc
> PROCESS_NAME:  erl.exe
> ADDITIONAL_DEBUG_TEXT:
>
> Use '!findthebuild' command to search for the target build information.
> If the build information is available, run '!findthebuild -s ; .reload' to
> set symbol path and load symbols.
> FAULTING_MODULE: 00007fff2c5b0000 ntdll
> DEBUG_FLR_IMAGE_TIMESTAMP:  6070400f
> ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced
> memory at 0x%08lx. The memory could not be %s.
> EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx
> referenced memory at 0x%08lx. The memory could not be %s.
> EXCEPTION_PARAMETER1:  0000000000000000
> EXCEPTION_PARAMETER2:  0000000000000000
> READ_ADDRESS:  0000000000000000
>
> FOLLOWUP_IP:
> beam_smp+4520
> 00007fff`0ab64520 488b11          mov     rdx,qword ptr [rcx]
> MOD_LIST: <ANALYSIS/>
> BUGCHECK_STR:  APPLICATION_FAULT_NULL_POINTER_READ_WRONG_SYMBOLS
> PRIMARY_PROBLEM_CLASS:  NULL_POINTER_READ
> DEFAULT_BUCKET_ID:  NULL_POINTER_READ
> LAST_CONTROL_TRANSFER:  from 00007fff0ab691d9 to 00007fff0ab64520
> STACK_TEXT:
> 0000003e`8f43f420 00007fff`0ab691d9 : 0000003e`8d699340 0000003e`91705610
> 0000003e`925135f0 0000003e`8d591dc0 : beam_smp+0x4520
> 0000003e`8f43f4c0 00007fff`0ab8e2b7 : 0000003e`8df802c0 0000003e`8d442240
> 00000000`00000000 00007fff`0ad803bd : beam_smp+0x91d9
> 0000003e`8f43f6c0 00007fff`0ad81f2d : 0000003e`8cc1dbd0 00000000`00000000
> 0000003e`8cc1dbd0 00000000`00000000 : beam_smp+0x2e2b7
> 0000003e`8f43f720 00007fff`19dfcd70 : 00000000`00000000 0000003e`8df801c0
> 00000000`00000000 00000000`00000000 : beam_smp!sys_primitive_init+0x5bb1d
> 0000003e`8f43f770 00007fff`2ba713d2 : 00000000`00000000 00007fff`19dfcd30
> 00000000`00000000 00000000`00000000 : ucrtbase!o__realloc_base+0x60
> 0000003e`8f43f7a0 00007fff`2c5c54f4 : 00007fff`2ba713b0 00000000`00000000
> 00000000`00000000 00000000`00000000 : KERNEL32!BaseThreadInitThunk+0x22
> 0000003e`8f43f7d0 00000000`00000000 : 00000000`00000000 00000000`00000000
> 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x34
> SYMBOL_STACK_INDEX:  0
> SYMBOL_NAME:  beam_smp+4520
> FOLLOWUP_NAME:  MachineOwner
> MODULE_NAME: beam_smp
> STACK_COMMAND:  ~4s ; kb
> BUCKET_ID:  WRONG_SYMBOLS
> IMAGE_NAME:  C:\PROGRA~1\xxxxxxx\erlang\erts-11.2\bin\beam.smp.dll
> =====================STACK TEXT=====================================
>
> Additional information:
>
> I built erlang with below mentioned environment:
>
> Packages                    Version used while building
> Windows OS               -  Microsoft Windows Server 2016 Standard
> Microsoft Visual Studio  -  2019 (14.27.29110)
> Microsoft Windows SDK    -  SDK 10 (10.0.18362.0)
> msys (64 bit)            -  msys2-x86_64-20180531
> NSIS                     -  3.06.1
> OpenSSL                  -  1.0.2u  - fips capable, loaded dynamically
> JDK & wxWidget           -  not installed
>
> Points to note:
> * On Unix platforms we are not seeing any issue with the same build &
> similar environment.
>
> Steps referred to build erlang referred from:
> https://erlang.org/doc/installation_guide/INSTALL-WIN32.html
> Also, note that OTP 23.1 built with same steps & configuration is not
> showing this above described crash.
>
> Please let me know if anyone has seen a similar issue in the past or if
> someone could provide me any pointers to solve this.
>
> Thanks,
> Pooja
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20210527/e7db223d/attachment.htm>


More information about the erlang-questions mailing list