[erlang-bugs] a crypto function call crashes VM

SUZUKI Tetsuya <>
Fri Aug 26 12:33:56 CEST 2011


BEAM crashes when the following message digest functions using
contexts in crypto module receive some binary as context.
(Environment: Erlang R14B03, Mac OS X 10.6.8 (i386, MacBook Air).

* md4_update/2
* md4_final/2
* md5_update/2
* md5_final/2
* sha_update/2
* sha_final/2
* hmac_update/2
* hmac_final/2

Conditions of the binary I tried are:

* The same length as context generated by
  context initialization functions (*_init/0 or hmac_init/2).
  For example in my environment, context length of
  md5_init/0 is 92 and sha_init/0 is 96.

* The content is reversed binary (context) generated by
  context initialization functions.

An example:

$ erl
Erlang R14B03 (erts-5.8.4) [source] [64-bit] [smp:2:2] [rq:2]
[async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.8.4  (abort with ^G)
1> C1 = crypto:sha_init().
2> C2 = list_to_binary(lists:reverse(binary_to_list(C1))).
3> crypto:sha_update(C2, "hello").
Segmentation fault

If length of context is different from one generated by
context initialization functions, BEAM does not crash:

1> C1 = crypto:sha_init().
2> C2 = list_to_binary(lists:reverse(binary_to_list(C1))).
3> C3 = list_to_binary(binary_to_list(C2) ++ "abc").
4> crypto:sha_update(C3, "hello").
** exception error: bad argument
     in function  crypto:sha_update/2
        called as

I think NIF implementation of the functions cause the crash
in otp_src_R14B03/lib/crypto/c_src/crypto.c.

For MD5, in static md5_init(), line 442:

  MD5_Init((MD5_CTX *) enif_make_new_binary(env, MD5_CTX_LEN, &ret));

In static md5_update(), line 450-451:

  MD5_CTX* new_ctx;
  ErlNifBinary ctx_bin, data_bin;
  if (!enif_inspect_binary(env, argv[0], &ctx_bin) /* here */
      || ctx_bin.size != MD5_CTX_LEN               /* here */

The md5_update() implementation uses a binary of first argument
as struct MD5_CTX with only checking length of the binary and MD5_CTX.
However, it is not ensured that the binary is
byte representation of struct MD5_CTX.
Other digest functions (MD4, SHA1, HMAC) have the same problem.

If the functions communicate with BEAM using resource objects
include MD5_CTX data, BEAM may not crash.

SUZUKI Tetsuya

More information about the erlang-bugs mailing list