[erlang-bugs] a crypto function call crashes VM

[SUZUKI Tetsuya]

Hi,

BEAM crashes when the following message digest functions using
contexts in crypto module receive some binary as context.
(Environment: Erlang R14B03, Mac OS X 10.6.8 (i386, MacBook Air).

* md4_update/2
* md4_final/2
* md5_update/2
* md5_final/2
* sha_update/2
* sha_final/2
* hmac_update/2
* hmac_final/2

Conditions of the binary I tried are:

* The same length as context generated by
  context initialization functions (*_init/0 or hmac_init/2).
  For example in my environment, context length of
  md5_init/0 is 92 and sha_init/0 is 96.

* The content is reversed binary (context) generated by
  context initialization functions.

An example:

$ erl
Erlang R14B03 (erts-5.8.4) [source] [64-bit] [smp:2:2] [rq:2]
[async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.8.4  (abort with ^G)
1> C1 = crypto:sha_init().
<<1,35,69,103,137,171,205,239,254,220,186,152,118,84,50,
  16,240,225,210,195,0,0,0,0,0,0,0,0,114,...>>
2> C2 = list_to_binary(lists:reverse(binary_to_list(C1))).
<<0,0,0,0,213,8,1,216,13,25,10,215,13,17,10,64,23,213,13,
  35,59,35,213,13,48,130,19,217,13,...>>
3> crypto:sha_update(C2, "hello").
Segmentation fault

If length of context is different from one generated by
context initialization functions, BEAM does not crash:

1> C1 = crypto:sha_init().
<<1,35,69,103,137,171,205,239,254,220,186,152,118,84,50,
  16,240,225,210,195,0,0,0,0,0,0,0,0,19,...>>
2> C2 = list_to_binary(lists:reverse(binary_to_list(C1))).
<<0,0,0,0,3,3,67,240,80,5,125,67,99,77,45,71,72,1,72,77,
  61,3,67,64,242,67,71,77,43,...>>
3> C3 = list_to_binary(binary_to_list(C2) ++ "abc").
<<0,0,0,0,3,3,67,240,80,5,125,67,99,77,45,71,72,1,72,77,
  61,3,67,64,242,67,71,77,43,...>>
4> crypto:sha_update(C3, "hello").
** exception error: bad argument
     in function  crypto:sha_update/2
        called as
crypto:sha_update(<<0,0,0,0,3,3,67,240,80,5,125,67,99,77,45,71,
                                      72,1,72,77,61,3,67,64,242,67,71,77,...>>,
                                    "hello")


I think NIF implementation of the functions cause the crash
in otp_src_R14B03/lib/crypto/c_src/crypto.c.

For MD5, in static md5_init(), line 442:

  MD5_Init((MD5_CTX *) enif_make_new_binary(env, MD5_CTX_LEN, &ret));

In static md5_update(), line 450-451:

  MD5_CTX* new_ctx;
  ErlNifBinary ctx_bin, data_bin;
  ERL_NIF_TERM ret;
  if (!enif_inspect_binary(env, argv[0], &ctx_bin) /* here */
      || ctx_bin.size != MD5_CTX_LEN               /* here */
  ...

The md5_update() implementation uses a binary of first argument
as struct MD5_CTX with only checking length of the binary and MD5_CTX.
However, it is not ensured that the binary is
byte representation of struct MD5_CTX.
Other digest functions (MD4, SHA1, HMAC) have the same problem.

If the functions communicate with BEAM using resource objects
include MD5_CTX data, BEAM may not crash.

---
SUZUKI Tetsuya