[erlang-questions] (ArgumentError) argument error :erlang.binary_to_term

Sat Jan 28 09:19:19 CET 2017

I don't think this would be zlib related. TCP is simply a stream protocol:
you put in a stream of bytes in arbitrary chunks, and on the other end you
receive the same sequence of bytes, but possibly in different chunks. If
you want to send a sequence of messages, it is up to the application layer
to detect the message boundaries.

You can use e.g. the {packet, 4} option when opening the TCP socket (on
both ends), so Erlang will add a 4-byte size header to every message you
send, and on the receiving end it will wait for a whole message to arrive
before handing off the bytes to your application. This will work only up to
4GB messages of course.

Alternatively, you may implement a streaming version of binary_to_term,
which will tell you if you gave it too few or too many bytes to parse a
whole term from them.

Cheers,
Daniel

On Sat, 28 Jan 2017 at 08:12 José Valim <jose.valim@REDACTED>
wrote:

> What is your zlib version? There has been a thread about a very similar
> issue and it was pinpointed to be related to the zlib version in your
> system:
> http://erlang.org/pipermail/erlang-questions/2017-January/091434.html
>
>
>
> *José Valim*
> www.plataformatec.com.br
> Skype: jv.ptec
> Founder and Director of R&D
>
> On Sat, Jan 28, 2017 at 1:55 AM, Kevin Johnson <johnson786.kevin@REDACTED
> > wrote:
>
> Hi,
>
> I am using Elixir to send data using :gen_tcp. The syntax presented here
> may be elementary Elixir syntax, however the issue is specifically related
> to the usage of term_to_binary and binary_to_term conversions over gen_tcp.
>
> This is the command I used on the client side:
>
> :gen_tcp.send(client_socket, :erlang.term_to_binary(data))
>
>
> This is the command I use on the server side:
>
>   defp handle_data(data) do
>     data
>     |> :erlang.binary_to_term
>   end
>
>
> When my data gets received on the server side, it seems that at a certain
> size threshold that it errors out like this:
>
> pry(6)> [error] GenServer #PID<0.502.0> terminating
>
> ** (ArgumentError) argument error
>
>  :erlang.binary_to_term(<<131, 116, 0, 0, 0, 28, 109, 0, 0, 0, 5, 66, 49,
> 95, 81, 49, 108, 0, 0, 0, 1, 116, 0, 0, 0, 1, 109, 0, 0, 0, 4, 116, 101,
> 120, 116, 109, 0, 0, 0, 11, 84, 69, 88, 84, 32, 65, 78, 83, 87, 69, ...>>)
>
> (stdlib) gen_server.erl:601: :gen_server.try_dispatch/4
>
> (stdlib) gen_server.erl:667: :gen_server.handle_msg/5
>
> (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
>
> Last message: {:tcp, #Port<0.10115>, <<131, 116, 0, 0, 0, 28, 109, 0, 0,
> 0, 5, 66, 49, 95, 81, 49, 108, 0, 0, 0, 1, 116, 0, 0, 0, 1, 109, 0, 0, 0,
> 4, 116, 101, 120, 116, 109, 0, 0, 0, 11, 84, 69, 88, 84, 32, 65, 78, 83,
> ...>>}
>
>
> Here is some example data that will just pass fine:
> %{"B2_Q18" => [%{"choice_id" => "B2_Q18_C1"}], "B3_B9_Q1" =>
> [%{"choice_id" => "B3_B9_Q1_C1"}], "B2_Q7" => [%{"choice_id" =>
> "B2_Q7_C1"}], "B5_B1_Q8" => [%{"choice_id" => "B5_B1_Q8_C1"}], "B3_B6_Q1"
> => [%{"choice_id" => "B3_B6_Q1_C1"}], "B5_B2_Q5" => [%{"choice_id" =>
> "B5_B2_Q5_C1"}, %{"choice_id" => "B5_B2_Q5_C2"}], "B3_B7_Q1" =>
> [%{"choice_id" => "B3_B7_Q1_C1"}], "B3_B4_Q1" => [%{"choice_id" =>
> "B3_B4_Q1_C1"}], "B2_Q16" => [%{"choice_id" => "B2_Q16_C1"}], "B2_Q20" =>
> [%{"choice_id" => "B2_Q20_C1"}], "B2_Q5" => [%{"choice_id" => "B2_Q5_C1"}],
> "B2_Q9" => [%{"choice_id" => "B2_Q9_C1"}], "B2_Q17" => [%{"choice_id" =>
> "B2_Q17_C1"}], "B3_B2_Q1" => [%{"choice_id" => "B3_B2_Q1_C1"}], "B5_B2_Q3"
> => [%{"choice_id" => "B5_B2_Q3_C1"}], "B3_B3_Q1" => [%{"choice_id" =>
> "B3_B3_Q1_C1"}], "B5_B1_Q1" => [%{"text" => "TEXT ANSWER"}], "B2_Q14" =>
> [%{"choice_id" => "B2_Q14_C1"}], "B5_B1_Q3" => [%{"choice_id" =>
> "B5_B1_Q3_C22"}], "B5_B2_Q2" => [%{"choice_id" => "B5_B2_Q2_C1"}], "B2_Q4"
> => [%{"choice_id" => "B2_Q4_C1"}], "B2_Q10" => [%{"choice_id" =>
> "B2_Q10_C1"}], "B4_Q1" => [%{"choice_id" => "B4_Q1_C1"}, %{"choice_id" =>
> "B4_Q1_C10"}], "B5_B1_Q6" => [%{"choice_id" => "B5_B1_Q6_C1"}], "B2_Q6" =>
> [%{"choice_id" => "B2_Q6_C1"}], "B2_Q2" => [%{"choice_id" => "B2_Q2_C1"}],
> "B2_Q1" => [%{"choice_id" => "B2_Q1_C1"}]}
>
> The moment I make it slightly longer, by like appending an extra key "a"
> => "b" it will give me the above error.
>
> It seems to be that the size of the keys plays an issue here, because if I
> convert all the above "choice_id" keys to a mere "c", then even with the
> extra entry "a" => "b" included like following below will work just fine:
> %{"B2_Q18" => [%{"c" => "B2_Q18_C1"}], "B3_B9_Q1" => [%{"c" =>
> "B3_B9_Q1_C1"}], "B2_Q7" => [%{"c" => "B2_Q7_C1"}], "B5_B1_Q8" => [%{"c" =>
> "B5_B1_Q8_C1"}], "B3_B6_Q1" => [%{"c" => "B3_B6_Q1_C1"}], "B5_B2_Q5" =>
> [%{"c" => "B5_B2_Q5_C1"}, %{"c" => "B5_B2_Q5_C2"}], "B3_B7_Q1" => [%{"c" =>
> "B3_B7_Q1_C1"}], "B3_B4_Q1" => [%{"c" => "B3_B4_Q1_C1"}], "B2_Q16" =>
> [%{"c" => "B2_Q16_C1"}], "B2_Q20" => [%{"c" => "B2_Q20_C1"}], "B2_Q5" =>
> [%{"c" => "B2_Q5_C1"}], "B2_Q9" => [%{"c" => "B2_Q9_C1"}], "B2_Q17" =>
> [%{"c" => "B2_Q17_C1"}], "B3_B2_Q1" => [%{"c" => "B3_B2_Q1_C1"}],
> "B5_B2_Q3" => [%{"c" => "B5_B2_Q3_C1"}], "B3_B3_Q1" => [%{"c" =>
> "B3_B3_Q1_C1"}], "B5_B1_Q1" => [%{"text" => "TEXT ANSWER"}], "B2_Q14" =>
> [%{"c" => "B2_Q14_C1"}], "B5_B1_Q3" => [%{"c" => "B5_B1_Q3_C22"}],
> "B5_B2_Q2" => [%{"c" => "B5_B2_Q2_C1"}], "B2_Q4" => [%{"c" => "B2_Q4_C1"}],
> "B2_Q10" => [%{"c" => "B2_Q10_C1"}], "B4_Q1" => [%{"c" => "B4_Q1_C1"},
> %{"c" => "B4_Q1_C10"}], "B5_B1_Q6" => [%{"c" => "B5_B1_Q6_C1"}], "B2_Q6" =>
> [%{"c" => "B2_Q6_C1"}], "B2_Q2" => [%{"c" => "B2_Q2_C1"}], "B2_Q1" =>
> [%{"c" => "B2_Q1_C1"}], "a" => "b"}
>
> The following on the other hand will just fail:
> data =
> %{"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
> => "q",
> "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
> => "q",
> "ccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc"
> => "q"}
> :gen_tcp.send(client_socket, :erlang.term_to_binary(data))
>
> The only issue that I was able to track down here is a conjunction of two
> things:
> 1) A limit to the size of total key length inside a map/hash
> 2) sending that hash as a binary over tcp and converting back to term
>
> In all of the above cases, a direct conversion of :erlang.term_to_binary
> followed by :erlang.binary_to_term works just fine without any issues. The
> issue only comes about after that binary was send over tcp and then
> :erlang.binary_to_term is attempted.
>
> I would greatly appreciate if anyone can provide me guidance in this
> matter, point out to me any relevant documentation that officially states
> any of these constraints if they are in fact constraints, and if they are
> not supposed to be constraints then kindly instruct me what additional
> information may be needed for me to get to the bottom of this.
>
> Thank you.
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20170128/75392081/attachment.htm>