[erlang-questions] (ArgumentError) argument error :erlang.binary_to_term

James Fish james@REDACTED
Sat Jan 28 13:52:45 CET 2017 

term_to_binary only uses zlib compression when the :compressed option is
used, and then it would start with <<131, 80, ...>>. The binary starts with
<<131, 116, ...>>, which is an uncompressed map in external term format (as
expected). I think Daniel gave the correct suggestion, especially if the
cutoff is 1460 bytes. This is the default internal buffer size for gen_tcp
sockets and so the maximum number of bytes received using an active socket
with  `packet: :raw` (the default).

On 28 January 2017 at 12:42, Michel Boaventura <michel.boaventura@REDACTED>
wrote:

> I do think it's related with zlib, since he is calling term_to_binary,
> which uses it by default.
>
>
> Em 28 de jan de 2017 06:19, "Dániel Szoboszlay" <dszoboszlay@REDACTED>
> escreveu:
>
> I don't think this would be zlib related. TCP is simply a stream protocol:
> you put in a stream of bytes in arbitrary chunks, and on the other end you
> receive the same sequence of bytes, but possibly in different chunks. If
> you want to send a sequence of messages, it is up to the application layer
> to detect the message boundaries.
>
> You can use e.g. the {packet, 4} option when opening the TCP socket (on
> both ends), so Erlang will add a 4-byte size header to every message you
> send, and on the receiving end it will wait for a whole message to arrive
> before handing off the bytes to your application. This will work only up to
> 4GB messages of course.
>
> Alternatively, you may implement a streaming version of binary_to_term,
> which will tell you if you gave it too few or too many bytes to parse a
> whole term from them.
>
> Cheers,
> Daniel
>
> On Sat, 28 Jan 2017 at 08:12 José Valim <jose.valim@REDACTED>
> wrote:
>
>> What is your zlib version? There has been a thread about a very similar
>> issue and it was pinpointed to be related to the zlib version in your
>> system: http://erlang.org/pipermail/erlang-questions/2017-
>> January/091434.html
>>
>>
>>
>> *José Valim*
>> www.plataformatec.com.br
>> Skype: jv.ptec
>> Founder and Director of R&D
>>
>> On Sat, Jan 28, 2017 at 1:55 AM, Kevin Johnson <
>> johnson786.kevin@REDACTED> wrote:
>>
>> Hi,
>>
>> I am using Elixir to send data using :gen_tcp. The syntax presented here
>> may be elementary Elixir syntax, however the issue is specifically related
>> to the usage of term_to_binary and binary_to_term conversions over gen_tcp.
>>
>> This is the command I used on the client side:
>>
>> :gen_tcp.send(client_socket, :erlang.term_to_binary(data))
>>
>>
>> This is the command I use on the server side:
>>
>>   defp handle_data(data) do
>>     data
>>     |> :erlang.binary_to_term
>>   end
>>
>>
>> When my data gets received on the server side, it seems that at a certain
>> size threshold that it errors out like this:
>>
>> pry(6)> [error] GenServer #PID<0.502.0> terminating
>>
>> ** (ArgumentError) argument error
>>
>>  :erlang.binary_to_term(<<131, 116, 0, 0, 0, 28, 109, 0, 0, 0, 5, 66, 49,
>> 95, 81, 49, 108, 0, 0, 0, 1, 116, 0, 0, 0, 1, 109, 0, 0, 0, 4, 116, 101,
>> 120, 116, 109, 0, 0, 0, 11, 84, 69, 88, 84, 32, 65, 78, 83, 87, 69, ...>>)
>>
>> (stdlib) gen_server.erl:601: :gen_server.try_dispatch/4
>>
>> (stdlib) gen_server.erl:667: :gen_server.handle_msg/5
>>
>> (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
>>
>> Last message: {:tcp, #Port<0.10115>, <<131, 116, 0, 0, 0, 28, 109, 0, 0,
>> 0, 5, 66, 49, 95, 81, 49, 108, 0, 0, 0, 1, 116, 0, 0, 0, 1, 109, 0, 0, 0,
>> 4, 116, 101, 120, 116, 109, 0, 0, 0, 11, 84, 69, 88, 84, 32, 65, 78, 83,
>> ...>>}
>>
>>
>> Here is some example data that will just pass fine:
>> %{"B2_Q18" => [%{"choice_id" => "B2_Q18_C1"}], "B3_B9_Q1" =>
>> [%{"choice_id" => "B3_B9_Q1_C1"}], "B2_Q7" => [%{"choice_id" =>
>> "B2_Q7_C1"}], "B5_B1_Q8" => [%{"choice_id" => "B5_B1_Q8_C1"}], "B3_B6_Q1"
>> => [%{"choice_id" => "B3_B6_Q1_C1"}], "B5_B2_Q5" => [%{"choice_id" =>
>> "B5_B2_Q5_C1"}, %{"choice_id" => "B5_B2_Q5_C2"}], "B3_B7_Q1" =>
>> [%{"choice_id" => "B3_B7_Q1_C1"}], "B3_B4_Q1" => [%{"choice_id" =>
>> "B3_B4_Q1_C1"}], "B2_Q16" => [%{"choice_id" => "B2_Q16_C1"}], "B2_Q20" =>
>> [%{"choice_id" => "B2_Q20_C1"}], "B2_Q5" => [%{"choice_id" => "B2_Q5_C1"}],
>> "B2_Q9" => [%{"choice_id" => "B2_Q9_C1"}], "B2_Q17" => [%{"choice_id" =>
>> "B2_Q17_C1"}], "B3_B2_Q1" => [%{"choice_id" => "B3_B2_Q1_C1"}], "B5_B2_Q3"
>> => [%{"choice_id" => "B5_B2_Q3_C1"}], "B3_B3_Q1" => [%{"choice_id" =>
>> "B3_B3_Q1_C1"}], "B5_B1_Q1" => [%{"text" => "TEXT ANSWER"}], "B2_Q14" =>
>> [%{"choice_id" => "B2_Q14_C1"}], "B5_B1_Q3" => [%{"choice_id" =>
>> "B5_B1_Q3_C22"}], "B5_B2_Q2" => [%{"choice_id" => "B5_B2_Q2_C1"}], "B2_Q4"
>> => [%{"choice_id" => "B2_Q4_C1"}], "B2_Q10" => [%{"choice_id" =>
>> "B2_Q10_C1"}], "B4_Q1" => [%{"choice_id" => "B4_Q1_C1"}, %{"choice_id" =>
>> "B4_Q1_C10"}], "B5_B1_Q6" => [%{"choice_id" => "B5_B1_Q6_C1"}], "B2_Q6" =>
>> [%{"choice_id" => "B2_Q6_C1"}], "B2_Q2" => [%{"choice_id" => "B2_Q2_C1"}],
>> "B2_Q1" => [%{"choice_id" => "B2_Q1_C1"}]}
>>
>> The moment I make it slightly longer, by like appending an extra key "a"
>> => "b" it will give me the above error.
>>
>> It seems to be that the size of the keys plays an issue here, because if
>> I convert all the above "choice_id" keys to a mere "c", then even with the
>> extra entry "a" => "b" included like following below will work just fine:
>> %{"B2_Q18" => [%{"c" => "B2_Q18_C1"}], "B3_B9_Q1" => [%{"c" =>
>> "B3_B9_Q1_C1"}], "B2_Q7" => [%{"c" => "B2_Q7_C1"}], "B5_B1_Q8" => [%{"c" =>
>> "B5_B1_Q8_C1"}], "B3_B6_Q1" => [%{"c" => "B3_B6_Q1_C1"}], "B5_B2_Q5" =>
>> [%{"c" => "B5_B2_Q5_C1"}, %{"c" => "B5_B2_Q5_C2"}], "B3_B7_Q1" => [%{"c" =>
>> "B3_B7_Q1_C1"}], "B3_B4_Q1" => [%{"c" => "B3_B4_Q1_C1"}], "B2_Q16" =>
>> [%{"c" => "B2_Q16_C1"}], "B2_Q20" => [%{"c" => "B2_Q20_C1"}], "B2_Q5" =>
>> [%{"c" => "B2_Q5_C1"}], "B2_Q9" => [%{"c" => "B2_Q9_C1"}], "B2_Q17" =>
>> [%{"c" => "B2_Q17_C1"}], "B3_B2_Q1" => [%{"c" => "B3_B2_Q1_C1"}],
>> "B5_B2_Q3" => [%{"c" => "B5_B2_Q3_C1"}], "B3_B3_Q1" => [%{"c" =>
>> "B3_B3_Q1_C1"}], "B5_B1_Q1" => [%{"text" => "TEXT ANSWER"}], "B2_Q14" =>
>> [%{"c" => "B2_Q14_C1"}], "B5_B1_Q3" => [%{"c" => "B5_B1_Q3_C22"}],
>> "B5_B2_Q2" => [%{"c" => "B5_B2_Q2_C1"}], "B2_Q4" => [%{"c" => "B2_Q4_C1"}],
>> "B2_Q10" => [%{"c" => "B2_Q10_C1"}], "B4_Q1" => [%{"c" => "B4_Q1_C1"},
>> %{"c" => "B4_Q1_C10"}], "B5_B1_Q6" => [%{"c" => "B5_B1_Q6_C1"}], "B2_Q6" =>
>> [%{"c" => "B2_Q6_C1"}], "B2_Q2" => [%{"c" => "B2_Q2_C1"}], "B2_Q1" =>
>> [%{"c" => "B2_Q1_C1"}], "a" => "b"}
>>
>> The following on the other hand will just fail:
>> data = %{"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
>> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
>> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
>> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
>> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
>> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
>> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
>> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
>> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
>> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
>> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
>> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
>> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
>> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
>> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
>> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
>> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
>> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
>> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
>> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
>> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" => "q",
>> "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
>> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
>> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
>> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
>> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
>> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
>> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
>> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
>> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
>> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
>> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
>> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
>> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
>> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
>> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
>> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
>> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
>> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
>> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
>> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
>> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
>> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
>> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" => "q",
>> "ccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc
>> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc
>> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc
>> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc
>> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc
>> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc
>> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc
>> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc
>> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc
>> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc
>> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc
>> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc
>> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc
>> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc
>> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc
>> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc
>> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc
>> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc
>> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc
>> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc
>> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc
>> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc
>> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc
>> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc
>> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc
>> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc
>> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc"
>> => "q"}
>> :gen_tcp.send(client_socket, :erlang.term_to_binary(data))
>>
>> The only issue that I was able to track down here is a conjunction of two
>> things:
>> 1) A limit to the size of total key length inside a map/hash
>> 2) sending that hash as a binary over tcp and converting back to term
>>
>> In all of the above cases, a direct conversion of :erlang.term_to_binary
>> followed by :erlang.binary_to_term works just fine without any issues. The
>> issue only comes about after that binary was send over tcp and then
>> :erlang.binary_to_term is attempted.
>>
>> I would greatly appreciate if anyone can provide me guidance in this
>> matter, point out to me any relevant documentation that officially states
>> any of these constraints if they are in fact constraints, and if they are
>> not supposed to be constraints then kindly instruct me what additional
>> information may be needed for me to get to the bottom of this.
>>
>> Thank you.
>>
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questions@REDACTED
>> http://erlang.org/mailman/listinfo/erlang-questions
>>
>>
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questions@REDACTED
>> http://erlang.org/mailman/listinfo/erlang-questions
>>
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
>
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20170128/6b2ec040/attachment.htm>

More information about the erlang-questions mailing list