From mikpe@REDACTED Sun Feb 1 16:21:41 2009 From: mikpe@REDACTED (Mikael Pettersson) Date: Sun, 1 Feb 2009 16:21:41 +0100 Subject: [erlang-patches] Remove executable stacks (HiPE/amd64) In-Reply-To: <18817.32964.133101.701435@harpo.it.uu.se> References: <2da21fe50901290139x5df2afccq3ef22bd8d9c04c7@mail.gmail.com> <18817.32964.133101.701435@harpo.it.uu.se> Message-ID: <18821.48645.897539.732198@harpo.it.uu.se> Mikael Pettersson writes: > Davide Pesavento writes: > > Hi, > > the attached patch fixes executable stacks in HiPE asm code for amd64. > > I guess a similar patch can easily be applied to the other > > architectures. > > > > References: > > [1] https://bugs.gentoo.org/show_bug.cgi?id=249328 > > [2] http://www.erlang.org/pipermail/erlang-bugs/2008-December/001125.html > > [3] http://www.gentoo.org/proj/en/hardened/gnu-stack.xml > > Thanks, this patch is what I ([2] above) suspected was needed. > > Note that HiPE never has used executable stacks, all this does is > annotate some assembly file modules so that the build tools don't > mark them as needing executable stack. > > This will be included in the R13B release, extended to cover all > architectures not just amd64. Here is the corresponding patch committed to HiPE CVS, extended to amd64, arm, ppc32, sparc32, and x86. Applies cleanly to R12B-5. If you intend to maintain an R12B-5 based package I also advise you to include the following bug fix: /Mikael The HiPE group --- otp-0125/erts/emulator/hipe/hipe_amd64_bifs.m4.~1~ 2008-10-27 20:07:08.000000000 +0100 +++ otp-0125/erts/emulator/hipe/hipe_amd64_bifs.m4 2009-02-01 11:36:47.000000000 +0100 @@ -531,3 +531,7 @@ define(gc_bif_interface_2,`standard_bif_ define(gc_nofail_primop_interface_1,`nofail_primop_interface_1($1, $2)') include(`hipe/hipe_bif_list.m4') + +`#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif' --- otp-0125/erts/emulator/hipe/hipe_amd64_glue.S.~1~ 2008-11-07 23:44:11.000000000 +0100 +++ otp-0125/erts/emulator/hipe/hipe_amd64_glue.S 2009-02-01 11:36:47.000000000 +0100 @@ -418,3 +418,7 @@ nbif_inc_stack_0: LOAD_ARG_REGS SWITCH_C_TO_ERLANG_QUICK NSP_RET0 + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif --- otp-0125/erts/emulator/hipe/hipe_arm_bifs.m4.~1~ 2008-10-27 20:07:08.000000000 +0100 +++ otp-0125/erts/emulator/hipe/hipe_arm_bifs.m4 2009-02-01 11:37:21.000000000 +0100 @@ -525,3 +525,7 @@ $1: #endif') include(`hipe/hipe_bif_list.m4') + +`#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif' --- otp-0125/erts/emulator/hipe/hipe_arm_glue.S.~1~ 2008-11-07 23:44:11.000000000 +0100 +++ otp-0125/erts/emulator/hipe/hipe_arm_glue.S 2009-02-01 11:38:25.000000000 +0100 @@ -392,3 +392,7 @@ hipe_arm_inc_stack: LOAD_ARG_REGS # this relies on LOAD_ARG_REGS not clobbering TEMP_ARG0 mov pc, TEMP_ARG0 + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif --- otp-0125/erts/emulator/hipe/hipe_ppc_bifs.m4.~1~ 2008-10-27 20:07:09.000000000 +0100 +++ otp-0125/erts/emulator/hipe/hipe_ppc_bifs.m4 2009-02-01 11:36:47.000000000 +0100 @@ -544,3 +544,7 @@ ASYM($1): #endif') include(`hipe/hipe_bif_list.m4') + +`#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif' --- otp-0125/erts/emulator/hipe/hipe_ppc_glue.S.~1~ 2008-11-07 23:44:12.000000000 +0100 +++ otp-0125/erts/emulator/hipe/hipe_ppc_glue.S 2009-02-01 11:36:47.000000000 +0100 @@ -557,3 +557,7 @@ ASYM(hipe_ppc_inc_stack): LOAD NSP, P_NSP(P) LOAD_ARG_REGS blr + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif --- otp-0125/erts/emulator/hipe/hipe_sparc_bifs.m4.~1~ 2008-10-27 20:07:09.000000000 +0100 +++ otp-0125/erts/emulator/hipe/hipe_sparc_bifs.m4 2009-02-01 11:36:47.000000000 +0100 @@ -554,3 +554,7 @@ $1: #endif') include(`hipe/hipe_bif_list.m4') + +`#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif' --- otp-0125/erts/emulator/hipe/hipe_sparc_glue.S.~1~ 2008-11-07 23:44:12.000000000 +0100 +++ otp-0125/erts/emulator/hipe/hipe_sparc_glue.S 2009-02-01 11:36:47.000000000 +0100 @@ -423,3 +423,7 @@ hipe_sparc_inc_stack: /* this relies on LOAD_ARG_REGS not clobbering TEMP_ARG0 */ jmp TEMP_ARG0+8 ld [P+P_NSP], NSP /* delay slot */ + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif --- otp-0125/erts/emulator/hipe/hipe_x86_bifs.m4.~1~ 2008-10-27 20:07:09.000000000 +0100 +++ otp-0125/erts/emulator/hipe/hipe_x86_bifs.m4 2009-02-01 11:36:47.000000000 +0100 @@ -611,3 +611,7 @@ define(gc_bif_interface_2,`standard_bif_ define(gc_nofail_primop_interface_1,`nofail_primop_interface_1($1, $2)') include(`hipe/hipe_bif_list.m4') + +`#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif' --- otp-0125/erts/emulator/hipe/hipe_x86_glue.S.~1~ 2008-11-07 23:44:12.000000000 +0100 +++ otp-0125/erts/emulator/hipe/hipe_x86_glue.S 2009-02-01 11:36:47.000000000 +0100 @@ -395,3 +395,7 @@ ASYM(nbif_inc_stack_0): LOAD_CALLER_SAVE SWITCH_C_TO_ERLANG_QUICK NSP_RET0 + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif From raimo+erlang-patches@REDACTED Thu Feb 5 09:35:01 2009 From: raimo+erlang-patches@REDACTED (Raimo Niskanen) Date: Thu, 5 Feb 2009 09:35:01 +0100 Subject: [erlang-patches] fread failing to handle newlines gracefully In-Reply-To: <20090130121223.GA11448@hezmatt.org> References: <20090130121223.GA11448@hezmatt.org> Message-ID: <20090205083501.GA11769@erix.ericsson.se> On Fri, Jan 30, 2009 at 11:12:23PM +1100, Matthew Palmer wrote: > Hi, > > I recently fielded a question on Stack Overflow regarding some strange > behaviour of fread, with the ~d format spec > (http://stackoverflow.com/questions/473327/unexpected-behavior-of-iofread-in-erlang/490023#490023). > Digging deep, it appears to be a bit of a bug in the way that newlines are > handled in the format parser, as demonstrated here: > > 3> io_lib_fread:fread([], "10 11\n12 13 14\n", "~d"). > {done,{ok,"\n"}," 1112 13 14\n"} > > It's eaten the newline between "11" and "12", which means that on the next > pass through it'll read the next number as "1112" instead of "11", which > poses some obvious problems. > > I've come up with the attached patch to fix the problem. I'm not very > experienced with Erlang yet, so it might not be the cleanest way to do it, > but it does definitely fix the problem, and without (apparently) breaking > any other common use case. Comments appreciated. If there's a test suite I > should be adding to as well, I'm more than happy to do that, I just couldn't > find one in my source tarball. > > - Matt Thank you for your effort! Your diagnosis and patch are basically right. I have now been digging into the matter and it took a while to ensure that. I will polish the patch a bit (the code needs some cleanup), and improve it for \r\n combinations. Unfortunately there are no tests in the source tarball. We are striving towards their release but so far large parts of them are too much depending on our internal environment to be globally useful... I will write tests too. When done I will release a new source code patch here. > --- ../erlang/erlang-12.b.3-dfsg/lib/stdlib/src/io_lib_fread.erl 2004-09-14 21:41:00.000000000 +1000 > +++ ./io_lib_fread.erl 2009-01-30 15:44:57.000000000 +1100 > @@ -35,9 +35,9 @@ > fread_collect(MoreChars, [], Rest, RestFormat, N, Inputs). > > fread_collect([$\r|More], Stack, Rest, RestFormat, N, Inputs) -> > - fread(RestFormat, Rest ++ reverse(Stack), N, Inputs, More); > + fread(RestFormat, Rest ++ reverse(Stack), N, Inputs, [$\r|More]); > fread_collect([$\n|More], Stack, Rest, RestFormat, N, Inputs) -> > - fread(RestFormat, Rest ++ reverse(Stack), N, Inputs, More); > + fread(RestFormat, Rest ++ reverse(Stack), N, Inputs, [$\n|More]); > fread_collect([C|More], Stack, Rest, RestFormat, N, Inputs) -> > fread_collect(More, [C|Stack], Rest, RestFormat, N, Inputs); > fread_collect([], Stack, Rest, RestFormat, N, Inputs) -> > @@ -55,8 +55,8 @@ > eof -> > fread(RestFormat,eof,N,Inputs,eof); > _ -> > - %% Don't forget to count the newline. > - {more,{More,RestFormat,N+1,Inputs}} > + %% Don't forget to strip and count the newline. > + {more,{tl(More),RestFormat,N+1,Inputs}} > end; > Other -> %An error has occurred > {done,Other,More} > _______________________________________________ > erlang-patches mailing list > erlang-patches@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-patches -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From mclaughlin77@REDACTED Fri Feb 6 21:42:09 2009 From: mclaughlin77@REDACTED (Kelly McLaughlin) Date: Fri, 6 Feb 2009 13:42:09 -0700 Subject: [erlang-patches] Patch for improperly set HTTP host header in inets http client Message-ID: <208c9bd70902061242t1365f017n895df4dd5ee04633@mail.gmail.com> The inets http client does not properly set the host header when making requests to a host listening on a port other than 80. This bug was reported here: http://www.erlang.org/pipermail/erlang-bugs/2009-January/001175.html This patch appends the port number if the request is directed to a port other than 80. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: http_host_header.patch Type: application/octet-stream Size: 939 bytes Desc: not available URL: From jan@REDACTED Mon Feb 9 00:47:52 2009 From: jan@REDACTED (Jan Lehnardt) Date: Mon, 9 Feb 2009 00:47:52 +0100 Subject: [erlang-patches] Patch for packet_parser.c to Include HTTP Methods Defined in RFC 2518 / WebDAV Message-ID: <2D918F25-4E4E-479B-BBC8-407A6028F698@apache.org> Dear OTP Team, the attached patch adds additional HTTP methods to packet_parser.c. The methods are defined in RFC 2518 / WebDAV. While not part of the HTTP/1.1 RFC 2616, these methods are useful when implementing a WebDAV (or similar) server in Erlang. I work on CouchDB. We borrowed the COPY and MOVE methods (but not the others) from RFC 2518 to allow additional operations on resources on the server. Until now, a parsed packet will include a list / string as the HTTP method if the method is not defined in packet_parser.c and as an atom otherwise. CouchDB uses this code to mitigate the issue: % Non standard HTTP verbs aren't atoms (COPY, MOVE etc) so convert when % possible (if any module references the atom, then it's existing). Meth -> couch_util:to_existing_atom(Meth) couch_util:to_existing_atom/1 handles the case: % works like list_to_existing_atom, except can be list or binary and it % gives you the original value instead of an error if no existing atom. to_existing_atom(V) when is_list(V)-> try list_to_existing_atom(V) catch _ -> V end; to_existing_atom(V) when is_binary(V)-> try list_to_existing_atom(?b2l(V)) catch _ -> V end; to_existing_atom(V) when is_atom(V)-> V. CouchDB (and other implementors of HTTP extending standards) would benefit from the inclusion of the methods in RFC 2518. While CouchDB only needs COPY and MOVE (for now), adding the complete suite of additional HTTP methods seems sensible. Please consider the patch for inclusion into the next OTP release. The patch is made against the preview release R13A but it equally applies to R12B-5. Cheers Jan -- --- a/erts/emulator/beam/packet_parser.c 2009-02-09 00:14:24.000000000 +0100 +++ b/erts/emulator/beam/packet_parser.c 2009-02-09 00:12:56.000000000 +0100 @@ -139,6 +139,13 @@ "PUT", "DELETE", "TRACE", + "PROPFIND", + "PROPPATCH", + "MKCOL", + "COPY", + "MOVE", + "LOCK", + "UNLOCK", NULL }; From ingela@REDACTED Mon Feb 9 10:06:07 2009 From: ingela@REDACTED (Ingela Anderton Andin) Date: Mon, 09 Feb 2009 10:06:07 +0100 Subject: [erlang-patches] Patch for improperly set HTTP host header in inets http client In-Reply-To: <208c9bd70902061242t1365f017n895df4dd5ee04633@mail.gmail.com> References: <208c9bd70902061242t1365f017n895df4dd5ee04633@mail.gmail.com> Message-ID: <498FF1FF.2060200@erix.ericsson.se> Hi thank you for the patch I will add it to the upcoming release. Ingela Andin Erlang/OTP - Ericsson Kelly McLaughlin wrote: > The inets http client does not properly set the host header when > making requests to a host listening on a port other than 80. This bug > was reported here: > http://www.erlang.org/pipermail/erlang-bugs/2009-January/001175.html > > This patch appends the port number if the request is directed to a > port other than 80. > ------------------------------------------------------------------------ > > _______________________________________________ > erlang-patches mailing list > erlang-patches@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-patches From michal.ptaszek@REDACTED Mon Feb 9 11:10:31 2009 From: michal.ptaszek@REDACTED (Michal Ptaszek) Date: Mon, 9 Feb 2009 10:10:31 +0000 (GMT) Subject: [erlang-patches] Patch for PUT and DELETE inets HTTP methods support In-Reply-To: <23184321.7521234174182450.JavaMail.root@zimbra> Message-ID: <5912359.7541234174231359.JavaMail.root@zimbra> Hi, In the times of RESTful addressing style it is a very inconvenient situation where when one wants to remove something on the server side must use 'POST' method instead of simple 'DELETE'. Are there any plans to widen the set of accepted HTTP methods? As far as I have understood the inets source code, RFC (http://www.w3.org/Protocols/rfc2616/rfc2616.html) and the patched server responses, those few lines I am attaching should solve the problem. Best regards, -- Michal Ptaszek www.erlang-consulting.com -------------- next part -------------- A non-text attachment was scrubbed... Name: httpd_request.erl.patch Type: text/x-patch Size: 475 bytes Desc: not available URL: From jan@REDACTED Mon Feb 9 12:35:42 2009 From: jan@REDACTED (Jan Lehnardt) Date: Mon, 9 Feb 2009 12:35:42 +0100 Subject: [erlang-patches] Patch for PUT and DELETE inets HTTP methods support In-Reply-To: <5912359.7541234174231359.JavaMail.root@zimbra> References: <5912359.7541234174231359.JavaMail.root@zimbra> Message-ID: <20F29D75-CF68-4CDB-9E3D-35167FF50C14@apache.org> On 9 Feb 2009, at 11:10, Michal Ptaszek wrote: > In the times of RESTful addressing style it is a very inconvenient > situation where when one wants to remove something on the server > side must use 'POST' method instead of simple 'DELETE'. > > Are there any plans to widen the set of accepted HTTP methods? We shipped our own patched version of inets to fix that back in the day with CouchDB. I agree that this is a useful addition. Also, if you are considering my patch about the WebDAV extension methods, you'd most likely want to update this list as well. Cheers Jan -- From raimo+erlang-patches@REDACTED Tue Feb 10 11:47:56 2009 From: raimo+erlang-patches@REDACTED (Raimo Niskanen) Date: Tue, 10 Feb 2009 11:47:56 +0100 Subject: [erlang-patches] : fread failing to handle newlines gracefully In-Reply-To: <20090205083501.GA11769@erix.ericsson.se> References: <20090130121223.GA11448@hezmatt.org> <20090205083501.GA11769@erix.ericsson.se> Message-ID: <20090210104756.GA24142@erix.ericsson.se> On Thu, Feb 05, 2009 at 09:35:01AM +0100, Raimo Niskanen wrote: > On Fri, Jan 30, 2009 at 11:12:23PM +1100, Matthew Palmer wrote: > > Hi, > > > > I recently fielded a question on Stack Overflow regarding some strange > > behaviour of fread, with the ~d format spec > > (http://stackoverflow.com/questions/473327/unexpected-behavior-of-iofread-in-erlang/490023#490023). > > Digging deep, it appears to be a bit of a bug in the way that newlines are > > handled in the format parser, as demonstrated here: > > > > 3> io_lib_fread:fread([], "10 11\n12 13 14\n", "~d"). > > {done,{ok,"\n"}," 1112 13 14\n"} > > > > It's eaten the newline between "11" and "12", which means that on the next > > pass through it'll read the next number as "1112" instead of "11", which > > poses some obvious problems. > > > > I've come up with the attached patch to fix the problem. I'm not very > > experienced with Erlang yet, so it might not be the cleanest way to do it, > > but it does definitely fix the problem, and without (apparently) breaking > > any other common use case. Comments appreciated. If there's a test suite I > > should be adding to as well, I'm more than happy to do that, I just couldn't > > find one in my source tarball. > > > > - Matt > > Thank you for your effort! > > Your diagnosis and patch are basically right. I have now been digging > into the matter and it took a while to ensure that. I will polish > the patch a bit (the code needs some cleanup), and improve it > for \r\n combinations. > > Unfortunately there are no tests in the source tarball. We are striving > towards their release but so far large parts of them are too much > depending on our internal environment to be globally useful... > > I will write tests too. When done I will release a new > source code patch here. OK. The patch is appended. I also found and corrected a bug for e.g io_lib:fread("~s", "\rabc"). > > > --- ../erlang/erlang-12.b.3-dfsg/lib/stdlib/src/io_lib_fread.erl 2004-09-14 21:41:00.000000000 +1000 > > +++ ./io_lib_fread.erl 2009-01-30 15:44:57.000000000 +1100 > > @@ -35,9 +35,9 @@ > > fread_collect(MoreChars, [], Rest, RestFormat, N, Inputs). > > > > fread_collect([$\r|More], Stack, Rest, RestFormat, N, Inputs) -> > > - fread(RestFormat, Rest ++ reverse(Stack), N, Inputs, More); > > + fread(RestFormat, Rest ++ reverse(Stack), N, Inputs, [$\r|More]); > > fread_collect([$\n|More], Stack, Rest, RestFormat, N, Inputs) -> > > - fread(RestFormat, Rest ++ reverse(Stack), N, Inputs, More); > > + fread(RestFormat, Rest ++ reverse(Stack), N, Inputs, [$\n|More]); > > fread_collect([C|More], Stack, Rest, RestFormat, N, Inputs) -> > > fread_collect(More, [C|Stack], Rest, RestFormat, N, Inputs); > > fread_collect([], Stack, Rest, RestFormat, N, Inputs) -> > > @@ -55,8 +55,8 @@ > > eof -> > > fread(RestFormat,eof,N,Inputs,eof); > > _ -> > > - %% Don't forget to count the newline. > > - {more,{More,RestFormat,N+1,Inputs}} > > + %% Don't forget to strip and count the newline. > > + {more,{tl(More),RestFormat,N+1,Inputs}} > > end; > > Other -> %An error has occurred > > {done,Other,More} > > > _______________________________________________ > > erlang-patches mailing list > > erlang-patches@REDACTED > > http://www.erlang.org/mailman/listinfo/erlang-patches > > -- > > / Raimo Niskanen, Erlang/OTP, Ericsson AB > _______________________________________________ > erlang-patches mailing list > erlang-patches@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-patches -- / Raimo Niskanen, Erlang/OTP, Ericsson AB -------------- next part -------------- A non-text attachment was scrubbed... Name: io_lib_fread.patch Type: text/x-patch Size: 4718 bytes Desc: not available URL: From matthias@REDACTED Thu Feb 12 14:35:09 2009 From: matthias@REDACTED (Matthias Lang) Date: Thu, 12 Feb 2009 14:35:09 +0100 Subject: [erlang-patches] patch to fix inet:ifset/2 breakage in R11B-5 (already fixed in R12) Message-ID: <20090212133509.GA14372@contorpis.lisalinda.com> Hi, inet:ifset/2 is an undocumented but useful function. It got broken somewhere between R11B-2 and R11B-5, it looks like it broke when SCTP got added. Here's what happens in R11B-5: 1> inet:ifset("eth0", {addr, {172,16,2,1}}). ** exited: {function_clause, [{prim_inet,encode_ifopt_val,[{addr,{172,16,2,1}},[]]}, {prim_inet,ifset,3}, {inet,withsocket,1}, {erl_eval,do_apply,5}, {shell,exprs,6}, {shell,eval_loop,3}]} ** I can't find any mention of the problem in the release notes for R12B-0, but it seems to be fixed. This patch is only of interest to people who specifically want to use R11B-5 and inet:ifset/2, i.e. probably only me. Matt --- lib/kernel/src/prim_inet.erl.11B5 2007-06-11 14:55:45.000000000 +0200 +++ lib/kernel/src/prim_inet.erl 2009-02-03 06:50:01.909368558 +0100 @@ -1707,9 +1707,9 @@ encode_ifopt_val([{Opt,Val}|Opts], Buf) -> Type = type_ifopt(Opt), - try type_value(Type, Val) of + try type_value(set, Type, Val) of true -> encode_ifopt_val(Opts, - [Buf,enc_ifopt(Opt),enc_value(Type,Val)]); + [Buf,enc_ifopt(Opt),enc_value(set, Type,Val)]); false -> {error,einval} catch Reason -> {error,Reason} From sverker@REDACTED Fri Feb 13 20:08:41 2009 From: sverker@REDACTED (Sverker Eriksson) Date: Fri, 13 Feb 2009 20:08:41 +0100 Subject: [erlang-patches] Patch for packet_parser.c to Include HTTP Methods Defined in RFC 2518 / WebDAV In-Reply-To: <2D918F25-4E4E-479B-BBC8-407A6028F698@apache.org> References: <2D918F25-4E4E-479B-BBC8-407A6028F698@apache.org> Message-ID: <4995C539.7000009@erix.ericsson.se> Jan Lehnardt wrote: > [...] > CouchDB (and other implementors of HTTP extending standards) would > benefit from the inclusion of the methods in RFC 2518. While CouchDB > only needs COPY and MOVE (for now), adding the complete suite of > additional HTTP methods seems sensible. Please consider the patch > for inclusion into the next OTP release. > > --- a/erts/emulator/beam/packet_parser.c 2009-02-09 00:14:24.000000000 > +0100 > +++ b/erts/emulator/beam/packet_parser.c 2009-02-09 00:12:56.000000000 > +0100 > @@ -139,6 +139,13 @@ > "PUT", > "DELETE", > "TRACE", > + "PROPFIND", > + "PROPPATCH", > + "MKCOL", > + "COPY", > + "MOVE", > + "LOCK", > + "UNLOCK", > NULL What do you think about the risk of applications that may break expecting any of these methods as strings and not as atoms? /Sverker, Erlang/OTP From jan@REDACTED Sat Feb 14 14:17:28 2009 From: jan@REDACTED (Jan Lehnardt) Date: Sat, 14 Feb 2009 14:17:28 +0100 Subject: [erlang-patches] Patch for packet_parser.c to Include HTTP Methods Defined in RFC 2518 / WebDAV In-Reply-To: <4995C539.7000009@erix.ericsson.se> References: <2D918F25-4E4E-479B-BBC8-407A6028F698@apache.org> <4995C539.7000009@erix.ericsson.se> Message-ID: On 13 Feb 2009, at 20:08, Sverker Eriksson wrote: > Jan Lehnardt wrote: >> [...] >> CouchDB (and other implementors of HTTP extending standards) would >> benefit from the inclusion of the methods in RFC 2518. While CouchDB >> only needs COPY and MOVE (for now), adding the complete suite of >> additional HTTP methods seems sensible. Please consider the patch >> for inclusion into the next OTP release. >> --- a/erts/emulator/beam/packet_parser.c 2009-02-09 >> 00:14:24.000000000 +0100 >> +++ b/erts/emulator/beam/packet_parser.c 2009-02-09 >> 00:12:56.000000000 +0100 >> @@ -139,6 +139,13 @@ >> "PUT", >> "DELETE", >> "TRACE", >> + "PROPFIND", >> + "PROPPATCH", >> + "MKCOL", >> + "COPY", >> + "MOVE", >> + "LOCK", >> + "UNLOCK", >> NULL > > > What do you think about the risk of applications that may break > expecting any of these methods as strings and not as atoms? Good point. I'm not aware of any Erlang WebDAV server but I do not constantly monitor the application space and I'm relatively new to Erlang (2 years now). I certainly have no idea about internal applications that use the WebDAV protocol or use any other non-standard HTTP extensions. I found http://code.google.com/p/webdav4erlang/ which doesn't have any code. I'd argue that if nobody has stepped up to "fix" this problem, as it is a relatively simple patch, this hasn't gotten a lot of attention at all. An inquiry on erlang-questions@ might be a good idea. If really needed, it might be worth the time to make the old or the new code optional. With say, inet:setopts(Socket, [{packet, http}]), % http + webdav, this patch inet:setopts(Socket, [{packet, http_strict}]), % RFC 2616 only, old behaviour or inet:setopts(Socket, [{packet, http}]), % RFC 2616 only, old behaviour inet:setopts(Socket, [{packet, webdav}]), % RFC 2518, this patch If at all, I'd go for the first option (the naming of the PacketType atom is just a suggestion). Legacy applications (the number I believe to be in the single digits) can be fixed by either removing their special handling of the non-standard methods or using `http_strict`. The second option is more defensive for legacy applications and would require us to patch MochiWeb (and hope for acceptance). I don't know how feasible or not this is. Cheers Jan -- From jan@REDACTED Sat Feb 14 14:49:25 2009 From: jan@REDACTED (Jan Lehnardt) Date: Sat, 14 Feb 2009 14:49:25 +0100 Subject: [erlang-patches] Patch for packet_parser.c to Include HTTP Methods Defined in RFC 2518 / WebDAV In-Reply-To: References: <2D918F25-4E4E-479B-BBC8-407A6028F698@apache.org> <4995C539.7000009@erix.ericsson.se> Message-ID: <18F8B848-F97B-4AA3-9768-0BAA82AFE594@apache.org> On 14 Feb 2009, at 14:17, Jan Lehnardt wrote: > > Good point. I'm not aware of any Erlang WebDAV server > but I do not constantly monitor the application space and > I'm relatively new to Erlang (2 years now). I certainly have > no idea about internal applications that use the WebDAV > protocol or use any other non-standard HTTP extensions. A little more digging. YAWS has a WevDAV extension and it uses this code to solve the issue: call_method(Method, CliSock, Req, H) -> case Method of F when atom(F) -> ?MODULE:F(CliSock, Req, H); L when list(L) -> handle_extension_method(L, CliSock, Req, H) end. (YAWS 1.79 source src/yaws_server.erl line 1155 ff.) YAWS hence, wouldn't break. Maybe because Klacke was smart enough to foresee that this is getting fixed eventually :) -- Erlware includes a WebDAV module and as far as I understand*, it does its own http parsing and always uses lists / strings, not atoms for methods. * http://git.erlware.org/web?p=crary_dav.git;a=blob_plain;f=crary_dav/src/crary_dav.erl;hb=HEAD and http://git.erlware.org/web?p=crary.git;a=blob_plain;f=src/crary_sock.erl;hb=HEAD Cheers, Jan -- From oscar@REDACTED Mon Feb 16 12:04:20 2009 From: oscar@REDACTED (=?ISO-8859-1?Q?Oscar_Hellstr=F6m?=) Date: Mon, 16 Feb 2009 11:04:20 +0000 Subject: [erlang-patches] Patch for packet_parser.c to Include HTTP Methods Defined in RFC 2518 / WebDAV In-Reply-To: <18F8B848-F97B-4AA3-9768-0BAA82AFE594@apache.org> References: <2D918F25-4E4E-479B-BBC8-407A6028F698@apache.org> <4995C539.7000009@erix.ericsson.se> <18F8B848-F97B-4AA3-9768-0BAA82AFE594@apache.org> Message-ID: <49994834.80306@erlang-consulting.com> Hi, Firstly, I fail to understand why this is an issue that need to be "fixed"? Except the increased size and a few more CPU cycles when matching (which should quite small any way) why is it it a problem having the methods as strings? Personally I wouldn't mind if *all* methods would have been strings since mixing types makes programming a bit bloated, but this obviously can't be changed. Anyway, for the legacy discussion. Looking at the yaws code, your assumption seems wrong. In handle_extension_method/4 the code assumes that the extension method is a string, at least in the rather old version that I looked at. It would certainly be possible to add clause before the last one such as: handle_extension_method(Method, CliSock, Req, Head) when is_atom(Methed) -> handle_extension_method(atom_to_list(Method), CliSock, Req, Head); but without this, it looks to me as if yaws would fail. Jan Lehnardt wrote: > On 14 Feb 2009, at 14:17, Jan Lehnardt wrote: > >> Good point. I'm not aware of any Erlang WebDAV server >> but I do not constantly monitor the application space and >> I'm relatively new to Erlang (2 years now). I certainly have >> no idea about internal applications that use the WebDAV >> protocol or use any other non-standard HTTP extensions. >> > > A little more digging. > > YAWS has a WevDAV extension and it uses this code to > solve the issue: > > call_method(Method, CliSock, Req, H) -> > case Method of > F when atom(F) -> > ?MODULE:F(CliSock, Req, H); > L when list(L) -> > handle_extension_method(L, CliSock, Req, H) > end. > > (YAWS 1.79 source src/yaws_server.erl line 1155 ff.) > > YAWS hence, wouldn't break. Maybe because Klacke > was smart enough to foresee that this is getting fixed > eventually :) > > -- > > Erlware includes a WebDAV module and as far as I > understand*, it does its own http parsing and always > uses lists / strings, not atoms for methods. > > * http://git.erlware.org/web?p=crary_dav.git;a=blob_plain;f=crary_dav/src/crary_dav.erl;hb=HEAD > and > http://git.erlware.org/web?p=crary.git;a=blob_plain;f=src/crary_sock.erl;hb=HEAD > > Cheers, > Jan > -- > > _______________________________________________ > erlang-patches mailing list > erlang-patches@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-patches > -- Oscar Hellstr?m, oscar@REDACTED Office: +44 20 7655 0337 Mobile: +44 798 45 44 773 Erlang Training and Consulting http://www.erlang-consulting.com/ From jan@REDACTED Mon Feb 16 12:46:46 2009 From: jan@REDACTED (Jan Lehnardt) Date: Mon, 16 Feb 2009 12:46:46 +0100 Subject: [erlang-patches] Patch for packet_parser.c to Include HTTP Methods Defined in RFC 2518 / WebDAV In-Reply-To: <49994834.80306@erlang-consulting.com> References: <2D918F25-4E4E-479B-BBC8-407A6028F698@apache.org> <4995C539.7000009@erix.ericsson.se> <18F8B848-F97B-4AA3-9768-0BAA82AFE594@apache.org> <49994834.80306@erlang-consulting.com> Message-ID: Hi Oscar, thanks for your comments. On 16 Feb 2009, at 12:04, Oscar Hellstr?m wrote: > Firstly, I fail to understand why this is an issue that need to be > "fixed"? Except the increased size and a few more CPU cycles when > matching (which should quite small any way) why is it it a problem > having the methods as strings? Personally I wouldn't mind if *all* > methods would have been strings since mixing types makes programming a > bit bloated, but this obviously can't be changed. I wouldn't mind atoms or strings either, what's weird is that the API returns both and clients have to deal with whether they use RFC 2616- defined methods or not. Either asking explicitly for WebDAV methods using `inet:setopts(Socket, [{packet, webdav}])` or just handling all known HTTP extension methods as atoms (for backwards compatibility), "fixes" that. The reasoning behind the patch is less code (less code is good), not speed. > Anyway, for the legacy discussion. Looking at the yaws code, your > assumption seems wrong. In handle_extension_method/4 the code assumes > that the extension method is a string, at least in the rather old > version that I looked at. It would certainly be possible to add clause > before the last one such as: > handle_extension_method(Method, CliSock, Req, Head) when > is_atom(Methed) -> > handle_extension_method(atom_to_list(Method), CliSock, Req, Head); > but without this, it looks to me as if yaws would fail. My version of `handle_extension_method()` reads: handle_extension_method("PROPFIND", CliSock, Req, Head) -> 'PROPFIND'(CliSock, Req, Head); handle_extension_method("MKCOL", CliSock, Req, Head) -> 'MKCOL'(CliSock, Req, Head); handle_extension_method("MOVE", CliSock, Req, Head) -> 'MOVE'(CliSock, Req, Head); handle_extension_method("COPY", CliSock, Req, Head) -> 'COPY'(CliSock, Req, Head); handle_extension_method(_Method, CliSock, Req, Head) -> not_implemented(CliSock, Req, Head). It gets called from: call_method(Method, CliSock, Req, H) -> case Method of F when atom(F) -> ?MODULE:F(CliSock, Req, H); L when list(L) -> handle_extension_method(L, CliSock, Req, H) end. I.e. "METHOD" is routed through `handle_extension_method()` to call `'METHOD'()` in `call_method()` while 'METHOD' directly calls `'METHOD'()`. Again, Yaws 1.79. So earlier versions of YAWS break. Cheers Jan -- > > > Jan Lehnardt wrote: >> On 14 Feb 2009, at 14:17, Jan Lehnardt wrote: >> >>> Good point. I'm not aware of any Erlang WebDAV server >>> but I do not constantly monitor the application space and >>> I'm relatively new to Erlang (2 years now). I certainly have >>> no idea about internal applications that use the WebDAV >>> protocol or use any other non-standard HTTP extensions. >>> >> >> A little more digging. >> >> YAWS has a WevDAV extension and it uses this code to >> solve the issue: >> >> call_method(Method, CliSock, Req, H) -> >> case Method of >> F when atom(F) -> >> ?MODULE:F(CliSock, Req, H); >> L when list(L) -> >> handle_extension_method(L, CliSock, Req, H) >> end. >> >> (YAWS 1.79 source src/yaws_server.erl line 1155 ff.) >> >> YAWS hence, wouldn't break. Maybe because Klacke >> was smart enough to foresee that this is getting fixed >> eventually :) >> >> -- >> >> Erlware includes a WebDAV module and as far as I >> understand*, it does its own http parsing and always >> uses lists / strings, not atoms for methods. >> >> * http://git.erlware.org/web?p=crary_dav.git;a=blob_plain;f=crary_dav/src/crary_dav.erl;hb=HEAD >> and >> http://git.erlware.org/web?p=crary.git;a=blob_plain;f=src/crary_sock.erl;hb=HEAD >> >> Cheers, >> Jan >> -- >> >> _______________________________________________ >> erlang-patches mailing list >> erlang-patches@REDACTED >> http://www.erlang.org/mailman/listinfo/erlang-patches >> > > > -- > Oscar Hellstr?m, oscar@REDACTED > Office: +44 20 7655 0337 > Mobile: +44 798 45 44 773 > Erlang Training and Consulting > http://www.erlang-consulting.com/ > > From oscar@REDACTED Mon Feb 16 13:08:17 2009 From: oscar@REDACTED (=?ISO-8859-1?Q?Oscar_Hellstr=F6m?=) Date: Mon, 16 Feb 2009 12:08:17 +0000 Subject: [erlang-patches] Patch for packet_parser.c to Include HTTP Methods Defined in RFC 2518 / WebDAV In-Reply-To: References: <2D918F25-4E4E-479B-BBC8-407A6028F698@apache.org> <4995C539.7000009@erix.ericsson.se> <18F8B848-F97B-4AA3-9768-0BAA82AFE594@apache.org> <49994834.80306@erlang-consulting.com> Message-ID: <49995731.2060409@erlang-consulting.com> Hi, Jan Lehnardt wrote: > Hi Oscar, > > thanks for your comments. > I wouldn't mind atoms or strings either, what's weird is that the API > returns both and clients have to deal with whether they use RFC 2616- > defined methods or not. Either asking explicitly for WebDAV methods > using `inet:setopts(Socket, [{packet, webdav}])` or just handling all > known HTTP extension methods as atoms (for backwards compatibility), > "fixes" that. > > The reasoning behind the patch is less code (less code is good), not > speed. I agree, but since you still have to check if you got another method, i.e. when is_atom(Method), you're just postponing the problem. Making all methods atoms isn't possible due to DOS attacks and I'm sure that there are a few other RFCs, more or less well known, that will add other HTTP methods. For instance, an HTTP proxy might wan to implement the CONNECT method... My approach is letting the receiving port of the code transform all methods to strings before actually dealing with the request. > My version of `handle_extension_method()` reads: > > handle_extension_method("PROPFIND", CliSock, Req, Head) -> > 'PROPFIND'(CliSock, Req, Head); > handle_extension_method("MKCOL", CliSock, Req, Head) -> > 'MKCOL'(CliSock, Req, Head); > handle_extension_method("MOVE", CliSock, Req, Head) -> > 'MOVE'(CliSock, Req, Head); > handle_extension_method("COPY", CliSock, Req, Head) -> > 'COPY'(CliSock, Req, Head); > handle_extension_method(_Method, CliSock, Req, Head) -> > not_implemented(CliSock, Req, Head). > > It gets called from: > > call_method(Method, CliSock, Req, H) -> > case Method of > F when atom(F) -> > ?MODULE:F(CliSock, Req, H); > L when list(L) -> > handle_extension_method(L, CliSock, Req, H) > end. > > I.e. "METHOD" is routed through `handle_extension_method()` > to call `'METHOD'()` in `call_method()` while 'METHOD' directly > calls `'METHOD'()`. > > Again, Yaws 1.79. > > So earlier versions of YAWS break. My bad, I think mine reads the same ;) Cheers -- Oscar Hellstr?m, oscar@REDACTED Office: +44 20 7655 0337 Mobile: +44 798 45 44 773 Erlang Training and Consulting http://www.erlang-consulting.com/ From jan@REDACTED Mon Feb 16 13:52:04 2009 From: jan@REDACTED (Jan Lehnardt) Date: Mon, 16 Feb 2009 13:52:04 +0100 Subject: [erlang-patches] Patch for packet_parser.c to Include HTTP Methods Defined in RFC 2518 / WebDAV In-Reply-To: <49995731.2060409@erlang-consulting.com> References: <2D918F25-4E4E-479B-BBC8-407A6028F698@apache.org> <4995C539.7000009@erix.ericsson.se> <18F8B848-F97B-4AA3-9768-0BAA82AFE594@apache.org> <49994834.80306@erlang-consulting.com> <49995731.2060409@erlang-consulting.com> Message-ID: On 16 Feb 2009, at 13:08, Oscar Hellstr?m wrote: > My approach is letting the receiving port of the code > transform all methods to strings before actually dealing with the > request. The whole idea of the patch is that there should be no extra code to do this :) If there are more methods that should be included, those who need it should speak up. Maybe `inet:setopts(Socket, [{packet, http_methods_as_lists}])` is some middle ground? (Note that the atom name is just to make the purpose clear :) Cheers Jan --