[erlang-questions] SSL Client memory usage

Tue Jun 11 12:00:37 CEST 2019

Please make sure to also pass {verify, verify_peer}, since any ssl_options
you pass will overwrite the hackney defaults, and therefore pick up the ssl
default of verify_none.

Eshell V10.2.5  (abort with ^G)
1> hackney:request(get, <<"https://self-signed.badssl.com/">>, [], <<>>,
[{ssl_options, [{cacertfile, certifi:cacertfile()}]}]).
{ok,200,[...],#Ref<...>} // Not good!
2> hackney:request(get, <<"https://self-signed.badssl.com/">>, [], <<>>,
[{ssl_options, [{cacertfile, certifi:cacertfile()}, {verify,
verify_peer}]}]).
{error,{tls_alert,"bad certificate"}}

Bram

On Tue, 11 Jun 2019 at 09:06, Jamie McClymont <
jamie+erlang-questions@REDACTED> wrote:

> Ahh that did it, thanks!
>
> I'm using hackney, which in turn uses certifi:cacerts() (binary blobs of
> cert data).
>
> Using hackney's ssl options to instead pass certifi:cacertfile() as an
> argument brings the memory usage of each keepalive'd tls connection (once
> the process is GCed) down to <30KB -- that's excellent :)
>
>
>
> On Mon, 10 Jun 2019, at 9:51 PM, Ingela Andin wrote:
> > Hi!
> >
> > How do you provide your certificates to ssl? If you provide your
> > CA-certs as binary DER blobs to the connection they
> > will be stored by the connection and be local to the connection. If you
> > provide them through a file they will be stored
> > in a ETS table and maybe referenced by other connections as there is a
> > way to refer to them.
> >
> > Regards Ingela Erlang/OTP Team
> >
> > Den mån 10 juni 2019 kl 05:10 skrev Jamie McClymont
> > <jamie+erlang-questions@REDACTED
> > <mailto:jamie%2Berlang-questions@REDACTED>>:
> > > Hi all
> > >
> > >  I'm building an app which needs to be able to make a good number of
> concurrent HTTPS requests (ideally I could keep high-hundreds/low-thousands
> of idle keepalive connections open at a time with modest memory usage).
> > >
> > >  Poking around (by making concurrent requests to 50 domains), I find
> that holding HTTP connections is effectively free memory-wise, while
> holding HTTPS connections costs around 2.5MB per connection! All this
> memory usage is in the tls_connection:init/1 processes, according to
> observer.
> > >
> > >  Looking at the state of these processes, they each hold a lot of
> information: the extracted/decoded certificate files for all the CA certs
> on my system (of which there are 134!). From some cursory googling about
> erlang's memory model, it seems that because this data is not a binary blob
> but a big tree of terms, it's going to be copied to every tls_connection
> process.
> > >
> > >  Any pointers on if there is an existing way to mitigate this (while
> keeping a full set of root certs available)? Otherwise, it seems to me the
> reasonable solution would be to keep the CA store in a central process
> (ETS?) and query that for the specific root cert that the server says it is
> signed by. If I'll be implementing it, could someone sanity-check this
> approach:
> > >  * provide an empty list of CAs to the ssl module
> > >  * provide a verify_fun which matches on {bad_cert, unknown_ca} and
> then gets the correct CA and performs all the remaining necessary checks
> (whatever those are...) before returning {valid, UserState} (or
> {valid_peer, UserState}?)
> > >
> > >
> > >  Thanks
> > >  - Jamie
> > >
> > >  P.s. sorry if this email is a duplicate: I think my last one was
> silently dropped since I sent it from a different address to the one I
> subscribed with, but not certain.
> > >
> > >  Appendix - one of the 134 decoded certs stored by every process:
> > >
> > >  {decoded,
> > >  {{#Ref<0.577909663.1295777797.255240>,
> > >  106100277556486529736699587978573607008,
> > >  {rdnSequence,
> > >  [[{'AttributeTypeAndValue',{2,5,4,6},"CN"}],
> > >  [{'AttributeTypeAndValue',
> > >  {2,5,4,10},
> > >  {utf8String,<<"UniTrust">>}}],
> > >  [{'AttributeTypeAndValue',
> > >  {2,5,4,3},
> > >  {utf8String,
> > >  <<"UCA Extend"...(28 B)>>}}]]}},
> > >  {<<48,130,5,90,48,130,3,66,160,3...(1 kB)>>,
> > >  {'OTPCertificate',
> > >  {'OTPTBSCertificate',v3,106100277556486529736699587978573607008,
> > >  {'SignatureAlgorithm',{1,2,840,113549,1,1,11},'NULL'},
> > >  {rdnSequence,
> > >  [[{'AttributeTypeAndValue',{2,5,4,6},"CN"}],
> > >  [{'AttributeTypeAndValue',
> > >  {2,5,4,10},
> > >  {utf8String,<<"UniTrust">>}}],
> > >  [{'AttributeTypeAndValue',
> > >  {2,5,4,3},
> > >  {utf8String,
> > >  <<"UCA Extend"...(28 B)>>}}]]},
> > >  {'Validity',{utcTime,"150313000000Z"},{utcTime,"381231000000Z"}},
> > >  {rdnSequence,
> > >  [[{'AttributeTypeAndValue',{2,5,4,6},"CN"}],
> > >  [{'AttributeTypeAndValue',
> > >  {2,5,4,10},
> > >  {utf8String,<<"UniTrust">>}}],
> > >  [{'AttributeTypeAndValue',
> > >  {2,5,4,3},
> > >  {utf8String,
> > >  <<"UCA Extend"...(28 B)>>}}]]},
> > >  {'OTPSubjectPublicKeyInfo',
> > >  {'PublicKeyAlgorithm',{1,2,840,113549,1,1,1},'NULL'},
> > >  {'RSAPublicKey',
> > >
> 689603717979852636831081426524261160465705539154429034072886964586226830541206711367761253770112725458465044497361739659482435934368052465546153546798794021727234724250491430601994902725573258440903769572247434719710136763019389835974660379560646189661431052777862620577580627735499519427983731365873646101116220703443112119094192330629089538925644308340292126955780669276375525797477380635249258541277353787220035677250408659616695990328739419944306797836041926101961617154636604738870383637100044978431121318943912528365357866046388476749437076957867974020718613998841641003305315809368105957269212277478816744375233680463674812053558530520195020911324127390436183137612053158831455696931169855471108960460199610930021503675868841756093916690737925568854144895439113023618139270733541586805299082563244530623575372402602522085830629158312024768857183218701826386841425816972120556588392270337545276474712543306659887836405319075385033422367187135702233624763183702955559196400
> > >
> 695312060899684142798408078723091214153237781607055313585627048184929762700635100359534649156945631489915475941635189136663455081054904713071171644045263631869342706234148072080520374089220391876551645591673277270360119934713372188503089344323356273673087,
> > >  65537}},
> > >  asn1_NOVALUE,asn1_NOVALUE,
> > >  [{'Extension',
> > >  {2,5,29,14},
> > >  false,
> > >  <<217,116,58,228,48,61,13,247,18,220...(20 B)>>},
> > >  {'Extension',
> > >  {2,5,29,19},
> > >  true,
> > >  {'BasicConstraints',true,asn1_NOVALUE}},
> > >  {'Extension',
> > >  {2,5,29,15},
> > >  true,
> > >  [digitalSignature,keyCertSign,cRLSign]}]},
> > >  {'SignatureAlgorithm',{1,2,840,113549,1,1,11},'NULL'},
> > >  <<54,141,151,204,66,21,100,41,55,155...(512 B)>>}}}},
> > >  _______________________________________________
> > >  erlang-questions mailing list
> > > erlang-questions@REDACTED
> > > http://erlang.org/mailman/listinfo/erlang-questions
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20190611/a28922cd/attachment.htm>