[erlang-questions] SSL Client memory usage

[Ingela Andin]

Hi!

How do you provide your certificates to ssl? If you provide your CA-certs
as binary DER blobs to the connection they
will be stored by the connection and be local to the connection. If you
provide them through a file they will be stored
in a ETS table and maybe referenced by other connections as there is a way
to refer to them.

Regards Ingela Erlang/OTP Team

Den mån 10 juni 2019 kl 05:10 skrev Jamie McClymont <
jamie+erlang-questions@REDACTED>:

> Hi all
>
> I'm building an app which needs to be able to make a good number of
> concurrent HTTPS requests (ideally I could keep high-hundreds/low-thousands
> of idle keepalive connections open at a time with modest memory usage).
>
> Poking around (by making concurrent requests to 50 domains), I find that
> holding HTTP connections is effectively free memory-wise, while holding
> HTTPS connections costs around 2.5MB per connection! All this memory usage
> is in the tls_connection:init/1 processes, according to observer.
>
> Looking at the state of these processes, they each hold a lot of
> information: the extracted/decoded certificate files for all the CA certs
> on my system (of which there are 134!). From some cursory googling about
> erlang's memory model, it seems that because this data is not a binary blob
> but a big tree of terms, it's going to be copied to every tls_connection
> process.
>
> Any pointers on if there is an existing way to mitigate this (while
> keeping a full set of root certs available)? Otherwise, it seems to me the
> reasonable solution would be to keep the CA store in a central process
> (ETS?) and query that for the specific root cert that the server says it is
> signed by. If I'll be implementing it, could someone sanity-check this
> approach:
> * provide an empty list of CAs to the ssl module
> * provide a verify_fun which matches on {bad_cert, unknown_ca} and then
> gets the correct CA and performs all the remaining necessary checks
> (whatever those are...) before returning {valid, UserState} (or
> {valid_peer, UserState}?)
>
>
> Thanks
> - Jamie
>
> P.s. sorry if this email is a duplicate: I think my last one was silently
> dropped since I sent it from a different address to the one I subscribed
> with, but not certain.
>
> Appendix - one of the 134 decoded certs stored by every process:
>
>      {decoded,
>       {{#Ref<0.577909663.1295777797.255240>,
>         106100277556486529736699587978573607008,
>         {rdnSequence,
>          [[{'AttributeTypeAndValue',{2,5,4,6},"CN"}],
>           [{'AttributeTypeAndValue',
>             {2,5,4,10},
>             {utf8String,<<"UniTrust">>}}],
>           [{'AttributeTypeAndValue',
>             {2,5,4,3},
>             {utf8String,
>              <<"UCA Extend"...(28 B)>>}}]]}},
>        {<<48,130,5,90,48,130,3,66,160,3...(1 kB)>>,
>         {'OTPCertificate',
>          {'OTPTBSCertificate',v3,106100277556486529736699587978573607008,
>           {'SignatureAlgorithm',{1,2,840,113549,1,1,11},'NULL'},
>           {rdnSequence,
>            [[{'AttributeTypeAndValue',{2,5,4,6},"CN"}],
>             [{'AttributeTypeAndValue',
>               {2,5,4,10},
>               {utf8String,<<"UniTrust">>}}],
>             [{'AttributeTypeAndValue',
>               {2,5,4,3},
>               {utf8String,
>                <<"UCA Extend"...(28 B)>>}}]]},
>           {'Validity',{utcTime,"150313000000Z"},{utcTime,"381231000000Z"}},
>           {rdnSequence,
>            [[{'AttributeTypeAndValue',{2,5,4,6},"CN"}],
>             [{'AttributeTypeAndValue',
>               {2,5,4,10},
>               {utf8String,<<"UniTrust">>}}],
>             [{'AttributeTypeAndValue',
>               {2,5,4,3},
>               {utf8String,
>                <<"UCA Extend"...(28 B)>>}}]]},
>           {'OTPSubjectPublicKeyInfo',
>            {'PublicKeyAlgorithm',{1,2,840,113549,1,1,1},'NULL'},
>            {'RSAPublicKey',
>
> 689603717979852636831081426524261160465705539154429034072886964586226830541206711367761253770112725458465044497361739659482435934368052465546153546798794021727234724250491430601994902725573258440903769572247434719710136763019389835974660379560646189661431052777862620577580627735499519427983731365873646101116220703443112119094192330629089538925644308340292126955780669276375525797477380635249258541277353787220035677250408659616695990328739419944306797836041926101961617154636604738870383637100044978431121318943912528365357866046388476749437076957867974020718613998841641003305315809368105957269212277478816744375233680463674812053558530520195020911324127390436183137612053158831455696931169855471108960460199610930021503675868841756093916690737925568854144895439113023618139270733541586805299082563244530623575372402602522085830629158312024768857183218701826386841425816972120556588392270337545276474712543306659887836405319075385033422367187135702233624763183702955559196400
>
>  695312060899684142798408078723091214153237781607055313585627048184929762700635100359534649156945631489915475941635189136663455081054904713071171644045263631869342706234148072080520374089220391876551645591673277270360119934713372188503089344323356273673087,
>             65537}},
>           asn1_NOVALUE,asn1_NOVALUE,
>           [{'Extension',
>             {2,5,29,14},
>             false,
>             <<217,116,58,228,48,61,13,247,18,220...(20 B)>>},
>            {'Extension',
>             {2,5,29,19},
>             true,
>             {'BasicConstraints',true,asn1_NOVALUE}},
>            {'Extension',
>             {2,5,29,15},
>             true,
>             [digitalSignature,keyCertSign,cRLSign]}]},
>          {'SignatureAlgorithm',{1,2,840,113549,1,1,11},'NULL'},
>          <<54,141,151,204,66,21,100,41,55,155...(512 B)>>}}}},
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20190610/984d814c/attachment.htm>