<div dir="ltr"><div>I've run better test of just hash(sha, Data) instead of hmac that does more things and got 2x speed (1 microsecond vs 2 microseconds).</div><div>Is there a way to improve the crypto:macN/5 speed to get it closer to 100% gain instead of just 25% gain?<br></div><div><br></div><div>We can clear the SHA hardware extension to test its effect by running the shell with the following env variable:</div><div>OPENSSL_ia32cap=:~0x20000000 erl</div><div><br></div><div>Benchmark code:<br></div><div>Data = crypto:strong_rand_bytes(1024),<br>TC = fun(TC_M, TC_F, TC_A, TC_N) when TC_N > 0 -> TC_L = tl([begin {TC_T, _Result} = timer:tc(TC_M, TC_F, TC_A), TC_T end || _ <- lists:seq(1, TC_N)]), TC_Min = lists:min(TC_L), TC_Max = lists:max(TC_L), TC_Med = lists:nth(round((TC_N - 1) / 2), lists:sort(TC_L)), TC_Avg = round(lists:foldl(fun(TC_X, TC_Sum) -> TC_X + TC_Sum end, 0, TC_L) / (TC_N - 1)), io:format("Range: ~b - ~b mics~nMedian: ~b mics ~nAverage: ~b mics~n", [TC_Min, TC_Max, TC_Med, TC_Avg]), TC_Med end.<br>TC(crypto, hash, [sha, Data], 1000000).</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Feb 21, 2020 at 2:38 AM Ben Browitt <<a href="mailto:ben.browitt@gmail.com">ben.browitt@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>I've compared:<br></div><div>Intel Skylake - no SHA hardware extension, N1 machine type on GCP [1]<br></div><div>Second generation AMD EPYC Rome processor - has SHA hardware extension, N2D machine type on GCP [2]<br></div><div><br></div><div>Ubuntu 18.04</div><div>OpenSSL 1.1.1</div><div>OTP-22.2.7 (erlang-solutions deb package)<br></div><div><br></div><div>openssl speed -evp sha1 on AMD EPYC is about 2X faster than Intel Skylake.</div><div>crypto:macN/5 on AMD EPYC is about 25% faster than Intel Skylake.</div><div><br></div><div>It doesn't seem like crypto:macN/5 on AMD is using the SHA hardware extension. The 25% increase is probably just because Skylake is several years older than AMD EPYC second genration.<br></div><div>Is my test correct?<br></div><div><br></div><div>Tests:</div><div>Key = crypto:strong_rand_bytes(20),<br>Data = crypto:strong_rand_bytes(1000),<br>MacLength = 10,<br>TC = fun(TC_M, TC_F, TC_A, TC_N) when TC_N > 0 -> TC_L = tl([begin {TC_T, _Result} = timer:tc(TC_M, TC_F, TC_A), TC_T end || _ <- lists:seq(1, TC_N)]), TC_Min = lists:min(TC_L), TC_Max = lists:max(TC_L), TC_Med = lists:nth(round((TC_N - 1) / 2), lists:sort(TC_L)), TC_Avg = round(lists:foldl(fun(TC_X, TC_Sum) -> TC_X + TC_Sum end, 0, TC_L) / (TC_N - 1)), io:format("Range: ~b - ~b mics~nMedian: ~b mics ~nAverage: ~b mics~n", [TC_Min, TC_Max, TC_Med, TC_Avg]), TC_Med end.<br>TC(crypto, macN, [hmac, sha, Key, Data, MacLength], 1000000).</div><div><br></div><div>1) Intel Skylake</div><div>TC(crypto, macN, [hmac, sha, Key, Data, MacLength], 1000000).<br>Range: 3 - 729 mics<br>Median: 4 mics <br>Average: 4 mics</div><div><br></div><div>openssl speed sha1<br>Doing sha1 for 3s on 16 size blocks: 19026044 sha1's in 2.99s<br>Doing sha1 for 3s on 64 size blocks: 11512925 sha1's in 2.98s<br>Doing sha1 for 3s on 256 size blocks: 5769743 sha1's in 2.98s<br>Doing sha1 for 3s on 1024 size blocks: 1927668 sha1's in 2.98s<br>Doing sha1 for 3s on 8192 size blocks: 265026 sha1's in 2.98s<br>Doing sha1 for 3s on 16384 size blocks: 133488 sha1's in 2.98s<br>OpenSSL 1.1.1 11 Sep 2018<br>built on: Tue Nov 12 16:58:35 2019 UTC<br>options:bn(64,64) rc4(16x,int) des(int) aes(partial) blowfish(ptr) <br>compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -Wa,--noexecstack -g -O2 -fdebug-prefix-map=/build/openssl-kxN_24/openssl-1.1.1=. -fstack-protector-strong -Wformat -Werror=format-security -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DRC4_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DX25519_ASM -DPADLOCK_ASM -DPOLY1305_ASM -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2<br>The 'numbers' are in 1000s of bytes per second processed.<br>type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes 16384 bytes<br>sha1 101811.61k 247257.45k 495655.77k 662393.30k 728554.70k 733915.23k</div><div><br></div><div>openssl speed -evp sha1<br>Doing sha1 for 3s on 16 size blocks: 11590063 sha1's in 2.99s<br>Doing sha1 for 3s on 64 size blocks: 8259388 sha1's in 2.97s<br>Doing sha1 for 3s on 256 size blocks: 4853323 sha1's in 2.99s<br>Doing sha1 for 3s on 1024 size blocks: 1796528 sha1's in 2.98s<br>Doing sha1 for 3s on 8192 size blocks: 259970 sha1's in 2.99s<br>Doing sha1 for 3s on 16384 size blocks: 131515 sha1's in 2.99s<br>OpenSSL 1.1.1 11 Sep 2018<br>built on: Tue Nov 12 16:58:35 2019 UTC<br>options:bn(64,64) rc4(16x,int) des(int) aes(partial) blowfish(ptr) <br>compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -Wa,--noexecstack -g -O2 -fdebug-prefix-map=/build/openssl-kxN_24/openssl-1.1.1=. -fstack-protector-strong -Wformat -Werror=format-security -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DRC4_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DX25519_ASM -DPADLOCK_ASM -DPOLY1305_ASM -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2<br>The 'numbers' are in 1000s of bytes per second processed.<br>type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes 16384 bytes<br>sha1 62020.40k 177980.08k 415535.35k 617330.43k 712265.63k 720649.42k</div><div><br></div><div>2) AMD EPYC</div><div>TC(crypto, macN, [hmac, sha, Key, Data, MacLength], 1000000).<br>Range: 3 - 515 mics<br>Median: 3 mics <br>Average: 3 mics</div><div><br></div><div>openssl speed sha1<br>Doing sha1 for 3s on 16 size blocks: 39862496 sha1's in 3.00s<br>Doing sha1 for 3s on 64 size blocks: 25451866 sha1's in 3.00s<br>Doing sha1 for 3s on 256 size blocks: 13073739 sha1's in 3.00s<br>Doing sha1 for 3s on 1024 size blocks: 4463324 sha1's in 3.00s<br>Doing sha1 for 3s on 8192 size blocks: 622138 sha1's in 3.00s<br>Doing sha1 for 3s on 16384 size blocks: 314316 sha1's in 3.00s<br>OpenSSL 1.1.1 11 Sep 2018<br>built on: Tue Nov 12 16:58:35 2019 UTC<br>options:bn(64,64) rc4(8x,int) des(int) aes(partial) blowfish(ptr) <br>compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -Wa,--noexecstack -g -O2 -fdebug-prefix-map=/build/openssl-kxN_24/openssl-1.1.1=. -fstack-protector-strong -Wformat -Werror=format-security -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DRC4_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DX25519_ASM -DPADLOCK_ASM -DPOLY1305_ASM -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2<br>The 'numbers' are in 1000s of bytes per second processed.<br>type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes 16384 bytes<br>sha1 212599.98k 542973.14k 1115625.73k 1523481.26k 1698851.50k 1716584.45k<br><br>openssl speed -evp sha1<br>Doing sha1 for 3s on 16 size blocks: 17719869 sha1's in 3.00s<br>Doing sha1 for 3s on 64 size blocks: 14559842 sha1's in 3.00s<br>Doing sha1 for 3s on 256 size blocks: 9433054 sha1's in 3.00s<br>Doing sha1 for 3s on 1024 size blocks: 3938020 sha1's in 3.00s<br>Doing sha1 for 3s on 8192 size blocks: 607605 sha1's in 2.99s<br>Doing sha1 for 3s on 16384 size blocks: 309279 sha1's in 3.00s<br>OpenSSL 1.1.1 11 Sep 2018<br>built on: Tue Nov 12 16:58:35 2019 UTC<br>options:bn(64,64) rc4(8x,int) des(int) aes(partial) blowfish(ptr) <br>compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -Wa,--noexecstack -g -O2 -fdebug-prefix-map=/build/openssl-kxN_24/openssl-1.1.1=. -fstack-protector-strong -Wformat -Werror=format-security -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DRC4_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DX25519_ASM -DPADLOCK_ASM -DPOLY1305_ASM -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2<br>The 'numbers' are in 1000s of bytes per second processed.<br>type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes 16384 bytes<br>sha1 94505.97k 310609.96k 804953.94k 1344177.49k 1664715.77k 1689075.71k</div><div><br></div><div>[1] <a href="https://cloud.google.com/compute/docs/machine-types#n1_machine_type" target="_blank">https://cloud.google.com/compute/docs/machine-types#n1_machine_type</a></div><div>[2] <a href="https://cloud.google.com/compute/docs/machine-types#n2d_machine_types_beta" target="_blank">https://cloud.google.com/compute/docs/machine-types#n2d_machine_types_beta</a></div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Feb 20, 2020 at 4:34 PM Ben Browitt <<a href="mailto:ben.browitt@gmail.com" target="_blank">ben.browitt@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hope to test soon. AMD servers on GCP will probably be available in the next few days.</div><div>This is how I'm going to benchmark unless someone have a better suggestion:<br></div><div>Key = crypto:strong_rand_bytes(20),<br>Data = crypto:strong_rand_bytes(1000),<br>MacLength = 10,<br>TC = fun(TC_M, TC_F, TC_A, TC_N) when TC_N > 0 -> TC_L = tl([begin {TC_T, _Result} = timer:tc(TC_M, TC_F, TC_A), TC_T end || _ <- lists:seq(1, TC_N)]), TC_Min = lists:min(TC_L), TC_Max = lists:max(TC_L), TC_Med = lists:nth(round((TC_N - 1) / 2), lists:sort(TC_L)), TC_Avg = round(lists:foldl(fun(TC_X, TC_Sum) -> TC_X + TC_Sum end, 0, TC_L) / (TC_N - 1)), io:format("Range: ~b - ~b mics~nMedian: ~b mics ~nAverage: ~b mics~n", [TC_Min, TC_Max, TC_Med, TC_Avg]), TC_Med end.<br>TC(crypto, macN, [hmac, sha, Key, Data, MacLength], 1000).</div><div><br></div><div>And with:</div><div>openssl speed -evp sha1</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Feb 20, 2020 at 2:15 PM Hans Nilsson R <<a href="mailto:hans.r.nilsson@ericsson.com" target="_blank">hans.r.nilsson@ericsson.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0)">
Well, it isn't super clear in the release notes, so it is not strange you didn't know it.</div>
<div style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0)">
I'm VERY interested in the results of your benchmarking!</div>
<div style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0)">
/Hans<br>
</div>
<div id="gmail-m_7541480852848273965gmail-m_-4448496650523220771gmail-m_-5648249121371218052appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_7541480852848273965gmail-m_-4448496650523220771gmail-m_-5648249121371218052divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>Från:</b> Ben Browitt <<a href="mailto:ben.browitt@gmail.com" target="_blank">ben.browitt@gmail.com</a>><br>
<b>Skickat:</b> den 19 februari 2020 18:14<br>
<b>Till:</b> Hans Nilsson R <<a href="mailto:hans.r.nilsson@ericsson.com" target="_blank">hans.r.nilsson@ericsson.com</a>><br>
<b>Kopia:</b> <a href="mailto:zxq9@zxq9.com" target="_blank">zxq9@zxq9.com</a> <<a href="mailto:zxq9@zxq9.com" target="_blank">zxq9@zxq9.com</a>>; <a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a> <<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a>><br>
<b>Ämne:</b> Re: [erlang-questions] crypto:hmac/3 using hardware acceleration</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div>Thank you Hans, that's great.</div>
<div>I probably missed it in the release notes.</div>
<div>I'll benchmark and compare hmac on a server with and without sha hardware accelerations.<br>
</div>
</div>
<br>
<div>
<div dir="ltr">On Wed, Feb 19, 2020 at 5:18 PM Hans Nilsson R <<a href="mailto:hans.r.nilsson@ericsson.com" target="_blank">hans.r.nilsson@ericsson.com</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0)">
Crypto uses the EVP interfase for hash and mac (as well ass ciphers) with some conditions:</div>
<div style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0)">
Since OTP-22.1:<br>
</div>
<div style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0)">
The hash functions in crypto (hash,/2, hash_init/1, hash_update/2 and hash_final/1) use the EVP interface if the underlying cryptolib is OpenSSL 1.0.0 or higher.</div>
<div style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0)">
Since OTP-22.1.3:<br>
</div>
<div style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0)">
The mac functions (mac, macN, mac_init, mac_update, mac_final and mac_finalN) use the EVP interface if the underlying cryptolib is OpenSSL 1.1.1 or higher.</div>
<div style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0)">
/Hans<br>
</div>
<div id="gmail-m_7541480852848273965gmail-m_-4448496650523220771gmail-m_-5648249121371218052x_gmail-m_1110423570732962710appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_7541480852848273965gmail-m_-4448496650523220771gmail-m_-5648249121371218052x_gmail-m_1110423570732962710divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>Från:</b> erlang-questions <<a href="mailto:erlang-questions-bounces@erlang.org" target="_blank">erlang-questions-bounces@erlang.org</a>>
för Ben Browitt <<a href="mailto:ben.browitt@gmail.com" target="_blank">ben.browitt@gmail.com</a>><br>
<b>Skickat:</b> den 18 februari 2020 17:55<br>
<b>Till:</b> <a href="mailto:zxq9@zxq9.com" target="_blank">zxq9@zxq9.com</a> <<a href="mailto:zxq9@zxq9.com" target="_blank">zxq9@zxq9.com</a>><br>
<b>Kopia:</b> <a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a> <<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a>><br>
<b>Ämne:</b> Re: [erlang-questions] crypto:hmac/3 using hardware acceleration</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div>AWS [1] and GCP [2] provide AMD EPYC servers with SHA hardware accelerations.</div>
<div>Intel Ice Lake servers will also have SHA hardware accelerations [3].</div>
<div>Is there a chance OTP 23 could use EVP for SHA? This will give a large performance boost.<br>
</div>
<div><br>
</div>
<div>[1] <a href="https://aws.amazon.com/ec2/amd/" target="_blank">https://aws.amazon.com/ec2/amd/</a></div>
<div>[2] <a href="https://protect2.fireeye.com/v1/url?k=5f4e9246-03c459b2-5f4ed2dd-86cd58c48020-a6f86d945d59dea5&q=1&e=ae3465f0-e6c3-4143-ae82-51afb4cdbf8e&u=https%3A%2F%2Fcloud.google.com%2Fblog%2Fproducts%2Fcompute%2Fannouncing-the-n2d-vm-family-based-on-amd" target="_blank">
https://cloud.google.com/blog/products/compute/announcing-the-n2d-vm-family-based-on-amd</a></div>
<div>[3] <a href="https://en.wikipedia.org/wiki/Ice_Lake_(microprocessor)" target="_blank">
https://en.wikipedia.org/wiki/Ice_Lake_(microprocessor)</a></div>
</div>
<br>
<div>
<div dir="ltr">On Wed, May 8, 2019 at 4:34 PM <<a href="mailto:zxq9@zxq9.com" target="_blank">zxq9@zxq9.com</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
On 2019年5月8日水曜日 14時15分51秒 JST Ben Browitt wrote:<br>
> I've tested the speed with and without evp. evp is slower because Intel<br>
> cpus don't have hardware acceleration for sha.<br>
> So it's best to leave it without evp for now. Thanks.<br>
> openssl speed sha1<br>
> openssl speed -evp sha1<br>
<br>
I think it depends on how your openssl was built and which processor<br>
family you have. IIRC Intel has SHA1 hardware support, and AMD has<br>
SHA1 and SHA256 hardware instructions since RyZen.<br>
<br>
May also depend on if you are running virtualized and whether the<br>
hypervisor is exposing the instructions.<br>
<br>
In the base case I imagine it would "just work", but not if this is<br>
disabled in a vanilla Linux/BSD/whatever distribution binary, or if<br>
your system is set to a mode that restricts some instructions.<br>
<br>
-Craig<br>
_______________________________________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a><br>
<a href="http://erlang.org/mailman/listinfo/erlang-questions" rel="noreferrer" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote></div>
</blockquote></div>
</blockquote></div>