<div dir="ltr">Rikard and Jonas,<div><br></div><div>Thank you for your response. With the patch at OTP PR-2027, we do observe that the dirty CPU schedulers do not drop to zero utilization, but in addition we see that the performance of the traditional schedulers suffers. Specifically, when we apply our load test, scheduler usage toggles in between the following two states. I've snipped the output of similar values for readability.</div><div><br></div><span id="gmail-docs-internal-guid-f4cc6740-7fff-72b7-7b59-6f35786d661d"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">[{1,0.8933020805456553},</span></p><p style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><font color="#000000" face="Arial"><span style="white-space:pre-wrap">...</span></font></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"> {36,0.8882068476700309},</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"> {37,0.4932569692588096},</span></p><p style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><font color="#000000" face="Arial"><span style="white-space:pre-wrap">... </span></font></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"> {72,0.49842881346116297}]</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"><br></span></p><p style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">and</span></p><p style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><br></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">[{1,1.0},</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><font color="#000000" face="Arial"><span style="white-space:pre-wrap">...</span></font></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"> {36,1.0},</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"> {37,0.42909590032946493},</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><font color="#000000" face="Arial"><span style="white-space:pre-wrap">...</span></font></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"> {72,0.4641339106204525}]</span></p></span><div><br></div>When applying this same load to the un-patched Erlang 21.2, the traditional schedulers are utilized at about 75%. So it appears that the 2027 patch may put undesirable pressure on the schedulers as a trade-off for consistent performance from the dirty CPU schedulers. Are you observing a similar response on your system?<br class="gmail-Apple-interchange-newline"><div><br></div><div>Thanks,</div><div>Jesse</div></div><br><div class="gmail_quote"><div dir="ltr">On Wed, Jan 9, 2019 at 3:33 AM Jonas Falkevik <<a href="mailto:jonas.falkevik@mobilearts.com">jonas.falkevik@mobilearts.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi,<br>
This sounds like the issue I bumped into.<br>
After some time, many of the dirty schedulers are not performing any work anymore.<br>
Which was related to a corrupted list of sleeping dirty schedulers.<br>
<br>
I haven’t found what triggered it in my case (maybe a GC?). For me it took approx. 1 day before it happened.<br>
<br>
Doing Multi Scheduler Block can trigger the problem with the list of sleeping dirty schedulers which gets corrupted.<br>
The sleepers list can contain only one or some schedulers. That list is later used to wake up schedulers when there is work to do.<br>
For each enqueued dirty-cpu or dirty-io job, it wakes the first scheduler in the sleepers list.<br>
But if this list only point to a few schedulers only those are woken up to pick jobs from the queue.<br>
<br>
Since fixing that specific problem I have not seen the collapse of dirty schedulers anymore.<br>
<br>
It would be great if you can test and see if it fixes your problem as well, <a href="https://github.com/erlang/otp/pull/2027" rel="noreferrer" target="_blank">https://github.com/erlang/otp/pull/2027</a><br>
Any feedback on the bug fix is appreciated.<br>
<br>
/Jonas<br>
<br>
> On 8 Jan 2019, at 17:13, Jesse Stimpson <<a href="mailto:jstimpson@republicwireless.com" target="_blank">jstimpson@republicwireless.com</a>> wrote:<br>
> <br>
> Hello,<br>
> <br>
> We run a large scale WebRTC-based VOIP application, and currently we are evaluating whether or not dirty CPU schedulers can help us get more consistent performance with some critical NIFs. We've created a test that mimics our production load and have run it on AWS EC2 instances of various sizes. When dirty CPU schedulers are working as expected, we do see measurable improvement in performance, but unfortunately we have seen that they can drop to 0 utilization. They do not recover from this state.<br>
> <br>
> The timings from 2 of our 12 NIFs are in the vicinity of the 1 msec rule of thumb, so we have experimented with them both on and off dirty CPU schedulers. We've left the other NIFs on the traditional schedulers. More detailed NIF timings can be found below.<br>
> <br>
> At the end of this message is the concerning data (from recon:scheduler_usage/1).We have schedulers 1-36 as the traditional schedulers bound to the first hardware thread of each core, and schedulers 37-72 are dirty CPU schedulers. Based on mpstat it appears the dirty CPU schedulers are not bound to threads. After about 10 minutes of running the load test, schedulers 42, 43, 44, 61, 63, 64, 66, 68, 72 went to at 0 utilization and stayed there indefinitely. As time goes on, more dirty CPU schedulers enter this 0 utilization state. Schedulers 1-36 do not exhibit this behaviour. We have reproduced this consistently on other EC2 c5 instances of various sizes. <br>
> <br>
> We have tried various beam adjustments, but we have seen the same behaviour each time. Some examples of things we have tried:<br>
> +sbwt very_long +sbwtdcpu very_long +swt very_low +swtdcpu very_low<br>
> +sfwi 500<br>
> <br>
> Is there anything else that we can try to avoid this 0-utilization issue? We are happy to provide more data if needed.<br>
> <br>
> NIF: webrtc_acm_add_data_10ms <br>
> median: 606 usec<br>
> 90 percentile: 1230 usec<br>
> 99 percentile: 2285 usec<br>
> 99.9 percentile: 3223 usec<br>
> <br>
> NIF: webrtc_acm_playout_data_10ms<br>
> median: 98 usec<br>
> 90 percentile: 371 usec<br>
> 99 percentile: 410 usec<br>
> 99.9 percentile: 449 usec<br>
> <br>
> EC2 instance: c5.18xlarge<br>
> <br>
> OS: Linux 4.4.0-1073-aws #83-Ubuntu SMP Sat Nov 17 00:26:27 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux<br>
> <br>
> beam flags: +P 4194304 +sbt db +SP 50:50 +sub true +A 128 +K true <br>
> <br>
> Erlang banner: Erlang/OTP 21 [erts-10.2] [source] [64-bit] [smp:36:36] [ds:36:36:10] [async-threads:128] [hipe]<br>
> <br>
> cpu_topology:<br>
> [{node,[{processor,[{core,[{thread,{logical,0}},<br>
> {thread,{logical,36}}]},<br>
> {core,[{thread,{logical,1}},{thread,{logical,37}}]},<br>
> {core,[{thread,{logical,2}},{thread,{logical,38}}]},<br>
> {core,[{thread,{logical,3}},{thread,{logical,39}}]},<br>
> {core,[{thread,{logical,4}},{thread,{logical,40}}]},<br>
> {core,[{thread,{logical,5}},{thread,{logical,41}}]},<br>
> {core,[{thread,{logical,6}},{thread,{logical,42}}]},<br>
> {core,[{thread,{logical,7}},{thread,{logical,43}}]},<br>
> {core,[{thread,{logical,8}},{thread,{logical,44}}]},<br>
> {core,[{thread,{logical,9}},{thread,{logical,45}}]},<br>
> {core,[{thread,{logical,10}},{thread,{logical,46}}]},<br>
> {core,[{thread,{logical,11}},{thread,{logical,47}}]},<br>
> {core,[{thread,{logical,12}},{thread,{logical,48}}]},<br>
> {core,[{thread,{logical,13}},{thread,{logical,49}}]},<br>
> {core,[{thread,{logical,14}},{thread,{logical,50}}]},<br>
> {core,[{thread,{logical,15}},{thread,{logical,51}}]},<br>
> {core,[{thread,{logical,16}},{thread,{logical,52}}]},<br>
> {core,[{thread,{logical,17}},{thread,{logical,53}}]}]}]},<br>
> {node,[{processor,[{core,[{thread,{logical,18}},<br>
> {thread,{logical,54}}]},<br>
> {core,[{thread,{logical,19}},{thread,{logical,55}}]},<br>
> {core,[{thread,{logical,20}},{thread,{logical,56}}]},<br>
> {core,[{thread,{logical,21}},{thread,{logical,57}}]},<br>
> {core,[{thread,{logical,22}},{thread,{logical,58}}]},<br>
> {core,[{thread,{logical,23}},{thread,{logical,59}}]},<br>
> {core,[{thread,{logical,24}},{thread,{logical,60}}]},<br>
> {core,[{thread,{logical,25}},{thread,{logical,61}}]},<br>
> {core,[{thread,{logical,26}},{thread,{logical,62}}]},<br>
> {core,[{thread,{logical,27}},{thread,{logical,63}}]},<br>
> {core,[{thread,{logical,28}},{thread,{logical,64}}]},<br>
> {core,[{thread,{logical,29}},{thread,{logical,65}}]},<br>
> {core,[{thread,{logical,30}},{thread,{logical,66}}]},<br>
> {core,[{thread,{logical,31}},{thread,{logical,67}}]},<br>
> {core,[{thread,{logical,32}},{thread,{logical,68}}]},<br>
> {core,[{thread,{logical,33}},{thread,{logical,69}}]},<br>
> {core,[{thread,{logical,34}},{thread,{logical,70}}]},<br>
> {core,[{thread,{logical,35}},{thread,{logical,71}}]}]}]}]<br>
> <br>
> scheduler_bindings: {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,<br>
> 23,24,25,26,27,28,29,30,31,32,33,34,35}<br>
> <br>
> CPU utilization (from mpstat) during our load test:<br>
> Linux 4.4.0-1073-aws (isw-core-x-qa-awsoh02) 01/04/2019 _x86_64_ (72 CPU)<br>
> <br>
> 08:55:15 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle<br>
> 08:55:15 PM all 51.58 0.00 1.64 0.01 0.00 0.42 0.00 0.00 0.00 46.36<br>
> 08:55:15 PM 0 62.34 0.00 2.69 0.06 0.00 0.00 0.00 0.00 0.00 34.90<br>
> 08:55:15 PM 1 62.84 0.00 2.58 0.02 0.00 0.00 0.00 0.00 0.00 34.55<br>
> 08:55:15 PM 2 62.04 0.00 2.52 0.00 0.00 0.00 0.00 0.00 0.00 35.44<br>
> 08:55:15 PM 3 62.17 0.00 2.62 0.00 0.00 0.00 0.00 0.00 0.00 35.21<br>
> 08:55:15 PM 4 62.14 0.00 2.54 0.02 0.00 0.00 0.00 0.00 0.00 35.30<br>
> 08:55:15 PM 5 62.06 0.00 2.51 0.00 0.00 0.00 0.00 0.00 0.00 35.43<br>
> 08:55:15 PM 6 61.93 0.00 2.58 0.01 0.00 0.00 0.02 0.00 0.00 35.46<br>
> 08:55:15 PM 7 61.99 0.00 2.54 0.00 0.00 0.00 0.00 0.00 0.00 35.48<br>
> 08:55:15 PM 8 62.01 0.00 2.52 0.00 0.00 0.00 0.00 0.00 0.00 35.47<br>
> 08:55:15 PM 9 61.88 0.00 2.49 0.00 0.00 0.00 0.00 0.00 0.00 35.63<br>
> 08:55:15 PM 10 62.18 0.00 2.48 0.00 0.00 0.00 0.00 0.00 0.00 35.34<br>
> 08:55:15 PM 11 61.89 0.00 2.47 0.00 0.00 0.00 0.00 0.00 0.00 35.64<br>
> 08:55:15 PM 12 61.98 0.00 2.50 0.00 0.00 0.00 0.00 0.00 0.00 35.52<br>
> 08:55:15 PM 13 61.80 0.00 2.54 0.00 0.00 0.00 0.00 0.00 0.00 35.66<br>
> 08:55:15 PM 14 61.96 0.00 2.56 0.00 0.00 0.00 0.00 0.00 0.00 35.48<br>
> 08:55:15 PM 15 62.11 0.00 2.57 0.00 0.00 0.00 0.00 0.00 0.00 35.31<br>
> 08:55:15 PM 16 62.03 0.00 2.55 0.00 0.00 0.00 0.00 0.00 0.00 35.42<br>
> 08:55:15 PM 17 61.90 0.00 2.53 0.00 0.00 0.00 0.00 0.00 0.00 35.57<br>
> 08:55:15 PM 18 58.36 0.00 2.41 0.09 0.00 3.72 0.00 0.00 0.00 35.43<br>
> 08:55:15 PM 19 58.29 0.00 2.45 0.02 0.00 4.00 0.00 0.00 0.00 35.24<br>
> 08:55:15 PM 20 58.45 0.00 2.36 0.02 0.00 3.73 0.00 0.00 0.00 35.44<br>
> 08:55:15 PM 21 58.56 0.00 2.34 0.04 0.00 3.54 0.00 0.00 0.00 35.51<br>
> 08:55:15 PM 22 58.12 0.00 2.46 0.00 0.00 3.84 0.00 0.00 0.00 35.58<br>
> 08:55:15 PM 23 58.24 0.00 2.36 0.01 0.00 3.82 0.00 0.00 0.00 35.57<br>
> 08:55:15 PM 24 58.84 0.00 2.42 0.00 0.00 3.60 0.00 0.00 0.00 35.14<br>
> 08:55:15 PM 25 58.35 0.00 2.43 0.00 0.00 3.72 0.00 0.00 0.00 35.51<br>
> 08:55:15 PM 26 62.09 0.00 2.47 0.00 0.00 0.00 0.00 0.00 0.00 35.45<br>
> 08:55:15 PM 27 62.09 0.00 2.46 0.00 0.00 0.00 0.00 0.00 0.00 35.45<br>
> 08:55:15 PM 28 62.15 0.00 2.48 0.00 0.00 0.00 0.00 0.00 0.00 35.37<br>
> 08:55:15 PM 29 62.09 0.00 2.46 0.00 0.00 0.00 0.00 0.00 0.00 35.45<br>
> 08:55:15 PM 30 62.74 0.00 2.45 0.00 0.00 0.00 0.00 0.00 0.00 34.81<br>
> 08:55:15 PM 31 62.14 0.00 2.47 0.00 0.00 0.00 0.00 0.00 0.00 35.39<br>
> 08:55:15 PM 32 62.35 0.00 2.52 0.00 0.00 0.00 0.00 0.00 0.00 35.13<br>
> 08:55:15 PM 33 62.13 0.00 2.42 0.00 0.00 0.00 0.00 0.00 0.00 35.46<br>
> 08:55:15 PM 34 62.08 0.00 2.47 0.00 0.00 0.00 0.00 0.00 0.00 35.45<br>
> 08:55:15 PM 35 62.15 0.00 2.49 0.00 0.00 0.00 0.00 0.00 0.00 35.35<br>
> 08:55:15 PM 36 43.19 0.00 0.80 0.00 0.00 0.00 0.00 0.00 0.00 56.01<br>
> 08:55:15 PM 37 43.09 0.00 0.83 0.01 0.00 0.00 0.00 0.00 0.00 56.06<br>
> 08:55:15 PM 38 42.59 0.00 0.78 0.01 0.00 0.00 0.00 0.00 0.00 56.63<br>
> 08:55:15 PM 39 42.60 0.00 0.82 0.00 0.00 0.00 0.00 0.00 0.00 56.58<br>
> 08:55:15 PM 40 42.94 0.00 0.76 0.02 0.00 0.00 0.00 0.00 0.00 56.28<br>
> 08:55:15 PM 41 42.11 0.00 0.76 0.01 0.00 0.00 0.00 0.00 0.00 57.13<br>
> 08:55:15 PM 42 43.26 0.00 0.85 0.01 0.00 0.00 0.00 0.00 0.00 55.88<br>
> 08:55:15 PM 43 42.23 0.00 0.77 0.01 0.00 0.00 0.00 0.00 0.00 56.99<br>
> 08:55:15 PM 44 42.24 0.00 0.80 0.00 0.00 0.00 0.00 0.00 0.00 56.95<br>
> 08:55:15 PM 45 41.65 0.00 0.79 0.01 0.00 0.00 0.00 0.00 0.00 57.55<br>
> 08:55:15 PM 46 41.76 0.00 0.77 0.01 0.00 0.00 0.00 0.00 0.00 57.46<br>
> 08:55:15 PM 47 42.66 0.00 0.82 0.00 0.00 0.00 0.00 0.00 0.00 56.52<br>
> 08:55:15 PM 48 41.83 0.00 0.76 0.01 0.00 0.00 0.00 0.00 0.00 57.40<br>
> 08:55:15 PM 49 42.04 0.00 0.82 0.00 0.00 0.00 0.00 0.00 0.00 57.13<br>
> 08:55:15 PM 50 41.55 0.00 0.78 0.01 0.00 0.00 0.00 0.00 0.00 57.67<br>
> 08:55:15 PM 51 42.37 0.00 0.78 0.00 0.00 0.00 0.00 0.00 0.00 56.84<br>
> 08:55:15 PM 52 41.66 0.00 0.77 0.00 0.00 0.00 0.00 0.00 0.00 57.57<br>
> 08:55:15 PM 53 41.77 0.00 0.79 0.00 0.00 0.00 0.00 0.00 0.00 57.45<br>
> 08:55:15 PM 54 40.33 0.00 0.88 0.01 0.00 0.00 0.00 0.00 0.00 58.79<br>
> 08:55:15 PM 55 41.62 0.00 0.78 0.01 0.00 0.00 0.00 0.00 0.00 57.59<br>
> 08:55:15 PM 56 41.51 0.00 0.82 0.00 0.00 0.00 0.00 0.00 0.00 57.67<br>
> 08:55:15 PM 57 40.35 0.00 0.76 0.00 0.00 0.00 0.00 0.00 0.00 58.89<br>
> 08:55:15 PM 58 41.92 0.00 0.79 0.01 0.00 0.00 0.00 0.00 0.00 57.28<br>
> 08:55:15 PM 59 41.48 0.00 0.80 0.00 0.00 0.00 0.00 0.00 0.00 57.72<br>
> 08:55:15 PM 60 40.83 0.00 0.75 0.01 0.00 0.00 0.00 0.00 0.00 58.41<br>
> 08:55:15 PM 61 40.95 0.00 0.78 0.01 0.00 0.00 0.00 0.00 0.00 58.26<br>
> 08:55:15 PM 62 40.85 0.00 0.80 0.01 0.00 0.00 0.00 0.00 0.00 58.35<br>
> 08:55:15 PM 63 42.11 0.00 0.77 0.01 0.00 0.00 0.00 0.00 0.00 57.12<br>
> 08:55:15 PM 64 42.22 0.00 0.78 0.01 0.00 0.00 0.00 0.00 0.00 56.99<br>
> 08:55:15 PM 65 41.55 0.00 0.78 0.00 0.00 0.00 0.00 0.00 0.00 57.67<br>
> 08:55:15 PM 66 41.68 0.00 0.81 0.00 0.00 0.00 0.00 0.00 0.00 57.51<br>
> 08:55:15 PM 67 41.06 0.00 0.87 0.01 0.00 0.00 0.00 0.00 0.00 58.07<br>
> 08:55:15 PM 68 41.80 0.00 0.79 0.00 0.00 0.00 0.00 0.00 0.00 57.41<br>
> 08:55:15 PM 69 41.38 0.00 0.75 0.00 0.00 0.00 0.00 0.00 0.00 57.87<br>
> 08:55:15 PM 70 42.17 0.00 0.75 0.00 0.00 0.00 0.00 0.00 0.00 57.08<br>
> 08:55:15 PM 71 41.26 0.00 0.81 0.00 0.00 0.00 0.01 0.00 0.00 57.92<br>
> <br>
> <br>
> 1> rp(recon:scheduler_usage(1000)).<br>
> [{1,0.8581586716045433},<br>
> {2,0.8607597847736659},<br>
> {3,0.8501543449289042},<br>
> {4,0.8619718426658023},<br>
> {5,0.8660226578705742},<br>
> {6,0.8555763933364806},<br>
> {7,0.8557195972097537},<br>
> {8,0.8668284625654983},<br>
> {9,0.858588675506532},<br>
> {10,0.8650501369640686},<br>
> {11,0.8615557214275474},<br>
> {12,0.8523392789683996},<br>
> {13,0.8469329348407398},<br>
> {14,0.8615637534991641},<br>
> {15,0.8591930739986557},<br>
> {16,0.8578596164201188},<br>
> {17,0.8549696086049403},<br>
> {18,0.8677098230685564},<br>
> {19,0.8556328340054443},<br>
> {20,0.8642542459479063},<br>
> {21,0.8670655179064826},<br>
> {22,0.8562638308551557},<br>
> {23,0.8511202058055239},<br>
> {24,0.8697339264798696},<br>
> {25,0.8594823462601853},<br>
> {26,0.8634550008001265},<br>
> {27,0.8568590130172198},<br>
> {28,0.864025757952233},<br>
> {29,0.8575614770622326},<br>
> {30,0.8611760239143281},<br>
> {31,0.8589789125669829},<br>
> {32,0.8537754468447621},<br>
> {33,0.8675538912040871},<br>
> {34,0.8607522803519515},<br>
> {35,0.8639389099445753},<br>
> {36,0.8743759133171144},<br>
> {37,0.9526507413773687},<br>
> {38,0.948925611917577},<br>
> {39,0.9511904612810083},<br>
> {40,0.9581015421004746},<br>
> {41,0.9534977201467516},<br>
> {42,0.0},<br>
> {43,0.0},<br>
> {44,0.0},<br>
> {45,0.9522518499948341},<br>
> {46,0.9453306755503674},<br>
> {47,0.9586374601140047},<br>
> {48,0.9574611845671873},<br>
> {49,0.950914304057791},<br>
> {50,0.9582124294398642},<br>
> {51,0.9448074463900029},<br>
> {52,0.9474771999511407},<br>
> {53,0.9450778098401229},<br>
> {54,0.0},<br>
> {55,0.9478167911711937},<br>
> {56,0.9612587860764551},<br>
> {57,0.943468696358506},<br>
> {58,0.9509498642900455},<br>
> {59,0.9603595134495414},<br>
> {60,0.9517397912369205},<br>
> {61,0.0},<br>
> {62,0.9587029756970658},<br>
> {63,0.0},<br>
> {64,0.0},<br>
> {65,0.9526167902065421},<br>
> {66,0.0},<br>
> {67,0.9475668680715805},<br>
> {68,0.0},<br>
> {69,0.9630301804839606},<br>
> {70,0.9475359592879946},<br>
> {71,0.9541718747681374},<br>
> {72,0.0}]<br>
> <br>
> <br>
> Thanks for your time,<br>
> <br>
> Jesse Stimpson<br>
> _______________________________________________<br>
> erlang-questions mailing list<br>
> <a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a><br>
> <a href="http://erlang.org/mailman/listinfo/erlang-questions" rel="noreferrer" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
<br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><span><div dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><table style="font-size:12.8px;border:none;border-collapse:collapse"><tbody><tr style="height:0px"><td style="border-width:0px 1px 0px 0px;border-style:solid;border-color:rgb(0,0,0) rgb(204,204,204) rgb(0,0,0) rgb(0,0,0);vertical-align:top;padding:0px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><a href="http://www.republicwireless.com/" style="color:rgb(17,85,204);text-decoration:none" target="_blank"><span style="font-size:14.6667px;font-family:Arial;background-color:transparent;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap"><img src="https://lh6.googleusercontent.com/OtQkJgG7cQsvQNGkkWBvl6NRdkaV7AlHeBvQeodb-MDWGY8uq3S6KESFRH6U1dN0-jIXYqrApf_ed4sYYvlmgenTmlizLyUGM9XFjzcS97_3AMyFf-78bzNS2yxkdABXzR3VhwWZul8N0H341w" width="105" height="31" style="border: none;"></span></a></p></td><td style="border-width:0px 0px 0px 1px;border-style:solid;border-color:rgb(0,0,0) rgb(0,0,0) rgb(0,0,0) rgb(204,204,204);vertical-align:top;padding:0px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt;margin-left:9.75pt"><span style="font-size:13.3333px;font-family:Arial;color:rgb(68,68,68);background-color:transparent;font-weight:700;vertical-align:baseline;white-space:pre-wrap">Jesse Stimpson</span></p><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt;margin-left:9.75pt"><span style="font-size:13.3333px;font-family:Arial;color:rgb(153,153,153);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Site Reliability Engineering</span></p><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt;margin-left:9.75pt"><span style="font-size:13.3333px;font-family:Arial;color:rgb(153,153,153);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">m: </span><span style="font-size:13.3333px;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><font color="#1155cc"><a href="tel:(919)%20995-0424" value="+19192971077" style="color:rgb(17,85,204)" target="_blank">9199950424</a></font></span><span style="font-size:13.3333px;font-family:Arial;color:rgb(17,85,204);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><br></span><span style="color:rgb(17,85,204);text-decoration:underline;font-size:13.3333px;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><a href="https://republicwireless.com/" style="color:rgb(17,85,204);text-decoration:none" target="_blank">RepublicWireless.com</a></span></p></td></tr></tbody></table></div></span><div><div dir="ltr"></div></div></div></div></div></div>