Big RAM Erlang stories/wisdom/anecdotes?

Scott Lystig Fritchie fritchie@REDACTED
Sun Sep 27 22:40:01 CEST 2009 

Greetings, all.  If all goes according to plan, I'll soon be getting my
hands on some 64GB RAM, 64-bit boxes to work with.  Their Erlang apps
will be using a large fraction of that RAM to store stuff in ETS
ordered_set tables.

Erlang process heap growth is much jumpier.  As noted by Ulf Wiger and
Erik Stenman on this list(*), an Erlang process's heap grows in steps
based on the Fibonacci sequence.  Looking at erl_gc.c, the sequence
starts at 34(***).

Actually, it stops being Fibonacci-based at index 22, 1346269 words.
>From then, it's based on 20% more than the previous index.

My Erlang apps will rarely have individual binaries larger than 4MB, and
we try very hard to create really big process heaps or really big terms,
e.g. list-of-ASCII-byte error messages created by the default
error_logger, which we don't use.

Does anyone have war stories about big RAM machines (more than 16GB,
for instance) + Erlang where you wish that the heap steps were a bit
smaller?  Once you're dealing with 1GB or more, "merely 20% more" ends
up being a pretty big number when talking about RAM allocation.

Or wisdom of past development experiences?  Or anecdotes?

I'm pondering playing with heap sizing in a couple ways:

    1. What if growth beyond index 46 (?) were smaller than 20%?

    2. What if a erts_gc.c:minor_collection() didn't automatically
       bump up to the next heap size?  Any idea of the comment below
       is still relevant today?

    /*
     * Allocate an old heap if we don't have one and if we'll need one.
     */

    if (OLD_HEAP(p) == NULL && mature != 0) {
        Eterm* n_old;

        /* Note: We choose a larger heap size than strictly needed,
         * which seems to reduce the number of fullsweeps.
         * This improved Estone by more than 1200 estones on my computer
         * (Ultra Sparc 10).
         */

Hrm, I've never tried calculating an Estone.  Maybe I should try.  :-)

-Scott

(*) And perhaps by others, sorry if my grep'ing didn't find your
postings.

(**) When putting millions of things into an ETS table via
        something simple like:

  [ets:insert(Tab, {X, B1, B2, B3, B4}) || X <- lists:seq(1, 10*1024*1024)].

       ... I see the VM's OS process size grow is small steps.  Yay.
       B1, B2, B3, and B4 are 63 byte binaries, if anyone cares.

       Yes, I've considered avoiding ETS and using a C driver, like
       Berkeley DB or Tokyo Cabinet.  I haven't seriously thought about
       it yet, but if they're more RAM-efficient by a factor of only 2,
       then that's only ... a factor of 2.  {shrug}

(***): Max heap size for 32-bit environments actually stops at
       index 53 or index 54, I forget which.
       Megabyte calculations are truncated, probably.
       Based on erl_gc.c:erts_init_gc().

index   0, heap size (words): 34 (0 MB @ 32bit, 0 MB @ 64bit)
index   1, heap size (words): 55 (0 MB @ 32bit, 0 MB @ 64bit)
index   2, heap size (words): 89 (0 MB @ 32bit, 0 MB @ 64bit)
index   3, heap size (words): 144 (0 MB @ 32bit, 0 MB @ 64bit)
index   4, heap size (words): 233 (0 MB @ 32bit, 0 MB @ 64bit)
index   5, heap size (words): 377 (0 MB @ 32bit, 0 MB @ 64bit)
index   6, heap size (words): 610 (0 MB @ 32bit, 0 MB @ 64bit)
index   7, heap size (words): 987 (0 MB @ 32bit, 0 MB @ 64bit)
index   8, heap size (words): 1597 (0 MB @ 32bit, 0 MB @ 64bit)
index   9, heap size (words): 2584 (0 MB @ 32bit, 0 MB @ 64bit)
index  10, heap size (words): 4181 (0 MB @ 32bit, 0 MB @ 64bit)
index  11, heap size (words): 6765 (0 MB @ 32bit, 0 MB @ 64bit)
index  12, heap size (words): 10946 (0 MB @ 32bit, 0 MB @ 64bit)
index  13, heap size (words): 17711 (0 MB @ 32bit, 0 MB @ 64bit)
index  14, heap size (words): 28657 (0 MB @ 32bit, 0 MB @ 64bit)
index  15, heap size (words): 46368 (0 MB @ 32bit, 0 MB @ 64bit)
index  16, heap size (words): 75025 (0 MB @ 32bit, 0 MB @ 64bit)
index  17, heap size (words): 121393 (0 MB @ 32bit, 0 MB @ 64bit)
index  18, heap size (words): 196418 (0 MB @ 32bit, 1 MB @ 64bit)
index  19, heap size (words): 317811 (1 MB @ 32bit, 2 MB @ 64bit)
index  20, heap size (words): 514229 (1 MB @ 32bit, 3 MB @ 64bit)
index  21, heap size (words): 832040 (3 MB @ 32bit, 6 MB @ 64bit)
index  22, heap size (words): 1346269 (5 MB @ 32bit, 10 MB @ 64bit)
index  23, heap size (words): 1682835 (6 MB @ 32bit, 12 MB @ 64bit)
index  24, heap size (words): 2103540 (8 MB @ 32bit, 16 MB @ 64bit)
index  25, heap size (words): 2629425 (10 MB @ 32bit, 20 MB @ 64bit)
index  26, heap size (words): 3286780 (12 MB @ 32bit, 25 MB @ 64bit)
index  27, heap size (words): 4108475 (15 MB @ 32bit, 31 MB @ 64bit)
index  28, heap size (words): 5135590 (19 MB @ 32bit, 39 MB @ 64bit)
index  29, heap size (words): 6419485 (24 MB @ 32bit, 48 MB @ 64bit)
index  30, heap size (words): 8024355 (30 MB @ 32bit, 61 MB @ 64bit)
index  31, heap size (words): 10030440 (38 MB @ 32bit, 76 MB @ 64bit)
index  32, heap size (words): 12538050 (47 MB @ 32bit, 95 MB @ 64bit)
index  33, heap size (words): 15672560 (59 MB @ 32bit, 119 MB @ 64bit)
index  34, heap size (words): 19590700 (74 MB @ 32bit, 149 MB @ 64bit)
index  35, heap size (words): 24488375 (93 MB @ 32bit, 186 MB @ 64bit)
index  36, heap size (words): 30610465 (116 MB @ 32bit, 233 MB @ 64bit)
index  37, heap size (words): 38263080 (145 MB @ 32bit, 291 MB @ 64bit)
index  38, heap size (words): 47828850 (182 MB @ 32bit, 364 MB @ 64bit)
index  39, heap size (words): 59786060 (228 MB @ 32bit, 456 MB @ 64bit)
index  40, heap size (words): 74732575 (285 MB @ 32bit, 570 MB @ 64bit)
index  41, heap size (words): 93415715 (356 MB @ 32bit, 712 MB @ 64bit)
index  42, heap size (words): 116769640 (445 MB @ 32bit, 890 MB @ 64bit)
index  43, heap size (words): 145962050 (556 MB @ 32bit, 1113 MB @ 64bit)
index  44, heap size (words): 182452560 (696 MB @ 32bit, 1392 MB @ 64bit)
index  45, heap size (words): 228065700 (870 MB @ 32bit, 1740 MB @ 64bit)
index  46, heap size (words): 285082125 (1087 MB @ 32bit, 2175 MB @ 64bit)
index  47, heap size (words): 356352655 (1359 MB @ 32bit, 2718 MB @ 64bit)
index  48, heap size (words): 445440815 (1699 MB @ 32bit, 3398 MB @ 64bit)
index  49, heap size (words): 556801015 (2124 MB @ 32bit, 4248 MB @ 64bit)
index  50, heap size (words): 696001265 (2655 MB @ 32bit, 5310 MB @ 64bit)
index  51, heap size (words): 870001580 (3318 MB @ 32bit, 6637 MB @ 64bit)
index  52, heap size (words): 1087501975 (4148 MB @ 32bit, 8296 MB @ 64bit)
index  53, heap size (words): 1359377465 (5185 MB @ 32bit, 10371 MB @ 64bit)
index  54, heap size (words): 1699221830 (6482 MB @ 32bit, 12964 MB @ 64bit)
index  55, heap size (words): 2124027285 (8102 MB @ 32bit, 16205 MB @ 64bit)
index  56, heap size (words): 2655034105 (10128 MB @ 32bit, 20256 MB @ 64bit)
index  57, heap size (words): 3318792630 (12660 MB @ 32bit, 25320 MB @ 64bit)
index  58, heap size (words): 4148490785 (15825 MB @ 32bit, 31650 MB @ 64bit)
index  59, heap size (words): 5185613480 (19781 MB @ 32bit, 39563 MB @ 64bit)
index  60, heap size (words): 6482016850 (24726 MB @ 32bit, 49453 MB @ 64bit)
index  61, heap size (words): 8102521060 (30908 MB @ 32bit, 61817 MB @ 64bit)
index  62, heap size (words): 10128151325 (38635 MB @ 32bit, 77271 MB @ 64bit)
index  63, heap size (words): 12660189155 (48294 MB @ 32bit, 96589 MB @ 64bit)
index  64, heap size (words): 15825236440 (60368 MB @ 32bit, 120736 MB @ 64bit)
index  65, heap size (words): 19781545550 (75460 MB @ 32bit, 150921 MB @ 64bit)
index  66, heap size (words): 24726931935 (94325 MB @ 32bit, 188651 MB @ 64bit)
index  67, heap size (words): 30908664915 (117907 MB @ 32bit, 235814 MB @ 64bit)
index  68, heap size (words): 38635831140 (147383 MB @ 32bit, 294767 MB @ 64bit)
index  69, heap size (words): 48294788925 (184229 MB @ 32bit, 368459 MB @ 64bit)
index  70, heap size (words): 60368486155 (230287 MB @ 32bit, 460574 MB @ 64bit)
index  71, heap size (words): 75460607690 (287859 MB @ 32bit, 575718 MB @ 64bit)
index  72, heap size (words): 94325759610 (359824 MB @ 32bit, 719648 MB @ 64bit)
index  73, heap size (words): 117907199510 (449780 MB @ 32bit, 899560 MB @ 64bit)
index  74, heap size (words): 147383999385 (562225 MB @ 32bit, 1124450 MB @ 64bit)
index  75, heap size (words): 184229999230 (702781 MB @ 32bit, 1405563 MB @ 64bit)
index  76, heap size (words): 230287499035 (878477 MB @ 32bit, 1756954 MB @ 64bit)
index  77, heap size (words): 287859373790 (1098096 MB @ 32bit, 2196192 MB @ 64bit)
index  78, heap size (words): 359824217235 (1372620 MB @ 32bit, 2745240 MB @ 64bit)
index  79, heap size (words): 449780271540 (1715775 MB @ 32bit, 3431551 MB @ 64bit)
index  80, heap size (words): 562225339425 (2144719 MB @ 32bit, 4289438 MB @ 64bit)
index  81, heap size (words): 702781674280 (2680899 MB @ 32bit, 5361798 MB @ 64bit)
index  82, heap size (words): 878477092850 (3351124 MB @ 32bit, 6702248 MB @ 64bit)
index  83, heap size (words): 1098096366060 (4188905 MB @ 32bit, 8377810 MB @ 64bit)
index  84, heap size (words): 1372620457575 (5236131 MB @ 32bit, 10472263 MB @ 64bit)
index  85, heap size (words): 1715775571965 (6545164 MB @ 32bit, 13090328 MB @ 64bit)
index  86, heap size (words): 2144719464955 (8181455 MB @ 32bit, 16362910 MB @ 64bit)
index  87, heap size (words): 2680899331190 (10226819 MB @ 32bit, 20453638 MB @ 64bit)
index  88, heap size (words): 3351124163985 (12783524 MB @ 32bit, 25567048 MB @ 64bit)
index  89, heap size (words): 4188905204980 (15979405 MB @ 32bit, 31958810 MB @ 64bit)
index  90, heap size (words): 5236131506225 (19974256 MB @ 32bit, 39948513 MB @ 64bit)
index  91, heap size (words): 6545164382780 (24967820 MB @ 32bit, 49935641 MB @ 64bit)
index  92, heap size (words): 8181455478475 (31209775 MB @ 32bit, 62419551 MB @ 64bit)
index  93, heap size (words): 10226819348090 (39012219 MB @ 32bit, 78024439 MB @ 64bit)
index  94, heap size (words): 12783524185110 (48765274 MB @ 32bit, 97530549 MB @ 64bit)
index  95, heap size (words): 15979405231385 (60956593 MB @ 32bit, 121913186 MB @ 64bit)
index  96, heap size (words): 19974256539230 (76195741 MB @ 32bit, 152391483 MB @ 64bit)
index  97, heap size (words): 24967820674035 (95244677 MB @ 32bit, 190489354 MB @ 64bit)
index  98, heap size (words): 31209775842540 (119055846 MB @ 32bit, 238111693 MB @ 64bit)
index  99, heap size (words): 39012219803175 (148819808 MB @ 32bit, 297639616 MB @ 64bit)
index 100, heap size (words): 48765274753965 (186024760 MB @ 32bit, 372049520 MB @ 64bit)
index 101, heap size (words): 60956593442455 (232530950 MB @ 32bit, 465061900 MB @ 64bit)
index 102, heap size (words): 76195741803065 (290663687 MB @ 32bit, 581327375 MB @ 64bit)
index 103, heap size (words): 95244677253830 (363329609 MB @ 32bit, 726659219 MB @ 64bit)
index 104, heap size (words): 119055846567285 (454162012 MB @ 32bit, 908324024 MB @ 64bit)
index 105, heap size (words): 148819808209105 (567702515 MB @ 32bit, 1135405030 MB @ 64bit)
index 106, heap size (words): 186024760261380 (709628144 MB @ 32bit, 1419256288 MB @ 64bit)
index 107, heap size (words): 232530950326725 (887035180 MB @ 32bit, 1774070360 MB @ 64bit)
index 108, heap size (words): 290663687908405 (1108793975 MB @ 32bit, 2217587950 MB @ 64bit)
index 109, heap size (words): 363329609885505 (1385992469 MB @ 32bit, 2771984938 MB @ 64bit)
index 110, heap size (words): 454162012356880 (1732490586 MB @ 32bit, 3464981173 MB @ 64bit)
index 111, heap size (words): 567702515446100 (2165613233 MB @ 32bit, 4331226466 MB @ 64bit)
index 112, heap size (words): 709628144307625 (2707016541 MB @ 32bit, 5414033083 MB @ 64bit)
index 113, heap size (words): 887035180384530 (3383770677 MB @ 32bit, 6767541354 MB @ 64bit)
index 114, heap size (words): 1108793975480660 (4229713346 MB @ 32bit, 8459426692 MB @ 64bit)
index 115, heap size (words): 1385992469350825 (5287141683 MB @ 32bit, 10574283366 MB @ 64bit)
index 116, heap size (words): 1732490586688530 (6608927103 MB @ 32bit, 13217854207 MB @ 64bit)
index 117, heap size (words): 2165613233360660 (8261158879 MB @ 32bit, 16522317759 MB @ 64bit)
index 118, heap size (words): 2707016541700825 (10326448599 MB @ 32bit, 20652897199 MB @ 64bit)
index 119, heap size (words): 3383770677126030 (12908060749 MB @ 32bit, 25816121499 MB @ 64bit)
index 120, heap size (words): 4229713346407535 (16135075936 MB @ 32bit, 32270151873 MB @ 64bit)
index 121, heap size (words): 5287141683009415 (20168844921 MB @ 32bit, 40337689842 MB @ 64bit)
index 122, heap size (words): 6608927103761765 (25211056151 MB @ 32bit, 50422112302 MB @ 64bit)
index 123, heap size (words): 8261158879702205 (31513820189 MB @ 32bit, 63027640378 MB @ 64bit)
index 124, heap size (words): 10326448599627755 (39392275236 MB @ 32bit, 78784550473 MB @ 64bit)
index 125, heap size (words): 12908060749534690 (49240344045 MB @ 32bit, 98480688091 MB @ 64bit)
index 126, heap size (words): 16135075936918360 (61550430057 MB @ 32bit, 123100860114 MB @ 64bit)
index 127, heap size (words): 20168844921147950 (76938037571 MB @ 32bit, 153876075143 MB @ 64bit)
index 128, heap size (words): 25211056151434935 (96172546964 MB @ 32bit, 192345093928 MB @ 64bit)
index 129, heap size (words): 31513820189293665 (120215683705 MB @ 32bit, 240431367410 MB @ 64bit)
index 130, heap size (words): 39392275236617080 (150269604631 MB @ 32bit, 300539209263 MB @ 64bit)
index 131, heap size (words): 49240344045771350 (187837005789 MB @ 32bit, 375674011579 MB @ 64bit)
index 132, heap size (words): 61550430057214185 (234796257237 MB @ 32bit, 469592514474 MB @ 64bit)
index 133, heap size (words): 76938037571517730 (293495321546 MB @ 32bit, 586990643093 MB @ 64bit)
index 134, heap size (words): 96172546964397160 (366869151933 MB @ 32bit, 733738303866 MB @ 64bit)
index 135, heap size (words): 120215683705496450 (458586439916 MB @ 32bit, 917172879833 MB @ 64bit)
index 136, heap size (words): 150269604631870560 (573233049895 MB @ 32bit, 1146466099791 MB @ 64bit)
index 137, heap size (words): 187837005789838200 (716541312369 MB @ 32bit, 1433082624739 MB @ 64bit)
index 138, heap size (words): 234796257237297750 (895676640462 MB @ 32bit, 1791353280924 MB @ 64bit)
index 139, heap size (words): 293495321546622185 (1119595800577 MB @ 32bit, 2239191601155 MB @ 64bit)
index 140, heap size (words): 366869151933277730 (1399494750722 MB @ 32bit, 2798989501444 MB @ 64bit)
index 141, heap size (words): 458586439916597160 (1749368438402 MB @ 32bit, 3498736876805 MB @ 64bit)
index 142, heap size (words): 573233049895746450 (2186710548003 MB @ 32bit, 4373421096006 MB @ 64bit)
index 143, heap size (words): 716541312369683060 (2733388185003 MB @ 32bit, 5466776370007 MB @ 64bit)
index 144, heap size (words): 895676640462103825 (3416735231254 MB @ 32bit, 6833470462509 MB @ 64bit)
index 145, heap size (words): 1119595800577629780 (4270919039068 MB @ 32bit, 8541838078137 MB @ 64bit)
index 146, heap size (words): 1399494750722037225 (5338648798835 MB @ 32bit, -6914888446744 MB @ 64bit)
index 147, heap size (words): 1749368438402546530 (6673310998544 MB @ 32bit, -4245564047326 MB @ 64bit)
index 148, heap size (words): 2186710548003183160 (8341638748181 MB @ 32bit, -908908548053 MB @ 64bit)
index 149, heap size (words): 2733388185003978950 (-7165137609189 MB @ 32bit, 3261910826036 MB @ 64bit)
index 150, heap size (words): 3416735231254973685 (-4558375500383 MB @ 32bit, 8475435043649 MB @ 64bit)
index 151, heap size (words): 4270919039068717105 (-1299922864374 MB @ 32bit, -2599845728749 MB @ 64bit)
index 152, heap size (words): 5338648798835896380 (2773142930635 MB @ 32bit, 5546285861270 MB @ 64bit)
index 153, heap size (words): 6673310998544870475 (7864475174398 MB @ 32bit, -1863235695619 MB @ 64bit)

More information about the erlang-questions mailing list