<br><br><div class="gmail_quote">On Thu, Jan 3, 2013 at 12:25 AM, J K <span dir="ltr"><<a href="mailto:jmakarlsson@yahoo.com" target="_blank">jmakarlsson@yahoo.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div><div style="font-size:12pt;font-family:times new roman,new york,times,serif"><div><span>Hi,</span></div><div style="font-style:normal;font-size:16px;background-color:transparent;font-family:'times new roman','new york',times,serif">
<span><br></span></div><div style="font-style:normal;font-size:16px;background-color:transparent;font-family:'times new roman','new york',times,serif"><span>I'm not sure I understand your 20.000 files example.</span></div>
</div></div></blockquote><div><br></div><div><font color="#333333" face="Verdana"><span style="line-height:14px"><i>This was just to give an estimate of the total memory size</i></span></font></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div><div style="font-size:12pt;font-family:times new roman,new york,times,serif"><div style="font-style:normal;font-size:16px;background-color:transparent;font-family:'times new roman','new york',times,serif">
<span> Are you suggesting that the user should limit the number of erlang processes to the number of cores or are you suggesting that the VM compresses the erlang process data when not running? </span></div></div></div></blockquote>
<div><br></div><div>You could experiment with limiting the number of active processes and</div><div>compressing data when it's not being used (you'd have to do this yourself as</div><div>part of the application).</div>
<div><br></div><div>To first approximation one process per parallel activity is a good rule of thumb (and you let the Erlang scheduler figure out who is to run where) - the alternative is</div><div>that you limit the number of parallel processes and decide when and where they execute - you are basically saying "because I know a lot about the specific details of my application I can do a better job of process management than the Erlang VM" - this is pretty tricky.</div>
<div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="font-size:12pt;font-family:times new roman,new york,times,serif"><div style="font-style:normal;font-size:16px;background-color:transparent;font-family:'times new roman','new york',times,serif">
<span>That would be really nice if it can be done with only a small performance penalty, say 10-20%.</span></div></div></div></blockquote><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div><div style="font-size:12pt;font-family:times new roman,new york,times,serif"><div style="font-style:normal;font-size:16px;background-color:transparent;font-family:'times new roman','new york',times,serif">
<span><br></span></div><div style="font-style:normal;font-size:16px;background-color:transparent;font-family:'times new roman','new york',times,serif"><span>In my case I start 50.000 to 100.000 processes , one per file (it's an (map reduce like) application to do feature extraction for some machine learning algorithms) . </span></div>
</div></div></blockquote><div><br></div><div>How big is each file? how much processing is needed per-file to do feature extraction?</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div><div style="font-size:12pt;font-family:times new roman,new york,times,serif"><div style="font-style:normal;font-size:16px;background-color:transparent;font-family:'times new roman','new york',times,serif">
<span>One erlang node uses about 7GB of memory. I can probably tune it a bit (a lot?) more by using binaries but it would be nice to have an option to compress process data when not running, for people that are lazy/not an erlang expert/does not have that much time (my case)/or just as an indication of how much memory that could be saved by using binaries.</span></div>
<div style="font-style:normal;font-size:16px;background-color:transparent;font-family:'times new roman','new york',times,serif"><br></div></div></div></blockquote><div><br></div><div>look up the manual entry for hibernate/3</div>
<div><br></div><div><a href="http://erlang.org/doc/man/erlang.html#hibernate-3">http://erlang.org/doc/man/erlang.html#hibernate-3</a> </div><div><br></div><div>The hibernate BIF minimise the size of a process before putting it to sleep. It won't</div>
<div>compress any process data - but it does trim the stack and heap before suspending a process. If you have a large number of processes that sleep for</div><div>long times this might be a good idea. If your processes sleep for short times and wake up at random then it probably won't help.</div>
<div><br></div><div>Performance tuning is a black art. Unfortunately it's *very* system dependent. If you change the number of cores, or operating system, or amount of memory you have to start again.</div><div><br></div>
<div>If in doubt measure !</div><div><br></div><div>/Joe</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="font-size:12pt;font-family:times new roman,new york,times,serif">
<div style="font-style:normal;font-size:16px;background-color:transparent;font-family:'times new roman','new york',times,serif"></div><div style="font-style:normal;font-size:16px;background-color:transparent;font-family:'times new roman','new york',times,serif">
JK</div><div style="font-style:normal;font-size:16px;background-color:transparent;font-family:'times new roman','new york',times,serif"><span><br></span></div><div><br></div> <div style="font-size:12pt;font-family:'times new roman','new york',times,serif">
<div style="font-size:12pt;font-family:'times new roman','new york',times,serif"> <div dir="ltr"> <font face="Arial"> <hr size="1"> <b><span style="font-weight:bold">From:</span></b> Joe Armstrong <<a href="mailto:erlang@gmail.com" target="_blank">erlang@gmail.com</a>><br>
<b><span style="font-weight:bold">To:</span></b> Dmitry Kolesnikov <<a href="mailto:dmkolesnikov@gmail.com" target="_blank">dmkolesnikov@gmail.com</a>> <br><b><span style="font-weight:bold">Cc:</span></b> Steve Davis <<a href="mailto:steven.charles.davis@gmail.com" target="_blank">steven.charles.davis@gmail.com</a>>; Erlang Questions <<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a>> <br>
<b><span style="font-weight:bold">Sent:</span></b> Saturday, December 29, 2012 4:30 PM<div class="im"><br> <b><span style="font-weight:bold">Subject:</span></b> Re: [erlang-questions] Strings and Text Processing<br> </div>
</font> </div> <br>
<div><br><div><div class="h5"><br><div>On Sat, Dec 29, 2012 at 3:20 PM, Dmitry Kolesnikov <span dir="ltr"><<a rel="nofollow" href="mailto:dmkolesnikov@gmail.com" target="_blank">dmkolesnikov@gmail.com</a>></span> wrote:<br>
<blockquote style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hello Steve,<br>
<br>
You have raised a good point here.<br>
One more reason for binary is memory consumption and IPC overhead.<br></blockquote><div><br></div><div>The point about memory consumption is raised *many* times - on a modern</div><div>machine this is not a problem.</div>
<div><br></div><div>Example: I am working on a text file of 84KB - in a 32 bit Erlang we use</div><div>8 bytes/character - so I use 0.6 MB - I have 4GB memory - so I use 0.015% of</div><div>memory - ie no problem.</div><div>
<br></div><div>My strategy is to keep large strings as binaries when I'm not working on them,</div><div>turn them into lists in order to work on them, and turn them back into binaries</div><div>when I'm done. Just because a string starts off in a binary does not mean</div>
<div>that it has to stay as a binary as you work on it.</div><div><br></div><div><br></div><div>Imagine I have a lot of text files, say each of 50KB, I can store 20 per/MB or</div><div>20,000 files per GB. Assume I have a quad core. I can only work on four things</div>
<div>at the same time - so having (say) 20,000 files (at 50K) and work on four of them</div><div>(unpacked) at a time is another 1.6 Meg.</div><div><br></div><div>Gigabyte memories mean (among other things) what saving the odd byte here are there is hardly relevant.</div>
<div><br></div><div> </div><blockquote style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On another hand list allows to represent a code point per element.<br></blockquote><div><br></div><div>yes - the convenience of having one character per list element far outweighs</div><div>the space saving of storing strings in binaries</div>
<div><br></div><div> </div><blockquote style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
iolists are also very handy to dynamically compose a complex strings.<br>
<br>
I am afraid that this is an application specific questions… However, I tend to use binary for strings...<br><span><font color="#888888"><br></font></span></blockquote><div><br></div><div>My strings change form depending on what I'm doing. Sometimes they are</div>
<div>binaries, sometimes lists, sometimes trees, ...</div><div><br></div><div>Cheers</div><div><br></div><div>/Joe</div><div><br></div><div> </div><blockquote style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<span><font color="#888888">
- Dmitry<br>
</font></span><div><div><br>
<br>
On Dec 29, 2012, at 4:08 PM, Steve Davis <<a rel="nofollow" href="mailto:steven.charles.davis@gmail.com" target="_blank">steven.charles.davis@gmail.com</a>> wrote:<br>
<br>
> Disclaimer :-) All the below is prefixed by a big IMHO<br>
><br>
> Erlang has been correctly criticized for the difficulty of handling "strings".<br>
><br>
> There are two reasons for this (fundamental decisions that were taken way-back-when):<br>
> 1) "strings" are "just lists of integers"<br>
> 2) "strings" are by default latin-1 representations<br>
><br>
> This introduces major inconveniences, some of which are not resolvable:<br>
> When faced with any list during pattern matching, it is not at all easy to determine whether that list is a "string".<br>
> Further, since strings are "only" a subset of the set of lists of integers, it can be impossible to determine programmatically whether the list is a list of integers or is meant to represent a string. Determining whether a particular list even qualifies as a string in a program requires non-trivial processing of the entire list.<br>
><br>
> It's rather unfortunate that Erlang has earned this reputation, since the truth is that Erlang is truly excellent at text processing. However, to benefit from this excellence, you need to do two things:<br>
> 1) Represent and process text as binaries.<br>
> 2) Assume that the text binary is UTF-8 encoded, unless otherwise stated (meaning, e.g. #text{encoding = cstring, value = <<116,101,120,116,0>>}).<br>
><br>
> Suddenly, thanks to binary syntax and pattern matching, processing text in your programs becomes deterministic and easy. (Note that part of the reason for this is that binaries are "expected" to be opaque, whereas general list processing is fundamental to writing any program in Erlang).<br>
><br>
> There's a couple of minor drawbacks, both of which are the result of the initial decisions about "strings":<br>
> 1) The code is littered with additional angle brackets <<"string">> (annoying, but definitely worth the inconvenience)<br>
> 2) The standard Erlang/OTP library functions require textual arguments as lists (requiring overuse of binary_to_list)<br>
><br>
> And there are further benefits:<br>
> 1) Parsing/transcoding different charset encodings is far more straightforward<br>
> 2) Internationalization/localization is far more straightfoward<br>
><br>
> I wonder if, had the current binary pattern matching/comprehensions been available "way-back-when", whether the decision about "string" representation in Erlang may have been different. (i.e. <<116,101,120,116>> = "text").<br>
><br>
> Finally, here's my two questions:<br>
> 1) Is there any benefit at all to the "list representation" of strings above binary text?<br>
> 2) If not, I wonder if there's any way to change our minds about "strings" as we enter 2013?<br>
><br>
> regs,<br>
> /s<br>
><br>
> _______________________________________________<br>
> erlang-questions mailing list<br>
> <a rel="nofollow" href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a><br>
> <a rel="nofollow" href="http://erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
<br>
_______________________________________________<br>
erlang-questions mailing list<br>
<a rel="nofollow" href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a><br>
<a rel="nofollow" href="http://erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
</div></div></blockquote></div><br>
</div></div></div><div><div class="h5"><br>_______________________________________________<br>erlang-questions mailing list<br><a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a><br>
<a href="http://erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br><br><br> </div></div></div> </div> </div></div></blockquote></div><br>