[erlang-questions] Why do we need modules at all?

Edmond Begumisa ebegumisa@REDACTED
Wed May 25 10:54:43 CEST 2011


Motivation:

If I can easily load modules or Joe's functions from different locations,  
then I can deploy my Erlang applications similarly to web-applications.  
I'm using Erlang at both the client and the server. Currently, reltool and  
appup I've found a too tedious. web-app style deployment would have big  
advantages...

* In corporate environments: clients can have a bare-bones installation,  
launch that, then get most of the Erlang code from a central server.  
Upgrading everyone is replacing a few files on the server. I could write  
some simple code to upgrade modules already loaded by clients.

* On the web: With code-signing supported, I can safely deploy all  
applications on my startup's web-server. I get all users (using different  
completely applications) to download and install the *same* single  
bare-bones installation. Erlang code for the different apps comes from my  
web-server. Users whose licenses run dry can't access newer code.

- Edmond -




On Wed, 25 May 2011 13:34:17 +1000, Edmond Begumisa  
<ebegumisa@REDACTED> wrote:

> Hello Joe,
>
> I've personally wanted an easy way to load modules from different places.
>
> If you started with making modules easy to load from different places,  
> then later provide support for functions, I imagine this would be easier  
> to implement immediately and easier for people to make use of  
> immediately ... where "people" refers to me :)
>
> I've got two suggestions (mostly inspired by the Mozilla Framework)...
>
> 1. How about letting the programmer choose where the functions come from  
> (kv store, fs, web-server, etc)...
>
> For instance, a module could start...
>
> -import(["file://misc/collect_int", "http://erlang.org/lists/*",  
> "mnesia://db1/misc/merge_kv"]).
>
> By default, the code loader would know what to do when it encounters  
> "file://", "http://", "https://", "ftp://", "sftp://" and "mnesia://"  
> URI schemes. Programmers could be allowed define their own URI schemes  
> by providing callbacks for loading code. For instance, one would be able  
> to define "ubf://" and have the code loader call his/her code when it  
> encounters "ubf://foo/bar"
>
> 2. Expanding on this: letting the programmer centralise the possible  
> locations of modules/functions and use this to define namespaces...
>
> One possible way of doing this is to steal from the "chrome  
> registration" system that the Mozilla Framework uses. A programmer could  
> define one-level deep namespaces in a configuration file which points to  
> targets that could come from anywhere, then use that in her modules. A  
> module could then start...
>
> -import(["erlang://misc/collect_int", "erlang://couch/mapreduce"]).
>
> Where "misc" and "couch" are namespaces and "erlang://" is a special URI  
> scheme telling the code loader to refer to the configuration file  
> "erlang.manifest" which would contain...
>
> [{"misc", "file://./ebin/misc.beam"},
>   {"couch", "http://localhost:7777/ebin/"}]
>
> Where "couch" points to a yaws server serving couch beams, or something  
> along those lines. Then she can easy switch between different code  
> sources using one file.
>
> A code-signing system could also be introduced.
>
> - Edmond -
>
>
> On Tue, 24 May 2011 18:06:19 +1000, Joe Armstrong <erlang@REDACTED>  
> wrote:
>
>> Why do we need modules at all?
>>
>> This is a brain-dump-stream-of-consciousness-thing. I've been
>> thinking about this for a while.
>>
>> I'm proposing a slightly different way of programming here
>> The basic idea is
>>
>>     - do away with modules
>>     - all functions have unique distinct names
>>     - all functions have (lots of) meta data
>>     - all functions go into a global (searchable) Key-value database
>>     - we need letrec
>>     - contribution to open source can be as simple as
>>       contributing a single function
>>     - there are no "open source projects" - only "the open source
>>       Key-Value database of all functions"
>>     - Content is peer reviewed
>>
>> These are discussed in no particular order below:
>>
>> Why does Erlang have modules?
>>
>> There's a good an bad side to modules:
>>
>> Good: Provides a unit of compilation, a unit of code
>> distribution. unit of code replacement
>>
>> Bad: It's very difficult to decide which module to put an individual
>> function in. Break encapsulation (see later)
>>
>> Aside: lib_misc.erl
>>
>> When I'm programming I often get to the point were I say there should
>> a function foo/2 in lists.erl but their isn't. There should be but
>> there isn't - foo/2 is a small self contained thing. Why should it be
>> in lists.erl because it "feels right".
>>
>> Strings are lists, so why do we have two modules lists.erl and
>> string.erl how should I decide in which module my new string/list
>> processing function should go.
>>
>> To avoid all mental anguish when I need a small function that
>> should be somewhere else and isn't I stick it in
>> a module elib1_misc.erl.
>>
>> My elib1_misc exports the following:
>>
>> added_files/2                 make_challenge/0
>> as_bits/1                     make_response/2
>> as_bits_test/0                make_response_test/0
>> bdump/2                       make_test_strings/1
>> bin2hex/1                     make_test_strings_test/0
>> bin2hex_test/0                make_tmp_filename/2
>> check_io_list/1               merge_kv/1
>> collect_atom/1                merge_kv_test/0
>> collect_atom_test/0           mini_shell/0
>> collect_int/1                 module_info/0
>> collect_int_test/0            module_info/1
>> collect_string/1              ndots/1
>> collect_string_test/0         nibble_to_hex_char/1
>> collect_word/1                nibble_to_hex_char_test/0
>> complete/2                    odd/1
>> complete_test/0               on_exit/2
>> dos2unix/1                    out_of_date/2
>> downcase_char/1               outfile/2
>> dump/2                        padd/2
>> dump_tmp/2                    perms/1
>> duplicates/1                  perms_test/0
>> ensure_started/2              pmap/2
>> eval_file/1                   pmap1/2
>> eval_file_test/0              pmap1_test/0
>> eval_string/1                 pmap_test/0
>> eval_string_test/0            priority_receive/0
>> every/3                       random_seed/0
>> expand_env_vars/1             random_string/1
>> expand_file_template/3        random_string/2
>> expand_string_template/2      read_at_most_n_lines/2
>> expand_tabs/1                 read_at_most_n_lines_test/0
>> expand_tabs_test/0            remove_duplicates/1
>> expand_template/2             remove_duplicates_test/0
>> extract_attribute/2           remove_leading_and_trailing_whitespace/1
>> extract_attribute_test/0       
>> remove_leading_and_trailing_whitespace_test/0
>> extract_prefix/2              remove_leading_whitespace/1
>> fetch/2                       remove_prefix/2
>> fetch_test/0                  remove_prefix_test/0
>> file2lines/1                  remove_trailing_whitespace/1
>> file2lines_test/0             replace/3
>> file2md5/1                    root_dir/0
>> file2numberedlines/1          rpc/2
>> file2numberedlines_test/0     safe/1
>> file2paras/1                  show_loaded/1
>> file2stream/1                 signed_byte_to_hex_string/1
>> file2string/1                 signed_byte_to_hex_string_test/0
>> file2template/1               skip_blanks/1
>> file2term/1                   skip_blanks_test/0
>> file_size_and_type/1          skip_to_nl/1
>> find_src/1                    skip_to_nl_test/0
>> first/1                       sleep/1
>> flatten_io_list/1             spawn_monitor/3
>> flush_buffer/0                split_at_char/2
>> for/3                         split_at_char_test/0
>> force/1                       split_list/2
>> foreach_chunk_in_file/3       split_list_test/0
>> foreach_word_in_file/2        string2exprs/1
>> foreach_word_in_string/2      string2exprs_test/0
>> forever/0                     string2html/1
>> get_erl_section/2             string2latex/1
>> get_line/1                    string2lines/1
>> get_line/2                    string2lines_test/0
>> have_common_prefix/1          string2stream/1
>> have_common_prefix_test/0     string2stream_test/0
>> hex2bin/1                     string2template/1
>> hex2bin_test/0                string2template_test/0
>> hex2list/1                    string2term/1
>> hex2list_test/0               string2term_test/0
>> hex_nibble2int/1              string2toks/1
>> hex_nibble2int_test/0         string2toks_test/0
>> id/1                          sub_binary/3
>> include_dir/0                 template2file/3
>> include_file/1                term2file/2
>> interleave/2                  term2string/1
>> is_alphanum/1                 test/0
>> is_blank_line/1               test1_test/0
>> is_prefix/2                   test_function_over_substrings/2
>> is_prefix_test/0              tex2pdf/1
>> is_response_correct/3         time_fun/2
>> keep_alive/2                  time_stamp/0
>> lines2para/1                  to_lower/1
>> list2frequency_distribution/1 to_lower_test/0
>> list2frequency_distribution_tetrim/1
>> longest_common_prefix/1       trim_test/0
>> longest_common_prefix_test/0  unconsult/2
>> lookup/2                      unsigned_byte_to_hex_string/1
>> lorem/1                       unsigned_byte_to_hex_string_test/0
>> ls/1                          which/1
>>                               which_added/1
>>
>> Now I find this very convenient when I write a new small utility  
>> function
>> I stick in in elib1_misc.erl - no mental anguish in choosing a module
>> name is involved.
>>
>> The observation that I find this very-convenient is telling me something
>> about modules - I like my elib1_misc it feels right.
>>
>> (aside - It seems many development projects have their own private
>> lib_miscs ...)
>>
>> Which brings me to the point of my question.
>>
>> Do we need module's at all? Erlang programs are composed of lots of  
>> small
>> functions, the only place where modules seem useful is to hide a letrec.
>>
>> The classic example is fibonacci. We want to expose fib/1 but hide the
>> helper function fib/3. Using modules we say
>>
>> -module(math).
>> -export([fib/1]).
>>
>> fib(N) ->
>>     fib(N, 1, 0).
>>
>> fib(N, A, B) when N < 2 -> A;
>> fib(N, A, B) -> fib(N-1, A+B, A).
>>
>> The downside is we have had to *invent* one module name math - whose  
>> *only*
>> purpose is to hide the definition of fib/3 which we don't want to be  
>> made
>> callable.
>>
>> If we put a second function into the module math, then this second  
>> function
>> could call fib/3 which breaks the encapsulation of fib/3.
>>
>> We could say:
>>
>> let fib = fun(N) -> fib(N, 1, 0) end
>> in
>>    fib(N, A, B) when N < 2 -> A;
>>    fib(N, A, B) -> fib(N-1, A+B, A).
>> end.
>>
>> I hardly dare suggest a syntax for this since I've been following
>> another thread in this forum where syntax discussion seem to encourage
>> much comment.
>>
>> ** Please do suggest alternative syntax's here - but do not comment on
>> other peoples suggestions ...
>>
>> I would like to just talk about why we have modules.
>>
>> Another question:
>>
>> Does the idea of a module come from the idea that functions have to be
>> stored somewhere, so we store them in a file, and we slurp the
>> file (as a unit) into the system, so the file becomes a module?
>>
>> If all the files were store by themselves in a database would this
>> change things.
>>
>> I am thinking more and more that if would be nice to have *all*  
>> functions in
>> a key_value database with unique names.
>>
>> lookup(foo,2) would get the definition foo foo/2 from a database.
>>
>> The unique names bit is interesting - is this a good idea. Qualified
>> names (ie names like xxx:foo/2) or (a.b.c.foo/2) sounds like a good
>> idea but but when I'm programming I have to invent the xxx or the
>> a.b.c which is very difficult. It also involves the "decision problem"
>> if the namespaces xxx and a.b.c already exist I have to *choose* which
>> to put my new function in.
>>
>> I think there might be a case for alises here joe:foo/2 could be used
>> while developing "joe" would expand to a horrible random local string  
>> the
>> real name being ab123aZwerasch123123_foo/2  but I would not be able to
>> publish my code or make it available to a third_part before I had
>> chosen a sensible name.
>>
>> (( managing namespaces seems really tricky, a lot of peoople seem
>> to thing that the problem goes away by adding "." 's to the name
>> but managing a namespace with namees like foo.bar.baz.z is just as  
>> complex
>> as managing a namespace with names like foo_bar_baz_z or names like
>> 0x3af312a78a3f1ae123 - the problem is that we have to go from a symbolic
>> name like www.a.b to a reference like 123.45.23.12 - but how do we  
>> discover
>> the initial name www.a.b? - there are two answers - a) we are given the  
>> name
>> (ie we click on a link) - we do not know the name but we search fo it ))
>>
>>
>> When programs are small we can live with "just the code" in "a few
>> modules" the ratio of code to meta data is high.
>>
>> When programs are large we need a lot of meta-data to understand them.
>>
>> I would like to see all functions with all meta-data in a data base.
>>
>> I'd like to say:
>>
>>    lookup(foo,2,Attribute) when Attribute =
>>
>>       code|source|documentation|type signatures|revision  
>> history|authors|...
>>
>> The more I think about it the more I think program development should
>> viewed as changing the state of a Key-Value database.
>>
>> So I imagine:
>>
>>     1) all functions have unique names
>>     2) there are no modules
>>     3) we discover the name of a function by searching metadata
>>        describing the function in a database
>>     4) all public functions (think open source) are in the same
>>        database
>>
>> We could make a system to do this.
>>
>> I think this would make open-source projects easier, since the
>> granularity of contribution goes down. You could contribute
>> a single function - not an entire application.
>>
>> (( A problem with GUT style open source projects is there is
>>    not one database of functions, I often what one function from
>>    this project, another function from another project -- the
>>    granularity of reusable parts should be the individual function.
>>
>>    functions are really easy to reuse
>>    modules are more difficult to reuse
>>    entire applications are very difficult to reuse
>>      (Unless there are isolated through a communication channel))
>>
>> Possible extensions.
>>
>>     1) Voting for promotion
>>     2) A review process
>>
>> Given a raw database will *all* functions in it - we could derive an
>> "approved" functions database.
>>
>> Popular functions could be moved to the approved database - the
>> review process would need to be discussed - so kind of peer-review/wiki
>> stuff.
>>
>> Comments?
>>
>> Volunteers?
>>
>> /Joe
>
>


-- 
Using Opera's revolutionary e-mail client: http://www.opera.com/mail/



More information about the erlang-questions mailing list