[erlang-questions] 1000+ users; 30+ data tables/user

[Joe Armstrong]

On Tue, Nov 11, 2014 at 10:28 PM,  <lloyd@REDACTED> wrote:
> Thank you, Joe,
>
>
>
> You've broadened my horizons and I love the simplification implied.

> For sake of clarification and deeper understanding:
>>
> -- Would, then, one directory and n files per user work?
>

Well I'd have to say "try it and see" - what I do know is that sooner or later
it will break - all things break eventually if you stress them enough
- but I don't know
where. So make it - break it and fix it. I used 1 directory per user
in my latest prototype.

>
>
>       One respondent to my question cited possible limits imposed by innodes
> and atoms. But I
>
>       see that my work station provides 30138368 inodes  and Erlang docs
> tell me that Erlang
>       provides 1048576 atoms, so I'm not convinced that these limits are a
> problem.

You can always start a second erlang node - buy a few exta GB of
memory or an SSD

If you get enough users that memory of any kind is a problem (be it
RAM, local storage or off-site storage)
then you will have made a successful product :-)

The choice of OS, the amount of RAM and whether the disk is a SSD or
spinning lump of
iron might made more differences than architectural differences. (I've
just swapped a magnetic disk
in my five year old macbook pro for an SSD and now it's going crazy
fast - a lot of algorithms
I wrote years ago and ditched because they were too slow are now feasible)

Personally I'd start with an API - sufficiently complex to meet your
needs. Something like:

     storage:open(Who) -> Handle | EXIT
     storage:read(Handle, Key) -> {ok, Val} | error
     storage:write(Handle, Key, Val) -> ok
     storage:close(handle) -> ok

Then implement this in the simplest possible way - it could be the
file system, an ets table, a dets table
or a old man in a cave who carves this on a big stone - the point is
you're not supposed to know.

Then write your code - scale it and see where it breaks - try to break
it and see what happens.

Once it's broken you can fix it.

If it never breaks then heave a sigh of relief and ship it.

>
> -- Would one directory and n dets tables per user work?

Yes - but again it will work for K dets tables whose average size is M
bytes - but we don't
know K and M - and no amount of guessing will work - it depends upon
your machine.

So now let me break my golden rule and guess ....

Let's suppose you have 10K users (and if you get 10K users for
anything - congratulations -
every new thing on the planet wants 10K users, so you're doing great).

Lets suppose each user has 10KBytes of data (which seems like a lot
for many apps - or could be really small if you're exchanging images)

(aside if they are exchanging images than assume 10K of meta data is
all that has to be kept in RAM
and all the images will be stored in the file system)

So we now have 10K users with 10K of meta data = 100M of data - but a
modern laptop comes with 4G of memory - so my 100M of data is 2.5% of
RAM - so now *everything* will fit into RAM
so you could have all user data in a single ets table and do
ets:tab2file and friends to store this.

Even a little rasperry PI mdel B with a 0.5G should be OK - but you
might break it.

If your users store big things (like images) then you will run out of
storage far faster than
you run out of processing power (say 100 images of 5MB = 0.5 GB/user x
10K users = 5TB)

So here the meta data of 100M can be kept in memory but the data
(images) would be better stored
off-site (in say S3 storage)

With 1GByte of meta data you could have 40K users - easy - but it
would be unwise to have these
on one machine - so really early on (say at 10 users) you should be
replicating data across machines
to recover from a crash ...

As I said,  I dislike guessing.


>
>
> -- Can sufficient availability be achieved with conventional backup
> strategies?

"conventional" means nothing to me - you have to think through the crash
scenarios and ask "what do I want to happen" - is it acceptable to loose data,
is it acceptable to loose latency, then build your application accordingly.

I don't really understand the difference between fault-tolerance and backup.

Backup seems to mean "I don't trust my fault tolerance" - storage
should be reliable
meaning that if I store stuff I can retrieve it later - so backup means
"I don't trust my storage layer".

Backup could be used in the sense of "a snapshot" - so if I
deliberately delete a file
then I can go back to an earlier snapshot and recover it. This should
not mean that the storage
abstraction is broken.

Rather than a snapshot it would be better if the storage abstraction
had no "delete" operation
so deleting something just removes it from the current namespace so I
can't see it any more.

Personally, it seems far easier to assume infinite write append
stores, with a trail somewhere else for
crash resilience (which is how most back-end stores work anyway)


>
> Yes, I know the answers are application specific, but are there
> metrics/rules of thumb/guidelines that can give me back-of-the-envelope
> answers short of all-out test-and-measurement?

My rule of thumb is that human textual data (by that I mean, text that
you type into
a computer) is easy to process. So anything that it typed into the
machine through a keyboard
can be stored in RAM -

Back of envelope - suppose I program like crazy and write 10K of code
per day (that's a lot)
and do this for 200 days a year - so I'm up in 2MBytes/year - so 1GB
will last for 500 years ....

Data produced by devices - photos videos etc. is what takes up space -
we need 100s of GB
to TB's of storage depending upon the resolution of the devices - but
this data will be stored
somewhere in the cloud - since they have far lower prices than we can
achieve at home.

I'd reckon on keeping all meta data in RAM locally - all big data
(images, whatever) in the cloud
with locally cached copies. I'd replicate the meta data across K
machines (for some K)

You don't need to do all-out measurements, think in terms of
"measurements that are appropriate to this phase in the design" -
start with a guess, measure to see if you are correct and refine your
estimates as you go along.

Estimating how long something takes is the the area were I am most
likely to be wrong.

Today I am very reluctant to guess how long something I have never
done will take. I have been
so wrong so many times in the past that it is laughable.

A few weeks ago I had a problem that I thought would take 5 minutes to
solve - it took three weeks ...
if stuff worked as it said on the label life would be simple - but
often strange and unexpected things happen
when you do something that you have never done before. Life would
however be rather boring
if we could predict the future :-)

Cheers

/Joe




>
>
>
> Many, many thanks for your insights.
>
>
>
> Lloyd
>
>
>
>
>
>
>
>
>
>
>
> -----Original Message-----
> From: "Joe Armstrong" <erlang@REDACTED>
> Sent: Tuesday, November 11, 2014 3:02pm
> To: "Garrett Smith" <g@REDACTED>
> Cc: "Lloyd R. Prentice" <lloyd@REDACTED>, "erlang-questions"
> <erlang-questions@REDACTED>
> Subject: Re: [erlang-questions] 1000+ users; 30+ data tables/user
>
> On Tue, Nov 11, 2014 at 7:41 PM, Garrett Smith <g@REDACTED> wrote:
>> Hi Lloyd,
>>
>> Sorry for the late reply here - I was interested in this thread last
>> week when you sent it but was in sleep deprived conference mode and
>> never got back to it.
>>
>> On Tue, Nov 4, 2014 at 8:49 PM, Lloyd R. Prentice <lloyd@REDACTED>
>> wrote:
>>> Hello,
>>>
>>> This is a naive question reflecting my inexperience with databases.
>>>
>>> I'm planning to offer my users a set of management/planning tools. Each
>>> user would be storing/retrieving user-specific data involving as many as 30
>>> data tables.
>>>
>>> --- Data fits well into Erlang records.
>>> --- We're not talking huge volumes of data per user.
>>> --- Nor do I expect much data analysis.
>>> --- Data integrity and availability are essential.
>>> --- Users may, however, wish to bundle up their data a some point and
>>> migrate to a different system.
>>>
>>> I'm attracted to mnesia because of it's it's tight integration with
>>> Erlang and it's replication features. I'm also considering riak.
>>>
>>> My first thought was that every user would own his/her own database. But
>>> this seems to rule out
>>> mnesia since:
>>>
>>> "Mnesia is a truly distributed DBMS and the schema is a system table that
>>> is replicated on all nodes in a Mnesia system. The function will fail if a
>>> schema is already present on any of the nodes in NodeList."
>>> http://www.erlang.org/doc/apps/mnesia/Mnesia_chap3.html
>>>
>>> An option would be to store data for all users in each of the 30 tables.
>>> But is there a better solution altogether?
>>>
>>> I'd much appreciate suggestions and guidance from wiser heads.
>>
>> I think we've all had this problem - a new project and nothing to hold
>> us back but our own imagination. So the question... which database to
>> pick. Which indeed? There are like 100 amazing options!
>>
>> My suggestion here is to stop this line of thinking immediately :)
>>
>> I would instead plan to throw your early work away. Pick something
>> that is the fastest and easiest imaginable for you to make progress on
>> your app. Treat it as a "this will almost certainly not be what I end
>> up with".
>>
>> _For me_ this means one of these:
>>
>> - Hard coded values or config files
>> - Dets
>> - SQLite
>> - MySQL
>>
>> The point is to keep it as simple as possible and just get stuff working.
>
> Excellent advice.
>
> For many systems - I use one file per user. The file contains
> term_to_binary(X)
> where X is whatever I feel like to represent the use data.
>
> (or you can use text files - then you can run amazing things like grep and
> find
> on them :-)
>
> The OS caches file access and I can easily analyse/dump the files.
>
> I've *never* got to the point where I need to change the file system for
> a database (but then again I've not built a really big system - and
> this works fine
> for several thousand files/users)
>
> If and when the design problems are solved you can change representations
> *if it is necessary* - choosing a database right at the start is
> "premature optimisation" - if you ever get to this point then the
> choice of representation
> should be dictated by measurement and not guesswork.
>
> /Joe
>
>> When I'm starting on something new, I just don't know enough about
>> anything to make the right decision - so I deliberately make the right
>> _wrong_ decision - that is, the decision that will let me move forward
>> quickly and get to the real problems. I might throw it away later, or
>> I might keep it. But in any case, I'm sure as hell not going to spend
>> a lot of time on it. Not until I'm facing real, hard, visible problems
>> that I can use to inform my next steps.
>>
>> Garrett
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questions@REDACTED
>> http://erlang.org/mailman/listinfo/erlang-questions