[erlang-questions] Storing images

[lloyd@REDACTED]

Thank you, Joe,

I get carried away with all the choices. The simplicity and logic of this approach is comforting.

All the best,

Lloyd

-----Original Message-----
From: "Joe Armstrong" <erlang@REDACTED>
Sent: Tuesday, May 20, 2014 4:31pm
To: "Lloyd R. Prentice" <lloyd@REDACTED>
Cc: "Erlang-Questions Questions" <erlang-questions@REDACTED>
Subject: Re: [erlang-questions] Storing images

I think you need to provide more data.
   - how many images ("unknown" thousands is not very precise) - how big is
the meta data?

I note you say thousands not millions. Say unknown = 100 then you have 100K
images

100K images @ 100K per image = 10^10 Bytes = 10 GBytes (= tiny)

Assume the meta data is small say 100Bytes/image then the
meta data is 10^5 * 100 = 10^7 Bytes = 10 MB (= tiny)

If you have a machine with 1GB of memory this is 1% of total memory.

I'd

   Make sure the meta data is small, ie no tags in it and possible
compressed and store the lot in RAM
   you could use an ets table or a custom binary.

You can store a heck of a lot of meta data in a GB and a heck of a lot of
images in a TB
so I'd keep the meta data in RAM and the image data on disk.

If you intend to use a SSD you could keep everything on disk

Start with the simplest possible correct solution and measure. You'll be
amazed what you can do with ets tables and
a GB of RAM

   Cheers

/Joe




On Mon, May 19, 2014 at 5:52 AM, Lloyd R. Prentice <lloyd@REDACTED>wrote:

> Hello,
>
> Suppose I anticipate unknown thousands of records, each containing
> meta-data associated with an image. Each image averages, say, 100KB.
> Retrieval is by key and queries on the meta data. Retrieval predominates
> over insert. Reliability and speed of access are high on the requirements
> list.
>
> I'm considering Riak. But...
>
> Questions:
>
> - Would it be best practice to store the each image in its associated
> record or would it be better to store the images separately in, say,
> directories or in a separate distributed file system such as leoFS?
>
> - Is there a better solution altogether?
>
> Many thanks,
>
> LRP
>
> Sent from my iPad
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>