[erlang-questions] HOW TO IMPROVE THE PERFORMANCE OF DETS

[chamila piyasena]

Hi,
Thank you again Cristian.

You said that indexing can be worked around although dets doesn't support
it. I still searching a way to do it.

Can you please tell me how it can be achieved? Are you telling like using
B-Tree or some data stucture?   But I don't got a way to combine this B-Tree
concept in to Dets table.

thanx

Chamila



On 12/21/06, Christian S <chsu79@REDACTED> wrote:
>
> On 12/21/06, chamila piyasena <tchamila@REDACTED> wrote:
> > Thank you again Cristian,
> >
> > actually it only stores the CDR records of the SMS(from number, to
> number ,
> > delivered or not, date,  etc.. ) not the SMS it self and a separate dets
> > file is created every hour .
> >
> > If some one want to see records  related to a particular phone number in
> a
> > given period,  suppose  in  a duration of month  there will be many rows
> in
> > corresponding files.
> >
> > going through all these records to select what we want is not efficient
> and
> > very slow.
> > thats why I thought about indexing (but there is no support for that in
> > dets)
>
> Are you calling select once per file for every phone number you want a
> report on? If this is for billing, dont you typically want a report
> for all phone numbers over the requested period?
>
> No there is no support for indexing but that can be worked around, but
> assuming you still want to scan each hourly dets file:
>
> I suggest you begin using dets:foldl or dets:foldr  [other readers,
> what is the difference, order isnt guaranteed anyway?]. Fold doesnt
> make you have to care about the optimal N. With fold you will process
> record after record and it allows you to accumulate information, or
> perform side effects, such as building up a report file.
>
> As for working around the lack of indexing. You can add information
> about the messages stored in an hourly dets file and write that down
> before you close it when replacing it with a new one. Just pick a
> unique dets key to store that information under. Imagine storing the
> phone numbers seen during the hour, and using this information to see
> if it is worth scanning the dets file for a given phone number at all.
> Imagine using fold as above to keep a report file for each hourly log,
> a file that maps phone number to a list of the records seen for that
> number. You can even store that in the same dets file if you make sure
> the keys do not collide.
>
> Similarly, you can keep a map up to date that maps phone numbers to
> dets files which contain messages with it in. From scanning 30*24 log
> files, touching ALL data, you will be looking up a phone number to
> find which dets files it has contributed to. And then looking up the
> phone number in those dets files to find which records that was. Only
> touching a fraction of the log data.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20061222/357d7a0e/attachment.htm>