[erlang-questions] High volume CDR analysis

[Ukyo Virgden]

Hi Christian,

You're right. What I'm imagining is to collect call detail records  
from several telco equipment and periodically create reports. At this  
moment I'm not thinking about real-time (by realtime I mean as-it- 
happens) reports.

So this basically means, collect input data in parallel, apply some  
transformation, store in mnesia and run a job to create reports.

Therefore, the only data I need to store is for only one period,  
which is 100-300 million records of input.

Any suggestions? I suppose there is a storage limit of 2gig for  
mnesia per node right?

Thanks in advance,
Ukyo.

On 15.Oca.2008, at 21:20, Christian S wrote:

> I spent some time googling these acronyms, and as far as I could
> figure out it is:
>
> CDR:  Caller data record
> ETL: Extract, transform, load
>
> My guess is that Ukyo gets vendor specific report formats from some
> telecom equipment
> and need to rectify all to one format that one can use to extract
> reports from, for billing
> and homeland insecurity.
>
> I'm not sure mnesia is a good fit. But then again I dont know what
> kind of system Ukyo
> imagined. I got the feeling he would want to store a hundred million
> CDRs in mnesia
> tables, that seems to require that you know how to rewrite the mnesia
> backend to add a
> sleepycat db table-type, then not release it because you think it
> wasnt done right.
>
> Having a high-availability cluster where each CDR is reported
> as-it-happens, updating
> an account's usage-data, seems quite doable. But having an actual log
> of transactions
> for each account is probably important.
>
> On Jan 15, 2008 7:42 PM, Ahmed Ali <ahmed.nawras@REDACTED> wrote:
>> Hi Ukyo,
>>
>> Just wondering, what's ETL?
>>
>> Best regards,
>> Ahmed
>>
>>
>> On Jan 15, 2008 3:52 PM, Ukyo Virgden <listproc@REDACTED> wrote:
>>> Hi,
>>>
>>> I'm working full-time on a system which involves a hundreds of
>>> millions of CDR records.
>>>
>>> Our current setup is a typical ETL infrastructure which NOBODY is
>>> happy about.
>>>
>>> After reading and experimenting with Erlang and Mnesia, wouldn't it
>>> be possible to do such ETL with Erlang and Mnesia?
>>>
>>> Any suggestions?
>>>
>>> Regards,
>>> Ukyo.
>>>
>>> _______________________________________________
>>> erlang-questions mailing list
>>> erlang-questions@REDACTED
>>> http://www.erlang.org/mailman/listinfo/erlang-questions
>>>
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questions@REDACTED
>> http://www.erlang.org/mailman/listinfo/erlang-questions
>>