[erlang-questions] List Question
Wed Aug 9 01:40:22 CEST 2017
The oo implementation we have is in 2 parts,
1. parse into a tree with no knowledge of semantics
2. Use a specific Object to provide an interface to the data based on
I am involved in standards process and there is a blog post here for
Essestially plan to do same in erlang
1. Parse into a tree
2. Create unit that has functions for reading the known values for a
specific segment - these are autogenerated eg have a msh.erl that has
a msh:Sending_Facility() function
3. Use reader functions that allow message to become more complex, but
permit old function to continue to work
Wednesday, August 9, 2017, 9:27:24 AM, you wrote:
RAOK> From what I've been able to glean about the HL7 message
RAOK> format, there are two aspects:
RAOK> * the basic syntax is the multi-level delimited thingy but
RAOK> * there is also *semantics*, predefined message types,
RAOK> an assortment of data types, rules for mapping data
RAOK> types to trees and so on.
RAOK> From a quick look at the Elixir HL7 parser, they have
RAOK> taken steps to handle some (but perhaps not all) of the
RAOK> *semantics* of HL7 and don't just give a tree of strings,
RAOK> but more structured data.
RAOK> Parsing the multi-level delimited syntax is trivial.
RAOK> Dealing with the semantics is not.
RAOK> I think that in figuring out for yourself how to work
RAOK> with HL7 messages in Erlang, the starting point would
RAOK> - what message types do you want to handle?
RAOK> - what kinds of data occur in them?
RAOK> - how do you want to represent those kinds of
RAOK> data in Erlang?
RAOK> - do you actually want to represent *every* field
RAOK> at all? Some might not be relevant to you.
RAOK> - would you be streaming messages through a
RAOK> system (like some sort of pub/sub queueing
RAOK> middleware), summarising messages, storing
RAOK> them, or what?
RAOK> - what does a type declaration for a message type
RAOK> look like in HL7? Is there some way to automatically
RAOK> derive parsing code from that?
RAOK> What I'm getting at with the last point is that there
RAOK> is ASN.1 support for Erlang. Give it an ASN.1
RAOK> definition, and you get Erlang code out the other end.
RAOK> I am particularly thinking of the PADS project:
RAOK> PADS: Processing Arbitrary Data Streams
RAOK> Kathleen Fisher and Bob Gruber, AT&T Labs
RAOK> Slides in ppt http://homepages.inf.ed.ac.uk/wadler/xmlbinding/
RAOK> Transactional data streams, such as sequences of stock-market
RAOK> buy/sell orders, credit-card purchase records, web server
RAOK> entries, and electronic fund transfer orders, can be mined very
RAOK> profitably. As an example, researchers at AT&T have built
RAOK> customer profiles from streams of call-detail records to significant financial effect.
RAOK> Often such streams are high-volume: AT&T's call-detail stream
RAOK> contains roughly 300 million calls per day requiring
RAOK> approximately 7GBs of storage space. Typically, such stream data
RAOK> arrives ``as is'' in ad hoc formats with poor documentation. In
RAOK> addition, the data frequently contains errors. The appropriate
RAOK> response to such errors is application-specific. Some
RAOK> applications can simply discard unexpected or erroneous values
RAOK> and continue processing. For other applications, however, errors
RAOK> in the data can be the most interesting part of the data.
RAOK> Understanding a new data source and producing a suitable parser
RAOK> are crucial first steps in any use of such data. Unfortunately,
RAOK> writing parsers for this kind of data is a difficult task, both
RAOK> tedious and error-prone. It is complicated by lack of
RAOK> documentation, convoluted encodings designed to save space, the
RAOK> need to handle errors robustly, and the need to produce
RAOK> efficient code to cope with the scale of the stream. Often, the
RAOK> hard-won understanding of the data ends up embedded in parsing
RAOK> code, making long-term maintenance difficult for the original
RAOK> writer and sharing the knowledge with others nearly impossible.
RAOK> The goal of the PADS project is to provide languages and tools
RAOK> for simplifying data processing. We have a preliminary design of
RAOK> a declarative data-description language, PADSL, expressive
RAOK> enough to describe the data feeds we see at AT&T in practice,
RAOK> including ASCII, binary, EBCDIC, Cobol, and mixed data formats.
RAOK> From PADSL we generate a tunable C library with functions for
RAOK> parsing, manipulating, and summarizing the data. In joint work
RAOK> with Mary Fernandez and Ricardo Medel, we are working to
RAOK> integrate PADS and XQuery to support declarative querying of
RAOK> data sources with PADS descriptions.
RAOK> The PADS project moved from AT&T to
RAOK> I say this in all seriousness: if I had a need to process
RAOK> GB of HL7 data, I would start by seeing if PADS was adequate
RAOK> to describe it, and if so I'd write an HL7->Erlang data
RAOK> translator in C or ML (as PADS has C and ML versions).
RAOK> If not, I'd see what ideas I could steal from PADS.
RAOK> Using a declarative data language to describe the message types
RAOK> I was interested in would be an up-front cost, but it would
RAOK> hugely simplify later maintenance.
sent from a real computer
More information about the erlang-questions