[erlang-questions] cost of integrating 3rd party SW

[Steve Vinoski]

On 5/24/08, Ulf Wiger <ulf@REDACTED> wrote:
> 2008/5/24 Steve Vinoski <vinoski@REDACTED>:
>
> > On 5/24/08, Per Melin <per.melin@REDACTED> wrote:
>  >> 2008/5/24 Ulf Wiger <ulf@REDACTED>:
>  >>
>  >> > It's something that pops up every once in a while,
>  >>  > e.g. in the Facebook chat server article: that combining
>  >>  > Erlang with components written in other languages is
>  >>  > hard.
>  >>
>  >> As a consultant I see a lot of components built in-house at different
>  >>  companies in different languages that use RDBMSs to interface with
>  >>  each other and COTSs. Often it's the completely wrong tool for the
>  >>  job, but it's used because it's easy. It doesn't matter if you mix
>  >>  Java, C++, Ruby, Python, PHP, C# and Perl; they usually (always?) have
>  >>  drivers/APIs to talk to the major RDBMSs, and you can assume that they
>  >>  are well used and therefor (probably) stable.
>  >>
>  >>  When I read or hear someone talk about using Erlang as part of a
>  >>  larger system the state of the RDBMS drivers usually comes into
>  >>  question.
>  >
>  > Hi Ulf, I agree with Per -- databases seem to be a weak link. I have
>  > an application where I had to front the database with a bridge written
>  > in a different language and then integrate my Erlang app with that
>  > through the network.
>
> This would seem to fit with Erlang's heritage - as the products
>  at Ericsson where Erlang has been used have not had any need
>  for external databases nor, indeed, any form of COM integration.
>  With the increased interest in ODBC, MySQL and PostgreSQL
>  connectivity, this will hopefully improve rather rapidly now.

Sounds good!

>  > Other than that, I've had great success with network-based
>  > integrations over various protocols.
>
> Would you say then that the dominant mode of integration
>  is still socket-based, even in environments where shared-
>  memory integration is possible (e.g. windows DLLs, CLR,
>  etc.)? I mean, is this the preferred way of integrating
>  components, even when not using Erlang, and even when
>  performance requirements are high?

As far as I've seen, the answer is yes. Regarding shared memory in
particular, in my CORBA-related work over the years, which included
efforts for two different vendors and a variety of co-development
efforts with other vendors, we always provided shared memory
capabilities, but all told only 1 or 2 customers, literally, ever used
them. We found that it really wasn't any faster than the local
loopback. I can honestly say that I can't recall using any systems,
CORBA or otherwise, that actually make effective use of shared memory,
though I of course accept that others might have different experiences
with other software.

Now that I've written the above, I see that you're using the term
"shared memory" more generally to include things like shared libraries
and DLLs. From that perspective, it depends on what the 3rd-party
software is like. If it's like an SDK, then you usually end up linking
with 3rd-party shared libraries/DLLs. If it's a networked service,
then of course network integration is the way to go. In my experience,
with 3rd-party systems you get much more of the latter, not the
former, mainly because of the pain of maintaining binary compatibility
across versions of the shared libraries/DLLs for languages like Java
and C++. Over the past few years, though, there have been some efforts
both in C#/CLR and in something called the Service Component
Architecture (SCA) to create "assemblies" that are supposed to help
address some of those binary compatibility issues. I'm unaware of how
widely used either of them are, though; I could be mistaken but I
don't think there's been a lot of uptake of either, relatively.

>  I'm hoping/assuming that shared-memory integration will
>  become less and less interesting with the increasing
>  availability of multicore computers. From Erlang's point
>  of view, the VM can do blocking IO in its own thread
>  rather than periodically polling like it has to do on a
>  single core.

That's an excellent point.

--steve