[erlang-questions] BEAM performance?

[Oliver Korpilla]

Hello.
 
We wrote a moderately complex telco application at work using elixir, but I think the performance we observe would be similar in Erlang since basically it all depends on BEAM and OTP anyway, so please don't mind me asking here.
 
We run message scenarios that involve ASN.1 encoding/decoding, several IP protocols, a proprietary transport we access through an Erlang port, etc. Message sequences will be handled through gen_fsm processes.
 
The application is split over several nodes.
 
What we've observed is:
* First time scenario runs is super slow.
* Second time, time is halved.
* Third time, performance is very good, especially given the complex scenario.
 
When analyzing bottlenecks in the software, we observed and tried the following:
* I preloaded all modules into the VM and times improved significantly. 
* ASN.1 encoding on 1st try can take as much as 20ms on first run but go <1ms after for same message with minimally different content.
* Using HiPE on the ASN.1 generated codec did worsen performance.
* Executing the message codec at least once during startup improved ASN.1 performance the most.
* Two DB writes to same table row in short order can significantly decrease performance by causing multi-ms delays.
 
(I know this is probably no surprise to anyone here but we're learning how the system behaves during runtime.)
 
The curve described above still persists - the totals are just less.
 
Our working assumption is that caching is impacting performance here, but we actually know too little about the BEAM runtime.
 
* Are there mechanisms internal to BEAM impacting this? Or is it purely a property of the CPU architecture?
* Could I stimulate BEAM to cache certain parts of the code ahead of time? 
 
The measurements are rough ones, done by log timestamps. ms granularity is enough for us to gauge overall performance of our signalling.
 
Message performance for user 3 and onward are okay but users 1 and/or 2 could be dropped because of timeouts, so optimizing these still matters.
 
Thank you for any advice you can give,
Oliver

(Resent on Raimo's request)