[erlang-questions] Performance issue with snmp

[Phillip Toland]

I am working on an application that uses SNMP heavily. This
application needs to make hundreds of SNMP queries during normal
operation and, this being Erlang, I tried to speed things up by
issuing queries in parallel. Unfortunately, the performance of snmpm
decreases considerably as you add more concurrent requests with
requests timing out above a certain threshold.  To illustrate the
problem I have created a minimal code sample to reproduce the problem.
Since snmpm requires a little configuration I have uploaded the code
and config files to Github: https://github.com/toland/snmp_perf

If I run snmp_perf:sync() with any number performance is very stable
with every request returning in about 0.002 seconds. I am testing with
a device on the network so this is reasonable performance. However,
running snmp_perf:async(10) results in requests taking on average 0.01
seconds and the performance continues to get worse until about 80
concurrent requests when I start getting timeouts and the average
request time is about 0.08 seconds. Increasing the number of requests
results in longer request times and more timeouts.

My question is this: is this performance degradation the expected
behavior or does it represent a bug in snmpm? It looks like a bug to
me but I wanted to get a second opinion. If this is the expected
behavior does anyone have a suggestion for ameliorating the problem in
a highly concurrent application? I have considered implementing a
queue and worker pool as a way to limit the concurrent requests to
snmpm, but was hoping there was a better solution.

Thanks
~p