[erlang-questions] Inter-node communication bottleneck
Jihyun Yu
yjh0502@REDACTED
Thu Aug 21 06:05:02 CEST 2014
The problem is reproducable *with loopback interface*. I tested with two
Erlang instances on same machine, communicating via loopback interface,
and results are same.
I changed contented lock - qlock in DistEntry - to spinlock, but it does
not affact on throughput(messages per second). 'perf' profiling tool [1]
shows that CPU cycle is moved from kernel to userspace.
I attach patch and 10-seconds sampling result on mutex and spinlock.
[1] https://perf.wiki.kernel.org/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: spinlock.patch
Type: text/x-diff
Size: 9161 bytes
Desc: not available
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20140821/827e911a/attachment.bin>
-------------- next part --------------
# Event count (approx.): 92786792854
#
# Overhead Command Shared Object Symbol
# ........ ............... ..................... ...............................................
#
24.69% beam.smp beam.smp [.] erts_dsig_send_msg
5.97% beam.smp beam.smp [.] scheduler_wait
2.05% beam.smp [ip_tables] [k] ipt_do_table
1.74% beam.smp [kernel.kallsyms] [k] nf_iterate
1.14% beam.smp [nf_conntrack] [k] ____nf_conntrack_find
1.11% swapper [kernel.kallsyms] [k] intel_idle
0.96% beam.smp [nf_conntrack] [k] tcp_packet
0.95% beam.smp [kernel.kallsyms] [k] tcp_recvmsg
0.93% beam.smp [kernel.kallsyms] [k] _raw_read_lock_bh
0.89% beam.smp [kernel.kallsyms] [k] _raw_spin_lock
0.85% beam.smp [kernel.kallsyms] [k] _raw_spin_lock_bh
0.81% beam.smp beam.smp [.] process_main
0.80% beam.smp [kernel.kallsyms] [k] tcp_ack
0.73% beam.smp [kernel.kallsyms] [k] default_send_IPI_mask_sequence_phys
0.69% beam.smp beam.smp [.] schedule
0.67% beam.smp [kernel.kallsyms] [k] tcp_transmit_skb
0.67% beam.smp [kernel.kallsyms] [k] __netif_receive_skb_core
0.65% beam.smp [kernel.kallsyms] [k] system_call
0.64% beam.smp [nf_conntrack] [k] tcp_in_window
0.59% beam.smp [kernel.kallsyms] [k] __inet_lookup_established
0.58% beam.smp [nf_conntrack] [k] nf_conntrack_in
0.57% beam.smp [kernel.kallsyms] [k] tcp_v4_rcv
0.57% beam.smp [kernel.kallsyms] [k] release_sock
0.56% beam.smp [kernel.kallsyms] [k] skb_release_data
0.55% beam.smp [kernel.kallsyms] [k] tcp_sendmsg
0.53% beam.smp [kernel.kallsyms] [k] local_bh_enable
0.51% beam.smp [kernel.kallsyms] [k] fget_light
0.48% beam.smp [bridge] [k] br_fdb_update
0.48% beam.smp [kernel.kallsyms] [k] put_prev_task_fair
0.48% beam.smp [kernel.kallsyms] [k] _raw_spin_lock_irqsave
0.47% beam.smp [kernel.kallsyms] [k] __alloc_skb
0.47% beam.smp [ebtables] [k] ebt_do_table
0.45% beam.smp [kernel.kallsyms] [k] __audit_syscall_exit
0.43% beam.smp [kernel.kallsyms] [k] tcp_rcv_established
0.43% beam.smp [kernel.kallsyms] [k] __ip_local_out
0.42% beam.smp [bridge] [k] br_handle_frame
0.41% beam.smp [nf_conntrack] [k] __nf_ct_refresh_acct
0.41% beam.smp [kernel.kallsyms] [k] net_rx_action
0.40% beam.smp [kernel.kallsyms] [k] copy_user_generic_string
0.39% beam.smp [iptable_nat] [k] nf_nat_ipv4_fn
0.39% beam.smp [kernel.kallsyms] [k] local_bh_enable_ip
0.39% beam.smp beam.smp [.] erts_thr_progress_leader_update
0.37% beam.smp [kernel.kallsyms] [k] ipv4_dst_check
0.37% beam.smp [kernel.kallsyms] [k] ip_finish_output
-------------- next part --------------
# Event count (approx.): 85477634040
#
# Overhead Command Shared Object Symbol
# ........ ............... ..................... ...............................................
#
14.05% beam.smp [kernel.kallsyms] [k] _raw_spin_lock
6.53% beam.smp beam.smp [.] scheduler_wait
2.58% beam.smp [ip_tables] [k] ipt_do_table
2.03% beam.smp [kernel.kallsyms] [k] nf_iterate
1.32% swapper [kernel.kallsyms] [k] intel_idle
1.07% beam.smp beam.smp [.] process_main
1.04% beam.smp [kernel.kallsyms] [k] default_send_IPI_mask_sequence_phys
0.89% beam.smp [kernel.kallsyms] [k] system_call
0.89% beam.smp [kernel.kallsyms] [k] _raw_spin_lock_bh
0.87% beam.smp [kernel.kallsyms] [k] __netif_receive_skb_core
0.77% beam.smp beam.smp [.] schedule
0.77% beam.smp [nf_conntrack] [k] ____nf_conntrack_find
0.74% beam.smp [kernel.kallsyms] [k] _raw_read_lock_bh
0.72% beam.smp [nf_conntrack] [k] tcp_packet
0.69% beam.smp [nf_conntrack] [k] tcp_in_window
0.67% beam.smp libpthread-2.17.so [.] pthread_mutex_unlock
0.66% beam.smp [ebtables] [k] ebt_do_table
0.65% beam.smp [kernel.kallsyms] [k] tcp_transmit_skb
0.65% beam.smp libpthread-2.17.so [.] pthread_mutex_lock
0.64% beam.smp [kernel.kallsyms] [k] tcp_recvmsg
0.60% beam.smp [kernel.kallsyms] [k] __audit_syscall_exit
0.59% beam.smp [kernel.kallsyms] [k] local_bh_enable
0.58% beam.smp [nf_conntrack] [k] nf_conntrack_in
0.57% beam.smp [kernel.kallsyms] [k] tcp_ack
0.56% beam.smp [kernel.kallsyms] [k] put_prev_task_fair
0.56% beam.smp beam.smp [.] erts_thr_progress_leader_update
0.54% beam.smp [kernel.kallsyms] [k] net_rx_action
0.54% beam.smp [kernel.kallsyms] [k] futex_wait_setup
0.53% beam.smp [kernel.kallsyms] [k] tcp_sendmsg
0.51% beam.smp [kernel.kallsyms] [k] tcp_v4_rcv
0.49% beam.smp [kernel.kallsyms] [k] _raw_spin_lock_irqsave
0.49% beam.smp [kernel.kallsyms] [k] copy_user_generic_string
0.49% beam.smp [kernel.kallsyms] [k] fget_light
0.48% beam.smp [kernel.kallsyms] [k] __do_softirq
0.48% beam.smp [kernel.kallsyms] [k] local_bh_enable_ip
0.47% beam.smp [bridge] [k] br_handle_frame
0.46% beam.smp [kernel.kallsyms] [k] __inet_lookup_established
0.44% beam.smp [kernel.kallsyms] [k] __schedule
0.43% beam.smp [kernel.kallsyms] [k] ip_rcv
0.42% beam.smp [kernel.kallsyms] [k] ip_finish_output
0.42% beam.smp beam.smp [.] erts_dsig_send_msg
0.42% beam.smp [kernel.kallsyms] [k] futex_wake
0.41% beam.smp beam.smp [.] encode_size_struct_int
0.40% beam.smp [kernel.kallsyms] [k] process_backlog
More information about the erlang-questions
mailing list