[erlang-questions] Inter-node communication bottleneck

Jihyun Yu <>
Thu Aug 21 06:05:02 CEST 2014


The problem is reproducable *with loopback interface*. I tested with two
Erlang instances on same machine, communicating via loopback interface,
and results are same.

I changed contented lock - qlock in DistEntry - to spinlock, but it does
not affact on throughput(messages per second). 'perf' profiling tool [1]
shows that CPU cycle is moved from kernel to userspace.

I attach patch and 10-seconds sampling result on mutex and spinlock.

[1] https://perf.wiki.kernel.org/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: spinlock.patch
Type: text/x-diff
Size: 9161 bytes
Desc: not available
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20140821/827e911a/attachment.bin>
-------------- next part --------------
# Event count (approx.): 92786792854
#
# Overhead          Command          Shared Object                                           Symbol
# ........  ...............  .....................  ...............................................
#
    24.69%         beam.smp  beam.smp               [.] erts_dsig_send_msg                         
     5.97%         beam.smp  beam.smp               [.] scheduler_wait                             
     2.05%         beam.smp  [ip_tables]            [k] ipt_do_table                               
     1.74%         beam.smp  [kernel.kallsyms]      [k] nf_iterate                                 
     1.14%         beam.smp  [nf_conntrack]         [k] ____nf_conntrack_find                      
     1.11%          swapper  [kernel.kallsyms]      [k] intel_idle                                 
     0.96%         beam.smp  [nf_conntrack]         [k] tcp_packet                                 
     0.95%         beam.smp  [kernel.kallsyms]      [k] tcp_recvmsg                                
     0.93%         beam.smp  [kernel.kallsyms]      [k] _raw_read_lock_bh                          
     0.89%         beam.smp  [kernel.kallsyms]      [k] _raw_spin_lock                             
     0.85%         beam.smp  [kernel.kallsyms]      [k] _raw_spin_lock_bh                          
     0.81%         beam.smp  beam.smp               [.] process_main                               
     0.80%         beam.smp  [kernel.kallsyms]      [k] tcp_ack                                    
     0.73%         beam.smp  [kernel.kallsyms]      [k] default_send_IPI_mask_sequence_phys        
     0.69%         beam.smp  beam.smp               [.] schedule                                   
     0.67%         beam.smp  [kernel.kallsyms]      [k] tcp_transmit_skb                           
     0.67%         beam.smp  [kernel.kallsyms]      [k] __netif_receive_skb_core                   
     0.65%         beam.smp  [kernel.kallsyms]      [k] system_call                                
     0.64%         beam.smp  [nf_conntrack]         [k] tcp_in_window                              
     0.59%         beam.smp  [kernel.kallsyms]      [k] __inet_lookup_established                  
     0.58%         beam.smp  [nf_conntrack]         [k] nf_conntrack_in                            
     0.57%         beam.smp  [kernel.kallsyms]      [k] tcp_v4_rcv                                 
     0.57%         beam.smp  [kernel.kallsyms]      [k] release_sock                               
     0.56%         beam.smp  [kernel.kallsyms]      [k] skb_release_data                           
     0.55%         beam.smp  [kernel.kallsyms]      [k] tcp_sendmsg                                
     0.53%         beam.smp  [kernel.kallsyms]      [k] local_bh_enable                            
     0.51%         beam.smp  [kernel.kallsyms]      [k] fget_light                                 
     0.48%         beam.smp  [bridge]               [k] br_fdb_update                              
     0.48%         beam.smp  [kernel.kallsyms]      [k] put_prev_task_fair                         
     0.48%         beam.smp  [kernel.kallsyms]      [k] _raw_spin_lock_irqsave                     
     0.47%         beam.smp  [kernel.kallsyms]      [k] __alloc_skb                                
     0.47%         beam.smp  [ebtables]             [k] ebt_do_table                               
     0.45%         beam.smp  [kernel.kallsyms]      [k] __audit_syscall_exit                       
     0.43%         beam.smp  [kernel.kallsyms]      [k] tcp_rcv_established                        
     0.43%         beam.smp  [kernel.kallsyms]      [k] __ip_local_out                             
     0.42%         beam.smp  [bridge]               [k] br_handle_frame                            
     0.41%         beam.smp  [nf_conntrack]         [k] __nf_ct_refresh_acct                       
     0.41%         beam.smp  [kernel.kallsyms]      [k] net_rx_action                              
     0.40%         beam.smp  [kernel.kallsyms]      [k] copy_user_generic_string                   
     0.39%         beam.smp  [iptable_nat]          [k] nf_nat_ipv4_fn                             
     0.39%         beam.smp  [kernel.kallsyms]      [k] local_bh_enable_ip                         
     0.39%         beam.smp  beam.smp               [.] erts_thr_progress_leader_update            
     0.37%         beam.smp  [kernel.kallsyms]      [k] ipv4_dst_check                             
     0.37%         beam.smp  [kernel.kallsyms]      [k] ip_finish_output                           
-------------- next part --------------
# Event count (approx.): 85477634040
#
# Overhead          Command          Shared Object                                           Symbol
# ........  ...............  .....................  ...............................................
#
    14.05%         beam.smp  [kernel.kallsyms]      [k] _raw_spin_lock                             
     6.53%         beam.smp  beam.smp               [.] scheduler_wait                             
     2.58%         beam.smp  [ip_tables]            [k] ipt_do_table                               
     2.03%         beam.smp  [kernel.kallsyms]      [k] nf_iterate                                 
     1.32%          swapper  [kernel.kallsyms]      [k] intel_idle                                 
     1.07%         beam.smp  beam.smp               [.] process_main                               
     1.04%         beam.smp  [kernel.kallsyms]      [k] default_send_IPI_mask_sequence_phys        
     0.89%         beam.smp  [kernel.kallsyms]      [k] system_call                                
     0.89%         beam.smp  [kernel.kallsyms]      [k] _raw_spin_lock_bh                          
     0.87%         beam.smp  [kernel.kallsyms]      [k] __netif_receive_skb_core                   
     0.77%         beam.smp  beam.smp               [.] schedule                                   
     0.77%         beam.smp  [nf_conntrack]         [k] ____nf_conntrack_find                      
     0.74%         beam.smp  [kernel.kallsyms]      [k] _raw_read_lock_bh                          
     0.72%         beam.smp  [nf_conntrack]         [k] tcp_packet                                 
     0.69%         beam.smp  [nf_conntrack]         [k] tcp_in_window                              
     0.67%         beam.smp  libpthread-2.17.so     [.] pthread_mutex_unlock                       
     0.66%         beam.smp  [ebtables]             [k] ebt_do_table                               
     0.65%         beam.smp  [kernel.kallsyms]      [k] tcp_transmit_skb                           
     0.65%         beam.smp  libpthread-2.17.so     [.] pthread_mutex_lock                         
     0.64%         beam.smp  [kernel.kallsyms]      [k] tcp_recvmsg                                
     0.60%         beam.smp  [kernel.kallsyms]      [k] __audit_syscall_exit                       
     0.59%         beam.smp  [kernel.kallsyms]      [k] local_bh_enable                            
     0.58%         beam.smp  [nf_conntrack]         [k] nf_conntrack_in                            
     0.57%         beam.smp  [kernel.kallsyms]      [k] tcp_ack                                    
     0.56%         beam.smp  [kernel.kallsyms]      [k] put_prev_task_fair                         
     0.56%         beam.smp  beam.smp               [.] erts_thr_progress_leader_update            
     0.54%         beam.smp  [kernel.kallsyms]      [k] net_rx_action                              
     0.54%         beam.smp  [kernel.kallsyms]      [k] futex_wait_setup                           
     0.53%         beam.smp  [kernel.kallsyms]      [k] tcp_sendmsg                                
     0.51%         beam.smp  [kernel.kallsyms]      [k] tcp_v4_rcv                                 
     0.49%         beam.smp  [kernel.kallsyms]      [k] _raw_spin_lock_irqsave                     
     0.49%         beam.smp  [kernel.kallsyms]      [k] copy_user_generic_string                   
     0.49%         beam.smp  [kernel.kallsyms]      [k] fget_light                                 
     0.48%         beam.smp  [kernel.kallsyms]      [k] __do_softirq                               
     0.48%         beam.smp  [kernel.kallsyms]      [k] local_bh_enable_ip                         
     0.47%         beam.smp  [bridge]               [k] br_handle_frame                            
     0.46%         beam.smp  [kernel.kallsyms]      [k] __inet_lookup_established                  
     0.44%         beam.smp  [kernel.kallsyms]      [k] __schedule                                 
     0.43%         beam.smp  [kernel.kallsyms]      [k] ip_rcv                                     
     0.42%         beam.smp  [kernel.kallsyms]      [k] ip_finish_output                           
     0.42%         beam.smp  beam.smp               [.] erts_dsig_send_msg                         
     0.42%         beam.smp  [kernel.kallsyms]      [k] futex_wake                                 
     0.41%         beam.smp  beam.smp               [.] encode_size_struct_int                     
     0.40%         beam.smp  [kernel.kallsyms]      [k] process_backlog                            


More information about the erlang-questions mailing list