Open Stack

Hi Stuart,

Please see below.


> -----Original Message-----
> From: Ahmed RAHAL [mailto:arahal at iweb.com]
> Sent: 18 août 2014 10:54
> To: openstack-operators at lists.openstack.org
> Subject: Re: [Openstack-operators] Neutron nf_conntrack performance
> 
> Hi,
> 
> Le 2014-08-16 02:44, Stuart Fox a écrit :
> > Hey other ops!
> >
> > Im having a serious problem with my neutron router getting spin locked
> > in nf_conntrack_tuple_taken.
> > Has anybody else experienced it?
> > As the incoming request rate goes up, so nf_conntrack_tuple_taken runs
> > very hot on CPU0 causing ksoftirqd/0 to run at 100%. At that point
> > internal pings on the GRE network go sky high and its game over.
> >
> > Ubuntu 14.04/Icehouse 2014.1.1 on an ibm x3550 with 4 10G intel nics.
> > eth0 - Mgt
> > eth1 - GRE
> > eth2 - Public
> > eth3 - unused

In order to deal w/ high softirq CPU usage that increases with the number of tracked connections, you need to scale the size of the hash table which is used to store the conntrack entries (/sys/module/nf_conntrack/parameters/hashsize). Preferably, you should have a 1:1 mapping between conntrack entries and hash buckets. 

This page, while a little old, is a great reference: http://wiki.khnet.info/index.php/Conntrack_tuning

That being said, I don't think you actually mention the number of conntrack'ed connections that you are dealing with.

-m

> 
> We ran into similar problems with softirq.
> While the problem often comes from load and how it's handled, an initial
> workaround we applied was this:
> 
> for i in $(find /sys/class/net/eth*/queues -type d -name "rx-*"); do echo $i;
> echo fffffff > $i/rps_cpus ; done
> 
> This sets all CPUs to handle recieved packets.
> https://access.redhat.com/documentation/en-
> US/Red_Hat_Enterprise_Linux/6/html/Performance_Tuning_Guide/network-
> rps.html
> 
> (note that this assumes your interfaces are called ethX. this turns out to no
> longer be true on ubuntu 14.04)
> 
> HTH.
> 
> --
> Ahmed Rahal
> 
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators