[Openstack-operators] Neutron nf_conntrack performance
marc.heckmann at ubisoft.com
Mon Aug 18 15:37:39 UTC 2014
Please see below.
> -----Original Message-----
> From: Ahmed RAHAL [mailto:arahal at iweb.com]
> Sent: 18 août 2014 10:54
> To: openstack-operators at lists.openstack.org
> Subject: Re: [Openstack-operators] Neutron nf_conntrack performance
> Le 2014-08-16 02:44, Stuart Fox a écrit :
> > Hey other ops!
> > Im having a serious problem with my neutron router getting spin locked
> > in nf_conntrack_tuple_taken.
> > Has anybody else experienced it?
> > As the incoming request rate goes up, so nf_conntrack_tuple_taken runs
> > very hot on CPU0 causing ksoftirqd/0 to run at 100%. At that point
> > internal pings on the GRE network go sky high and its game over.
> > Ubuntu 14.04/Icehouse 2014.1.1 on an ibm x3550 with 4 10G intel nics.
> > eth0 - Mgt
> > eth1 - GRE
> > eth2 - Public
> > eth3 - unused
In order to deal w/ high softirq CPU usage that increases with the number of tracked connections, you need to scale the size of the hash table which is used to store the conntrack entries (/sys/module/nf_conntrack/parameters/hashsize). Preferably, you should have a 1:1 mapping between conntrack entries and hash buckets.
This page, while a little old, is a great reference: http://wiki.khnet.info/index.php/Conntrack_tuning
That being said, I don't think you actually mention the number of conntrack'ed connections that you are dealing with.
> We ran into similar problems with softirq.
> While the problem often comes from load and how it's handled, an initial
> workaround we applied was this:
> for i in $(find /sys/class/net/eth*/queues -type d -name "rx-*"); do echo $i;
> echo fffffff > $i/rps_cpus ; done
> This sets all CPUs to handle recieved packets.
> (note that this assumes your interfaces are called ethX. this turns out to no
> longer be true on ubuntu 14.04)
> Ahmed Rahal
> OpenStack-operators mailing list
> OpenStack-operators at lists.openstack.org
More information about the OpenStack-operators