Open Stack

We have memcache enabled on the metadata servers.  Part of our load is
because we have a cron job that pulls the metadata and does some stuff on
the server every ~10 minutes.  We staggered the start times so that the
requests are spread out over a period of time and in general concurrent
requests are relatively low, under 10 per second max.

We have looked at FD for the neutron process as well as the database.  We
are not hitting a db deadlock and the slow query log - the slowest query
is ~1 second.

We are using the following version of oslo components: messaging 1.4.1, db
1.3.0, config 1.4.0, serialization 1.1.0, utils 1.1.0

In general we do not have any process using high amount of cpu (asside
from rabbit).  We see load on multiple neutron processes but its usually
under 7% per process and only 7-8 processes show up in the process list at
a time.

We increased the neutron timeout value to 120 seconds and it still get
read timeouts.  I am about to standup a neutron api server local on the
metadata server so that it specifically talks to that neutron instance and
see if that makes things better.
____________________________________________
 
Kris Lindgren
Senior Linux Systems Engineer
GoDaddy, LLC.



On 2/20/15, 12:27 AM, "Robert van Leeuwen"
<Robert.vanLeeuwen at spilgames.com> wrote:

>> After our icehouse -> juno upgrade we are noticing sporadic but
>>frequent errors from nova-metadata when trying to > serve metadata
>>requests.  The error is the following:
>
>>Is anyone else noticing this or frequent read timeouts when talking to
>>neutron?  Have you found a solution?
>> What  have you tried?
>
>If it is metadata agent load related:
>Are you caching the metadata info? (configure memcached_servers in
>nova.conf)
>We noticed a huge benefit on the metadata agent performance.
>Within our cloud the biggest load is created from facter (puppet) which
>will query the metadata agent.
>This adds up to quite a lot of requests to the metadata agent quickly.
>
>Adding caching significantly decreases the load on all systems.
>(Without caching we have a VERY high nova-conductor load)
>
>Cheers,
>Rober van Leeuwen