Open Stack

That looks promising.  I'll report back to confirm the solution.

Thanks!

---
v/r

Chris Apsey
bitskrieg at bitskrieg.net
https://www.bitskrieg.net

On 2018-01-31 04:40 PM, Matt Riedemann wrote:
> On 1/31/2018 3:16 PM, Chris Apsey wrote:
>> All,
>> 
>> Running in to a strange issue I haven't seen before.
>> 
>> Randomly, the nova-compute services on compute nodes are disabling 
>> themselves (as if someone ran openstack compute service set --disable 
>> hostX nova-compute.  When this happens, the node continues to report 
>> itself as 'up' - the service is just disabled.  As a result, if enough 
>> of these occur, we get scheduling errors due to lack of available 
>> resources (which makes sense).  Re-enabling them works just fine and 
>> they continue on as if nothing happened.  I looked through the logs 
>> and I can find the API calls where we re-enable the services (PUT 
>> /v2.1/os-services/enable), but I do not see any API calls where the 
>> services are getting disabled initially.
>> 
>> Is anyone aware of any cases where compute nodes will automatically 
>> disable their nova-compute service on their own, or has anyone seen 
>> this before and might know a root cause?  We have plenty of spare 
>> vcpus and RAM on each node - like less than 25% utilization (both in 
>> absolute terms and in terms of applied ratios).
>> 
>> We're seeing follow-on errors regarding rmq messages getting lost and 
>> vif-plug failures, but we think those are a symptom, not a cause.
>> 
>> Currently running pike on Xenial.
>> 
>> ---
>> v/r
>> 
>> Chris Apsey
>> bitskrieg at bitskrieg.net
>> https://www.bitskrieg.net
>> 
>> _______________________________________________
>> OpenStack-operators mailing list
>> OpenStack-operators at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
> 
> 
> This is actually a feature added in Pike:
> 
> https://review.openstack.org/#/c/463597/
> 
> This came up in discussion with operators at the Forum in Boston.
> 
> The vif-plug failures are likely the reason those computes are getting 
> disabled.
> 
> There is a config option "consecutive_build_service_disable_threshold"
> which you can set to disable the auto-disable behavior as some have
> experienced issues with it:
> 
> https://bugs.launchpad.net/nova/+bug/1742102