Open Stack

On 08/25/15 at 06:08pm, Gary Kotton wrote:
>
>
>On 8/25/15, 9:10 AM, "Matt Riedemann" <mriedem at linux.vnet.ibm.com> wrote:
>
>>
>>
>>On 8/25/2015 10:03 AM, Gary Kotton wrote:
>>>
>>>
>>> On 8/25/15, 7:04 AM, "Matt Riedemann" <mriedem at linux.vnet.ibm.com>
>>>wrote:
>>>
>>>>
>>>>
>>>> On 8/24/2015 9:32 PM, Gary Kotton wrote:
>>>>> In item #2 below the reboot is down via the guest and not the nova
>>>>> api¹s :)
>>>>>
>>>>> From: Gary Kotton <gkotton at vmware.com <mailto:gkotton at vmware.com>>
>>>>> Reply-To: OpenStack List <openstack-dev at lists.openstack.org
>>>>> <mailto:openstack-dev at lists.openstack.org>>
>>>>> Date: Monday, August 24, 2015 at 7:18 PM
>>>>> To: OpenStack List <openstack-dev at lists.openstack.org
>>>>> <mailto:openstack-dev at lists.openstack.org>>
>>>>> Subject: [openstack-dev] [nova] periodic task
>>>>>
>>>>> Hi,
>>>>> A couple of months ago I posted a patch for bug
>>>>> https://launchpad.net/bugs/1463688. The issue is as follows: the
>>>>> periodic task detects that the instance state does not match the state
>>>>> on the hypervisor and it shuts down the running VM. There are a number
>>>>> of ways that this may happen and I will try and explain:
>>>>>
>>>>>   1. Vmware driver example: a host where the instances are running
>>>>>goes
>>>>>      down. This could be a power outage, host failure, etc. The first
>>>>>      iteration of the perdioc task will determine that the actual
>>>>>      instacne is down. This will update the state of the instance to
>>>>>      DOWN. The VC has the ability to do HA and it will start the
>>>>>instance
>>>>>      up and running again. The next iteration of the periodic task
>>>>>will
>>>>>      determine that the instance is up and the compute manager will
>>>>>stop
>>>>>      the instance.
>>>>>   2. All drivers. The tenant decides to do a reboot of the instance
>>>>>and
>>>>>      that coincides with the periodic task state validation. At this
>>>>>      point in time the instance will not be up and the compute node
>>>>>will
>>>>>      update the state of the instance as DWON. Next iteration the
>>>>>states
>>>>>      will differ and the instance will be shutdown
>>>>>
>>>>> Basically the issue hit us with our CI and there was no CI running
>>>>>for a
>>>>> couple of hours due to the fact that the compute node decided to
>>>>> shutdown the running instances. The hypervisor should be the source of
>>>>> truth and it should not be the compute node that decides to shutdown
>>>>> instances. I posted a patch to deal with this
>>>>> https://review.openstack.org/#/c/190047/. Which is the reason for this
>>>>> mail. The patch is backwards compatible so that the existing
>>>>>deployments
>>>>> and random shutdown continues as it works today and the admin now has
>>>>>an
>>>>> ability just to do a log if there is a inconsistency.
>>>>>
>>>>> We do not want to disable the periodic task as knowing the current
>>>>>state
>>>>> of the instance is very important and has a ton of value, we just do
>>>>>not
>>>>> want the periodic to task to shut down a running instance.
>>>>>
>>>>> Thanks
>>>>> Gary
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>_______________________________________________________________________
>>>>>__
>>>>> _
>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>> Unsubscribe:
>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>
>>>> In #2 the guest shouldn't be rebooted by the user (tenant) outside of
>>>> the nova-api.  I'm not sure if it's actually formally documented in the
>>>> nova documentation, but from what I've always heard/known, nova is the
>>>> control plane and you should be doing everything with your instances
>>>>via
>>>> the nova-api.  If the user rebooted via nova-api, the task_state would
>>>> be set and the periodic task would ignore the instance.
>>>
>>> Matt, this is one case that I showed where the problem occurs. There are
>>> others and I can invest time to see them. The fact that the periodic
>>>task
>>> is there is important. What I don¹t understand is why having an option
>>>of
>>> log indication for an admin is something that is not useful and instead
>>>we
>>> are going with having the compute node shutdown instance when this
>>>should
>>> not happen. Our infrastructure is behaving like cattle. That should not
>>>be
>>> the case and the hypervisor should be the source of truth.
>>>
>>> This is a serious issue and instances in production can and will go
>>>down.
>>>
>>>>
>>>> --
>>>>
>>>> Thanks,
>>>>
>>>> Matt Riedemann
>>>>
>>>>
>>>>
>>>>________________________________________________________________________
>>>>__
>>>> OpenStack Development Mailing List (not for usage questions)
>>>> Unsubscribe:
>>>>OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>>
>>>_________________________________________________________________________
>>>_
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>>OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>>For the HA case #1, the periodic task checks to see if the instance.host
>>doesn't match the compute service host [1] and skips if they don't match.
>>
>>Shouldn't your HA scenario be updating which host the instance is
>>running on?  Or is this a vCenter-ism?
>
>The nova compute node has not changed. It is not the compute nodes host.
>The host that the instance was running on was down and those instances
>were moved.

So this is a case where a single compute node is managing multiple 
hypervisors?  It sounds like there is an assumption being made in the 
periodic task that doesn't hold true for the VMware driver, that a 
request for the power state of an instance would fail if the host was 
down.  This may be a better fix here: to not sync the state if the host 
is down.



>
>For libvirt the same issues could happen if a process goes down and is
>restarted (there may be some race conditions). But I am not familiar
>enough with the ins and outs there. Just the fact that suggesting in some
>cases that people disable the periodic task indicates that this too is an
>issue.
>
>But seriously, we need this and the change is non intrusive, configuarble
>and backwards compatible. Honestly I see no reason why this is bing
>blocked.

The change seems to be under discussion here because this is adding more 
complexity to an already quite complex method.  I believe the desire is 
to find a model that simplifies, or at least doesn't add to the 
complexity of, the way that syncs are handled.

>
>
>>
>>[1]
>>http://git.openstack.org/cgit/openstack/nova/tree/nova/compute/manager.py#
>>n5871
>>
>>--
>>
>>Thanks,
>>
>>Matt Riedemann
>>
>>
>>__________________________________________________________________________
>>OpenStack Development Mailing List (not for usage questions)
>>Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>__________________________________________________________________________
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev