Open Stack

Hi Operators,

Could you please let me know your opinion on below scenario? It will help me to proceed my work.

Thank you,

Abhishek

-----Original Message-----
From: Kekane, Abhishek [mailto:Abhishek.Kekane at nttdata.com] 
Sent: Tuesday, July 04, 2017 2:52 PM
To: OpenStack Development Mailing List (not for usage questions); openstack-operators at lists.openstack.org
Subject: Re: [Openstack-operators] [openstack-dev] [masakari][nova] Allow evacuation of instances in resized state

Hi operators,

I want to know how evacuation of resized instances is handled in real environment.
For example if the vm is in resized state and if the compute host on which the vm is resized goes down, then how will operator evacuate the vm.

One possible way is to reset that vm state to error and then evacuate it to new compute host.
Please refer below scenario for reference:

Scenario:
=========

Pre-conditions:
--------------------
1. Config option allow_resize_to_same_host is False.
2. Instance path is not mounted on shared storage.
3. Three compute nodes: "compute node A", "compute node B" and "compute node C"

Steps:
--------
1. Boot an instance on "compute node A".
2. User tries to resize the newly created instance and nova-scheduler selects "compute node B" as a destination node for resize.
   In this case nova creates a instance directory on destination "compute node B" and mark the instance directory which is present on the source "compute node A" as "*_resize".

Note that the resize operation is yet not confirmed and "compute node B" goes down.

3. Reset instance state to ERROR as nova allows evacuation only if instance state is 'ACTIVE', 'STOPPED' or 'ERROR'.
4. Evacuate the instance to "compute node C" using target_host option.
   As a result, instance files which were on "compute node B" will be cleaned up after compute service on it is up again, but instance files which were on "compute node A" marked as "*_resize" will never be cleaned up. As of now there is no periodic task in nova to perform cleanup of these kinds of scenarios.


Questions:
1. is this the only possible way of evacuating the resized instances in real world scenario?
2. If yes is there any way to cleanup unused (*_resize) instance files from the source compute node other than cleaning up it manually?
3. Should we add support of evacuating of resized instances in nova?

Please let me know your opinions about the same.


Thank you,

Abhishek Kekane



-----Original Message-----
From: Kekane, Abhishek [mailto:Abhishek.Kekane at nttdata.com]
Sent: Thursday, June 29, 2017 5:57 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [masakari][nova] Allow evacuation of instances in resized state

Hi Chris,

IMO we cannot perform auto-confirm as confirming or reverting is user's choice, whereas reverting is not possible as the node where the instance is resized is down.
As suggested by you allowing this in nova require additional work. It is possible if we take power-state into consideration for evacuation operation, i.e. while evacuation if instance vmstate is resized and power-state is shutoff then we can stop that instance after evacuation.

Thank you,

Abhishek Kekane 

-----Original Message-----
From: Chris Friesen [mailto:chris.friesen at windriver.com]
Sent: Wednesday, June 28, 2017 8:54 PM
To: openstack-dev at lists.openstack.org
Subject: Re: [openstack-dev] [masakari][nova] Allow evacuation of instances in resized state

On 06/28/2017 05:50 AM, Kekane, Abhishek wrote:

> In masakari, we are setting an instance to an error state if the 
> vmstate is resized before evacuating it to a new host.

Arguably the instance should be set to an error state as soon as you notice that the compute node is down.

> Once an instance (which was in
> resized state) is evacuated then it becomes active on the new host. 
> The main problem with this implementation from user’s point of view is 
> the instance goes into active state after evacuation, it should be in 
> stopped state if the prior action on the instance before resizing was 
> stop. In masakari, It’s possible to set the vm state to stopped state 
> after evacuation but for a short period the instance will go into the active state which is unacceptable.

That's a valid point, I think.

> *Proposing changes to Nova:*
>
> In the current nova code, if the instance is in stopped state before 
> evacuation, then it remains in the stopped state after evacuation is 
> complete. On the similar lines, we are proposing nova should allow 
> instance to be evacuated in resized state and after evacuation the 
> instance should remain in stopped state if the prior action on the instance is stopped before resizing.

The current nova code looks at the vm_state to decide whether or not it's allowable to evacuate, and while "stopped" is a valid state to evacuate from "resized" is not.  In your scenario it's both "stopped" *and* "resized" 
simultaneously, but there's no way to represent that in the vmstate so I think we'd have to check the power state, which would mean extending the
check_instance_state() routine since it doesn't currently handle the power state.

The trickier question is how to handle the "resized" state...after evacuating an instance in the "resized" state should you be able to revert the resize?  If so, how would that work in the case where the instance was resized on the same host originally and that host is no longer available?  If not, then you'll end up with resources permanently reserved on the host the instance was on before the resize.  I suppose one option would be to auto-confirm the resize in the case of a resize-to-same-host, but that'll be tricky to process with the host not available.

Also, it should be noted that when rebuilding/evacuating a "stopped" instance the nova code just boots it up as normal and sets the vm_state to "active", then realizes that it's supposed to be stopped and sets the task_state to "powering_off" and goes down the normal path to stop the instance, eventually setting the vm_state to "stopped".  So you're still going to end up with the same state transitions as what you have now, though the timing will probably be a bit tighter.  If you really want a stopped instance to not actually start up on a rebuild/evacuate then that would be additional work.

Chris

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

______________________________________________________________________
Disclaimer: This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding.
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

______________________________________________________________________
Disclaimer: This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding.
_______________________________________________
OpenStack-operators mailing list
OpenStack-operators at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

______________________________________________________________________
Disclaimer: This email and any attachments are sent in strictest confidence
for the sole use of the addressee and may contain legally privileged,
confidential, and proprietary data. If you are not the intended recipient,
please advise the sender by replying promptly to this email and then delete
and destroy this email and any attachments without any further use, copying
or forwarding.