Open Stack

Hi Matthew,

Thanks for bringing some light on what problems has nova with evacuation of an instance. It is very important to have those limitations in mind when preparing final solution. Or to fix them, as you proposed.

Nevertheless, I would say that evacuationD does more than what calling 'nova host-evacuate' do. Let's consider such scenario:

1. Call 'nova host evacuate HostX'
2. Caller dies during call - information that some VMs are still to be evacuated is lost.

Such thing would not happen with evacuationD, because it prepares one rabbitMQ message for each VM that needs to be evacuated. Moreover, it deals with situation, when process that lists VMs crashes. In such case, whole operation would be continued by another daemon.

EvacD may also handle another problem that you mentioned: failure of target host of evacuation. In such scenario, 'evacuate host' message will be send for a new host and EvacD will try to evacuate all of it's vms - even those in rebuild state. Of course, evacuation of such instances fails, but they would eventually enter error state and evacuationD would start resurrection process. This can be speed up by setting instances state to 'error' (despite these which are in 'active' state) on the beginning of whole 'evacuate host' process.

Finally, another action - called 'Look for VM' - could be added. It would check if given VM ended up in active state on new hosts; if no, VM could be rebuild. I hope this would give us as much certainty that VM is alive as possible.

Dawid

On Tue, 2015-10-06 at 16:34 +0100, Matthew Booth wrote:
Hi, Roman,

Evacuated has been on my radar for a while and this post has prodded me to take a look at the code. I think it's worth starting by explaining the problems in the current solution. Nova client is currently responsible for doing this evacuate. It does:

1. List all instances on the source host
2. Initiate evacuate for each instance

Evacuating a single instance does:

API:
1. Set instance task state to rebuilding
2. Create a migration record with source and dest if specified

Conductor:
3. Call the scheduler to get a destination host if not specified
4. Get the migration object from the db

Compute:
5. Rebuild the instance on dest
6. Update instance.host to dest

Examining single instance evacuation, the first obvious thing to look at is what if 2 happen simultaneously. Because step 1 is atomic, it should not be possible to initiate 2 evacuations simultaneously of a single instance. However, note that this atomic action hasn't updated the instance host, meaning the source host remains the owner of this instance. If the evacuation process fails to complete, the source host will automatically delete it if it comes back up because it will find a migration record, but it will not be rebuilt anywhere else. Evacuating it again will fail, because its task state is already rebuilding.

Also, let's imagine that the conductor crashes. There is not enough state for any tool, whether internal or external, to be able to know if the rebuild is ongoing somewhere or not, and therefore whether it is safe to retry even if that retry would succeed, which it wouldn't.

Which is to say that we can't currently robustly evacuate one instance!

Looking at the nova client side, there is an obvious race there: there is no guarantee in step 2 that instances returned in step one have not already been evacuated by another process. We're protected here, though because evacuating a single instance twice will fail the second time. Note that the process isn't idempotent, though, because an evacuation which falls into a hole will never be retried.

Moving on to what evacuated does. Evacuated uses rabbit to distribute jobs reliably. There are 2 jobs in evacuated:

1. Evacuate host:
  1.1 Get list of all instances on the source host from Nova
  1.2 Send an evacuate vm job for each instance
2. Evacuate vm:
  2.1 Tell Nova to start evacuating an instance

Because we're using rabbit as a reliable message bus, the initiator of one of the tasks knows that it will eventually run to completion at least once. Note that there's nothing to prevent the task being executed more than once per call, though. A task may crash before sending an ack, or may just be really slow. However, in both cases, for exactly the same reasons as for the implementation in nova client, running more than once should not race. It is still not idempotent, though, again for exactly the same reasons as nova client.

Also notice that, exactly as in the nova client implementation, we are not asserting that an instance has been evacuated. We are only asserting that we called nova.evacuate, which is to say that we got as far as step 2 in the evacuation sequence above.

In other words, in terms of robustness, calling evacuated's evacuate host is identical to asserting that nova client's evacuate host ran to completion at least once, which is quite a lot simpler to do. That's still not very robust, though: we don't recover from failures, and we don't ensure that an instance is evacuated, only that we started an attempt to evacuate at least once. I'm obviously not satisfied with nova client, however as the implementation is simpler I would favour it over evacuated.

I believe we can solve this problem, but I think that without fixing single-instance evacuate we're just pushing the problem around (or creating new places for it to live). I would base the robustness of my implementation on a single principal:

  An instance has a single owner, which is exclusively responsible for rebuilding it.

In outline, I would redefine the evacuate process to do:

API:
1. Call the scheduler to get a destination for the evacuate if none was given.
2. Atomically update instance.host to this destination, and task state to rebuilding.

Compute:
3. Rebuild the instance.

This would be supported by a periodic task on the compute host which looks for rebuilding instances assigned to this host which aren't currently rebuilding, and kicks off a rebuild for them. This would cover the compute going down during a rebuild, or the api going down before messaging the compute.

Implementing this gives us several things:

1. The list instances, evacuate all instances process becomes idempotent, because as soon as the evacuate is initiated, the instance is removed from the source host.
2. We get automatic recovery of failure of the target compute. Because we atomically moved the instance to the target compute immediately, if the target compute also has to be evacuated, our instance won't fall through the gap.
3. We don't need an additional place for the code to run, because it will run on the compute. All the work has to be done by the compute anyway. By farming the evacuates out directly and immediately to the target compute we reduce both overhead and complexity.

The coordination becomes very simple. If we've run the nova client evacuation anywhere at least once, the actual evacuations are now Sombody Else's Problem (to quote h2g2), and will complete eventually. As evacuation in any case involves a forced change of owner it requires fencing of the source and implies an external agent such as pacemaker. The nova client evacuation can run in pacemaker.

Matt

On Fri, Oct 2, 2015 at 2:05 PM, Roman Dobosz <roman.dobosz at intel.com<mailto:roman.dobosz at intel.com>> wrote:
Hi all,

The case of automatic evacuation (or resurrection currently), is a topic
which surfaces once in a while, but it isn't yet fully supported by
OpenStack and/or by the cluster services. There was some attempts to
bring the feature into OpenStack, however it turns out it cannot be
easily integrated with. On the other hand evacuation may be executed
from the outside using Nova client or Nova API calls for evacuation
initiation.

I did some research regarding the ways how it could be designed, based
on Russel Bryant blog post[1] as a starting point. Apart from it, I've
also taken high availability and reliability into consideration when
designing the solution.

Together with coworker, we did first PoC[2] to enable cluster to be able
to perform evacuation. The idea behind that PoC was simple - providing
additional, small service which would trigger and supervise the
evacuation process, which would be triggered from the outside (in this
example we were using Pacemaker fencing facility, but it might be
anything) using RabbitMQ directly. Those services are running on the
control plane in AA fashion.

That work well for us. So we started exploring other possibilities like
oslo.messaging just to use it in the same manner as we did in the poc.
It turns out that the implementation will not be as easy, because there
is no facility in the oslo.messaging for letting sending an ACK from the
client after the job is done (not as soon as it gets the message). We
also looked at the existing OpenStack projects for a candidate which
provide service for managing long running tasks.

There is the Mistral project, which gives us almost all the features we
need. The one missing feature is the HA of the Mistral tasks execution.

The question is, how such problem (long running tasks) could be resolved
in OpenStack?

[1] http://blog.russellbryant.net/2014/10/15/openstack-instance-ha-proposal/
[2] https://github.com/dawiddeja/evacuationd

--
Cheers,
Roman Dobosz

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe<http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org<mailto:OpenStack-dev-request at lists.openstack.org>?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev