Open Stack

On 12/04/2015 11:50 PM, Michał Jastrzębski wrote:
> Hey guys,
>
> Orchiestrated upgrades is one of our highest priorities for M in kolla,
> so following up after discussion on summit I'd like to suggest an approach:
>
> Instead of creating playbook called "upgrade my openstack" we will
> create "upgrade my nova" instead and approach to each service case by
> case (since all of our running services are in dockers, this is possible).
> We will also make use of image tags as version carriers, so ansible will
> deploy new container only if version tag differs from what we ask it to
> deploy. This will help with idempotency of upgrade process.
>
> So let's start with nova. Upgrade-my-nova playbook will do something
> like this:
>
> 0. We create snapshot of our mariadb-data container. This will affect
> every service, but it's always good to have backup and rollback of db
> will be manual action
>
> 1. Nova bootstrap will be called and it will perform db-migration. Since
> current approach to nova code is add-only we shouldn't need to stop
> services and old services should keep working on newer database. Also
> for minor version upgrades there will be no action here unless there is
> migration.
> 2. We upgrade all conductor at the same time. This should take mere
> seconds since we'll have prebuilt containers
> 3. We will upgrade rest of controller services with using "series: 1" in
> ansible to ensure rolling upgrades.
> 4. We will upgrade all of nova-compute services on it's own pace.
>
> This workflow should be pretty robust (I think it is) and it should also
> provide idempotency.

 From Nova's perspective, most of the above looks OK to me, however, I 
am in more in favor of an upgrade strategy that builds a "new" set of 
conductor or controller service containers all at once and then flips 
the VIP or managed IP from the current controller pod containers to the 
newly-updated controller pod containers.

This seems to me to be the most efficient (in terms of minimal overall 
downtime) and most robust (because a "rollback" is simply flipping the 
VIP back to the original controller/conductor service pod containers) 
solution for upgrading Nova.

Dan Smith, can you comment on this since you are the God of Upgrades?

Best,
-jay