Open Stack

On 6 December 2015 at 09:33, Jay Pipes <jaypipes at gmail.com> wrote:
>
> On 12/04/2015 11:50 PM, Michał Jastrzębski wrote:
>>
>> Hey guys,
>>
>> Orchiestrated upgrades is one of our highest priorities for M in kolla,
>> so following up after discussion on summit I'd like to suggest an approach:
>>
>> Instead of creating playbook called "upgrade my openstack" we will
>> create "upgrade my nova" instead and approach to each service case by
>> case (since all of our running services are in dockers, this is possible).
>> We will also make use of image tags as version carriers, so ansible will
>> deploy new container only if version tag differs from what we ask it to
>> deploy. This will help with idempotency of upgrade process.
>>
>> So let's start with nova. Upgrade-my-nova playbook will do something
>> like this:
>>
>> 0. We create snapshot of our mariadb-data container. This will affect
>> every service, but it's always good to have backup and rollback of db
>> will be manual action
>>
>> 1. Nova bootstrap will be called and it will perform db-migration. Since
>> current approach to nova code is add-only we shouldn't need to stop
>> services and old services should keep working on newer database. Also
>> for minor version upgrades there will be no action here unless there is
>> migration.
>> 2. We upgrade all conductor at the same time. This should take mere
>> seconds since we'll have prebuilt containers
>> 3. We will upgrade rest of controller services with using "series: 1" in
>> ansible to ensure rolling upgrades.
>> 4. We will upgrade all of nova-compute services on it's own pace.
>>
>> This workflow should be pretty robust (I think it is) and it should also
>> provide idempotency.
>
>
> From Nova's perspective, most of the above looks OK to me, however, I am in more in favor of an upgrade strategy that builds a "new" set of conductor or controller service containers all at once and then flips the VIP or managed IP from the current controller pod containers to the newly-updated controller pod containers.

Well, we can't really do that because that would affect all other
services on controller host, and we want to minimize impact on rest of
cluster during upgrade. Containers gives us option to upgrade "just
nova", or even "just nova conductors" without affecting anything else.
On the bright side, redeploying pre-built container is a matter of
seconds (or even less). Whole process from turning off last conductor
to spawn first new conductor should take less than 10s, and that's
only downtime we should get there.

> This seems to me to be the most efficient (in terms of minimal overall downtime) and most robust (because a "rollback" is simply flipping the VIP back to the original controller/conductor service pod containers) solution for upgrading Nova.


>
> Dan Smith, can you comment on this since you are the God of Upgrades?
>
> Best,
> -jay
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev