Open Stack

On 12/07/2015 04:28 AM, Michał Jastrzębski wrote:
> On 6 December 2015 at 09:33, Jay Pipes <jaypipes at gmail.com> wrote:
>>
>> On 12/04/2015 11:50 PM, Michał Jastrzębski wrote:
>>>
>>> Hey guys,
>>>
>>> Orchiestrated upgrades is one of our highest priorities for M in
>>> kolla, so following up after discussion on summit I'd like to
>>> suggest an approach:
>>>
>>> Instead of creating playbook called "upgrade my openstack" we
>>> will create "upgrade my nova" instead and approach to each
>>> service case by case (since all of our running services are in
>>> dockers, this is possible). We will also make use of image tags
>>> as version carriers, so ansible will deploy new container only if
>>> version tag differs from what we ask it to deploy. This will help
>>> with idempotency of upgrade process.
>>>
>>> So let's start with nova. Upgrade-my-nova playbook will do
>>> something like this:
>>>
>>> 0. We create snapshot of our mariadb-data container. This will
>>> affect every service, but it's always good to have backup and
>>> rollback of db will be manual action
>>>
>>> 1. Nova bootstrap will be called and it will perform
>>> db-migration. Since current approach to nova code is add-only we
>>> shouldn't need to stop services and old services should keep
>>> working on newer database. Also for minor version upgrades there
>>> will be no action here unless there is migration. 2. We upgrade
>>> all conductor at the same time. This should take mere seconds
>>> since we'll have prebuilt containers 3. We will upgrade rest of
>>> controller services with using "series: 1" in ansible to ensure
>>> rolling upgrades. 4. We will upgrade all of nova-compute services
>>> on it's own pace.
>>>
>>> This workflow should be pretty robust (I think it is) and it
>>> should also provide idempotency.
>>
>>
>> From Nova's perspective, most of the above looks OK to me, however,
>> I am in more in favor of an upgrade strategy that builds a "new"
>> set of conductor or controller service containers all at once and
>> then flips the VIP or managed IP from the current controller pod
>> containers to the newly-updated controller pod containers.
>
> Well, we can't really do that because that would affect all other
> services on controller host, and we want to minimize impact on rest
> of cluster during upgrade. Containers gives us option to upgrade
> "just nova", or even "just nova conductors" without affecting
> anything else. On the bright side, redeploying pre-built container is
> a matter of seconds (or even less). Whole process from turning off
> last conductor to spawn first new conductor should take less than
> 10s, and that's only downtime we should get there.

Sorry, I guess I wasn't clear. I was not proposing to put all controller 
services in a single container. I was proposing simply to build the 
various container sets (nova-conductor, nova-api, etc) with the updated 
code for that service and then "flip the target IP or port" for that 
particular service from the existing container set's VIP to the 
new/updated container set's VIP.

In Kubernetes-speak, you would create a new Pod with containers having 
the updated Nova conductor code, and would set the new Pod's selector to 
something different than the existing Pod's selector -- using the Git 
SHA1 hash for the selector would be perfect... You would then update the 
Service object's selector to target the new Pod instead of the old one.

Or, alternately, you could use Kubernetes' support for rolling updates 
[1] of a service using a ReplicationController that essentially does the 
Pod and Service orchestration for you.

Hope that is a little more clear what I was referring to...

Best,
-jay

[1] 
https://github.com/kubernetes/kubernetes/blob/master/docs/user-guide/replication-controller.md#rolling-updates

>> This seems to me to be the most efficient (in terms of minimal
>> overall downtime) and most robust (because a "rollback" is simply
>> flipping the VIP back to the original controller/conductor service
>> pod containers) solution for upgrading Nova.
>
>
>>
>> Dan Smith, can you comment on this since you are the God of
>> Upgrades?
>>
>> Best, -jay
>>
>>
>> __________________________________________________________________________
>>
>>
OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __________________________________________________________________________
>
>
OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>