Open Stack

On 12/6/15, 9:47 PM, "Jay Pipes" <jaypipes at gmail.com> wrote:

>On 12/07/2015 04:28 AM, Michał Jastrzębski wrote:
>> On 6 December 2015 at 09:33, Jay Pipes <jaypipes at gmail.com> wrote:
>>>
>>> On 12/04/2015 11:50 PM, Michał Jastrzębski wrote:
>>>>
>>>> Hey guys,
>>>>
>>>> Orchiestrated upgrades is one of our highest priorities for M in
>>>> kolla, so following up after discussion on summit I'd like to
>>>> suggest an approach:
>>>>
>>>> Instead of creating playbook called "upgrade my openstack" we
>>>> will create "upgrade my nova" instead and approach to each
>>>> service case by case (since all of our running services are in
>>>> dockers, this is possible). We will also make use of image tags
>>>> as version carriers, so ansible will deploy new container only if
>>>> version tag differs from what we ask it to deploy. This will help
>>>> with idempotency of upgrade process.
>>>>
>>>> So let's start with nova. Upgrade-my-nova playbook will do
>>>> something like this:
>>>>
>>>> 0. We create snapshot of our mariadb-data container. This will
>>>> affect every service, but it's always good to have backup and
>>>> rollback of db will be manual action
>>>>
>>>> 1. Nova bootstrap will be called and it will perform
>>>> db-migration. Since current approach to nova code is add-only we
>>>> shouldn't need to stop services and old services should keep
>>>> working on newer database. Also for minor version upgrades there
>>>> will be no action here unless there is migration. 2. We upgrade
>>>> all conductor at the same time. This should take mere seconds
>>>> since we'll have prebuilt containers 3. We will upgrade rest of
>>>> controller services with using "series: 1" in ansible to ensure
>>>> rolling upgrades. 4. We will upgrade all of nova-compute services
>>>> on it's own pace.
>>>>
>>>> This workflow should be pretty robust (I think it is) and it
>>>> should also provide idempotency.
>>>
>>>
>>> From Nova's perspective, most of the above looks OK to me, however,
>>> I am in more in favor of an upgrade strategy that builds a "new"
>>> set of conductor or controller service containers all at once and
>>> then flips the VIP or managed IP from the current controller pod
>>> containers to the newly-updated controller pod containers.
>>
>> Well, we can't really do that because that would affect all other
>> services on controller host, and we want to minimize impact on rest
>> of cluster during upgrade. Containers gives us option to upgrade
>> "just nova", or even "just nova conductors" without affecting
>> anything else. On the bright side, redeploying pre-built container is
>> a matter of seconds (or even less). Whole process from turning off
>> last conductor to spawn first new conductor should take less than
>> 10s, and that's only downtime we should get there.
>
>Sorry, I guess I wasn't clear. I was not proposing to put all controller
>services in a single container. I was proposing simply to build the
>various container sets (nova-conductor, nova-api, etc) with the updated
>code for that service and then "flip the target IP or port" for that
>particular service from the existing container set's VIP to the
>new/updated container set's VIP.
>
>In Kubernetes-speak, you would create a new Pod with containers having
>the updated Nova conductor code, and would set the new Pod's selector to
>something different than the existing Pod's selector -- using the Git
>SHA1 hash for the selector would be perfect... You would then update the
>Service object's selector to target the new Pod instead of the old one.
>
>Or, alternately, you could use Kubernetes' support for rolling updates
>[1] of a service using a ReplicationController that essentially does the
>Pod and Service orchestration for you.
>
>Hope that is a little more clear what I was referring to...
>
>Best,
>-jay

Jay,

Forgive me if this is a little incoherent - I have a root canal scheduled
for tomorrow with the fun-filled day that comes  comes with :(

We don't use docker-proxy in kolla.  Instead we use net=host mode.  I
think what you propose would require an extra global VIP for upgrades (or
using docker-proxy, which we don't want to do). I am not really hot on
this approach for that reason.

Regards
-stee

>
>[1] 
>https://github.com/kubernetes/kubernetes/blob/master/docs/user-guide/repli
>cation-controller.md#rolling-updates
>
>>> This seems to me to be the most efficient (in terms of minimal
>>> overall downtime) and most robust (because a "rollback" is simply
>>> flipping the VIP back to the original controller/conductor service
>>> pod containers) solution for upgrading Nova.
>>
>>
>>>
>>> Dan Smith, can you comment on this since you are the God of
>>> Upgrades?
>>>
>>> Best, -jay
>>>
>>>
>>> 
>>>________________________________________________________________________
>>>__
>>>
>>>
>OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>> 
>>_________________________________________________________________________
>>_
>>
>>
>OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>__________________________________________________________________________
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev