Open Stack

Jay's approach is really neat and that how it done in big productions,
sadly there is a problem with implementation of it in Kolla. I'd consider
to dig into it, maybe there is nice way to do so. If not, original approach
is ok too.

On Mon, Dec 7, 2015 at 10:06 AM, Steven Dake (stdake) <stdake at cisco.com>
wrote:

>
>
> On 12/6/15, 9:47 PM, "Jay Pipes" <jaypipes at gmail.com> wrote:
>
> >On 12/07/2015 04:28 AM, Michał Jastrzębski wrote:
> >> On 6 December 2015 at 09:33, Jay Pipes <jaypipes at gmail.com> wrote:
> >>>
> >>> On 12/04/2015 11:50 PM, Michał Jastrzębski wrote:
> >>>>
> >>>> Hey guys,
> >>>>
> >>>> Orchiestrated upgrades is one of our highest priorities for M in
> >>>> kolla, so following up after discussion on summit I'd like to
> >>>> suggest an approach:
> >>>>
> >>>> Instead of creating playbook called "upgrade my openstack" we
> >>>> will create "upgrade my nova" instead and approach to each
> >>>> service case by case (since all of our running services are in
> >>>> dockers, this is possible). We will also make use of image tags
> >>>> as version carriers, so ansible will deploy new container only if
> >>>> version tag differs from what we ask it to deploy. This will help
> >>>> with idempotency of upgrade process.
> >>>>
> >>>> So let's start with nova. Upgrade-my-nova playbook will do
> >>>> something like this:
> >>>>
> >>>> 0. We create snapshot of our mariadb-data container. This will
> >>>> affect every service, but it's always good to have backup and
> >>>> rollback of db will be manual action
> >>>>
> >>>> 1. Nova bootstrap will be called and it will perform
> >>>> db-migration. Since current approach to nova code is add-only we
> >>>> shouldn't need to stop services and old services should keep
> >>>> working on newer database. Also for minor version upgrades there
> >>>> will be no action here unless there is migration. 2. We upgrade
> >>>> all conductor at the same time. This should take mere seconds
> >>>> since we'll have prebuilt containers 3. We will upgrade rest of
> >>>> controller services with using "series: 1" in ansible to ensure
> >>>> rolling upgrades. 4. We will upgrade all of nova-compute services
> >>>> on it's own pace.
> >>>>
> >>>> This workflow should be pretty robust (I think it is) and it
> >>>> should also provide idempotency.
> >>>
> >>>
> >>> From Nova's perspective, most of the above looks OK to me, however,
> >>> I am in more in favor of an upgrade strategy that builds a "new"
> >>> set of conductor or controller service containers all at once and
> >>> then flips the VIP or managed IP from the current controller pod
> >>> containers to the newly-updated controller pod containers.
> >>
> >> Well, we can't really do that because that would affect all other
> >> services on controller host, and we want to minimize impact on rest
> >> of cluster during upgrade. Containers gives us option to upgrade
> >> "just nova", or even "just nova conductors" without affecting
> >> anything else. On the bright side, redeploying pre-built container is
> >> a matter of seconds (or even less). Whole process from turning off
> >> last conductor to spawn first new conductor should take less than
> >> 10s, and that's only downtime we should get there.
> >
> >Sorry, I guess I wasn't clear. I was not proposing to put all controller
> >services in a single container. I was proposing simply to build the
> >various container sets (nova-conductor, nova-api, etc) with the updated
> >code for that service and then "flip the target IP or port" for that
> >particular service from the existing container set's VIP to the
> >new/updated container set's VIP.
> >
> >In Kubernetes-speak, you would create a new Pod with containers having
> >the updated Nova conductor code, and would set the new Pod's selector to
> >something different than the existing Pod's selector -- using the Git
> >SHA1 hash for the selector would be perfect... You would then update the
> >Service object's selector to target the new Pod instead of the old one.
> >
> >Or, alternately, you could use Kubernetes' support for rolling updates
> >[1] of a service using a ReplicationController that essentially does the
> >Pod and Service orchestration for you.
> >
> >Hope that is a little more clear what I was referring to...
> >
> >Best,
> >-jay
>
> Jay,
>
> Forgive me if this is a little incoherent - I have a root canal scheduled
> for tomorrow with the fun-filled day that comes  comes with :(
>
> We don't use docker-proxy in kolla.  Instead we use net=host mode.  I
> think what you propose would require an extra global VIP for upgrades (or
> using docker-proxy, which we don't want to do). I am not really hot on
> this approach for that reason.
>
> Regards
> -stee
>
> >
> >[1]
> >
> https://github.com/kubernetes/kubernetes/blob/master/docs/user-guide/repli
> >cation-controller.md#rolling-updates
> >
> >>> This seems to me to be the most efficient (in terms of minimal
> >>> overall downtime) and most robust (because a "rollback" is simply
> >>> flipping the VIP back to the original controller/conductor service
> >>> pod containers) solution for upgrading Nova.
> >>
> >>
> >>>
> >>> Dan Smith, can you comment on this since you are the God of
> >>> Upgrades?
> >>>
> >>> Best, -jay
> >>>
> >>>
> >>>
> >>>________________________________________________________________________
> >>>__
> >>>
> >>>
> >OpenStack Development Mailing List (not for usage questions)
> >>> Unsubscribe:
> >>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>
> >>
> >>_________________________________________________________________________
> >>_
> >>
> >>
> >OpenStack Development Mailing List (not for usage questions)
> >> Unsubscribe:
> >> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>
> >
> >__________________________________________________________________________
> >OpenStack Development Mailing List (not for usage questions)
> >Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Best regards,
Proskurin Kirill
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20151207/f7f0d722/attachment.html>