<div dir="ltr">Hey,<div><br></div><div>Well, example you describe is similar to a situation "what if my computer dies when I'm in the middle of an alter database". I'd argue that we should at least for start assume that operators won't do stupid things intentionally (but bad things might happen, hence step 0). I don't assume we'll ever be able to solve this kind of problems, apart from making restore backup as easy as possible, which fortunately docker allows us to do.</div><div>Also I'd rather start from getting "normal" upgrades ready and keep improving them.</div><div><br></div><div>Cheers,</div><div>Michal</div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On 4 December 2015 at 16:18, Steven Dake (stdake) <span dir="ltr"><<a href="mailto:stdake@cisco.com" target="_blank">stdake@cisco.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word;color:rgb(0,0,0);font-size:14px;font-family:Calibri,sans-serif">
<div>Michal,</div>
<div><br>
</div>
<div>It looks pretty good but I want neo-style [1] upgrades. If someone rebuilds a container with a migration, I want to make sure that container isn't deployed until nova DB has been upgraded.</div>
<div><br>
</div>
<div>Can you work this into your model?</div>
<div><br>
</div>
<div>Regards</div>
<div>-steve</div>
<div><br>
</div>
<div>[1] <a href="https://www.youtube.com/watch?v=guVAeFs5XwE" target="_blank">https://www.youtube.com/watch?v=guVAeFs5XwE</a></div>
<div><br>
</div>
<span>
<div style="font-family:Calibri;font-size:11pt;text-align:left;color:black;BORDER-BOTTOM:medium none;BORDER-LEFT:medium none;PADDING-BOTTOM:0in;PADDING-LEFT:0in;PADDING-RIGHT:0in;BORDER-TOP:#b5c4df 1pt solid;BORDER-RIGHT:medium none;PADDING-TOP:3pt">
<span style="font-weight:bold">From: </span>Michał Jastrzębski <<a href="mailto:inc007@gmail.com" target="_blank">inc007@gmail.com</a>><br>
<span style="font-weight:bold">Reply-To: </span>"<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a>" <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Date: </span>Friday, December 4, 2015 at 1:50 PM<br>
<span style="font-weight:bold">To: </span>"<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a>" <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Subject: </span>[openstack-dev] [kolla][nova] Orchiestrated upgrades in kolla<br>
</div><div><div class="h5">
<div><br>
</div>
<div>
<div>
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>Hey guys,<br>
<br>
</div>
Orchiestrated upgrades is one of our highest priorities for M in kolla, so following up after discussion on summit I'd like to suggest an approach:<br>
<br>
</div>
Instead of creating playbook called "upgrade my openstack" we will create "upgrade my nova" instead and approach to each service case by case (since all of our running services are in dockers, this is possible).<br>
</div>
<div>We will also make use of image tags as version carriers, so ansible will deploy new container only if version tag differs from what we ask it to deploy. This will help with idempotency of upgrade process.<br>
</div>
<div><br>
</div>
So let's start with nova. Upgrade-my-nova playbook will do something like this:<br>
<br>
</div>
0. We create snapshot of our mariadb-data container. This will affect every service, but it's always good to have backup and rollback of db will be manual action<br>
<br>
</div>
<div>1. Nova bootstrap will be called and it will perform db-migration. Since current approach to nova code is add-only we shouldn't need to stop services and old services should keep working on newer database. Also for minor version upgrades there will be
no action here unless there is migration.<br>
</div>
2. We upgrade all conductor at the same time. This should take mere seconds since we'll have prebuilt containers<br>
</div>
3. We will upgrade rest of controller services with using "series: 1" in ansible to ensure rolling upgrades.<br>
</div>
4. We will upgrade all of nova-compute services on it's own pace.<br>
<br>
</div>
This workflow should be pretty robust (I think it is) and it should also provide idempotency.<br>
<br>
</div>
<div>Thoughts?<br>
<br>
</div>
<div>Regards,<br>
</div>
<div>Michal<br>
</div>
</div>
</div>
</div>
</div></div></span>
</div>
<br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div>