<div dir="ltr">Thanks Crag on starting the thread. Few comments inline.<br><br>On Fri, Oct 21, 2016 at 5:32 AM, Crag Wolfe <span dir="ltr"><<a href="mailto:cwolfe@redhat.com" target="_blank">cwolfe@redhat.com</a>></span> wrote:<br><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">At Summit, folks will be discussing the rolling upgrade issue across a<br>
couple of sessions. I personally won't be able to attend, but thought<br>
I would share my thoughts on the subject.<br>
<br>
To handle rolling upgrades, there are two general cases to consider:<br>
database model changes and RPC method signature changes.<br>
<br>
For DB Model changes (this has already been well discussed on the<br>
mailing list, see the footnotes), let's assume for the moment we don't<br>
want to use triggers. If we are moving data from one column/table to<br>
another, the pattern looks like:<br>
<br>
legacy release: write to old location<br>
release+1: write to old and new location, read from old<br>
release+2: write to old and new location, read from new,<br>
provide migration utility<br>
release+3: write to new location, read from new<br></blockquote><div>Not sure I understand this. Is it always about changing the table name or column name of a table? <br></div><div>What about adding a new column to an existing table? I assume the db api<br></div><div>implementation have to ignore the additional column values when writing to old location. <br></div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Works great! The main issue is if the duplicated old and new data<br>
happens to be large. For a heat-specific example (one that is close to<br>
my heart), consider moving resource/event properties data into a<br>
separate table.<br>
<br>
We could speed up the process by adding config variables that specify<br>
where to read from, but that is putting a burden on the operator,<br>
creating a risk that data is lost if the config variables are not<br>
updated in the correct order after each full rolling restart, etc.<br>
<br>
Which brings us back to triggers. AFAIK, only sqlalchemy+mariadb is<br>
being used in production, so we really only have one backend we would<br>
have to write triggers for. If the data duplication is too unpalatable<br>
for a given migration (using the +1, +2, +3 pattern above), we may<br>
have to wade into the less simple world of triggers.<br></blockquote><div>I think we can only enable the trigger during the upgrade process and then disable it.<br> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
For RPC changes, we don't have a great solution right now (looking<br>
specifically at heat/engine/service.py). If we add a field, an older<br>
running heat-engine will break if it receives a request from a newer<br>
running heat-engine. For a relevant example, consider adding the<br>
"root_id" as an argument (<br>
<a href="https://review.openstack.org/#/c/354621/13/heat/engine/service.py" rel="noreferrer" target="_blank">https://review.openstack.org/#<wbr>/c/354621/13/heat/engine/<wbr>service.py</a> ).<br>
<br>
Looking for the simplest solution -- if we introduce a mandatory<br>
"future_args" arg (a dict) now to all rpc methods (perhaps provide a<br>
decorator to do so), then we could follow this pattern post-Ocata:<br>
<br>
legacy release: accepts the future_args param (but does nothing with it).<br>
release+1: accept the new parameter with a default of None,<br>
pass the value of the new parameter in future_args.<br>
release+2: accept the new parameter, pass the value of the new parameter<br>
in its proper placeholder, no longer in future_args.<br></blockquote><div>This is something similar to the one is being used by neutron for the agents,<br></div><div>i.e consistently capturing those new/unknown
arguments with keyword arguments<br>and ignoring them on agent side; and by not
enforcing newer RPC entry point<br>versions on server side. However, this makes the rpc api less strict and not ideal.<br><br></div><div>The best way would be do some kind of rpc pinning on the new engines when<br></div><div>they send messages(new engines can receive old messages). I was also wondering<br>if it's possible/good idea to restrict engines not to communicate with other engines<br>only during the upgrade process.<br><br> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
But, we don't have a way of deleting args. That's not super<br>
awful... old args never die, they just eventually get ignored. As for<br>
adding new api's, the pattern would be to add them in release+1, but<br>
not call them until release+2. [If we really have a case where we need<br>
to add and use a new api in release+1, the solution may be to have two<br>
rpc api messaging targets in release+1, one for the previous<br>
major.minor release and another for the major+1.0 release that has the<br>
new api. Then, we of course we could remove outdated args in<br>
major+1.0.]<br></blockquote><div>I'm not sure we ever delete args, as we make the rpc servers backward compatible.<br> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Finally, a note about Oslo versioned objects: they don't really help<br>
us. They work great for nova where there is just nova-conductor<br>
reading and writing to the DB, but we have multiple heat-engines doing<br>
that that need to be restarted in a rolling manner. See the references<br>
below for greater detail.<br>
<br>
--Crag<br>
<br>
References<br>
----------<br>
<br>
[openstack-dev] [Heat] Versioned objects upgrade patterns<br>
<a href="http://lists.openstack.org/pipermail/openstack-dev/2016-May/thread.html#95245" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>pipermail/openstack-dev/2016-<wbr>May/thread.html#95245</a><br>
<br>
[openstack-dev] [keystone][nova][neutron][all] Rolling upgrades:<br>
database triggers and oslo.versionedobjects<br>
<a href="http://lists.openstack.org/pipermail/openstack-dev/2016-September/102698.html" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>pipermail/openstack-dev/2016-<wbr>September/102698.html</a><br>
<a href="http://lists.openstack.org/pipermail/openstack-dev/2016-October/105764.html" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>pipermail/openstack-dev/2016-<wbr>October/105764.html</a><br>
<br>
<br>
______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br>
</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature"><div dir="ltr"><div>Regards,</div>Rabi Misra<div><br></div></div></div>
</div></div>