<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">On Mon, Apr 15, 2019 at 7:48 PM Bence Romsics <<a href="mailto:bence.romsics@gmail.com">bence.romsics@gmail.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Heat and Neutron folks,<br>
<br>
I'm wondering how could I improve support for extra routes for one of<br>
our users. In this email I'd like to get your input (from both Heat<br>
and Neutron folks) in which direction should I start before I sit down<br>
to write the relevant blueprints and RFEs. This will be a bit long,<br>
please bear with me.<br>
<br>
As you know a neutron router can have extra static routing table<br>
entries as the 'routes' attribute like this (please see the api-ref in<br>
[1]):<br>
<br>
$ openstack router show router1 -f value -c routes<br>
destination='<a href="http://10.1.0.0/24" rel="noreferrer" target="_blank">10.1.0.0/24</a>', gateway='10.0.0.11'<br>
destination='<a href="http://10.2.0.0/24" rel="noreferrer" target="_blank">10.2.0.0/24</a>', gateway='10.0.0.12'<br>
...<br>
<br>
To my best knowledge the whole set of extra routes must be updated at<br>
once which easily leads to lost updates if multiple clients modify<br>
this attribute concurrently (imagine client A and B working<br>
concurrently, for example both reads routes [r1], client A writes [r1,<br>
r2], client B writes [r1, r3]; depending on timing either r2 or r3<br>
will be lost).<br>
<br>
Heat has an unsupported resource OS::Neutron::ExtraRoute [2] to front<br>
Neutron extra routes. For example:<br>
<br>
heat_template_version: '2015-04-30'<br>
resources:<br>
Extra_route_1:<br>
type: OS::Neutron::ExtraRoute<br>
properties:<br>
destination: '<a href="http://10.1.0.0/24" rel="noreferrer" target="_blank">10.1.0.0/24</a>'<br>
nexthop: '10.0.0.11'<br>
router_id: 'd544b5c5-d4d2-4b52-aaa8-ef4b9fa3e154'<br>
<br>
The Heat resource represents a single extra route out of the many in<br>
the router's routes attribute. If you have multiple<br>
OS::Neutron::ExtraRoute resources Heat by default handles them<br>
concurrently leading to the lost update problem. You can easily<br>
reproduce the problem just by having 10 extra routes in a template.<br>
The stack creation will succeed, but the router in fact will have only<br>
6-7 (the exact number changes based on timing) extra routes. AFAIU<br>
this is exactly why this was marked unsupported. (Please let me know<br>
if you know other reasons.)<br>
<br></blockquote><div><br></div><div>I think this had been discussed on several occasions[1] before and one of the<br></div><div>suggestions from the neutron team was to use <span class="gmail-yui3-editable_text-text ellipsis" style="max-width:95%" id="gmail-yui_3_10_3_1_1555339055534_51">compare-and-swap update api </span>[2]</div><div>to avoid race conditions.</div><div><br></div><div>[1] <a href="http://lists.openstack.org/pipermail/openstack-dev/2018-June/131020.html">http://lists.openstack.org/pipermail/openstack-dev/2018-June/131020.html</a><br></div><div> [2] <a href="https://bugs.launchpad.net/neutron/+bug/1703234">https://bugs.launchpad.net/neutron/+bug/1703234</a><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
As a workaround I suggested our user a hack to serialize the<br>
OS::Neutron::ExtraRoute resources by 'depends_on'.<br>
<br>
But which way should I go if I want to fix this properly?<br>
<br>
way #1 - changes in Heat only: Introduce a new Heat resource<br>
OS::Neutron::ExtraRoutes (please notice the plural) which can<br>
represent the whole routes attribute of a router (i.e. multiple extra<br>
routes in a single resource). This will not solve the design<br>
shortcomings of the neutron extraroute API, but it can at least<br>
express what can be done in the Neutron API as is. Lost updates of<br>
multiple stacks editing the same router's routes are still possible.<br>
<br></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
way #2 - changes in Heat and Neutron: Introduce a new Neutron<br>
extension with a better API (either extraroutes as a top level<br>
resource, each object with a router_id or an action map a'la<br>
add/remove_router_interface to edit extra routes on the server side).<br>
This new API would work on the same DB tables and backends as the old.<br>
The old API would be kept for backwards compatibility. But if a new<br>
user exclusively uses the new API it can work concurrently. Then<br>
introduce a new Heat resource (or change the existing unsupported one)<br>
to use this new Neutron API. This is obviously more work than the<br>
first option.<br>
<br>
Which one do you prefer and why? Or let me know please if I missed a<br>
better alternative.<br>
<br>
In this thread I hope to get a cross-project agreement of this between<br>
Heat and Neutron so I can go ahead and open the relevant Heat and<br>
Neutron blueprints where we can focus only on each projects' internal<br>
questions.<br>
<br>
Thanks in advance,<br>
Bence Romsics<br>
irc: rubasov<br>
<br>
[1] <a href="https://developer.openstack.org/api-ref/network/v2/?expanded=update-router-detail#update-router" rel="noreferrer" target="_blank">https://developer.openstack.org/api-ref/network/v2/?expanded=update-router-detail#update-router</a><br>
[2] <a href="https://docs.openstack.org/heat/latest/template_guide/unsupported.html#OS::Neutron::ExtraRoute" rel="noreferrer" target="_blank">https://docs.openstack.org/heat/latest/template_guide/unsupported.html#OS::Neutron::ExtraRoute</a><br>
<br>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>Regards,</div>Rabi Mishra<div><br></div></div></div></div></div></div></div></div>