<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Arial;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:DengXian;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman";}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:Calibri;
color:windowtext;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body bgcolor="white" lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-family:Calibri">Thanks Numan & Amitabha, this may be the right direction to solve the bug [1].
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Calibri"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:Calibri">It basically implements Neutron API as async call, and queuing the request within DB transaction, and the ordering is preserved by the journal thread "lock" that is implemented with state PROCESSING plus
DB transaction "with_for_update", with the help of validation functions for dependency checking (e.g. same object cannot be updated by 2 journal threads at the same time, etc.).<o:p></o:p></span></p>
<p><span style="font-family:Calibri">However, I didn't figure out how errors are handled with this approach. For example, a port is created in Neutron but ODL controller failed to create it although the journal thread successfully sent the request to ODL. And
I didn't see how the port states (UP & DOWN) are handled (I didn’t see any call to ProvisioningBlock, so does it mean it will just be UP from the beginning?) It would be great if anyone can help answer this question.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Calibri"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:Calibri">[1] <a href="https://bugs.launchpad.net/networking-ovn/+bug/1605089">
https://bugs.launchpad.net/networking-ovn/+bug/1605089</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Calibri"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:Calibri">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Calibri">Han Zhou<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-family:Calibri;color:black">From: </span>
</b><span style="font-family:Calibri;color:black">Numan Siddique <nusiddiq@redhat.com><br>
<b>Reply-To: </b>"OpenStack Development Mailing List (not for usage questions)" <openstack-dev@lists.openstack.org><br>
<b>Date: </b>Friday, July 22, 2016 at 4:51 AM<br>
<b>To: </b>"OpenStack Development Mailing List (not for usage questions)" <openstack-dev@lists.openstack.org><br>
<b>Subject: </b>Re: [openstack-dev] [Neutron][networking-ovn][networking-odl] Syncing neutron DB and OVN DB<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:Arial">Thanks for the comments Amitabha.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Arial">Please see comments inline</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Fri, Jul 22, 2016 at 5:50 AM, Amitabha Biswas <<a href="mailto:azbiswas@gmail.com" target="_blank">azbiswas@gmail.com</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<p class="MsoNormal">Hi Numan, <o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Thanks for the proposal. We have also been thinking about this use-case.<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">If I’m reading this accurately (and I may not be), it seems that the proposal is to not have any OVN NB (CUD) operations (R operations outside the scope) done by the api_worker threads but rather by a new journal thread.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:Arial">Correct.</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:Courier"><o:p></o:p></span></p>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<p class="MsoNormal">If this is indeed the case, I’d like to consider the scenario when there any N neutron nodes, each node with M worker threads. The journal thread at the each node contain list of pending operations. Could there be (sequence) dependency
in the pending operations amongst each the journal threads in the nodes that prevents them from getting applied (for e.g. Logical_Router_Port and Logical_Switch_Port inter-dependency), because we are returning success on neutron operations that have still
not been committed to the NB DB.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Arial">I <o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-family:Arial">ts a valid scenario and should be designed properly to handle such scenarios in case we take this approach. <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:Courier"><o:p> </o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:Courier"><o:p></o:p></span></p>
</div>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<p class="MsoNormal">Couple of clarifications and thoughts below.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Thanks<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Amitabha <<a href="mailto:abiswas@us.ibm.com" target="_blank">abiswas@us.ibm.com</a>><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On Jul 13, 2016, at 1:20 AM, Numan Siddique <<a href="mailto:nusiddiq@redhat.com" target="_blank">nusiddiq@redhat.com</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:Arial">Adding the proper tags in subject</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Wed, Jul 13, 2016 at 1:22 PM, Numan Siddique <<a href="mailto:nusiddiq@redhat.com" target="_blank">nusiddiq@redhat.com</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:Arial">Hi Neutrinos,</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Arial">Presently, In the OVN ML2 driver we have 2 ways to sync neutron DB and OVN DB</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Arial"> - At neutron-server startup, OVN ML2 driver syncs the neutron DB and OVN DB if sync mode is set to repair.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Arial"> - Admin can run the "neutron-ovn-db-sync-util" to sync the DBs.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Arial">Recently, in the v2 of networking-odl ML2 driver (Please see (1) below which has more details). (ODL folks please correct me if I am wrong here)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Arial"> - a journal thread is created which does the CRUD operations of neutron resources asynchronously (i.e it sends the REST APIs to the ODL controller).</span><o:p></o:p></p>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
<p class="MsoNormal"><br>
Would this be the equivalent of making OVSDB transactions to the OVN NB DB?<o:p></o:p></p>
</div>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:Arial">Correct.</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:Courier"><o:p></o:p></span></p>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<div>
<div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:Arial"> - a maintenance thread is created which does some cleanup periodically and at startup does full sync if it detects ODL controller cold reboot.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Arial">Few question I have</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Arial"> - can OVN ML2 driver take same or similar approach. Are there any advantages in taking this approach ? One advantage is neutron resources can be created/updated/deleted even if the OVN ML2 driver has lost
connection to the ovsdb-server. The journal thread would eventually sync these resources in the OVN DB. I would like to know the communities thoughts on this.</span><o:p></o:p></p>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">If we can make it work, it would indeed be a huge plus for system wide upgrades and some corner cases in the code (ACL specifically), where the post_commit relies on all transactions to be successful and doesn’t revert the neutron db if
something fails.<o:p></o:p></p>
</div>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<div>
<div>
<div>
<div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Arial"> - Are there are other ML2 drivers which might have to handle the DB sync's (cases where the other controllers also maintain their own DBs) and how they are handling it ?</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Arial"> - Can a common approach be taken to sync the neutron DB and controller DBs ?</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Arial">-----------------------------------------------------------------------------------------------------------</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Arial">(1)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Arial">Sync threads created by networking-odl ML2 driver</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Arial">--------------------------------------------------</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Arial">ODL ML2 driver creates 2 threads (threading.Thread module) at init</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Arial"> - Journal thread</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Arial"> - Maintenance thread</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Arial">Journal thread</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Arial">----------------</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Arial">The journal module creates a new journal table by name “opendaylightjournal” -
<a href="https://github.com/openstack/networking-odl/blob/master/networking_odl/db/models.py#L23" target="_blank">
https://github.com/openstack/networking-odl/blob/master/networking_odl/db/models.py#L23</a></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Arial">Journal thread will be in loop waiting for the sync event from the ODL ML2 driver.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Arial"> - ODL ML2 driver resource (network, subnet, port) precommit functions when called by the ML2 plugin adds an entry in the “opendaylightjournal” table with the resource data and sets the journal operation
state for this entry to “PENDING”.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Arial"> - The corresponding resource postcommit function of the ODL ML2 plugin when called, sets the sync event flag.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Arial"> - A timer is also created which sets the sync event flag when it expires (the default value is 10 seconds).</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Arial"> - Journal thread wakes up, looks into the “opendaylightjournal” table with the entries with state “pending” and runs the CRUD operation on those resources in the ODL DB. Once done, it sets the state to “completed”.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Arial">Maintenance thread</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Arial">------------------</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Arial">Maintenance thread does 3 operations</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Arial"> - JournalCleanup - Delete completed rows from journal table “opendaylightjournal”.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Arial"> - CleanupProcessing - Mark orphaned processing rows to pending.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Arial"> - Full sync - Re-sync when detecting an ODL "cold reboot”.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Arial">Thanks</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Arial;color:#888888">Numan</span><span style="color:#888888"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Courier;color:#888888"><o:p> </o:p></span></p>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</div>
<p class="MsoNormal">__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="mailto:OpenStack-dev-request@lists.openstack.org" target="_blank">
OpenStack-dev-request@lists.openstack.org</a>?subject:unsubscribe<br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><o:p></o:p></p>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</div>
</div>
</body>
</html>