<div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jul 27, 2016 at 4:15 PM, Zhou, Han <span dir="ltr"><<a href="mailto:hzhou8@ebay.com" target="_blank">hzhou8@ebay.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span class="">On Wed, Jul 27, 2016 at 7:15 AM, Russell Bryant <span dir="ltr"><<a href="mailto:rbryant@redhat.com" target="_blank">rbryant@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_extra"><br><div class="gmail_quote"><span>On Wed, Jul 27, 2016 at 5:58 AM, Kevin Benton <span dir="ltr"><<a href="mailto:kevin@benton.pub" target="_blank">kevin@benton.pub</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><span>><span style="font-family:arial,helvetica,sans-serif;font-size:12.8px"> I'd like to see if we can solve the problems more generally.</span><div><span style="font-family:arial,helvetica,sans-serif;font-size:12.8px"><br></span></div></span><div><span style="font-family:arial,helvetica,sans-serif;font-size:12.8px">We've tried before but we very quickly run into competing requirements with regards to eventual consistency. For example, asynchronous background sync doesn't work if someone wants their backend to confirm that port details are acceptable (e.g. mac isn't in use by some other system outside of openstack). Then each backend has different methods for detecting what is out of sync (e.g. config numbers, hashes, or just full syncs on startup) that each come with their own requirements for how much data needs to be resent when an inconsistency is detected.</span></div><div><span style="font-family:arial,helvetica,sans-serif;font-size:12.8px"><br></span></div><div><span style="font-family:arial,helvetica,sans-serif;font-size:12.8px">If we can come to some common ground of what is required by all of them, then I would love to get some of this built into the ML2 framework. However, we've discussed this at meetups/mid-cycles/summits and it inevitably ends up with two people drawing furiously on a whiteboard, someone crying in the corner, and everyone else arguing about the lack of parametric polymorphism in Go.</span></div></div></blockquote><div><br></div></span><div><div style="font-family:arial,helvetica,sans-serif;display:inline">Ha, yes, makes sense that this is really hard to solve in a way that works for everyone ...</div></div><span><div><div style="font-family:arial,helvetica,sans-serif;display:inline"></div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Even between OVN and ODL in this thread, it sounds like the only thing in common is a background worker that consumes from a queue of tasks in the db. Maybe realistically the only common thing we can come up with is a taskflow queue stored in the DB to solve the multiple workers issue...</div></div></blockquote><div><br></div></span><div><div style="font-family:arial,helvetica,sans-serif">To clarify, ODL has this background worker and the discussion was whether OVN should try to follow a similar approach.</div><div style="font-family:arial,helvetica,sans-serif"><br></div><div style="font-family:arial,helvetica,sans-serif">So far, my gut feeling is that it's far too complicated for the problems it would solve. There's one identified multiple-worker related race condition on updates, but I think we can solve that another way.</div><br></div></div></div></div></blockquote><div><br></div></span><div>Russell, in fact I think this background worker is the good way to solve both problems:<br><br></div><div>Problem 1. When something failed when updating OVN DB in post-commit: With the help of background worker, it can do the retries and the job state can be tracked, and with the information proper actions can be taken against failure jobs, e.g. cleanups. It is basically a declarative way of implementation, which IMHO is a particularly good way in ML2 context, because we cannot just rollback Neutron DB changes at failure because it is shared by all mech-drivers. (Even in a monolithic plugin, handling the partial failures and doing rollback is a big headache).<br><br></div><div>Problem 2. Race condition because of lack of critical section between Neutron DB transaction and post-commit: With the help of journal, the ordering is ensured to be the same as DB transaction commits. Protection against the journal processing between multiple background workers can be properly enforced with the help of DB transaction.<br><br></div><div>I think ODL and OVN are not the only ones facing these problems. They are pretty general to most drivers if not all. It would be great to have a common task flow mechanism in ML2, but I'd like to try it in OVN first (if no better solution to the problems above).</div></div></div></div></blockquote><div><br></div><div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">I had another idea for problem 2, at least. I posted a more detailed description of the idea on the bug you posted [1].</div></div><div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline"><br></div></div><div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">This is unrelated to problem 1, though. I guess I was hoping we could just come up with a better way of doing rollbacks when necessary.</div></div><div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline"><br></div></div><div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">I also had a long term dream of not using the Neutron DB at all, and only relying on the OVN database. That seems much less practical now that we've moved back to ML2. Maybe it was a crazy idea, anyway. :-)</div></div><div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline"><br></div></div><div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">[1] <a href="https://bugs.launchpad.net/networking-ovn/+bug/1605089">https://bugs.launchpad.net/networking-ovn/+bug/1605089</a></div> </div><div><br></div><div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">-- </div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Russell Bryant</div><br></div></div>
</div></div>