On Jun 7, 2019, at 7:17 AM, Chris Dent cdent+os@anticdent.org wrote:
Ed Leafe's ongoing work with using a graph database probably needs some kind of report or update.
Sure, be happy to.
A few weeks ago I completed the code changes to remove sqlalchemy from the objects, and replace it with code to talk to the graph DB (Neo4j). One big issue is that there isn’t a 1:1 relationship between what is needed in the two database approaches. For example, there is no need to use the two-ID design that (IMO) overly complicates placement data. There is also no need to store the IDs of parents and root providers, but so much of the code depends on these values, I left them in there for now. One other twist is that an Allocation cannot exist without a Consumer, so all the code to handle the early microversions that support that was removed.
I then moved on to getting the functional tests passing. Some early runs revealed holes in my understanding of what the code was supposed to be doing, so I fixed those. Most of the failures were in the tests/functional/db directory. I mentioned that to Chris in a side conversation, and he agreed that those tests would not be relevant, as the system had a completely different database, so I removed those.
I tried to integrate Neo4j’s transaction model into the transaction framework of oslo_db and sqla, and while it works for the most part, it fails when running tox. I get “Invalid transaction” messages, which you would expect when one process closes another process’s transaction. Since the Python adapter I’m using (py2neo) creates a pool for connections, I suspect that the way tox runs is causing py2neo to reuse live connections. I haven’t had time to dig into this yet, but it is my current focus. When I run the tests individually, they pass without a problem.
I am also planning on doing some performance measurement, so I guess I’ll look into the perfload stuff to see if it can work for this.
One thing that is clear to me from all this work is that the way Placement is coded is very much a result of its relational DB roots. There were so many places I encountered when converting the code where I needed to do what felt like unnecessary steps to make the objects continue to work the way that are currently designed. Had this been a greenfield effort, the code for implementing Placement with a graph DB would have been much more direct and understandable. But the converted objects are working, and working well.
-- Ed Leafe