Open Stack

On Tue, Dec 4, 2018 at 11:42 AM Ben Nemec <openstack at nemebean.com> wrote:
>
> Copying Mike Bayer since he's our resident DB expert. One more comment
> inline.

so the level of abstraction oslo.db itself provides is fairly light -
it steps in for the initial configuration of the database engine, for
the job of reworking exceptions into something more locallized, and
then for supplying a basic transactional begin/commit pattern that
includes concepts that openstack uses a lot.   it also has some
helpers for things like special datatypes, test frameworks, and stuff
like that.

That is, oslo.db is not a full blown "abstraction" layer, it exposes
the SQLAlchemy API which is then where you have the major level of
abstraction.

Given that, making oslo.db do for etcd3 what it does for SQLAlchemy
would be an appropriate place for such a thing.  It would be all new
code and not really have much overlap with anything that's there right
now, but still would be feasible at least at the level of, "get a
handle to etcd3, here's the basic persistence / query pattern we use
with it, here's a test framework that will allow test suites to use
it".

At the level of actually reading and writing data to etcd3 as well as
querying, that's a bigger task, and certainly that is not a SQLAlchemy
thing either.    If etcd3's interface is a simple enough "get" / "put"
/ "query" and then some occasional special operations, those kinds of
abstraction APIs are often not too terrible to write.

Also note that we have a key/value database interface right now in
oslo.cache which uses dogpile.cache against both memcached and redis
right now.    If you really only needed put/get with etcd3, it could
do that also, but I would assume we have the need for more of a fine
grained interface than that.   Haven't studied etcd3 as of yet.   But
I'd be interested in supporting it in oslo somewhere.




>
> On 12/4/18 4:08 AM, Gorka Eguileor wrote:
> > On 03/12, Julia Kreger wrote:
> >> Indeed it is a considered a base service, but I'm unaware of why it was
> >> decided to not have any abstraction layer on top. That sort of defeats the
> >> adoption of tooz as a standard in the community. Plus with the rest of our
> >> code bases, we have a number of similar or identical patterns and it would
> >> be ideal to have a single library providing the overall interface for the
> >> purposes of consistency. Could you provide some more background on that
> >> decision?
> >>
> >> I guess what I'd really like to see is an oslo.db interface into etcd3.
> >>
> >> -Julia
> >
> > Hi,
> >
> > I think that some projects won't bother with the etcd interface since it
> > would require some major rework of the whole service to get it working.
>
> I don't think Julia was suggesting that every project move to etcd, just
> that we make it available for projects that want to use it this way.
>
> >
> > Take Cinder for example.  We do complex conditional updates that, as far
> > as I know, cannot be satisfied with etcd's Compare-and-Swap
> > functionality.  We could modify all our code to make it support both
> > relational databases and key-value stores, but I'm not convinced it
> > would be worthwhile considering the huge effort it would require.
> >
> > I believe there are other OpenStack projects that have procedural code
> > stored on the database, which would probably be hard to make compatible
> > with key-value stores.
> >
> > Cheers,
> > Gorka.
> >
> >>
> >> On Mon, Dec 3, 2018 at 4:55 PM Fox, Kevin M <Kevin.Fox at pnnl.gov> wrote:
> >>
> >>> It is a full base service already:
> >>> https://governance.openstack.org/tc/reference/base-services.html
> >>>
> >>> Projects have been free to use it for quite some time. I'm not sure if any
> >>> actually are yet though.
> >>>
> >>> It was decided not to put an abstraction layer on top as its pretty simple
> >>> and commonly deployed.
> >>>
> >>> Thanks,
> >>> Kevin
> >>> ------------------------------
> >>> *From:* Julia Kreger [juliaashleykreger at gmail.com]
> >>> *Sent:* Monday, December 03, 2018 3:53 PM
> >>> *To:* Ben Nemec
> >>> *Cc:* Davanum Srinivas; geguileo at redhat.com;
> >>> openstack-discuss at lists.openstack.org
> >>> *Subject:* Re: [all] Etcd as DLM
> >>>
> >>> I would like to slightly interrupt this train of thought for an
> >>> unscheduled vision of the future!
> >>>
> >>> What if we could allow a component to store data in etcd3's key value
> >>> store like how we presently use oslo_db/sqlalchemy?
> >>>
> >>> While I personally hope to have etcd3 as a DLM for ironic one day, review
> >>> bandwidth permitting, it occurs to me that etcd3 could be leveraged for
> >>> more than just DLM. If we have a common vision to enable data storage, I
> >>> suspect it might help provide overall guidance as to how we want to
> >>> interact with the service moving forward.
> >>>
> >>> -Julia
> >>>
> >>> On Mon, Dec 3, 2018 at 2:52 PM Ben Nemec <openstack at nemebean.com> wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> I wanted to revisit this topic because it has come up in some downstream
> >>>> discussions around Cinder A/A HA and the last time we talked about it
> >>>> upstream was a year and a half ago[1]. There have certainly been changes
> >>>> since then so I think it's worth another look. For context, the
> >>>> conclusion of that session was:
> >>>>
> >>>> "Let's use etcd 3.x in the devstack CI, projects that are eventlet based
> >>>> an use the etcd v3 http experimental API and those that don't can use
> >>>> the etcd v3 gRPC API. Dims will submit a patch to tooz for the new
> >>>> driver with v3 http experimental API. Projects should feel free to use
> >>>> the DLM based on tooz+etcd3 from now on. Others projects can figure out
> >>>> other use cases for etcd3."
> >>>>
> >>>> The main question that has come up is whether this is still the best
> >>>> practice or if we should revisit the preferred drivers for etcd. Gorka
> >>>> has gotten the grpc-based driver working in a Cinder driver that needs
> >>>> etcd[2], so there's a question as to whether we still need the HTTP
> >>>> etcd-gateway or if everything should use grpc. I will admit I'm nervous
> >>>> about trying to juggle eventlet and grpc, but if it works then my only
> >>>> argument is general misgivings about doing anything clever that involves
> >>>> eventlet. :-)
> >>>>
> >>>> It looks like the HTTP API for etcd has moved out of experimental
> >>>> status[3] at this point, so that's no longer an issue. There was some
> >>>> vague concern from a downstream packaging perspective that the grpc
> >>>> library might use a funky build system, whereas the etcd3-gateway
> >>>> library only depends on existing OpenStack requirements.
> >>>>
> >>>> On the other hand, I don't know how much of a hassle it is to deploy and
> >>>> manage a grpc-gateway. I'm kind of hoping someone has already been down
> >>>> this road and can advise about what they found.
> >>>>
> >>>> Thanks.
> >>>>
> >>>> -Ben
> >>>>
> >>>> 1: https://etherpad.openstack.org/p/BOS-etcd-base-service
> >>>> 2:
> >>>>
> >>>> https://github.com/embercsi/ember-csi/blob/5bd4dffe9107bc906d14a45cd819d9a659c19047/ember_csi/ember_csi.py#L1106-L1111
> >>>> 3: https://github.com/grpc-ecosystem/grpc-gateway
> >>>>
> >>>>