Open Stack

Wed Feb 26 16:40:38 UTC 2014

I'm not sure I understand the first question, but it seems like you are asking how the default capabilities are determined. First, they should all be documented. Secondly, we can have defaults setup by migrations. I agree that this is not as seamless as a default config file, and that segways into question two.

We want to eventually serve this data to the customers. As far as I know, there is no precedent in Openstack to serve configuration file data to customers. All customer presented data should be contained in the database, not in flat files. Additionally, management via the api becomes much more complex when I have to write to files instead of adding removing database rows.

As far as enabled/disabled capabilities, having enabled/disabled for each feature is more prone to errors or inconsistencies. Having a list of allowed features solves the same problem without needing to maintain a full set of features and associated flags for each datastore/version. We agreed this approach much cleaner and simpler than the prior.

Implementation will be coming shortly - I'm still working on that. I'll let you know as soon as I have something ready for review.

Also, supposedly, the discussions we had last week are recorded and available via some means, but I don't know where. A lot of this was discussed there if you weren't able to make it.

________________________________
From: Denis Makogon [dmakogon at mirantis.com]
Sent: Wednesday, February 26, 2014 5:18 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [TROVE] Trove capabilities

First of all, how does Trove knows what kind of capabilities it can give to users ?

We should have some kind of identity, that stores the restricted description of all capabilities. Suggested variant is not acceptable, because you could register any kind of capabilities in database.

    Main point: Trove should have restrict description of all possible capabilities, and it’s should not be stored in backend table.

    Design: What is a capability? - Almost totally disagree.

Only datastore vesrion association is significant, common capabilities table is not needed at all, because all new features(capabilities) assums that Trove will be re-deployed from scratch. It means that capabilities description would be updated only 2 or 3 time per release.

    Design: Why isn't this in config files? - Agreed, my solution design doesn’t required restart. Magic happes in runtime.

Development: How are capabilities checked? - Option 2 is more preferable, because capabilities should be stored as dictionary object that has next structure:

API call: ALLOWED/BLOCKED

Unfortunately given design is too broad, i dont see any implementation steps description. Please provide more details and flow examples with pseudo-code.

Best regards,
Denis Makogon

On Wed, Feb 26, 2014 at 12:13 AM, Kaleb Pomeroy <kaleb.pomeroy at rackspace.com<mailto:kaleb.pomeroy at rackspace.com>> wrote:

Design: What is a capability?
A capabilities is an attribute of a datastore/version that allows dynamic code flow for different datastore/versions, The presence of a capability determines which code path is taken. A capability is an ID, a friendly name (used in our code), and a lengthy description (for displaying to the user).

Design: Where do they live?
Capabilities will live in their own table. In addition, there will be an association table to map between datastore versions and capabilities.

Design: Why isn't this in config files?
First, they will be able to be viewed by customers, and modified by operations. That does not lend itself to config files. Additionally, we should be able to change a datastores capabilities without a restart.

Operations: What happens if the capabilities table is not populated?
We are setting everything to be default off for all features. For mysql to support ephemeral_volumes it would need to be added to the capabilities table and properly associated to each version of mysql.

Operations: How do you manage them?
For the first pass, trove-manage commands. There will be a add_capability, remove_capability, associate_capability_with_datastore, and unassociate_capability_from_datastore (name are completely off the top of my head and subject to discussion/change).

Development: When/where are capabilities checked/determined?
Any time you would want to fork paths. The check for ephemeral_volume would happen right before the cinder volume is created, whereas the database capability would be checked as soon as possible in the api layer.

Development: How are capabilities checked?
This is still to be determined.

Option one (the one discussed at the meetup):
    if(cfg.cababilites.mysql.ephemeral_volumes):
        do_code()

Option two:
    if "ephemeral_volumes" in datastore.manager.capabilities)
        do_code()

The advantages of option one is consistency in checking the cfg object for configuration like things. I think that is worse than checking a property of the datastore. The advantage of the second way is that is more cleanly expresses what is happening under the hood without any additional cost. I have the second option about complete, so we can see what that looks like soon.

Most of this was discussed and agreed upon last week, I just wanted to recap and confirm with everyone on what our objectives and goals are. If there are unanswered questions, lets discuss sooner rather than later.

- KPom

Bonus: API discussion for future development (not right now)
I suggest we expose only the capabilities that belong to a specific datastore. As a customer, I shouldn't know every feature available, just the ones on the datastore they are looking at.

GET /{tenant_id}/datastores/{datastore_version_id}/capabilities

Eventually, we would want the following admin routes available, but that is mitigated by the trove-manage stuff.

[GET | PUT | POST | DELETE] /capabilities
[POST | DELETE] /datastores/datastore_version_id/capabilities

________________________________
From: Denis Makogon [dmakogon at mirantis.com<mailto:dmakogon at mirantis.com>]
Sent: Wednesday, February 19, 2014 1:00 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [TROVE] Trove capabilities

I know that Kaleb is working on that.

But we(me, Kaleb, rest of guys) decided that i'm gonna present my own vision of Capabilities.
That is what i'm doing now. Basicaly, at meetup you all could discuss design in general, i suppose.

Best regards,
Denis Makogon.

On Wed, Feb 19, 2014 at 7:42 PM, Daniel Salinas <imsplitbit at gmail.com<mailto:imsplitbit at gmail.com>> wrote:
We're at the trove meetup for the rest of this week but you should speak with Kaleb Pomeroy from Rackspace.  He's generated some code on this particular subject.

On Wed, Feb 19, 2014 at 11:11 AM, Denis Makogon <dmakogon at mirantis.com<mailto:dmakogon at mirantis.com>> wrote:

Goodday OpenStack DBaaS communtiy.

I’d like to start topic related to design of capabilities [1].

This question was raised when multiple datastores where integrated into Trove (such as redis, cassandra, mongo in instance modes). We encountered a the problem when specific datastore doesn’t support operations that are the part of Trove API (core and extension). After some discussions with the community we decided to implement “capabilities” feature. Basicaly, “capabilities” is the list of operations acceptable for chosen “object”. To be accurate,  “capabilities” is the API for listing available Trove API(core and extensions) per some kind of an identifier.

Design.

I’d like to split up all tasks into small peaces:

  1.  Nameing convintion.<https://docs.google.com/a/mirantis.com/document/d/1FDhenfF47UYdiqTr5dGHSG_Gk828yFuvew92I9qil5I/edit#heading=h.dqx5s9nxd86g>

  2.  DSL for writing reference to available capabilities.<https://docs.google.com/a/mirantis.com/document/d/1FDhenfF47UYdiqTr5dGHSG_Gk828yFuvew92I9qil5I/edit#heading=h.a86gyngvxgek>

  3.  Сustody strategy. How and what to store at the back-end?<https://docs.google.com/a/mirantis.com/document/d/1FDhenfF47UYdiqTr5dGHSG_Gk828yFuvew92I9qil5I/edit#heading=h.p3kja3s8zex>

  4.  Pinning to datastore or version?<https://docs.google.com/a/mirantis.com/document/d/1FDhenfF47UYdiqTr5dGHSG_Gk828yFuvew92I9qil5I/edit#heading=h.mvusz3lj5res>

  5.  How to load  actual API reference?<https://docs.google.com/a/mirantis.com/document/d/1FDhenfF47UYdiqTr5dGHSG_Gk828yFuvew92I9qil5I/edit#heading=h.df6yrbqny8te>

  6.  Default capabilities. What operations are allowed by default?<https://docs.google.com/a/mirantis.com/document/d/1FDhenfF47UYdiqTr5dGHSG_Gk828yFuvew92I9qil5I/edit#heading=h.gjahbyoqoq5d>

  7.  Capablities API.<https://docs.google.com/a/mirantis.com/document/d/1FDhenfF47UYdiqTr5dGHSG_Gk828yFuvew92I9qil5I/edit#heading=h.ihzh5pv7pmci>

  8.  Capabilities Management API.<https://docs.google.com/a/mirantis.com/document/d/1FDhenfF47UYdiqTr5dGHSG_Gk828yFuvew92I9qil5I/edit#heading=h.paouley6yndv>

Naming convention

Suggestions:

  1.  Filename convension:

{datastore or datastore_version_manager}.capabilities

  1.  Attribute convension:

Trove-”API-section”:

- method: Identifier

    Available sections: instance, backip, users, schemes.

    Capability Identifier: ALLOWED/BLOCKED

DSL for writing reference to available capabilities

There are several options of DSL’s that can be chosen for writing capabilities references, options:

  1.  JSON.

  2.  YAML.

  3.  XML.

    Personaly i’d like to choose YAML as appropriate DSL. Described capabilities would look like: mysql.capabilities

Trove-Instance:

- create: ALLOWED

- list: BLOCKED

- delete: ALLOWED

Trove-Backup:

- create: BLOCKED

Trove-Users:

- create: BLOCKED

Сustody strategy. How and what to store at the back-end ?

Before suggesting database table scheme i’d like to describe what we are going to store. It would be better to store only blocked capabilities, its cheaper, instead of storing all capabilities, either blocked and allowed.

Backend table scheme.

Table 1 - Capabilities table scheme

Datastore or Datastore version manager

Capabilities

datastore or manager

“Trove-”API-section”:

                     - method: Identifier“

    Where Identifier could be ALLOWED or BLOCKED

Pinning to datastore or datastore version?

As you can see from previous topics, it would be better to pin capabilities directly to datastore version, since one manager could be assigned to multiple datastore versions.

    So, capabilities backend table scheme would have next desription:

  *   datastrove_version_manager:

name: manager, Foreing key from DatastoreVersion table;

type: String.

  *   capabilities:

name: capabilities;

type: Text.

How to load actual API reference?

    According to previous topics i forsee the next way of discovering actual capabilities at runtime:

    Since YAML format perfectly could be loaded as Python dictionary, the easiest way is to merge two dictionaries.

    Example:

    default_capabilities = (CapabilitiesModel.

            load_defaults(manager=datastore_version.manager))

blocked_capabilities = (CapabilitiesModel.

load_blocked(manager=datastore_version.manager))

    actual_capabilities = dict(chain(

default_capabilities.iteritems(),

blocked_capabilities.iteritems()

)

)

Default capabilities. What operations are allowed by default?

Lets take a look at those datastore types that Trove supports. For now it supports: mysql, redis, cassandra, mongo.

    Not so long ago community team decided to bring up Capablities matrix [2]. By default each datastore should support core API [3] and extensions API [4] are optional. So, i’d like to suggest to mark core API capabilities as ALLOWED, and extensions API capabilities as BLOCKED.

Capabilities API

Task:     describe actual capabilities per datastore version manager

HTTP method:     GET

Method name:     show

Route:     /{tenant_id}/capabilities/{datastore_version_id}

CLI call:     trove capabilities-show --datastore-version <UUID>

Task:     list all actual capabilities

HTTP method:     GET

Method name:     index

Route:     /{tenant_id}/capabilities/

CLI call:     trove capabilities-list

Capabilities Management API

My suggestion is to add capabilities management API to trove-manage utility (for the first iteration).

trove-manage capabilities_update <datastore_manager> <Trove-”API-section”> <method> <Identifier>

where:

<datastore_manager> - datastore manager (variants: mysql, redis, cassandra, mongo, percona);

<Trove-”API-section”>  - part of API (core or extension, possible variants: Trove-Instance, Trove-Backup, Trove-Users, etc);

<method> - method available for given section of Trove API (variants: create, delete, list, show etc)

    <Identifier> - identifier that marks method from API section as ALLOWED/BLOCKED

[1] https://wiki.openstack.org/wiki/Trove/trove-capabilities

[2] https://wiki.openstack.org/wiki/Trove/DatastoreCompatibilityMatrix

[3] https://wiki.openstack.org/wiki/Trove/DatastoreCompatibilityMatrix#API_Matrix

[4] https://wiki.openstack.org/wiki/Trove/DatastoreCompatibilityMatrix#Extensions_Matrix

Best regards

Denis Makogon.
<https://wiki.openstack.org/wiki/Trove/DatastoreCompatibilityMatrix#Extensions_Matrix>

_______________________________________________
OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org<mailto:OpenStack-dev at lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

_______________________________________________
OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org<mailto:OpenStack-dev at lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

_______________________________________________
OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org<mailto:OpenStack-dev at lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140226/2f37c58b/attachment.html>

Open Stack

[openstack-dev] [TROVE] Trove capabilities

OpenStack

Community

Documentation

Branding & Legal