[openstack-dev] [TROVE] Trove capabilities
Craig Vyvial
cp16net at gmail.com
Thu Feb 27 00:51:37 UTC 2014
Kaleb,
I've noticed something that i think is quickly getting out of control using
the trove-manage cmd. I saw this with datastores and now similar to what
you are describing for the capabilities.
What i would like to see is more like categorizing the cmds.
example:
trove-manage datastores [update, version_update]
trove-manage capabilities [add, remove, associate, unassociate]
I mainly suggest this so that we wont have 20 top level cmds to run from
trove-manage. I'd like to see this something that capabilities sets as a
new standard. It also falls in line with the nova-manage cmd as well.
example:
nova-manage services [describe_resource, list, enable, disable]
I'll respond to some of the other things later.
-Craig
On Wed, Feb 26, 2014 at 10:40 AM, Kaleb Pomeroy <kaleb.pomeroy at rackspace.com
> wrote:
> I'm not sure I understand the first question, but it seems like you are
> asking how the default capabilities are determined. First, they should all
> be documented. Secondly, we can have defaults setup by migrations. I agree
> that this is not as seamless as a default config file, and that segways
> into question two.
>
> We want to eventually serve this data to the customers. As far as I know,
> there is no precedent in Openstack to serve configuration file data to
> customers. All customer presented data should be contained in the database,
> not in flat files. Additionally, management via the api becomes much more
> complex when I have to write to files instead of adding removing database
> rows.
>
> As far as enabled/disabled capabilities, having enabled/disabled for each
> feature is more prone to errors or inconsistencies. Having a list of
> allowed features solves the same problem without needing to maintain a full
> set of features and associated flags for each datastore/version. We agreed
> this approach much cleaner and simpler than the prior.
>
> Implementation will be coming shortly - I'm still working on that. I'll
> let you know as soon as I have something ready for review.
>
> Also, supposedly, the discussions we had last week are recorded and
> available via some means, but I don't know where. A lot of this was
> discussed there if you weren't able to make it.
>
>
> ------------------------------
> *From:* Denis Makogon [dmakogon at mirantis.com]
> *Sent:* Wednesday, February 26, 2014 5:18 AM
>
> *To:* OpenStack Development Mailing List (not for usage questions)
> *Subject:* Re: [openstack-dev] [TROVE] Trove capabilities
>
> First of all, how does Trove knows what kind of capabilities it can
> give to users ?
>
> We should have some kind of identity, that stores the restricted
> description of all capabilities. Suggested variant is not acceptable,
> because you could register any kind of capabilities in database.
>
> Main point: Trove should have restrict description of all possible
> capabilities, and it's should not be stored in backend table.
>
> Design: What is a capability? - Almost totally disagree.
>
> Only datastore vesrion association is significant, common capabilities
> table is not needed at all, because all new features(capabilities) assums
> that Trove will be re-deployed from scratch. It means that capabilities
> description would be updated only 2 or 3 time per release.
>
> Design: Why isn't this in config files? - Agreed, my solution design
> doesn't required restart. Magic happes in runtime.
>
> Development: How are capabilities checked? - Option 2 is more
> preferable, because capabilities should be stored as dictionary object that
> has next structure:
>
> API call: ALLOWED/BLOCKED
>
> Unfortunately given design is too broad, i dont see any implementation
> steps description. Please provide more details and flow examples with
> pseudo-code.
>
> Best regards,
> Denis Makogon
>
>
> On Wed, Feb 26, 2014 at 12:13 AM, Kaleb Pomeroy <
> kaleb.pomeroy at rackspace.com> wrote:
>
>>
>> *Design: What is a capability?*
>> A capabilities is an attribute of a datastore/version that allows dynamic
>> code flow for different datastore/versions, The presence of a capability
>> determines which code path is taken. A capability is an ID, a friendly name
>> (used in our code), and a lengthy description (for displaying to the user).
>>
>> *Design: Where do they live?*
>> Capabilities will live in their own table. In addition, there will be an
>> association table to map between datastore versions and capabilities.
>>
>>
>> *Design: Why isn't this in config files? *First, they will be able to be
>> viewed by customers, and modified by operations. That does not lend itself
>> to config files. Additionally, we should be able to change a datastores
>> capabilities without a restart.
>>
>> *Operations: What happens if the capabilities table is not populated?*
>> We are setting everything to be *default off* for all features. For
>> mysql to support ephemeral_volumes it would need to be added to the
>> capabilities table and properly associated to each version of mysql.
>>
>> *Operations: How do you manage them?*
>> For the first pass, trove-manage commands. There will be a
>> add_capability, remove_capability, associate_capability_with_datastore, and
>> unassociate_capability_from_datastore (name are completely off the top of
>> my head and subject to discussion/change).
>>
>> *Development: When/where are capabilities checked/determined?*
>> Any time you would want to fork paths. The check for ephemeral_volume
>> would happen right before the cinder volume is created, whereas the
>> database capability would be checked as soon as possible in the api layer.
>>
>> *Development: How are capabilities checked?*
>> This is still to be determined.
>>
>> Option one (the one discussed at the meetup):
>> if(cfg.cababilites.mysql.ephemeral_volumes):
>> do_code()
>>
>> Option two:
>> if "ephemeral_volumes" in datastore.manager.capabilities)
>> do_code()
>>
>> The advantages of option one is consistency in checking the cfg object
>> for configuration like things. I think that is worse than checking a
>> property of the datastore. The advantage of the second way is that is more
>> cleanly expresses what is happening under the hood without any additional
>> cost. I have the second option about complete, so we can see what that
>> looks like soon.
>>
>> Most of this was discussed and agreed upon last week, I just wanted to
>> recap and confirm with everyone on what our objectives and goals are. If
>> there are unanswered questions, lets discuss sooner rather than later.
>>
>>
>> - KPom
>>
>>
>>
>> * Bonus: API discussion for future development (not right now) *I
>> suggest we expose only the capabilities that belong to a specific
>> datastore. As a customer, I shouldn't know every feature available, just
>> the ones on the datastore they are looking at.
>>
>> GET /{tenant_id}/datastores/{datastore_version_id}/capabilities
>>
>>
>> Eventually, we would want the following admin routes available, but that
>> is mitigated by the trove-manage stuff.
>>
>> [GET | PUT | POST | DELETE] /capabilities
>> [POST | DELETE] /datastores/datastore_version_id/capabilities
>>
>>
>>
>>
>>
>> ------------------------------
>> *From:* Denis Makogon [dmakogon at mirantis.com]
>> *Sent:* Wednesday, February 19, 2014 1:00 PM
>> *To:* OpenStack Development Mailing List (not for usage questions)
>> *Subject:* Re: [openstack-dev] [TROVE] Trove capabilities
>>
>> I know that Kaleb is working on that.
>>
>> But we(me, Kaleb, rest of guys) decided that i'm gonna present my own
>> vision of Capabilities.
>> That is what i'm doing now. Basicaly, at meetup you all could discuss
>> design in general, i suppose.
>>
>> Best regards,
>> Denis Makogon.
>>
>>
>> On Wed, Feb 19, 2014 at 7:42 PM, Daniel Salinas <imsplitbit at gmail.com>wrote:
>>
>>> We're at the trove meetup for the rest of this week but you should speak
>>> with Kaleb Pomeroy from Rackspace. He's generated some code on this
>>> particular subject.
>>>
>>>
>>> On Wed, Feb 19, 2014 at 11:11 AM, Denis Makogon <dmakogon at mirantis.com>wrote:
>>>
>>>> Goodday OpenStack DBaaS communtiy.
>>>>
>>>> I'd like to start topic related to design of capabilities [1].
>>>>
>>>> This question was raised when multiple datastores where integrated
>>>> into Trove (such as redis, cassandra, mongo in instance modes). We
>>>> encountered a the problem when specific datastore doesn't support
>>>> operations that are the part of Trove API (core and extension). After some
>>>> discussions with the community we decided to implement "capabilities"
>>>> feature. Basicaly, "capabilities" is the list of operations acceptable for
>>>> chosen "object". To be accurate, "capabilities" is the API for listing
>>>> available Trove API(core and extensions) per some kind of an identifier.
>>>>
>>>> Design.
>>>>
>>>> I'd like to split up all tasks into small peaces:
>>>>
>>>>
>>>> 1.
>>>>
>>>> Nameing convintion.<https://docs.google.com/a/mirantis.com/document/d/1FDhenfF47UYdiqTr5dGHSG_Gk828yFuvew92I9qil5I/edit#heading=h.dqx5s9nxd86g>
>>>> 2.
>>>>
>>>> DSL for writing reference to available capabilities.<https://docs.google.com/a/mirantis.com/document/d/1FDhenfF47UYdiqTr5dGHSG_Gk828yFuvew92I9qil5I/edit#heading=h.a86gyngvxgek>
>>>> 3.
>>>>
>>>> Сustody strategy. How and what to store at the back-end?<https://docs.google.com/a/mirantis.com/document/d/1FDhenfF47UYdiqTr5dGHSG_Gk828yFuvew92I9qil5I/edit#heading=h.p3kja3s8zex>
>>>> 4.
>>>>
>>>> Pinning to datastore or version?<https://docs.google.com/a/mirantis.com/document/d/1FDhenfF47UYdiqTr5dGHSG_Gk828yFuvew92I9qil5I/edit#heading=h.mvusz3lj5res>
>>>> 5.
>>>>
>>>> How to load actual API reference?<https://docs.google.com/a/mirantis.com/document/d/1FDhenfF47UYdiqTr5dGHSG_Gk828yFuvew92I9qil5I/edit#heading=h.df6yrbqny8te>
>>>> 6.
>>>>
>>>> Default capabilities. What operations are allowed by default?<https://docs.google.com/a/mirantis.com/document/d/1FDhenfF47UYdiqTr5dGHSG_Gk828yFuvew92I9qil5I/edit#heading=h.gjahbyoqoq5d>
>>>> 7.
>>>>
>>>> Capablities API.<https://docs.google.com/a/mirantis.com/document/d/1FDhenfF47UYdiqTr5dGHSG_Gk828yFuvew92I9qil5I/edit#heading=h.ihzh5pv7pmci>
>>>> 8.
>>>>
>>>> Capabilities Management API.<https://docs.google.com/a/mirantis.com/document/d/1FDhenfF47UYdiqTr5dGHSG_Gk828yFuvew92I9qil5I/edit#heading=h.paouley6yndv>
>>>>
>>>>
>>>> Naming convention
>>>>
>>>> Suggestions:
>>>>
>>>> 1.
>>>>
>>>> Filename convension:
>>>>
>>>> {datastore or datastore_version_manager}.capabilities
>>>>
>>>> 1.
>>>>
>>>> Attribute convension:
>>>>
>>>> Trove-"API-section":
>>>>
>>>> - method: Identifier
>>>>
>>>> Available sections: instance, backip, users, schemes.
>>>>
>>>> Capability Identifier: ALLOWED/BLOCKED
>>>> DSL for writing reference to available capabilities
>>>>
>>>> There are several options of DSL's that can be chosen for writing
>>>> capabilities references, options:
>>>>
>>>> 1.
>>>>
>>>> JSON.
>>>> 2.
>>>>
>>>> YAML.
>>>> 3.
>>>>
>>>> XML.
>>>>
>>>> Personaly i'd like to choose YAML as appropriate DSL. Described
>>>> capabilities would look like: mysql.capabilities
>>>>
>>>> Trove-Instance:
>>>>
>>>> - create: ALLOWED
>>>>
>>>> - list: BLOCKED
>>>>
>>>> - delete: ALLOWED
>>>>
>>>> Trove-Backup:
>>>>
>>>> - create: BLOCKED
>>>>
>>>> Trove-Users:
>>>>
>>>> - create: BLOCKED
>>>>
>>>> Сustody strategy. How and what to store at the back-end ?
>>>>
>>>> Before suggesting database table scheme i'd like to describe what we
>>>> are going to store. It would be better to store only blocked capabilities,
>>>> its cheaper, instead of storing all capabilities, either blocked and
>>>> allowed.
>>>>
>>>> Backend table scheme.
>>>>
>>>> Table 1 - Capabilities table scheme
>>>>
>>>> Datastore or Datastore version manager
>>>>
>>>> Capabilities
>>>>
>>>> datastore or manager
>>>>
>>>> "Trove-"API-section":
>>>>
>>>> - method: Identifier"
>>>>
>>>> Where Identifier could be ALLOWED or BLOCKED
>>>> Pinning to datastore or datastore version?
>>>>
>>>> As you can see from previous topics, it would be better to pin
>>>> capabilities directly to datastore version, since one manager could be
>>>> assigned to multiple datastore versions.
>>>>
>>>> So, capabilities backend table scheme would have next desription:
>>>>
>>>>
>>>> -
>>>>
>>>> datastrove_version_manager:
>>>>
>>>> name: manager, Foreing key from DatastoreVersion table;
>>>>
>>>> type: String.
>>>>
>>>> -
>>>>
>>>> capabilities:
>>>>
>>>> name: capabilities;
>>>>
>>>> type: Text.
>>>>
>>>> How to load actual API reference?
>>>>
>>>> According to previous topics i forsee the next way of discovering
>>>> actual capabilities at runtime:
>>>>
>>>> Since YAML format perfectly could be loaded as Python dictionary,
>>>> the easiest way is to merge two dictionaries.
>>>>
>>>> Example:
>>>>
>>>> default_capabilities = (CapabilitiesModel.
>>>>
>>>> load_defaults(manager=datastore_version.manager))
>>>>
>>>> blocked_capabilities = (CapabilitiesModel.
>>>>
>>>> load_blocked(manager=datastore_version.manager))
>>>>
>>>> actual_capabilities = dict(chain(
>>>>
>>>> default_capabilities.iteritems(),
>>>>
>>>> blocked_capabilities.iteritems()
>>>>
>>>> )
>>>>
>>>> )
>>>>
>>>>
>>>>
>>>> Default capabilities. What operations are allowed by default?
>>>>
>>>>
>>>>
>>>> Lets take a look at those datastore types that Trove supports. For now
>>>> it supports: mysql, redis, cassandra, mongo.
>>>>
>>>> Not so long ago community team decided to bring up Capablities
>>>> matrix [2]. By default each datastore should support core API [3] and
>>>> extensions API [4] are optional. So, i'd like to suggest to mark core API
>>>> capabilities as ALLOWED, and extensions API capabilities as BLOCKED.
>>>> Capabilities API
>>>>
>>>> Task: describe actual capabilities per datastore version manager
>>>>
>>>> HTTP method: GET
>>>>
>>>> Method name: show
>>>>
>>>> Route: /{tenant_id}/capabilities/{datastore_version_id}
>>>>
>>>> CLI call: trove capabilities-show --datastore-version <UUID>
>>>>
>>>> Task: list all actual capabilities
>>>>
>>>> HTTP method: GET
>>>>
>>>> Method name: index
>>>>
>>>> Route: /{tenant_id}/capabilities/
>>>>
>>>> CLI call: trove capabilities-list
>>>> Capabilities Management API
>>>>
>>>>
>>>>
>>>> My suggestion is to add capabilities management API to trove-manageutility (for the first iteration).
>>>>
>>>> trove-manage capabilities_update <datastore_manager>
>>>> <Trove-"API-section"> <method> <Identifier>
>>>>
>>>> where:
>>>>
>>>> <datastore_manager> - datastore manager (variants: mysql, redis,
>>>> cassandra, mongo, percona);
>>>>
>>>> <Trove-"API-section"> - part of API (core or extension, possible
>>>> variants: Trove-Instance, Trove-Backup, Trove-Users, etc);
>>>>
>>>> <method> - method available for given section of Trove API (variants:
>>>> create, delete, list, show etc)
>>>>
>>>> <Identifier> - identifier that marks method from API section as
>>>> ALLOWED/BLOCKED
>>>>
>>>>
>>>> [1] https://wiki.openstack.org/wiki/Trove/trove-capabilities
>>>>
>>>> [2] https://wiki.openstack.org/wiki/Trove/DatastoreCompatibilityMatrix
>>>>
>>>> [3]
>>>> https://wiki.openstack.org/wiki/Trove/DatastoreCompatibilityMatrix#API_Matrix
>>>>
>>>> [4]
>>>> https://wiki.openstack.org/wiki/Trove/DatastoreCompatibilityMatrix#Extensions_Matrix
>>>>
>>>>
>>>>
>>>> Best regards
>>>>
>>>> Denis Makogon.
>>>> <https://wiki.openstack.org/wiki/Trove/DatastoreCompatibilityMatrix#Extensions_Matrix>
>>>>
>>>>
>>>> _______________________________________________
>>>> OpenStack-dev mailing list
>>>> OpenStack-dev at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>>
>>>
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140226/059912f9/attachment-0001.html>
More information about the OpenStack-dev
mailing list