Open Stack

Let me elaborate a bit.

Ephemeral volumes described in proposed DSL

Trove-volumes:

             - provisioning: ALLOWED

             - ephemeral_volumes: BLOCKED


Model is not changes. Flat file is the only way to restrict API contract of
all Trove. Same thing with datastore configuration file - we store in a
flat file, not in the database.

Storing API contract file would give an ability to protect trove from
registering incorrect capabilities, example:

Trove-instances:

             - create ALLOWED

             - make_a_butterfly: BLOCKED

Without storing static API contract capabilities gonna be 100% corrupted,
because there's no API description inside Trove.

About reloading, file should not be reloaded, file is a static unmodifiable
template. As the response users sees the combinition of described contract
(from file) and all capabilities inside database table (as i described in
my first emai). To simplify file usage we could deserialize it into dict
object and make it as singleton to use less memory.

About Kalebs design and whole appoach. Without blocking/allowing certain
capabilities for datastores whole approach is nothing else than "help
document" and this is not acceptable.

Approach of capabilities is: "In runtime block/allow tasks to be executed
over certain objects". That why i stand on re-reviewing whole design from
scratch with all contributors.

Best regards,
Denis Makogon


On Thu, Feb 27, 2014 at 2:51 AM, Craig Vyvial <cp16net at gmail.com> wrote:

> Kaleb,
>
> I've noticed something that i think is quickly getting out of control
> using the trove-manage cmd. I saw this with datastores and now similar to
> what you are describing for the capabilities.
>
> What i would like to see is more like categorizing the cmds.
>
> example:
> trove-manage datastores [update, version_update]
> trove-manage capabilities [add, remove, associate, unassociate]
>
> I mainly suggest this so that we wont have 20 top level cmds to run from
> trove-manage. I'd like to see this something that capabilities sets as a
> new standard. It also falls in line with the nova-manage cmd as well.
>
> example:
> nova-manage services [describe_resource, list, enable, disable]
>
> I'll respond to some of the other things later.
>
> -Craig
>
>
> On Wed, Feb 26, 2014 at 10:40 AM, Kaleb Pomeroy <
> kaleb.pomeroy at rackspace.com> wrote:
>
>>  I'm not sure I understand the first question, but it seems like you are
>> asking how the default capabilities are determined. First, they should all
>> be documented. Secondly, we can have defaults setup by migrations. I agree
>> that this is not as seamless as a default config file, and that segways
>> into question two.
>>
>> We want to eventually serve this data to the customers. As far as I know,
>> there is no precedent in Openstack to serve configuration file data to
>> customers. All customer presented data should be contained in the database,
>> not in flat files. Additionally, management via the api becomes much more
>> complex when I have to write to files instead of adding removing database
>> rows.
>>
>> As far as enabled/disabled capabilities, having enabled/disabled for each
>> feature is more prone to errors or inconsistencies. Having a list of
>> allowed features solves the same problem without needing to maintain a full
>> set of features and associated flags for each datastore/version. We agreed
>> this approach much cleaner and simpler than the prior.
>>
>> Implementation will be coming shortly - I'm still working on that. I'll
>> let you know as soon as I have something ready for review.
>>
>> Also, supposedly, the discussions we had last week are recorded and
>> available via some means, but I don't know where. A lot of this was
>> discussed there if you weren't able to make it.
>>
>>
>>  ------------------------------
>> *From:* Denis Makogon [dmakogon at mirantis.com]
>> *Sent:* Wednesday, February 26, 2014 5:18 AM
>>
>> *To:* OpenStack Development Mailing List (not for usage questions)
>> *Subject:* Re: [openstack-dev] [TROVE] Trove capabilities
>>
>>    First of all, how does Trove knows what kind of capabilities it can
>> give to users ?
>>
>>  We should have some kind of identity, that stores the restricted
>> description of all capabilities. Suggested variant is not acceptable,
>> because you could register any kind of capabilities in database.
>>
>>     Main point: Trove should have restrict description of all possible
>> capabilities, and it's should not be stored in backend table.
>>
>>      Design: What is a capability? - Almost totally disagree.
>>
>> Only datastore vesrion association is significant, common capabilities
>> table is not needed at all, because all new features(capabilities) assums
>> that Trove will be re-deployed from scratch. It means that capabilities
>> description would be updated only 2 or 3 time per release.
>>
>>      Design: Why isn't this in config files? - Agreed, my solution
>> design doesn't required restart. Magic happes in runtime.
>>
>>  Development: How are capabilities checked? - Option 2 is more
>> preferable, because capabilities should be stored as dictionary object that
>> has next structure:
>>
>> API call: ALLOWED/BLOCKED
>>
>> Unfortunately given design is too broad, i dont see any implementation
>> steps description. Please provide more details and flow examples with
>> pseudo-code.
>>
>> Best regards,
>> Denis Makogon
>>
>>
>> On Wed, Feb 26, 2014 at 12:13 AM, Kaleb Pomeroy <
>> kaleb.pomeroy at rackspace.com> wrote:
>>
>>>
>>> *Design: What is a capability?*
>>> A capabilities is an attribute of a datastore/version that allows
>>> dynamic code flow for different datastore/versions, The presence of a
>>> capability determines which code path is taken. A capability is an ID, a
>>> friendly name (used in our code), and a lengthy description (for displaying
>>> to the user).
>>>
>>> *Design: Where do they live?*
>>> Capabilities will live in their own table. In addition, there will be an
>>> association table to map between datastore versions and capabilities.
>>>
>>>
>>> *Design: Why isn't this in config files? *First, they will be able to
>>> be viewed by customers, and modified by operations. That does not lend
>>> itself to config files. Additionally, we should be able to change a
>>> datastores capabilities without a restart.
>>>
>>> *Operations: What happens if the capabilities table is not populated?*
>>> We are setting everything to be *default off* for all features. For
>>> mysql to support ephemeral_volumes it would need to be added to the
>>> capabilities table and properly associated to each version of mysql.
>>>
>>> *Operations: How do you manage them?*
>>> For the first pass, trove-manage commands. There will be a
>>> add_capability, remove_capability, associate_capability_with_datastore, and
>>> unassociate_capability_from_datastore (name are completely off the top of
>>> my head and subject to discussion/change).
>>>
>>> *Development: When/where are capabilities checked/determined?*
>>> Any time you would want to fork paths. The check for ephemeral_volume
>>> would happen right before the cinder volume is created, whereas the
>>> database capability would be checked as soon as possible in the api layer.
>>>
>>> *Development: How are capabilities checked?*
>>> This is still to be determined.
>>>
>>> Option one (the one discussed at the meetup):
>>>     if(cfg.cababilites.mysql.ephemeral_volumes):
>>>         do_code()
>>>
>>> Option two:
>>>     if "ephemeral_volumes" in datastore.manager.capabilities)
>>>         do_code()
>>>
>>> The advantages of option one is consistency in checking the cfg object
>>> for configuration like things. I think that is worse than checking a
>>> property of the datastore. The advantage of the second way is that is more
>>> cleanly expresses what is happening under the hood without any additional
>>> cost. I have the second option about complete, so we can see what that
>>> looks like soon.
>>>
>>> Most of this was discussed and agreed upon last week, I just wanted to
>>> recap and confirm with everyone on what our objectives and goals are. If
>>> there are unanswered questions, lets discuss sooner rather than later.
>>>
>>>
>>> - KPom
>>>
>>>
>>>
>>> * Bonus: API discussion for future development (not right now) *I
>>> suggest we expose only the capabilities that belong to a specific
>>> datastore. As a customer, I shouldn't know every feature available, just
>>> the ones on the datastore they are looking at.
>>>
>>> GET /{tenant_id}/datastores/{datastore_version_id}/capabilities
>>>
>>>
>>> Eventually, we would want the following admin routes available, but that
>>> is mitigated by the trove-manage stuff.
>>>
>>> [GET | PUT | POST | DELETE] /capabilities
>>> [POST | DELETE] /datastores/datastore_version_id/capabilities
>>>
>>>
>>>
>>>
>>>
>>>  ------------------------------
>>> *From:* Denis Makogon [dmakogon at mirantis.com]
>>> *Sent:* Wednesday, February 19, 2014 1:00 PM
>>> *To:* OpenStack Development Mailing List (not for usage questions)
>>> *Subject:* Re: [openstack-dev] [TROVE] Trove capabilities
>>>
>>>     I know that Kaleb is working on that.
>>>
>>>  But we(me, Kaleb, rest of guys) decided that i'm gonna present my own
>>> vision of Capabilities.
>>> That is what i'm doing now. Basicaly, at meetup you all could discuss
>>> design in general, i suppose.
>>>
>>>  Best regards,
>>> Denis Makogon.
>>>
>>>
>>> On Wed, Feb 19, 2014 at 7:42 PM, Daniel Salinas <imsplitbit at gmail.com>wrote:
>>>
>>>> We're at the trove meetup for the rest of this week but you should
>>>> speak with Kaleb Pomeroy from Rackspace.  He's generated some code on this
>>>> particular subject.
>>>>
>>>>
>>>>  On Wed, Feb 19, 2014 at 11:11 AM, Denis Makogon <dmakogon at mirantis.com
>>>> > wrote:
>>>>
>>>>>   Goodday OpenStack DBaaS communtiy.
>>>>>
>>>>>  I'd like to start topic related to design of capabilities [1].
>>>>>
>>>>>  This question was raised when multiple datastores where integrated
>>>>> into Trove (such as redis, cassandra, mongo in instance modes). We
>>>>> encountered a the problem when specific datastore doesn't support
>>>>> operations that are the part of Trove API (core and extension). After some
>>>>> discussions with the community we decided to implement "capabilities"
>>>>> feature. Basicaly, "capabilities" is the list of operations acceptable for
>>>>> chosen "object". To be accurate,  "capabilities" is the API for listing
>>>>> available Trove API(core and extensions) per some kind of an identifier.
>>>>>
>>>>>  Design.
>>>>>
>>>>> I'd like to split up all tasks into small peaces:
>>>>>
>>>>>
>>>>>    1.
>>>>>
>>>>>    Nameing convintion.<https://docs.google.com/a/mirantis.com/document/d/1FDhenfF47UYdiqTr5dGHSG_Gk828yFuvew92I9qil5I/edit#heading=h.dqx5s9nxd86g>
>>>>>    2.
>>>>>
>>>>>    DSL for writing reference to available capabilities.<https://docs.google.com/a/mirantis.com/document/d/1FDhenfF47UYdiqTr5dGHSG_Gk828yFuvew92I9qil5I/edit#heading=h.a86gyngvxgek>
>>>>>    3.
>>>>>
>>>>>    Сustody strategy. How and what to store at the back-end?<https://docs.google.com/a/mirantis.com/document/d/1FDhenfF47UYdiqTr5dGHSG_Gk828yFuvew92I9qil5I/edit#heading=h.p3kja3s8zex>
>>>>>    4.
>>>>>
>>>>>    Pinning to datastore or version?<https://docs.google.com/a/mirantis.com/document/d/1FDhenfF47UYdiqTr5dGHSG_Gk828yFuvew92I9qil5I/edit#heading=h.mvusz3lj5res>
>>>>>    5.
>>>>>
>>>>>    How to load  actual API reference?<https://docs.google.com/a/mirantis.com/document/d/1FDhenfF47UYdiqTr5dGHSG_Gk828yFuvew92I9qil5I/edit#heading=h.df6yrbqny8te>
>>>>>    6.
>>>>>
>>>>>    Default capabilities. What operations are allowed by default?<https://docs.google.com/a/mirantis.com/document/d/1FDhenfF47UYdiqTr5dGHSG_Gk828yFuvew92I9qil5I/edit#heading=h.gjahbyoqoq5d>
>>>>>    7.
>>>>>
>>>>>    Capablities API.<https://docs.google.com/a/mirantis.com/document/d/1FDhenfF47UYdiqTr5dGHSG_Gk828yFuvew92I9qil5I/edit#heading=h.ihzh5pv7pmci>
>>>>>    8.
>>>>>
>>>>>    Capabilities Management API.<https://docs.google.com/a/mirantis.com/document/d/1FDhenfF47UYdiqTr5dGHSG_Gk828yFuvew92I9qil5I/edit#heading=h.paouley6yndv>
>>>>>
>>>>>
>>>>>  Naming convention
>>>>>
>>>>> Suggestions:
>>>>>
>>>>>    1.
>>>>>
>>>>>    Filename convension:
>>>>>
>>>>>  {datastore or datastore_version_manager}.capabilities
>>>>>
>>>>>    1.
>>>>>
>>>>>    Attribute convension:
>>>>>
>>>>>  Trove-"API-section":
>>>>>
>>>>> - method: Identifier
>>>>>
>>>>>     Available sections: instance, backip, users, schemes.
>>>>>
>>>>>     Capability Identifier: ALLOWED/BLOCKED
>>>>>   DSL for writing reference to available capabilities
>>>>>
>>>>> There are several options of DSL's that can be chosen for writing
>>>>> capabilities references, options:
>>>>>
>>>>>    1.
>>>>>
>>>>>    JSON.
>>>>>    2.
>>>>>
>>>>>    YAML.
>>>>>    3.
>>>>>
>>>>>    XML.
>>>>>
>>>>>      Personaly i'd like to choose YAML as appropriate DSL. Described
>>>>> capabilities would look like: mysql.capabilities
>>>>>
>>>>> Trove-Instance:
>>>>>
>>>>> - create: ALLOWED
>>>>>
>>>>> - list: BLOCKED
>>>>>
>>>>> - delete: ALLOWED
>>>>>
>>>>> Trove-Backup:
>>>>>
>>>>> - create: BLOCKED
>>>>>
>>>>> Trove-Users:
>>>>>
>>>>> - create: BLOCKED
>>>>>
>>>>>  Сustody strategy. How and what to store at the back-end ?
>>>>>
>>>>> Before suggesting database table scheme i'd like to describe what we
>>>>> are going to store. It would be better to store only blocked capabilities,
>>>>> its cheaper, instead of storing all capabilities, either blocked and
>>>>> allowed.
>>>>>
>>>>> Backend table scheme.
>>>>>
>>>>>  Table 1 - Capabilities table scheme
>>>>>
>>>>> Datastore or Datastore version manager
>>>>>
>>>>> Capabilities
>>>>>
>>>>> datastore or manager
>>>>>
>>>>>   "Trove-"API-section":
>>>>>
>>>>>                      - method: Identifier"
>>>>>
>>>>>      Where Identifier could be ALLOWED or BLOCKED
>>>>>   Pinning to datastore or datastore version?
>>>>>
>>>>> As you can see from previous topics, it would be better to pin
>>>>> capabilities directly to datastore version, since one manager could be
>>>>> assigned to multiple datastore versions.
>>>>>
>>>>>     So, capabilities backend table scheme would have next desription:
>>>>>
>>>>>
>>>>>    -
>>>>>
>>>>>    datastrove_version_manager:
>>>>>
>>>>>  name: manager, Foreing key from DatastoreVersion table;
>>>>>
>>>>> type: String.
>>>>>
>>>>>    -
>>>>>
>>>>>    capabilities:
>>>>>
>>>>>  name: capabilities;
>>>>>
>>>>> type: Text.
>>>>>
>>>>>  How to load actual API reference?
>>>>>
>>>>>     According to previous topics i forsee the next way of discovering
>>>>> actual capabilities at runtime:
>>>>>
>>>>>     Since YAML format perfectly could be loaded as Python dictionary,
>>>>> the easiest way is to merge two dictionaries.
>>>>>
>>>>>     Example:
>>>>>
>>>>>      default_capabilities = (CapabilitiesModel.
>>>>>
>>>>>             load_defaults(manager=datastore_version.manager))
>>>>>
>>>>> blocked_capabilities = (CapabilitiesModel.
>>>>>
>>>>> load_blocked(manager=datastore_version.manager))
>>>>>
>>>>>     actual_capabilities = dict(chain(
>>>>>
>>>>> default_capabilities.iteritems(),
>>>>>
>>>>> blocked_capabilities.iteritems()
>>>>>
>>>>> )
>>>>>
>>>>> )
>>>>>
>>>>>
>>>>>
>>>>>   Default capabilities. What operations are allowed by default?
>>>>>
>>>>>
>>>>>
>>>>> Lets take a look at those datastore types that Trove supports. For now
>>>>> it supports: mysql, redis, cassandra, mongo.
>>>>>
>>>>>     Not so long ago community team decided to bring up Capablities
>>>>> matrix [2]. By default each datastore should support core API [3] and
>>>>> extensions API [4] are optional. So, i'd like to suggest to mark core API
>>>>> capabilities as ALLOWED, and extensions API capabilities as BLOCKED.
>>>>>  Capabilities API
>>>>>
>>>>> Task:     describe actual capabilities per datastore version manager
>>>>>
>>>>> HTTP method:     GET
>>>>>
>>>>> Method name:     show
>>>>>
>>>>> Route:     /{tenant_id}/capabilities/{datastore_version_id}
>>>>>
>>>>> CLI call:     trove capabilities-show --datastore-version <UUID>
>>>>>
>>>>>  Task:     list all actual capabilities
>>>>>
>>>>> HTTP method:     GET
>>>>>
>>>>> Method name:     index
>>>>>
>>>>> Route:     /{tenant_id}/capabilities/
>>>>>
>>>>> CLI call:     trove capabilities-list
>>>>>  Capabilities Management API
>>>>>
>>>>>
>>>>>
>>>>> My suggestion is to add capabilities management API to trove-manageutility (for the first iteration).
>>>>>
>>>>>  trove-manage capabilities_update <datastore_manager>
>>>>> <Trove-"API-section"> <method> <Identifier>
>>>>>
>>>>> where:
>>>>>
>>>>> <datastore_manager> - datastore manager (variants: mysql, redis,
>>>>> cassandra, mongo, percona);
>>>>>
>>>>> <Trove-"API-section">  - part of API (core or extension, possible
>>>>> variants: Trove-Instance, Trove-Backup, Trove-Users, etc);
>>>>>
>>>>> <method> - method available for given section of Trove API (variants:
>>>>> create, delete, list, show etc)
>>>>>
>>>>>     <Identifier> - identifier that marks method from API section as
>>>>> ALLOWED/BLOCKED
>>>>>
>>>>>
>>>>>  [1] https://wiki.openstack.org/wiki/Trove/trove-capabilities
>>>>>
>>>>> [2] https://wiki.openstack.org/wiki/Trove/DatastoreCompatibilityMatrix
>>>>>
>>>>> [3]
>>>>> https://wiki.openstack.org/wiki/Trove/DatastoreCompatibilityMatrix#API_Matrix
>>>>>
>>>>> [4]
>>>>> https://wiki.openstack.org/wiki/Trove/DatastoreCompatibilityMatrix#Extensions_Matrix
>>>>>
>>>>>
>>>>>
>>>>>  Best regards
>>>>>
>>>>> Denis Makogon.
>>>>> <https://wiki.openstack.org/wiki/Trove/DatastoreCompatibilityMatrix#Extensions_Matrix>
>>>>>
>>>>>
>>>>>  _______________________________________________
>>>>> OpenStack-dev mailing list
>>>>> OpenStack-dev at lists.openstack.org
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> OpenStack-dev mailing list
>>>> OpenStack-dev at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>>
>>>
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140227/f432fb52/attachment.html>