Open Stack

​The email thread below is an one that I have been having with Luke and
some others at IBM who have been working on Trove.

I feel that these kind(s) of conversations are better had in the mailing
list, and in replying to this email, I let the participants know that
absent any objections, I would like to continue this conversation on the
ML. Hearing nothing in the past two days, I am forwarding this to the ML.

-amrith
​


> On Thu, Aug 17, 2017 at 4:24 PM, Luke Browning <LukeBrowning at us.ibm.com>
> wrote:
>
>> Hi Amrith,
>>
>> I understand that the Trove project will be going into maintenance mode
>> soon. I have a number of Trove related enhancements and bug fixes that I
>> would like to submit to the community, but I don't know if they would be
>> considered given that the project is going into maintenance mode. Please
>> elaborate on what you mean by maintenance mode.
>>
>
> ​See: https://review.openstack.org/#/c/488947/
>
> It is not a foregone conclusion that the project will go into maintenance
> mode. Thierry and I (for example) are not convinced that this is the right
> course of action, but if there are no offers to contribute to the project,
> it may be a decision by default.
>
> Do you subscribe to the OpenStack-dev mailing list​. Please also see
> http://openstack.markmail.org/thread/4p6gp263fei4mbru
>
> Finally, for a description of what maintenance mode means, please refer:
> https://governance.openstack.org/tc/reference/tags/status_ma
> intenance-mode.html
>
>>
>>
>> Would Trove be updated to support new OpenStack versions?
>>
> Would support be provided for Trove gate testing?
>>
> Would elements be enhanced to support Xenial and newer versions of
>> databases?
>>
> Is there a time limit associated with maintenance mode? For example, would
>> it remain in effect for a year or two after the new stuff is released?
>>
>> ​For the four questions above, I direct your attention to the definition
> of the maintenance-mode tag (https://governance.openstack.
> org/tc/reference/tags/status_maintenance-mode.html) and remind you that
> support for different versions from the community, for gate testing and for
> elements is dependent on people participating (and their employers allowing
> them to do this). At this point, I have no one who has stepped up to do
> this in any significant way. There are a couple of people from China Mobile
> who want to help but who are largely in the weeds, trying to fix this and
> that bug but have no idea what Trove is all about. IBM has a core reviewer
> on the project ​(Mariam is copied on this email). I am happy that she's
> able to devote what time she can to the project but absent competent
> reviewers, whether your changes ever get merged or not remains an
> interesting question. Since you will be contributing elements and propose
> to contribute a tool, will you be (or will IBM be) offering to support it?
>
>  Let me provide some back ground on my code changes, so you will have a
> better idea of what might be submitted in a patch.
>
>>
>> I have written a fully automated command that creates a virtual guest
>> image based on Ubuntu 16.04 containing a user specified version of a
>> database. The user's selection is validated against an internal list of
>> databases built into the tool. Once validated, the tool creates the image,
>> registers it with Glance, and then creates a Trove Datastore definition for
>> it. This is done in a single command invocation that typically takes about
>> 25 minutes to complete. For some of the databases, it takes considerably
>> longer (2 hours for mongodb) as I have to compile source code. The guest
>> image that is produced is complete from a Trove perspective. It includes
>> the Cloud specific trove-guestagent.conf file and SSH public keys for the
>> DBA and OpenStack controller nodes.
>>
>> ​What mechanism does it use to develop the image? is it diskimage-builder
> or some other?
>
>
> Is there something the matter with the existing tool that exists, it
> already works, it does more than just build an image, and it is already
> integrated in the gate. Why not use that?​
>
> ​
> This approach is nice (having a full guest image with the guest agent and
> all that but as you observe, it takes about 2 hours per build and from a
> developers perspective, this is a pain.
>
> I assume that the tool is therefore for production use cases.
>
> Could the elements that you have developed be used with the existing tool
> or is your tool a complete replacement for the one that already exists?​
>
> ​Is there some reason that this tool was not developed in the open, using
> the normal open development process?
>
> There are ways to accelerate the compiling of source code and such through
> the use of image layers (if you are using DIB). I have elements and code
> that will build most of the trove elements in minutes and I'm waiting to
> see whether there is any point in contributing them to the upstream or not,
> at this point. I could show you how to do that, and it would be awesome to
> have you build your tool into the build process somehow. That would be a
> win-win for all ...​
>
> I also have commands that setup Nova flavors on a per database basis.
>> These flavors can be customized by the user before they are instantiated in
>> the cloud.
>>
> ​The existing command (trovestack) already does this.​
>
>
>>
>> I wouldn't expect to drop these commands to the Trove community. Only the
>> changes to the Trove-guestagent and elements enabling databases.
>>
>> Here's the list of databases that are currently supported for ppc64le. In
>> some cases, I have had to add Xenial support.
>>
>> mariadb 10.1 --- package comes from the community
>> mongodb 3.4 community edition --- source code from the community is
>> compiled as there is no ppc64le package
>> mongodb 3.4 enterprise edition --- package comes from community
>> mysql 5.7 --- package comes from Ubuntu 16.04. percona-xtrabackup-24 is
>> compiled as there is no power support in community
>> postgresql 9.5 --- package comes from Ubuntu 16.04
>> postgresql 9.6 --- package comes from community
>> redis 3.0 --- package comes from Ubuntu 16.04
>> redis 3.2 --- source code from the community is compiled
>>
> ​Why not just contribute the elements to the existing project and have the
> current tool use them?​
>
> ​What of other databases currently supported? galera cluster (pxc),
> vertica, db2, percona, couchdb, and couchbase?
>
>
>>
>> My element related code changes add environment variables that control
>> the database version, the source of the database software (distro,
>> community, enterprise), and logging options. In this way, the logic in the
>> element is more generalized and can be made to support multiple versions. I
>> also added an environment variable that specifies the name of the source
>> package to be downloaded and compiled when that is applicable. I also have
>> environment variables to enable database and trove guest agent logging.
>>
> ​Which is nice except that your guest agent code may or may not know how
> to deal with that.​
>
>
>>
>>
>> My trove-guestagent code changes are mostly about not reconfiguring the
>> database with configuration data remotely passed to it from the task
>> manager. My code changes are isolated to the prepare step. This logic looks
>> something like: if database version x, add or remove configuration
>> parameter. Note there is relatively very little of this code, ~150 LOCs, as
>> database configuration parameters rarely change. Note there is already some
>> of this DB version specific logic in the trove guest agent, so I am just
>> adding more of it. But I believe a better way is not to alter the database
>> configuration data when the database is fully configured in a Trove
>> compliant manner via elements. This would be implemented through a new
>> command argument to the Trove Guest agent which says ignore configuration
>> changes. It would be set by the elements as well.
>>
> ​Hard to say whether this is a good approach or not with out more detail.​
>
>
>>
>>
>> I haven't looked into the scale out configuration data yet, but this can
>> be handled differently as required.
>>
>> ​Not sure what you mean.
>
> ​
>
>
>> So, are these the sorts of changes that I could add in a maintenance
>> release?
>>
>> ​Maintenance release of what, trove? Are you asking whether these changes
> can be made in a stable branch? if yes, which branch? See
> https://docs.openstack.org/project-team-guide/stable-branches.html and
> the section "Appropriate Fixes".​
>
>
>> 1) conditionally disable the dynamic installation and configuration of
>> database software in the trove-guest agent via a new command argument
>> 2) add environment variables to trove elements enabling multiple versions
>> of databases to be installed and configured
>> 3) add environment variables to trove elements to enable database and
>> trove-guest agent logging
>> 4) add xenial support to elements
>> 5) compiling source code in elements for missing stack elements for
>> ppc64le
>>
>> These changes are implemented at: https://github.com/open-power-
>> ref-design-toolkit/os-services/tree/master/osa/dbaas/dbimage-builder
>>
> ​OK, so this is a whole new tool, sure you can propose it but how do we
> make sure that it gets integrated into the current testing/gate? If you
> want to drop this then I'd also have to assume that there would be some
> code that would make it part of the current gate, and ideally make it
> generate the images for the current gate. But for that, the fact that you
> install the guest agent onto the image is a serious drag.
>
>
> Also, why this wasn't proposed as a blueprint or spec in the upstream so
> it could be developed in the open, I don't quite understand. After all, you
> are now showing up with a complete tool and asking that it be included
> which largely runs counter to the whole notion of open development, at
> least as I understand it.​
>
> ​
> Finally, once committed, do you (or IBM) commit to maintaining this code
> and fixing bugs in it should they arise, and upgrade/update it from time to
> time?
>
>
>>
>> Thanks, Luke Browning
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170819/a4c6dcc1/attachment.html>