Open Stack

Luke, please see inline below.

--
Amrith Kumar
mailto:amrith.kumar at gmail.com


>From: Luke Browning [mailto:LukeBrowning at us.ibm.com] 
>Sent: Thursday, August 31, 2017 1:00 PM
>To: openstack-dev at lists.openstack.org
>Subject: Re: [openstack-dev] Future Trove development
>
>Amrith, 
>
>See answers to your questions below.
>
>Cheers,
>Luke  
>
>>>
>>> I understand that the Trove project will be going into maintenance mode
>>> soon. I have a number of Trove related enhancements and bug fixes that I
>>> would like to submit to the community, but I don't know if they would be
>>> considered given that the project is going into maintenance mode. Please
>>> elaborate on what you mean by maintenance mode.
>>>
>>
>> See: https://review.openstack.org/#/c/488947/
>>
>> It is not a foregone conclusion that the project will go into maintenance
>> mode. Thierry and I (for example) are not convinced that this is the right
>> course of action, but if there are no offers to contribute to the project,
>> it may be a decision by default.
>>
>> Do you subscribe to the OpenStack-dev mailing list. Please also see
>> http://openstack.markmail.org/thread/4p6gp263fei4mbru
>>
>> Finally, for a description of what maintenance mode means, please refer:
>> https://governance.openstack.org/tc/reference/tags/status_ma
>> intenance-mode.html
>>
>>>
>>>
>>> Would Trove be updated to support new OpenStack versions?
>>>
>> Would support be provided for Trove gate testing?
>>>
>> Would elements be enhanced to support Xenial and newer versions of
>>> databases?
>>>
>> Is there a time limit associated with maintenance mode? For example, would
>>> it remain in effect for a year or two after the new stuff is released?
>>>
>>> For the four questions above, I direct your attention to the definition
>> of the maintenance-mode tag (https://governance.openstack/.
>> org/tc/reference/tags/status_maintenance-mode.html) and remind you that
>> support for different versions from the community, for gate testing and for
>> elements is dependent on people participating (and their employers allowing
>> them to do this). At this point, I have no one who has stepped up to do
>> this in any significant way. There are a couple of people from China Mobile
>> who want to help but who are largely in the weeds, trying to fix this and
>> that bug but have no idea what Trove is all about. IBM has a core reviewer
>> on the project (Mariam is copied on this email). I am happy that she's
>> able to devote what time she can to the project but absent competent
>> reviewers, whether your changes ever get merged or not remains an
>> interesting question. Since you will be contributing elements and propose
>> to contribute a tool, will you be (or will IBM be) offering to support it?
>>
>>  Let me provide some back ground on my code changes, so you will have a
>> better idea of what might be submitted in a patch.
>>
>>>
>>> I have written a fully automated command that creates a virtual guest
>>> image based on Ubuntu 16.04 containing a user specified version of a
>>> database. The user's selection is validated against an internal list of
>>> databases built into the tool. Once validated, the tool creates the image,
>>> registers it with Glance, and then creates a Trove Datastore definition for
>>> it. This is done in a single command invocation that typically takes about
>>> 25 minutes to complete. For some of the databases, it takes considerably
>>> longer (2 hours for mongodb) as I have to compile source code. The guest
>>> image that is produced is complete from a Trove perspective. It includes
>>> the Cloud specific trove-guestagent.conf file and SSH public keys for the
>>> DBA and OpenStack controller nodes.
>>>
>>> What mechanism does it use to develop the image? is it diskimage-builder
>> or some other?
>
>Yes, diskimage-builder and Trove elements
>
>>
>>
>> Is there something the matter with the existing tool that exists, it
>> already works, it does more than just build an image, and it is already
>> integrated in the gate. Why not use that?
>


I should have said this before (like in the last reply) and I may not have, but I am broadly in favor of the approach you proposed (I did look briefly at the repository that you provided). The place(s) where I had reservations relate primarily to how this new tool would interface/interact with the existing Trove. More details about that below.


>Is your question why don't we use trovestack?    If so, we were looking for
>something that was more streamlined with fewer dependencies.   The tool
>would need to interface with the client's cloud.   Our tool is directed at
>customers and IBM lab based services, neither of whom is expected to be
>an OpenStack or Trove expert.

The guts of image building in trovestack are easy to abstract out. There is a trovestack command (image-build) which is a code path that can be easily leveraged for this. The reason I bring this up is that I would much rather this new tool, and the elements it comes with be the thing that is actively tested by the CI each time a Trove build is tested. I would therefore, much rather that the existing image building stuff be somehow integrated with the new tool that you have created, and the old stuff can be junked.

>
>Beyond that, there is a difference in how are images our built.   First, we need
>to control which versions of databases are built as opposed to letting the stack
>determine that.    ppc64le support in the database communities is at varying
>stages.   In some cases, there are packages and in others there are not.  In
>addition, there are specific versions that we have tested and enhanced from
>a performance perspective that we would like to include in our offering. 
>  

Makes perfect sense.

>The second part of this is related to the Trove Guest Agent.   We needed to
>make a few code changes to support different versions and to fix bugs that
>we have found.   Anyway, this led to us carrying patches in the tool, which is
>part of a broader strategy to stage our database deliverables.   We would
>like to add support for new databases over time.   To deliver these to our
>clients without them having to re-install Trove on the controller side.
>Fundamentally, this means isolating the database specific log to the Trove
>Guest Agent and Trove elements.   
>

I don't completely understand the issue with isolating log files but I would much rather see the fixes you have incorporated into the upstream, and not carried in your elements. 

>So, this is a rather long explanation, but that is why we couldn't use Trovestack
>or the existing tools as is.   There are external considerations such as the
>state of the database communities and delivery strategies that warranted
>a slightly different approach. 
>

Makes sense, why not make that the thing that Trove ships instead of creating a support nightmare with a Trove way and the IBM tool way?

>>
>> 
>> This approach is nice (having a full guest image with the guest agent and
>> all that but as you observe, it takes about 2 hours per build and from a
>> developers perspective, this is a pain.
>
>It takes 2 hours to build mongodb, because there is no package provided 
>by the community for ppc64le.   In general, it takes 25 minutes to generate
>images and create the datastore.    I expect that the times are roughly the
>same as with the existing tools. 
>

So your tool will likely not run in the CI as it will timeout. What about building and caching images? Also from the perspective of your users, they would much rather have a tool that finishes in 15 minutes than 2 hours so you may benefit from refactoring your elements to create an image which is all-but-guest-agent, and then have a second set of elements which takes that image as an input (not a cloud image) and just slaps the guest agent into it and does the required configurations. Happy to help with that, I've been working on this for trovestack anyway.

>>
>> I assume that the tool is therefore for production use cases.
>
>Our offering is intended for development use only, but it is a big step
>in the direction of production use cases.   The images are built for specific
>users in a more prescribed manner.
>

So, why not make that the default tool chain for trove?

>>
>> Could the elements that you have developed be used with the existing tool
>> or is your tool a complete replacement for the one that already exists?
>
>With a little bit of work they may be used with the existing tool.   I am not 
>planning on replacing existing tools.   In fact, I wasn't planning on dropping
>the new tool but I would if there is interest.   The new tool is at a public
>which I noted in my original note.
>

I don't understand this comment.

>>
>> Is there some reason that this tool was not developed in the open, using
>> the normal open development process?
>
>We were working towards a short release schedule for our new OpenDBaaS offering.
>It took three months to develop the tool and provide support for 5 databases,
>including multiple versions of some of them.   This is our second step
>to engage with the community. 
>
>>
>> There are ways to accelerate the compiling of source code and such through
>> the use of image layers (if you are using DIB). I have elements and code
>> that will build most of the trove elements in minutes and I'm waiting to
>> see whether there is any point in contributing them to the upstream or not,
>> at this point. I could show you how to do that, and it would be awesome to
>> have you build your tool into the build process somehow. That would be a
>> win-win for all ...
>>
>> I also have commands that setup Nova flavors on a per database basis.
>>> These flavors can be customized by the user before they are instantiated in
>>> the cloud.That's good to hear.
>>>
>> The existing command (trovestack) already does this.
>
>I didn't see that in the Trovestack.   Does it allow the user to customize
>the values.   Can you provide a  provide a link?
>

trovestack image-build (or build-image) is the command, if you follow that code path it will just setup environment variables (used by the elements) and run dib.

>>
>>
>>>
>>> I wouldn't expect to drop these commands to the Trove community. Only the
>>> changes to the Trove-guestagent and elements enabling databases.
>>>
>>> Here's the list of databases that are currently supported for ppc64le. In
>>> some cases, I have had to add Xenial support.
>>>
>>> mariadb 10.1 --- package comes from the community
>>> mongodb 3.4 community edition --- source code from the community is
>>> compiled as there is no ppc64le package
>>> mongodb 3.4 enterprise edition --- package comes from community
>>> mysql 5.7 --- package comes from Ubuntu 16.04. percona-xtrabackup-24 is
>>> compiled as there is no power support in community
>>> postgresql 9.5 --- package comes from Ubuntu 16.04
>>> postgresql 9.6 --- package comes from community
>>> redis 3.0 --- package comes from Ubuntu 16.04
>>> redis 3.2 --- source code from the community is compiled
>>>
>> Why not just contribute the elements to the existing project and have the
>> current tool use them?
>
>That is my plan.
>

Wait, you will have the existing tool use your elements? That's awesome.


>>
>> What of other databases currently supported? galera cluster (pxc),
>> vertica, db2, percona, couchdb, and couchbase?
>
>I haven't messed with them.   My changes are hidden under environment
>variables so the absence of environment variables should yield the old code.
>

Sounds like a support challenge in the making.

>>
>>
>>>
>>> My element related code changes add environment variables that control
>>> the database version, the source of the database software (distro,
>>> community, enterprise), and logging options. In this way, the logic in the
>>> element is more generalized and can be made to support multiple versions. I
>>> also added an environment variable that specifies the name of the source
>>> package to be downloaded and compiled when that is applicable. I also have
>>> environment variables to enable database and trove guest agent logging.
>>>
>> Which is nice except that your guest agent code may or may not know how
>> to deal with that.
>
>The guest agent and image logic are coordinated.  
> 
>>
>>
>>>
>>>
>>> My trove-guestagent code changes are mostly about not reconfiguring the
>>> database with configuration data remotely passed to it from the task
>>> manager. My code changes are isolated to the prepare step. This logic looks
>>> something like: if database version x, add or remove configuration
>>> parameter. Note there is relatively very little of this code, ~150 LOCs, as
>>> database configuration parameters rarely change. Note there is already some
>>> of this DB version specific logic in the trove guest agent, so I am just
>>> adding more of it. But I believe a better way is not to alter the database
>>> configuration data when the database is fully configured in a Trove
>>> compliant manner via elements. This would be implemented through a new
>>> command argument to the Trove Guest agent which says ignore configuration
>>> changes. It would be set by the elements as well.
>>>
>> Hard to say whether this is a good approach or not with out more detail.
>>
>>
>>>
>>>
>>> I haven't looked into the scale out configuration data yet, but this can
>>> be handled differently as required.
>>>
>>> Not sure what you mean.
>
>I was talking about the database cluster configuration code which is a future
>deliverable for our offering.   We are starting simple and will incrementally expand the
>offering as time goes by.      
>
>>
>> 
>>
>>
>>> So, are these the sorts of changes that I could add in a maintenance
>>> release?
>>>
>>> Maintenance release of what, trove? Are you asking whether these changes
>> can be made in a stable branch? if yes, which branch? See
>> https://docs.openstack.org/project-team-guide/stable-branches.htmland
>> the section "Appropriate Fixes".
>>
>>
>>> 1) conditionally disable the dynamic installation and configuration of
>>> database software in the trove-guest agent via a new command argument
>>> 2) add environment variables to trove elements enabling multiple versions
>>> of databases to be installed and configured
>>> 3) add environment variables to trove elements to enable database and
>>> trove-guest agent logging
>>> 4) add xenial support to elements
>>> 5) compiling source code in elements for missing stack elements for
>>> ppc64le
>>>
>>> These changes are implemented at: https://github.com/open-power-
>>> ref-design-toolkit/os-services/tree/master/osa/dbaas/dbimage-builder
>>>
>> OK, so this is a whole new tool, sure you can propose it but how do we
>> make sure that it gets integrated into the current testing/gate? If you
>> want to drop this then I'd also have to assume that there would be some
>> code that would make it part of the current gate, and ideally make it
>> generate the images for the current gate. But for that, the fact that you
>> install the guest agent onto the image is a serious drag.
>>
>>
>> Also, why this wasn't proposed as a blueprint or spec in the upstream so
>> it could be developed in the open, I don't quite understand. After all, you
>> are now showing up with a complete tool and asking that it be included
>> which largely runs counter to the whole notion of open development, at
>> least as I understand it.
>>
>
>Above, in my original node, I said that I was only planning on dropping code
>changes  to the element and that I had a small design change to the guest agent
>that I would pursue with the community.   I am not proposing to drop the 
>tool, unless there was interest in that from the community.    
>
>> 
>> Finally, once committed, do you (or IBM) commit to maintaining this code
>> and fixing bugs in it should they arise, and upgrade/update it from time to
>> time?
>>
>
>Yes
>
>>
>>>
>>> Thanks, Luke Browning
>>>
>>>
>>