Open Stack

Amrith, 

See answers to your questions below.

Cheers,
Luke 

>>
>> I understand that the Trove project will be going into maintenance mode
>> soon. I have a number of Trove related enhancements and bug fixes that 
I
>> would like to submit to the community, but I don't know if they would 
be
>> considered given that the project is going into maintenance mode. 
Please
>> elaborate on what you mean by maintenance mode.
>>
>
> ​See: https://review.openstack.org/#/c/488947/
>
> It is not a foregone conclusion that the project will go into 
maintenance
> mode. Thierry and I (for example) are not convinced that this is the 
right
> course of action, but if there are no offers to contribute to the 
project,
> it may be a decision by default.
>
> Do you subscribe to the OpenStack-dev mailing list​. Please also see
> http://openstack.markmail.org/thread/4p6gp263fei4mbru
>
> Finally, for a description of what maintenance mode means, please refer:
> https://governance.openstack.org/tc/reference/tags/status_ma
> intenance-mode.html
>
>>
>>
>> Would Trove be updated to support new OpenStack versions?
>>
> Would support be provided for Trove gate testing?
>>
> Would elements be enhanced to support Xenial and newer versions of
>> databases?
>>
> Is there a time limit associated with maintenance mode? For example, 
would
>> it remain in effect for a year or two after the new stuff is released?
>>
>> ​For the four questions above, I direct your attention to the 
definition
> of the maintenance-mode tag (https://governance.openstack.
> org/tc/reference/tags/status_maintenance-mode.html) and remind you that
> support for different versions from the community, for gate testing and 
for
> elements is dependent on people participating (and their employers 
allowing
> them to do this). At this point, I have no one who has stepped up to do
> this in any significant way. There are a couple of people from China 
Mobile
> who want to help but who are largely in the weeds, trying to fix this 
and
> that bug but have no idea what Trove is all about. IBM has a core 
reviewer
> on the project ​(Mariam is copied on this email). I am happy that she's
> able to devote what time she can to the project but absent competent
> reviewers, whether your changes ever get merged or not remains an
> interesting question. Since you will be contributing elements and 
propose
> to contribute a tool, will you be (or will IBM be) offering to support 
it?
>
>  Let me provide some back ground on my code changes, so you will have a
> better idea of what might be submitted in a patch.
>
>>
>> I have written a fully automated command that creates a virtual guest
>> image based on Ubuntu 16.04 containing a user specified version of a
>> database. The user's selection is validated against an internal list of
>> databases built into the tool. Once validated, the tool creates the 
image,
>> registers it with Glance, and then creates a Trove Datastore definition 
for
>> it. This is done in a single command invocation that typically takes 
about
>> 25 minutes to complete. For some of the databases, it takes 
considerably
>> longer (2 hours for mongodb) as I have to compile source code. The 
guest
>> image that is produced is complete from a Trove perspective. It 
includes
>> the Cloud specific trove-guestagent.conf file and SSH public keys for 
the
>> DBA and OpenStack controller nodes.
>>
>> ​What mechanism does it use to develop the image? is it 
diskimage-builder
> or some other?

Yes, diskimage-builder and Trove elements

>
>
> Is there something the matter with the existing tool that exists, it
> already works, it does more than just build an image, and it is already
> integrated in the gate. Why not use that?​

Is your question why don't we use trovestack?    If so, we were looking 
for
something that was more streamlined with fewer dependencies.   The tool
would need to interface with the client's cloud.   Our tool is directed at
customers and IBM lab based services, neither of whom is expected to be
an OpenStack or Trove expert.

Beyond that, there is a difference in how are images our built.   First, 
we need
to control which versions of databases are built as opposed to letting the 
stack
determine that.    ppc64le support in the database communities is at 
varying
stages.   In some cases, there are packages and in others there are not. 
In
addition, there are specific versions that we have tested and enhanced 
from
a performance perspective that we would like to include in our offering. 
 
The second part of this is related to the Trove Guest Agent.   We needed 
to
make a few code changes to support different versions and to fix bugs that
we have found.   Anyway, this led to us carrying patches in the tool, 
which is
part of a broader strategy to stage our database deliverables.   We would
like to add support for new databases over time.   To deliver these to our
clients without them having to re-install Trove on the controller side.
Fundamentally, this means isolating the database specific log to the Trove
Guest Agent and Trove elements. 

So, this is a rather long explanation, but that is why we couldn't use 
Trovestack
or the existing tools as is.   There are external considerations such as 
the
state of the database communities and delivery strategies that warranted
a slightly different approach. 

>
> ​
> This approach is nice (having a full guest image with the guest agent 
and
> all that but as you observe, it takes about 2 hours per build and from a
> developers perspective, this is a pain.

It takes 2 hours to build mongodb, because there is no package provided 
by the community for ppc64le.   In general, it takes 25 minutes to 
generate
images and create the datastore.    I expect that the times are roughly 
the
same as with the existing tools. 

>
> I assume that the tool is therefore for production use cases.

Our offering is intended for development use only, but it is a big step
in the direction of production use cases.   The images are built for 
specific
users in a more prescribed manner.

>
> Could the elements that you have developed be used with the existing 
tool
> or is your tool a complete replacement for the one that already 
exists?​

With a little bit of work they may be used with the existing tool.   I am 
not 
planning on replacing existing tools.   In fact, I wasn't planning on 
dropping
the new tool but I would if there is interest.   The new tool is at a 
public
which I noted in my original note.

>
> ​Is there some reason that this tool was not developed in the open, 
using
> the normal open development process?

We were working towards a short release schedule for our new OpenDBaaS 
offering.
It took three months to develop the tool and provide support for 5 
databases,
including multiple versions of some of them.   This is our second step
to engage with the community. 

>
> There are ways to accelerate the compiling of source code and such 
through
> the use of image layers (if you are using DIB). I have elements and code
> that will build most of the trove elements in minutes and I'm waiting to
> see whether there is any point in contributing them to the upstream or 
not,
> at this point. I could show you how to do that, and it would be awesome 
to
> have you build your tool into the build process somehow. That would be a
> win-win for all ...​
>
> I also have commands that setup Nova flavors on a per database basis.
>> These flavors can be customized by the user before they are 
instantiated in
>> the cloud.That's good to hear.
>>
> ​The existing command (trovestack) already does this.​

I didn't see that in the Trovestack.   Does it allow the user to customize
the values.   Can you provide a  provide a link?

>
>
>>
>> I wouldn't expect to drop these commands to the Trove community. Only 
the
>> changes to the Trove-guestagent and elements enabling databases.
>>
>> Here's the list of databases that are currently supported for ppc64le. 
In
>> some cases, I have had to add Xenial support.
>>
>> mariadb 10.1 --- package comes from the community
>> mongodb 3.4 community edition --- source code from the community is
>> compiled as there is no ppc64le package
>> mongodb 3.4 enterprise edition --- package comes from community
>> mysql 5.7 --- package comes from Ubuntu 16.04. percona-xtrabackup-24 is
>> compiled as there is no power support in community
>> postgresql 9.5 --- package comes from Ubuntu 16.04
>> postgresql 9.6 --- package comes from community
>> redis 3.0 --- package comes from Ubuntu 16.04
>> redis 3.2 --- source code from the community is compiled
>>
> ​Why not just contribute the elements to the existing project and have 
the
> current tool use them?​

That is my plan.

>
> ​What of other databases currently supported? galera cluster (pxc),
> vertica, db2, percona, couchdb, and couchbase?

I haven't messed with them.   My changes are hidden under environment
variables so the absence of environment variables should yield the old 
code.

>
>
>>
>> My element related code changes add environment variables that control
>> the database version, the source of the database software (distro,
>> community, enterprise), and logging options. In this way, the logic in 
the
>> element is more generalized and can be made to support multiple 
versions. I
>> also added an environment variable that specifies the name of the 
source
>> package to be downloaded and compiled when that is applicable. I also 
have
>> environment variables to enable database and trove guest agent logging.
>>
> ​Which is nice except that your guest agent code may or may not know 
how
> to deal with that.​

The guest agent and image logic are coordinated. 
 
>
>
>>
>>
>> My trove-guestagent code changes are mostly about not reconfiguring the
>> database with configuration data remotely passed to it from the task
>> manager. My code changes are isolated to the prepare step. This logic 
looks
>> something like: if database version x, add or remove configuration
>> parameter. Note there is relatively very little of this code, ~150 
LOCs, as
>> database configuration parameters rarely change. Note there is already 
some
>> of this DB version specific logic in the trove guest agent, so I am 
just
>> adding more of it. But I believe a better way is not to alter the 
database
>> configuration data when the database is fully configured in a Trove
>> compliant manner via elements. This would be implemented through a new
>> command argument to the Trove Guest agent which says ignore 
configuration
>> changes. It would be set by the elements as well.
>>
> ​Hard to say whether this is a good approach or not with out more 
detail.​
>
>
>>
>>
>> I haven't looked into the scale out configuration data yet, but this 
can
>> be handled differently as required.
>>
>> ​Not sure what you mean.

I was talking about the database cluster configuration code which is a 
future
deliverable for our offering.   We are starting simple and will 
incrementally expand the
offering as time goes by. 

>
> ​
>
>
>> So, are these the sorts of changes that I could add in a maintenance
>> release?
>>
>> ​Maintenance release of what, trove? Are you asking whether these 
changes
> can be made in a stable branch? if yes, which branch? See
> https://docs.openstack.org/project-team-guide/stable-branches.html and
> the section "Appropriate Fixes".​
>
>
>> 1) conditionally disable the dynamic installation and configuration of
>> database software in the trove-guest agent via a new command argument
>> 2) add environment variables to trove elements enabling multiple 
versions
>> of databases to be installed and configured
>> 3) add environment variables to trove elements to enable database and
>> trove-guest agent logging
>> 4) add xenial support to elements
>> 5) compiling source code in elements for missing stack elements for
>> ppc64le
>>
>> These changes are implemented at: https://github.com/open-power-
>> ref-design-toolkit/os-services/tree/master/osa/dbaas/dbimage-builder
>>
> ​OK, so this is a whole new tool, sure you can propose it but how do we
> make sure that it gets integrated into the current testing/gate? If you
> want to drop this then I'd also have to assume that there would be some
> code that would make it part of the current gate, and ideally make it
> generate the images for the current gate. But for that, the fact that 
you
> install the guest agent onto the image is a serious drag.
>
>
> Also, why this wasn't proposed as a blueprint or spec in the upstream so
> it could be developed in the open, I don't quite understand. After all, 
you
> are now showing up with a complete tool and asking that it be included
> which largely runs counter to the whole notion of open development, at
> least as I understand it.​
>

Above, in my original node, I said that I was only planning on dropping 
code
changes  to the element and that I had a small design change to the guest 
agent
that I would pursue with the community.   I am not proposing to drop the 
tool, unless there was interest in that from the community. 

> ​
> Finally, once committed, do you (or IBM) commit to maintaining this code
> and fixing bugs in it should they arise, and upgrade/update it from time 
to
> time?
>

Yes

>
>>
>> Thanks, Luke Browning
>>
>>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170831/addf451d/attachment.html>