Open Stack

Please see inline.

TL;DR - Thanks for the bash script and a kinda round about way of how you
got your configs for Debian.  You are missing the point that the packaging
tooling that we want to build - should make your life as a the only
package maintainer for Debian easier.
____________________________________________
 
Kris Lindgren
Senior Linux Systems Engineer
GoDaddy, LLC.


On 12/11/14, 9:29 AM, "Thomas Goirand" <zigo at debian.org> wrote:

>Hi,
>
>FYI, I'm the person behind all Debian packages of OpenStack. I'm working
>full-time on packaging OpenStack for Debian, and I've been doing so for
>the last few years (I started to get involved in the Cactus release,
>which is years ago now...). It's been a long time I wanted to reply to
>this thread, and I'm finally finding the time in (the very long trip on)
>the plane today.
>
>tl;dr: The generation of config files is a non-issue which doesn't need
>any tox or pip stuff. Please join me in the packaging effort in Debian
>and let's work together.
>
>Longer version:
>
>Before I start quoting and replying, I'd like to point out a few things.
>For many years, I've been trying to push for working on packages within
>the community, with Debian in mind. It is my view that all the work has
>to happen in Debian, since Ubuntu derives from it, and that this is
>where everything happens anyway.
>
>During this last Juno cycle, I've done absolutely *ALL* of the Python
>module dependency packaging by myself, with the help of nobody. Zero,
>nada, black-hole help from anyone. Even guys from Canonical didn't help,
>they just synched absolutely all of the packages I made into the last
>version of Ubuntu, and that's it. When I complained about it to both
>Mark Shuttleworth and James Page, then the answer was "Thanks!!!".
>
>It's been a long long time that I would like to be able to track trunk
>in my package git repositories, and be able to do gating using packages.
>But given the amount of work that it would represent, and the fact that
>I'd be alone working on all of that, I currently gave up, until the
>company that currently pays for my salary (Mirantis, since last
>September) gets someone else on-board so that we may finally be a team
>of more than a single person.
>
>At the same time, I know that on the Red Hat side, they are only "2.75
>persons" (if you count 0.75 for Alan Pevec, which is also doing some
>Keystone work). I'd also consider this as understaffed.
>
>So, when I have heard from my colleagues that a small group of people
>from the operators mailing list thought it was a good idea to do another
>set of packages of their own, I found it both amusing, irritating, and
>huge time waste.
>
>Why do you think it's a good idea to restart doing the work of
>distributions by yourself? Why not joining a common effort?

Well for a lot of reasons.  Some of us carry our own patch sets - some of
us want to run certain services on trunk.  But mainly the bigger push is
that if we come up with a standard set of openstack packages - then we can
build a standard set of tooling around dealing/installing those packages.
The point of this is that we are trying to build that common effort and
not leave it up to the distro's to solve all the problems?
>
>I'd be more than happy to have some more contributors. We could open a
>new branch on git.debian.org so that we track stuff from trunk (let's
>call it debian/trunk instead of debian/kilo?). I already have all the
>infrastructure to build using Jenkins, cowbuilder, etc, and
>automatically package repositories for both Debian and Ubuntu. The only
>thing that would be needed would be enough workforce so that we maintain
>this at the same time as trunk, and fix issues as we see them (like
>adding a new {build,runtime}-dependency when there's a new one, upgrade
>versions when needed, and things like that).
>
>Note that the Debian packaging is already happening in the open on
>git.debian.org (ie: service managed by http://alioth.debian.org), and it
>is very easy to add new contributors.

That’s cool.  I don't use Debian/Ubuntu.

>
>Ok, let me quote what I've seen in this thread now...
>
>On 12/09/2014 10:46 AM, Kris G. Lindgren wrote:
>> Hello all,
>> 
>> Since somewhere around icehouse projects have started to stop shipping
>> sample configuration files with their projects and instead create a
>> README-sample.conf.txt that usually contains something like:
>> To generate the sample nova.conf file, run the following
>> command from the top level of the nova directory:
>> 
>> tox ­egenconfig
>> 
>> The problem that I am running into is that tox ­egenconfig now requires
>> a newer version of tox than what is available for the build distro:
>> [root at localhost ceilometer-2014.2.1]# tox -egenconfig
>> ERROR: tox version is 1.4.2, required is at least 1.6
>
>First of all, you do *not* need tox to generate the config files. In
>fact, you should assign yourself to *not* use tox or pip when building a
>package. If you follow what tox -egenconfig (by reading the tox.ini),
>you will find out it does:
>
>bash tools/config/generate_sample.sh -b . -p nova -o etc/nova
>
>that's the only thing you need to run, not tox!
>
>> I know I can do a pip install blah and blindly hope that I get
>> everything installed locally on my build system and that I don’t install
>> something that conflicts with the system but that seems like a less
>> than desirable solution.
>
>Once you figure out what command tox -egenconfig, and what the
>build-dependencies are, it's actually fine.
>
>> So how are people who are doing packaging
>> handling this? Are you now building a venv per package for tox just to
>> generate a sample configuration?
>
>Last time I did use a VM to figure out what was doing on with "tox
>-egenconfig". Then once it was done, I just copied the nova.conf file
>into the debian folder of the nova package, though next time, I'll just
>do things properly like I just wrote above: find out the correct
>build-depends, and hard-wire the generation of the /conf.sample file.
>
>Though you also have to realize that a generated config file isn't so
>much useable by our users. That's not enough. What I ship in Debian is
>better than just all directives that tox -egenconfig spits out. The
>default nova.conf in Debian has sensible defaults, comments, etc.
>
>> Shouldn't this be part of the python
>> build/install steps per project?
>> 
>> It seems redhat/rdo is simply including a sample configuration that they
>> generated somehow.
>> 
>> What are you other packagers doing?
>
>You may want to use the words "package maintainers" rather than
>"packagers". It is as if you were using "programator" or something...
>Yes, it's a common mistake that everyone within OpenStack seems to
>insist on doing.

Seriously?  This added what to the discussion?

>
>> Also, is it just me or does this seem wrong?  Most of the commits that
>> made this change seem to indicate it was because the sample config file
>> was separate from the project and that it was breaking gate when it
>> wasn't kept up to date.  Shouldn't this be something that each project
>> generates?  This seemed to work for 8 previous releases ­ now its too
>> difficult?  I don’t get the motivations behind this change.
>
>The motivation is to have always config files which are up-to-date, and
>not just rely on whatever is in the Git, plus issues with libraries (see
>below).
>
>On 12/09/2014 11:23 AM, Fischer, Matt wrote:
>> Not only is the sample
>> config a useful reference, but having it in a git repo allows me to
>> see when new options were added or defaults were changed.
>
>Yeah, though if you've been doing packages for a long time, you do have
>this information. At least, I do have it for all the last stable releases.

Great - some of us also are busy actually running clouds here...

>
>On 12/09/2014 11:23 AM, Fischer, Matt wrote:
>> I don’t really know the reason. This is one of my larger
>> annoyances as an operator
>
>As a package maintainer, I found that the type of config file from
>projects like neutron is a *way* more annoying. It's not even parse-able
>by default, things that it sets are sometimes the default (and
>therefore, it should be commented-out, but it's not), some options
>simply don't work (ever tried to remove the "gre" driver, just for
>fun?), and loads of this kind of issues. I very much prefer Nova's
>approach.
>
>> Maybe someone can setup a cron job to do a pull, build the sample
>> configs and post them to git hub every day.
>
>What would work a way better would be to do that job when doing "python
>setup.py install", though it's also fine to let package maintainer deal
>with it. At least, I'm happy to do it the way I feel right.
>
>On 12/09/2014 12:01 PM, Kris G. Lindgren wrote:
>> [...] sample config and replace it with some tox script. Which may or
>> may not run on your current operating system without going to pip to
>> get newer versions of stuff.
>
>Excuse my words, but that's bullshit. If your operating system doesn't
>have a version high enough for one of your build-dependency, then this
>should be fixed. It should be your work as a package maintainer. For
>example, if we need a newer version of tox (which is *not* the case, as
>tox isn't really needed for generating config files), then the tox
>package should be updated. It's as simple as that. And if you want this
>to happen, then I would strongly suggest you to get involved in Debian
>and/or Fedora packaging, because that's where things happen.

I guess from my point of view - not providing a sample config file for
your project is bullshit - unless said project can work without any config
file at all (even then its still bullshit).  Providing a script to do it -
ok that’s live-able.  Providing documentation to use tox that installs all
the projects deps into a venv then generates a script from those deps
which don't match what on the system.  That’s bullshit as well.

>
>On 12/09/2014 12:01 PM, Kris G. Lindgren wrote:
>> I see that whats being said is that a  valid sample configuration file
>> depends on the version of the other dependent libraries that are
>> installed on the system as well.  I assume this is talking about
>> different versions of oslo.config and oslo.messaging take different
>> configuration options and as such we can't generate a generic sample
>> config file because we don’t know what versions of dependent libraries
>> you actually have installed.
>
>Let me give you an example. In most (if not *all* projects) you will
>find some [keystone:auth_token] section. These are directly derived from
>what's in python-keystoneclient. Indeed, the output .conf.sample file
>will depend on that... Now let's say the maintainers of
>python-keystoneclient would like to update something (let's say, update
>a comment on one of the directives, or simply a *new* directive),
>obviously, you want that change to propagate on all projects, but *only*
>if we have a new version of python-keystoneclient.

I guess I can get that for staying on trunk.  But when openstack does a
stable release... Shouldn't the client versions be pretty well defined?
And not just >= 1.4 - but should be capped to some specific version?

>
>Obviously, this is a management issue for package maintainers, because
>you have to make sure that you rebuild the packages *after* all
>libraries have been updated. But lucky, considering the fact that we do
>a bunch of RC releases before the final stable versions, that's how it
>practically happens, without even needing to care about it.
>
>Though if we are to build from trunk in a CI/CD way, some kind of
>trigger mechanisms have to happen. Like, touching any option on any lib
>would have to trigger the rebuild of all OpenStack projects and validate
>against the rebuilt packages.
>
>On 12/09/2014 12:01 PM, Kris G. Lindgren wrote:
>> Just because some options change depending
>> on the oslo.config version doesn't mean the configuration options that
>> are specific to that project change as well…
>
>Correct. The issue isn't in oslo.config, but in all libraries that
>projects are using. I gave above an example with keystoneclient, but the
>more we're splitting things into libraries, the more we have such
>examples to give (another example could be the [database]/connection
>directive...). I didn't take the time to survey it though, but I'm sure
>it must be an increasing number of libraries that are concerned.
>
>On 12/09/2014 12:01 PM, Kris G. Lindgren wrote:
>> The problem that I have with this change is
>> that in order to package a sample configuration I need to basically do
>> the following:
>> 1.) get the current build/commit run
>> 2.) run python build
>
>No, you *never* need to care about this in the context of a package
>building (at least in Debian, this may happen by itself with the dh auto
>sequencer).

So... You don't do something like:
https://github.com/stackforge/anvil/blob/master/conf/templates/packaging/sp
ecs/openstack-nova.spec#L393-L394
That’s interesting....
>
>> 3.) strip away the relevant built parts of the package
>
>I never did have to "strip away" anything. What are you talking about
>exactly here?

Should be irrelevant... Sorry...
Something like: 
https://github.com/stackforge/anvil/blob/master/conf/templates/packaging/sp
ecs/openstack-nova.spec#L407

>
>> 4.) install on the build machine all the python runtime deps that I
>> am going to need for this package
>
>You don't need to do that. This is taken care of by your
>build-dependencies (which btw are not trivial to get right). And your
>build environment (cowbuilder in my case, but maybe sbuild on the buildd
>network or many others) will install them automatically. If some
>(build-)dependencies are missing, just fix that...

Anvil figures out the build dependencies for me - it also figures out if
those deps are satisfiable via yum, if not downloads the version from pip
and packages that.  So for me requirements is pretty trivial.  But to this
point and your previous one where you ran everything on a vm to figure out
WTF it was doing.  The point is now to get a sample config I need to as
part of the package build process - build/install all the *RUNTIME* deps
for that service on the box so I can generate a config file.

What I am getting from you is that you basically install all the runtime
deps for the project on your build machine and then build the config using
the bash script.

>
>> 5.) install tox and other packages as needed to generate a sample
>> configuration
>
>Same as above: this is taken care of by your build-dependencies, you
>should *never* need to install anything manually. And tox isn't needed,
>*ever*.
>
>> 6.) generate sample config
>> 7.) build a new package with the exact same requirements as what I
>> installed in step 4
>> 8.) add sample configuration generated in step 6 to the package.
>
>Why wouldn't the process of building your package include building a
>sample configuration file? I don't get your reasoning here...

Getting to the point that what tox installs and what are available on the
system are different. So the only way to get a valid config is to either
use the same package versions that tox does or to duplicate what "tox" is
suppose to do for you.  Which sounds like you duplicate what tox does for
you to avoid that mess.

>
>> Then I need to make sure I also package all of the python-versions
>> that was used in step 4, making sure that I don’t have conflicting
>> system level dependencies from other openstack projects.
>
>Of course all build-dependencies and runtime dependencies need to be
>packaged, and available in the system. That's the basics of packaging,
>no? Making sure this happens is about 90% of my Debian packaging work.
>So far, I haven't seen anyone in the community volunteering to help on
>packaging Python modules. Why not focus on that rather than wasting your
>time on non-issues such as generating sample config files? I'd
>appreciate a lot some help you know...

That is what this effort is for? Coming up with tooling to package
openstack and its python modules and if we can't simply include a sample
config like we have done for the past 3 years, then the tooling (which we
are trying to consolidate) should help us here.

>
>On 12/09/2014 12:01 PM, Kris G. Lindgren wrote:
>> I don’t think its too much to ask for each project to include a
>> script that will build a venv that includes tox and the other
>> relevant deps to build the sample configuration.
>
>A venv, seriously?!?
>
>No, it's not that. What need to happen is to have an easy and *OpenStack
>unified way* of building the config files, and preferably with things
>integrated directly in a oslo.config new command line. Not just a black
>magic tox thing, but something documented. But I believe that's already
>the direction that is being taken (did you notice
>/usr/bin/oslo-config-generator ?).

Or projects could maintain a configuration file...  But I guess if
everyone uses the bash script for config generation then I could work with
that...
I am also agreeing that the process should be uniform.

>
>On 12/10/2014 01:02 AM, Kris G. Lindgren wrote:
>> So a few things to point out in order to even get tox -egenconfig I
>> had to update the system packages versions using pip.
>
>If you are being serious about packaging, then don't use pip. If you
>don't have a tox version high enough (again, that's not the case that
>you need tox to generate nova.conf.sample), then you should package that
>version if it's not available for your system.
>
>On 12/10/2014 01:02 AM, Kris G. Lindgren wrote:
>> Since we have other python
>> packages using virtualenv I have no idea if the updated venvironment
>> package is going to break those systems or not.  So the included
>> script/command is already a barrier to getting a sample config.
>
>Again, you don't need tox.

Thanks for driving the point home - I didn't get it the first 10 times.
>
>> If the configuration is so dynamic that getting a different version
>> of oslo.config could generate a sample configuration that wont work
>> on my system then how am I suppose to deal with:
>> Tox installed version:
>> oslo.config-1.5.0
>
>Which is why you should never use tox/pip when building packages.
Thanks for agreeing with my point.
>
>> Also python-libvrit failed to build because I don¹t have libvrit
>> installed on this system.  So am I to assume that there are no
>> libvrit options (which we both know is false)?
>> Now I can get a example config - that wont work with my system - per
>> what everyone else has been saying.  Also, at what point would the
>> average user just say "F it"? - because at the point I feel like if I
>> was an average user - I would be there right now.
>
>Yeah, it's annoying, but deal-able with.

I guess the average user uses upstream packages so they never see this.
>
>On 12/10/2014 01:25 AM, Michael Dorman wrote:
>> Well I think we can all agree this is an irritation.  But how are
>> others  actually dealing with this problem?  (Maybe it’s less
>> complicated in Ubuntu.)
>
>Since exactly when Ubuntu people started caring about the configuration
>files that they ship? Last time I checked, Nova doesn't install a
>workable nova.conf by default, it only has a few directives and that's
>about it.
>
>> The sense I get is that most people using Anvil, or other custom-ish
>> packaging tools, are also running config management which handles
>> generating the config files, anyway.  So you don’t so much care about
>> the  contents of the config file shipped with the package.
>
>Yeah, it has been the excuse for years so that 1/ upstream project
>doesn't care 2/ downstream distribution don't care 3/ users give-up on
>installing by hand. I DON'T BUY THIS CRAP! And as a package maintainer,
>I strongly believe that it's my duty to make packages at least a little
>bit useable by default.
>
>> Is that accurate for most people?  Or are folks doing some other
>> magic to get a good config file in the packages?
>
>No magic. Only hard work can make it happen (unfortunately, currently
>limited by the time I have available, which isn't much given the amount
>of work OpenStack packaging represents). The way to do things is to get
>OpenStack installed (by hand, without any helper, using the package
>defaults), then make sure the package ships with sensible defaults that
>do work. Which is why I've been working on the "openstack-deploy"
>package and scripts in Debian. Contributions are welcome there too!
>Also, reading the install-guide and making sure what it recommends as
>options in config files are pre-wired in the packages may help as well.
>
>Cheers,
>
>Thomas Goirand (zigo)
>
>P.S: Matt Fischer, could you *PLEASE* stop posting with a footer that
>makes you look like a fool?

I dunno who looks like a fool - but from my view its not Matt Fischer.

>
>On 12/09/2014 11:39 AM, Fischer, Matt wrote:
>> This E-mail and any of its attachments may contain Time Warner Cable
>> proprietary information, which is privileged, confidential, or
>> subject to copyright belonging to Time Warner Cable. This E-mail is
>> intended solely for the use of the individual or entity to which it
>> is addressed. If you are not the intended recipient of this E-mail,
>> you are hereby notified that any dissemination, distribution,
>> copying, or action taken in relation to the contents of and
>> attachments to this E-mail is strictly prohibited and may be
>> unlawful. If you have received this E-mail in error, please notify
>> the sender immediately and permanently delete the original and any
>> copy of this E-mail and any printout.
>
>I am hereby notifying you: you're sending emails to a public list.
>Therefore it *WILL* be reproduced, distributed, quoted, indexed, etc.,
>And I challenge you (or your company) to dare threatening me again of
>doing unlawful miss-handling of contents which you are willingly sending
>to a public list... Should my lawyer get in touch with yours? :)

That really added a lot to this discussion - thanks.
>
>
>_______________________________________________
>OpenStack-operators mailing list
>OpenStack-operators at lists.openstack.org
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators