Open Stack

Real hard work, man. Thanks.

I think the problem is that most of ops does not know enough about whole 
stuff needed. I do some tiny CI in my company: repackaging ubuntu 
packages with debian-jenkins-glue (plus backported patches 
icehouse->havana), but I can't say I understand all stuff happens under 
the hood of pbr/tox/etc.  And I spend about two weeks trying and 
learning in the process before 1st build was done.

If I can help somehow, I'm ready to do something, but.... What should I 
do, exactly?

On 12/11/2014 06:29 PM, Thomas Goirand wrote:
> Hi,
>
> FYI, I'm the person behind all Debian packages of OpenStack. I'm working
> full-time on packaging OpenStack for Debian, and I've been doing so for
> the last few years (I started to get involved in the Cactus release,
> which is years ago now...). It's been a long time I wanted to reply to
> this thread, and I'm finally finding the time in (the very long trip on)
> the plane today.
>
> tl;dr: The generation of config files is a non-issue which doesn't need
> any tox or pip stuff. Please join me in the packaging effort in Debian
> and let's work together.
>
> Longer version:
>
> Before I start quoting and replying, I'd like to point out a few things.
> For many years, I've been trying to push for working on packages within
> the community, with Debian in mind. It is my view that all the work has
> to happen in Debian, since Ubuntu derives from it, and that this is
> where everything happens anyway.
>
> During this last Juno cycle, I've done absolutely *ALL* of the Python
> module dependency packaging by myself, with the help of nobody. Zero,
> nada, black-hole help from anyone. Even guys from Canonical didn't help,
> they just synched absolutely all of the packages I made into the last
> version of Ubuntu, and that's it. When I complained about it to both
> Mark Shuttleworth and James Page, then the answer was "Thanks!!!".
>
> It's been a long long time that I would like to be able to track trunk
> in my package git repositories, and be able to do gating using packages.
> But given the amount of work that it would represent, and the fact that
> I'd be alone working on all of that, I currently gave up, until the
> company that currently pays for my salary (Mirantis, since last
> September) gets someone else on-board so that we may finally be a team
> of more than a single person.
>
> At the same time, I know that on the Red Hat side, they are only "2.75
> persons" (if you count 0.75 for Alan Pevec, which is also doing some
> Keystone work). I'd also consider this as understaffed.
>
> So, when I have heard from my colleagues that a small group of people
> from the operators mailing list thought it was a good idea to do another
> set of packages of their own, I found it both amusing, irritating, and
> huge time waste.
>
> Why do you think it's a good idea to restart doing the work of
> distributions by yourself? Why not joining a common effort?
>
> I'd be more than happy to have some more contributors. We could open a
> new branch on git.debian.org so that we track stuff from trunk (let's
> call it debian/trunk instead of debian/kilo?). I already have all the
> infrastructure to build using Jenkins, cowbuilder, etc, and
> automatically package repositories for both Debian and Ubuntu. The only
> thing that would be needed would be enough workforce so that we maintain
> this at the same time as trunk, and fix issues as we see them (like
> adding a new {build,runtime}-dependency when there's a new one, upgrade
> versions when needed, and things like that).
>
> Note that the Debian packaging is already happening in the open on
> git.debian.org (ie: service managed by http://alioth.debian.org), and it
> is very easy to add new contributors.
>
> Ok, let me quote what I've seen in this thread now...
>
> On 12/09/2014 10:46 AM, Kris G. Lindgren wrote:
>> Hello all,
>>
>> Since somewhere around icehouse projects have started to stop shipping
>> sample configuration files with their projects and instead create a
>> README-sample.conf.txt that usually contains something like:
>> To generate the sample nova.conf file, run the following
>> command from the top level of the nova directory:
>>
>> tox –egenconfig
>>
>> The problem that I am running into is that tox –egenconfig now requires
>> a newer version of tox than what is available for the build distro:
>> [root at localhost ceilometer-2014.2.1]# tox -egenconfig
>> ERROR: tox version is 1.4.2, required is at least 1.6
> First of all, you do *not* need tox to generate the config files. In
> fact, you should assign yourself to *not* use tox or pip when building a
> package. If you follow what tox -egenconfig (by reading the tox.ini),
> you will find out it does:
>
> bash tools/config/generate_sample.sh -b . -p nova -o etc/nova
>
> that's the only thing you need to run, not tox!
>
>> I know I can do a pip install blah and blindly hope that I get
>> everything installed locally on my build system and that I don’t install
>> something that conflicts with the system but that seems like a less
>> than desirable solution.
> Once you figure out what command tox -egenconfig, and what the
> build-dependencies are, it's actually fine.
>
>> So how are people who are doing packaging
>> handling this? Are you now building a venv per package for tox just to
>> generate a sample configuration?
> Last time I did use a VM to figure out what was doing on with "tox
> -egenconfig". Then once it was done, I just copied the nova.conf file
> into the debian folder of the nova package, though next time, I'll just
> do things properly like I just wrote above: find out the correct
> build-depends, and hard-wire the generation of the /conf.sample file.
>
> Though you also have to realize that a generated config file isn't so
> much useable by our users. That's not enough. What I ship in Debian is
> better than just all directives that tox -egenconfig spits out. The
> default nova.conf in Debian has sensible defaults, comments, etc.
>
>> Shouldn't this be part of the python
>> build/install steps per project?
>>
>> It seems redhat/rdo is simply including a sample configuration that they
>> generated somehow.
>>
>> What are you other packagers doing?
> You may want to use the words "package maintainers" rather than
> "packagers". It is as if you were using "programator" or something...
> Yes, it's a common mistake that everyone within OpenStack seems to
> insist on doing.
>
>> Also, is it just me or does this seem wrong?  Most of the commits that
>> made this change seem to indicate it was because the sample config file
>> was separate from the project and that it was breaking gate when it
>> wasn't kept up to date.  Shouldn't this be something that each project
>> generates?  This seemed to work for 8 previous releases – now its too
>> difficult?  I don’t get the motivations behind this change.
> The motivation is to have always config files which are up-to-date, and
> not just rely on whatever is in the Git, plus issues with libraries (see
> below).
>
> On 12/09/2014 11:23 AM, Fischer, Matt wrote:
>> Not only is the sample
>> config a useful reference, but having it in a git repo allows me to
>> see when new options were added or defaults were changed.
> Yeah, though if you've been doing packages for a long time, you do have
> this information. At least, I do have it for all the last stable releases.
>
> On 12/09/2014 11:23 AM, Fischer, Matt wrote:
>> I don’t really know the reason. This is one of my larger
>> annoyances as an operator
> As a package maintainer, I found that the type of config file from
> projects like neutron is a *way* more annoying. It's not even parse-able
> by default, things that it sets are sometimes the default (and
> therefore, it should be commented-out, but it's not), some options
> simply don't work (ever tried to remove the "gre" driver, just for
> fun?), and loads of this kind of issues. I very much prefer Nova's approach.
>
>> Maybe someone can setup a cron job to do a pull, build the sample
>> configs and post them to git hub every day.
> What would work a way better would be to do that job when doing "python
> setup.py install", though it's also fine to let package maintainer deal
> with it. At least, I'm happy to do it the way I feel right.
>
> On 12/09/2014 12:01 PM, Kris G. Lindgren wrote:
>> [...] sample config and replace it with some tox script. Which may or
>> may not run on your current operating system without going to pip to
>> get newer versions of stuff.
> Excuse my words, but that's bullshit. If your operating system doesn't
> have a version high enough for one of your build-dependency, then this
> should be fixed. It should be your work as a package maintainer. For
> example, if we need a newer version of tox (which is *not* the case, as
> tox isn't really needed for generating config files), then the tox
> package should be updated. It's as simple as that. And if you want this
> to happen, then I would strongly suggest you to get involved in Debian
> and/or Fedora packaging, because that's where things happen.
>
> On 12/09/2014 12:01 PM, Kris G. Lindgren wrote:
>> I see that whats being said is that a  valid sample configuration file
>> depends on the version of the other dependent libraries that are
>> installed on the system as well.  I assume this is talking about
>> different versions of oslo.config and oslo.messaging take different
>> configuration options and as such we can't generate a generic sample
>> config file because we don’t know what versions of dependent libraries
>> you actually have installed.
> Let me give you an example. In most (if not *all* projects) you will
> find some [keystone:auth_token] section. These are directly derived from
> what's in python-keystoneclient. Indeed, the output .conf.sample file
> will depend on that... Now let's say the maintainers of
> python-keystoneclient would like to update something (let's say, update
> a comment on one of the directives, or simply a *new* directive),
> obviously, you want that change to propagate on all projects, but *only*
> if we have a new version of python-keystoneclient.
>
> Obviously, this is a management issue for package maintainers, because
> you have to make sure that you rebuild the packages *after* all
> libraries have been updated. But lucky, considering the fact that we do
> a bunch of RC releases before the final stable versions, that's how it
> practically happens, without even needing to care about it.
>
> Though if we are to build from trunk in a CI/CD way, some kind of
> trigger mechanisms have to happen. Like, touching any option on any lib
> would have to trigger the rebuild of all OpenStack projects and validate
> against the rebuilt packages.
>
> On 12/09/2014 12:01 PM, Kris G. Lindgren wrote:
>> Just because some options change depending
>> on the oslo.config version doesn't mean the configuration options that
>> are specific to that project change as well…
> Correct. The issue isn't in oslo.config, but in all libraries that
> projects are using. I gave above an example with keystoneclient, but the
> more we're splitting things into libraries, the more we have such
> examples to give (another example could be the [database]/connection
> directive...). I didn't take the time to survey it though, but I'm sure
> it must be an increasing number of libraries that are concerned.
>
> On 12/09/2014 12:01 PM, Kris G. Lindgren wrote:
>> The problem that I have with this change is
>> that in order to package a sample configuration I need to basically do
>> the following:
>> 1.) get the current build/commit run
>> 2.) run python build
> No, you *never* need to care about this in the context of a package
> building (at least in Debian, this may happen by itself with the dh auto
> sequencer).
>
>> 3.) strip away the relevant built parts of the package
> I never did have to "strip away" anything. What are you talking about
> exactly here?
>
>> 4.) install on the build machine all the python runtime deps that I
>> am going to need for this package
> You don't need to do that. This is taken care of by your
> build-dependencies (which btw are not trivial to get right). And your
> build environment (cowbuilder in my case, but maybe sbuild on the buildd
> network or many others) will install them automatically. If some
> (build-)dependencies are missing, just fix that...
>
>> 5.) install tox and other packages as needed to generate a sample
>> configuration
> Same as above: this is taken care of by your build-dependencies, you
> should *never* need to install anything manually. And tox isn't needed,
> *ever*.
>
>> 6.) generate sample config
>> 7.) build a new package with the exact same requirements as what I
>> installed in step 4
>> 8.) add sample configuration generated in step 6 to the package.
> Why wouldn't the process of building your package include building a
> sample configuration file? I don't get your reasoning here...
>
>> Then I need to make sure I also package all of the python-versions
>> that was used in step 4, making sure that I don’t have conflicting
>> system level dependencies from other openstack projects.
> Of course all build-dependencies and runtime dependencies need to be
> packaged, and available in the system. That's the basics of packaging,
> no? Making sure this happens is about 90% of my Debian packaging work.
> So far, I haven't seen anyone in the community volunteering to help on
> packaging Python modules. Why not focus on that rather than wasting your
> time on non-issues such as generating sample config files? I'd
> appreciate a lot some help you know...
>
> On 12/09/2014 12:01 PM, Kris G. Lindgren wrote:
>> I don’t think its too much to ask for each project to include a
>> script that will build a venv that includes tox and the other
>> relevant deps to build the sample configuration.
> A venv, seriously?!?
>
> No, it's not that. What need to happen is to have an easy and *OpenStack
> unified way* of building the config files, and preferably with things
> integrated directly in a oslo.config new command line. Not just a black
> magic tox thing, but something documented. But I believe that's already
> the direction that is being taken (did you notice
> /usr/bin/oslo-config-generator ?).
>
> On 12/10/2014 01:02 AM, Kris G. Lindgren wrote:
>> So a few things to point out in order to even get tox -egenconfig I
>> had to update the system packages versions using pip.
> If you are being serious about packaging, then don't use pip. If you
> don't have a tox version high enough (again, that's not the case that
> you need tox to generate nova.conf.sample), then you should package that
> version if it's not available for your system.
>
> On 12/10/2014 01:02 AM, Kris G. Lindgren wrote:
>> Since we have other python
>> packages using virtualenv I have no idea if the updated venvironment
>> package is going to break those systems or not.  So the included
>> script/command is already a barrier to getting a sample config.
> Again, you don't need tox.
>
>> If the configuration is so dynamic that getting a different version
>> of oslo.config could generate a sample configuration that wont work
>> on my system then how am I suppose to deal with:
>> Tox installed version:
>> oslo.config-1.5.0
> Which is why you should never use tox/pip when building packages.
>
>> Also python-libvrit failed to build because I don¹t have libvrit
>> installed on this system.  So am I to assume that there are no
>> libvrit options (which we both know is false)?
>> Now I can get a example config - that wont work with my system - per
>> what everyone else has been saying.  Also, at what point would the
>> average user just say "F it"? - because at the point I feel like if I
>> was an average user - I would be there right now.
> Yeah, it's annoying, but deal-able with.
>
> On 12/10/2014 01:25 AM, Michael Dorman wrote:
>> Well I think we can all agree this is an irritation.  But how are
>> others  actually dealing with this problem?  (Maybe it’s less
>> complicated in Ubuntu.)
> Since exactly when Ubuntu people started caring about the configuration
> files that they ship? Last time I checked, Nova doesn't install a
> workable nova.conf by default, it only has a few directives and that's
> about it.
>
>> The sense I get is that most people using Anvil, or other custom-ish
>> packaging tools, are also running config management which handles
>> generating the config files, anyway.  So you don’t so much care about
>> the  contents of the config file shipped with the package.
> Yeah, it has been the excuse for years so that 1/ upstream project
> doesn't care 2/ downstream distribution don't care 3/ users give-up on
> installing by hand. I DON'T BUY THIS CRAP! And as a package maintainer,
> I strongly believe that it's my duty to make packages at least a little
> bit useable by default.
>
>> Is that accurate for most people?  Or are folks doing some other
>> magic to get a good config file in the packages?
> No magic. Only hard work can make it happen (unfortunately, currently
> limited by the time I have available, which isn't much given the amount
> of work OpenStack packaging represents). The way to do things is to get
> OpenStack installed (by hand, without any helper, using the package
> defaults), then make sure the package ships with sensible defaults that
> do work. Which is why I've been working on the "openstack-deploy"
> package and scripts in Debian. Contributions are welcome there too!
> Also, reading the install-guide and making sure what it recommends as
> options in config files are pre-wired in the packages may help as well.
>
> Cheers,
>
> Thomas Goirand (zigo)
>
> P.S: Matt Fischer, could you *PLEASE* stop posting with a footer that
> makes you look like a fool?
>
> On 12/09/2014 11:39 AM, Fischer, Matt wrote:
>> This E-mail and any of its attachments may contain Time Warner Cable
>> proprietary information, which is privileged, confidential, or
>> subject to copyright belonging to Time Warner Cable. This E-mail is
>> intended solely for the use of the individual or entity to which it
>> is addressed. If you are not the intended recipient of this E-mail,
>> you are hereby notified that any dissemination, distribution,
>> copying, or action taken in relation to the contents of and
>> attachments to this E-mail is strictly prohibited and may be
>> unlawful. If you have received this E-mail in error, please notify
>> the sender immediately and permanently delete the original and any
>> copy of this E-mail and any printout.
> I am hereby notifying you: you're sending emails to a public list.
> Therefore it *WILL* be reproduced, distributed, quoted, indexed, etc.,
> And I challenge you (or your company) to dare threatening me again of
> doing unlawful miss-handling of contents which you are willingly sending
> to a public list... Should my lawyer get in touch with yours? :)
>
>
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators