Open Stack

On 06/12/2016 10:45 PM, Monty Taylor wrote:
> On 06/11/2016 10:13 PM, Robert Collins wrote:
>> On 12 June 2016 at 11:31, Thomas Goirand <zigo at debian.org> wrote:
>>> On 06/11/2016 05:02 AM, Robert Collins wrote:
>> ...
>>> Then keystone.conf ends up installed in /usr/etc/keystone when setup.py
>>> install is used. With this configuration, it would be installed
>>> correctly within a venv.
>>
>> Except that noone uses 'setup.py install' - *except* for rpm and dpkg
>> packagers. Everyone else uses pip, and pip won't pass new options.
>> Worse, pip creates wheels and then installs the wheels, and until PEP
>> 491 [1] is finalised and implemented, wheels won't change their
>> behaviour.
>>
>>> If instead we have:
>>> [files]
>>> data_files =
>>>     /etc/keystone/keystone.conf = etc/keystone.conf.sample
>>>
>>> [ notice the leading / for the destination... ]
>>>
>>> Then keystone.conf gets installed on the root of the filesystem instead
>>> of properly be installed within the venv. Though it's installed properly
>>> for distributions in /etc/keystone.
>>>
>>> To address this problem, Julien Danjou attempted to add a --sysconfdir
>>> option in PBR. Though it was rejected by Robert, who wrote in the code
>>> review that it should be addressed within setuptools or disutils.
>>
>> Not should. Must be. A local change to pbr won't solve the problem,
>> except for the very narrow case of package creators (who already have
>> excellent tools - its a single one-line file to fix it on Debian, and
>> I presume roughly the same on RPM distributions). It certainly won't
>> solve it for virtual envs.
>>
>>> If I got the above facts wrong, feel free to let everyone know. Now, my
>>> reply to Robert...
>>> === End of context ===
>>>
>>> Robert,
>>>
>>> I do not agree that we *must* wait until politics are discussed and a
>>
>> I'm going to quote what I said back in February - 4 months ago: "Ok,
>> so here's where I got to:
>>
>> - wheel doesn't support or understand /etc per se
>> - setuptools and distutils disagree on the meaning of data files paths
>> - wheel takes the distutils value today
>> - pep 491, which seems moribund, is an attempt to explicitly support
>> sysconf dir's like /etc, and thats what is needed in general.
>>
>> I think the way forward to get this available as a sensible,
>> supportable feature is to discuss it on the distutils-sig list, and
>> that community can help figure out a design, including the backwards
>> compat implications."
>>
>> I don't -2 lightly, and I did considerable research into this patch
>> before leaving the -2 in place.
>>
>> With regards to politics and consensus in the upstream packaging
>> community - well, it would move faster if folk who are distro folk and
>> thus want to see the mess sorted were collaborating there: its been 4
>> months since Julien's original patch and no discussion has taken
>> place, nor has anyone volunteered in that community to do the work.
>> Its not super hard, and I'd be happy to guide anyone interested in
>> reviving PEP 491 (or drafting an alternative). What I'm not interested
>> in doing is adding an ineffective hack to pbr which we'll have to
>> maintain for an extended period, which isn't actually in pbr's domain
>> (pbr exists to prevent duplication amongst OpenStack projects and
>> provide solid and sane policy, not to replace wheel and setuptools).
>> I'm sorry if that sounds like a hard line, but we had an ongoing
>> series of bugs in pbr - the gift that keeps on giving - from the
>> previous attempts to work around defects upstream of pbr rather than
>> fixing those defects directly, which is a much better strategy : less
>> cost, less technical debt, benefits the entire community, gets better
>> outcomes.
> 
> I'm going to have to second what Robert says here.
> 
> Also - I would like to add the following.
> 
> Having pip or setup.py install install a config file is at best a
> strange thing to want, as it does not have the mechanisms to deal with
> templating in values, or dealing with conflicts with locally changed
> file values on upgrades.
> 
> Consider the case where a person does "pip install keystone" to install
> keystone.
> 
> If that command installs keystone.conf into /etc/keystone - there is
> _NO_ way that that config file will have any useful values. The user
> will then edit the file to add the values into the file that need to be
> added. Then, the user will, at a later point in time do "pip install -U
> keystone" to upgrade the keystone installation.
> 
> What does pip do? Does it overwrite /etc/keystone/keystone.conf? What
> about the user's values? Does it NOT overwrite
> /etc/keystone/keystone.conf? Why not?
> 
> This is, btw, why dpkg and rpm and friends exist. It's their job.
> 
> Now, you might argue, there are files, such as the paste config files
> which are configs that the software needs to install to be useful.
> 
> I will call hogwash on that. That we install files such as that which
> are actually code has been a deployer nightmare since day one, and the
> solution is not to make pip smarter, it's to stop installing such files.
> 
> NOW - if it's quite literally a file that just exists better as a file
> and not direct python code - that's fine - just put it into the package
> itself and then load it either by relative paths from the package or via
> pkg_resources. For instance, one could put paste.conf into
> /usr/local/lib/python2.7/site-packages/keystone/paste.ini instead of
> into etc.
> 
> Which is my way of saying:
> 
> - If it's a file that one expects the user to edit, having pip/setup.py
> install it is a bad idea
> - If it's a file that one does NOT expect the user to edit, put it into
> the package.
> 
> There is, of course, a class of files that meet neither condition:
> manpages. Yup. Manpage handling is bong. But we already have as much
> special casing in pbr as I think is reasonable to do the best we can
> with manpages. That best is crappy anyway, which brings us back to
> Robert's point about fixing things in distutils/setuptools.
> 
> But seriously - figuring out _why_ we want pip to be able to deal with
> config files I think would be much better served by reducing the amount
> of boilerplate config we expect users to plop down. No amount of code in
> pbr is going to help that problem.
> 
> Monty

I proposed to use the --install-layout=deb parameter, so that *ONLY*
distros would be affected. But this was also rejected. Could you explain
why it would be a bad idea?

Also, I fail to understand why it's ok to have "python setup.py install"
to install things in etc/ in the case of a virtualenv, but you think
it's forbidden to fix the silly /usr/etc path when called by packaging.

Here, I'm not trying to add a feature, just address a bug. Because
that's really what we are dealing with here (unless someone is insane
enough to believe that installing stuff in /usr/etc is ok, somehow...).

Also, Monty, I read you writing that nobody should ever attempt to use
pbr to install config files in etc, not even using virtualenv. However,
it's already a common practice in OpenStack. So like it or not, we
should deal with it.

As for the answer of Robert, yes, please explain to me how I can attempt
to fix it in setuptools / pip / whatever. In the long run, I agree it
should be nice, and that's the way to go.

Though probably you're maybe being misslead when writing:

On 06/12/2016 04:13 AM, Robert Collins wrote:
> Except that noone uses 'setup.py install' - *except* for rpm and dpkg
> packagers. Everyone else uses pip, and pip won't pass new options.

In terms of actual code having to deal with this issue, the folks using
pip count as 1 type of setup. Package maintainers count as 5 currently
(Debian, Ubuntu, RDO, SuSE and Gento). It doesn't mater how many *users*
these groups of packaging system have. So IMO, trying to fix the work of
5 type of programming vs a single one (even if that single one has more
developers using it) isn't that silly.

If on top, we're protecting it with --install-layout=deb, or reading
anything else that would work for other distro (an env var?) then I'm
sure it's going to be safe for everyone, and I do believe that this is
the pragmatic thing to do, until upper level of tooling can be fixed.

Your thoughts?

Cheers,

Thomas Goirand (zigo)