Open Stack

On 7/14/15, 10:41, "Monty Taylor" <mordred at inaugust.com> wrote:

>On 07/14/2015 11:08 AM, Ian Cordasco wrote:
>> 
>> 
>> On 7/13/15, 16:19, "Joshua Harlow" <harlowja at outlook.com> wrote:
>> 
>>> Ian Cordasco wrote:
>>>>
>>>> On 7/13/15, 15:09, "Dave Walker"<email at daviey.com>  wrote:
>>>>
>>>>> On 13 Jul 2015 8:52 pm, "Ian Cordasco"<ian.cordasco at rackspace.com>
>>>>> wrote:
>>>>>> On 7/13/15, 03:38, "Thierry Carrez"<thierry at openstack.org>  wrote:
>>>>> <SNIP>
>>>>>>> By "counter-productive", I meant: likely to generate more confusion
>>>>>> than
>>>>>>> clarity. If you provide an epoch in the version and it doesn't
>>>>>>>match
>>>>>>> downstream packagers ones, it's hard to rely on it.
>>>>>> I see what you mean now. The thing is that for Debian/Fedora the
>>>>>>epoch
>>>>>> syntax is different from PEP 440
>>>>>>
>>>>>> For them it's
>>>>>>
>>>>>> [distro-epoch]:[upstream-version][otherstuff]
>>>>>>
>>>>>> PEP 440 epochs are separated by a !, so let's say $(distro) has an
>>>>>> epoch
>>>>>> value of "1" and we choose "2", for glance the version would look
>>>>>>ugly
>>>>>> but
>>>>>> would be:
>>>>>>
>>>>>> 1:2!11.0.0
>>>>>>
>>>>> <SNIP>
>>>>> This would be a problem for at least Ubuntu and Debian as the version
>>>>> string is specifically not allowed to contain a '!'.
>>>>> "The upstream_version may contain only alphanumerics and the
>>>>> characters .
>>>>> +- : ~ (full stop, plus, hyphen, colon, tilde) and should start with
>>>>>a
>>>>> digit." [0]
>>>>
>>>> Thanks for the input Dave. I didn't see that part since I was
>>>> specifically
>>>> looking for how epochs are denoted. I still am quite certain that we
>>>> have
>>>> no choice but to use the proper versioning tools upstream though and
>>>> that
>>>> means using ! to indicate the epoch in our version strings because
>>>>these
>>>> are fundamentally Python packages.
>>>
>>> +1 we produce python versions, and people use them (whether they are
>>> uploaded to pypi or not) so we should try to do the right thing here no
>>> matter what. If downstream distro (rh, ubuntu, other...) wants to
>>> convert that into its local epoch scheme that's cool (and I would
>>>expect
>>> them to do that in the correct manner that is appropriate for there
>>> packages).
>> 
>> Since the mailing list reaches the largest number of people, I thought
>>I'd
>> continue some of this discussion here.
>> 
>> Yesterday some further discussion happened in #openstack-infra (on IRC),
>> and the crux of the argument against Epochs seems to be that they make
>>it
>> hard to deal with archives on the command line, e.g., you'd have an
>> archive generated that looks like:
>> 
>>     nova-1!12.0.0.tar.gz
>> 
>> Perhaps I'm misunderstanding, but any relatively modern shell that
>>allows
>> for tab completion will properly tab-complete that when using the
>>filename.
>> 
>> Another argument is that this is something ugly that we'll have to live
>> with for the rest of OpenStack's life. Without it, however, we'll have
>>to
>> live with 12.0.0 < (the last four years of version numbers) as a sorting
>> mechanism as far as Python packaging is concerned.
>> 
>> There's also the argument that (paraphrasing) this isn't really a
>>problem
>> for people using pip because you can do `pip install -U nova==12.0.0`
>>once
>> and you're done. Except that if you have a local package index and
>>you're
>> support reproducible builds of multiple versions, say, Juno (2014.2.x),
>> Kilo (2015.1.y), Liberty (your version number depends on project but is
>> strictly less than both Juno and Kilo), and M. If you upgrade a project
>> from Liberty to M, you have to still use `pip install -U nova==13.0.0`
>> because `pip install -U nova` will install Kilo because neither Liberty
>> nor M are using an epoch. We're effectively saying "if you support more
>> than one release at a time using the python packaging ecosystem, you're
>>a
>> second class citizen in OpenStack".
>> 
>> In other words, this will be an ongoing problem. Epochs are clearly a
>> necessity, otherwise our downstream distributors wouldn't be using them
>> and the PEP 440 authors wouldn't have included them. Epochs were
>>designed
>> exactly for this situation, and while they're ugly, they're the
>>absolutely
>> correct approach here.
>> 
>> As a sidenote, in case people are wondering how epochs even work with
>>pip,
>> 2015.1.0 has an implicit epoch of 0, e.g., the version can be
>>equivalently
>> written as 0!2015.1.0. 1!12.0.0 would sort higher because 1 has a higher
>> value than the implicit 0. 1!13.0.0 then defers to the rest of the
>>version
>> string because they belong to the same epoch.
>
>Those are all only physical manifestations of the concern I shared, not
>the concerns themselves.
>
>Here are my actual thoughts and concerns:
>
>pip is not a downstream distribution. It is not as rich as dpkg or rpm.
>The issues that they solve by using epochs are not the same problem
>space pip exists in.

So, pip may not have as many features as dpkg or rpm. That said, it is
(apparently) being chosen increasingly by operators over dpkg and rpm.
There was a lot of talk at the last OPS mid cycle (in Philadelphia) about
deploying from source (and using pip) from what I've heard about that
midcycle. Granted, it's a mid cycle so not every operator will be able to
attend (just like not every operator is able to attend the Summit). There
was still a consensus to move towards installing from source though.

>In downstream distros, such as debian and fedora, the work of and
>complexity of epochs are packager concerns. End users are not generally
>exposed to them, because a release of a Linux distro has one version of
>a given thing at any time. So the problem space as a user is "I want to
>install the version of the software that exists in the current release
>of my distro contains" In that problem space, epochs provide a tool when
>an upstream (like us) does something that  conflicts with the version
>sorting rules in the distro.

Sure, and in a way we are packaging it (even just by offering source
tarballs). Anyone trying to sort them naively (without knowing anything
about this versioning change) will only be able to understand that nova's
12.0.0 is newer than 2015.1.0 by examining when the files were generated
or tags created.

>With pip/pypi, all of the versions are there, and people interact with
>versions more directly. That means that having human understandable
>versions is more important.
>
>An epoch, being a tool designed for package maintainers to solve package
>maintainer problems, is not a tool designed to solve ease of
>communication issues for regular users. Most regular users immediately
>grasp that 2.0.0 > 1.0.0. Most regular users do NOT intuitively grasp
>that 1!1.0.0 > 0!2.0.0 - nor why the heck there is a ! in the version.

Fun fact, pip wanted to use : instead of ! but couldn't because : can't be
part of a path on Windows.

>We're moving to semver versions to better align with understandable
>version numbering as people regularly use it. Yes, this is going to
>cause a short term grief for a set of people. However, it is a VERY
>simple problem for those people to overcome, if it is actually a problem
>for them at all.
>
>To be clear, since we have never published the servers to PyPI, it means
>that there are the following ways in which people can install the servers:
>
>a) from the distros. Yes. this means that the distro packagers will have
>to add an epoch in their packaging. Guess what? That's what they do and
>why they have that tool at their disposal.
>
>b) from git. Not a problem. "git pull ; pip install -U ." will work no
>matter what the version number is"
>
>c) from locally built distro packages. If they're also following
>upstream packaging, no problem. If they're doing their own thing, then
>they get to add an epoch line.
>
>d) in docker containers. Not a problem. Build a new container.
>
>e) in venvs. Not a problem. Build a new venv.
>
>f) from locally built packages that are created programatically and have
>decided to base their versioning on a direct translation of the python
>version string that also publish into a local distro package repository
>from which deployments are run. Ok. Fair - this one will hurt. However,
>I believe the sum-total of the list of that is "anvil" - so I'm pretty
>sure that adding a quick "if version == blah" into the anvil code would
>solve this.

It's not just Anvil. Openstack-ansible is going to have this problem. And
operators there appear to be salt states being discussed in one of the
Tokyo summit talk proposals.

>So when I say it's not an actual problem, but instead a pedantic
>theoretical problem - this is what I mean. There are no real instances
>where this is anything other than bikeshedding over the "right" way to
>do things.

No it is a problem. The fastest way to install python packages at scale is
by generating wheels and installing those. That means installing from git
is out of the question. Building new containers can work for some but not
for all. Same for virtualenvs. If you're not using docker or virtualenvs
and you're using a wheel repository for speed and reproducibility you're
out of luck basically. From the talk proposals for Tokyo it looks like
more than one group of people have found this way of installing OpenStack
(building a wheel, pushing to a repository, pip installing it).

>Are there a few people who this will annoy? TOTALLY. Am I sorry about
>that? TOTALLY.

Are you minimizing the actual effect for to satisfy personal preferences?
Maybe.

>Do I think that we should introduce an epoch into our upstream version
>numbering even though it will help close to zero people? Nope. I do not.
>
>Monty

Quite frankly, the only real arguments I'm hearing against setting an
epoch is that it's ugly. Not that there isn't technical merit or that it
isn't a real use case. We can all ignore the way people are actually
deploying OpenStack today and pat ourselves on the back, but that won't
garner us much with operators.

Beyond the projects described above, it's entirely plausible that there
are more operators, not subscribed to openstack-dev because of the volume
of communication, who are going to have to go through this pain with
whatever tooling they're using.

Is there some kind of middle ground to be reached? A way we can set the
version appropriately while not generating ugly tarballs for those who
find it too ugly to be worthy? This would be a middle ground where our
tags/tarballs all exclude the epoch while anything that is a package built
with python includes it.

I'm not trying to be unreasonable here, I'm just trying to find a
resolution that isn't a large ball of hacks and that respects python
packaging tools.

--
Ian