Open Stack

On 04/19/2016 03:10 PM, Doug Hellmann wrote:
>> On your etherpad, you wrote:
>>
>> "During the lead-up to preparing the final releases, one of the tracking
>> tasks we have is to ensure all projects have synced their global
>> requirements updates. This is another area where we could reduce the
>> burden on the release team."
>>
>> Well, don't bother, this doesn't reflect a reality anyway (ie: maybe
>> service X can use an older version of oslo.utils), so that's not really
>> helpful in any way.
> 
> Right, that's why I proposed stopping the whole business with managing
> the ranges. Your proposal seems to be somewhere in the middle between
> doing what we do today (obsessively keeping in sync) and what I'm
> proposing (abandoning any pretense of keeping in sync). Is that right?

Absolutely ! :)

>> Though having a single version that projects are allowed to test with is
>> super important, so we can check everything can work together. IMO,
>> that's the part which should absolutely not change. Dropping that is
>> like opening a Pandora box. Relying on containers and/or venv will
>> unfortunately, not work, from my stand point.
> 
> Again, as I pointed out elsewhere in the thread, using some sort of
> environment isolation tool in upstream CI testing does not imply that
> the same or a similar tool needs to be used by distros for packaging.

But then, we still need co-installability and avoid lib version
conflicts. Aren't we then back to square one?

It feels like I'm missing something in the reasoning here...

>> (think: we still have loads of Python 2.6 stuff like discover, argparse
>> and such that needs to be cleaned-up).
> 
> Aside: We should figure out a way to make a list of those things,
> so we can work on the cleanup.

I've pushed a bit of patches here and there, but I sometimes give-up
forwarding the Debian work (because lacking time). I'm hereby making the
promise to always do it for the Newton cycle (as there shouldn't be too
many remaining).

> I'm certainly not proposing that. My proposal is point out the
> economics of our current situation, which is that upstream we're
> doing a lot of work that IMHO we don't need to do *for ourselves*.
> I do see its usefulness, but it's not necessary for our own CI
> testing.

Thanks for making this explicit. I was kind of double-guessing something
like that was in your mind. I'm with you now.

>>> Another alternate that might work is for downstream folks to do
>>> their own dependency management.
>>
>> I fail to understand what you think distros could do to fix the
>> situation of conflicting versions, other than bundling, which isn't
>> acceptable. Could you explain what you mean by "downstream folks to do
>> their own dependency management"?
> 
> How do you ensure that a given version of a python lib you package
> is compatible with independent tools written to use that lib that
> you also want to include in your distro?

Well, you're pointing here at a major pain point. I can give you 2
examples where it has been extremely painful for me:
- SQLA upgrades from 0.7.x to 0.8.x, then to 0.9.x. a year ago.
- Django upgrade to 1.9 last december.

Both SQLA and Django are widely used in Debian, and when one gets
upgraded, package maintainers have no choice but to fix their packages
ASAP, to support the new version.

I don't maintain SQLA or Django, both are maintained by package
maintainers who don't really care about OpenStack, and therefore, they
uploaded new versions to Sid before OpenStack was ready.

The result is that most of OpenStack was broken in Sid for maybe 4 or 5
months when the SQLA upgrade happened.

As for Django, this broke Horizon from last December to a few days
before the release. Without heroic efforts from Horizon contributors,
(mostly by Rob, the new Horizon PTL. but not only him), then Horizon
would still be broken. I did a bunch of Django 1.9 patches (blind hacks
looking at other fixes and upstream doc, really...) myself too

> Or for that matter, anything
> written in any other language that supports shared libraries.

For shared libraries breaking ABI, we have "binNMU" that the release
team can triggers (in fact, just a rebuild of packages). For API
breakage of shared libs, we have "transitions" which the Debian release
team manages. It's a major pain point. Last summer, Matthias Klose (aka
Doko) upgraded gcc from version 4.9 to 5.x, which included (if I'm not
mistaking) a breakage in libstdc++. This broke Debian unstable for more
than a month, with hundreds of packages to fix. When all was fixed,
everything migrated at once from unstable to testing (yes, that's the
difference between Debian testing and unstable: testing isn't affected
by library transition bugs).

Oh, and a few month later, Doko decided it would be fun to upgrade
Python 3 from 3.4 to 3.5! :)

>>> I started the discussion to solicit ideas, but I would prefer to
>>> avoid proposing what downstream should do because (a) I'm sure
>>> folks want options and (b) I'm sure I'm not the person to identify
>>> those options.
>>
>> Well, I'm doing the packaging, and I'm not sure if we have any option
>> either, if 2 python dependencies are conflicting.
> 
> Surely OpenStack is not the only time when that situation arises?

Of course not. I suppose you're asking what happens in distro in such
cases, right?

When the latest version of a given lib is uploaded, and some package
break (either they can't be rebuild, or they have new bug, or anything
else release critical), then they get auto-removed from Debian testing,
and therefore, from the next Debian stable release if nothing is done.

The *only* solution that there is, is to fix that one component that's
not accepting the new version.

After Django 1.9 was uploaded to Sid last December, a month and a half
later (which is the normal delay), Horizon (and all its plugins) got
therefore removed from Debian testing... until I uploaded the Mitaka
final release of Horizon, including a not-yet-merged Django 1.9 compat
patch.

Oh, btw, horizon is producing loads of Django 1.10 deprecation warnings,
so history may repeat itself. Except that this time, it may be the last
time before the Debian 9 (aka: Stretch) release... :/

>> I've done some of that, and used to be reactive on it. But mostly, the
>> way it's been done so far is very good (thanks to a lot of folks working
>> on it), and it's taking too much of my time to review them. So I just
>> pick it up before packaging a (beta-)version of OpenStack.
> 
> I think you've made my point here. Reviewing these changes *is*
> time consuming.

I'm kind of reviewing it: I'm subscribed to the requirements repository
change, and receive everything in my mail. About once a week, I read the
resulting emails, and sometimes, even go to look at the gerrit review
(when the subject of the patch isn't helpful enough, which is
unfortunately often the case). This doesn't help getting
global-requirements.txt being reviewed, but this is super helpful for my
packaging workflow, as I can see what new package there is to do.

Cheers,

Thomas Goirand (zigo)