Open Stack

Sean,

Before everything, I'd like to thank you for insisting in making the
transition to SQLA 0.8.x.

Since it has been uploaded to Sid, this SQLA <0.7.99 has been without
any doubt the biggest reoccurring pain in the but with the packaging of
OpenStack. Without people like you, insisting again and again, I would
have loose hope that progress could happen in OpenStack! So thanks
again, Sean.

On 01/03/2014 11:26 PM, Sean Dague wrote:
> Given that sqla 0.9 just came out, I wanted to explore, again, what the
> state of the world was with sqla 0.8 (especially given that Ubuntu and
> Red Hat are both shipping 0.8 in their OpenStack bundles) -
> https://review.openstack.org/#/c/64831/
> 
> The answer is not great. But more importantly, the answer is actually
> worse than the last time we checked, which I think gives this some
> urgency to move forward.

For me, it's been urgent since the 6th of July...

SQLA 0.8.2 was uploaded to Sid on the 6th of July. Since then, I've been
bugging everyone on this list about it, explaining that I have no choice
but to have Debian packages to support it (since I upload in Sid, and
Sid has SQL 0.8.x). It seems I still haven't been heard.

Now, we're 6 months after that, and after the release of Havana which
happened more than 3 months after this, and after everything was fixed
in all core packages (the last one was heat, at the end of August). So,
in January 2014, I'm still fixing manually most requirements.txt, which
still advertize for support of only SQL <0.7.99. I currently have
fixes-SQLAlchemy-requirement.patch in Cinder, Neutron, and Nova (for
Havana), just to allow it and stop the software to break because it was
decided it is the case, even though they perfectly work with SQLA 0.8.
Some other project have SQLAlchemy>=0.7.8,<=0.7.99 in the
requirements.txt, but do not break as badly as these 3 just because of
the bad declaration that doesn't reflect the reality (that's the case
for Heat Keystone and Glance, for example).

Something is going extremely wrong here. And seeing the actual result of
the SQLA transition, I am really leaning toward thinking we have a
process problem. What I believe is wrong, is that instead of having
project wide decisions imposing some constraints, we have leaf packages
that do. Until absolutely all of OpenStack is ready and fixed, then
there's no constraint applied. This is the thing that must change.

It shouldn't be this way. It should be from top to bottom. While I do
understand that we do need the gate to be able to continue working with
all projects at any given time, we still must find a solution so that
this kind of 6 months transition never happens again. This really goes
against the culture that we have inside OpenStack, and our common belief
that things must be able to move fast.

On 01/04/2014 04:13 AM, Sean Dague wrote:>
> Because of entry points any library that specifies any dependencies
> that OpenStack components specify, at incompatible levels, means that
> library effectively puts a hold on the rest of OpenStack and prevents
> it from being able to move forward.

That's exactly the kind of *very bad* habits that needs to stop in
OpenStack.

On 01/04/2014 04:13 AM, Sean Dague wrote:
> The only other option is that libraries we own (stackforge / oslo),
> for condition to be included in global- requirements, *can never*
> specify a maximum version of any dependency (and I really mean
> never), and can never specify a minimum greater than current global
> requirements.

PLEASE !!! Let's do this !!! :)

On 01/04/2014 05:29 AM, Sean Dague wrote:
> It actually tells us that a human, somewhere, decided that their
> software did not work with some combination of other software

Often it's even worse. Sometimes, a human decide that, just it case, the
software will declare itself incompatible with some non-existent future
version of another software, even if there's no way to know.

We're even more into sci-fi when we see stuff like:

pbr>=0.5.21,<1

Monty, did you decide you would release 1.0 with lots of backward
incompatibility? Has the topic been raised and I missed it??? I'm
convinced this isn't the case (and let's pretend it isn't, just until
the end of this message).

So, how does one know that the thing he's using in PBR will be the thing
that will cause trouble later on? For a version which hasn't been
released yet?!? Who's the person with such a looking glass, who can
predict the future? I'd like to know as well some stuff in my personal
future...

Please, let's stop this kind of non-sense and stop pretending we can
tell if something is incompatible with a version of something else that
doesn't even exist yet. It's hurting and slowing down the whole of
OpenStack for no reason.

> One of the key problems is taskflow, which has an sqla pin, which breaks
> all the cinder entry points. This was actually entirely the problem
> global requirements was meant to address, but it really can't when there
> are nested requirements like this, in stackforge projects that aren't
> installing via git (so we can rewrite their requirements).
> 
> So why does taskflow have the SQLA pin? And if the answer is global
> requirements, then taskflow can't be installing via pypi like it is now,
> because it's by nature taking a wedge. So we need a real solution here
> to un bind us, because right now it's actually impossible to upgrade
> sqla in requirements because of taskflow.

A colleague and myself have been struggling with the packaging of
taskflow. Whatever we do with its dependencies, there's some unit tests
errors. One of them is that it can't find the "futures" module:

DistributionNotFound: futures>=2.1.3

That's weird because I have python-concurrent.futures 2.1.3-1~bpo70+1
installed on my system.

There's also lots of:

RuntimeError: No 'taskflow.engines' driver found, looking for 'parallel'
or
RuntimeError: No 'taskflow.engines' driver found, looking for 'default'

which we can't figure out the reason (but probably related to the
detection of futures missing...).

Though we do have python-concurrent.futures installed. Is this really
the module that taskflow expects, or does it really want "futures" and
not concurrent.futures? If it really wants "futures" then we have a
problem, because python-futures (which isn't in Debian yet) would
conflict with concurrent.futures (they would share the same files). I'd
be very happy to have upstream point of view, and probably some help.

Cheers,

Thomas Goirand (zigo)