Open Stack

On 03/15/2017 05:32 AM, Thierry Carrez wrote:
> Mike Bayer wrote:
>> As mentioned previously, SQLAlchemy 1.1 has now been released for about
>> six months.   My work now is on SQLAlchemy 1.2 which should hopefully
>> see initial releases in late spring.    SQLAlchemy 1.1 includes tons of
>> features, bugfixes, and improvements, and in particular the most recent
>> versions contain some critical performance improvements focused around
>> the "joined eager loading" feature, most typically encountered when an
>> application makes many, many queries for small, single-row result sets
>> with lots of joined eager loading.   In other words, exactly the kinds
>> of queries that Openstack applications do a lot; the fixes here were
>> identified as a direct result of Neutron query profiling by myself and a
>> few other contributors.
>>
>> For many weeks now, various patches to attempt to bump requirements for
>> SQLAlchemy 1.1 have been languishing with little interest, and I do not
>> have enough knowledge of the requirements system to get exactly the
>> correct patch that will accomplish the goal (nor do others).  The
>> current gerrit is at https://review.openstack.org/#/c/423192/, where you
>> can see that not just me, but a bunch of folks, have no idea what
>> incantations we need to put here that will make this happen.  Tony
>> Breeds has chimed in thusly:
>>
>>> To get this in we'll need to remove the cap in global-requirements
>> *and* at the same time add a heap of entries to
>> upper-constratints-xfails.txt. this will allow us to merge the cap
>> removal and keep the constraint in the 1.0 family while we wait for the
>> requirements sync to propagate out.
>>
>> I'm not readily familiar with what goes into upper-constraints-xfails
>> and this file does not appear to be documented in common places like
>> https://wiki.openstack.org/wiki/Requirements or
>> https://git.openstack.org/cgit/openstack/requirements/tree/README.rst .
>>
>> I'm asking on the list here for some assistance in moving this forward.
>> SQLAlchemy development these days is closely attuned to the needs of
>> Openstack now, a series of Openstack test suites are part of
>> SQLAlchemy's own CI servers to ensure backwards compatibility with all
>> changes, and 1.2 will have even more features that are directly
>> consumable by oslo.db (features everyone will want, I promise you).
>> Being able to bump requirements across Openstack so that new versions
>> can be tested and integrated in a timely manner would be very helpful.
> 
> It sounds like a transition that the requirements team should directly
> help with (since it's tricky). Would be a shame to pass on the
> performance improvements just because the process to work around the cap
> is dark magic. I would help but it's the first time I hear of that
> xfails file, so I'm probably not current enough :)

I think there was just a communication gap here, I'm hoping the updated
review (which I just approved) gets us past it.

The question was: How can we get newer SQLA into the system?

The problem was the original patch kept a cap on SQLA, just moved it up
to the next pre-release, not realizing the caps in general are the
concern by the requirements team. So instead of upping the cap, I just
removed it entirely. (It also didn't help on clarity that there was a
completely unrelated fail in the tests which made it look like the
system was stopping this.)

This should hopefully let new SQLA releases very naturally filter out to
all our services and libraries.

	-Sean

-- 
Sean Dague
http://dague.net