Open Stack

I've seen this pop up in various channels now, so I figure I better get more
visibility on what is going on to avoid wasted troubleshooting.

We have a couple of issues causing failures with stable/pike and stable/ocata.
Actually, it also affects stable/queens as well due to grenade jobs needing to
run stable/pike first.

The first is an issue with setuptools since the 39.0.0 release. There were some
deprecations for the type of Version objects returned from setuptools that have
now been removed to new objects that no longer allow iterating. This impacted
oslo.utilsin versionutils.is_compatible, causing that method to raise the
exception:

    TypeError: 'Version' object does not support indexing

This has been already addressed in master, so there are two backports for that
fix to the stable branches.

The second issue is with a new release of Pip. Basically, this change
deprecated and removed support for importing pip and calling internal methods
in 9.0.2. This manifests itself by neutron agent failing to load with the
following in the q-agt log file:

    KeyError: 'pip._vendor.urllib3.contrib'

This actually bubbles up from the ryu package. Luckily, they had refactored
some things such that they are still importing pip, but we are not calling the
parts of the ryu code where this is still an issue.

To make things even more fun, these changes are in two different repos, and
neither can merge without the other fix.

I think we have a full working plan in place. The oslo.util patches would fail
just the legacy-tempest-dsvm-neutron-src job, so that has been marked as
non-voting for now. Next, the oslo.util fixes need to merge and a new stable
release done for them. Then, requirements updates to both stable branches can
pass that raise the upper-constraints for ryu to 4.18 which includes the
changes we need.

Once all that is done, we can merge the last patch that reverts the change
making legacy-tempest-dsvm-neutron-src voting again.

The set up patches (other than the upcoming release requests) can be found
under the pip/5081 topic:

https://review.openstack.org/#/q/topic:pip/5081+(status:open+OR+status:merged)

As far as I can tell, once all that is done, the stable branches should be
unblocked and we should be back in business. If anything else crops up, I'll
post updates here.

Thanks,
Sean