Open Stack

On 06/06/2018 10:04 PM, Doug Hellmann wrote:
> I have started submitting a series of patches to fix up the tox.ini
> settings for projects as a step towards running "python3 first"
> [1]. The point of doing this now is to give teams a head start on
> understanding the work involved as we consider whether to make this
> a community goal.
> 
> The current patches are all mechanically generated changes to the
> basepython value for environments that seem to be likely candidates.
> They're basically the "easy" part of the transition. I've left any
> changes that will need more discussion alone for now.
> 
> In particular, I've skipped over any tox environments with "functional"
> in the name, since I thought those ran functional tests. Teams will
> need to decide whether to change those job definitions, or duplicate
> them and run them under python 2 and 3. Since we are not dropping
> python 2 support until the U cycle, I suggest going ahead and running
> the jobs twice.
> 
> Note that changing the tox settings won't actually change some of the
> jobs. For example, with our current PTI definition, the documentation
> and releasenotes jobs do not run under tox. That means those will need
> to be changed by editing the zuul configuration for the repository.
> 
> I have started to make notes for tracking the work in
> https://etherpad.openstack.org/p/python3-first -- including some notes
> about taking the next step to update the zuul job definitions and common
> issues we've already encountered to help folks debug job failures.
> 
> I could use some help keeping an eye on these changes and getting
> them through the gate. If you are interested in helping, please
> leave a comment on the review you are willing to shepherd.
> 
> Doug
> 
> [1] https://review.openstack.org/#/q/topic:python3-first+(status:open+OR+status:merged)

tl;dr: all projects must move their functional tests under Py3, uwsgi
and SSL, otherwise there's undetected issues.

Doug,

Attempting to make puppet-openstack fully work for Debian (more on this
soon as hopefully I'll be able to announce soon the general availability
and success with testing), and having all Debian packages running in
Python 3, what I have discovered is that there's a general issue when
running tests: they do not take into account the fact that we cannot run
with Eventlet, because Python 3 + Eventlet + SSL = handshake crash.

In other words, all projects need to run their API functional tests
using either mod_wsgi or uwsgi (and more likely the later), using Python
3 *AND* SSL, as this is how operators will run. Unfortunately, this is
far from being the case at the moment, and I have been able to find
defects in at least 3 services, and sometimes, helped, or contributed
patches myself.

More in details, projects after projects, just so you understand what I
mean. No finger pointing here, these are just to illustrate my
argumentation and show the kind of problems we're having.

sahara-api looks completely broken under Python 3 (ie: error 500 on each
GET requests), unless this tiny 3 liner patch is applied
https://review.openstack.org/#/c/573179/ (hopefully, a better patch will
happen upstream, I'm just a package maintainer not having enough time to
get involved deep enough in each individual projects...)

Cinder-volume fails to load with Ceph driver if using Luminous and
Python 3 unless this patch is applied:
https://review.openstack.org/#/c/568813/

neutron-server, under Python 3 and SSL, has to be transformed into
neutron-api served over uwsgi (probably also works using mod_wsgi). In
such case, you must run neutron-rpc-server to handle messages from the
bus. But then, the rpc-server doesn't understand plugin modules unless
this patch is added: https://review.openstack.org/555608 and also,
firewall rules aren't received by the openvswitch-agent (and therefore,
security group rules aren't being applied to iptables) unless this patch
is reverted: https://review.openstack.org/434678 and then it works for
openvswitch agent, though for linuxbridge, I still couldn't ping
floating IPs (I'm still investigating the problem, I'm not sure yet if
it's a routing problem or what...). And that's not even over, as it
looks like  tempest's scenario
test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops is
still broken with openvswitch for me:
http://logs.openstack.org/28/564328/48/check/puppet-openstack-integration-4-scenario001-tempest-debian-stable/5e64694/job-output.txt.gz#_2018-06-07_12_42_50_760463
Help is desperately wanted here as I spent a really long time on this
already, it's still failing, and I don't understand what's going on!

All this to say, the more general issue here is that projects, while
unit testing under Python 3, aren't running functional tests the same
way that operators will (ie: with SSL and uwsgi/mod_wsgi), and because
of that, some problems aren't being detected. Just a hint, to run
something like neutron-api with ipv4, 6 and SSL, you'd run it this way:

/usr/bin/uwsgi_python35 --https-socket
[::]:9696,/usr/local/share/ca-certificates/puppet_openstack.crt,/etc/neutron/ssl/private/debian-stretch-ovh-gra1-0004341140.pem
--ini /etc/neutron/neutron-api-uwsgi.ini --pyargv
"--config-file=/etc/neutron/neutron.conf
--config-file=/etc/neutron/plugins/ml2/ml2_conf.ini
--config-file=/etc/neutron/fwaas_driver.ini"

Notice the [::] meaning ipv4 AND ipv6 at the same time, and how --pyargv
needs a quoted list of arguments.

Hopefully, this is slowly being addressed, though I would very much like
if there was a general move to using uwsgi, preferably with SSL and ipv6
turned on and tested.

It'd be also nice if projects could ship the uwsgi ini file that they
use for functional tests, as it looks like it sometimes depend on the
project (for example, I had to add rem-header: content-length for
nova-placement-api to avoid connection reset by peer, but I'm not sure
if it even is harmful for other projects). Here's an example from the
Debian packaging:

https://salsa.debian.org/openstack-team/services/neutron/blob/debian/queens/debian/neutron-api-uwsgi.ini

See how I'm deliberately *not* setting "pyargv" there, and prefer it to
be set by the init script / systemd service (so that it can be dynamic
and load the configuration file needed for the activated plugin), and
that http-socket / https-socket is also dynamic, ie: --https-socket is
used on the command line if a pair of certificate + private key is found
on the hard disk under /etc/neutron/ssl. See
https://salsa.debian.org/openstack-team/services/neutron/blob/debian/queens/debian/neutron-api.init.in
and
https://salsa.debian.org/openstack-team/debian/openstack-pkg-tools/tree/debian/queens/init-template
to understand how it's built.

Hoping that these packaging-related insights are helpful for the project
at large, cheers,

Thomas Goirand (zigo)