Open Stack

Excerpts from Thomas Goirand's message of 2018-06-07 22:17:11 +0200:
> On 06/06/2018 10:04 PM, Doug Hellmann wrote:
> > I have started submitting a series of patches to fix up the tox.ini
> > settings for projects as a step towards running "python3 first"
> > [1]. The point of doing this now is to give teams a head start on
> > understanding the work involved as we consider whether to make this
> > a community goal.
> > 
> > The current patches are all mechanically generated changes to the
> > basepython value for environments that seem to be likely candidates.
> > They're basically the "easy" part of the transition. I've left any
> > changes that will need more discussion alone for now.
> > 
> > In particular, I've skipped over any tox environments with "functional"
> > in the name, since I thought those ran functional tests. Teams will
> > need to decide whether to change those job definitions, or duplicate
> > them and run them under python 2 and 3. Since we are not dropping
> > python 2 support until the U cycle, I suggest going ahead and running
> > the jobs twice.
> > 
> > Note that changing the tox settings won't actually change some of the
> > jobs. For example, with our current PTI definition, the documentation
> > and releasenotes jobs do not run under tox. That means those will need
> > to be changed by editing the zuul configuration for the repository.
> > 
> > I have started to make notes for tracking the work in
> > https://etherpad.openstack.org/p/python3-first -- including some notes
> > about taking the next step to update the zuul job definitions and common
> > issues we've already encountered to help folks debug job failures.
> > 
> > I could use some help keeping an eye on these changes and getting
> > them through the gate. If you are interested in helping, please
> > leave a comment on the review you are willing to shepherd.
> > 
> > Doug
> > 
> > [1] https://review.openstack.org/#/q/topic:python3-first+(status:open+OR+status:merged)
> 
> tl;dr: all projects must move their functional tests under Py3, uwsgi
> and SSL, otherwise there's undetected issues.
> 
> Doug,
> 
> Attempting to make puppet-openstack fully work for Debian (more on this
> soon as hopefully I'll be able to announce soon the general availability
> and success with testing), and having all Debian packages running in
> Python 3, what I have discovered is that there's a general issue when
> running tests: they do not take into account the fact that we cannot run
> with Eventlet, because Python 3 + Eventlet + SSL = handshake crash.
> 
> In other words, all projects need to run their API functional tests
> using either mod_wsgi or uwsgi (and more likely the later), using Python
> 3 *AND* SSL, as this is how operators will run. Unfortunately, this is
> far from being the case at the moment, and I have been able to find
> defects in at least 3 services, and sometimes, helped, or contributed
> patches myself.
> 
> More in details, projects after projects, just so you understand what I
> mean. No finger pointing here, these are just to illustrate my
> argumentation and show the kind of problems we're having.
> 
> sahara-api looks completely broken under Python 3 (ie: error 500 on each
> GET requests), unless this tiny 3 liner patch is applied
> https://review.openstack.org/#/c/573179/ (hopefully, a better patch will
> happen upstream, I'm just a package maintainer not having enough time to
> get involved deep enough in each individual projects...)
> 
> Cinder-volume fails to load with Ceph driver if using Luminous and
> Python 3 unless this patch is applied:
> https://review.openstack.org/#/c/568813/
> 
> neutron-server, under Python 3 and SSL, has to be transformed into
> neutron-api served over uwsgi (probably also works using mod_wsgi). In
> such case, you must run neutron-rpc-server to handle messages from the
> bus. But then, the rpc-server doesn't understand plugin modules unless
> this patch is added: https://review.openstack.org/555608 and also,
> firewall rules aren't received by the openvswitch-agent (and therefore,
> security group rules aren't being applied to iptables) unless this patch
> is reverted: https://review.openstack.org/434678 and then it works for
> openvswitch agent, though for linuxbridge, I still couldn't ping
> floating IPs (I'm still investigating the problem, I'm not sure yet if
> it's a routing problem or what...). And that's not even over, as it
> looks like  tempest's scenario
> test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops is
> still broken with openvswitch for me:
> http://logs.openstack.org/28/564328/48/check/puppet-openstack-integration-4-scenario001-tempest-debian-stable/5e64694/job-output.txt.gz#_2018-06-07_12_42_50_760463
> Help is desperately wanted here as I spent a really long time on this
> already, it's still failing, and I don't understand what's going on!
> 
> All this to say, the more general issue here is that projects, while
> unit testing under Python 3, aren't running functional tests the same
> way that operators will (ie: with SSL and uwsgi/mod_wsgi), and because
> of that, some problems aren't being detected. Just a hint, to run
> something like neutron-api with ipv4, 6 and SSL, you'd run it this way:
> 
> /usr/bin/uwsgi_python35 --https-socket
> [::]:9696,/usr/local/share/ca-certificates/puppet_openstack.crt,/etc/neutron/ssl/private/debian-stretch-ovh-gra1-0004341140.pem
> --ini /etc/neutron/neutron-api-uwsgi.ini --pyargv
> "--config-file=/etc/neutron/neutron.conf
> --config-file=/etc/neutron/plugins/ml2/ml2_conf.ini
> --config-file=/etc/neutron/fwaas_driver.ini"
> 
> Notice the [::] meaning ipv4 AND ipv6 at the same time, and how --pyargv
> needs a quoted list of arguments.
> 
> Hopefully, this is slowly being addressed, though I would very much like
> if there was a general move to using uwsgi, preferably with SSL and ipv6
> turned on and tested.
> 
> It'd be also nice if projects could ship the uwsgi ini file that they
> use for functional tests, as it looks like it sometimes depend on the
> project (for example, I had to add rem-header: content-length for
> nova-placement-api to avoid connection reset by peer, but I'm not sure
> if it even is harmful for other projects). Here's an example from the
> Debian packaging:
> 
> https://salsa.debian.org/openstack-team/services/neutron/blob/debian/queens/debian/neutron-api-uwsgi.ini
> 
> See how I'm deliberately *not* setting "pyargv" there, and prefer it to
> be set by the init script / systemd service (so that it can be dynamic
> and load the configuration file needed for the activated plugin), and
> that http-socket / https-socket is also dynamic, ie: --https-socket is
> used on the command line if a pair of certificate + private key is found
> on the hard disk under /etc/neutron/ssl. See
> https://salsa.debian.org/openstack-team/services/neutron/blob/debian/queens/debian/neutron-api.init.in
> and
> https://salsa.debian.org/openstack-team/debian/openstack-pkg-tools/tree/debian/queens/init-template
> to understand how it's built.
> 
> Hoping that these packaging-related insights are helpful for the project
> at large, cheers,
> 
> Thomas Goirand (zigo)
> 

Thanks for all of these details, Thomas,

I will work on summarizing these notes in
https://etherpad.openstack.org/p/python3-first so project teams can
consider which parts relate to their services.

Doug