Open Stack

Hi,

Here's my summary of the discussions we had in the Oslo room at the PTG. 
  Please feel free to reply with any additions if I missed something or 
correct anything I've misrepresented.

oslo.config drivers for secret management
-----------------------------------------

The oslo.config implementation is in progress, while the Castellan 
driver still needs to be written.  We want to land this early in Rocky 
as it is a significant change in architecture for oslo.config and we 
want it to be well-exercised before release.

There are discussions with the TripleO team around adding support for 
this feature to its deployment tooling and there will be a functional 
test job for the Castellan driver with Custodia.

There is a weekly meeting in #openstack-meeting-3 on Tuesdays at 1600 
UTC for discussion of this feature.

oslo.config driver implementation: https://review.openstack.org/#/c/513844
spec: 
https://specs.openstack.org/openstack/oslo-specs/specs/queens/oslo-config-drivers.html
Custodia key management support for Castellan: 
https://review.openstack.org/#/c/515190/

"stable" libraries
------------------

Some of the Oslo libraries are in a mature state where there are very 
few, if any, meaningful changes to them.  With the removal of the 
requirements sync process in Rocky, we may need to change the release 
process for these libraries.  My understanding was that there were no 
immediate action items for this, but it was something we need to be 
aware of.

dropping support for mox3
-------------------------

There was some concern that no one from the Oslo team is actually in a 
position to support mox3 if something were to break (such as happened in 
some libraries with Python 3.6).  Since there is a community goal to 
remove mox from all OpenStack projects in Rocky this will hopefully not 
be a long-term problem, but there was some discussion that if projects 
needed to keep mox for some reason that they would be asked to provide a 
maintainer for mox3.  This topic is kind of on hold pending the outcome 
of the community goal this cycle.

automatic configuration migration on upgrade
--------------------------------------------

There is a desire for oslo.config to provide a mechanism to 
automatically migrate deprecated options to their new location on 
version upgrades.  This is a fairly complex topic that I can't cover 
adequately in a summary email, but there is a spec proposed at 
https://review.openstack.org/#/c/520043/ and POC changes at 
https://review.openstack.org/#/c/526314/ and 
https://review.openstack.org/#/c/526261/

One outcome of the discussion was that in the initial version we would 
not try to handle complex migrations, such as the one that happened when 
we combined all of the separate rabbit connection opts into a single 
connection string.  To start with we will just raise a warning to the 
user that they need to handle those manually, but a templated or 
hook-based method of automating those migrations could be added as a 
follow-up if there is sufficient demand.

oslo.messaging plans
--------------------

There was quite a bit discussed under this topic.  I'm going to break it 
down into sub-topics for clarity.

oslo.messaging heartbeats
=========================

Everyone seemed to be in favor of this feature, so we anticipate 
development moving forward in Rocky.  There is an initial patch proposed 
at https://review.openstack.org/546763

We felt that it should be possible to opt in and out of the feature, and 
that the configuration should be done at the application level.  This 
should _not_ be an operator decision as they do not have the knowledge 
to make it sanely.

There was also a desire to have a TTL for messages.

bug cleanup
===========

There are quite a few launchpad bugs open against oslo.messaging that 
were reported against old, now unsupported versions.  Since we have the 
launchpad bug expirer enabled in Oslo the action item proposed for such 
bugs was to mark them incomplete and ask the reporter to confirm that 
they still occur against a supported version.  This way bugs that don't 
reproduce or where the reporter has lost interest will eventually be 
closed automatically, but bugs that do still exist can be updated with 
more current information.

deprecations
============

The Pika driver will be deprecated in Rocky.  To our knowledge, no one 
has ever used it and there are no known benefits over the existing 
Rabbit driver.

Once again, the ZeroMQ driver was proposed for deprecation as well.  The 
CI jobs for ZMQ have been broken for a while, and there doesn't seem to 
be much interest in maintaining them.  Furthermore, the breakage seems 
to be a fundamental problem with the driver that would require 
non-trivial work to fix.

Given that ZMQ has been a consistent pain point in oslo.messaging over 
the past few years, it was proposed that if someone does step forward 
and want to maintain it going forward then we should split the driver 
off into its own library which could then have its own core team and 
iterate independently of oslo.messaging.  However, at this time the plan 
is to propose the deprecation and start that discussion first.

CI
==

Need to migrate oslo.messaging to zuulv3 native jobs.  The 
openstackclient library was proposed as a good example of how to do so.

We also want to have voting hybrid messaging jobs (where the 
notification and rpc messages are sent via different backends).  We will 
define a devstack job variant that other projects can turn on if desired.

We also want to add amqp1 support to pifpaf for functional testing.

Low level messaging API
=======================

A proposal for a new oslo.messaging API to expose lower level messaging 
functionality was proposed.  There is a presentation at 
https://docs.google.com/presentation/d/1mCOGwROmpJvsBgCTFKo4PnK6s8DkDVCp1qnRnoKL_Yo/edit?usp=sharing

This seemed to generally be well-received by the room, and dragonflow 
and neutron reviewers were suggested for the spec.

Kafka
=====

Andy Smith gave an update on the status of the Kafka driver.  Currently 
it is still experimental, and intended to be used for notifications 
only.  There is a presentation with more details in 
https://docs.google.com/presentation/d/e/2PACX-1vQpaSSm7Amk9q4sBEAUi_IpyJ4l07qd3t5T_BPZkdLWfYbtSpSmF7obSB1qRGA65wjiiq2Sb7H2ylJo/pub?start=false&loop=false&delayms=3000&slide=id.p

testing for Edge/FEMDC use cases
================================

Matthieu Simonin gave a presentation about the testing he has done 
related to messaging in the Edge/FEMDC scenario where messaging targets 
might be widely distributed.  The slides can be found at 
https://docs.google.com/presentation/d/1LcF8WcihRDOGmOPIU1aUlkFd1XkHXEnaxIoLmRN4iXw/edit#slide=id.p3

In short, there is a desire to build clouds that have widely distributed 
nodes such that content can be delivered to users from a location as 
close as possible.  This puts a lot of pressure on the messaging layer 
as compute nodes (for example) could be halfway around the world from 
the control nodes, which is problematic for a broker-based system such 
as Rabbit.  There is some very interesting data comparing Rabbit with a 
more distributed AMQP1 system based on qpid-dispatch-router.  In short, 
the distributed system performed much better for this use case, although 
there was still some concern raised about the memory usage on the client 
side with both drivers.  Some followup is needed on the oslo.messaging 
side to make sure we aren't leaking/wasting resources in some messaging 
scenarios.

For further details I suggest taking a look at the presentation.

mutable configuration
---------------------

This is also a community goal for Rocky, and Chang Bo is driving its 
adoption.  There was some discussion of how to test it, and also that we 
should provide an example of turning on mutability for the debug option 
since that is the target of the community goal.  The cinder patch can be 
found here: https://review.openstack.org/#/c/464028/  Turns out it's 
really simple!

Nova is also using this functionality for more complex options related 
to upgrades, so that would be a good place to look for more advanced use 
cases.

Full documentation for the mutable config options is at 
https://docs.openstack.org/oslo.config/latest/reference/mutable.html

The goal status is being tracked in 
https://storyboard.openstack.org/#!/story/2001545

Chang Bo was also going to talk to Lance about possibly coming up with a 
burndown chart like the one he had for the policy in code work.

oslo healthcheck middleware
---------------------------

As this ended up being the only major topic for the afternoon, the 
session was unfortunately lightly attended.  However, the self-healing 
SIG was talking about related topics at the same time so we ended up 
moving to that room and had a good discussion.

Overall the feature seemed to be well-received.  There is some security 
concern with exposing service information over an un-authenticated 
endpoint, but because there is no authentication supported by the health 
checking functionality in things like Kubernetes or HAProxy this is 
unavoidable.  The feature won't be mandatory, so if this exposure is 
unacceptable it can be turned off (with a corresponding loss of 
functionality, of course).

There was also some discussion of dropping the asynchronous nature of 
the checks in the initial version in order to keep the complexity to a 
minimum.  Asynchronous testing can always be added later if it proves 
necessary.

The full spec is at https://review.openstack.org/#/c/531456

oslo.config strict validation
-----------------------------

I actually had discussions with multiple people about this during the 
week.  In both cases, they were just looking for a minimal amount of 
validation that would catch an error such at "devug=True".  Such a 
validation might be fairly simple to write now that we have the 
YAML-based sample config with (ideally) information about all the 
options available to set in a project.  It should be possible to compare 
the options set in the config file with the ones listed in the sample 
config and raise warnings for any that don't exist.

There is also a more complete validation spec at 
http://specs.openstack.org/openstack/oslo-specs/specs/ocata/oslo-validator.html 
and a patch proposed at https://review.openstack.org/#/c/384559/

Unfortunately there has been little movement on that as of late, so it 
might be worthwhile to implement something more minimalist initially and 
then build from there.  The existing patch is quite significant and 
difficult to review.

Conclusion
----------

I feel like there were a lot of good discussions at the PTG and we have 
plenty of work to keep the small Oslo team busy for the Rocky cycle. :-)

Thanks to everyone who participated and I look forward to seeing how 
much progress we've made at the next Summit and PTG.

-Ben