March 2019 - openstack-discuss - lists.openstack.org

[designate] Release Patches
by Graham Hayes 19 Mar '19

19 Mar '19

Hi all, I have flagged patches I think we need to merge for this cycle's RC [1] Please have a look at these patches and we can try and get them in for the release. - Graham 1 - https://graham.hayes.ie/designate/release-dashboard

1 0

[placement] update 19-10
by Chris Dent 19 Mar '19

19 Mar '19

HTML: https://anticdent.org/placement-update-19-10.html Placement update 19-10 is here. We're fast approaching placement's first official release. # Most Important There are several tasks left before we can cut the release, mostly related to documentation and other writing related things. I've attempted to enumerate them in a "Prepping the RC" section below. These are things that need to be done before next Thursday, preferably sooner. It's also important to be thinking about how placement would like to engage (as a group) with the PTG (the Forum is already decided: there will be an extraction related Forum session). # What's Changed * Oh, hey, I'm, like, the placement PTL. Mel and I decided early in the week that whatever the official timetable, I'll take the baton from here. Thanks to everyone who helped to get placement to where we are now. * The stack of code that got rid of the List classes in favor of module level methods, which also happened to move each "object" type to its own module, has merged. I'm glad we got this in before release as it ought to make debugging and digging around a bit easier. * Lots of little documentation tuneups (from [story 2005190](https://storyboard.openstack.org/#!/story/2005190)) have merged, including pointing to storyboard for bugs. These chanages scrape the surface of what remains (listed below). * I wrote up a blog post on [profiling wsgi apps](https://anticdent.org/profiling-wsgi-apps.html) which I'd been doing to confirm that the many refactorings that have happened recently weren't having a negative impact (they are not). * We decided to wait for Train for the [negative-member-of](https://review.openstack.org/#/q/topic:bp/negative-aggr… functionality and the [allocation ratio](https://review.openstack.org/#/c/640898/) change in osc-placement. * Kolla has merged several changes for extracted placement. Thanks! # Specs/Blueprint/Features Skipping this section until after the release candidate(s) are done. # Bugs We've got a StoryBoard [project group](https://storyboard.openstack.org/#!/project_group/placement) now. I've started using it. Tagging bugs with `bug` and also making use of a `cleanup` tag to indicate things that needs to be cleaned up. There are worklists for both of these: * [bugs](https://storyboard.openstack.org/#!/worklist/574) * [cleanups](https://storyboard.openstack.org/#!/worklist/575) Please be prepared for these structures to evolve as we gain some understanding of how StoryBoard works. There are still bugs in launchpad and we need to continue to watch there: * Placement related [bugs not yet in progress](https://goo.gl/TgiPXb):13. No change. * [In progress placement bugs](https://goo.gl/vzGGDQ) 7. -2. Many of these are about nova's use of placement. At some point after RC we should do a bug review, and port placement-only things to StoryBoard. # osc-placement osc-placement is currently behind by 13 microversions. Pending changes: * [support for 1.19](https://review.openstack.org/#/c/641094/) * [support for 1.21](https://review.openstack.org/#/c/641123/) * [aggregate allocation ratio tool](https://review.openstack.org/#/c/640898/) # Prepping the RC Things that need to happen so we can cut a placement release candidate: * Anything currently open that we want in. There are only 6 pending patches that might be options (everything else is either waiting for Train or already +W), so a quick look [at them](https://review.openstack.org/#/q/project:openstack/placement+status:o… is worth the effort. * We've started a cycle-highlights etherpad, as announced by [this email](http://lists.openstack.org/pipermail/openstack-discuss/2019-March/00…. We've probably got enough, but feel free to add to it if you think of something. * There's a story for [preparing placement docs for stein](https://storyboard.openstack.org/#!/story/2005190). The story includes several tasks, many of which are already merged. Have a look and assign yourself a task if you can commit to having it done by early next week. There are some biggies: * Creating the canonical how to upgrade from placement-in-nova to placement-in-placement document. As stated very well by Tetsuoro, this is effectively translating the [grenade upgrade script](http://git.openstack.org/cgit/openstack-dev/grenade/tree/projects/6… to English. * Ensuring the install docs are sane and complete. [I have asked](http://lists.openstack.org/pipermail/openstack-discuss/2019-March/00… packaging-related people for their input, as they're the ones who know how their packages are (or will be) set up, but there's also an "install from-pypi" hole that needs to be filled. * The [releasenotes](https://docs.openstack.org/project-team-guide/release-managem… need to be evaluated for correctness and effective annotation of upgrade concerns. They will also need a prelude, probably pointing to the "upgrading from nova" doc mentioned above. For a sample, see nova's [rocky prelude](https://review.openstack.org/#/c/589303/). # Main Themes We'll come back to themes once the RC is cut. # Other Placement * <https://review.openstack.org/#/q/topic:cd/gabbi-tempest-job> Gabbi-based integration tests of placement. These recently found a bug that none of the functional, grenade, nor tempest tests did. Not release related, but useful testing. * <https://review.openstack.org/#/q/topic:bp/negative-aggregate-membership> Negative member of aggregate filtering resource providers and allocation candidates. Holding until Train opens. # Other Service Users We'll also hold off here until the RC is cut. In the future if you stick "placement" somewhere in your commit message I'll probably eventually find your in-progress placement-related changes. # End Once the release is released, it will be time to start thinking about what we want Train to look like. There are pending Stein feature specs that we will want to do (and will need to be put in our specs directory, once it exists), but other than the various ideas about ways to do multi-nova/cloud partitioning of resource providers and multi-service partitioning of allocations (both of which need much more well-defined uses cases before we start thinking about the solutions) I've not heard a lot of clamouring from services and operators for features in Placement. If you have heard, or are clamouring, please make yourself known. I'd personally like us to focus on enabling existing services that use or want to use placement (nova, neutron, blazar, cyborg) and its existing features rather than new features. No need to have any immediate thoughts or decisions on this, but some background thinking is warranted. Also, OMG, we need a logo. How about an Australian Magpie? They make a cool noise. <https://www.youtube.com/watch?v=oYEYc8Ge3nw> -- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent tw: @anticdent

1 2

[telemetry][ptl][volunteer][tc] Telemetry PTL for Train
by Trinh Nguyen 19 Mar '19

19 Mar '19

Hi all, Because no one want to nominate for the PTL in Train [1], Telemetry has been marked as a leaderless project and may be removed of OpenStack. We and some companies are still using Ceilometer, AODH (parts of Telemetry) so we don't want that happen. Lingxian Kong (Trove and Qinling PTL), I (cc Julien Danjou, the current PTL of Telemetry) had a conversation about this and agree to work together to maintain the Telemetry projects. So I'm writing this email stating that I would like to volunteer to be the PTL for Telemetry in Train. Though I haven't been directly involved in the development of Telemetry ever, my experience as Searchlight PTL and core in Fenix and Freezer give me some technical understanding of Telemetry as well as OpenStack development procedure. You're in the good hands. Thanks, [1] https://governance.openstack.org/election/ -- *Trinh Nguyen* *www.edlab.xyz <https://www.edlab.xyz>*

9 14

[cinder] setting `rbd_exclusive_cinder_pool = True` with multiple cinder-volume writing to the same pool
by Krzysztof Klimonda 19 Mar '19

19 Mar '19

Hi, Just doing a quick sanity check - I can set `rbd_exclusive_cinder_pool = True` (in pike, where the change landed first) even if more than one cinder-volume is writing to the same Ceph pool, right? Assuming nothing writes to the pool outside of cinder, we should be good as far as I understand the code. -Chris

2 3

[neutron] [dev] ml2plugin doesn't use new security group rpc api
by Just FooBar 19 Mar '19

19 Mar '19

Hello everyone, I'm having problems with Security Group Rule updates not being applied to vms on hypervisors and I think I know why this is happening. I see that there's already a bug about that: https://bugs.launchpad.net/neutron/+bug/1814209 but there isn't much action going on there. In my case, neutron-server is also using the queue q-agent-notifier-security_group-update (as seen from neutron-server logs in debug mode). The neutron version is the same, 12.0.4. I went to check the code for my version 12.0.4 and I've found some suspicious part which might be the cause of this bug. Let me explain my understanding of the situation. I started with ovs agent code and found that it is using SecurityGroupServerAPIShim ( https://github.com/openstack/neutron/blob/12.0.4/neutron/plugins/ml2/driver…) class for, well, Security Group RPC. Comments in this class definition ( https://github.com/openstack/neutron/blob/12.0.4/neutron/api/rpc/handlers/s…) explain that it is a replacement for an older interface, SecurityGroupServerRpcApi ( https://github.com/openstack/neutron/blob/12.0.4/neutron/api/rpc/handlers/s… ). SecurityGroupServerAPIShim inherits from SecurityGroupInfoAPIMixin ( https://github.com/openstack/neutron/blob/12.0.4/neutron/db/securitygroups_…) which is also a parent of the server side Ml2Plugin ( https://github.com/openstack/neutron/blob/12.0.4/neutron/plugins/ml2/plugin…) >From this I make a conclusion that Ml2Plugin was also switched to the new SG RPC interface. Now, there are other details that suggest that Ml2Plugin wasn't switched to the new interface entirely and continues to use the old-style SG RPC classes. There's a class AgentNotifierApi ( https://github.com/openstack/neutron/blob/12.0.4/neutron/plugins/ml2/rpc.py…) used by neutron-server's Ml2Plugin ( https://github.com/openstack/neutron/blob/12.0.4/neutron/plugins/ml2/plugin…) to send notifications (to agents, I suppose). It inherits from the class SecurityGroupAgentRpcApiMixin ( https://github.com/openstack/neutron/blob/12.0.4/neutron/api/rpc/handlers/s…) which has been marked for removal starting from Pike 3 years ago in this commit: https://github.com/openstack/neutron/commit/97338258967d3b95f382f188ab2ab57…. This AgentNotifierApi class wasn't switched to a new Shim RPC interface for SG (as it was done for the ovs agent and Ml2Plugin itself). All previous links are for 12.0.4 version, the one used in my system currently. And here's the same class AgentNotifierApi from the Rocky release: https://github.com/openstack/neutron/blob/stable/rocky/neutron/plugins/ml2/… . As you can see, it still inherits from the class marked for removal and isn't using new style SG RPC API. >From all this I conclude that until AgentNotifierApi is using new style API or the way that Ml2Plugin is sending notifications isn't changed, the bug will still be present. Please let me know if I'm getting this wrong. If I'm right, I'm interested in helping to fix the bug. Thank you for your attention!

3 4

[neutron] Bug deputy report (2019-03-11 week)
by Takashi Yamamoto 18 Mar '19

18 Mar '19

This week was not too busy and LIU was quicker to triage bugs than I was. So I looked at some older bugs (mostly from the last week) as well. The following is the list of bugs I observed. High https://bugs.launchpad.net/neutron/+bug/1819477 [OVS] When cleaning min QoS, queue could be already deleted https://bugs.launchpad.net/neutron/+bug/1820744 conntrack v1.4.4 (conntrack-tools): `0' unsupported protocol Medium https://bugs.launchpad.net/neutron/+bug/1819897 Intermittent "Neutron did not start" failures in the gate https://bugs.launchpad.net/neutron/+bug/1818651 UnsupportedVersion: Endpoint does not support RPC version 1.2. Attempted method: report_state Low https://bugs.launchpad.net/neutron/+bug/1819591 Fwaas-v2 not fully suport "source_firewall_group_id" for creation of firewall_rule https://bugs.launchpad.net/neutron/+bug/1818824 When a fip is added to a vm with dvr, previous connections loss the connectivity https://bugs.launchpad.net/neutron/+bug/1818805 Conntrack rules in the qrouter are not deleted when a fip is removed with dvr https://bugs.launchpad.net/neutron/+bug/1820297 os-ken error when BGP peer goes down Incomplete https://bugs.launchpad.net/neutron/+bug/1818960 IPv6 not working with iptables https://bugs.launchpad.net/neutron/+bug/1818697 neutron fullstack frequently times out waiting on qos ports https://bugs.launchpad.net/neutron/+bug/1818696 frequent ci failures trying to delete qos port

1 0

[tc] weekly update
by Mohammed Naser 18 Mar '19

18 Mar '19

Hi everyone! Thanks fo reading this, if you are :) This is a weekly email update on the current state of the TC. Over the last week, there hasn't been much activity over the governance repository other than a change we merged today to declare the Train supported runtimes for Python and Golang, this has officially been published here: https://governance.openstack.org/tc/reference/runtimes/train.html There was a few emails that were sent out by Zane with regards on how we're going to go about this, the versions we're going to use for testing, etc. Ghanshyam also worked alongside the infrastructure team to get our CI jobs to move to bionic to help with the Python 3 bits as well: http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003610.ht… http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003584.ht… Other than that, we've started working a document to provide feedback for the board regarding the upcoming confirmation of OIPs, the effort in setting up the etherpads and reaching out was started by Jeremy: http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003938.ht… We also have a few on-going discussions with the introduction of OpenDev, however those are still very early discussions. Jim is the point of contact of the OpenStack community within the transition process so he's provided some questions to be answered from our community: http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003943.ht… The process to get grab our goals is also about to wrap up, we have an upcoming few office hours and we're trying to pick the best one to find to discuss the upcoming cycle goals, please join if you're interested! http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003932.ht… We've got our space at the PTG setup for us for all Saturday thanks to the foundation, we're getting the Etherpads slowly up and going and I think it would be great to solicit subjects for discussion in the PTG that involve the community as well: http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003916.ht… With the summit comes a joint leadership meeting, we are looking for subjects to bring up to the board, so please chime in there if that interests you: http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003810.ht… Finally, I hope you forgive my long email. I'm trying different formats to make it more digestable and/or consumable and/or interesting. If you have feedback on those emails, positive or negative, please send them out (feel free to do that personally) so that we can make those updates more meaningful to our community. Thanks, Mohammed

1 0

[ironic][neutron] nf_conntrack_helper now disabled by default
by Derek Higgins 18 Mar '19

18 Mar '19

Hi All, Shortly before the holidays CI jobs moved from xenial to bionic, for Ironic this meant a bunch failures[1], all have now been dealt with, with the exception of the UEFI job. It turns out that during this job our (virtual) baremetal nodes use tftp to download a ipxe image. In order to track these tftp connections we have been making use of the fact that nf_conntrack_helper has been enabled by default. In newer kernel versions[2] this is no longer the case and I'm now trying to figure out the best way to deal with the new behaviour. I've put together some possible solutions along with some details on why they are not ideal and would appreciate some opinions 1. Why not enable the conntrack helper with echo 1 > /proc/sys/net/netfilter/nf_conntrack_helper The router namespace is still created with nf_conntrack_helper==0 as it follows the default the nf_conntrack module was loaded with 2. Enable it in modprobe.d # cat /etc/modprobe.d/conntrack.conf options nf_conntrack nf_conntrack_helper=1 This works but requires the nf_conntrack module to be unloaded if it has already been loaded, for devstack and I guess in the majority of cases (including CI nodes) this means a reboot stage or a potentially error prone sequence of stopping the firewall and unloading nf_conntrack modules. This also globally turns on the helper on the host reintroducing the security concerns it comes with 3. Enable the contrack helper in the router network namespace when it is created[3] This works for ironic CI, but there may be better solutions that can be worked within neutron that I'm not aware of. Of the 3 options above this would be most transparent to other operators as the original behaviour would be maintained. thoughts on any of the above? or better solutions? 1 - https://storyboard.openstack.org/#!/story/2004604 2 - https://kernel.googlesource.com/pub/scm/linux/kernel/git/horms/ipvs-next/+/… 3 - https://review.openstack.org/#/c/628493/1

6 10

[nova][openstack-charms] One day until PTL elections end
by Jeremy Stanley 18 Mar '19

18 Mar '19

Hello Nova, OpenStack_Charms contributors, Just a quick reminder that elections are closing soon, if you haven't already you should use your right to vote and pick your favourite candidate! You have until Mar 19, 2019 23:45 UTC. Thanks for your time! -- Jeremy Stanley

2 1

senlin 7.0.0.0rc1 (stein)
by no-reply＠openstack.org 18 Mar '19

18 Mar '19

Hello everyone, A new release candidate for senlin for the end of the Stein cycle is available! You can find the source code tarball at: https://tarballs.openstack.org/senlin/ Unless release-critical issues are found that warrant a release candidate respin, this candidate will be formally released as the final Stein release. You are therefore strongly encouraged to test and validate this tarball! Alternatively, you can directly test the stable/stein release branch at: https://git.openstack.org/cgit/openstack/senlin/log/?h=stable/stein Release notes for senlin can be found at: https://docs.openstack.org/releasenotes/senlin/ If you find an issue that could be considered release-critical, please file it at: https://bugs.launchpad.net/senlin-tempest-plugin and tag it *stein-rc-potential* to bring it to the senlin release crew's attention.

1 0