November 2019 - openstack-discuss

[nova] splitting "ERROR" into 2 states
by Mohammed Naser 25 Nov '19

25 Nov '19

Hi everyone, This is just a very open ended discussion that might bring up some interesting ideas, either that I'm going the wrong way about this, or perhaps this is something we need to think more about. As we go about increasing the instrumentation of the clouds we run, one of the interesting ideas was to measure the "ERROR" instance rate to see if more VMs than usual are hitting ERROR state. The problem with this right now is that there are a few factors at the moment where an instance can hit an ERROR state which are *not* cause of concern for the operator. For example, if you're booting an instance from a volume and you're at your quota, the instance will fail to boot, end up in "ERROR" state. If we're instrumenting that, we are likely going to be alerted on a high # of ERROR state instances but there's not much that we can do about it realistically. However, if we're getting a lot of ERROR instances because of "NoValidHost" or because some other valid failures such as in libvirt or RBD, then we'd probably want to be alerted on those. Does anyone have any ideas on how we can either better instrument this, or perhaps seeing how inside Nova, we have a "system error" and a "user error" Thanks :) Mohammed

2 1

[neutron] Bug deputy report week of November 18th
by Miguel Lavalle 25 Nov '19

25 Nov '19

Hi, I was the bugs deputy for the week of November 18th. Here's the list of reported bugs: High: - https://bugs.launchpad.net/neutron/+bug/1853223 [VPNaaS]: Python3 RuntimeError: dictionary changed size during iteration Fix in progress: https://review.opendev.org/#/c/695130 - https://bugs.launchpad.net/neutron/+bug/1853603 SG rules not enforced This happens with Kuryr tests. Per https://bugzilla.redhat.com/show_bug.cgi?id=1688323, Nate Johnson is reported working on it in the Red Hat bugzilla Medium: - https://bugs.launchpad.net/neutron/+bug/1853171 Deprecate and remove any "ofctl" code in Neutron and related projects ralonsoh reported it and working on it - https://bugs.launchpad.net/neutron/+bug/1853637 Assign floating IP to port owned by another tenant is not override-able with RBAC policy Reported by Octavia team. Needs owner Low: - https://bugs.launchpad.net/neutron/+bug/1853582 ovs_all_ports option used by ovs cleanup has a misleading help string Requires further investigation: - https://bugs.launchpad.net/neutron/+bug/1853613 VMs don't get ip from dhcp after compute restart Submitter was asked for more information Incomplete: - https://bugs.launchpad.net/neutron/+bug/1853632 designate dns driver does not use domain settings for auth More information requested from submitter. Potential fix proposed: https://review.opendev.org/#/c/695726 Invalid: - https://bugs.launchpad.net/neutron/+bug/1853089 openstack env in Stein, CLI or Dashboard response slowly Report is too vague. More data requested

2 1

[Kolla] Configure docker service fails during bootstrap_servers
by Bhupathi, Ramakrishna 25 Nov '19

25 Nov '19

Folks, Looking for help on this. When I am attempting to install OpenStack Kolla (development mode) I am running into this error when running bootstrap_servers ./kolla-ansible -i ../../multinode bootstrap-servers TASK [baremetal : Configure docker service] **************************************************************************************************** fatal: [localhost]: FAILED! => {"changed": false, "msg": "AnsibleUndefinedVariable: 'ansible.vars.hostvars.HostVarsVars object' has no attribute u'ansible_eth0'"} PLAY RECAP ************************************************************************************************************************************* localhost : ok=29 changed=3 unreachable=0 failed=1 skipped=16 rescued=0 ignored=0 Command failed ansible-playbook -i all-in-one -e @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml -e CONFIG_DIR=/etc/kolla -e kolla_action= bootstrap-servers /home/ubuntu/kolla-env/share/kolla-ansible/ansible/kolla-host.yml Can someone tell me what is going wrong here? --RamaK E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.

3 2

[kolla] PTG summary
by Mark Goddard 25 Nov '19

25 Nov '19

Hi, A couple of weeks ago we had our partially virtual PTG. The full notes are in Etherpad [1], but I will try to summarise some of the discussions here. We had a good turnout, and good engagement in the discussions. # Priorities I'll start at the end, as this is arguably the most tangible outcome of the discussions. At the end of the sessions, we gathered all potential work items for Ussuri, and voted on them in another Etherpad [2]. Afterwards these were cleaned up, ordered, and assigned blueprints. The priorities were transferred to the whiteboard [3] for tracking. Of course other things will come along, and some things won't get done, but these priorities should be used as a guide for anyone reviewing or developing patches during the Ussuri cycle # General topics Sustainability is a key theme in kolla recently. There are two main aspects to this - keeping the community healthy, and ensuring the scope of the project is maintainable. We discussed having some additional community meetings on a regular basis, as a way to keep in touch with operators and to build up more of a community outside of IRC. I proposed having a virtual onboarding session which a number of people expressed an interest in attending. Pulling this session out of the summit should allow for a more level playing field. ## Ceph Ansible We are continuing to investigate Ceph Ansible as an alternative to our native Ceph deployment. The work to migrate from an existing kolla deployment is still ongoing. There are some potential blockers in the form of no Ubuntu container image support in ceph-ansible, and no ARM container images published by the ceph-container project. ## Kolla-cli This project has been restored in the Train cycle, and we continue to keep an eye on its progress. ## Extensibility We discussed making kolla and kolla-ansible more extensible as a way to avoid the need to support every service under the sun. This should be simple in kolla, but kolla-ansible will require more effort to define how custom playbooks or roles are hooked in. ## Inverting kolla images There was a proposal to use more upstream docker images, and potentially add our tooling on top where necessary. This is an interesting idea, but could add complexity with more image distros to track. ## Cloud native logging The cloud-native folk seem to have agreed on capturing container logs from stdout, which doesn't align with our file-based model. We also miss some logs during startup that don't get logged to files which could be interesting. We agreed to try ingesting these into fluentd as a starting point. ## Tracking bug fixes in release notes We agreed to start experimenting with tracking bug fixes in release notes from the Ussuri release. Previously we have just tracked features, deprecations and upgrade notes. This will raise the bar for contribution slightly, so we will keep an eye on it. # Kolla ## CentOS 8 Supporting CentOS 8 base container images is key for us this cycle, as it allows us to move to python 3 based CentOS images. These images may start to break at any moment as services drop python 2 support. The Ussuri release will not support CentOS 7 base container images. We are currently blocked by a number of missing yum repositories for CentOS 8. Supporting running kolla-build on a CentOS 8 host is a simpler task, although getting Docker installed requires a few contortions at this point. ## Drop python 2 This depends on CentOS 8 host and container support. We are heavily at risk of being broken as other projects drop python 2 support. We plan to keep CentOS CI happy for as long as possible but expect it may break at some point during the cycle. ## Zuul proposal bot We talked about adding a zuul proposal bot to update source package versions on stable branches (running tools/version-check.py). I looked into it and put forward a PoC, but we are going to try switching to a YAML format definition before proceeding with this. ## Remove EPEL Everyone likes to hate EPEL, so we will try to remove it. We also discussed an off by default approach for our custom package repositories to provide some damage limitation if they go AWOL. ## Support matrix We made a good start with the support matrix [4] this cycle. We'd like to continue this effort, and continue to evaluate which images we support. The next step seems to be to better define our categories of images, and use these to define voting vs. non-voting build failures. We'd like to find community owners for some of our 'community maintained' images. ## RabbitMQ upgrade RabbitMQ 3.8 brings a prometheus exporter, which a number of people have expressed an interest in. This will require an erlang upgrade. ## Prometheus 2 There is no migration path between prometheus 1 and 2. We discussed a few options for a smooth transition, and I think we landed on this: * keep old prometheus container around, configure it as a remote read source * configure haproxy to flip to the new prometheus when ready # Kolla Ansible ## CentOS 8 This is where it gets interesting. How do we migrate a running system from CentOS 7 to 8? Ideally we would not couple this to an OpenStack upgrade, so at least one release needs to support both CentOS7 and 8 hosts. I will follow up on this topic separately as it's a big one, and I'd like to try a cross-project approach. ## Drop python 2 The main interesting decision here is: don't drop py2 for remote hosts in Ussuri, until we are sure that Ussuri will only need to run against CentOS 8 hosts (see above). ## OVN support Neutron seems to be making moves to deprecate OVS and LinuxBridge ML2 drivers, replacing them with OVN in tree. We have OVN images, but no deployment support in kolla ansible. We'd like to add it this cycle. Interesting questions around migration from OVS to OVN came up. Tripleo has some tooling which might help here. ## More host-level commands (day 2 ops) We have the bootstrap-servers command for bootstrapping hosts, but lack some commands for ongoing operations. Common examples include: * reconfiguring or upgrading docker in a safe manner (without live-restore, a docker restart takes down your containers). * adding new hosts. This requires updating /etc/hosts everywhere, but running bootstrap-servers again is heavy handed and risks a docker restart. Containers don't automatically pick up changes to /etc/hosts, so we need to address that. * pruning docker images ## Restarting services There was a request for a command to restart services. It could probably be cobbled together from existing code quite easily. ## More destruction It should be possible to run the destroy command against a subset of services. We could also do more to thoroughly clean up. ## More security friendliness (especially transport security) * Could we integrate with letsencrypt? Possibly. * Should we default to use TLS with self-signed certs? Probably, but expiry could cause some surprises without explicit buy-in from the operator. * Can we use per-host RabbitMQ usernames and passwords? Potentially... ## SELinux Comes up often, but never gets voted for in our priorities. It would be nice to get this one sorted though. ## Fluentd reconfiguration Currently it's not possible to deploy the common services (cron, kolla-toolbox, fluentd) without also deploying another service. There are a few fiddly details, but it should be possible to resolve. ## Ansible lint We agreed to try running ansible-lint on our codebase. The group has had mixed results with it before, but was open to trying again. ## Ansible maximum version pinning We agreed to define a maximum version of Ansible that we support. This will help to prevent breakage out of our control. ## Nova cells v2 Work continues this cycle on cells with support for shared cell controllers, and deployment of multiple RabbitMQ and MariaDB clusters. ## Config file audit We should use the oslo config validator [5] to ensure our config is valid. ## Podman This one keeps coming up, but we never agree to implement it. Possible issues include a lack of a full-featured Python library, and lack of a supported package for Debian/Ubuntu. We agreed to start thinking about how we might perform a migration from Docker one day, given the direction of Red Hat. [1] https://etherpad.openstack.org/p/kolla-ussuri-ptg [2] https://etherpad.openstack.org/p/kolla-ussuri-priorities [3] https://etherpad.openstack.org/p/KollaWhiteBoard [4] https://docs.openstack.org/kolla/latest/support_matrix.html [5] https://docs.openstack.org/oslo.config/latest/cli/validator.html Well that turned into more of an exhaustive list than I'd expected. Well done for reading (or scrolling) to the end. Cheers, Mark

3 5

[nova] unshelve image_ref bugs
by Alexandre Arents 25 Nov '19

25 Nov '19

Hey all, We are using more and more shelve feature and we recently hit this referenced shelve/unshelve bug: https://bugs.launchpad.net/nova/+bug/1732428 http://lists.openstack.org/pipermail/openstack-dev/2017-December/125124.html To summup the issue: -Once we shelve/unshelve a qcow2 instance, we cannot anymore live-migrate/cold-migrate/resize without breaking it, involving data loss.. ->This is because the unshelved instance have a backing file corresponding to the deleted snapshot(shelve-snapshot), not the original image_ref in glance. So when we migrate the instance, destination host fetch original image from glance, not the one currently used by instance. I've tried to summup possible solution: A) Use patch proposed by Matt (abandonned): https://review.opendev.org/#/c/524726/ This change assume the existence of the snapshot image created by a shelve, that is to say don't delete image during unshelve and update instane.image_ref with it. PROS: -there is no more "hidden image" CONS: -each shelved/unshelved create images in glance(it uses space, can be a problem or not) -It breaks the assumption: "I want my instance remains unchanged after unshelve and keep original image_ref, so when I rebuild my instance, I use the original image (at spawn time), not the one used during last shelve" Note: Matt mention a possible workaround of the rebuild issue by using image_id in in request.spec instead of image_ref[1]) B) Someone propose to rebase backing file [2]: I did not check feasibility/complexity of this ? CONS: -What we do when original image is deleted? C) Change create_image()/imagebackend driver behavior, to create a flatten qcow2 file in case of unshelving. flattening disk may be a solution because there will be no more "orphan backing file". (Basicly doing like "flat" backend driver except we need to stay in qcow2 instead of RAW) PROS: -we keep orignal unshelve behavior/assumption CONS: -It means that in your infra configured in COW some instances will be in "qcow2 flat", Flat qcow2 instance works great (livemigration/resize..). Would all installation ok with that ? Ok it seems a little odd to ask COW driver to not do COW in some case. Alternatilevy we can force using flat driver if unshelving, but we need to change flat driver to support also qcow2. D) During spawn() if unshelving we convert "qcow2 disk with backing file" to a "flatten qcow2 disk", just after self._create_image(). It looks more like a workaround than a long term solution as it need to convert something created before, that do not meet the need(better to do C). E) Any other idea ? Currently to make short term OPS and User happy, we are about to do (D) as it works great in our environment, but we are looking for the project solution. Any recommendation? [1] http://lists.openstack.org/pipermail/openstack-dev/2018-September/134855.ht… [2] https://bugs.launchpad.net/nova/+bug/1732428/comments/6 -- Alexandre Arents aarents

2 2

[cinder] weekly meeting time and location change
by Brian Rosmaita 24 Nov '19

24 Nov '19

Beginning with the 4 December 2019 meeting, the Cinder weekly meeting will be held as follows: Day: Wednesday Time: 1400 UTC Location: #openstack-meeting-4 Agenda: https://wiki.openstack.org/wiki/CinderMeetings Updated ICS file: http://eavesdrop.openstack.org/calendars/cinder-team-meeting.ics Note that the *time* and *IRC chat room* have changed. The day, agenda, meeting log locations, etc., remain the same. Thanks to Liang Fang for initiating this change, which will bring our meeting into a more reasonable time frame for contributors in Asia time zones. cheers, brian

2 1

[qa][form][ptg] QA Summary for Forum & PTG
by Ghanshyam Mann 24 Nov '19

24 Nov '19

Hello Everyone, I am summarizing the QA PTG and Forum sessions held in Shanghai. * OpenStack QA – Project Update: We gave the updates on what we finished on Train and draft plan for the Ussuri cycle. due to fewer contributors in QA, Train cycle activities are decreased as compare to Stein. We tried to maintain the daily QA activity and finished a few important things. Slides are uploaded on the schedule site. * Users / Operators adoption of QA tools / plugins : Etherpad: https://etherpad.openstack.org/p/PVG-forum-qa-ops-user-feedback. This is another useful session for QA to get feedback as well as information about downstream tooling. Few tools we talked about: - Fault injection tests - https://opendev.org/x/tobiko/ - whitebox testing(Whitebox Tempest Plugin) - high availability testing: https://review.opendev.org/#/c/443504/ Run patrole jobs in the neutron upstream(Keystone was interested before) One big concern shared from a few people about a long time to get merged tempest patches. One idea to solve this is to bring critical reviews in Office hours. QA PTG: It was a small gathering this time for one day for PTG on Wednesday. Even with the small number of developers, we had good discussions on many topics. Etherpad: https://etherpad.openstack.org/p/shanghai-ptg-qa * Train Retrospective: Retrospective bought up the few key issues where we need improvement. We collected the below action items including bug triage. Untriage QA bugs are increasing day by day. Action: - need to discuss blacklist plugins and how to notify and remove them if dead - gmann - start the process of community-goal work in QA - masayuki - sprint for bug triage with number of volunteers - - (chandankumar)Include one bug in each sprint in TripleO CI tempest member - Traige the new bug and then pick the bug based on priority - For tripleo Ci team we will track here: https://tree.taiga.io/project/tripleo-ci-board/ - chandankumar * How to deal with an aging testing stack: With testtools being not so active, we need to think on the alternate or best suitable options to solve this issue. We discussed the few options which need to be discussed further on ML. - Can we fork the dependecies of testtools in Temepst or stestr ? - As we are removing the py2.7 support in tempest, we can completly ignore/remove the unittest2 things but that is not case for testtools ? - Similar example done in different project: https://github.com/plone/Products.CMFPlone/issues/1882 - Remove the support of unittest2 from testtools ? py2.7 is going away from everywhere and testools can create tag or something for py2.7 usage ? - Since Python2 is going EOL on 01st Jan, 2020, so let's create a tag and remove the unitest2 with unitest for python3 release only Action: - Document the official supported test runner by Tempest. - Soniya Vyas/Chandan Kumar - ML to discuss the above options - gmann * Remove/migrate the .testr.conf to .stestr: 60 openstack/* repositories have .stestr.conf AND .testr.conf. We don't need to have both files at least. Let's take a look some of them and make a plan to remove if we can. - http://paste.openstack.org/show/785832/ - If both exist then remove the .testr.conf and Then verify that .stestr conf has the correct test path. If only .testr.conf then migrate to stestr.conf - We need to figure out the purpose of pbr .testr.conf code before removing. Is this just old codes or necessary? - https://opendev.org/openstack/pbr/src/branch/master/pbr/hooks/commands.py#L… - https://opendev.org/openstack/pbr/src/branch/master/pbr/testr_command.py * Moving subunit2html script from os-testr: Since os-testr runner piece in os-testr project is deprecated but subunit2html project still exists there, it is widely used across the OpenStack ecosystem, Can we move to somewhere else? I do not find any benefits to move those scripts to other places. We asked chandan to open an issue on stestr to discuss moving to stestr repo. mtreinish replied on this: os-testr meant to be the place in openstack that we could host the ostestr runner wrapper/script subunit2html, generate_subunit, etc. Just because ostestr is deprecated and being removed doesn't mean it's not the proper home for those other tools. * Separate integrated services tests can be used in TriplO CI: TriplO CI maintains a separate file to run dependent tests per service. Tempest has dependent services tox and integrated jobs and the same can be used in TriplO CI. For example: - blacklist file for networking testing [1]. - tox for networking[2]. * RBAC testing strategy: This was a cross-project strategy for positive/negative testing for system scope and new defaults in keystone. Keystone has implemented the new defaults and system scope in its policy and added a unit test to cover the new policies. Nova is implementing the same in Ussuri cycle. As discussed in Denver PTG also, Tempest will implement the new credential for all 9 personas available in keystone. Slowly migrate the tests start using the new policies. That will be done via a flag switching Tempest to use system scope or new defaults and that flag will be false to keep using the old policies for stable branch testing. We can use patrole tests or implement new tests in the Tempest plugin and verify the response. Both have the issue of performing the complete operation which is not required always for policy verification. Running full functional tests is expensive and duplicates existing tests. One solution for that (we talked about it in Denver PTG also) is via some flag like os-profiler by just do the policy check and return the API response with specific return code. AGREE: - Tempest to provide the all 9 personas available from keystone. Slowly migrate Tempest existing tests to run with new policies. - We agreed to have two ways to test the policy: - Tempest like tests in tempest plugins with the complete operation and verify the things on response, not just policy return code. It depends on the project if they want to implement such tests. - Unit/Functional tests on the project's side. - Document the both way so that project can adopt the best suitable one. * How to remove tempest plugin sanity BLACKLIST: We have Tempest plugin blacklist. It should be removed in the future if possible. Some of them shouldn't be as a tempest-plugin because they're just neutron studium things which already moved to neutron-tempest-plugin but still exiting in repo also. Some of them are less active. Remove below plugins from BLACKLIST: - x/group-based-policy - https://opendev.org/x/group-based-policy - openstack/networking-generic-switch needs to be checked (setup.py/cfg?) Action: - Add the start date in blacklist doc so that we can know how long a plugin is blacklisted. - After 60 days: we send the email notification to openstack-discuss, PTL, maitainer and TC to either fix it or remove it from the governance. * Python 2.7 drop plan: We discussed the next steps to drop the py2 from Tempest and other QA tools. AGREE: - Will be doing before milestone 2 - Create a new tag for python-2.7 saying it is the last tag and document that the Tempest tag needs Train u-c. - Test the Tempest tag with Train u-c, if fail then we will disucss. - TripleO and OSA is going to use CentOS 8 for train and master * Adding New glance tests to Tempest: We discussed on testing the new glance v2 api and feature. Below are the glance features and agreed points on how to test them. - Hide old images: Test can be added in Tempest. Hide the image and try to boot the server from the image in scenario tests. - Delete barbican secrets from glance images: This test belongs to barbican-tempest-plugin which can be run as part of the barbican gate using an existing job. Running barbican job on glance gate is not required, we can add a new job (multi stores) on glance gate which can run this + other new features tests. - Multiple stores: DevStack patch is already up, add a new zuul job to set up multiple stores and run on the glance gate with api and scenario tests. gmann to setup the zuulv3 job for that. * Tempest volunteers for reviewing patches: We've noticed that the amount of merged patches in October is less than in September and much less than it was during the summer. This has been brought in feedback sessions also. There is no perfect solution for this. Nowadays QA has less active core developers. We encourage people to bring up the critical or stuck patches in office hours. * Improving Tempest cleanup: Tempest cleanup is not so stable and not a perfect design. We have spec up to redesign that but could not get a consensus on that. I am ok to move with resource prefix with UUID. We should extend the cleanup tool for plugins also. * Ussuri Priority & Planning: This was the last session for the PTG which could not happen on Wed due to strict time-up policy of the conference place which I really liked. Time-based working is much needed for IT people :). We met on Thursday morning in coffee area and discussed about priority for Ussuri cycle. QA Ussuri Priority Etherpad[3] has the priority items with the assignee. [1] https://github.com/openstack/tempest/blob/9cdd5250615bb6ab26a1a9a80743a03cc… [2] https://github.com/openstack/tempest/blob/9cdd5250615bb6ab26a1a9a80743a03cc… [3] https://etherpad.openstack.org/p/qa-ussuri-priority -gmann

1 0

[cinder] Virtual PTG reminder--Monday 25 Nov at 1500 UTC
by Brian Rosmaita 24 Nov '19

24 Nov '19

This is a quick reminder that we're having the first session of the Cinder Virtual PTG for Ussuri from 1500-1700 UTC on Monday 25 November. We'll meet in this BlueJeans room: https://bluejeans.com/3228528973 The session will be recorded. The rough schedule is here: https://etherpad.openstack.org/p/cinder-ussuri-virtual-ptg-planning It's not too late to add a topic if there's something you'd like to have discussed. The second session is Wednesday 27 November from 1500-1700 UTC. See you there! brian

1 0

[cinder] Ussuri Shanghai PTG summary
by Brian Rosmaita 24 Nov '19

24 Nov '19

Sorry for the delay. A summary of what happened at the Cinder part of the Shanghai PTG is available: https://wiki.openstack.org/wiki/CinderUssuriPTGSummary Some topics are flagged for follow-up at this week's Cinder Virtual PTG, but any topics that you think could use more discussion are fair game to add to the Virtual PTG planning etherpad: https://etherpad.openstack.org/p/cinder-ussuri-virtual-ptg-planning The list of topics for the Virtual PTG is unordered right now. I'll send an email out tonight (New York time) when I've got them ordered so you'll know what to expect (roughly). cheers, brian

1 0

[ptg][neutron] Ussuri PTG summary
by Slawek Kaplonski 24 Nov '19

24 Nov '19

Hi Neutron team, First if all thank to all of You for great and very productive week during the PTG in Shanghai. Below is summary of our discussions from whole 3 days. If I forgot about something, please respond to the email and update missing informations. But if You want to have follow up discussion about one of the topics from this summary, please start a new thread to keep this one only as high level summary of the PTG. On boarding =========== Slides from onboarding session can be found at [1] If You have any follow up questions to us about onboarding, or You need help with starting any work in Neutron team, please contact me or Miguel Lavalle by email or on IRC. My IRC nick is slaweq and Miguel's nick is mlavalle. We are available on #openstack-neutron channel @freenode. Train retrospective =================== Good things in Train cycle: * working with this team is still good experience * core team is stable, and we didn't lost any core reviewers during the cycle, * networking is still one of key reasons why people use OpenStack Not good things: * dimished vitality in stadium projects - we had also forum session and follow discussion about this later during the PTG, * gate instability - we have seen many issues which were out of our control, like infra problems, grenade jobs failures, other projects failures, but also many bugs on our side, * we have really a lot of jobs in our check/gate queue. If each of them is failing 5% of times, it's hard to merge any patch as almost every time, one of jobs will fail. Later during the PTG we also discussed that topic and we were looking for some jobs which we maybe can potentially drop from our queues. See below for summary about that, Action items/improvements: * many team meetings each week. We decided to limit number of meetings by: ** consolidate performance subteam meeting into weekly team meeting - this topic will be added to the team meeting's agenda for team meetings on Monday, ** consolidate ovn convergence meeting into weekly team meeting - this topic will be added to the team meeting's agenda for team meetings on Tuesday, ** we need to check if QoS subteam meeting is still needed, * Review process: list of actual review priorities would be useful for the team, we will add "Review-Priority" label to the Neutron reviews board and try to use it during the Ussuri cycle. Openvswitch agent enhancements ============================== We had bunch of topics related to potential improvements for neutron-openvswitch-agent proposed mostly by Liu Yulong. Slides with his proposals are available at [2]. * retire DHCP agent - resyncs of DHCP agent are problematic, especially when agent hosts many networks. Proposal was to add new L2 agent's extension which could be used instead of "regular" DHCP agent and to provide only basic DHCP functionalities. Such solutions would work in the way quite similar to how networking-ovn works today but we would need to implement and maintain own dhcp server application. Problems of this solution are: ** problems with compatibility e.g. with Ironic, ** how it would work with mixed deployments, e.g. with ovs and sriov agents, ** support for dhcp options, Advantages of this solution: ** fully distributed DHCP service, ** no DHCP agents, so less RPC messages on the bus and easier maintanance of the agents, Team's feedback for that is that this is potentially nice solution which may helps in some specific, large scale deploymnets. We can continue discussion about this during Ussuri cycle for sure. * add accepted egress fdb flows We agreed that this is a bug and we should continue work on this to propose some way to fix it. Solution proposed by LIU during this discussion wasn't good as it could potentially break some corner cases. * new API and agent for L2 traffic health check The team asked to add to the spec some more detailed and concrete use cases with explanation how this new API may help operator of the cloud to investigate where the problem actually is. * Local flows cache and batch updating The team agreed that as long as this will be optional solution which operator can opt-in we can give it a try. But spec and discuss details there will be necessary. * stop processing ports twice in ovs-agent We all agreed that this is a bug and should be fixed. But we have to be careful as fixing this bug may cause some other problems e.g. with live-migration - see nova-neutron cross project session. * ovs-agent: batch flow updates with --bundle We all agreed that this can be done as an improvement of existing code. Similar option is already used in openvswitch firewall driver. Neutron - Cyborg cross project session ====================================== Etherpad for the session is at [3]. Cyborg team wants to include Neutron in workflow of spawning VMs with Smart NICs or accelerator cards. From Neutron's side, required change is to allow including "accel" data in port binding profile. As long as this will be well documented what can be placed there, there should be no problem with doing that. Technically we can place almost anything there. Neutron - Kuryr cross project session ===================================== Etherpad for the session is at [4]. Kuryr team proposed 4 improvements for Neutron which would help a lot Kuryr. Ideas are: * Network cascade deletion, * Force subport deletion, * Tag resources at creation time, * Security group creation with rules & bulk security group rule creation All of those ideas makes sense for Neutron team. Tag resources at creation time is even accepted rfe already - see [5] but there was no volunteer to implement it. We will add it to list of our BPs tracked weekly on team meeting. Miguel Lavalle is going to take a look at it during this cycle. For other proposals we need to have RFEs reported first. Starting the process of removing ML2/Linuxbridge ================================================ Currently in Neutron tree we have 4 drivers: * Linuxbridge, * Openvswitch, * macvtap, * sriov. SR-IOV driver is out of discussion here as this driver is addressing slightly different use case than other out drivers. We started discussion about above topic because we don't want to end up with too many drivers in-tree and we also had some discussions (and we have spec for that already) about include networking-ovn as in-tree driver. So with networking-ovn in-tree we would have already 4 drivers which can be used on any hardware: linuxbridge, ovs, macvtap and ovn. Conclusions from the discussion are: * each driver requires proper testing in the gate, so we need to add many new jobs to our check/gate queue, * currently linuxbridge driver don't have a lot of development and feature parity gaps between linuxbridge and ovs drivers is getting bigger and bigger (e.g. dvr, trunk ports), * also macvtap driver don't have a lot of activity in last few cycles. Maybe this one could be also considered as candidate to deprecation, * we need to have process of deprecating some drivers and time horizon for such actions should be at least 2 cycles. * we will not remove any driver completly but rather we will move it to be in stadium process first so it still can be maintained by people who are interested in it. Actions to do after this discussion: * Miguel Lavalle will contact RAX and Godaddy (we know that those are Linuxbridge users currently) to ask about their feedback about this, * if there are any other companies using LB driver, Nate Johnston is willing to help conctating them, please reach to him in such case. * we may ratify marking linuxbridge as deprecated in the team meeting during Ussuri cycle if nothing surprising pops in. Encrypted(IPSec) tenant networks ================================ Interesting topic proposed but we need to have RFE and spec with more detailed informations about it to continue discussions. Medatada service over IPv6 ========================== This is continuation of old RFE [6]. The only real problem is to choose proper IPv6 address which will be well known address used e.g. by cloud-init. Original spec proposed fe80::a9fe:a9fe as IPv6 address to access metadata service. We decided to be bold and define the standard. Bence Romsics and Miguel Lavalle volunteered to reach out to cloud-init maintainers to discuss that. walkthrough of OVN ================== Since some time we have in review spec about ml2/ovs and ovn convergence. See [7] for details. List of parity gaps between those backends is available at [8]. During the discussion we talked about things like: * migration from ml2/ovs to ml2/ovn - some scripts are already done in [9], * migration from ml2/lb to ml2/ovn - there was no any work done in this topic so far but it should be doable also if someone would need it and want to invest own time for that, * include networking-ovn as in-tree neutron driver and reasons why it could be good idea. Main reasons of that are: ** that would help growing networking-ovn community, ** would help to maintain a healthy project team, ** the default drivers have always been in-tree, However such inclusion may also hurt modularity/logical separation/dependency management/packaging/etc so we need to consider it really carefully and consider all points of view and opinions. Next action item on this topic is to write more detailed summary of this topic and send it to ML and ask wider audience for feedback. IPv6 devstack tempest test configuration vs OVN =============================================== Generally team supports idea which was described during this session and we should change sligtly IPv6 config on e.g. devstack deployments. Neutron - Edge SIG session ========================== We discussed about RFE [10]. This will require also changes on placement side. See [11] for details. Also some cyborg and ovn related changes may be relevant to topics related to Edge. Currently specs which we have are only related to ML2/OVS solution. Neutron - Nova cross project session ==================================== Etherpad for this session is on [12]. Summary written already by gibi can be found at [13]. On [14] You can find image which shows in visual way problem with live-migration of instances with SR-IOV ports. Policy handling in Neutron ========================== The goal of the session was to plan on Neutron's side similar effort to what services like nova are doing now to use new roles like reader and scopes, like project, domain, system provided by Keystone. Miguel Lavalle volunteered to work on this for Neutron and to be part of popup team for cross project collaboration on this topic. Neutron performance improvements ================================ Miguel Lavalle shown us his new profiling decorator [15] and how we all can use it to profile some of API calls in Neutron. Reevaluate Stadium projects =========================== This was follow up discussion after forum session. Notes from forum session can be found at [16]. Nate also prepared some good data about stadium projects activity in last cycles. See at [17] and [18] for details. We all agreed that projects which are in (relatively) good condition now are: * networking-ovn, * networking-odl, * ovsdbapp Projects in bad condition are other projects, like: * neutron-interconnection, * networking-sfc, * networking-bagpipe/bgpvpn, * networking-midonet, * neutron-fwaas and neutron-fwaas-dashboard, * neutron-dynamic-routing, * neutron-vpnaas and neutron-vpnaas-dashboard, We decided to immediately remove neutron-interconnection project as it was never really implemented. For other of those projects, we will send emails to ML to ask for potential maintainers of those projects. If there will be no any volunteers to maintain some of those projects, we will deprecated them and move to "x/" namespace in 2 cycles. Floating IP's On Routed Networks ================================ There is still interest of doing this. Lajos Katona started adding some scenario tests for routed networks already as we need improved test coverage for this feature. Miguel Lavalle said that he will possibly try to work on implementing this in Ussuri cycle. L3 agent enhancement ==================== We talked about couple potential improvements of existing L3 agent, all proposed by LIU Yulong. * retire metering-agent It seems that there is some interest in metering agent recently so we shouldn't probably consider of retiring it for now. We also talked about adding new "tc based" driver to the metering agent and this discussion can be continue on rfe bug [19]. * Centralized DNAT (non-DVR) traffic (floating IP) Scale-out This is proposal of new DVR solution. Some details of this new solution are available at [20]. We agreed that this proposal is trying to solve some very specific use case, and it seems to be very complicated solution with many potential corner cases to address. As a community we don't want to introduce and maintain such complicated new L3 design. * Lazy-load agent side router resources when no related service port Team wants to see RFE with detailed description of the exact problem which this is trying to solve and than continue discussion on such RFE. Zuul jobs ========= In this session we talked about jobs which we can potentially promote to be voting (and we didn't found any of such) and about jobs which we maybe can potentially remove from our queues. Here is what we agreed: * we have 2 iptables_hybrid jobs - one on Fedora and one on Ubuntu - we will drop one of those jobs and left only one of them, * drop neutron-grenade job as it is running still on py27 - we have grenade-py3 which is the same job but run on py36 already, * as it is begin of the cycle, we will switch in devstack neutron uwsgi to be default choice and we will remove "-uwsgi" jobs from queue, * we should compare our single node and multinode variants of same jobs and maybe promote multinode jobs to be voting and then remove single node job - I volunteered to do that, * remove our existing experimental jobs as those jobs are mostly broken and nobody is run those jobs in experimental queue actually, * Yamamoto will check failing networking-midonet job and propose patch to make it passing again, * we will change neutron-tempest-plugin jobs for branch in EM phase to always use certain tempest-plugin and tempest tag, than we will remove those jobs from check and gate queue in master branch, Stateless security groups ========================= Old RFE [21] was approved for neutron-fwaas project but we all agreed that this should be now implemented for security groups in core Neutron. People from Nuage are interested in work on this in upstream. We should probably also explore how easy/hard it will be to implement it in networking-ovn backend. Old, stagnant specs =================== During this session we decided to abandon many of old specs which were proposed long time ago and there is currently no any activity and interest in continue working on them. If anyone would be interested in continue work on some of them, feel free to contact neutron core team on irc or through email and we can always reopen such patch. Community Goal things ===================== We discussed about currently proposed community goals and who can take care of which goal on Neutron's side. Currently there are proposals of community goals as below: * python3 readiness - Nate will take care of this, * move jobs definitions to zuul v3 - I will take care of it. In core neutron and neutron-tempest-plugin we are (mostly) done. On stadium projects' side this will require some work to do, * Project specific PTL and contributor guides - Miguel Lavalle will take care of this goal as former PTL, We will track progress of community goals weekly in our team meetings. Neutron-lib =========== As some time ago our main neutron-lib maintainer (Boden) leaved from the project, we need some new volunteers to continue work on it. Todo list is available on [22]. This should be mostly important for people who are maintaining stadium projects or some 3rd party drivers/plugins so if You are doing things like that, please check list from [22] and reach out to us on ML or #openstack-neutron IRC channel. [1] https://www.slideshare.net/SawomirKaposki/neutron-on-boarding-room [2] https://github.com/gotostack/shanghai_ptg/blob/master/shanghai_neutron_ptg_… [3] https://etherpad.openstack.org/p/Shanghai-Neutron-Cyborg-xproj [4] https://etherpad.openstack.org/p/kuryr-neutron-nice-to-have [5] https://bugs.launchpad.net/neutron/+bug/1815933 [6] https://bugs.launchpad.net/neutron/+bug/1460177 [7] https://review.opendev.org/#/c/658414/ [8] https://etherpad.openstack.org/p/ML2-OVS-OVN-Convergence [9] https://github.com/openstack/networking-ovn/tree/master/migration [10] https://bugs.launchpad.net/neutron/+bug/1832526 [11] http://lists.openstack.org/pipermail/openstack-discuss/2019-October/009991.… [12] https://etherpad.openstack.org/p/ptg-ussuri-xproj-nova-neutron [13] http://lists.openstack.org/pipermail/openstack-discuss/2019-November/010654… [14] https://imgur.com/a/12PrQ9W [15] https://review.opendev.org/678438 [16] https://etherpad.openstack.org/p/PVG-Neutron-stadium-projects-the-path-forw… [17] https://ethercalc.openstack.org/neutron-stadium-train-metrics [18] https://ibb.co/SBzDGdD [19] https://bugs.launchpad.net/neutron/+bug/1817881 [20] https://imgur.com/a/6MeNUNb [21] https://bugs.launchpad.net/neutron/+bug/1753466 [22] https://etherpad.openstack.org/p/neutron-lib-volunteers-and-punch-list -- Slawek Kaplonski Senior software engineer Red Hat

7 8