November 2019 - openstack-discuss

[neutron] [all] Networking-ovn and neutron convergence
by Brian Haley 24 Nov '19

24 Nov '19

Hi, For some time we have been discussing in the Neutron community the possibility of including the networking-ovn driver [1] as one of the in-tree Neutron drivers. There is already a spec [2] describing in detail why we want to do this and why we think it is good idea. We also discussed this during the PTG in Shanghai within our team [3], and had a discussion at the ops-meetup as well [4]. The OVN backend is free of many well-known issues which are impacting the existing ML2/OVS reference implementation today with the OVS agent. For example, OVN provides: * Control plane performance optimizations by not using rabbitmq; * DVR (and HA) by default, based on OpenFlow so it can be easy offloaded e.g. by SmartNICs; * Distributed DHCP; There are some feature parity gaps when comparing it to ML2/OVS that we plan to address. See [2] for details. We think that merging this code into the neutron repository will help to grow the networking-ovn community and will help us to keep a healthy Neutron team as well by increasing the number or contributors. Our current plan is to start merging code from networking-ovn into the neutron repository as soon as possible in the Ussuri cycle. But we also wanted to get any additional opinions from the wider community about this plan. What do users and operators of Neutron think about this? [1] https://opendev.org/openstack/networking-ovn [2] https://review.opendev.org/#/c/658414/ [3] https://etherpad.openstack.org/p/Shanghai-Neutron-Planning-restored Lines 252 - 295 [4] https://etherpad.openstack.org/p/shanghai-ptg-ops-meetup Lines 62 - 69 Thanks for any feedback, -Brian (and Slawek)

5 4

[all][neutron][networking-bagpipe][networking-bgpvpn] Maintainers needed
by Slawek Kaplonski 24 Nov '19

24 Nov '19

Hi, Over the past couple of cycles we have noticed that new contributions and maintenance efforts for networking-bagpipe and networking-bgpvpn were almost non existent. This impacts patches for bug fixes, new features and reviews. The Neutron core team is trying to at least keep the CI of this project healthy, but we don’t have enough knowledge about the details of the <stadium project name> code base to review more complex patches. During the PTG in Shanghai we discussed that with operators and TC members during the forum session [1] and later within the Neutron team during the PTG session [2]. During these discussions, with the help of operators and TC members, we reached the conclusion that we need to have someone responsible for maintaining those projects. This doesn’t mean that the maintainer needs to spend full time working on those projects. Rather, we need someone to be the contact person for the project, who takes care of the project’s CI and review patches. Of course that’s only a minimal requirement. If the new maintainer works on new features for the project, it’s even better :) If we don’t have any new maintainer(s) before milestone Ussuri-2, which is Feb 10 - Feb 14 according to [3], we will need to mark networking-bgpvpn and networking-bagpipe as deprecated and in “V” cycle we will propose to move the projects from the Neutron stadium, hosted in the “openstack/“ namespace, to the unofficial projects hosted in the “x/“ namespace. So if You are using this project now, or if You have customers who are using it, please consider the possibility of maintaining it. Otherwise, please be aware that it is highly possible that the project will be deprecated and moved out from the official OpenStack projects. [1] https://etherpad.openstack.org/p/PVG-Neutron-stadium-projects-the-path-forw… [2] https://etherpad.openstack.org/p/Shanghai-Neutron-Planning-restored - Lines 379-421 [3] https://releases.openstack.org/ussuri/schedule.html -- Slawek Kaplonski Senior software engineer Red Hat

2 2

[neutron][ovn] networking-ovn-metadata-agent and neutron agent liveness
by Chris Apsey 23 Nov '19

23 Nov '19

All, Currently experimenting with networking-ovn (rdo/train packages on centos7) and I've managed to cobble together a functional deployment with two exceptions: metadata agents and agent liveness. Ref: the metadata issues, it appears that the local compute node ovsdb server listens on a unix socket at /var/run/openvswitch/db.sock as openvswitch:hugetlbfs 0750. Since networking-ovn-metadata-agent runs as neutron, it's not able to interact with the local ovs database and gets stuck in a restart loop and complains about the inaccessible database socket. If I edit the systemd unit file and let the agent run as root, it functions as expected. This obviously isn't a real solution, but indicates to me a possible packaging bug? Not sure what the correct mix of permissions is, or if the local database should be listening on tcp:localhost:6640 as well and that's how the metadata agent should connect. The docs are sparse in this area, but I would imagine that something like the metadata-agent should 'just work' out of the box without having to change systemd unit files or mess with unix socket permissions. Thoughts? Secondly, ```openstack network agent list``` shows that all agents (ovn-controller) are all dead, all the time. However, if I display a single agent ```openstack network agent show $foo```, it shows as live. I looked around and saw some discussions about getting networking-ovn to deal with this better, but as of now the agents are reported as dead consistently unless they are explicitly polled, at least on centos 7. I haven't noticed any real impact, but the testing I'm doing is small scale. Other than those two issues, networking-ovn is great, and based on the discussions around possibly deprecating linuxbridge as an in-tree driver, it would make a great 'default' networking configuration option upstream, given the docs get cleaned up. Thanks in advance, r Chris Apsey

2 2

[kolla] all-in-one install with swift failing
by Ryan Phillips 23 Nov '19

23 Nov '19

Hi All, I've been trying to install Kolla (train) as an all-in-one install. The base install works great: I can get to the dashboard and create a nova instance. I am attempting to add swift support and followed the directions to configure swift. The problem is all the swift containers are coming up and exiting with: + sudo -E kolla_set_configs sudo: no tty present and no askpass program specified + sudo -E kolla_set_configs sudo: no tty present and no askpass program specified + sudo -E kolla_set_configs sudo: no tty present and no askpass program specified + sudo -E kolla_set_configs sudo: no tty present and no askpass program specified Does anyone know what is wrong? Regards, Ryan CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES cea6edf16348 kolla/ubuntu-binary-horizon:train "dumb-init --single-…" 7 minutes ago Up 7 minutes horizon 6c88d426a07d kolla/ubuntu-binary-heat-engine:train "dumb-init --single-…" 7 minutes ago Up 7 minutes heat_engine c28e5ce612dc kolla/ubuntu-binary-heat-api-cfn:train "dumb-init --single-…" 7 minutes ago Up 7 minutes heat_api_cfn bfc6cdd6b969 kolla/ubuntu-binary-heat-api:train "dumb-init --single-…" 7 minutes ago Up 7 minutes heat_api aa5c37b09733 kolla/ubuntu-binary-neutron-metadata-agent:train "dumb-init --single-…" 8 minutes ago Up 8 minutes neutron_metadata_agent 07154a5af33b kolla/ubuntu-binary-neutron-l3-agent:train "dumb-init --single-…" 8 minutes ago Up 8 minutes neutron_l3_agent 17e2265f8d34 kolla/ubuntu-binary-neutron-dhcp-agent:train "dumb-init --single-…" 8 minutes ago Up 8 minutes neutron_dhcp_agent 1fdb7bea5434 kolla/ubuntu-binary-neutron-openvswitch-agent:train "dumb-init --single-…" 8 minutes ago Up 8 minutes neutron_openvswitch_agent e1fe4fffe34d kolla/ubuntu-binary-neutron-server:train "dumb-init --single-…" 8 minutes ago Up 8 minutes neutron_server cc20097e0787 kolla/ubuntu-binary-openvswitch-vswitchd:train "dumb-init --single-…" 9 minutes ago Up 9 minutes openvswitch_vswitchd 3cd5b3ac32a9 kolla/ubuntu-binary-openvswitch-db-server:train "dumb-init --single-…" 9 minutes ago Up 9 minutes openvswitch_db 22684295751d kolla/ubuntu-binary-nova-compute:train "dumb-init --single-…" 10 minutes ago Up 10 minutes nova_compute 535e6c81e8f5 kolla/ubuntu-binary-nova-libvirt:train "dumb-init --single-…" 10 minutes ago Up 10 minutes nova_libvirt 59e06d362509 kolla/ubuntu-binary-nova-ssh:train "dumb-init --single-…" 10 minutes ago Up 10 minutes nova_ssh 575ea9e95e33 kolla/ubuntu-binary-nova-novncproxy:train "dumb-init --single-…" 10 minutes ago Up 10 minutes nova_novncproxy 150d2d5723e6 kolla/ubuntu-binary-nova-conductor:train "dumb-init --single-…" 10 minutes ago Up 10 minutes nova_conductor e6cd7b383972 kolla/ubuntu-binary-nova-api:train "dumb-init --single-…" 10 minutes ago Up 10 minutes nova_api af9df9d41b0f kolla/ubuntu-binary-nova-scheduler:train "dumb-init --single-…" 10 minutes ago Up 10 minutes nova_scheduler 1892ab9d3d19 kolla/ubuntu-binary-placement-api:train "dumb-init --single-…" 12 minutes ago Up 12 minutes placement_api eed26f6ed1b2 kolla/ubuntu-binary-glance-api:train "dumb-init --single-…" 13 minutes ago Up 13 minutes glance_api 983b379c9d52 kolla/ubuntu-binary-swift-proxy-server:train "dumb-init --single-…" 13 minutes ago Restarting (1) 55 seconds ago swift_proxy_server 0a7e6a59b493 kolla/ubuntu-binary-swift-object-expirer:train "dumb-init --single-…" 13 minutes ago Restarting (1) Less than a second ago swift_object_expirer d426ebe62e71 kolla/ubuntu-binary-swift-object:train "dumb-init --single-…" 13 minutes ago Restarting (1) 3 seconds ago swift_object_updater 8aa837f99b40 kolla/ubuntu-binary-swift-object:train "dumb-init --single-…" 13 minutes ago Restarting (1) 4 seconds ago swift_object_replicator 06e5000f82ce kolla/ubuntu-binary-swift-object:train "dumb-init --single-…" 13 minutes ago Restarting (1) 4 seconds ago swift_object_auditor ac17c224738b kolla/ubuntu-binary-swift-object:train "dumb-init --single-…" 14 minutes ago Restarting (1) 4 seconds ago swift_object_server fca502a360fb kolla/ubuntu-binary-swift-container:train "dumb-init --single-…" 14 minutes ago Restarting (1) 8 seconds ago swift_container_updater 8def0c6efde9 kolla/ubuntu-binary-swift-container:train "dumb-init --single-…" 14 minutes ago Restarting (1) 8 seconds ago swift_container_replicator e2aa84ebed7d kolla/ubuntu-binary-swift-container:train "dumb-init --single-…" 14 minutes ago Restarting (1) 8 seconds ago swift_container_auditor e7aabf20c11a kolla/ubuntu-binary-swift-container:train "dumb-init --single-…" 14 minutes ago Restarting (1) 11 seconds ago swift_container_server cc754157a930 kolla/ubuntu-binary-swift-account:train "dumb-init --single-…" 14 minutes ago Restarting (1) 14 seconds ago swift_account_reaper b3ef7952d2d3 kolla/ubuntu-binary-swift-account:train "dumb-init --single-…" 14 minutes ago Restarting (1) 14 seconds ago swift_account_replicator 7b984325b708 kolla/ubuntu-binary-swift-account:train "dumb-init --single-…" 14 minutes ago Restarting (1) 14 seconds ago swift_account_auditor 06454711e2f9 kolla/ubuntu-binary-swift-account:train "dumb-init --single-…" 14 minutes ago Restarting (1) 14 seconds ago swift_account_server 226002e50961 kolla/ubuntu-binary-swift-rsyncd:train "dumb-init --single-…" 14 minutes ago Restarting (1) 19 seconds ago swift_rsyncd 06ede7f32329 kolla/ubuntu-binary-keystone-fernet:train "dumb-init --single-…" 18 minutes ago Up 18 minutes keystone_fernet b6f258bfee93 kolla/ubuntu-binary-keystone-ssh:train "dumb-init --single-…" 18 minutes ago Up 18 minutes keystone_ssh c237390ab2d3 kolla/ubuntu-binary-keystone:train "dumb-init --single-…" 18 minutes ago Up 18 minutes keystone 9d55b6233854 kolla/ubuntu-binary-rabbitmq:train "dumb-init --single-…" 18 minutes ago Up 18 minutes rabbitmq 037db9b550d9 kolla/ubuntu-binary-memcached:train "dumb-init --single-…" 19 minutes ago Up 19 minutes memcached 747026ed2053 kolla/ubuntu-binary-mariadb:train "dumb-init -- kolla_…" 19 minutes ago Up 19 minutes mariadb 07148ecdfbd9 kolla/ubuntu-binary-chrony:train "dumb-init --single-…" 19 minutes ago Up 19 minutes chrony cc6883333991 kolla/ubuntu-binary-cron:train "dumb-init --single-…" 19 minutes ago Up 19 minutes cron 2eb7cb285418 kolla/ubuntu-binary-kolla-toolbox:train "dumb-init --single-…" 20 minutes ago Up 19 minutes kolla_toolbox b21c793b31d3 kolla/ubuntu-binary-fluentd:train "dumb-init --single-…" 20 minutes ago Up 20 minutes fluentd

2 2

[ops] topics for London ops meetup jan 202
by Chris Morgan 22 Nov '19

22 Nov '19

https://twitter.com/osopsmeetup/status/1197920312555429889?s=20 -- Chris Morgan <mihalis68(a)gmail.com>

1 0

[oslo] Reminder: Virtual PTG on Monday the 25th
by Ben Nemec 22 Nov '19

22 Nov '19

Time: 1500-1700 UTC (starts at the regular meeting time) Etherpad: https://etherpad.openstack.org/p/oslo-shanghai-topics Location: https://meet.jit.si/oslo-ptg (note that I've had trouble using Firefox with Jitsi, Chrome seems to work fine) I look forward to seeing everyone! -Ben

1 0

[dev] Upgrading flake8 to support f-strings
by Thierry Carrez 22 Nov '19

22 Nov '19

Hi everyone, TL;DR: is it time for us to update flake8 and break the world? Long version: Ussuri-supported Python runtimes will be python 3.6 and 3.7, which opens the marvelous world of f-strings to us. For those not familiar with f-strings, I recommend reading: https://realpython.com/python-f-strings/ In a recent project[1] I tried to use them, only to get pep8 job failures[2]. The old flake8 version we are using in hacking is exploding trying to parse f-strings, with a cryptic "AttributeError: 'FlakesChecker' object has no attribute 'JOINEDSTR'" error. Of course that was long-fixed in pyflakes (>=1.4.0) but that version is only used starting in flake8>=3.3.0, and hacking is capping flake8<2.7.0. We do cap those aggressively for a reason: bumping that cap triggers pep8 job failures everywhere as flake8 decides to pay attention to slightly different things, and we lose a lot of time collectively chasing down those syntax glitches everywhere. Which is why we haven't really bumped those since 2016... But proper support of f-strings might be a good reason to update? Thoughts? [1] https://review.opendev.org/#/c/695457/1 [2] https://zuul.opendev.org/t/openstack/build/bd97e5397b184176aac94251cc0a9220 -- Thierry Carrez (ttx)

10 21

[nova] Why don't we have a SNAPSHOTTING status?
by Matt Riedemann 22 Nov '19

22 Nov '19

This came up in IRC today [1]. There are two issues: 1. As far as I can see, when creating a snapshot of a volume-backed server the API does not change the instance task_state so technically the user could perform some other action on the server while we're creating the volume snapshot(s), like try to resize it. My guess is that would fail since we likely can't be doing things like creating and deleting volume attachments on a volume that is actively being snapshot. 2. For image-backed servers, we set the progress the task_states through a few states [2] but those aren't conveyed in the overall server "status" field in the API [3]. Does anyone have any historical context on either of those issues? Issue #1 reminds me that we don't have a task_state transition while confirming a resized server either [4] but I guess that's less of an issue because there are not a lot of things you can do to a server in VERIFY_RESIZE status (delete and revert the resize I think), but that does mean I could fire off separate confirmResize and then revertResize actions on the same server at the same time and the confirm will probably fail down in compute rather than in the API with a 409 error. [1] http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.… [2] https://github.com/openstack/nova/blob/991d675675c1c6bb87a2b9d19327e2b4473f… [3] https://github.com/openstack/nova/blob/991d675675c1c6bb87a2b9d19327e2b4473f… [4] https://github.com/openstack/nova/blob/991d675675c1c6bb87a2b9d19327e2b4473f… -- Thanks, Matt

1 0

[sig] Forming a Large scale SIG
by Thierry Carrez 22 Nov '19

22 Nov '19

Hi everyone, In Shanghai we held a forum session to gauge interest in a new SIG to specifically address cluster scaling issues. In the past we had several groups ("Large deployments", "Performance", LCOO...) but those efforts were arguably a bit too wide and those groups are now abandoned. My main goal here is to get large users directly involved in a domain where their expertise can best translate into improvements in the software. It's easy for such a group to go nowhere while trying to boil the ocean. To maximize its chances of success and make it sustainable, the group should have a narrow focus, and reasonable objectives. My personal idea for the group focus was to specifically address scaling issues within a single cluster: basically identify and address issues that prevent scaling a single cluster (or cell) past a number of nodes. By sharing analysis and experience, the group could identify common pain points that, once solved, would help raising that number. There was a lot of interest in that session[1], and it predictably exploded in lots of different directions, including some that are definitely past a single cluster (like making Neutron better support cells). I think it's fine: my initial proposal was more of a strawman. Active members of the group should really define what they collectively want to work on. And the SIG name should be picked to match that. I'd like to help getting that group off the ground and to a place where it can fly by itself, without needing external coordination. The first step would be to identify interested members and discuss group scope and objectives. Given the nature of the group (with interested members in Japan, Europe, Australia and the US) it will be hard to come up with a synchronous meeting time that will work for everyone, so let's try to hold that discussion over email. So to kick this off: if you are interested in that group, please reply to this email, introduce yourself and tell us what you would like the group scope and objectives to be, and what you can contribute to the group. Thanks! [1] https://etherpad.openstack.org/p/PVG-large-scale-SIG -- Thierry Carrez (ttx)

9 9

[tripleo] TripleO CI Summary: Sprint 39
by Rafael Folco 22 Nov '19

22 Nov '19

Greetings, The TripleO CI team has just completed Sprint 39 / Unified Sprint 18 (Oct 31 thru Nov 20). The following is a summary of completed work during this sprint cycle: - Evaluated and implemented CI jobs in Zuul that deal with RPM build artifacts for ceph-ansible and podman 3rd party testing. - Explored design options and created a PoC for individual component testing in the promotion pipeline. This effort will add an additional verification layer to check OpenStack components (compute, networking, storage, etc) with stable builds, and ease root cause determination when it breaks the code. - Fixed the docker login module in the new promotion code. - Improve tests for verifying a full promotion workflow running on the staging environment. The planned work for the next sprint [1] are: - Close-out work on 3rd-party testing jobs for podman and ceph-ansible. - Close-out work on the design enhancements to the promotion server. - Implement component pipeline running daily and promoting separated from the integration jobs. - Address issues and technical debt tasks in TripleO CI realm. The Ruck and Rover for this sprint are Marios Andreou (marios) and Chandan Kumar (chkumar). Please direct questions or queries to them regarding CI status or issues in #tripleo, ideally to whomever has the ‘|ruck’ suffix on their nick. Ruck/rover notes are being tracked in etherpad [2]. Thanks, rfolco [1] https://tree.taiga.io/project/tripleo-ci-board/taskboard/unified-sprint-19 [2] https://etherpad.openstack.org/p/ruckroversprint19

1 0