August 2021 - openstack-discuss - lists.openstack.org

[all][tc] What's happening in Technical Committee: summary 6th Aug, 21: Reading: 5 min
by Ghanshyam Mann 06 Aug '21

06 Aug '21

Hello Everyone, Here is last week's summary of the Technical Committee activities. 1. What we completed this week: ========================= * Added tap-as-a-service back to neutron stadium[1] * Defined Yoga release testing runtime[2] * Migration from Freenode to OFTC[3]. Only a few repo patches are pending to merge and can be merged in their own time[4]. 2. TC Meetings: ============ * TC held this week meeting on Thursday; you can find the full meeting logs in the below link: - https://meetings.opendev.org/meetings/tc/2021/tc.2021-08-05-15.00.log.html * We will have next week's meeting on Aug 12th, Thursday 15:00 UTC[5]. 3. Activities In progress: ================== TC Tracker for Xena cycle ------------------------------ TC is using the etherpad[6] for Xena cycle working item. We will be checking and updating the status biweekly in the same etherpad. Open Reviews ----------------- * Two open reviews for ongoing activities[7]. Wallaby testing runtime for centos8 vs centos8-stream ----------------------------------------------------------------- * Centos8 which is testing runtime for stable/wallaby cannot run on stable/wallaby testing anymore[8]. * We are discussing in TC whether to update the wallaby testing runtime from centos8->centos8-stream or stop testing centos8 itself. We will continue discussing it in the next TC meeting, feel free to join us if you have any input on this topic. Project Skyline(dashboard) proposal ------------------------------------------ * This is a new proposal about a new GUI for OpenStack name 'Skyline'. Discussion is in the initial stage and TC is not so clear if it is a proposal for a new separate project application or new repo/deliverables within the Horizon project. As the next step, TC asked to start the ML thread with all the details and then we can continue the discussion. * Meanwhile we will continue the discussion of 'required things (zuul, devstack integration etc ) we need to know about this proposal' in the next TC meeting Project updates ------------------- * Retiring js-openstack-lib [9] Yoga release community-wide goal ----------------------------------------- * Please add the possible candidates in this etherpad [10]. * Current status: Proposed "Secure RBAC" to select for Yoga cycle[11]. PTG planning ---------------- * TC booked three slots: ** 1. Monday 15-17 UTC for TC + Community Leaders ** 2. Thursday 13-17 UTC ** 3. Friday 13-17UTC * We are collecting the PTG topics in etherpad[12], feel free to add any topic you want to discuss. Test support for TLS default: ---------------------------------- * Rico has started a separate email thread over testing with tls-proxy enabled[13], we encourage projects to participate in that testing and help to enable the tls-proxy in gate testing. 4. How to contact the TC: ==================== If you would like to discuss or give feedback to TC, you can reach out to us in multiple ways: 1. Email: you can send the email with tag [tc] on openstack-discuss ML[14]. 2. Weekly meeting: The Technical Committee conduct a weekly meeting every Thursday 15 UTC [15] 3. Office hours: The Technical Committee offers a weekly office hour every Tuesday at 0100 UTC [16] 4. Ping us using 'tc-members' nickname on #openstack-tc IRC channel. [1] https://review.opendev.org/c/openstack/governance/+/802833 [2] https://review.opendev.org/c/openstack/governance/+/799927 [3] https://etherpad.opendev.org/p/openstack-irc-migration-to-oftc [4] https://review.opendev.org/q/topic:%2522oftc%2522+status:open [5] https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting [6] https://etherpad.opendev.org/p/tc-xena-tracker [7] https://review.opendev.org/q/project:openstack/governance+status:open [8] https://review.opendev.org/c/openstack/devstack/+/803071 [9] https://review.opendev.org/c/openstack/governance/+/798540 [10] https://etherpad.opendev.org/p/y-series-goals [11] https://review.opendev.org/c/openstack/governance/+/803783 [12] https://etherpad.opendev.org/p/tc-yoga-ptg [13] http://lists.openstack.org/pipermail/openstack-discuss/2021-June/023000.html [14] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss [15] http://eavesdrop.openstack.org/#Technical_Committee_Meeting [16] http://eavesdrop.openstack.org/#Technical_Committee_Office_hours -gmann

2 4

[neutron][ops] VLAN Provider Network
by DHilsbos＠performair.com 06 Aug '21

06 Aug '21

All; We're using OpenStack Victoria, with OVS networking (no OVN). I'm having some problems with a VLAN provider network that I added to my OpenStack. I have other VLAN provider networks, that I based this one off of. Instances are getting the tap interfaces, but these interfaces are not getting IP addresses, and when a static IP is given, they can't ping either the DHCP address or the gateway address. I have checked the switch configuration, to verify the VLAN is bound appropriately. I rebooted the whole cluster to ensure the configuration change applied. Here's a snip our plugin.ini [ml2_type_vlan] network_vlan_ranges = provider_core:55:55,provider_core:64:64,... 64 works, 55, doesn't. I'm not seeing anything concerning in /var/log/openvswitch/ovs-vswitchd.log I'm sure it's something dumb, but I'm not seeing it. Any suggestions for what else to look at? Thank you, Dominic L. Hilsbos, MBA Vice President - Information Technology Perform Air International Inc. DHilsbos(a)PerformAir.com www.PerformAir.com

2 2

[release] Release countdown for week R-8, Aug 09 - Aug 13
by Herve Beraud 06 Aug '21

06 Aug '21

General Information ------------------- The following cycle-with-intermediary deliverables have not done any intermediary release yet during this cycle. The cycle-with-rc release model is more suited for deliverables that plan to be released only once per cycle. As a result, we have proposed[1] to change the release model for the following deliverables: - python-openstackclient ( https://review.opendev.org/c/openstack/releases/+/803221) - ansible-role-atos-hsm ( https://review.opendev.org/c/openstack/releases/+/803216) - ansible-role-atos-lunasa ( https://review.opendev.org/c/openstack/releases/+/803216) - ansible-role-thales-hsm ( https://review.opendev.org/c/openstack/releases/+/803216) [1] https://review.opendev.org/q/topic:%22not-yet-released%22 PTLs and release liaisons for each of those deliverables can either +1 the release model change, or propose an intermediary release for that deliverable. In absence of answer by the end of R-10 week we'll consider that the switch to cycle-with-rc is preferable. We also published a proposed release schedule for the upcoming yoga cycle. Please check out the separate thread: https://review.opendev.org/c/openstack/releases/+/800663 Upcoming Deadlines & Dates -------------------------- Non-client library freeze: August 26 (R-6 week) Client library freeze: September 2 (R-5 week) Xena-3 milestone: September 2 (R-5 week) -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud

1 0

[wallaby][neutron][ovn] Multiple VLAN ranges and FLAT network
by Malik Obaid 06 Aug '21

06 Aug '21

Hi, I am using Openstack Wallaby release on Ubuntu 20.04. I am configuring openstack neutron for production and just want to know is there a way to specify different vlan ranges with multiple physical networks. I would really appreciate any input in this regard. Thank you. Regards, Malik Obaid

2 1

[nova][placement] PTL is on vacation next week
by Balazs Gibizer 06 Aug '21

06 Aug '21

Hi, I will be on PTO next week. The weekly nova meeting has been canceled. cheers, gibi

1 0

[i18n] Zanata
by Jimmy McArthur 05 Aug '21

05 Aug '21

Has Zanata been deprecated? I have a person trying to log in and they're unable. It's a new user that would like to contribute to the translation team. What's the best path to send them? Thank you, Jimmy

4 5

gerrit no longer lading from iphone
by Sorin Sbarnea 05 Aug '21

05 Aug '21

I am wondering if something was changed on our gerrit deployment recently as it seems to no longer load on ios safari. I know,… not the most common use case… still adding the usual recheck can prove useful after checking zuul comment. /zbr -- -- /zbr

5 7

[neutron] Drivers team meeting - agenda for Friday 05.08.2021
by Slawek Kaplonski 05 Aug '21

05 Aug '21

Hi, Agenda for our tomorrow's drivers meeting is at [1]. We have one RFE to discuss: https://bugs.launchpad.net/neutron/+bug/1936408 - [RFE] Neutron quota change should check available existing resources [1] https://wiki.openstack.org/wiki/Meetings/NeutronDrivers -- Slawek Kaplonski Principal Software Engineer Red Hat

1 0

[all][tc] Technical Committee next weekly meeting on Aug 5th at 1500 UTC
by Ghanshyam Mann 04 Aug '21

04 Aug '21

Hello Everyone, Technical Committee's next weekly meeting is scheduled for Aug 5th at 1500 UTC. If you would like to add topics for discussion, please add them to the below wiki page by Wednesday, Aug 4th, at 2100 UTC. https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting -gmann

1 1

[Ironic] Users Needed! OpenInfra Live Opportunity
by Erin Disney 04 Aug '21

04 Aug '21

Hey All- We have an OpenInfra Live episode coming up next Thursday, August 12th at 14:00 UTC focused on Ironic. We will be kicking off the episode with a project overview and look at the development roadmap, then would like to feature companies running it in production to discuss their use cases, and wrap up the episode discussing FAQs regarding Ironic as a group. We currently have one user signed up but would love to add one or two more to the line up. If this is something you or your team would be interested in participating in, please reach out to me directly (erin(a)openstack.org) If the day/time doesn’t work but you are still willing to discuss your deployment, a pre-recorded video is an option as well. Thanks in advance! For more on OpenInfra Live, check out https://openinfra.dev/live. Erin Disney Event Marketing Open Infrastructure Foundation

1 0

[largescale-sig] Next meeting: Aug 4, 15utc
by Thierry Carrez 04 Aug '21

04 Aug '21

Hi everyone, Our next Large Scale SIG meeting will be this Wednesday in #openstack-operators on OFTC IRC, at 15UTC. You can doublecheck how that time translates locally at: https://www.timeanddate.com/worldclock/fixedtime.html?iso=20210804T15 A number of topics have already been added to the agenda, including discussing the contents of our next OpenInfra.Live show. Feel free to add other topics to our agenda at: https://etherpad.openstack.org/p/large-scale-sig-meeting Regards, -- Thierry Carrez

1 1

[cinder] Bug deputy report for week of 2021-04-08
by Sofia Enriquez 04 Aug '21

04 Aug '21

Hello, This is a bug report from 2021-28-07 to 2021-04-08. We won't have the Cinder Bug Meeting today but we're welcome to attend the cinder mid cycle. Agenda: https://etherpad.opendev.org/p/cinder-bug-squad-meeting ----------------------------------------------------------------------------------------- Low - https://bugs.launchpad.net/cinder/+bug/1938869 "max_over_subscription_ratio set auto mode rbd backend will have a negative value". Unassigned. Incomplete - https://bugs.launchpad.net/cinder/+bug/1938488 "When restarting the c-bak service snapshot status is always backing-up". Unassigned Cheers, Sofi -- L. Sofía Enriquez she/her Software Engineer Red Hat PnT <https://www.redhat.com> IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 0

Migrating VM's
by Derek O keeffe 04 Aug '21

04 Aug '21

Hi all, I have a question. If we have an old openstack deployment (Queens) that have VM's (with & without volumes attached) and we want to migrate those to a new deploy (Victoria) can this be achieved by the owners of the VM's through horizon in some way? We can take snapshots of the vm's that don't have volumes attached, copy them to the new controller and re instantiate them but we haven't found a solution for the volumes yet. This give us more work though as it all needs to be done over cli so the user is taken out of the loop and it falls on us 3 admins to move over 300 VM's. Any advice or info would be greatly appreciated. Thanks in advance. Derek

3 7

[nova][neutron][deployment-projects] Re secbug #1734320
by Radosław Piliszek 03 Aug '21

03 Aug '21

Hello Folks! I am writing this because a recent patch proposed to DevStack [1] mentioned "when using ml2/ovs vif isolation should always be used to prevent cross tenant traffic during a live migration" which is related to secbug #1734320 "Eavesdropping private traffic" [2]. However, I've found that none of the publicly-available deployment projects seem to be using ``isolate_vif``. [3] [4] Should this be corrected? PS: I used the deployment-projects tag as a collective tag to avoid mentioning all the projects (as it is too long to write :-) ). I hope that relevant people see this if need be or someone passes the information to them. For now, I am curious whether this should actually be enforced by default with ML2/OVS. [1] https://review.opendev.org/c/openstack/devstack/+/796826 [2] https://bugs.launchpad.net/neutron/+bug/1734320 [3] https://codesearch.opendev.org/?q=%5Cbisolate_vif%5Cb&i=nope&files=&exclude… [4] https://github.com/search?p=1&q=isolate_vif&type=Code -yoctozepto

3 2

Re: [centos][kolla-ansible][wallaby][vpnaas] I need Help
by Björn Puttmann 03 Aug '21

03 Aug '21

Hi! We had similar problems while testing VPNaaS with Victoria in HA mode. The main takeaway was the following: - the path to the pidfile of the libreswan process in the driver is wrong. This causes the process.active check to always fail. diff --git a/neutron_vpnaas/services/vpn/device_drivers/libreswan_ipsec.py b/neutron_vpnaas/services/vpn/device_drivers/libreswan_ipsec.py index 90731f7a4..48b178d05 100644 --- a/neutron_vpnaas/services/vpn/device_drivers/libreswan_ipsec.py +++ b/neutron_vpnaas/services/vpn/device_drivers/libreswan_ipsec.py @@ -30,6 +30,9 @@ class LibreSwanProcess(ipsec.OpenSwanProcess): self._rootwrap_cfg = self._get_rootwrap_config() super(LibreSwanProcess, self).__init__(conf, process_id, vpnservice, namespace) + self.pid_path = os.path.join( + self.config_dir, 'var', 'run', 'pluto', 'pluto') + self.pid_file = '%s.pid' % self.pid_path def _ipsec_execute(self, cmd, check_exit_code=True, extra_ok_codes=None): """Execute ipsec command on namespace. - when deploying VPNaaS in HA mode, takeover fails as the standby node is missing the necessary configuration files for Libreswan. diff --git a/neutron_vpnaas/services/vpn/device_drivers/ipsec.py b/neutron_vpnaas/services/vpn/device_drivers/ipsec.py index 424c0eea0..7b1c96b3d 100644 --- a/neutron_vpnaas/services/vpn/device_drivers/ipsec.py +++ b/neutron_vpnaas/services/vpn/device_drivers/ipsec.py @@ -1077,12 +1077,19 @@ class IPsecDriver(device_drivers.DeviceDriver, metaclass=abc.ABCMeta): def report_status(self, context): status_changed_vpn_services = [] for process_id, process in list(self.processes.items()): - # NOTE(mnaser): It's not necessary to check status for processes - # of a backup L3 agent router = self.routers.get(process_id) - if router and router.router['ha'] and router.ha_state == 'backup': - LOG.debug("%s router in backup state, skipping", process_id) - continue + if router and router.router['ha']: + # NOTE(mnaser): It's not necessary to check status for processes + # of a backup L3 agent + if router.ha_state == 'backup': + LOG.debug("%s router in backup state, skipping", process_id) + continue + # When a ha takeover took place, router will initially not be active. Enable it now. + if router.ha_state == 'primary' and not process.active: + LOG.debug("%s is primary but not active. Activating.", process_id) + process.ensure_configs() + process.enable() + continue if not self.should_be_reported(context, process): continue previous_status = self.get_process_status_cache(process) With these patches applied, VPNaaS worked as expected, at least in Victoria. It seems, as Bodo already pointed out, that CentOS 8 is coming with Libreswan 4.4 which apparently dropped the "—use-netkey" parameter and now expects "--use-xfrm" So you could try to patch the corresponding driver files: diff --git a/neutron_vpnaas/services/vpn/device_drivers/ipsec.py b/neutron_vpnaas/services/vpn/device_drivers/ipsec.py index 424c0eea0..051719d40 100644 --- a/neutron_vpnaas/services/vpn/device_drivers/ipsec.py +++ b/neutron_vpnaas/services/vpn/device_drivers/ipsec.py @@ -647,7 +647,7 @@ class OpenSwanProcess(BaseSwanProcess): 'pluto', '--ctlbase', self.pid_path, '--ipsecdir', self.etc_dir, - '--use-netkey', + '--use-xfrm', '--uniqueids', '--nat_traversal', '--secretsfile', self.secrets_file] diff --git a/neutron_vpnaas/services/vpn/device_drivers/libreswan_ipsec.py b/neutron_vpnaas/services/vpn/device_drivers/libreswan_ipsec.py index 90731f7a4..b68e05f81 100644 --- a/neutron_vpnaas/services/vpn/device_drivers/libreswan_ipsec.py +++ b/neutron_vpnaas/services/vpn/device_drivers/libreswan_ipsec.py @@ -106,7 +106,7 @@ class LibreSwanProcess(ipsec.OpenSwanProcess): def start_pluto(self): cmd = ['pluto', - '--use-netkey', + '--use-xfrm', '--uniqueids'] if self.conf.ipsec.enable_detailed_logging: You could apply the patches to the device driver files on a local machine and copy the patched files into the corresponding containers via: docker cp /path/to/patched/{{ driver_file }} neutron_l3_agent:/usr/lib/python3.6/site-packages/neutron_vpnaas/services/vpn/device_drivers/{{ driver_file }} After that, restart the container. At least, that worked for us. Some commands, that proved quite useful for debugging the VPN stuff (please ignore if you already know this, just found this info very useful myself ;-) Get host for VPN tunnel: export PROJECT_ID=THE_PROJECT_ID ROUTER_ID=$(openstack --os-cloud THE_PROJECT_NAME vpn service list --long -f json | jq -r ".[] | select(.Project == \"${PROJECT_ID}\").Router") echo ${ROUTER_ID} openstack --os-cloud THE_PROJECT_NAME port list --router ${ROUTER_ID} --device-owner network:ha_router_replicated_interface -c binding_host_id -f value | sort -u Enable detailed logging in /etc/kolla/neutron-l3-agent/l3_agent.ini (restart container after changing this): [ipsec] enable_detailed_logging = True Logfile can now be found in neutron_l3_agent under /var/lib/neutron/ipsec/ROUTER_ID/log/ Get ipsec status (execute in neutron_l3_agent container): export ROUTER_ID=THE_ROUTER_ID ip netns exec qrouter-${ROUTER_ID} neutron-vpn-netns-wrapper --mount_paths="/etc:/var/lib/neutron/ipsec/${ROUTER_ID}/etc,/var/run:/var/lib/neutron/ipsec/${ROUTER_ID}/var/run" --rootwrap_config=/etc/neutron/rootwrap.conf --cmd="ipsec,whack,--status" ipsec.conf for router can be found under /var/lib/neutron/ipsec/ROUTER_ID/etc/ipsec.conf If you want to change the configuration directly (e.g. for testing cipher), the connection configuration needs to be reloaded: export ROUTER_ID=THE_ROUTER_ID ip netns exec qrouter-${ROUTER_ID} neutron-vpn-netns-wrapper --mount_paths="/etc:/var/lib/neutron/ipsec/${ROUTER_ID}/etc,/var/run:/var/lib/neutron/ipsec/${ROUTER_ID}/var/run" --rootwrap_config=/etc/neutron/rootwrap.conf --cmd="ipsec,auto,--replace,CONNECTION_ID" CONNECTION_ID can be found in ipsec.conf, e.g. conn 2b965b6b-2e03-4a89-acff-74cb56335f22 To initiate the tunnel after configuration change: export ROUTER_ID=THE_ROUTER_ID ip netns exec qrouter-${ROUTER_ID} neutron-vpn-netns-wrapper --mount_paths="/etc:/var/lib/neutron/ipsec/${ROUTER_ID}/etc,/var/run:/var/lib/neutron/ipsec/${ROUTER_ID}/var/run" --rootwrap_config=/etc/neutron/rootwrap.conf --cmd="ipsec,whack,--initiate,--name,CONNECTION_ID" To reread secrets: export ROUTER_ID=THE_ROUTER_ID ip netns exec qrouter-${ROUTER_ID} neutron-vpn-netns-wrapper --mount_paths="/etc:/var/lib/neutron/ipsec/${ROUTER_ID}/etc,/var/run:/var/lib/neutron/ipsec/${ROUTER_ID}/var/run" --rootwrap_config=/etc/neutron/rootwrap.conf --cmd="ipsec,auto,--rereadsecrets" Hope this is of some use. All the best and take care, Björn Puttmann -- NEOZO Cloud GmbH

2 1

[nova][placement]Weekly meeting canceled on 10th of Aug
by Balazs Gibizer 03 Aug '21

03 Aug '21

Hi, As it seems many of us will be on PTO next week I cancel the weekly meeting. Cheers, gibi

1 0

stackalytics.com/api/1.0/activity
by Csatari, Gergely (Nokia - FI/Espoo) 03 Aug '21

03 Aug '21

Hi, We plan to use the API of stackalytics.com and we realized that the data it provides ([1<https://www.stackalytics.com/api/1.0/activity?page_size=100>]) is never more recent than March 2021. Does anyone know what is the reason for this? Thanks, Gergely 1: https://www.stackalytics.com/api/1.0/activity?page_size=100

5 5

Automagically find LP bugs that should be marked as "Fix Released"
by Cyril Roelandt 03 Aug '21

03 Aug '21

Hello, I recently realized that some of my Launchpad bugs had a status different from "Fix Released" even though a fix had been released, and still appeared in my query of "open bugs". I wrote a simple script[1] to find bugs in a similar state: $ cd ~/openstack/glance_store/ # We assume the repository is up-to-date # Let's find the glance_store bugs fixed between 2.4.0 and 2.5.0 $ fix-released.sh 2.4.0 2.5.0 glance-store https://bugs.launchpad.net/glance-store/+bug/1915163 (Opinion) https://bugs.launchpad.net/glance-store/+bug/1915602 (New) In both these bug reports the following comment can be found: "This issue was fixed in the openstack/glance_store 2.5.0 release.", but the bug statuses have not been changed to "Fix Released". Comments are welcome, and I hope this helps. Cyril [1] https://gist.github.com/CyrilRoelandteNovance/1af6a5a23112ce444bda0ca2010d3…

1 0

[nova][glance] nova-lvm job constantly fails on glance image upload
by Sylvain Bauza 03 Aug '21

03 Aug '21

Hi, I recently spotted a constant failure on the nova-lvm job [1] which seems to occur since July-19 at least, and which prevents us to merge all changes touching the nova-libvirt code section [2] Evidences show a glance image upload HTTP 502 failure when we want to snapshot [3] which corresponds to the Glance API getting issues with staged image data not found [4] Until we figure out how to solve this, recheck against nova changes touching nova/virt/libvirt/ module are useless. Any help would be appreciated on debugging the Glance failure, tho. Thanks, -Sylvain [1] https://zuul.openstack.org/builds?job_name=nova-lvm&branch=master [2] https://zuul.openstack.org/job/nova-lvm [3] https://zuul.openstack.org/build/a04b0d2e184a4d3e9c39bd6110b454ea/log/contr… [4] https://zuul.openstack.org/build/a04b0d2e184a4d3e9c39bd6110b454ea/log/contr…

2 3

[wallaby][neutron] OVN ovs-system: deferred action limit reached, drop recirc action
by Ammad Syed 03 Aug '21

03 Aug '21

Hi, I am using openstack with OVN 20.12 and OVS 2.15.0 on ubuntu 20.04. I am using geneve tenant network and vlan provider network. I am continuously getting below messages in my dmesg logs continuously on compute node 1 only the other two compute nodes have no such messages. [275612.826698] openvswitch: ovs-system: deferred action limit reached, drop recirc action [275683.750343] openvswitch: ovs-system: deferred action limit reached, drop recirc action [276102.200772] openvswitch: ovs-system: deferred action limit reached, drop recirc action [276161.575494] openvswitch: ovs-system: deferred action limit reached, drop recirc action [276210.262524] openvswitch: ovs-system: deferred action limit reached, drop recirc action I have tried by reinstalling (OS everything) compute node 1 but still having same errors. Need your advice. Ammad

1 0

[swift] EC data left in old location after rebalance
by Reid Guyett 02 Aug '21

02 Aug '21

Hello, We started using EC policies in a new cluster a few months ago and added more capacity. During the rebalance (started June 30), it seems that all the data was copied to the new locations but it didn't clean up the old locations. This was identified through our handoff monitoring. OS: Ubuntu 18.04 Swift: 2.17.1 Example: List of devices for partition ~$ swift-get-nodes /etc/swift/object-4.ring.gz -p 14242 ... removed ... Server:Port Device x.x.x.31:6031 d31 Server:Port Device x.x.x.66:6030 d30 Server:Port Device x.x.x.25:6029 d29 Server:Port Device x.x.x.33:6027 d27 Server:Port Device x.x.x.36:6020 d20 Server:Port Device x.x.x.29:6018 d18 Server:Port Device x.x.x.21:6033 d33 Server:Port Device x.x.x.27:6025 d25 Server:Port Device x.x.x.35:6022 d22 Server:Port Device x.x.x.39:6031 d31 Server:Port Device x.x.x.28:6032 d32 Server:Port Device x.x.x.23:6021 d21 Server:Port Device x.x.x.26:6022 d22 Server:Port Device x.x.x.34:6023 d23 Server:Port Device x.x.x.37:6019 d19 Server:Port Device x.x.x.30:6017 d17 Server:Port Device x.x.x.22:6027 d27 Server:Port Device x.x.x.24:6031 d31 Server:Port Device x.x.x.32:6032 d32 Partitions look to have the correct data on them: ~$ ssh root(a)x.x.x.31 "ls -lah ${DEVICE:-/srv/node*}/d31/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.66 "ls -lah ${DEVICE:-/srv/node*}/d30/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.25 "ls -lah ${DEVICE:-/srv/node*}/d29/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.33 "ls -lah ${DEVICE:-/srv/node*}/d27/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.36 "ls -lah ${DEVICE:-/srv/node*}/d20/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.29 "ls -lah ${DEVICE:-/srv/node*}/d18/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.21 "ls -lah ${DEVICE:-/srv/node*}/d33/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.27 "ls -lah ${DEVICE:-/srv/node*}/d25/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.35 "ls -lah ${DEVICE:-/srv/node*}/d22/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.39 "ls -lah ${DEVICE:-/srv/node*}/d31/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.28 "ls -lah ${DEVICE:-/srv/node*}/d32/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.23 "ls -lah ${DEVICE:-/srv/node*}/d21/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.26 "ls -lah ${DEVICE:-/srv/node*}/d22/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.34 "ls -lah ${DEVICE:-/srv/node*}/d23/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.37 "ls -lah ${DEVICE:-/srv/node*}/d19/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.30 "ls -lah ${DEVICE:-/srv/node*}/d17/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.22 "ls -lah ${DEVICE:-/srv/node*}/d27/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.24 "ls -lah ${DEVICE:-/srv/node*}/d31/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.32 "ls -lah ${DEVICE:-/srv/node*}/d32/objects-4/14242 | wc -l" 664 >From one of the nodes that does not belong to the list above. This partition should not exist on this node after the rebalance. x.x.x.20:~# ls /srv/node/d28/objects-4/14242 | wc -l 627 The reconstructor is throwing a lot of these unexpected response errors in the logs. Manually running it from the node that should not have the partition, I can reproduce the error. x.x.y.0/24 is the replication network. x.x.x.20:~# swift-object-reconstructor /etc/swift/object-server.conf -d d28 -p 14242 -o -v object-reconstructor: x.x.y.42:6200/d30/14242 Unexpected response: ":ERROR: 500 'ERROR: With :UPDATES: 36 failures to 0 successes'" There looked to have been some partition locations cleaned up around July 11. Our expectation is that the old partition locations should be cleaned gradually since June 30 but we're not seeing that. I Was hoping for some ideas on what may be the problem (if any) and how we can make sure that the old partitions are cleaned up. Thanks, Reid

2 4

[ironic] Xena Mid-cycle
by Iury Gregory 02 Aug '21

02 Aug '21

Hello Ironicers! In our upstream meeting we decided we should do a virtual mid-cycle! If you are interested please fill out the doodle[1] and add topics in the etherpad[2]. Keep in mind we have contributors in many time zones, if you think we should add more time windows, please let me know. Maybe we can have multiple sessions each day so we can accommodate APAC TZ. I will close the doodle before our upstream meeting next week. [1] https://doodle.com/poll/mhh959u4s4rturxi [2] https://etherpad.opendev.org/p/ironic-xena-midcycle -- *Att[]'sIury Gregory Melo Ferreira * *MSc in Computer Science at UFCG* *Part of the ironic-core and puppet-manager-core team in OpenStack* *Software Engineer at Red Hat Czech* *Social*: https://www.linkedin.com/in/iurygregory *E-mail: iurygregory(a)gmail.com <iurygregory(a)gmail.com>*

1 1

[nova][gate] nova-tox-functional-* jobs failing on master branch
by melanie witt 02 Aug '21

02 Aug '21

Hey all, The nova-tox-functional-* jobs on the openstack/nova master branch are currently failing 100% of the time: https://zuul.openstack.org/builds?job_name=nova-tox-functional-py38&branch=… This is due to another issue with time overrides (a lack thereof) in the test_archive_task_logs functional tests. The same bug has happened in the past and the fix is now known to have been unsuccessful: https://bugs.launchpad.net/nova/+bug/1934519 Another fix has been proposed here: https://review.opendev.org/c/openstack/nova/+/803106 Please hold your rechecks until the fix merges and apologies for the trouble. -melwitt

2 2

[neutron] Bug deputy report July 26 - Aug 2
by Jakub Libosvar 02 Aug '21

02 Aug '21

Hi, I was bug deputy for the last week, here is the report. There are couple of unassigned bugs, I would highlight the first on the list related to stable/ussuri functional tests failures that needs an assignee. Kuba Critical * [stable/ussuri] Functional jobs timeout, many tests with fixtures._fixtures.timeout.TimeoutException https://bugs.launchpad.net/neutron/+bug/1938262 Needs assignee * [fullstack] OOM mysql service https://bugs.launchpad.net/neutron/+bug/1938455 Patch up for review: https://review.opendev.org/c/openstack/neutron/+/802907 Assigned to Rodolfo High * L3 agent fails to process a DVR router external network change https://bugs.launchpad.net/neutron/+bug/1938191 Needs assignee * [ovn]Router scheduler failing for config "default_availability_zones" https://bugs.launchpad.net/neutron/+bug/1938261 Patch up for review: https://review.opendev.org/c/openstack/neutron/+/802665 * FT "test_restart_rpc_on_sighup_multiple_workers" failing recurrently https://bugs.launchpad.net/neutron/+bug/1938428 Patch up for review: https://review.opendev.org/c/openstack/neutron/+/802860 Assigned to Rodolfo * vpnaas problem:ipsec pluto not running centos 8 victoria wallaby https://bugs.launchpad.net/neutron/+bug/1938571 Needs assignee * ofctl timeouts lead to dvr-ha-multinode-full failures https://bugs.launchpad.net/neutron/+bug/1938685 Needs assignee Medium * Cannot delete vlan type network via network generic switch (ngs) https://bugs.launchpad.net/neutron/+bug/1938023 Needs assignee * Cannot adjust number of resources in one agent step during device handling https://bugs.launchpad.net/neutron/+bug/1938202 Assigned to Mitya Eremeev * [ovn]agents alive status error after restarting neutron server https://bugs.launchpad.net/neutron/+bug/1938478 Needs assignee * Typo in OVN SUPPORTED_DHCP_OPTS_MAPPING dictionary https://bugs.launchpad.net/neutron/+bug/1938569 Patch up for review: https://review.opendev.org/c/openstack/neutron/+/803037 Assigned to Eduardo Low * Misalignment with extra-dhcp-options between neutronclient & openstackclient https://bugs.launchpad.net/neutron/+bug/1938575 Needs assignee Wishlist * [networking-bgpvpn] Port events now use payload Patch up for review: https://review.opendev.org/c/openstack/networking-bgpvpn/+/802916 Assigned to Rodolfo * Missing Diffie-Hellman-Groups vpnaas: https://bugs.launchpad.net/neutron/+bug/1938284 Patch up for review: https://review.opendev.org/c/openstack/neutron-vpnaas/+/802714

1 0

[TripleO] centos7 gate blocker (stable/train) please hold rechecks
by Marios Andreou 02 Aug '21

02 Aug '21

Hello tripleo o/ new gate blocker for centos7 hitting stable/train tripleo repos https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-7-co… bug is at https://bugs.launchpad.net/tripleo/+bug/1938079 - hitting stable/train (we only run c7 jobs there). investigation/resolution ongoing please hold rechecks for centos7 jobs. thank you

2 3

Diversity and Inclusion Meeting Reminder - OFTC
by Amy Marrich 01 Aug '21

01 Aug '21

The Diversity & Inclusion WG invites members of all OIF projects to attend our next meeting Monday Auguat 2nd, at 17:00 UTC in the #openinfra-diversity channel on OFTC. The agenda can be found at https://etherpad.openstack.org/p/ diversity-wg-agenda. Please feel free to add any topics you wish to discuss at the meeting. Thanks, Amy (apotz)

1 0

VPNaaS for Openstack
by Vinh Nguyen Duc 01 Aug '21

01 Aug '21

1 0