August 2021 - openstack-discuss - lists.openstack.org

[swift] EC data left in old location after rebalance
by Reid Guyett 02 Aug '21

02 Aug '21

Hello, We started using EC policies in a new cluster a few months ago and added more capacity. During the rebalance (started June 30), it seems that all the data was copied to the new locations but it didn't clean up the old locations. This was identified through our handoff monitoring. OS: Ubuntu 18.04 Swift: 2.17.1 Example: List of devices for partition ~$ swift-get-nodes /etc/swift/object-4.ring.gz -p 14242 ... removed ... Server:Port Device x.x.x.31:6031 d31 Server:Port Device x.x.x.66:6030 d30 Server:Port Device x.x.x.25:6029 d29 Server:Port Device x.x.x.33:6027 d27 Server:Port Device x.x.x.36:6020 d20 Server:Port Device x.x.x.29:6018 d18 Server:Port Device x.x.x.21:6033 d33 Server:Port Device x.x.x.27:6025 d25 Server:Port Device x.x.x.35:6022 d22 Server:Port Device x.x.x.39:6031 d31 Server:Port Device x.x.x.28:6032 d32 Server:Port Device x.x.x.23:6021 d21 Server:Port Device x.x.x.26:6022 d22 Server:Port Device x.x.x.34:6023 d23 Server:Port Device x.x.x.37:6019 d19 Server:Port Device x.x.x.30:6017 d17 Server:Port Device x.x.x.22:6027 d27 Server:Port Device x.x.x.24:6031 d31 Server:Port Device x.x.x.32:6032 d32 Partitions look to have the correct data on them: ~$ ssh root(a)x.x.x.31 "ls -lah ${DEVICE:-/srv/node*}/d31/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.66 "ls -lah ${DEVICE:-/srv/node*}/d30/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.25 "ls -lah ${DEVICE:-/srv/node*}/d29/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.33 "ls -lah ${DEVICE:-/srv/node*}/d27/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.36 "ls -lah ${DEVICE:-/srv/node*}/d20/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.29 "ls -lah ${DEVICE:-/srv/node*}/d18/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.21 "ls -lah ${DEVICE:-/srv/node*}/d33/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.27 "ls -lah ${DEVICE:-/srv/node*}/d25/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.35 "ls -lah ${DEVICE:-/srv/node*}/d22/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.39 "ls -lah ${DEVICE:-/srv/node*}/d31/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.28 "ls -lah ${DEVICE:-/srv/node*}/d32/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.23 "ls -lah ${DEVICE:-/srv/node*}/d21/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.26 "ls -lah ${DEVICE:-/srv/node*}/d22/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.34 "ls -lah ${DEVICE:-/srv/node*}/d23/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.37 "ls -lah ${DEVICE:-/srv/node*}/d19/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.30 "ls -lah ${DEVICE:-/srv/node*}/d17/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.22 "ls -lah ${DEVICE:-/srv/node*}/d27/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.24 "ls -lah ${DEVICE:-/srv/node*}/d31/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.32 "ls -lah ${DEVICE:-/srv/node*}/d32/objects-4/14242 | wc -l" 664 >From one of the nodes that does not belong to the list above. This partition should not exist on this node after the rebalance. x.x.x.20:~# ls /srv/node/d28/objects-4/14242 | wc -l 627 The reconstructor is throwing a lot of these unexpected response errors in the logs. Manually running it from the node that should not have the partition, I can reproduce the error. x.x.y.0/24 is the replication network. x.x.x.20:~# swift-object-reconstructor /etc/swift/object-server.conf -d d28 -p 14242 -o -v object-reconstructor: x.x.y.42:6200/d30/14242 Unexpected response: ":ERROR: 500 'ERROR: With :UPDATES: 36 failures to 0 successes'" There looked to have been some partition locations cleaned up around July 11. Our expectation is that the old partition locations should be cleaned gradually since June 30 but we're not seeing that. I Was hoping for some ideas on what may be the problem (if any) and how we can make sure that the old partitions are cleaned up. Thanks, Reid

2 4

[ironic] Xena Mid-cycle
by Iury Gregory 02 Aug '21

02 Aug '21

Hello Ironicers! In our upstream meeting we decided we should do a virtual mid-cycle! If you are interested please fill out the doodle[1] and add topics in the etherpad[2]. Keep in mind we have contributors in many time zones, if you think we should add more time windows, please let me know. Maybe we can have multiple sessions each day so we can accommodate APAC TZ. I will close the doodle before our upstream meeting next week. [1] https://doodle.com/poll/mhh959u4s4rturxi [2] https://etherpad.opendev.org/p/ironic-xena-midcycle -- *Att[]'sIury Gregory Melo Ferreira * *MSc in Computer Science at UFCG* *Part of the ironic-core and puppet-manager-core team in OpenStack* *Software Engineer at Red Hat Czech* *Social*: https://www.linkedin.com/in/iurygregory *E-mail: iurygregory(a)gmail.com <iurygregory(a)gmail.com>*

1 1

[nova][gate] nova-tox-functional-* jobs failing on master branch
by melanie witt 02 Aug '21

02 Aug '21

Hey all, The nova-tox-functional-* jobs on the openstack/nova master branch are currently failing 100% of the time: https://zuul.openstack.org/builds?job_name=nova-tox-functional-py38&branch=… This is due to another issue with time overrides (a lack thereof) in the test_archive_task_logs functional tests. The same bug has happened in the past and the fix is now known to have been unsuccessful: https://bugs.launchpad.net/nova/+bug/1934519 Another fix has been proposed here: https://review.opendev.org/c/openstack/nova/+/803106 Please hold your rechecks until the fix merges and apologies for the trouble. -melwitt

2 2

[neutron] Bug deputy report July 26 - Aug 2
by Jakub Libosvar 02 Aug '21

02 Aug '21

Hi, I was bug deputy for the last week, here is the report. There are couple of unassigned bugs, I would highlight the first on the list related to stable/ussuri functional tests failures that needs an assignee. Kuba Critical * [stable/ussuri] Functional jobs timeout, many tests with fixtures._fixtures.timeout.TimeoutException https://bugs.launchpad.net/neutron/+bug/1938262 Needs assignee * [fullstack] OOM mysql service https://bugs.launchpad.net/neutron/+bug/1938455 Patch up for review: https://review.opendev.org/c/openstack/neutron/+/802907 Assigned to Rodolfo High * L3 agent fails to process a DVR router external network change https://bugs.launchpad.net/neutron/+bug/1938191 Needs assignee * [ovn]Router scheduler failing for config "default_availability_zones" https://bugs.launchpad.net/neutron/+bug/1938261 Patch up for review: https://review.opendev.org/c/openstack/neutron/+/802665 * FT "test_restart_rpc_on_sighup_multiple_workers" failing recurrently https://bugs.launchpad.net/neutron/+bug/1938428 Patch up for review: https://review.opendev.org/c/openstack/neutron/+/802860 Assigned to Rodolfo * vpnaas problem:ipsec pluto not running centos 8 victoria wallaby https://bugs.launchpad.net/neutron/+bug/1938571 Needs assignee * ofctl timeouts lead to dvr-ha-multinode-full failures https://bugs.launchpad.net/neutron/+bug/1938685 Needs assignee Medium * Cannot delete vlan type network via network generic switch (ngs) https://bugs.launchpad.net/neutron/+bug/1938023 Needs assignee * Cannot adjust number of resources in one agent step during device handling https://bugs.launchpad.net/neutron/+bug/1938202 Assigned to Mitya Eremeev * [ovn]agents alive status error after restarting neutron server https://bugs.launchpad.net/neutron/+bug/1938478 Needs assignee * Typo in OVN SUPPORTED_DHCP_OPTS_MAPPING dictionary https://bugs.launchpad.net/neutron/+bug/1938569 Patch up for review: https://review.opendev.org/c/openstack/neutron/+/803037 Assigned to Eduardo Low * Misalignment with extra-dhcp-options between neutronclient & openstackclient https://bugs.launchpad.net/neutron/+bug/1938575 Needs assignee Wishlist * [networking-bgpvpn] Port events now use payload Patch up for review: https://review.opendev.org/c/openstack/networking-bgpvpn/+/802916 Assigned to Rodolfo * Missing Diffie-Hellman-Groups vpnaas: https://bugs.launchpad.net/neutron/+bug/1938284 Patch up for review: https://review.opendev.org/c/openstack/neutron-vpnaas/+/802714

1 0

[TripleO] centos7 gate blocker (stable/train) please hold rechecks
by Marios Andreou 02 Aug '21

02 Aug '21

Hello tripleo o/ new gate blocker for centos7 hitting stable/train tripleo repos https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-7-co… bug is at https://bugs.launchpad.net/tripleo/+bug/1938079 - hitting stable/train (we only run c7 jobs there). investigation/resolution ongoing please hold rechecks for centos7 jobs. thank you

2 3

Diversity and Inclusion Meeting Reminder - OFTC
by Amy Marrich 01 Aug '21

01 Aug '21

The Diversity & Inclusion WG invites members of all OIF projects to attend our next meeting Monday Auguat 2nd, at 17:00 UTC in the #openinfra-diversity channel on OFTC. The agenda can be found at https://etherpad.openstack.org/p/ diversity-wg-agenda. Please feel free to add any topics you wish to discuss at the meeting. Thanks, Amy (apotz)

1 0

VPNaaS for Openstack
by Vinh Nguyen Duc 01 Aug '21

01 Aug '21

1 0