August 2021 - openstack-discuss - lists.openstack.org

Migrating VM's
by Derek O keeffe 04 Aug '21

04 Aug '21

Hi all, I have a question. If we have an old openstack deployment (Queens) that have VM's (with & without volumes attached) and we want to migrate those to a new deploy (Victoria) can this be achieved by the owners of the VM's through horizon in some way? We can take snapshots of the vm's that don't have volumes attached, copy them to the new controller and re instantiate them but we haven't found a solution for the volumes yet. This give us more work though as it all needs to be done over cli so the user is taken out of the loop and it falls on us 3 admins to move over 300 VM's. Any advice or info would be greatly appreciated. Thanks in advance. Derek

3 7

[nova][neutron][deployment-projects] Re secbug #1734320
by Radosław Piliszek 04 Aug '21

04 Aug '21

Hello Folks! I am writing this because a recent patch proposed to DevStack [1] mentioned "when using ml2/ovs vif isolation should always be used to prevent cross tenant traffic during a live migration" which is related to secbug #1734320 "Eavesdropping private traffic" [2]. However, I've found that none of the publicly-available deployment projects seem to be using ``isolate_vif``. [3] [4] Should this be corrected? PS: I used the deployment-projects tag as a collective tag to avoid mentioning all the projects (as it is too long to write :-) ). I hope that relevant people see this if need be or someone passes the information to them. For now, I am curious whether this should actually be enforced by default with ML2/OVS. [1] https://review.opendev.org/c/openstack/devstack/+/796826 [2] https://bugs.launchpad.net/neutron/+bug/1734320 [3] https://codesearch.opendev.org/?q=%5Cbisolate_vif%5Cb&i=nope&files=&exclude… [4] https://github.com/search?p=1&q=isolate_vif&type=Code -yoctozepto

3 2

Re: [centos][kolla-ansible][wallaby][vpnaas] I need Help
by Björn Puttmann 03 Aug '21

03 Aug '21

Hi! We had similar problems while testing VPNaaS with Victoria in HA mode. The main takeaway was the following: - the path to the pidfile of the libreswan process in the driver is wrong. This causes the process.active check to always fail. diff --git a/neutron_vpnaas/services/vpn/device_drivers/libreswan_ipsec.py b/neutron_vpnaas/services/vpn/device_drivers/libreswan_ipsec.py index 90731f7a4..48b178d05 100644 --- a/neutron_vpnaas/services/vpn/device_drivers/libreswan_ipsec.py +++ b/neutron_vpnaas/services/vpn/device_drivers/libreswan_ipsec.py @@ -30,6 +30,9 @@ class LibreSwanProcess(ipsec.OpenSwanProcess): self._rootwrap_cfg = self._get_rootwrap_config() super(LibreSwanProcess, self).__init__(conf, process_id, vpnservice, namespace) + self.pid_path = os.path.join( + self.config_dir, 'var', 'run', 'pluto', 'pluto') + self.pid_file = '%s.pid' % self.pid_path def _ipsec_execute(self, cmd, check_exit_code=True, extra_ok_codes=None): """Execute ipsec command on namespace. - when deploying VPNaaS in HA mode, takeover fails as the standby node is missing the necessary configuration files for Libreswan. diff --git a/neutron_vpnaas/services/vpn/device_drivers/ipsec.py b/neutron_vpnaas/services/vpn/device_drivers/ipsec.py index 424c0eea0..7b1c96b3d 100644 --- a/neutron_vpnaas/services/vpn/device_drivers/ipsec.py +++ b/neutron_vpnaas/services/vpn/device_drivers/ipsec.py @@ -1077,12 +1077,19 @@ class IPsecDriver(device_drivers.DeviceDriver, metaclass=abc.ABCMeta): def report_status(self, context): status_changed_vpn_services = [] for process_id, process in list(self.processes.items()): - # NOTE(mnaser): It's not necessary to check status for processes - # of a backup L3 agent router = self.routers.get(process_id) - if router and router.router['ha'] and router.ha_state == 'backup': - LOG.debug("%s router in backup state, skipping", process_id) - continue + if router and router.router['ha']: + # NOTE(mnaser): It's not necessary to check status for processes + # of a backup L3 agent + if router.ha_state == 'backup': + LOG.debug("%s router in backup state, skipping", process_id) + continue + # When a ha takeover took place, router will initially not be active. Enable it now. + if router.ha_state == 'primary' and not process.active: + LOG.debug("%s is primary but not active. Activating.", process_id) + process.ensure_configs() + process.enable() + continue if not self.should_be_reported(context, process): continue previous_status = self.get_process_status_cache(process) With these patches applied, VPNaaS worked as expected, at least in Victoria. It seems, as Bodo already pointed out, that CentOS 8 is coming with Libreswan 4.4 which apparently dropped the "—use-netkey" parameter and now expects "--use-xfrm" So you could try to patch the corresponding driver files: diff --git a/neutron_vpnaas/services/vpn/device_drivers/ipsec.py b/neutron_vpnaas/services/vpn/device_drivers/ipsec.py index 424c0eea0..051719d40 100644 --- a/neutron_vpnaas/services/vpn/device_drivers/ipsec.py +++ b/neutron_vpnaas/services/vpn/device_drivers/ipsec.py @@ -647,7 +647,7 @@ class OpenSwanProcess(BaseSwanProcess): 'pluto', '--ctlbase', self.pid_path, '--ipsecdir', self.etc_dir, - '--use-netkey', + '--use-xfrm', '--uniqueids', '--nat_traversal', '--secretsfile', self.secrets_file] diff --git a/neutron_vpnaas/services/vpn/device_drivers/libreswan_ipsec.py b/neutron_vpnaas/services/vpn/device_drivers/libreswan_ipsec.py index 90731f7a4..b68e05f81 100644 --- a/neutron_vpnaas/services/vpn/device_drivers/libreswan_ipsec.py +++ b/neutron_vpnaas/services/vpn/device_drivers/libreswan_ipsec.py @@ -106,7 +106,7 @@ class LibreSwanProcess(ipsec.OpenSwanProcess): def start_pluto(self): cmd = ['pluto', - '--use-netkey', + '--use-xfrm', '--uniqueids'] if self.conf.ipsec.enable_detailed_logging: You could apply the patches to the device driver files on a local machine and copy the patched files into the corresponding containers via: docker cp /path/to/patched/{{ driver_file }} neutron_l3_agent:/usr/lib/python3.6/site-packages/neutron_vpnaas/services/vpn/device_drivers/{{ driver_file }} After that, restart the container. At least, that worked for us. Some commands, that proved quite useful for debugging the VPN stuff (please ignore if you already know this, just found this info very useful myself ;-) Get host for VPN tunnel: export PROJECT_ID=THE_PROJECT_ID ROUTER_ID=$(openstack --os-cloud THE_PROJECT_NAME vpn service list --long -f json | jq -r ".[] | select(.Project == \"${PROJECT_ID}\").Router") echo ${ROUTER_ID} openstack --os-cloud THE_PROJECT_NAME port list --router ${ROUTER_ID} --device-owner network:ha_router_replicated_interface -c binding_host_id -f value | sort -u Enable detailed logging in /etc/kolla/neutron-l3-agent/l3_agent.ini (restart container after changing this): [ipsec] enable_detailed_logging = True Logfile can now be found in neutron_l3_agent under /var/lib/neutron/ipsec/ROUTER_ID/log/ Get ipsec status (execute in neutron_l3_agent container): export ROUTER_ID=THE_ROUTER_ID ip netns exec qrouter-${ROUTER_ID} neutron-vpn-netns-wrapper --mount_paths="/etc:/var/lib/neutron/ipsec/${ROUTER_ID}/etc,/var/run:/var/lib/neutron/ipsec/${ROUTER_ID}/var/run" --rootwrap_config=/etc/neutron/rootwrap.conf --cmd="ipsec,whack,--status" ipsec.conf for router can be found under /var/lib/neutron/ipsec/ROUTER_ID/etc/ipsec.conf If you want to change the configuration directly (e.g. for testing cipher), the connection configuration needs to be reloaded: export ROUTER_ID=THE_ROUTER_ID ip netns exec qrouter-${ROUTER_ID} neutron-vpn-netns-wrapper --mount_paths="/etc:/var/lib/neutron/ipsec/${ROUTER_ID}/etc,/var/run:/var/lib/neutron/ipsec/${ROUTER_ID}/var/run" --rootwrap_config=/etc/neutron/rootwrap.conf --cmd="ipsec,auto,--replace,CONNECTION_ID" CONNECTION_ID can be found in ipsec.conf, e.g. conn 2b965b6b-2e03-4a89-acff-74cb56335f22 To initiate the tunnel after configuration change: export ROUTER_ID=THE_ROUTER_ID ip netns exec qrouter-${ROUTER_ID} neutron-vpn-netns-wrapper --mount_paths="/etc:/var/lib/neutron/ipsec/${ROUTER_ID}/etc,/var/run:/var/lib/neutron/ipsec/${ROUTER_ID}/var/run" --rootwrap_config=/etc/neutron/rootwrap.conf --cmd="ipsec,whack,--initiate,--name,CONNECTION_ID" To reread secrets: export ROUTER_ID=THE_ROUTER_ID ip netns exec qrouter-${ROUTER_ID} neutron-vpn-netns-wrapper --mount_paths="/etc:/var/lib/neutron/ipsec/${ROUTER_ID}/etc,/var/run:/var/lib/neutron/ipsec/${ROUTER_ID}/var/run" --rootwrap_config=/etc/neutron/rootwrap.conf --cmd="ipsec,auto,--rereadsecrets" Hope this is of some use. All the best and take care, Björn Puttmann -- NEOZO Cloud GmbH

2 1

[nova][placement]Weekly meeting canceled on 10th of Aug
by Balazs Gibizer 03 Aug '21

03 Aug '21

Hi, As it seems many of us will be on PTO next week I cancel the weekly meeting. Cheers, gibi

1 0

stackalytics.com/api/1.0/activity
by Csatari, Gergely (Nokia - FI/Espoo) 03 Aug '21

03 Aug '21

Hi, We plan to use the API of stackalytics.com and we realized that the data it provides ([1<https://www.stackalytics.com/api/1.0/activity?page_size=100>]) is never more recent than March 2021. Does anyone know what is the reason for this? Thanks, Gergely 1: https://www.stackalytics.com/api/1.0/activity?page_size=100

5 5

Automagically find LP bugs that should be marked as "Fix Released"
by Cyril Roelandt 03 Aug '21

03 Aug '21

Hello, I recently realized that some of my Launchpad bugs had a status different from "Fix Released" even though a fix had been released, and still appeared in my query of "open bugs". I wrote a simple script[1] to find bugs in a similar state: $ cd ~/openstack/glance_store/ # We assume the repository is up-to-date # Let's find the glance_store bugs fixed between 2.4.0 and 2.5.0 $ fix-released.sh 2.4.0 2.5.0 glance-store https://bugs.launchpad.net/glance-store/+bug/1915163 (Opinion) https://bugs.launchpad.net/glance-store/+bug/1915602 (New) In both these bug reports the following comment can be found: "This issue was fixed in the openstack/glance_store 2.5.0 release.", but the bug statuses have not been changed to "Fix Released". Comments are welcome, and I hope this helps. Cyril [1] https://gist.github.com/CyrilRoelandteNovance/1af6a5a23112ce444bda0ca2010d3…

1 0

[nova][glance] nova-lvm job constantly fails on glance image upload
by Sylvain Bauza 03 Aug '21

03 Aug '21

Hi, I recently spotted a constant failure on the nova-lvm job [1] which seems to occur since July-19 at least, and which prevents us to merge all changes touching the nova-libvirt code section [2] Evidences show a glance image upload HTTP 502 failure when we want to snapshot [3] which corresponds to the Glance API getting issues with staged image data not found [4] Until we figure out how to solve this, recheck against nova changes touching nova/virt/libvirt/ module are useless. Any help would be appreciated on debugging the Glance failure, tho. Thanks, -Sylvain [1] https://zuul.openstack.org/builds?job_name=nova-lvm&branch=master [2] https://zuul.openstack.org/job/nova-lvm [3] https://zuul.openstack.org/build/a04b0d2e184a4d3e9c39bd6110b454ea/log/contr… [4] https://zuul.openstack.org/build/a04b0d2e184a4d3e9c39bd6110b454ea/log/contr…

2 3

[wallaby][neutron] OVN ovs-system: deferred action limit reached, drop recirc action
by Ammad Syed 03 Aug '21

03 Aug '21

Hi, I am using openstack with OVN 20.12 and OVS 2.15.0 on ubuntu 20.04. I am using geneve tenant network and vlan provider network. I am continuously getting below messages in my dmesg logs continuously on compute node 1 only the other two compute nodes have no such messages. [275612.826698] openvswitch: ovs-system: deferred action limit reached, drop recirc action [275683.750343] openvswitch: ovs-system: deferred action limit reached, drop recirc action [276102.200772] openvswitch: ovs-system: deferred action limit reached, drop recirc action [276161.575494] openvswitch: ovs-system: deferred action limit reached, drop recirc action [276210.262524] openvswitch: ovs-system: deferred action limit reached, drop recirc action I have tried by reinstalling (OS everything) compute node 1 but still having same errors. Need your advice. Ammad

1 0

[swift] EC data left in old location after rebalance
by Reid Guyett 02 Aug '21

02 Aug '21

Hello, We started using EC policies in a new cluster a few months ago and added more capacity. During the rebalance (started June 30), it seems that all the data was copied to the new locations but it didn't clean up the old locations. This was identified through our handoff monitoring. OS: Ubuntu 18.04 Swift: 2.17.1 Example: List of devices for partition ~$ swift-get-nodes /etc/swift/object-4.ring.gz -p 14242 ... removed ... Server:Port Device x.x.x.31:6031 d31 Server:Port Device x.x.x.66:6030 d30 Server:Port Device x.x.x.25:6029 d29 Server:Port Device x.x.x.33:6027 d27 Server:Port Device x.x.x.36:6020 d20 Server:Port Device x.x.x.29:6018 d18 Server:Port Device x.x.x.21:6033 d33 Server:Port Device x.x.x.27:6025 d25 Server:Port Device x.x.x.35:6022 d22 Server:Port Device x.x.x.39:6031 d31 Server:Port Device x.x.x.28:6032 d32 Server:Port Device x.x.x.23:6021 d21 Server:Port Device x.x.x.26:6022 d22 Server:Port Device x.x.x.34:6023 d23 Server:Port Device x.x.x.37:6019 d19 Server:Port Device x.x.x.30:6017 d17 Server:Port Device x.x.x.22:6027 d27 Server:Port Device x.x.x.24:6031 d31 Server:Port Device x.x.x.32:6032 d32 Partitions look to have the correct data on them: ~$ ssh root(a)x.x.x.31 "ls -lah ${DEVICE:-/srv/node*}/d31/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.66 "ls -lah ${DEVICE:-/srv/node*}/d30/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.25 "ls -lah ${DEVICE:-/srv/node*}/d29/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.33 "ls -lah ${DEVICE:-/srv/node*}/d27/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.36 "ls -lah ${DEVICE:-/srv/node*}/d20/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.29 "ls -lah ${DEVICE:-/srv/node*}/d18/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.21 "ls -lah ${DEVICE:-/srv/node*}/d33/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.27 "ls -lah ${DEVICE:-/srv/node*}/d25/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.35 "ls -lah ${DEVICE:-/srv/node*}/d22/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.39 "ls -lah ${DEVICE:-/srv/node*}/d31/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.28 "ls -lah ${DEVICE:-/srv/node*}/d32/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.23 "ls -lah ${DEVICE:-/srv/node*}/d21/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.26 "ls -lah ${DEVICE:-/srv/node*}/d22/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.34 "ls -lah ${DEVICE:-/srv/node*}/d23/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.37 "ls -lah ${DEVICE:-/srv/node*}/d19/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.30 "ls -lah ${DEVICE:-/srv/node*}/d17/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.22 "ls -lah ${DEVICE:-/srv/node*}/d27/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.24 "ls -lah ${DEVICE:-/srv/node*}/d31/objects-4/14242 | wc -l" 664 ~$ ssh root(a)x.x.x.32 "ls -lah ${DEVICE:-/srv/node*}/d32/objects-4/14242 | wc -l" 664 >From one of the nodes that does not belong to the list above. This partition should not exist on this node after the rebalance. x.x.x.20:~# ls /srv/node/d28/objects-4/14242 | wc -l 627 The reconstructor is throwing a lot of these unexpected response errors in the logs. Manually running it from the node that should not have the partition, I can reproduce the error. x.x.y.0/24 is the replication network. x.x.x.20:~# swift-object-reconstructor /etc/swift/object-server.conf -d d28 -p 14242 -o -v object-reconstructor: x.x.y.42:6200/d30/14242 Unexpected response: ":ERROR: 500 'ERROR: With :UPDATES: 36 failures to 0 successes'" There looked to have been some partition locations cleaned up around July 11. Our expectation is that the old partition locations should be cleaned gradually since June 30 but we're not seeing that. I Was hoping for some ideas on what may be the problem (if any) and how we can make sure that the old partitions are cleaned up. Thanks, Reid

2 4

[ironic] Xena Mid-cycle
by Iury Gregory 02 Aug '21

02 Aug '21

Hello Ironicers! In our upstream meeting we decided we should do a virtual mid-cycle! If you are interested please fill out the doodle[1] and add topics in the etherpad[2]. Keep in mind we have contributors in many time zones, if you think we should add more time windows, please let me know. Maybe we can have multiple sessions each day so we can accommodate APAC TZ. I will close the doodle before our upstream meeting next week. [1] https://doodle.com/poll/mhh959u4s4rturxi [2] https://etherpad.opendev.org/p/ironic-xena-midcycle -- *Att[]'sIury Gregory Melo Ferreira * *MSc in Computer Science at UFCG* *Part of the ironic-core and puppet-manager-core team in OpenStack* *Software Engineer at Red Hat Czech* *Social*: https://www.linkedin.com/in/iurygregory *E-mail: iurygregory(a)gmail.com <iurygregory(a)gmail.com>*

1 1