August 2021 - openstack-discuss - lists.openstack.org

[largescale-sig] Next meeting: Aug 4, 15utc
by Thierry Carrez 04 Aug '21

04 Aug '21

Hi everyone, Our next Large Scale SIG meeting will be this Wednesday in #openstack-operators on OFTC IRC, at 15UTC. You can doublecheck how that time translates locally at: https://www.timeanddate.com/worldclock/fixedtime.html?iso=20210804T15 A number of topics have already been added to the agenda, including discussing the contents of our next OpenInfra.Live show. Feel free to add other topics to our agenda at: https://etherpad.openstack.org/p/large-scale-sig-meeting Regards, -- Thierry Carrez

1 1

[cinder] Bug deputy report for week of 2021-04-08
by Sofia Enriquez 04 Aug '21

04 Aug '21

Hello, This is a bug report from 2021-28-07 to 2021-04-08. We won't have the Cinder Bug Meeting today but we're welcome to attend the cinder mid cycle. Agenda: https://etherpad.opendev.org/p/cinder-bug-squad-meeting ----------------------------------------------------------------------------------------- Low - https://bugs.launchpad.net/cinder/+bug/1938869 "max_over_subscription_ratio set auto mode rbd backend will have a negative value". Unassigned. Incomplete - https://bugs.launchpad.net/cinder/+bug/1938488 "When restarting the c-bak service snapshot status is always backing-up". Unassigned Cheers, Sofi -- L. Sofía Enriquez she/her Software Engineer Red Hat PnT <https://www.redhat.com> IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 0

Migrating VM's
by Derek O keeffe 04 Aug '21

04 Aug '21

Hi all, I have a question. If we have an old openstack deployment (Queens) that have VM's (with & without volumes attached) and we want to migrate those to a new deploy (Victoria) can this be achieved by the owners of the VM's through horizon in some way? We can take snapshots of the vm's that don't have volumes attached, copy them to the new controller and re instantiate them but we haven't found a solution for the volumes yet. This give us more work though as it all needs to be done over cli so the user is taken out of the loop and it falls on us 3 admins to move over 300 VM's. Any advice or info would be greatly appreciated. Thanks in advance. Derek

3 7

[nova][neutron][deployment-projects] Re secbug #1734320
by Radosław Piliszek 03 Aug '21

03 Aug '21

Hello Folks! I am writing this because a recent patch proposed to DevStack [1] mentioned "when using ml2/ovs vif isolation should always be used to prevent cross tenant traffic during a live migration" which is related to secbug #1734320 "Eavesdropping private traffic" [2]. However, I've found that none of the publicly-available deployment projects seem to be using ``isolate_vif``. [3] [4] Should this be corrected? PS: I used the deployment-projects tag as a collective tag to avoid mentioning all the projects (as it is too long to write :-) ). I hope that relevant people see this if need be or someone passes the information to them. For now, I am curious whether this should actually be enforced by default with ML2/OVS. [1] https://review.opendev.org/c/openstack/devstack/+/796826 [2] https://bugs.launchpad.net/neutron/+bug/1734320 [3] https://codesearch.opendev.org/?q=%5Cbisolate_vif%5Cb&i=nope&files=&exclude… [4] https://github.com/search?p=1&q=isolate_vif&type=Code -yoctozepto

3 2

Re: [centos][kolla-ansible][wallaby][vpnaas] I need Help
by Björn Puttmann 03 Aug '21

03 Aug '21

Hi! We had similar problems while testing VPNaaS with Victoria in HA mode. The main takeaway was the following: - the path to the pidfile of the libreswan process in the driver is wrong. This causes the process.active check to always fail. diff --git a/neutron_vpnaas/services/vpn/device_drivers/libreswan_ipsec.py b/neutron_vpnaas/services/vpn/device_drivers/libreswan_ipsec.py index 90731f7a4..48b178d05 100644 --- a/neutron_vpnaas/services/vpn/device_drivers/libreswan_ipsec.py +++ b/neutron_vpnaas/services/vpn/device_drivers/libreswan_ipsec.py @@ -30,6 +30,9 @@ class LibreSwanProcess(ipsec.OpenSwanProcess): self._rootwrap_cfg = self._get_rootwrap_config() super(LibreSwanProcess, self).__init__(conf, process_id, vpnservice, namespace) + self.pid_path = os.path.join( + self.config_dir, 'var', 'run', 'pluto', 'pluto') + self.pid_file = '%s.pid' % self.pid_path def _ipsec_execute(self, cmd, check_exit_code=True, extra_ok_codes=None): """Execute ipsec command on namespace. - when deploying VPNaaS in HA mode, takeover fails as the standby node is missing the necessary configuration files for Libreswan. diff --git a/neutron_vpnaas/services/vpn/device_drivers/ipsec.py b/neutron_vpnaas/services/vpn/device_drivers/ipsec.py index 424c0eea0..7b1c96b3d 100644 --- a/neutron_vpnaas/services/vpn/device_drivers/ipsec.py +++ b/neutron_vpnaas/services/vpn/device_drivers/ipsec.py @@ -1077,12 +1077,19 @@ class IPsecDriver(device_drivers.DeviceDriver, metaclass=abc.ABCMeta): def report_status(self, context): status_changed_vpn_services = [] for process_id, process in list(self.processes.items()): - # NOTE(mnaser): It's not necessary to check status for processes - # of a backup L3 agent router = self.routers.get(process_id) - if router and router.router['ha'] and router.ha_state == 'backup': - LOG.debug("%s router in backup state, skipping", process_id) - continue + if router and router.router['ha']: + # NOTE(mnaser): It's not necessary to check status for processes + # of a backup L3 agent + if router.ha_state == 'backup': + LOG.debug("%s router in backup state, skipping", process_id) + continue + # When a ha takeover took place, router will initially not be active. Enable it now. + if router.ha_state == 'primary' and not process.active: + LOG.debug("%s is primary but not active. Activating.", process_id) + process.ensure_configs() + process.enable() + continue if not self.should_be_reported(context, process): continue previous_status = self.get_process_status_cache(process) With these patches applied, VPNaaS worked as expected, at least in Victoria. It seems, as Bodo already pointed out, that CentOS 8 is coming with Libreswan 4.4 which apparently dropped the "—use-netkey" parameter and now expects "--use-xfrm" So you could try to patch the corresponding driver files: diff --git a/neutron_vpnaas/services/vpn/device_drivers/ipsec.py b/neutron_vpnaas/services/vpn/device_drivers/ipsec.py index 424c0eea0..051719d40 100644 --- a/neutron_vpnaas/services/vpn/device_drivers/ipsec.py +++ b/neutron_vpnaas/services/vpn/device_drivers/ipsec.py @@ -647,7 +647,7 @@ class OpenSwanProcess(BaseSwanProcess): 'pluto', '--ctlbase', self.pid_path, '--ipsecdir', self.etc_dir, - '--use-netkey', + '--use-xfrm', '--uniqueids', '--nat_traversal', '--secretsfile', self.secrets_file] diff --git a/neutron_vpnaas/services/vpn/device_drivers/libreswan_ipsec.py b/neutron_vpnaas/services/vpn/device_drivers/libreswan_ipsec.py index 90731f7a4..b68e05f81 100644 --- a/neutron_vpnaas/services/vpn/device_drivers/libreswan_ipsec.py +++ b/neutron_vpnaas/services/vpn/device_drivers/libreswan_ipsec.py @@ -106,7 +106,7 @@ class LibreSwanProcess(ipsec.OpenSwanProcess): def start_pluto(self): cmd = ['pluto', - '--use-netkey', + '--use-xfrm', '--uniqueids'] if self.conf.ipsec.enable_detailed_logging: You could apply the patches to the device driver files on a local machine and copy the patched files into the corresponding containers via: docker cp /path/to/patched/{{ driver_file }} neutron_l3_agent:/usr/lib/python3.6/site-packages/neutron_vpnaas/services/vpn/device_drivers/{{ driver_file }} After that, restart the container. At least, that worked for us. Some commands, that proved quite useful for debugging the VPN stuff (please ignore if you already know this, just found this info very useful myself ;-) Get host for VPN tunnel: export PROJECT_ID=THE_PROJECT_ID ROUTER_ID=$(openstack --os-cloud THE_PROJECT_NAME vpn service list --long -f json | jq -r ".[] | select(.Project == \"${PROJECT_ID}\").Router") echo ${ROUTER_ID} openstack --os-cloud THE_PROJECT_NAME port list --router ${ROUTER_ID} --device-owner network:ha_router_replicated_interface -c binding_host_id -f value | sort -u Enable detailed logging in /etc/kolla/neutron-l3-agent/l3_agent.ini (restart container after changing this): [ipsec] enable_detailed_logging = True Logfile can now be found in neutron_l3_agent under /var/lib/neutron/ipsec/ROUTER_ID/log/ Get ipsec status (execute in neutron_l3_agent container): export ROUTER_ID=THE_ROUTER_ID ip netns exec qrouter-${ROUTER_ID} neutron-vpn-netns-wrapper --mount_paths="/etc:/var/lib/neutron/ipsec/${ROUTER_ID}/etc,/var/run:/var/lib/neutron/ipsec/${ROUTER_ID}/var/run" --rootwrap_config=/etc/neutron/rootwrap.conf --cmd="ipsec,whack,--status" ipsec.conf for router can be found under /var/lib/neutron/ipsec/ROUTER_ID/etc/ipsec.conf If you want to change the configuration directly (e.g. for testing cipher), the connection configuration needs to be reloaded: export ROUTER_ID=THE_ROUTER_ID ip netns exec qrouter-${ROUTER_ID} neutron-vpn-netns-wrapper --mount_paths="/etc:/var/lib/neutron/ipsec/${ROUTER_ID}/etc,/var/run:/var/lib/neutron/ipsec/${ROUTER_ID}/var/run" --rootwrap_config=/etc/neutron/rootwrap.conf --cmd="ipsec,auto,--replace,CONNECTION_ID" CONNECTION_ID can be found in ipsec.conf, e.g. conn 2b965b6b-2e03-4a89-acff-74cb56335f22 To initiate the tunnel after configuration change: export ROUTER_ID=THE_ROUTER_ID ip netns exec qrouter-${ROUTER_ID} neutron-vpn-netns-wrapper --mount_paths="/etc:/var/lib/neutron/ipsec/${ROUTER_ID}/etc,/var/run:/var/lib/neutron/ipsec/${ROUTER_ID}/var/run" --rootwrap_config=/etc/neutron/rootwrap.conf --cmd="ipsec,whack,--initiate,--name,CONNECTION_ID" To reread secrets: export ROUTER_ID=THE_ROUTER_ID ip netns exec qrouter-${ROUTER_ID} neutron-vpn-netns-wrapper --mount_paths="/etc:/var/lib/neutron/ipsec/${ROUTER_ID}/etc,/var/run:/var/lib/neutron/ipsec/${ROUTER_ID}/var/run" --rootwrap_config=/etc/neutron/rootwrap.conf --cmd="ipsec,auto,--rereadsecrets" Hope this is of some use. All the best and take care, Björn Puttmann -- NEOZO Cloud GmbH

2 1

[nova][placement]Weekly meeting canceled on 10th of Aug
by Balazs Gibizer 03 Aug '21

03 Aug '21

Hi, As it seems many of us will be on PTO next week I cancel the weekly meeting. Cheers, gibi

1 0

stackalytics.com/api/1.0/activity
by Csatari, Gergely (Nokia - FI/Espoo) 03 Aug '21

03 Aug '21

Hi, We plan to use the API of stackalytics.com and we realized that the data it provides ([1<https://www.stackalytics.com/api/1.0/activity?page_size=100>]) is never more recent than March 2021. Does anyone know what is the reason for this? Thanks, Gergely 1: https://www.stackalytics.com/api/1.0/activity?page_size=100

5 5

Automagically find LP bugs that should be marked as "Fix Released"
by Cyril Roelandt 03 Aug '21

03 Aug '21

Hello, I recently realized that some of my Launchpad bugs had a status different from "Fix Released" even though a fix had been released, and still appeared in my query of "open bugs". I wrote a simple script[1] to find bugs in a similar state: $ cd ~/openstack/glance_store/ # We assume the repository is up-to-date # Let's find the glance_store bugs fixed between 2.4.0 and 2.5.0 $ fix-released.sh 2.4.0 2.5.0 glance-store https://bugs.launchpad.net/glance-store/+bug/1915163 (Opinion) https://bugs.launchpad.net/glance-store/+bug/1915602 (New) In both these bug reports the following comment can be found: "This issue was fixed in the openstack/glance_store 2.5.0 release.", but the bug statuses have not been changed to "Fix Released". Comments are welcome, and I hope this helps. Cyril [1] https://gist.github.com/CyrilRoelandteNovance/1af6a5a23112ce444bda0ca2010d3…

1 0

[nova][glance] nova-lvm job constantly fails on glance image upload
by Sylvain Bauza 03 Aug '21

03 Aug '21

Hi, I recently spotted a constant failure on the nova-lvm job [1] which seems to occur since July-19 at least, and which prevents us to merge all changes touching the nova-libvirt code section [2] Evidences show a glance image upload HTTP 502 failure when we want to snapshot [3] which corresponds to the Glance API getting issues with staged image data not found [4] Until we figure out how to solve this, recheck against nova changes touching nova/virt/libvirt/ module are useless. Any help would be appreciated on debugging the Glance failure, tho. Thanks, -Sylvain [1] https://zuul.openstack.org/builds?job_name=nova-lvm&branch=master [2] https://zuul.openstack.org/job/nova-lvm [3] https://zuul.openstack.org/build/a04b0d2e184a4d3e9c39bd6110b454ea/log/contr… [4] https://zuul.openstack.org/build/a04b0d2e184a4d3e9c39bd6110b454ea/log/contr…

2 3

[wallaby][neutron] OVN ovs-system: deferred action limit reached, drop recirc action
by Ammad Syed 03 Aug '21

03 Aug '21

Hi, I am using openstack with OVN 20.12 and OVS 2.15.0 on ubuntu 20.04. I am using geneve tenant network and vlan provider network. I am continuously getting below messages in my dmesg logs continuously on compute node 1 only the other two compute nodes have no such messages. [275612.826698] openvswitch: ovs-system: deferred action limit reached, drop recirc action [275683.750343] openvswitch: ovs-system: deferred action limit reached, drop recirc action [276102.200772] openvswitch: ovs-system: deferred action limit reached, drop recirc action [276161.575494] openvswitch: ovs-system: deferred action limit reached, drop recirc action [276210.262524] openvswitch: ovs-system: deferred action limit reached, drop recirc action I have tried by reinstalling (OS everything) compute node 1 but still having same errors. Need your advice. Ammad

1 0