- openstack-discuss - lists.openstack.org

Propose Dale Smith as Magnum Caore
by Jake Yip 04 Apr '23

04 Apr '23

Hi all, I would like to propose Dale Smith as a Magnum Core Reviewer. Dale is a contributor to Magnum, and also takes care of Magnum at his place of employment - Catalyst Cloud NZ. As many Magnum contributors are ourselves operators, I think it is important that we have representation from different cloud providers to help guide and shape Magnum going forward. He will replace Feilong, who is the former PTL and used to work in Catalyst Cloud, in the Core Team. We thank Feilong for his contributions and welcome Dale. Regards, Jake -- Jake Yip Technical Lead, ARDC Nectar Research Cloud

1 0

[nova][heat] The next steps to "fix" libvirt problems in Ubuntu Jammy
by Takashi Kajinami 04 Apr '23

04 Apr '23

Hello, Since we migrated our jobs from Ubuntu Focal to Ubuntu Jammy, heat gate jobs have become very flaky. Further investigation revealed that the issue is related to something in libvirt from Ubuntu Jammy and that prevents detaching devices from instances[1]. The same problem appears in different jobs[2] and we workaround the problem by disabling some affected jobs. In heat we also disabled some flaky tests but because of this we no longer run basic scenario tests which deploys instance/volume/network in a single stack, which means we lost the quite basic test coverage. My question is, is there anyone in the Nova team working on "fixing" this problem ? We might be able to implement some workaround (like checking status of the instances before attempting to delete it) but this should be fixed in libvirt side IMO, as this looks like a "regression" in Ubuntu Jammy. Probably we should report a bug against the libvirt package in Ubuntu but I'd like to hear some thoughts from the nova team because they are more directly affected by this problem. I'm now trying to set up a centos stream 9 job in Heat repo to see whether this can be reproduced if we use centos stream 9. I've been running that specific scenario test in centos stream 9 jobs in puppet repos but I've never seen this issue, so I suspect the issue is really specific to libvirt in Jammy. [1] https://bugs.launchpad.net/nova/+bug/1998274 [2] https://bugs.launchpad.net/nova/+bug/1998148 Thank you, Takashi

3 5

[horizon]display quota volume type on horizon
by Nguyễn Hữu Khôi 04 Apr '23

04 Apr '23

Hello. I see that blueprint is very real case but it has not deployed yet. https://blueprints.launchpad.net/horizon/+spec/quotas-per-cinder-volume-type I am curious why it is abanddoned .

2 2

[designate][ptg] Designate Bobcat (2023.2) PTG summary
by Michael Johnson 03 Apr '23

03 Apr '23

We accomplished some big tasks in Antelope: Finishing the shared zones feature, adding support for SQL Alchemy 2.x, and hard zone deletes. For the Bobcat series, Designate had a pretty straightforward PTG session with good discussion. The focus areas for Bobcat are going to be on the batch serial number update patch[1] (deferred from Antelope) and improving the multiple pools feature[2][3]. Multiple DNS server pools have been a feature in Designate for a number of years now, but we need to improve the test coverage and documentation. Other minor work anticipated for Bobcat is to update the RBAC testing to account for the RBAC "Direction Change"[4] in support of enabling the new Keystone default roles as the default RBAC configuration. We also discussed the new specification that merged during Antelope[5] to add support for catalog zones. This specification is merged and ready for development, but it was not clear that there are developers planning to work on it during Bobcat. We hope this feature can at least get started during the Bobcat series. Thank you to everyone that participated! Michael [1] https://review.opendev.org/c/openstack/designate/+/871255 [2] https://review.opendev.org/c/openstack/designate/+/857978 [3] https://review.opendev.org/c/openstack/designate-tempest-plugin/+/871184 [4] https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rb… [5] https://specs.openstack.org/openstack/designate-specs/specs/antelope/catalo…

1 0

[ptls] PyPI maintainer cleanup - Action needed: Contact extra maintainers
by Jay Faulkner 03 Apr '23

03 Apr '23

Hi PTLs, The TC recently voted[1] to require humans be removed from PyPI access for OpenStack-managed projects. This helps ensure all releases are created via releases team tooling and makes it less likely for a user account compromise to impact OpenStack packages. Many projects have already updated https://etherpad.opendev.org/p/openstack-pypi-maintainers-cleanup#L33 with a list of packages that contain extra maintainers. We'd like to request that PTLs, or their designate, reach out to any extra maintainers listed for projects you are responsible for and request they remove their access in accordance with policy. An example email, and detailed steps to follow have been provided at https://etherpad.opendev.org/p/openstack-pypi-maintainers-cleanup-email-tem… . Thank you for your cooperation as we work to improve our security posture and harden against supply chain attacks. Thank you, Jay Faulkner TC Vice-Chair 1: https://opendev.org/openstack/governance/commit/979e339f899ef62d2a6871a99c9…

3 6

[barbican] database is growing and can not be purged
by Pavlo Shchelokovskyy 03 Apr '23

03 Apr '23

Hi all, we are observing the following behavior in Barbican: - OpenStack environment is using both encrypted Cinder volumes and encrypted local storage (lvm) for Nova instances - over the time, the secrets and orders tables are growing - many soft-deleted entries in secrets DB can not be purged by the db cleanup script As I understand what is happening - both Cinder and Nova create secrets in Barbican on behalf of the user when creating an encrypted volume or booting an instance with encrypted local storage. They both do it via castellan library, that under the hood creates orders in Barbican, waits for them to become active and returns to the caller only the ID of the generated secret. When time comes to delete the thing (volume or instance) Cinder/Nova again use castellan, but only delete the secret, not the order (they are not aware that there was any 'order' created anyway). As a result, the orders are left in place, and DB cleanup procedure does not delete soft-deleted secrets when there's an ACTIVE order referencing such secret. This is troublesomes on many levels - users who use Cinder or Nova may not even be aware that they are creating something in Barbican. Orders accumulating like that may eventually result in cryptic errors when e.g. when you run out of quota for orders. And what's more, default Barbican policies do allow 'normal' (creator) users to create an order, but not delete it (only project admin can do it), so even if the users are aware of Barbican involvement, they can not delete those orders manually anyway. Plus there's no good way in API to determine outright which orders are referencing deleted secrets. I see several ways of dealing with that and would like to ask for your opinion on what would be the best one: 1. Amend Barbican API to allow filtering orders by the secrets, when castellan deletes a secret - search for corresponding order and delete it as well, change default policy to actually allow order deletion by the same users who can create them. 2. Cascade-delete orders when deleting secrets - this is easy but probably violates that very policy that disallowed normal users to delete orders. 3. improve the database cleanup so it first marks any order that references a deleted secret also as deleted, so later when time comes both could be purged (or something like that). This also has a similar downside to the previous option by not being explicit enough. I've filed a bug for that https://storyboard.openstack.org/#!/story/2010625 and proposed a patch for option 2 (cascade delete), but would like to ask what would you see as the most appropriate way or may be there's something else that I've missed. Btw, the problem is probably even more pronounced with keypairs - when castellan is used to create those, under the hood both order and container are created besides the actual secrets, and again only the secret ids are returned to the caller. When time comes to delete things, the caller only knows about secret IDs, and can only delete them, leaving both container and order behind. Luckily, I did not find any place across OpenStack that actually creates keypairs using castellan... but the problem is definitely there. Best regards, -- Dr. Pavlo Shchelokovskyy Principal Software Engineer Mirantis Inc www.mirantis.com

2 2

[SWIFT] ssync receiver crash
by Reid Guyett 03 Apr '23

03 Apr '23

Hello, We recently upgraded our swift servers to 20.04 from 18.04. After the upgrades were completed we are seeing 1 server where the ssync receiver keeps crashing. The crash is preventing hand offs from being cleanup up in the cluster (50 servers). Looking for some advice to fix/workaround the issue. We have 20 clusters but are only seeing this issue in 1 cluster. It is strange to me that we only see this ssync receiver error in one server out of the ~1000 we are running. OS: Ubuntu 20.04.5 Swift version: 2.25.2 On the receiver side we are seeing the following trace: object-server: senderIp/d1/2166 EXCEPTION in ssync.Receiver: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/eventlet/wsgi.py", line 209, in _chunked_read self.chunk_length = int(rfile.readline().split(b";",1)[0], 16) ValueError: invalid literal for int() with base 16: b'' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/swift/obj/ssync_receiver.py", line 166, in __call__ for data in self.missing_check(): File "/usr/lib/python3/dist-packages/swift/obj/ssync_receiver.py", line 340, in missing_check line = self.fp.readline(self.app.network_chunk_size) File "/usr/lib/python3/dist-packages/eventlet/wsgi.py", line 226, in readline return self._chunked_read(self.rfile, size, True) File "/usr/lib/python3/dist-packages/eventlet/wsgi.py", line 211, in _chunked_read raise ChunkReadError(err) eventlet.wsgi.ChunkReadError: invalid literal for int() with base 16: b'' On the sender side we are seeing this message when I try to run the reconstructor manually with debug logging: root@use1-saas-p6-paco-9:~# swift-object-reconstructor object-server.conf -p 27999 -v -o object-reconstructor: Starting 271723 object-reconstructor: Spawned worker 271755 with {'override_partitions': [27999], 'override_devices': ['d10', 'd17', 'd19', 'd9', 'd29', 'd15', 'd2', 'd24', ' d32', 'd1', 'd8', 'd14', 'd11', 'd28', 'd27', 'd6', 'd7', 'd16', 'd13', 'd25', 'd26', 'd20', 'd18', 'd12', 'd4', 'd0', 'd22', 'd3', 'd5', 'd21', 'd33', 'd23', 'd30', 'd34', 'd31'], 'multiprocess_worker_index': 0} object-reconstructor: [worker 1/1 pid=271755] Running object reconstructor in script mode. object-reconstructor: [worker 1/1 pid=271755] Run listdir on /srv/node/d22/objects-4/27999 object-reconstructor: [worker 1/1 pid=271755] recieverIP:6200/d31/27999 10.0 seconds: connect receive object-reconstructor: [worker 1/1 pid=271755] 1/26820 (0.00%) partitions reconstructed in 10.14s (0.10/sec, 75h remaining) object-reconstructor: [worker 1/1 pid=271755] Object reconstruction complete (once). (0.17 minutes) object-reconstructor: Forked worker 271755 finished object-reconstructor: Worker 271755 exited object-reconstructor: Finished 271723 object-reconstructor: Exited 271723 Thanks! Reid Guyett Staff Storage Engineer Datto, A Kaseya Company www.datto.com<http://www.datto.com/> Important Notice: This email is intended to be received only by persons entitled to receive the confidential and legally privileged information it presumptively contains, and this notice constitutes identification as such. Any reading, disclosure, copying, distribution or use of this information by or to someone who is not the intended recipient, is prohibited. If you received this email in error, please notify us immediately at legal(a)kaseya.com, and then delete it. To opt-out of receiving emails Please click here<https://info.kaseya.com/email-subscription-center.html>. The term 'this e-mail' includes any and all attachments.

1 0

[nova][dev] image to local block device mapping not supported
by Nemanja Miletic 03 Apr '23

03 Apr '23

Hello, Nova does not support image to local block device mapping since issue in this bug is fixed: https://bugs.launchpad.net/nova/+bug/1377958 In particular I am interested in scenario best described with this command: openstack server create --flavor m1.small --network mgmt --image XXX \ --block-device uuid=YYY,source_type=image,\ destination_type=local,disk_bus=ide,device_type=cdrom,boot_index=1 testvm300 For above command nova gives error: Block Device Mapping is Invalid: Mapping image to local is not supported. Libvirt does not have issues with having two block devices which have file source. My question is what am I missing here? Why Nova does not support image to local block device mapping? Thank you for your help. Nemanja

2 2

[kolla] Image building question
by Satish Patel 03 Apr '23

03 Apr '23

Folks, I am playing with kolla image building to understand how it works. I am using the following command to build images and wanted to check with you folks if that is the correct way to do it. $ kolla-build -b ubuntu -t source keystone nova neutron glance Does the above command compile code from source or just download images from remote repositories and re-compile them? because in command output I've not noticed anything related to the compiling process going on. Here is the output of all images produced by kolla-build command. Do I need anything else or is this enough to deploy kolla? root@docker-reg:~# docker images REPOSITORY TAG IMAGE ID CREATED SIZE kolla/mariadb-server 15.1.0 2a497eee8269 26 minutes ago 595MB kolla/cron 15.1.0 342877f26a8a 30 minutes ago 250MB kolla/memcached 15.1.0 0d19a4902644 31 minutes ago 250MB kolla/mariadb-clustercheck 15.1.0 d84427d3c639 31 minutes ago 314MB kolla/mariadb-base 15.1.0 34447e3e59b6 31 minutes ago 314MB kolla/keepalived 15.1.0 82133b09fbf0 31 minutes ago 260MB kolla/prometheus-memcached-exporter 15.1.0 6c2d605f70ee 31 minutes ago 262MB <none> <none> e66b228c2a07 31 minutes ago 248MB kolla/rabbitmq 15.1.0 8de5c39379d3 32 minutes ago 309MB kolla/fluentd 15.1.0 adfd19027862 33 minutes ago 519MB kolla/haproxy-ssh 15.1.0 514357ac4d36 36 minutes ago 255MB kolla/haproxy 15.1.0 e5b9cfdf6dfc 37 minutes ago 257MB kolla/prometheus-haproxy-exporter 15.1.0 a679f65fd735 37 minutes ago 263MB kolla/prometheus-base 15.1.0 afeff3ed5dce 37 minutes ago 248MB kolla/glance-api 15.1.0 a2241f68f23a 38 minutes ago 1.04GB kolla/glance-base 15.1.0 7286772a03a4 About an hour ago 1.03GB kolla/neutron-infoblox-ipam-agent 15.1.0 f90ffc1a3326 About an hour ago 1.05GB kolla/neutron-server 15.1.0 69c844a2e3a9 About an hour ago 1.05GB kolla/neutron-l3-agent 15.1.0 4d87e6963c96 About an hour ago 1.05GB <none> <none> 486da9a6562e About an hour ago 1.05GB kolla/neutron-linuxbridge-agent 15.1.0 e5b3ca7e099c About an hour ago 1.04GB kolla/neutron-bgp-dragent 15.1.0 ac37377820c6 About an hour ago 1.04GB kolla/ironic-neutron-agent 15.1.0 90993adcd74b About an hour ago 1.04GB kolla/neutron-metadata-agent 15.1.0 8522f147f88d About an hour ago 1.04GB kolla/neutron-sriov-agent 15.1.0 8a92ce7d13c0 About an hour ago 1.04GB kolla/neutron-dhcp-agent 15.1.0 5c214b0171f5 About an hour ago 1.04GB kolla/neutron-metering-agent 15.1.0 7b3b91ecd77b About an hour ago 1.04GB kolla/neutron-openvswitch-agent 15.1.0 1f8807308814 About an hour ago 1.04GB kolla/neutron-base 15.1.0 f85b6a2e2725 About an hour ago 1.04GB kolla/nova-libvirt 15.1.0 0f3ecefe4752 About an hour ago 987MB kolla/nova-compute 15.1.0 241b7e7fafbe About an hour ago 1.47GB kolla/nova-spicehtml5proxy 15.1.0 b740820a7ad1 About an hour ago 1.15GB kolla/nova-novncproxy 15.1.0 1ba2f443d5c3 About an hour ago 1.22GB kolla/nova-compute-ironic 15.1.0 716612107532 About an hour ago 1.12GB kolla/nova-ssh 15.1.0 ae2397f4e1c1 About an hour ago 1.11GB kolla/nova-api 15.1.0 2aef02667ff8 About an hour ago 1.11GB kolla/nova-conductor 15.1.0 6f1da3400901 About an hour ago 1.11GB kolla/nova-scheduler 15.1.0 628326776b1d About an hour ago 1.11GB kolla/nova-serialproxy 15.1.0 28eb7a4a13f8 About an hour ago 1.11GB kolla/nova-base 15.1.0 e47420013283 About an hour ago 1.11GB kolla/keystone 15.1.0 e5530d829d5f 2 hours ago 947MB kolla/keystone-ssh 15.1.0 eaa7e3f3985a 2 hours ago 953MB kolla/keystone-fernet 15.1.0 8a4fa24853a8 2 hours ago 951MB kolla/keystone-base 15.1.0 b6f9562364a9 2 hours ago 945MB kolla/barbican-base 15.1.0 b2fdef1afb44 2 hours ago 915MB kolla/barbican-keystone-listener 15.1.0 58bd59de2c63 2 hours ago 915MB kolla/openstack-base 15.1.0 c805b4b3b1c1 2 hours ago 893MB kolla/base 15.1.0 f68e9ef3dd30 2 hours ago 248MB registry 2 8db46f9d7550 19 hours ago 24.2MB ubuntu 22.04 08d22c0ceb15 3 weeks ago 77.8MB

4 8

[neutron][ptg] Neutron PTG summary
by Rodolfo Alonso Hernandez 03 Apr '23

03 Apr '23

Hello Neutrinos: This is a quick summary of the PTG sessions last week. This is the etherpad used: https://etherpad.opendev.org/p/neutron-bobcat-ptg *Retrospective.* - Good: - Speed on RFE spec reviewal. - Continue the deprecation of non-supported code: Linux Bridge, neutronclient - Good CI tracking of external library dependencies: pyroute2, sqlalchemy - Good progress on TC goals, in particular sRBAC (now supported in Zed). - Close of stale bugs - Bad: - The use of devstack for newcomers, in particular because of Neutron. It was proposed to implement vagrant files that will work with ant IDE and will allow to deploy OpenStack easily. - Scenario CI jobs still running in Ubuntu Focal due to the nested virt provider issues: https://bugs.launchpad.net/neutron/+bug/1999249 - To be improved: - Engagement with students to join the Neutron community *Stadium projects.* SQLAlchemy 2.0 is the main problem right now. The CI "-sqlalchmey-master" jobs should help us to prepare these projects for the next framework version and we need to start now. Train branch is broken in most of the CIs. With https://review.opendev.org/c/openstack/releases/+/878202, all projects will be migrated to EOL. networking-odl will be deprecated: it is still testing against ODL 11, while the latest version is 16. Due to the lack of maintainers, we are no longer delivering a new release. *Release cadence.* New meeting the first Tuesday of each month (Neutron team meeting) to decide when to release a new project version. Transition Train to EOL: https://review.opendev.org/c/openstack/releases/+/878202 *python-neutronclient deprecation.* Both the CLI and the python bindings code: https://etherpad.opendev.org/p/python-neutronclient_deprecation We decided not to spend too much time on this goal, as long as there are other priorities. But we are going to mark this project as deprecated (README and mail); that means we'll only push patches for bugs but no new feature will be accepted. If any project is still using the CLI or the bindings and needs a new feature, it will need to migrate to SDK (that will help us on this common effort). *Port hints feature.* https://review.opendev.org/q/topic:port-hints It was considered to use the "tags" resource attribute but this functionality is very specific to the backend and the property is by default admin-only. We'll keep the current implementation (under review right now). *Stateless security groups.* This issue was found during the implementation of stateless SG in ML2/OVN. Main problems: - Semantics and mechanics defined in the documentation and the API. - DHCP and DHCPv6: as long as other FWs (ML2/OVS native and iptables) support both by default with stateless SGs, it will be implicitly supported too in ML2/OVN. That will be documented. Rationale: DHCP is a networking service and we should be able to provide IP in any case. - Metadata: no support. Users will need to explicitly define the egress/ingress rules to allow the metadata traffic (as is now in ML2/OVS FW drivers). *IPv6 prefix delegation.* https://bugs.launchpad.net/neutron/+bug/1895972 It was decided not to implement this feature during this release. At the same time it was decided to define L3 agent PD feature as "experimental" because the client used, dibbler, ended the support: https://bugs.launchpad.net/neutron/+bug/1916428 *Neutron agent status report.* https://bugs.launchpad.net/neutron/+bug/2011422 It was accepted to have a better report from the agent side of the current status, including if it was during the initial transient period, if the agent was doing a full sync, if it was restarted, etc. This functionality should also provide the ability of reporting customized status for particular agents. *DHCP agents with IPv6 metadata services.* https://bugs.launchpad.net/neutron/+bug/1953165 Several options were considered. The approved one was (3): to schedule a single IPv6 metadata agent per network, regardless of the HA number of DHCP agents. That will prevent the IPv6 DAD error now present; the DHCP agent will use that to stop spawning a new metadata agent (it will consider another DHCP agent has a running metadata server). No DB/API changes will be needed. *sRBAC, phase 2.* A new role called "service" will be created. It will identify the server to service calls during the normal operation. For example: Nova to Neutron port binding activation. During this cycle we need to identify the server-2-server calls, performed by other projects against the Neutron API and update these actions with the new role. *HWOL API for ports.* Same topic as commented during the last PTG: the HWOL functionality implies modifying the port binding, that is an admin-only property (and should always be like that). Normal users cannot create and request a port with this functionality if that port was not created before by an admin. It was also suggested (but at least rejected by me: config driven API is something now recommended) to have a configuration option to allow the creation of HWOL ports by default. *Delete on termination for Neutron ports.* The goal of this new feature is to mark the port in order to know if it needs to be deleted after the VM is deleted or not. This functionality is valid for user created ports, not Nova created ports (the last ones are always deleted when the VM is). Nova folks are investigating if "tags" API is valid or not for this RFE. If not, a new API will be proposed for Neutron ports. *Allow to specify the hardware model in the Neutron port.* The goal of this feature is to tint the port with a HW model. For example "hw_vif_model=virtio" or "hw_vif_model=e1000". Same as before, investigate if "tags" is valid or not. We are also considering improving "tags" to be a key-value model, instead of a list of strings only. *Add support for Napatech LinkVirt SmartNICs* https://review.opendev.org/c/openstack/nova-specs/+/859290 The OVS service is running on the NIC HW. It uses DPDK port representors (similar to devlink port representors but not present in kernel). There is a POC in gerrit to be reviewed (patches in the etherpad). The VNIC type was added some releases ago. The main issue of this RFE: the lack of CI support. This is why the main request for this company is to implement an external CI for zuul, adding some basic tempest tests. *Port binding activation duplication.* https://bugs.launchpad.net/neutron/+bug/1986003 It was decided to implement the current solution ( https://review.opendev.org/c/openstack/neutron/+/853281/) if a duplicated call is received by Neutron, it will raise a Conflict (409) exception, handled by Nova. *SR-IOV port tracking in Placement.* Still under definition. Several alternatives to the current ML2/SRIOV agent-resource reporting (for QoS) are presented, some of them exclusive to the current model. There is a etherpad to track the spec definition: https://etherpad.opendev.org/p/track-sriov-nics-in-placement *"skip-level" jobs this cycle (non SLURP).* It was decided: - To continue testing it but testing master against Zed. - To move the jobs to experimental and periodic queues. *ovn-bgp-agent roadmap.* It was decided to add ovn-bgp-agent to the Neutron stadium. The migration will be done during this release. The current API, provided by networking-bgpvpn, is not enough. It is under consideration to improve it or create a new one, just for OVN. It is also needed to improve the current documentation: installation, current uses, drivers available, etc. *Extend OVN db sync tool.* Now the OVN DB sync tool removes anything in the OVN NB DB that is not present in the Neutron DB. That clashes with some deployments, for example when using the OVN-IC to connect several nodes. E.g.: http://dani.foroselectronica.es/ovn-cluster-interconnection-567/ It will be proposed a pluggable interface to make the sync tool aware of these NB objects that have no counterpart in the Neutron DB but in other OVN DB (ovn-ic-nb, for example). *FIPS jobs, OS support.* Currently this functionality is only tested in CentOS8. CentOS9 has an issue with the conntrack module: https://bugs.launchpad.net/neutron/+bug/1976323 These jobs are not tested in Ubuntu in the CI because this is a paid feature. *Vancouver PTG.* We have registered the Neutron team for the Vancouver PTG. The topics to be discussed will be tracked in https://etherpad.opendev.org/p/neutron-vancouver-2023. Regards.

1 0