August 2023 - openstack-discuss - lists.openstack.org

[openstack][kolla-ansible]about group node config overwrite
by Nguyễn Hữu Khôi 29 Oct '25

29 Oct '25

Hello Koller. I want to ask if we can overwrite config for a group of nodes. For example, compute01 >> compute10 will have the same nova configuration compute11 >> compute20 will have the same nova configuration. It will be very nice if we support this way. Thank you. Regards Nguyen Huu Khoi

4 7

Fwd: DPDK+OVS with OpenStack
by Mark Wittling 23 Apr '25

23 Apr '25

Looking for someone who knows OpenStack with OpenVSwitch, and in addition to that, DPDK with OpenStack and OVS.I am using OpenStack Queens, with OpenVSwitch. The architecture I am using is documented here: https://docs.openstack.org/neutron/queens/admin/deploy-ovs-provider.htmlThe OVS I am using on the Compute Node, is compiled with DPDK, and I have enabled the datapath to netdev (DPDK) on br-prv (provider network bridge), and br-tun (tunneling bridge). But these two bridges, br-tun and br-prv, are patched into another OpenStack bridge, called br-int. I wasn’t actually sure about whether to tinker with this bridge, and wondered what datapath it was using.Then, I realized there is a parameter in the openvswitch_agent.ini file, which I will list here: # OVS datapath to use. 'system' is the default value and corresponds to the # kernel datapath. To enable the userspace datapath set this value to 'netdev'. # (string value) # Possible values: # system - <No description provided> # netdev - <No description provided> #datapath_type = system datapath_type = netdev So in tinkering with this, what I realized, is that when you set this datapath_type to system or netdev, it will adjust the br-int bridge to that datapath type.So here is my question. How can I launch a non-DPDK VM, if all of the bridges are using the netdev datapath type?Here is another question. What if one of the flavors don’t have the largepages property set on them? I assumed OpenStack would revert to a system datapath and not use DPDK for those VM interfaces. Well, I found out in testing, that is not the case. If you set all your bridges up for netdev, and you don’t set the property on the Flavor of the VM (largepages), the VM will launch, but it simply won’t work.Is there no way, on a specific Compute Host, to support both DPDK (netdev datapaths) and non-DPDK (system datapaths)?Either on a VM interface level (VM has one interface that is netdev DPDK and another that is system datapath non-DPDK)?Or on a VM by VM basis (VM 1 has 1 or more netdev datapath interfaces and VM 2 has 1 or more system datapath interfaces)?Am I right here? Once you set up a Compute Host for DPDK, it’s DPDK or nothing on that Compute Host? (edited)

3 2

How to config DPDK compute in OVN
by dangerzone ar 22 Apr '25

22 Apr '25

Hi....as far I understand ovs does support dpdk compute to be deployed. I'm referring to the guideline ' *https://docs.openstack.org/networking-ovn/queens/admin/dpdk.html <https://docs.openstack.org/networking-ovn/queens/admin/dpdk.html>* ' ... It's too brief.. May I know how to config dpdk compute in OVN networking openstack? Hope someone could help what is the config to be done.... Currently compute is run without dpdk over openstack train ovn based. Please need some info on how I can proceed. Thank you

2 1

ERROR openstack [-] Resource OS::TripleO::Network::Ports::ControlPlaneVipPort maps to type OS::Neutron::Port and the Neutron service is not available when using ephemeral Heat.| Openstack tripleo wallaby version
by Swogat Pradhan 23 Dec '24

23 Dec '24

Hi, I am currently trying to deploy openstack wallaby using tripleo arch. I created the network jinja templates, ran the following commands also: #openstack overcloud network provision --stack overcloud --output networks-deployed-environment.yaml custom_network_data.yaml # openstack overcloud network vip provision --stack overcloud --output vip-deployed-environment.yaml custom_vip_data.yaml # openstack overcloud node provision --stack overcloud --overcloud-ssh-key /home/stack/sshkey/id_rsa overcloud-baremetal-deploy.yaml and used the environment files in the openstack overcloud deploy command: (undercloud) [stack@hkg2director ~]$ cat deploy.sh #!/bin/bash THT=/usr/share/openstack-tripleo-heat-templates/ CNF=/home/stack/ openstack overcloud deploy --templates $THT \ -r $CNF/templates/roles_data.yaml \ -n $CNF/workplace/custom_network_data.yaml \ -e ~/containers-prepare-parameter.yaml \ -e $CNF/templates/node-info.yaml \ -e $CNF/templates/scheduler-hints.yaml \ -e $CNF/workplace/networks-deployed-environment.yaml \ -e $CNF/workplace/vip-deployed-environment.yaml \ -e $CNF/workplace/overcloud-baremetal-deployed.yaml \ -e $CNF/workplace/custom-net-bond-with-vlans.yaml Now when i run the ./deploy.sh script i encounter an error stating: ERROR openstack [-] Resource OS::TripleO::Network::Ports::ControlPlaneVipPort maps to type OS::Neutron::Port and the Neutron service is not available when using ephemeral Heat. The generated environments from 'openstack overcloud baremetal provision' and 'openstack overcloud network provision' must be included with the deployment command.: tripleoclient.exceptions.InvalidConfiguration: Resource OS::TripleO::Network::Ports::ControlPlaneVipPort maps to type OS::Neutron::Port and the Neutron service is not available when using ephemeral Heat. The generated environments from 'openstack overcloud baremetal provision' and 'openstack overcloud network provision' must be included with the deployment command. 2022-04-19 13:47:16.582 735924 INFO osc_lib.shell [-] END return value: 1 Can someone tell me where the mistake is? With regards, Swogat Pradhan

4 4

[kolla] [openeuler] CI jobs status
by Michał Nasiadka 26 Nov '24

26 Nov '24

Hello Koalas, OpenEuler jobs in master (to be Antelope) are failing for a long time. If no one is interested in maintaining OpenEuler support in Kolla-Ansible - I will propose dropping it’s support on Wednesday Kolla meeting. Best regards, Michal

4 3

Kolla Ansible on Ubuntu 20.04 - cloud-init & other network issues
by Tobias McNulty 10 Oct '24

10 Oct '24

Hi, I'm attempting to use Kolla Ansible 14.6.0 to deploy OpenStack Yoga on a small 3-node Ubuntu 20.04 cluster. The nodes have 128 GB RAM each, dual Xeon processors, and dual 10G Intel NICs. The NICs are connected to access ports on a 10G switch with separate VLANs for the local and external networks. All the playbooks run cleanly, but cloud-init is failing in the Ubuntu 20.04 and 22.04 VMs I attempt to boot. The VM images are unmodified from https://cloud-images.ubuntu.com/, and cloud-init works fine if I mount a second volume with user-data. The error is a timeout attempting to reach 169.254.169.254. This occurs both when booting a VM in an internal routed network and directly in an external network. I tried various neutron plugin agents (ovn, linuxbridge, and openvswitch both with and without firewall_driver = openvswitch <https://docs.openstack.org/kolla-ansible/latest/reference/networking/neutro…>) first with a clean install of the entire OS each time, all with the same result. Running tcpdump looking for 169.254.169.254 shows nothing. As a possible clue, the virtual NICs are unable to pass any traffic (e.g., to reach an external DHCP server) unless I completely disable port security on the interface (even if the associated security group is wide open). But disabling port security does not fix cloud-init (not to mention I don't really want to disable port security). Are there any additional requirements related to deploying OpenStack with Kolla on Ubuntu 20.04? This is a fairly vanilla configuration using the multinode inventory as a starting point. I tried to follow the Quick Start <https://docs.openstack.org/kolla-ansible/yoga/user/quickstart.html> as closely as possible; the only material difference I see is that I'm using the same 3 nodes for control + compute. I am using MAAS so it's easy to get a clean OS install on all three nodes ahead of each attempt. I plan to try again with the standard (non-HWE) kernel just in case, but otherwise I am running out of ideas. In case of any additional clues, here are my globals.yml and inventory file, along with the playbook I'm using to configure the network, images, VMs, etc., after bootstrapping the cluster: https://gist.github.com/tobiasmcnulty/7dbbdbc67abc08cbb013bf5983852ed6 Thank you in advance for any advice! Cheers, Tobias

5 9

Live migration fails
by Szabo, Istvan (Agoda) 06 Aug '24

06 Aug '24

Hi, I have couple of compute nodes where the live migration fails with existing vms. When I quickly spawn a vm and try live migration it works so I assume shouldn't be a big problem with the compute node. However I have many existing vms where it fails with a servername not found. /var/log/nova/nova-conductor.log:2021-04-21 14:47:12.605 227612 ERROR nova.conductor.tasks.migrate [req-f4067a26-a233-4673-8c07-9a8a290980b0 dce35e6eceea4312bb0baa0510cef363 ca7e35079f4440c78bd9870724b9638b - default default] [instance: 1517a2ac-3b51-4d8d-80b3-89a5614d1ae0] Unable to find record for source node servername on servername: ComputeHostNotFound: Compute host servername could not be found. /var/log/nova/nova-conductor.log:2021-04-21 14:47:12.605 227612 WARNING nova.scheduler.utils [req-f4067a26-a233-4673-8c07-9a8a290980b0 dce35e6eceea4312bb0baa0510cef363 ca7e35079f4440c78bd9870724b9638b - default default] Failed to compute_task_migrate_server: Compute host servername could not be found.: ComputeHostNotFound: Compute host servername could not be found. /var/log/nova/nova-conductor.log:2021-04-21 14:47:12.605 227612 WARNING nova.scheduler.utils [req-f4067a26-a233-4673-8c07-9a8a290980b0 dce35e6eceea4312bb0baa0510cef363 ca7e35079f4440c78bd9870724b9638b - default default] [instance: 1517a2ac-3b51-4d8d-80b3-89a5614d1ae0] Setting instance to ACTIVE state.: ComputeHostNotFound: Compute host servername could not be found. /var/log/nova/nova-conductor.log:2021-04-21 14:47:12.672 227612 ERROR oslo_messaging.rpc.server [req-f4067a26-a233-4673-8c07-9a8a290980b0 dce35e6eceea4312bb0baa0510cef363 ca7e35079f4440c78bd9870724b9638b - default default] Exception during message handling: ComputeHostNotFound: Compute host am-osfecn-4025 Tried with this command: nova live-migration --block-migrate id. Any idea? Thank you. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

5 16

[ptl][tc] OpenStack packages PyPi additional external maintainers audit & cleanup
by Ghanshyam Mann 16 May '24

16 May '24

Hi PTLs, As you might know or have seen for your project package on PyPi, OpenStack deliverables on PyPi have additional maintainers, For example, https://pypi.org/project/murano/, https://pypi.org/project/glance/ We should keep only 'openstackci' as a maintainer in PyPi so that releases of OpenStack deliverables can be managed in a single place. Otherwise, we might face the two sets of maintainers' places and packages might get released in PyPi by additional maintainers without the OpenStack project team knowing about it. One such case is in Horizon repo 'xstatic-font-awesome' where a new maintainer is added by an existing additional maintainer and this package was released without the Horizon team knowing about the changes and release. - https://github.com/openstack/xstatic-font-awesome/pull/2 To avoid the 'xstatic-font-awesome' case for other packages, TC discussed it in their weekly meetings[1] and agreed to audit all the OpenStack packages and then clean up the additional maintainers in PyPi (keep only 'openstackci' as maintainers). To help in this task, TC requests project PTL to perform the audit for their project's repo and add comments in the below etherpad. - https://etherpad.opendev.org/p/openstack-pypi-maintainers-cleanup Thanks to knikolla to automate the listing of the OpenStack packages with additional maintainers in PyPi which you can find the result in output.txt at the bottom of this link. I have added the project list of who needs to check their repo in etherpad. - https://gist.github.com/knikolla/7303a65a5ddaa2be553fc6e54619a7a1 Please complete the audit for your project before March 15 so that TC can discuss the next step in vPTG. [1] https://meetings.opendev.org/meetings/tc/2023/tc.2023-01-11-16.00.log.html#… -gmann

18 27

Accessing virsh in openstack wallaby | tripleo
by Swogat Pradhan 24 Mar '24

24 Mar '24

Hi, Can someone please tell me how can i access the virsh utility on the compute node on openstack wallaby? Previously we used nova_libvirt but in wallaby there are multiple containers and i am unable to use virsh utility to edit my domain.xml file. List of containers: [root@dcn01-hci-2 ~]# podman ps | grep nova 10d981158842 172.25.201.68:8787/tripleomaster/openstack-nova-libvirt:current-tripleo kolla_start 2 months ago Up 2 months ago nova_virtlogd_wrapper e91667516afc 172.25.201.68:8787/tripleomaster/openstack-nova-libvirt:current-tripleo kolla_start 2 months ago Up 2 months ago nova_virtsecretd a4b07fe0e833 172.25.201.68:8787/tripleomaster/openstack-nova-libvirt:current-tripleo kolla_start 2 months ago Up 2 months ago nova_virtnodedevd d7e1db393e9c 172.25.201.68:8787/tripleomaster/openstack-nova-libvirt:current-tripleo kolla_start 2 months ago Up 2 months ago nova_virtstoraged 5308a171793e 172.25.201.68:8787/tripleomaster/openstack-nova-libvirt:current-tripleo kolla_start 2 months ago Up 2 months ago nova_virtqemud f490a6249ba1 172.25.201.68:8787/tripleomaster/openstack-nova-libvirt:current-tripleo kolla_start 2 months ago Up 6 weeks ago nova_virtproxyd ab0441e56957 172.25.201.68:8787/tripleomaster/openstack-nova-compute:current-tripleo kolla_start 2 months ago Up 2 months ago (healthy) nova_migration_target f35fbba5b690 172.25.201.68:8787/tripleomaster/openstack-nova-compute:current-tripleo kolla_start 2 months ago Up 6 weeks ago (unhealthy) nova_compute 82bc01993106 172.25.201.68:8787/tripleomaster/openstack-nova-libvirt:current-tripleo /usr/sbin/virtlog... 2 months ago Up 2 months ago Can someone please suggest how I can i edit an instance xml file in this setup? With regards, Swogat Pradhan

2 5

[Watcher] How use Watcher to scheduler Openstack resource without cpu_usage
by Nguyễn Hữu Khôi 06 Mar '24

06 Mar '24

Hello guys. I want to use watcher to scheduler openstack resources but some openstack versions are missing cpu_usage metric. How do we use Watcher without it? Thank you. Nguyen Huu Khoi

3 5

[neutron][vpnaas][ovn] Pushing VPNaaS support through OVN
by Mohammed Naser 12 Feb '24

12 Feb '24

Hi all! One of the biggest show stoppers for us for OVN is VPNaaS but I think it’s time that we’d like to be involved in getting it over the line, I want to thank Bodo who has shown a lot of patience in getting the code necessary, and it seems that the code works for them in production as well. I’d like to ask if possible for folks to review these two outstanding changes: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/847007 https://review.opendev.org/c/openstack/neutron-vpnaas/+/765353 I know Slawek had some concerns, so hoping that raising via the email list if Bodo can address those and over the next bit of time we can get this landed in, since it even has actual proper testing both in CI and with user feedback. Thanks! Mohammed

6 8

[designate] Proposal to deprecate the agent framework and agent based backends
by Michael Johnson 09 Jan '24

09 Jan '24

TLDR: The Designate team would like to deprecate the backend agent framework and the agent based backends due to lack of development and design issues with the current implementation. The following backends would be deprecated: Bind9 (Agent), Denominator, Microsoft DNS (Agent), Djbdns (Agent), Gdnsd (Agent), and Knot2 (Agent). Designate includes many backend DNS server drivers[1], many of which are "native" (also known as xfr type backends) backend implementations. In addition to the "native" backends, Designate has an agent backend[2] that supports other backends via an agent process. To quote the agent backend documentation[2]: This backend uses an extension[3] of the DNS protocol itself to send management requests to the remote agent processes, where the requests will be actioned. The rpc traffic between designate and the agent is both unauthenticated and unencrypted. Do not run this traffic over unsecured networks. Here are the reasons we are proposing to deprecate the agent framework now: 1. The agent protocol used by Designate is using an "unassigned"[4][5] DNS opcode (14) that is causing problems with the dnspython library >= 2.3.0 which is now validating the opcode when building DNS messages. It is a bad practice to use "unassigned" values as they may be officially assigned at any time, likely with an incompatible message format. 2. The agent backends are not tested in the OpenStack jobs[1]. 3. Many of the agent backends have been marked as "Experimental" since 2016 with no additional contributions beyond general repository code maintenance. 4. The protocol between the Designate worker process and the agent process is unauthenticated and unencrypted. 5. We do not know of a development resource to rewrite the agent framework protocol to address issue #1 and #4 above. 6. The introduction of catalog zones[6] may eliminate the need for some of the agent based backend drivers. By marking the agent framework and agent based backends "deprecated" in the Antelope cycle, we would remove the code no earlier than the "C" release of OpenStack (per the OpenStack deprecation policy[7]). In the meantime, issue #1 has been worked around by overriding the dnspython opcode validation[7] as needed in the Designate code (similar to a monkey patch). This is not a sustainable long term solution. The following backend agent based drivers would be marked "deprecated" in addition to the agent framework itself: Bind9 (Agent) Denominator Microsoft DNS (Agent) Djbdns (Agent) Gdnsd (Agent) Knot2 (Agent) We plan to propose patches for the deprecation over the next week, but will not merge them until at least January 24th to allow time for comment from the community. If you have concerns about this deprecation plan or are interested in rewriting the agent framework protocol to address the above issues, please reply to this announcement. Michael [1] https://docs.openstack.org/designate/latest/admin/support-matrix.html [2] https://docs.openstack.org/designate/latest/admin/backends/agent.html [3] https://github.com/openstack/designate/blob/master/designate/backend/privat… [4] https://www.rfc-editor.org/rfc/rfc6895.html#section-2.2 [5] https://www.iana.org/assignments/dns-parameters/dns-parameters.xhtml#dns-pa… [6] https://www.ietf.org/archive/id/draft-ietf-dnsop-dns-catalog-zones-08.txt [7] https://docs.openstack.org/project-team-guide/deprecation.html [8] https://review.opendev.org/c/openstack/designate/+/870678

3 13

[swift][OpenStackSDK][keystone] Warnings about (my) swift endpoint URL by keystoneauth1 discovery
by Christian Rohmann 13 Nov '23

13 Nov '23

Hey openstack-discuss, we use Ceph RadosGW (Quincy release) to provide Swift as part of an OpenStack cloud. The endpoints are configured as such: > # openstack endpoint list > +----------------------------------+--------+--------------+----------------+---------+-----------+----------------------------------------------------------------------+ > | ID | Region | Service Name | Service > Type | Enabled | Interface | URL | > +----------------------------------+--------+--------------+----------------+---------+-----------+----------------------------------------------------------------------+ > | 1234567890 | region | swift | object-store | True | > public | > https://object-store.region.cloud.example.com/swift/v1/AUTH_%(tenant_id)s > | > | 2345678901 | region | swift | object-store | True | > internal | > https://object-store.region.cloud.example.com/swift/v1/AUTH_%(tenant_id)s > | > | 3456789012 | region | swift | object-store | True | > admin | > https://object-store.region.cloud.example.com/swift/v1/AUTH_%(tenant_id)s > | > so according to https://docs.ceph.com/en/latest/radosgw/keystone/#cross-project-tenant-acce…, if I am not mistaken. I can also use the openstack client to access containers: > # openstack container list > +---------------+ > | Name | > +---------------+ > | containera | > | containerb | > +---------------+ and there is no warnings or other apparent issues. But when using "project cleanup" via e.g. "openstack project cleanup --dry-run --project $PROJECT" I see multiple warnings like > Failed to contact the endpoint at > https://object-store.region.cloud.example.com/swift/v1/AUTH_f2bc4bd34567ddc… > for discovery. Fallback to using that endpoint as the base url. > but only mentioning the swift endpoint. Following the warning string this originates from the discovery method of keystoneauth1, see https://github.com/openstack/keystoneauth/blob/28048af9593740315df8d9027c3b…. Could anybody help me understand what is issue might be? Why does this warning not appear for every use of object storage / swift via openstackclient? Is there any way for me to "fix" this? Could this even be a bug? Thanks and with kind regards Christian

4 10

[kolla] [train] [cinder] Cinder issues during controller replacement
by Albert Braden 19 Oct '23

19 Oct '23

We’re replacing controllers, and it takes a few hours to build the new controller. We’re following this procedure to remove the old controller: https://docs.openstack.org/kolla-ansible/train/user/adding-and-removing-hos… After that the cluster seems to run fine on 2 controllers, but approximately 1/3 of our volumes can’t be attached to a VM. When we look at those volumes, we see this: | os-vol-host-attr:host | qde4-ctrl1.cloud.ourdomain.com@rbd-1#rbd-1 | Ctrl1 is the controller that is being replaced. Is it possible to change the os-vol-host-attr on a volume? How can we work around this issue while we are replacing controllers? Do we need to disable the API for the duration of the replacement process, or is there a better way?

6 9

[OPENSTACK][rabbitmq] using quorum queues
by Nguyễn Hữu Khôi 17 Oct '23

17 Oct '23

Hello guys. IS there any guy who uses the quorum queue for openstack? Could you give some feedback to compare with classic queue? Thank you. Nguyen Huu Khoi

9 19

[all][tc] python 3.11 testing plan
by Ghanshyam Mann 27 Sep '23

27 Sep '23

Hi All, Some of you are part of discussion for python 3.11 testing but If you are not aware of it, below is the plan for python 3.11 testing in OpenStack. Non voting in 2023.2 ------------------------- You might have seen that python 3.11 job is now running as non voting in all projects[1]. Idea is to run it as non voting for this (2023.2) cycle which will give projects to fix the issue and make it green. As it is running on debian (frickler mentioned the reason of running it in debian in gerrit[2]), it need some changes in bindep.txt file to pass. Here is the example of fix[3] which you can do in your project also. Voting in 2024.1 -------------------- In next cycle (2024.1), I am proposing to make py3.11 testing mandatory [4] and voting (automatically via common python job template). You need to fix the failure in this cycle otherwise it will block the gate once the next cycle development start (basically once 891238 is merged). [1] https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/891227/5 [2] https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/891146/1 [3] https://review.opendev.org/c/openstack/nova/+/891256 [4] https://review.opendev.org/c/openstack/governance/+/891225 [5] https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/891238 -gmann

12 27

[ironic] Mass retirement of old branches due to config errs
by Jay Faulkner 18 Sep '23

18 Sep '23

Hi all, Ironic is one of the major offenders remaining in the work to cleanup Zuul config errors. To resolve these issues, it's my plan to retire any impacted branches unless a contributor volunteers to clean them up. I will give folks until at least September 1, 2023, to object before I begin to take action. These branches have been broken for at least one year, and have gotten no patches or releases in that time. It's clear they have little need to continue existing. Please look at https://zuul.opendev.org/t/openstack/config-errors for the full list of potentially impacted branches; I will list here as best I can: openstack/ironic-prometheus-exporter - bugfix/2.1 - stable/train openstack/networking-generic-switch - stable/pike - stable/train - stable/ussuri - stable/victoria - stable/wallaby - stable/xena openstack/python-ironicclient - stable/train - stable/ussuri - stable/victoria - stable/wallaby - stable/xena - Note: stable/yoga is also in the config errors list, but is considered maintained and cannot be retired at this time. I will work to remove failing jobs and get this zuul config fixed manuallt. As an additional note, the following branches were retired months ago, but due to a bug were recreated. They have been fully deleted (again): openstack/ironic-python-agent - bugfix/8.0 openstack/ironic - bugfix/18.0

3 4

[cinder][kolla-ansible] cinder-volume active/active usage
by 韩光宇 14 Sep '23

14 Sep '23

Hello, I saw that kolla-ansible had allow cinder coordination backend to be configured. And it can be set to "redis" or "etcd" in /etc/kolla/gloabls.yml [1]. Based on my current understanding, this option enables cinder-volume active/active. 1. I would like to ask, between Redis and Etcd, which one is more recommended or what are their pros and cons? 2. And I have saw the spec of cinder-volume a/a[2]. Is it sufficient to configure [coordination]backend_url and [DEFAULT]cluster to use Cinder volume active/active, or are there any other considerations? Could any reference for Cinder volume active/active configuration and usage documentation be recommended ? I would appreciate for any help. Best wishes. Han Guangyu [1] https://github.com/openstack/kolla-ansible/blob/master/etc/kolla/globals.ym… [2] https://specs.openstack.org/openstack/cinder-specs/specs/ocata/ha-aa-job-di…

3 2

[ptl][tc][winstackers] Final call for Winstackers PTL and maintainers
by James Page 11 Sep '23

11 Sep '23

Hi All As announced by Lucian last November (see [0]) Cloudbase Solutions are no longer in a position to maintain support for running OpenStack on Windows and have also ceased operation of their 3rd party CI for the windows support across a number of OpenStack projects. This situation has resulted in the Winstackers project becoming PTL-less for the 2023.2 cycle with no volunteers responding to the TC's call to fill this role and take this feature in OpenStack forward (see [1]). This is the final call for any maintainers to step forward if this feature is important to them in OpenStack. The last user survey in 2022 indicated that 2% of respondents were running on Hyper-V so this might be important enough to warrant a commitment from someone operating OpenStack on Windows to maintain these features going forward. Here is a reminder from Lucian's original email on the full list of projects which are impacted in some way: * nova hyper-v driver - in-tree plus out-of-tree compute-hyperv driver * os-win - common Windows library for Openstack * neutron hyperv ml2 plugin and agent * ovs on Windows and neutron ovs agent support * cinder drivers - SMB and Windows iSCSI * os-brick Windows connectors - iSCSI, FC, SMB, RBD * ceilometer Windows poller * manila Windows driver * glance Windows support * freerdp gateway The lack of 3rd party CI for testing all of this really needs to be addressed as well. If no maintainers are forthcoming between now and the next PTG in June the TC will need to officially retire the project and start the process of removing support for Windows across the various projects that support this operating system in some way - either directly or through the use of os-win. For clarity this call refers to the use of the Hyper-V virtualisation driver and associated Windows server components to provide WIndows based OpenStack Hypervisors and does not relate to the ability to run Windows images as guests on OpenStack. Regards James [0] https://lists.openstack.org/pipermail/openstack-discuss/2022-November/03104… [1] https://lists.openstack.org/pipermail/openstack-discuss/2023-March/032888.h…

5 9

[Skyline] error username or password incorrect
by Satish Patel 10 Sep '23

10 Sep '23

Folks, Try to install skyline UI to replace horizon using doc: https://docs.openstack.org/skyline-apiserver/latest/install/docker-install-… Everything went well and I got a login page on http://x.x.x.x:9999 also it pulled Region/Domains. When I am trying to login with my account, I get an error: Username or Password is incorrect. I am using sqlite DB for skyline as per documents. No errors in logs command $ docker logs skyline When I use Chrome Developer Tools then it was indicating an error in these URLs. http://openstack.example.com:9999/api/openstack/skyline/api/v1/profile http://openstack.example.com:9999/api/openstack/skyline/api/v1/policies 401 Unauthorized ( {"detail":"no such table: revoked_token"} )

2 16

[openstack-ansible] Recommended branch for a production environment?
by Roger Rivera 05 Sep '23

05 Sep '23

Hello, We have deployed Openstack-Ansible in a test environment and we were wondering what the recommended repository branch is to deploy in a production environment that will be integrated with a separate Ceph cluster. 1. We are thinking about pinning to the stable/zed. Is that recommended over the master branch? 2. Are bugs ironed out on both master and stable/zed with the same cadence? 3. Additionally, is Debian 11 a better alternative than Ubuntu 22 for target hosts? We noticed Ubuntu 22.04 support was added recently, whereas Debian 11 has been supported for quite some time now which leads us to believe its stability/integration could be more mature at this point? Any suggestions will be appreciated. Thank you, -- *Roger *

3 7

[SDK][CLI][API][OpenAPI] Machine readable OpenStack API spec - time to do a next step?
by Artem Goncharov 01 Sep '23

01 Sep '23

Hey, Some of you may know that for certain time I was doing experiments on the sdk/cli/ansible_mod side with regards to the code generation. Reason for that is simple: huge codebases and too few resources. While not being the primary goal of this initiative, but still a pointer into the same direction is that most of the tools we need to maintain do the same stuff with some adaptations necessary for the framework in question. I did a lot of diverse attempts trying to take current OpenStackSDK codebase as a “source of truth” regarding what exists in which service and use it to generate code (OSC, AnsibleModule, etc). Pretty quickly it became clear that we do not have enough information to cover all the different needs: - when query parameter has been added into or dropped from the API - what is the type of the query parameter and whether in the case it is list it should be rendered as "?tags=a,b" or "?tags=a&tags=b” - which fields are necessary to be send in the request, which are optional and how to deal when you send name but get back UUID in the same field - what are the parameter/resource fields types in detail with respect to strongly typed programming languages (“int” is not good to differentiate between u8 and f64) - how to determine list of supported “actions” on the resource which are above the plain CRUD Firstly I tried to extend our current model of things in SDK, but even that was not so helpful. And then I decided to give another look at OpenAPI and our desires since multiple years to move into that direction. First of all it provides all the required information to do what is necessary. And so I tried to gather all the things we said about OpenAPI at OpenStack and which challenges we have and compare it against the OpenAPI 3.1. Problems: 1) OpenAPI does not support model of microversions. To be more precise on that you can’t describe different request/response body based on the header differentiator 2) URL + Method are representing unique combination making it impossible to represent different actions based on the content 3) All relevant Headers are supposed to be described in the spec. For example Swift is using X-Object-Meta-* prefix to allow user customisations what is not currently supported by the spec Actually that’s it, everything else was doable. Have I missed something? So what can we do about it: 1) OpenAPI 3.1 is now based upon jsonschema, what allows huge flexibility in describing object schema including inheritance and polymorphism (with composition, discriminator, etc). So we can describe the necessary Microversion Header and schema using “oneOf” whenever necessary. Moreover it is possible to extend type schema with “x-“ prefixed custom properties. So I went and started adding “x-openstack-“ prefixed properties where necessary 2) I personally agree with OpenAPI that URL+method should be enough to uniquely identify what is expected (while not a big deal for the API consumer, it would have been easier on the server side to have URL uniquely tied with the data and operation instead of dynamically looking at the request to figure out what to do. This is actually a most complex thing to deal with and the only solution that comes to my mind would be not to try to put everything into single spec, but rather have a dedicated specs for different operations. It makes it anyway easier to maintain such spec. Maybe at some next OpenAPI iteration this gets resolved and we can adopt changes. 3) A tricky thing, but there is nothing preventing us specifying headers as “x-object-meta-*” and defining custom processors for things like that to for example compose/decompose this into dictionary on the client side OpenAPI now even allows defining spec extensions if necessary, but I think at the moment we can cover all issues without that. Demo time: - server live migration (in my eyes one of the trickiest things I have seen so far from the OpenStack APIs). Here you have an action requiring a dedicated spec, multiple revisions with micro versions and even parameter type bool which can be a string as well: [2] https://paste.opendev.org/show/bcxFi2CrNX1YNEuTzoEh/ More real-life examples in [4] - image create and upload: [3] https://paste.opendev.org/show/bNN1t0qVmHEdVHu0Mtrn/ Here it is possible to put few methods into the single spec. You can also see here samples of different media-types, capture headers, readOnly/writeOnly properties (You can actually take those specs and paste them into https://editor-next.swagger.io/ and see how it looks like) Closing notes: I have implemented a WIP change [1] in SDK that consumes list of specs and performs requested operation. Series of SDK commands are converted to the new way dealing with APIs. This is backwards compatible, but opens a road to heavily reduce amount of code in SDK (precisely by dropping all those customisations that are necessary now). Honestly speaking my drafts even show performance improvement on SDK side in the area of 100-300ms for diverse API operations, but I am still not really able to explain why. Maybe because of avoiding most black magic of SDK. I assume after completion we could get another performance boost by dropping other unnecessary logic and abstractions. The work is not completed as the change title states. Once we have specs for the commands code generators can be updated to consume this data and produce much better code. BTW, I have now also a prototype of the new CLI written with Rust and this is a hypersonic bullet compared to the current OSC. But please do not push me more on that, it is still in a very early stages and depends heavily on the available specs. I can only say that it is now designed in the way that every API call can have a thin CLI coverage just by providing a spec, when additional logic is desired - surely will require human implementation. Code generators in the pipe: OSC, AnsibleModules, RustSDK (sync/async), RustCLI. Next thing that are on the radar: gopher cloud, terraform modules, async python sdk, JS SDK(?) If all of that gets executed properly and with some community traction we can all have following things covered: - improve standardisation of OpenStack internals and externals: glance and nova (at least those 2) are already using jsonschema internally in different areas to describe requests/responses. Why not to make this standard reaching the service consumers? - getting rid of api-ref work by updating our sphinx machinery to consume our customised specs and produce nice docs matching the reality - sharing specs between teams to improve interface (not like currently we need to read the api-ref with tons of bugs plus source code to understand how to cover new feature in service X). Maybe even a central repo with the specs per release. - there are plenty of code generators and server bindings for OpenAPI specs so that we can potentially align frameworks used by different teams to maintain less - less work for all of us who needs services talking to each other (not immediately right now, but once the code is switched on consuming specs) - request verification already on the client side not waiting for the response - finally show something to customers often annoying asking “where are your openapi specs” (no offence here ;-))? I know it is a long message. But I am pretty excited with the progress and would like to hear community opinions. For the more detailed discussion consider this as a pre-announcement of the topic for PTG in sdk/cli slots. Huge invest but huge outcome P.S. it can result in a good chunk of relatively easy work for students Regards, Artem [1] https://review.opendev.org/c/openstack/openstacksdk/+/892161 [2] https://paste.opendev.org/show/bcxFi2CrNX1YNEuTzoEh/ [3] https://paste.opendev.org/show/bNN1t0qVmHEdVHu0Mtrn/ [4] https://review.opendev.org/c/openstack/openstacksdk/+/893365

4 5

[all] osprofiler requirements on opentelemetry: is this reasonable?
by Thomas Goirand 01 Sep '23

01 Sep '23

Hi, I just saw that the last version of osprofiler requires opentelemetry-exporter-otlp and opentelemetry-sdk. Since osprofiler is used in almost all projects, and that opentelemetry is really HUGE, I wonder if this is all reasonable. This forces downstream distros to do a lot of packaging only for that. Is this avoidable? I started doing the packaging, and then stopped to write this message, seeing how much work that would represent. Your thoughts? Cheers, Thomas Goirand (zigo)

3 3

[Rally]
by Aayushi Gautam 31 Aug '23

31 Aug '23

Hello, I am Aayushi , an intern with Redhat working on ESI ( Elastic secure Infracstructure) group. We were trying to use Rally to test the performance of our codebase. I have created a Plugin and task . And getting an error. The same bug is also asked on the bug page of Rally, but it was not answered. I have attached the code of the plugin and task and the error message. Looking forward to hearing from you.

2 5

Issue with Live Migration
by Chris DiLorenzo 31 Aug '23

31 Aug '23

We are running zed deployed with kolla-ansible. We are running into an issue with Live Migration. These are the steps to reproduce the issue: - Create a VM - Attach a Cinder Volume. This will create a new Storage Controller. - Start writing data to the new disk. We used FIO. Let FIO run until it starts reporting IOPS data. - Live migrate the VM - The VM will migrate and FIO will continue to run. FIO will show 0 or no IOPS - lspci output will show that the new storage device is in error. - The VM will remain in this state until the user issues a hard reset command - Once the VM is rebooted, the issue can not be reproduced until another device is attached This also happens if we attach a new VNIC. Prior to migration, this is what we see: root@test-vm-cd:~# cat before_migration_lspci.txt | grep block 05:00.0 SCSI storage controller: Red Hat, Inc. Virtio block device (rev 01) Subsystem: Red Hat, Inc. Virtio block device 08:00.0 SCSI storage controller: Red Hat, Inc. Virtio block device (rev 01) Subsystem: Red Hat, Inc. Virtio block device Post migration we see this: 08:00.0 SCSI storage controller: Red Hat, Inc. Virtio block device (rev ff) (prog-if ff) !!! Unknown header type 7f Kernel driver in use: virtio-pci The original SCSI controller is fine. Outside of the above error, we don't see any other errors. From Openstack's perspective, the migration went fine. Any suggestions on where to look? Has anyone else seen this? Thanks, Chris

1 0

[elections][sahara] PTL Candidacy for 2024.1
by Jerry Zhou (周鑫勇) 31 Aug '23

31 Aug '23

Hello everyone! I would like to announce my candidacy for the role of Sahara PTL during the 2024.1 cycle. I have been in this role for the last cycle, and I would like to continue serving as PTL. During the past cycles, there was a lot of effort put into promoting contributor collaboration through various activities. We also revised unstable unit/functional tests to improve the quality of development, and created documents for users to demonstrate use cases with practical examples. In the next cycle, I'd like to continue improving unit testing and documentation quality.

1 0

TripleO centos9 wallaby IPA job is failing
by Bhagyashri Shewale 31 Aug '23

31 Aug '23

Hi folks, All the centos9 wallaby IPA jobs are failing due to [1] and blocking gate. We are currently working on it. [1]: https://bugs.launchpad.net/tripleo/+bug/2033357 Thanks and regards Bhagyashri Shewale

2 1

Openstack Horizon Login Issues
by Asma Naz Shariq 31 Aug '23

31 Aug '23

Hi Team, Can anyone help me to resolve this issue of openstack dashboard-Yoga release? You do not have permission to access the resource: /horizon/ Login as different user or go back to home page. How to resolve this issue in openstack while accessing web interface. Can anyone guide me how to resolve this issue? when i entered the credentials to access to openstack dashboard, it display mesasge: Something went wrong! An unexpected error has occurred. Try refreshing the page. If that doesn't help, contact your local administrator. Regards, Asma

2 1

migration error
by Karera Tony 31 Aug '23

31 Aug '23

Hello Team, I am migrating an instance from one Cpmpute to another but I am getting this error *Error: *Failed to live migrate instance to host "compute1". Details <http://10.10.13.31/admin/instances/#message_details> Migration pre-check error: Unacceptable CPU info: CPU doesn't have compatibility. 0 Refer to http://libvirt.org/html/libvirt-libvirt-host.html#virCPUCompareResult (HTTP 400) (Request-ID: req-6bc103a8-f1f1-489c-a620-ec535b651280) Please advise Regards Tony Karera

1 0

Re: openstack-discuss Digest, Vol 58, Issue 108
by Karera Tony 31 Aug '23

31 Aug '23

My issue was resolved after upgrading kolla-ansible. Regards Tony Karera On Wed, Aug 30, 2023 at 3:53 PM < openstack-discuss-request(a)lists.openstack.org> wrote: > Send openstack-discuss mailing list submissions to > openstack-discuss(a)lists.openstack.org > > To subscribe or unsubscribe via the World Wide Web, visit > > https://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss > > or, via email, send a message with subject or body 'help' to > openstack-discuss-request(a)lists.openstack.org > > You can reach the person managing the list at > openstack-discuss-owner(a)lists.openstack.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of openstack-discuss digest..." > > > Today's Topics: > > 1. [keystone][election] Self-nomination for Keystone PTL for > 2024.1 cycle (Dave Wilde) > 2. Re: openstack-discuss Digest, Vol 58, Issue 107 (Karera Tony) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 30 Aug 2023 08:35:00 -0500 > From: Dave Wilde <dwilde(a)redhat.com> > To: OpenStack Discuss <openstack-discuss(a)lists.openstack.org> > Subject: [keystone][election] Self-nomination for Keystone PTL for > 2024.1 cycle > Message-ID: <54d82977-aee0-4df2-a7c3-2be3b8c2788c@Spark> > Content-Type: text/plain; charset="utf-8" > > Hey folks, > > It's Dave here, your current OpenStack Keystone PTL. I'd like to submit my > candidacy to again act as the PTL for the 2024.1 cycle.??We're making great > progress with Keystone, and I would like to continue with that excellent > work. > > As PTL in this cycle here are some of the things I'd like to focus on: > > ??- Finish the manager role and ensure that the SRBAC implied roles are > correct > ?? ?for manager and member > ??- Continue the OAuth 2.0 Implementation > ??- Start a known issues section in the Keystone documentation > ??- Start a documentation audit to ensure that our documentation is of the > ?? ?highest quality > > Of course we will continue the weekly meetings and the reviewathons.??I > think > the reviewathons have been very successful and I'm keen on keeping them > going. > > I'm looking forward to another successful cycle and working with everyone > again! > > Thank you! > > /Dave >

1 0

[elections][venus] PTL Candidacy for 2024.1
by Eric Zhang (张连法)-浪潮信息 30 Aug '23

30 Aug '23

Hello everyone! I would like to announce my candidacy for the role of Venus PTL during the Caracal/2024.1 cycle. I have been in this role for the last 2 cycles and I would like to continue serving as PTL. In the upcoming cycle, I would like to focus on the following: * Continuous improvement in documents, unit tests, clients and dashboard. * Enhance log anomaly detection, such as more matching rules, improving performance. Thank you for your consideration! Lianfa Zhang

1 0

[all][elections][tc] TC Election Campaigning Kickoff
by Ian Y. Choi 30 Aug '23

30 Aug '23

Developers, The TC Election Campaigning Period has now started[1]. During the next couple days, you are all encouraged to ask the candidates questions about their platforms[2], opinions on OpenStack, community governance, and anything else that will help you to better determine how you will vote. This is the time to raise any issues you wish the future TC to consider, and to evaluate the opinions of the nominees prior to their election. Candidates, Each of you has posted a platform[2], and announced your nomination to the developers. From this point, you are encouraged to ask each other questions about the posted platforms, and begin discussion of any points that you feel are particularly important during the next cycle. While you are not yet TC members, your voices and opinions about the issues raised in your platforms and questions raised by the wider community will help ensure that the future TC has the widest possible input on the matters of community concern, and the electorate has the best information to determine the ideal TC composition to address these and other issues that may arise. [1] https://governance.openstack.org/election/ [2] https://opendev.org/openstack/election/src/branch/master/candidates/2024.1/…

1 0

[elections][cyborg] PTL Candidacy for 2024.1
by Alex Song (宋文平) 30 Aug '23

30 Aug '23

Hello everyone! I would like to announce my candidacy for the role of Cyborg PTL during the Caracal/2024.1 cycle. I have been in this role for the last 3 cycles and I would like to continue serving as PTL. During the past cycles, there was a lot of effort on promoting contributor collaboration through different activities and it helped us to make progress on closing gaps between the Cyborg APIs and other clients consuming them, as well as tackling tech debt and getting new features and drivers merged. I feel energized to continue pushing on these areas, but also put some focus on: * Complete the new features such as vgpu and mig management, disable and enable devices, update the device attribute and so on. * Working on the stability of our upstream CI first party jobs and also checking with third party CI maintainers to ensure their CI is healthy and reporting valid results. * Enhancing the review bandwidth and doing more actions to encourage more reviews from people submitting code to the Cyborg code base. * Work on documentation to help new contributors understand what are the requirements when proposing new features, and also continue the effort to cover the lack of documentation we have on third party CI setups for Cyborg, helping potential new vendors to quickly set up their CI systems. Thank you for your consideration! Wenping Song IRC: wenping1

1 0

[all][elections][ptl][tc] Combined PTL/TC 2024.1 cycle Election Nominations End
by Ian Y. Choi 30 Aug '23

30 Aug '23

The PTL and TC Nomination period is now over. The official candidate lists for PTLs [0] and TC seats [1] are available on the election website. -- PTL Election Details -- There are 12 projects without candidates, so according to this resolution[2], the TC will have to decide how the following projects will proceed: Cyborg, Ec2_Api, Mistral, Monasca, Openstack_Chef, Rally, Sahara, Senlin, Skyline, Sunbeam, Trove, Venus There are 1 projects that will have elections: OpenStack_Helm. The details for those will be posted around polling start date after we setup the CIVS system. -- TC Election Details -- Now begins the campaigning period where candidates and electorate may debate their statements. Polling will start Sep 06, 2023 23:45 UTC. Thank you, [0] https://governance.openstack.org/election/#ptl-candidates [1] https://governance.openstack.org/election/#tc-candidates [2] https://governance.openstack.org/resolutions/20141128-elections-process-for…

1 0

Cinder Bug Report 2023-08-30
by Jon Bernard 30 Aug '23

30 Aug '23

Hello Argonauts, Cinder Bug Meeting Etherpad <https://etherpad.opendev.org/p/cinder-bug-squad-meeting> Undecided - [StorPool Storage] Live VM Migration Fails <https://bugs.launchpad.net/cinder/+bug/2033292> - Status: In Progress - Dell PowerMax - Online migration can fail <https://bugs.launchpad.net/cinder/+bug/2033398> - Status: New - Hitachi: send unnecessary request to unlock storage resource group <https://bugs.launchpad.net/cinder/+bug/2033448> - Status: In Progress Thanks -- Jon

1 0

Re: openstack-discuss Digest, Vol 58, Issue 107
by Karera Tony 30 Aug '23

30 Aug '23

Hi Michael, I have realized that the docker version I have on the new compute nodes after running the bootstrap command is 24 while the previous compute and controller nodes have 20. Could that be a problem? Regards Regards Tony Karera On Wed, Aug 30, 2023 at 3:22 PM < openstack-discuss-request(a)lists.openstack.org> wrote: > Send openstack-discuss mailing list submissions to > openstack-discuss(a)lists.openstack.org > > To subscribe or unsubscribe via the World Wide Web, visit > > https://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss > > or, via email, send a message with subject or body 'help' to > openstack-discuss-request(a)lists.openstack.org > > You can reach the person managing the list at > openstack-discuss-owner(a)lists.openstack.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of openstack-discuss digest..." > > > Today's Topics: > > 1. Re: [elections][Cinder] PTL Candidacy for 2024.1 (Caracal) > (Brian Rosmaita) > 2. Re: [kolla] [train] [cinder] Cinder issues during controller > replacement (Danny Webb) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 30 Aug 2023 08:31:32 -0400 > From: Brian Rosmaita <rosmaita.fossdev(a)gmail.com> > To: openstack-discuss(a)lists.openstack.org > Subject: Re: [elections][Cinder] PTL Candidacy for 2024.1 (Caracal) > Message-ID: <fa39fe25-6651-688a-4bfa-39abbdbb7e04(a)gmail.com> > Content-Type: text/plain; charset=UTF-8; format=flowed > > On 8/29/23 10:17 AM, Rajat Dhasmana wrote: > > Hi All, > > > > I would like to nominate myself to be Cinder?PTL during the 2024.1 > (Caracal) > > cycle. > [snip] > > Here are some work items we are planning for the next cycle (2024.1): > > > > * We still lack review bandwidth since one of our active cores, Sofia, > > couldn't contribute to Cinder anymore > > hence we will be looking out for potential core reviewers. > > * Continue working on migrating from cinderclient to OSC (and SDK) > support. > > * Continue with the current cinder events like festival of XS reviews, > > midcycle, video meeting once a month etc > > that provides regular team interaction and helps tracking and discussion > > of ongoing work throughout the cycle. > > All this sounds great to me! Thanks for stepping up once again to take > on the task of herding the Argonauts and keeping the Cinder project on > track. > > > > > [1] https://review.opendev.org/q/topic:cinderclient-sdk-migration > > <https://review.opendev.org/q/topic:cinderclient-sdk-migration> > > [2] https://review.opendev.org/q/topic:cinder-sdk-gap > > <https://review.opendev.org/q/topic:cinder-sdk-gap> > > [3] > > > https://docs.google.com/spreadsheets/d/1yetPti2XImRnOXvvJH48yQogdKDzRWDl-NY… > < > https://docs.google.com/spreadsheets/d/1yetPti2XImRnOXvvJH48yQogdKDzRWDl-NY… > > > > [4] > > > https://governance.openstack.org/tc/resolutions/20230724-unmaintained-branc… > < > https://governance.openstack.org/tc/resolutions/20230724-unmaintained-branc… > > > > > > Thanks > > Rajat Dhasmana > > > > > ------------------------------ > > Message: 2 > Date: Wed, 30 Aug 2023 13:19:49 +0000 > From: Danny Webb <Danny.Webb(a)thehutgroup.com> > To: Eugen Block <eblock(a)nde.ag>, > "openstack-discuss(a)lists.openstack.org" > <openstack-discuss(a)lists.openstack.org> > Subject: Re: [kolla] [train] [cinder] Cinder issues during controller > replacement > Message-ID: > < > LO2P265MB5773FCAFE86B26D1E45E31459AE6A(a)LO2P265MB5773.GBRP265.PROD.OUTLOOK.COM > > > > Content-Type: text/plain; charset="windows-1252" > > Just to add to the cornucopia of configuration methods for cinder > backends. We use a variation of this for active/active controller setup > which seems to work (on pure, ceph rbd and nimble). We use both > backend_host and volume_backend_name so any of our controllers can action a > request for a given volume. We've not had any issues with this setup since > ussuri (we're on Zed / Yoga now). > > eg: > > [rbd] > backend_host = ceph-nvme > volume_backend_name = high-performance > ... > > [pure] > backend_host = pure > volume_backend_name = high-performance > ... > > which leaves us with a set of the following: > > $ openstack volume service list > > +------------------+-------------------------+----------+---------+-------+----------------------------+ > | Binary | Host | Zone | Status | State > | Updated At | > > +------------------+-------------------------+----------+---------+-------+----------------------------+ > | cinder-volume | nimble@nimble-az1 | gb-lon-1 | enabled | up > | 2023-08-30T12:34:18.000000 | > | cinder-volume | nimble@nimble-az2 | gb-lon-2 | enabled | up > | 2023-08-30T12:34:14.000000 | > | cinder-volume | nimble@nimble-az3 | gb-lon-3 | enabled | up > | 2023-08-30T12:34:17.000000 | > | cinder-volume | ceph-nvme@ceph-nvme-az3 | gb-lon-3 | enabled | up > | 2023-08-30T12:34:14.000000 | > | cinder-volume | pure@pure-az1 | gb-lon-1 | enabled | up > | 2023-08-30T12:34:11.000000 | > | cinder-volume | pure@pure-az2 | gb-lon-2 | enabled | up > | 2023-08-30T12:34:11.000000 | > > +------------------+-------------------------+----------+---------+-------+----------------------------+ > > > ________________________________ > From: Eugen Block <eblock(a)nde.ag> > Sent: 30 August 2023 11:36 > To: openstack-discuss(a)lists.openstack.org < > openstack-discuss(a)lists.openstack.org> > Subject: Re: [kolla] [train] [cinder] Cinder issues during controller > replacement > > CAUTION: This email originates from outside THG > > Just to share our config, we don't use "backend_host" but only "host" > on all control nodes. Its value is the hostname (shortname) pointing > to the virtual IP which migrates in case of a failure. We only use > Ceph as storage backend with different volume types. The volume > service list looks like this: > > controller02:~ # openstack volume service list > > +------------------+--------------------+------+---------+-------+----------------------------+ > | Binary | Host | Zone | Status | State | > Updated At | > > +------------------+--------------------+------+---------+-------+----------------------------+ > | cinder-scheduler | controller | nova | enabled | up | > 2023-08-30T10:32:36.000000 | > | cinder-backup | controller | nova | enabled | up | > 2023-08-30T10:32:34.000000 | > | cinder-volume | controller@rbd | nova | enabled | up | > 2023-08-30T10:32:28.000000 | > | cinder-volume | controller@rbd2 | nova | enabled | up | > 2023-08-30T10:32:36.000000 | > | cinder-volume | controller@ceph-ec | nova | enabled | up | > 2023-08-30T10:32:28.000000 | > > +------------------+--------------------+------+---------+-------+----------------------------+ > > We haven't seen any issue with this setup in years during failover. > > Zitat von Gorka Eguileor <geguileo(a)redhat.com>: > > > On 29/08, Albert Braden wrote: > >> What does backend_host look like? Should it match my internal API > >> URL, i.e. api-int.qde4.ourdomain.com? > >> On Tuesday, August 29, 2023 at 10:44:48 AM EDT, Gorka Eguileor > >> <geguileo(a)redhat.com> wrote: > >> > > > > Hi, > > > > If I remember correctly you can set it to anything you want, but for > > convenience I would recommend setting it to the hostname that currently > > has more volumes in your system. > > > > Let's say you have 3 hosts: > > qde4-ctrl1.cloud.ourdomain.com > > qde4-ctrl2.cloud.ourdomain.com > > qde4-ctrl3.cloud.ourdomain.com > > > > And you have 100 volumes on ctrl1, 20 on ctrl2, and 10 on ctrl3. > > > > Then it would be best to set it to ctrl1: > > [rbd-1] > > backend_host = qde4-ctrl1.cloud.ourdomain.com > > > > And then use the cinder-manage command to modify the other 2. > > > > For your information the value you see as "os-vol-host-attr:host" when > > seeing the detailed information of a volume is in the form of: > > <HostName>@<BackendName>#<PoolName> > > > > In your case: > > <HostName> = qde4-ctrl1.cloud.ourdomain.com > > <BackendName> = rbd-1 > > <PoolName> = rbd-1 > > > > In the RBD case the poolname will always be the same as the backendname. > > > > Cheers, > > Gorka. > > > >> On 29/08, Albert Braden wrote: > >> > We?re replacing controllers, and it takes a few hours to build > >> the new controller. We?re following this procedure to remove the > >> old controller: > >> > https://docs.openstack.org/kolla-ansible/train/user/adding-and-removing-hos… > < > https://docs.openstack.org/kolla-ansible/train/user/adding-and-removing-hos… > > > >> > > >> > After that the cluster seems to run fine on 2 controllers, but > >> approximately 1/3 of our volumes can?t be attached to a VM. When we > >> look at those volumes, we see this: > >> > > >> > | os-vol-host-attr:host | > >> qde4-ctrl1.cloud.ourdomain.com@rbd-1#rbd-1 > >> | > >> > > >> > Ctrl1 is the controller that is being replaced. Is it possible to > >> change the os-vol-host-attr on a volume? How can we work around > >> this issue while we are replacing controllers? Do we need to > >> disable the API for the duration of the replacement process, or is > >> there a better way? > >> > > >> > >> Hi, > >> > >> Assuming you are running cinder volume in Active-Passive mode (which I > >> believe was the only way back in Train) then you should be hardcoding > >> the host name in the cinder.conf file to avoid losing access to your > >> volumes when the volume service starts in another host. > >> > >> This is done with the "backend_host" configuration option within the > >> specific driver section in cinder.conf. > >> > >> As for how to change the value of all the volumes to the same host > >> value, you can use the "cinder-manage" command: > >> > >> cinder-manage volume update_host \ > >> --currenthost <current host> \ > >> --newhost <new host> > >> > >> Cheers, > >> Gorka. > >> > >> > >> > > > >

1 0

[keystone][election] Self-nomination for Keystone PTL for 2024.1 cycle
by Dave Wilde 30 Aug '23

30 Aug '23

Hey folks, It's Dave here, your current OpenStack Keystone PTL. I'd like to submit my candidacy to again act as the PTL for the 2024.1 cycle. We're making great progress with Keystone, and I would like to continue with that excellent work. As PTL in this cycle here are some of the things I'd like to focus on: - Finish the manager role and ensure that the SRBAC implied roles are correct for manager and member - Continue the OAuth 2.0 Implementation - Start a known issues section in the Keystone documentation - Start a documentation audit to ensure that our documentation is of the highest quality Of course we will continue the weekly meetings and the reviewathons. I think the reviewathons have been very successful and I'm keen on keeping them going. I'm looking forward to another successful cycle and working with everyone again! Thank you! /Dave

1 0

[elections][Cinder] PTL Candidacy for 2024.1 (Caracal)
by Rajat Dhasmana 30 Aug '23

30 Aug '23

Hi All, I would like to nominate myself to be Cinder PTL during the 2024.1 (Caracal) cycle. I have been the Cinder PTL for the last 3 cycles namely Zed, 2023.1 (Antelope) and 2023.2 (Bobcat). Few of the notable things we achieved in the past cycle are: ** OpenStack Client and SDK work: *We started two efforts in parallel: ** Migrate OSC commands to SDK[1] ** Add support of missing operations in SDK[2] We are making good progress in this area but looking at the work items[3], this effort will span across the next couple of cycles to be completed. ** Security Vulnerabilities:* We were able to address a major security vulnerability CVE-2023-2088 that prevents end users to delete volume attachments directly leaving left over devices on compute hosts resulting in data leaks. ** Transition from EM to Unmaintained: *We noticed the inactive contribution in extended maintenance branches and the need to fix gate issues from time to time. Also the project team is responsible to merge changes into these EM branches. After the recent CVE-2023-2088 wasn't proposed to EM branches, it was clear that the efforts were not worth the outcome so the Cinder team initiated a discussion regarding EOLing the current EM branches. There is a resolution from TC to migrate EM to unmaintained branches[4]. ** Outreachy Mentorship: *We had a successful quarter of outreachy internship from May-August 2023 with 2 interns, Desire and Toheeb. They were able to get a good amount of work done on the project "Automating generation of api-ref samples". ** Review bandwidth and quality:* We have been able to maintain good quantity and quality of reviews from contributors contributing from different organizations. Here are some work items we are planning for the next cycle (2024.1): * We still lack review bandwidth since one of our active cores, Sofia, couldn't contribute to Cinder anymore hence we will be looking out for potential core reviewers. * Continue working on migrating from cinderclient to OSC (and SDK) support. * Continue with the current cinder events like festival of XS reviews, midcycle, video meeting once a month etc that provides regular team interaction and helps tracking and discussion of ongoing work throughout the cycle. [1] https://review.opendev.org/q/topic:cinderclient-sdk-migration [2] https://review.opendev.org/q/topic:cinder-sdk-gap [3] https://docs.google.com/spreadsheets/d/1yetPti2XImRnOXvvJH48yQogdKDzRWDl-NY… [4] https://governance.openstack.org/tc/resolutions/20230724-unmaintained-branc… Thanks Rajat Dhasmana

2 1

Adding compute nodes to openstack cluster running wallaby
by Karera Tony 30 Aug '23

30 Aug '23

Hello Team, I am running openstack wallaby deployed using kolla-ansible and I am trying to add two compute node but I am getting this error. failed: [compute3] (item={'key': 'cron', 'value': {'container_name': 'cron', 'group': 'cron', 'enabled': True, 'image': 'kolla/ubuntu-source-cron:wallaby', 'environment': {'DUMMY_ENVIRONMENT': 'kolla_useless_env'}, 'volumes': ['/etc/kolla/cron/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/etc/timezone:/etc/timezone:ro', 'kolla_logs:/var/log/kolla/'], 'dimensions': {}}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "cron", "value": {"container_name": "cron", "dimensions": {}, "enabled": true, "environment": {"DUMMY_ENVIRONMENT": "kolla_useless_env"}, "group": "cron", "image": "kolla/ubuntu-source-cron:wallaby", "volumes": ["/etc/kolla/cron/:/var/lib/kolla/config_files/:ro", "/etc/localtime:/etc/localtime:ro", "/etc/timezone:/etc/timezone:ro", "kolla_logs:/var/log/kolla/"]}}, "msg": "'Traceback (most recent call last):\\n File \"/tmp/ansible_kolla_docker_payload_e49mzp45/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py\", line 1241, in main\\n File \"/tmp/ansible_kolla_docker_payload_e49mzp45/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py\", line 355, in compare_container\\n File \"/tmp/ansible_kolla_docker_payload_e49mzp45/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py\", line 376, in check_container_differs\\n File \"/tmp/ansible_kolla_docker_payload_e49mzp45/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py\", line 571, in compare_dimensions\\nKeyError: \\'KernelMemory\\'\\n'"} Please advise Tony Regards Tony Karera

2 1

[release][ironic][OpenStackSDK][swift] cycle-with-intermediary deliverables without any release
by Herve Beraud 30 Aug '23

30 Aug '23

Hi everyone, Quick reminder that we'll need a release very soon for a number of deliverables following a cycle-with-intermediary release model but which have not done *any* release yet in the 2023.2 Bobcat cycle: The following deliverables have done a bobcat release, but it was not refreshed in the last two months: [ironic] bifrost ironic-prometheus-exporter ironic-python-agent-builder ironic-ui networking-baremetal networking-generic-switch [OpenStackSDK] python-openstackclient [swift] swift Those should be released ASAP, and in all cases before September 15, so that we have a release to include in the final 2023.2 Bobcat release. Thanks in advance, -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/

1 0

[horizon] Cancelling Today's Weekly meeting
by vishal manchanda 30 Aug '23

30 Aug '23

Hello Team, As it is a holiday for me, I will not be able to host today's horizon weekly meeting. So let's cancel today's weekly meeting. If anything urgent, please reach out to the horizon core team. Thanks & regards, Vishal Manchanda

1 0

[election][ptl][openstack-helm] PTL Candidacy for 2024.1
by Vladimir Kozhukalov 29 Aug '23

29 Aug '23

Hello Everyone, I am pleased to announce my candidacy to continue as the PTL for OpenStack-Helm in the upcoming 2024.1 release cycle. ## Key accomplishments we have collectively achieved during the Bobcat cycle ### New Features: - Integrated a Manila chart, expanding the range of out-of-the-box use-cases. - Introduced OVN chart and support in Neutron, as OVN is emerging as the de facto standard for OpenStack networking. - Launched a Tacker chart to engage telecom sectors as potential users and contributors. - Upgraded the Elasticsearch chart to version 8.9.0. - Initiated the transition to Rook operator for Ceph management, with completion expected by cycle end. ### Pipeline and Compatibility: - Added Compute-kit and Cinder jobs to the check pipeline for Yoga, Zed, and Antelope releases, thereby extending the range of officially supported Openstack releases. ### Platform Support: - We stay Ubuntu-centric due to lack of resources for support of multiple platforms. During this cycle we introduced Ubuntu Jammy images, and initiated extensive testing for Jammy-based deployments. ### Other Improvements: - Enhanced Compute-kit job stability by adopting a multinode test environment for specific releases. - Cleaned up the OpenStack-Helm storyboard. Over 100 stories were reviewed, verified and fixed or closed. as outdated. - Numerous minor fixes and adjustments, including updates for recent Kubernetes releases. A big thank you to everyone for your contributions. ## Looking Ahead ### Documentation: The documentation still needs to be improved and aligned with all the recent changes. Users need clear understanding of how to deploy/upgrade Openstack components and which releases and scenarios are thoroghly tested and expected to work. ### Compatibility: We plan to continue expanding support for recent OpenStack and Kubernetes releases and additional Linux distributions. I am eager to continue in my role as PTL to ensure these initiatives move forward. Thanks. -- Best regards, Kozhukalov Vladimir

1 0

Nominations closing soon
by Amy Marrich 29 Aug '23

29 Aug '23

Hey all, There is less than 24 hours left for nominations to be committed to the election repos for PTL and TC. If you are a diverse member of the community and see a role you would like to try, especially if you’re a member of a leaderless project, step up and put your name in for a leadership role. There are plenty of us in the community who are willing to help mentor you in these roles. Amy(spotz)

1 0

[election][ptl][tacker] PTL Candidacy for 2024.1
by Yasufumi Ogawa 29 Aug '23

29 Aug '23

Hi, I'd like to nominate myself for Tacker PTL for 2024.1 cycle. I've contributed to Tacker as a PTL to provide a generic VNF Manager to industiries. We have implemented or improved several features such as monitoring with Prometheus for containers or policy management for multi domain usecases in addition to introduce the latest ETSI-NVF standard compliant features. We also revised unstable unit/functional tests for which we have been suffered several times for unexpected failures and waste of time as one of the our goals for efficiency of our development, and documents for users to demonstrate usecases in practical examples. In the next cycle, I'd like to continue to improving Tacker for the 5G network includes both a radio-access network and a core network to provide operators as we've focused on. Deploying Tacker as Service Management and Orchestration (SMO) in O-RAN architecture is one of our plans for realizing the scenario. Tacker team has already contributed to ETSI NFV and O-RAN SC communities while getting feedbacks from them and enforced relationship among them gradually. I'd appreciate if we make some more progress for OpenStack and these communities. Regards, Yasufumi (yasufum)

1 0

[scientific-sig] Fwd: Attract Top Talent at UC Berkeley’s Annual World of Data Career Fair for OpenInfra Foundation
by T. Nichole Williams 29 Aug '23

29 Aug '23

Greetings OpenStackers! If you or your organization is on the hunt for new talent, University of California Berkeley is holding an event called World of Data that may be of interest. If you’d like more info, please contact Sarah Son (sarahson(a)berkeley.edu <mailto:sarahson@berkeley.edu>) for more info. Have a lovely week everyone! Cheers, trilliams > Begin forwarded message: > > From: Sarah Son <sarahson(a)berkeley.edu> > Subject: Attract Top Talent at UC Berkeley’s Annual World of Data Career Fair for OpenInfra Foundation > Date: August 3, 2023 at 11:38:56 AM EDT > To: Treva Williams <tribecca(a)tribecc.us> > > Hi Treva! > > My name is Sarah and I am the External Vice President at UC Berkeley’s Student Association for Applied Statistics (SAAS). Nice to meet you! > > I’m reaching out to share some details about World of Data, an annual in-person Data Science Career Fair at UC Berkeley hosted by SAAS <https://saas.berkeley.edu/>, DSS <https://dssberkeley.com/>, and BDAB <https://bd.berkeley.edu/>, Berkeley’s leading student-run organizations for projects and education in data science. We are selective organizations that have helped several hundred Berkeley students cultivate their knowledge in data-related career fields and land positions at outstanding companies. Last September, the first-ever World of Data Career Fair reached an audience of over 400+ with companies such as Micron Technology and Accenture in attendance. We are looking forward to hosting the event again this fall at an even greater scale! > > This year, we would love to have OpenInfra Foundation participate in World of Data on September 20th, 2023. > > Our goal, as always, is to show our talented student body how data science can be applied to solve real-world problems in a diverse range of industries. > > We would love to collaborate with OpenInfra Foundation and learn how you leverage data science techniques to advance your company’s mission. The event will revolve around networking and connecting between students and recruiters at individual company booths, with a focus on data science/machine learning internships and full-time opportunities for summer 2024. > > In the past, our organizations have hosted events such as career fairs with 500+ attendees, annual Data Science Forums with 800+ attendees, annual Datathons for Social Good, and much more. Additionally, with university departmental support and our organization’s existing networks, we once again expect strong participation from promising students in the Statistics, Computer Science, and Data Science communities. > > I’ve attached a brochure below with more details and I'm happy to answer any questions you might have via email or over a call. Looking forward to your response! > > Warm regards, > Sarah > > – > Sarah Son > External Vice President | SAAS > UC Berkeley | Data Science > > >

1 0

[elections][manila] PTL Candidacy for 2024.1
by Carlos Silva 29 Aug '23

29 Aug '23

Hello everyone! I would like to announce my candidacy for the role of Manila PTL during the Caracal/2024.1 cycle. I have been in this role for the last 3 cycles and I would like to continue serving as PTL. During the past cycles, there was a lot of effort on promoting contributor collaboration through different activities and it helped us to make progress on closing gaps between the Manila APIs and other clients consuming them, as well as tackling tech debt and getting new features and drivers merged. I feel energized to continue pushing on these areas, but also put some focus on: * Working on the stability of our upstream CI first party jobs and also checking with third party CI maintainers to ensure their CI is healthy and reporting valid results. * Enhancing the review bandwidth and doing more actions to encourage more reviews from people submitting code to the Manila code base. * Work on documentation to help new contributors understand what are the requirements when proposing new features, and also continue the effort to cover the lack of documentation we have on third party CI setups for Manila, helping potential new vendors to quickly set up their CI systems. Thank you for your consideration! Carlos da Silva IRC: carloss

1 0

Query on Enabling Horizon Support for Creating SR-IOV VMs
by Aditya Sathish 29 Aug '23

29 Aug '23

Hello, We have enabled SR-IOV external interfaces on our server cluster running OpenStack Yoga. Unfortunately, there is a known limitation on the Horizon dashboard where we cannot create virtual machines through the dashboard - forcing us to rely on the OpenStack CLI to create the VMs. I wanted to understand what precisely the limitation was to support this from Horizon and if there were any cherry-picks I could pull from the later releases that would allow me to provide this support through the dashboard (or upgrade my release to 2023.1). I did see that we can now select the VNIC_Type from the dashboard, but that doesn't resolve the issue of VM creation with SR-IOV ports. Any points would be greatly appreciated. Regards, Aditya

1 0

[election][cloudkitty] PTL candidacy for Caracal
by Rafael Weingärtner 29 Aug '23

29 Aug '23

Hi all, I would like to self-nominate for the role of PTL of CloudKitty for the Caracal release cycle. I have been PTL since the Victoria cycle and I am willing to continue in this role. I will keep working with existing contributors to maintain CloudKitty for their use cases and encourage the addition of new functionalities. I have opened the patch with my candidacy: https://review.opendev.org/c/openstack/election/+/893061 -- Rafael Weingärtner

1 0

[kolla] using local package mirror for image builds
by Jonathan Proulx 29 Aug '23

29 Aug '23

HI All, Using `kolla-build -b ubuntu` to build container images keeps timing out (well hanging for literal days) seemingly on package downloads from remote mirrors. I'm new to kolla and kolla-ansible so I may be misdiagnosing the cause of my current issue but we de have local ubuntu package mirros which would be faster in any case. Is there a way to specify using only a local package mirror? Thanks, -Jon

2 2

[tc] Technical Committee next weekly meeting today, August 29, 2023
by Nikolla, Kristi 29 Aug '23

29 Aug '23

Hi all, This is a reminder that the next weekly Technical Committee meeting is to be held today, Tuesday, 29 August 2023 at 1800 UTC on #openstack-tc on OFTC IRC. The agenda can be found at https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting and attached below: • Roll call • Follow up on past action items • No action items • Gate health check • 2022 user survey analysis • #link https://review.opendev.org/c/openstack/governance/+/892670/ • Documenting implementation processes for Unmaintained • With the resolution being merged, the next step is defining and documenting the processes and implementing the required tooling. • Open Discussion and Reviews • Register for the PTG • #link https://openinfra.dev/ptg/ • #link https://review.opendev.org/q/projects:openstack/governance+is:open Thank you, Kristi Nikolla

1 0