June 2023 - openstack-discuss - lists.openstack.org

Fwd: DPDK+OVS with OpenStack
by Mark Wittling 23 Apr '25

23 Apr '25

Looking for someone who knows OpenStack with OpenVSwitch, and in addition to that, DPDK with OpenStack and OVS.I am using OpenStack Queens, with OpenVSwitch. The architecture I am using is documented here: https://docs.openstack.org/neutron/queens/admin/deploy-ovs-provider.htmlThe OVS I am using on the Compute Node, is compiled with DPDK, and I have enabled the datapath to netdev (DPDK) on br-prv (provider network bridge), and br-tun (tunneling bridge). But these two bridges, br-tun and br-prv, are patched into another OpenStack bridge, called br-int. I wasn’t actually sure about whether to tinker with this bridge, and wondered what datapath it was using.Then, I realized there is a parameter in the openvswitch_agent.ini file, which I will list here: # OVS datapath to use. 'system' is the default value and corresponds to the # kernel datapath. To enable the userspace datapath set this value to 'netdev'. # (string value) # Possible values: # system - <No description provided> # netdev - <No description provided> #datapath_type = system datapath_type = netdev So in tinkering with this, what I realized, is that when you set this datapath_type to system or netdev, it will adjust the br-int bridge to that datapath type.So here is my question. How can I launch a non-DPDK VM, if all of the bridges are using the netdev datapath type?Here is another question. What if one of the flavors don’t have the largepages property set on them? I assumed OpenStack would revert to a system datapath and not use DPDK for those VM interfaces. Well, I found out in testing, that is not the case. If you set all your bridges up for netdev, and you don’t set the property on the Flavor of the VM (largepages), the VM will launch, but it simply won’t work.Is there no way, on a specific Compute Host, to support both DPDK (netdev datapaths) and non-DPDK (system datapaths)?Either on a VM interface level (VM has one interface that is netdev DPDK and another that is system datapath non-DPDK)?Or on a VM by VM basis (VM 1 has 1 or more netdev datapath interfaces and VM 2 has 1 or more system datapath interfaces)?Am I right here? Once you set up a Compute Host for DPDK, it’s DPDK or nothing on that Compute Host? (edited)

3 2

How to config DPDK compute in OVN
by dangerzone ar 22 Apr '25

22 Apr '25

Hi....as far I understand ovs does support dpdk compute to be deployed. I'm referring to the guideline ' *https://docs.openstack.org/networking-ovn/queens/admin/dpdk.html <https://docs.openstack.org/networking-ovn/queens/admin/dpdk.html>* ' ... It's too brief.. May I know how to config dpdk compute in OVN networking openstack? Hope someone could help what is the config to be done.... Currently compute is run without dpdk over openstack train ovn based. Please need some info on how I can proceed. Thank you

2 1

ERROR openstack [-] Resource OS::TripleO::Network::Ports::ControlPlaneVipPort maps to type OS::Neutron::Port and the Neutron service is not available when using ephemeral Heat.| Openstack tripleo wallaby version
by Swogat Pradhan 23 Dec '24

23 Dec '24

Hi, I am currently trying to deploy openstack wallaby using tripleo arch. I created the network jinja templates, ran the following commands also: #openstack overcloud network provision --stack overcloud --output networks-deployed-environment.yaml custom_network_data.yaml # openstack overcloud network vip provision --stack overcloud --output vip-deployed-environment.yaml custom_vip_data.yaml # openstack overcloud node provision --stack overcloud --overcloud-ssh-key /home/stack/sshkey/id_rsa overcloud-baremetal-deploy.yaml and used the environment files in the openstack overcloud deploy command: (undercloud) [stack@hkg2director ~]$ cat deploy.sh #!/bin/bash THT=/usr/share/openstack-tripleo-heat-templates/ CNF=/home/stack/ openstack overcloud deploy --templates $THT \ -r $CNF/templates/roles_data.yaml \ -n $CNF/workplace/custom_network_data.yaml \ -e ~/containers-prepare-parameter.yaml \ -e $CNF/templates/node-info.yaml \ -e $CNF/templates/scheduler-hints.yaml \ -e $CNF/workplace/networks-deployed-environment.yaml \ -e $CNF/workplace/vip-deployed-environment.yaml \ -e $CNF/workplace/overcloud-baremetal-deployed.yaml \ -e $CNF/workplace/custom-net-bond-with-vlans.yaml Now when i run the ./deploy.sh script i encounter an error stating: ERROR openstack [-] Resource OS::TripleO::Network::Ports::ControlPlaneVipPort maps to type OS::Neutron::Port and the Neutron service is not available when using ephemeral Heat. The generated environments from 'openstack overcloud baremetal provision' and 'openstack overcloud network provision' must be included with the deployment command.: tripleoclient.exceptions.InvalidConfiguration: Resource OS::TripleO::Network::Ports::ControlPlaneVipPort maps to type OS::Neutron::Port and the Neutron service is not available when using ephemeral Heat. The generated environments from 'openstack overcloud baremetal provision' and 'openstack overcloud network provision' must be included with the deployment command. 2022-04-19 13:47:16.582 735924 INFO osc_lib.shell [-] END return value: 1 Can someone tell me where the mistake is? With regards, Swogat Pradhan

4 4

[kolla] [openeuler] CI jobs status
by Michał Nasiadka 26 Nov '24

26 Nov '24

Hello Koalas, OpenEuler jobs in master (to be Antelope) are failing for a long time. If no one is interested in maintaining OpenEuler support in Kolla-Ansible - I will propose dropping it’s support on Wednesday Kolla meeting. Best regards, Michal

4 3

Kolla Ansible on Ubuntu 20.04 - cloud-init & other network issues
by Tobias McNulty 10 Oct '24

10 Oct '24

Hi, I'm attempting to use Kolla Ansible 14.6.0 to deploy OpenStack Yoga on a small 3-node Ubuntu 20.04 cluster. The nodes have 128 GB RAM each, dual Xeon processors, and dual 10G Intel NICs. The NICs are connected to access ports on a 10G switch with separate VLANs for the local and external networks. All the playbooks run cleanly, but cloud-init is failing in the Ubuntu 20.04 and 22.04 VMs I attempt to boot. The VM images are unmodified from https://cloud-images.ubuntu.com/, and cloud-init works fine if I mount a second volume with user-data. The error is a timeout attempting to reach 169.254.169.254. This occurs both when booting a VM in an internal routed network and directly in an external network. I tried various neutron plugin agents (ovn, linuxbridge, and openvswitch both with and without firewall_driver = openvswitch <https://docs.openstack.org/kolla-ansible/latest/reference/networking/neutro…>) first with a clean install of the entire OS each time, all with the same result. Running tcpdump looking for 169.254.169.254 shows nothing. As a possible clue, the virtual NICs are unable to pass any traffic (e.g., to reach an external DHCP server) unless I completely disable port security on the interface (even if the associated security group is wide open). But disabling port security does not fix cloud-init (not to mention I don't really want to disable port security). Are there any additional requirements related to deploying OpenStack with Kolla on Ubuntu 20.04? This is a fairly vanilla configuration using the multinode inventory as a starting point. I tried to follow the Quick Start <https://docs.openstack.org/kolla-ansible/yoga/user/quickstart.html> as closely as possible; the only material difference I see is that I'm using the same 3 nodes for control + compute. I am using MAAS so it's easy to get a clean OS install on all three nodes ahead of each attempt. I plan to try again with the standard (non-HWE) kernel just in case, but otherwise I am running out of ideas. In case of any additional clues, here are my globals.yml and inventory file, along with the playbook I'm using to configure the network, images, VMs, etc., after bootstrapping the cluster: https://gist.github.com/tobiasmcnulty/7dbbdbc67abc08cbb013bf5983852ed6 Thank you in advance for any advice! Cheers, Tobias

5 9

Live migration fails
by Szabo, Istvan (Agoda) 06 Aug '24

06 Aug '24

Hi, I have couple of compute nodes where the live migration fails with existing vms. When I quickly spawn a vm and try live migration it works so I assume shouldn't be a big problem with the compute node. However I have many existing vms where it fails with a servername not found. /var/log/nova/nova-conductor.log:2021-04-21 14:47:12.605 227612 ERROR nova.conductor.tasks.migrate [req-f4067a26-a233-4673-8c07-9a8a290980b0 dce35e6eceea4312bb0baa0510cef363 ca7e35079f4440c78bd9870724b9638b - default default] [instance: 1517a2ac-3b51-4d8d-80b3-89a5614d1ae0] Unable to find record for source node servername on servername: ComputeHostNotFound: Compute host servername could not be found. /var/log/nova/nova-conductor.log:2021-04-21 14:47:12.605 227612 WARNING nova.scheduler.utils [req-f4067a26-a233-4673-8c07-9a8a290980b0 dce35e6eceea4312bb0baa0510cef363 ca7e35079f4440c78bd9870724b9638b - default default] Failed to compute_task_migrate_server: Compute host servername could not be found.: ComputeHostNotFound: Compute host servername could not be found. /var/log/nova/nova-conductor.log:2021-04-21 14:47:12.605 227612 WARNING nova.scheduler.utils [req-f4067a26-a233-4673-8c07-9a8a290980b0 dce35e6eceea4312bb0baa0510cef363 ca7e35079f4440c78bd9870724b9638b - default default] [instance: 1517a2ac-3b51-4d8d-80b3-89a5614d1ae0] Setting instance to ACTIVE state.: ComputeHostNotFound: Compute host servername could not be found. /var/log/nova/nova-conductor.log:2021-04-21 14:47:12.672 227612 ERROR oslo_messaging.rpc.server [req-f4067a26-a233-4673-8c07-9a8a290980b0 dce35e6eceea4312bb0baa0510cef363 ca7e35079f4440c78bd9870724b9638b - default default] Exception during message handling: ComputeHostNotFound: Compute host am-osfecn-4025 Tried with this command: nova live-migration --block-migrate id. Any idea? Thank you. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

5 16

[ptl][tc] OpenStack packages PyPi additional external maintainers audit & cleanup
by Ghanshyam Mann 16 May '24

16 May '24

Hi PTLs, As you might know or have seen for your project package on PyPi, OpenStack deliverables on PyPi have additional maintainers, For example, https://pypi.org/project/murano/, https://pypi.org/project/glance/ We should keep only 'openstackci' as a maintainer in PyPi so that releases of OpenStack deliverables can be managed in a single place. Otherwise, we might face the two sets of maintainers' places and packages might get released in PyPi by additional maintainers without the OpenStack project team knowing about it. One such case is in Horizon repo 'xstatic-font-awesome' where a new maintainer is added by an existing additional maintainer and this package was released without the Horizon team knowing about the changes and release. - https://github.com/openstack/xstatic-font-awesome/pull/2 To avoid the 'xstatic-font-awesome' case for other packages, TC discussed it in their weekly meetings[1] and agreed to audit all the OpenStack packages and then clean up the additional maintainers in PyPi (keep only 'openstackci' as maintainers). To help in this task, TC requests project PTL to perform the audit for their project's repo and add comments in the below etherpad. - https://etherpad.opendev.org/p/openstack-pypi-maintainers-cleanup Thanks to knikolla to automate the listing of the OpenStack packages with additional maintainers in PyPi which you can find the result in output.txt at the bottom of this link. I have added the project list of who needs to check their repo in etherpad. - https://gist.github.com/knikolla/7303a65a5ddaa2be553fc6e54619a7a1 Please complete the audit for your project before March 15 so that TC can discuss the next step in vPTG. [1] https://meetings.opendev.org/meetings/tc/2023/tc.2023-01-11-16.00.log.html#… -gmann

18 27

[Watcher] How use Watcher to scheduler Openstack resource without cpu_usage
by Nguyễn Hữu Khôi 06 Mar '24

06 Mar '24

Hello guys. I want to use watcher to scheduler openstack resources but some openstack versions are missing cpu_usage metric. How do we use Watcher without it? Thank you. Nguyen Huu Khoi

3 5

[designate] Proposal to deprecate the agent framework and agent based backends
by Michael Johnson 09 Jan '24

09 Jan '24

TLDR: The Designate team would like to deprecate the backend agent framework and the agent based backends due to lack of development and design issues with the current implementation. The following backends would be deprecated: Bind9 (Agent), Denominator, Microsoft DNS (Agent), Djbdns (Agent), Gdnsd (Agent), and Knot2 (Agent). Designate includes many backend DNS server drivers[1], many of which are "native" (also known as xfr type backends) backend implementations. In addition to the "native" backends, Designate has an agent backend[2] that supports other backends via an agent process. To quote the agent backend documentation[2]: This backend uses an extension[3] of the DNS protocol itself to send management requests to the remote agent processes, where the requests will be actioned. The rpc traffic between designate and the agent is both unauthenticated and unencrypted. Do not run this traffic over unsecured networks. Here are the reasons we are proposing to deprecate the agent framework now: 1. The agent protocol used by Designate is using an "unassigned"[4][5] DNS opcode (14) that is causing problems with the dnspython library >= 2.3.0 which is now validating the opcode when building DNS messages. It is a bad practice to use "unassigned" values as they may be officially assigned at any time, likely with an incompatible message format. 2. The agent backends are not tested in the OpenStack jobs[1]. 3. Many of the agent backends have been marked as "Experimental" since 2016 with no additional contributions beyond general repository code maintenance. 4. The protocol between the Designate worker process and the agent process is unauthenticated and unencrypted. 5. We do not know of a development resource to rewrite the agent framework protocol to address issue #1 and #4 above. 6. The introduction of catalog zones[6] may eliminate the need for some of the agent based backend drivers. By marking the agent framework and agent based backends "deprecated" in the Antelope cycle, we would remove the code no earlier than the "C" release of OpenStack (per the OpenStack deprecation policy[7]). In the meantime, issue #1 has been worked around by overriding the dnspython opcode validation[7] as needed in the Designate code (similar to a monkey patch). This is not a sustainable long term solution. The following backend agent based drivers would be marked "deprecated" in addition to the agent framework itself: Bind9 (Agent) Denominator Microsoft DNS (Agent) Djbdns (Agent) Gdnsd (Agent) Knot2 (Agent) We plan to propose patches for the deprecation over the next week, but will not merge them until at least January 24th to allow time for comment from the community. If you have concerns about this deprecation plan or are interested in rewriting the agent framework protocol to address the above issues, please reply to this announcement. Michael [1] https://docs.openstack.org/designate/latest/admin/support-matrix.html [2] https://docs.openstack.org/designate/latest/admin/backends/agent.html [3] https://github.com/openstack/designate/blob/master/designate/backend/privat… [4] https://www.rfc-editor.org/rfc/rfc6895.html#section-2.2 [5] https://www.iana.org/assignments/dns-parameters/dns-parameters.xhtml#dns-pa… [6] https://www.ietf.org/archive/id/draft-ietf-dnsop-dns-catalog-zones-08.txt [7] https://docs.openstack.org/project-team-guide/deprecation.html [8] https://review.opendev.org/c/openstack/designate/+/870678

3 13

[OPENSTACK][rabbitmq] using quorum queues
by Nguyễn Hữu Khôi 17 Oct '23

17 Oct '23

Hello guys. IS there any guy who uses the quorum queue for openstack? Could you give some feedback to compare with classic queue? Thank you. Nguyen Huu Khoi

9 19

[ptl][tc][winstackers] Final call for Winstackers PTL and maintainers
by James Page 11 Sep '23

11 Sep '23

Hi All As announced by Lucian last November (see [0]) Cloudbase Solutions are no longer in a position to maintain support for running OpenStack on Windows and have also ceased operation of their 3rd party CI for the windows support across a number of OpenStack projects. This situation has resulted in the Winstackers project becoming PTL-less for the 2023.2 cycle with no volunteers responding to the TC's call to fill this role and take this feature in OpenStack forward (see [1]). This is the final call for any maintainers to step forward if this feature is important to them in OpenStack. The last user survey in 2022 indicated that 2% of respondents were running on Hyper-V so this might be important enough to warrant a commitment from someone operating OpenStack on Windows to maintain these features going forward. Here is a reminder from Lucian's original email on the full list of projects which are impacted in some way: * nova hyper-v driver - in-tree plus out-of-tree compute-hyperv driver * os-win - common Windows library for Openstack * neutron hyperv ml2 plugin and agent * ovs on Windows and neutron ovs agent support * cinder drivers - SMB and Windows iSCSI * os-brick Windows connectors - iSCSI, FC, SMB, RBD * ceilometer Windows poller * manila Windows driver * glance Windows support * freerdp gateway The lack of 3rd party CI for testing all of this really needs to be addressed as well. If no maintainers are forthcoming between now and the next PTG in June the TC will need to officially retire the project and start the process of removing support for Windows across the various projects that support this operating system in some way - either directly or through the use of os-win. For clarity this call refers to the use of the Hyper-V virtualisation driver and associated Windows server components to provide WIndows based OpenStack Hypervisors and does not relate to the ability to run Windows images as guests on OpenStack. Regards James [0] https://lists.openstack.org/pipermail/openstack-discuss/2022-November/03104… [1] https://lists.openstack.org/pipermail/openstack-discuss/2023-March/032888.h…

5 9

[openstack-ansible] Recommended branch for a production environment?
by Roger Rivera 05 Sep '23

05 Sep '23

Hello, We have deployed Openstack-Ansible in a test environment and we were wondering what the recommended repository branch is to deploy in a production environment that will be integrated with a separate Ceph cluster. 1. We are thinking about pinning to the stable/zed. Is that recommended over the master branch? 2. Are bugs ironed out on both master and stable/zed with the same cadence? 3. Additionally, is Debian 11 a better alternative than Ubuntu 22 for target hosts? We noticed Ubuntu 22.04 support was added recently, whereas Debian 11 has been supported for quite some time now which leads us to believe its stability/integration could be more mature at this point? Any suggestions will be appreciated. Thank you, -- *Roger *

3 7

[puppet] Proposing core team clean up
by Takashi Kajinami 23 Aug '23

23 Aug '23

Hello, I went through the current core reviewers and found out that the current list includes some people no longer working in the OpenStack community. I'd like to propose removing the following persons from the core reviewer team as we haven't seen these people making any activity in the community recently. * Pradeep Kilambi - puppet-aodh, puppet-ceilometer, puppet-gnocchi, puppet-panko * Richard Megginson - puppet-keystone * Denis Egorenko - puppet-murano * Serg Melikyan - puppet-murano * Sergey Kolekonov - puppet-neutron * David Moreau Simard - puppet-openstack-integration * Yu xingchao - all modules Also, I confirmed that Wes Hayutin is no longer working for OpenStack and he agreed to retire from the core reviewer position. Thus I'll remove him as well. I'll be open for any feedback for one week to hear any thoughts/feedback about this removal. I'd like to take this moment to thank all people for their past work. We always welcome them back if they resume their work in this project. Thank you, Takashi KAJINAMI

3 4

[cloudkitty] question to the scope_key in cloudkitty
by Noah Elias Feldt 28 Jul '23

28 Jul '23

Hello! I am currently trying to get CloudKitty to work with the Prometheus Collector and PyScripts. I noticed that the scope_key for Prometheus can only be changed once in the global configuration file. This was a problem for me because we need different scope_keys for different metrics. Many of the metrics still need the scope key "tenant_id" and some others "project_id". I have found in the documentation of CloudKitty no way to change this for each metric, so I have internally changed something in the source code and added the function to set the scope_key for each metric individually. I just wanted to ask if you would be interested if I submit the small change upstream? Thank you. Regards Noah Feldt

3 3

[cinder] cinder-backup volume stuck in creating
by Satish Patel 27 Jul '23

27 Jul '23

Folks, I have configured nova and cinder with ceph storage. VMs running on ceph storage but now when i am trying to create a backup of cinder volume its getting stuck on creating and doing nothing. Logs also do not give any indication of bad. My cinder.conf [DEFAULT] enabled_backends = rbd-1 backup_driver = cinder.backup.drivers.ceph.CephBackupDriver backup_ceph_conf = /etc/ceph/ceph.conf backup_ceph_user = cinder-backup backup_ceph_chunk_size = 134217728 backup_ceph_pool = backups backup_ceph_stripe_unit = 0 backup_ceph_stripe_count = 0 restore_discard_excess_bytes = true osapi_volume_listen = 10.73.0.181 osapi_volume_listen_port = 8776 Output of "openstack volume service list" showing cinder-backup service is up but when i create a backup it's getting stuck in this stage and no activity. I am not seeing anything getting transferred to the ceph backups pool also. Any clue? or method to debug? # openstack volume backup list --all +--------------------------------------+------+-------------+----------+------+ | ID | Name | Description | Status | Size | +--------------------------------------+------+-------------+----------+------+ | bc844d55-8c5a-4bd3-b0e9-7c4c780c95ad | foo1 | | creating | 20 | +--------------------------------------+------+-------------+----------+------+

5 7

Re: [all] [oslo.messaging] Interest in collaboration on a NATS driver
by Tobias Urdin 22 Jul '23

22 Jul '23

Hello Kai, Great to hear that you’re interested! I’m currently just testing this as a POC directly in devstack so there is real-world testing (not should it yet I think, needs more development first). Best regards Tobias > On 30 Aug 2022, at 18:32, Kai Bojens <kb(a)kbojens.de> wrote: > > Am 29.08.22 um 15:46 schrieb Tobias Urdin: > >> If anybody, or any company, out there would be interested in collaborating in a project to bring this support and maintain it feel free to >> reach out. I’m hoping somebody will bite but atleast I’ve put it out there for all of you. > > Hi, > I am very much interested to take a closer look at your work and maybe contribute to it. Although I'm working with OpenStack for my employer at the moment, I'd do this in my spare time as I'm not sure that I can convince him to add another staging system with a full OpenStack installation just for developing a RabbitMQ replacement. That's not on our agenda as we are mostly users and not developers of OpenStack. > > As I'm pretty new to the messaging topic and had just heard of NATS and your idea, I'll first would have to dive into NATS and then I would take a closer look at your code and maybe try to create some documentation or tests. So, I can't promise anything but as I said I'm very interested in your approach as I also see the massive load from RabbitMQ. > > This brings me to the most important question: Where do you run and test your code? > > Greetings, > Kai

8 30

SNAT failure with OVN under Antelope
by Gary Molenkamp 13 Jul '23

13 Jul '23

Good morning, I'm having a problem with snat routing under OVN but I'm not sure if something is mis-configured or just my understanding of how OVN is architected is wrong. I've built a Zed cloud, since upgraded to Antelope, using the Neutron Manual install method here: https://docs.openstack.org/neutron/latest/install/ovn/manual_install.html I'm using a multi-tenent configuration using geneve and the flat provider network is present on each hypervisor. Each hypervisor is connected to the physical provider network, along with the tenent network and is tagged as an external chassis under OVN. br-int exists, as does br-provider ovs-vsctl set open . external-ids:ovn-cms-options=enable-chassis-as-gw For most cases, distributed FIP based connectivity is working without issue, but I'm having an issue where VMs without a FIP are not always able to use the SNAT services of the tenent network router. Scenario: Internal network named cs3319: with subnet 172.31.100.0/23 Has a router named cs3319_router with external gateway set (snat enabled) This network has 3 vms: - #1 has a FIP and can be accessed externally - #2 has no FIP, can be accessed via VM1 and can access external resources via SNAT (ie OS repos, DNS, etc) - #3 has no FIP, can be accessed via VM1 but has no external SNAT connectivity From what I can tell, the chassis config is correct, compute05 is the hypervisor and the faulty VM has a port binding on this hypervisor: ovn-sbctl show ... Chassis "8e0fa17c-e480-4b60-9015-bd8833412561" hostname: compute05.cloud.sci.uwo.ca Encap geneve ip: "192.168.0.105" options: {csum="true"} Port_Binding "7a5257eb-caea-45bf-b48c-620c5dff4b39" Port_Binding "50e16602-78e6-429b-8c2f-e7e838ece1b4" Port_Binding "f121c9f4-c3fe-4ea9-b754-a809be95a3fd" The router has the candidate gateways, and the snat set: ovn-nbctl show 92df19a7-4ebe-43ea-b233-f4e9f5a46e7c router 92df19a7-4ebe-43ea-b233-f4e9f5a46e7c (neutron-389439b5-07f8-44b6-a35b-c76651b48be5) (aka cs3319_public_router) port lrp-44ae1753-845e-4822-9e3d-a41e0469e257 mac: "fa:16:3e:9a:db:d8" networks: ["129.100.21.94/22"] gateway chassis: [5c039d38-70b2-4ee6-9df1-596f82c68106 99facd23-ad17-4b68-a8c2-1ff6da15ac5f 1694116c-6d30-4c31-b5ea-0f411878316e 2a4bbaf9-228a-462e-8970-0cdbf59086e6 9332c61b-93e1-4a70-9547-701a014bfd98] port lrp-509bba37-fa06-42d6-9210-2342045490db mac: "fa:16:3e:ff:0f:3b" networks: ["172.31.100.1/23"] nat 11e0565a-4695-4f67-b4ee-101f1b1b9a4f external ip: "129.100.21.94" logical ip: "172.31.100.0/23" type: "snat" nat 21e4be02-d81c-46e8-8fa8-3f94edb4aed1 external ip: "129.100.21.87" logical ip: "172.31.100.49" type: "dnat_and_snat" Each network agent on the hypervisors shows the ovn controller up : OVN Controller Gateway agent | compute05.cloud.sci.uwo.ca | | :-) | UP | ovn-controller The ovs vswitch on the hypervisor looks correct afaict and ovn ports bfd status are all forwarding to other hypervisors. ie: Port ovn-2a4bba-0 Interface ovn-2a4bba-0 type: geneve options: {csum="true", key=flow, remote_ip="192.168.0.106"} bfd_status: {diagnostic="No Diagnostic", flap_count="1", forwarding="true", remote_diagnostic="No Diagnostic", remote_state=up, state=up} Any advice on where to look would be appreciated. PS. Version info: Neutron 22.0.0-1 OVN 22.12 neutron options: enable_distributed_floating_ip = true ovn_l3_scheduler = leastloaded Thanks Gary -- Gary Molenkamp Science Technology Services Systems/Cloud Administrator University of Western Ontario molenkam(a)uwo.ca http://sts.sci.uwo.ca (519) 661-2111 x86882 (519) 661-3566

5 14

[cinder]cinder qos question.
by Nguyễn Hữu Khôi 13 Jul '23

13 Jul '23

Hello guys. I am trying to use cinder qos. But i have question. - I added qos after a volume was create and I see that qos did not apply to this volume. - i create new volume with assosiate qos then it is ok but when i changed extra specs in qos, volume keep old specs. Is there any way to make qos apply with two above cases. Many thanks.

2 5

[cinder-backup][kolla] mix cinder-volume and cinder-backup types (nfs, ceph)
by garcetto 13 Jul '23

13 Jul '23

good evening, it is possible to mix cinder-volume backend (say ceph) and cinder-backup backend (say nfs)? and also do incrementals backup? thank you

3 3

instance console something went wrong, connection is closed | Wallaby DCN
by Swogat Pradhan 12 Jul '23

12 Jul '23

Hi, I am creating instances in my DCN site and i am unable to get the console sometimes, error: something went wrong, connection is closed I have 3 instances now running on my hci02 node and there is console access on 1 of the vm's and the rest two i am not getting the console, i have used the same flavor, same image same security group for the VM's. Please suggest what can be done. With regards, Swogat Pradhan

3 13

[ptl] Need PTL volunteer for OpenStack Vitrage
by Ghanshyam Mann 11 Jul '23

11 Jul '23

Hi Eyal, I am reaching out to you as you were PTL for OpenStack Vitrage project in the last cycle. There is no PTL candidate for the next cycle (2023.2), and it is on the leaderless project list. Please check if you or anyone you know would like to lead this project. - https://etherpad.opendev.org/p/2023.2-leaderless Also, if anyone else would like to help leading this project, this is time to let TC knows. -gmann

3 3

[oslo] HTTP base direct RPC oslo.messaging driver contribution
by 室井雅仁 10 Jul '23

10 Jul '23

Hi oslo team, We'd like to contribute HTTP base direct RPC driver to the oslo.messaging community. We have developed the HTTP base driver internally. We have been using the driver in the production with over 10K hypervisors now. I checked the IRC meeting log of the oslo team[1], but there is no regluar meeting in 2023. Is it okay to submit oslo-spec[2] to propose the driver directly, or is there another good place to discuss the feature before submitting a spec? 1. https://meetings.opendev.org/#Oslo_Team_Meeting 2. https://opendev.org/openstack/oslo-specs best regards, Masahito

8 13

[ironic] Capping off storyboard migration
by Jay Faulkner 03 Jul '23

03 Jul '23

Hey all, I've written a small tool[1] which should (I think; I can't test it all the way until I'm ready to actually change a thing) iterate through stories in a storyboard project, and close any open tasks as invalid with a message. I'd like to point this script at all the Ironic-related Storyboard projects, closing all the issues with a message like the following: "Hello Ironic contributor, thank you for filing this bug! We have migrated to the Launchpad bugtracker, located at https://bugs.launchpad.net/ironic, If this bug remains valid, please open an issue in launchpad with the URL to this story for context." Are folks onboard for having all the tasks closed as invalid in these Ironic-related projects? Any feedback on the messaging? Thanks, Jay Faulkner P.S. For other projects who might want to use the tool as well, wait until I use it for Ironic to help me smooth out any rough edges and then have at it! 1: https://github.com/jayofdoom/sb-issue-closer

5 9

[Neutron] Tap as a service
by altidor JB 03 Jul '23

03 Jul '23

Hello, Can anyone help me with the setup of Tap as a Service on Openstack with Juju and Maas? I've seen tutorials for installing it on DevStack but nothing for large scale setups. So far I've tried doing a similar process to this: https://zhuanlan.zhihu.com/p/101599786 basically install the apt on the neutron_api lxd and on the compute nodes, with configurations that are done in the above tutorial. Sounds simple enough. But the tap_services and tap_flows stay in Down state. Any ideas??? Also I'm open to another way to do port mirroring if this doesn't work. JB

2 3

[kolla] kolla-toolbox Ubuntu 22 Erlang RMQ dependencies
by Bram Kranendonk 03 Jul '23

03 Jul '23

Hi all, I'm trying to build the kolla-toolbox image for Ubuntu 22 source for 2023.1 using kolla v16.0.0 This is however not possible due to broken packages for Erlang/RabbitMQ (tried without package cache): INFO:kolla.common.utils.kolla-toolbox: rabbitmq-server : Depends: erlang-base (< 1:26.0) but 1:26.0.2-1rmq1ppa1~ubuntu22.04.1 is to be installed or INFO:kolla.common.utils.kolla-toolbox: erlang-base-hipe (< 1:26.0) but it is not installable or INFO:kolla.common.utils.kolla-toolbox: esl-erlang (< 1:26.0) but it is not installable INFO:kolla.common.utils.kolla-toolbox: Depends: erlang-crypto (< 1:26.0) but 1:26.0.2-1rmq1ppa1~ubuntu22.04.1 is to be installed or INFO:kolla.common.utils.kolla-toolbox: esl-erlang (< 1:26.0) but it is not installable INFO:kolla.common.utils.kolla-toolbox: Depends: erlang-eldap (< 1:26.0) but 1:26.0.2-1rmq1ppa1~ubuntu22.04.1 is to be installed or INFO:kolla.common.utils.kolla-toolbox: esl-erlang (< 1:26.0) but it is not installable INFO:kolla.common.utils.kolla-toolbox: Depends: erlang-inets (< 1:26.0) but 1:26.0.2-1rmq1ppa1~ubuntu22.04.1 is to be installed or INFO:kolla.common.utils.kolla-toolbox: esl-erlang (< 1:26.0) but it is not installable INFO:kolla.common.utils.kolla-toolbox: Depends: erlang-mnesia (< 1:26.0) but 1:26.0.2-1rmq1ppa1~ubuntu22.04.1 is to be installed or INFO:kolla.common.utils.kolla-toolbox: esl-erlang (< 1:26.0) but it is not installable INFO:kolla.common.utils.kolla-toolbox: Depends: erlang-os-mon (< 1:26.0) but 1:26.0.2-1rmq1ppa1~ubuntu22.04.1 is to be installed or INFO:kolla.common.utils.kolla-toolbox: esl-erlang (< 1:26.0) but it is not installable INFO:kolla.common.utils.kolla-toolbox: Depends: erlang-parsetools (< 1:26.0) but 1:26.0.2-1rmq1ppa1~ubuntu22.04.1 is to be installed or INFO:kolla.common.utils.kolla-toolbox: esl-erlang (< 1:26.0) but it is not installable INFO:kolla.common.utils.kolla-toolbox: Depends: erlang-public-key (< 1:26.0) but 1:26.0.2-1rmq1ppa1~ubuntu22.04.1 is to be installed or INFO:kolla.common.utils.kolla-toolbox: esl-erlang (< 1:26.0) but it is not installable INFO:kolla.common.utils.kolla-toolbox: Depends: erlang-runtime-tools (< 1:26.0) but 1:26.0.2-1rmq1ppa1~ubuntu22.04.1 is to be installed or INFO:kolla.common.utils.kolla-toolbox: esl-erlang (< 1:26.0) but it is not installable INFO:kolla.common.utils.kolla-toolbox: Depends: erlang-ssl (< 1:26.0) but 1:26.0.2-1rmq1ppa1~ubuntu22.04.1 is to be installed or INFO:kolla.common.utils.kolla-toolbox: esl-erlang (< 1:26.0) but it is not installable INFO:kolla.common.utils.kolla-toolbox: Depends: erlang-syntax-tools (< 1:26.0) but 1:26.0.2-1rmq1ppa1~ubuntu22.04.1 is to be installed or INFO:kolla.common.utils.kolla-toolbox: esl-erlang (< 1:26.0) but it is not installable INFO:kolla.common.utils.kolla-toolbox: Depends: erlang-tools (< 1:26.0) but 1:26.0.2-1rmq1ppa1~ubuntu22.04.1 is to be installed or INFO:kolla.common.utils.kolla-toolbox: esl-erlang (< 1:26.0) but it is not installable INFO:kolla.common.utils.kolla-toolbox: Depends: erlang-xmerl (< 1:26.0) but 1:26.0.2-1rmq1ppa1~ubuntu22.04.1 is to be installed or INFO:kolla.common.utils.kolla-toolbox: esl-erlang (< 1:26.0) but it is not installable INFO:kolla.common.utils.kolla-toolbox:E: Unable to correct problems, you have held broken packages. I saw a recent commit that pins the RabbitMQ version, but this commit was reverted later on. Does anyone has a idea how I can fix this issue? Thanks, . Bram Kranendonk System Engineer <tel:> Oostmaaslaan 71 (15e etage) 3063 AN Rotterdam The Netherlands [team.blue logo]<https://team.blue/?utm_source=signature&utm_medium=email&utm_campaign=team-…>

2 2

[IRONIC] Firewall drivers / implementation
by Karl Kloppenborg 30 Jun '23

30 Jun '23

Hi Team, We have Ironic deployed and configured to deploy baremetal on vlans attached to the neutron routers of a tenancy/project. However, when assigned a floating IP, there’s no firewall and the server is completely exposed. I cannot seem to see any information on Ironic Firewall’s, how are others achieving this? Any suggestions would be greatly appreciated. Thanks, Karl Kloppenborg. Openstack-Helm Team.

3 3

[release] Release countdown for week R-13, July 03-07
by Elõd Illés 30 Jun '23

30 Jun '23

Development Focus ----------------- The Bobcat-2 milestone is next week, on July 6th, 2023! 2023.2 Bobcat-related specs should now be finalized so that teams can move to implementation ASAP. Some teams observe specific deadlines on the second milestone (mostly spec freezes): please refer to https://releases.openstack.org/bobcat/schedule.html for details. General Information ------------------- Libraries need to be released at least once per milestone period. Next week, the release team will propose releases for any library that has not been otherwise released since milestone 1. PTL's and release liaisons, please watch for these and give a +1 to acknowledge them. If there is some reason to hold off on a release, let us know that as well. A +1 would be appreciated, but if we do not hear anything at all by the end of the week, we will assume things are OK to proceed. Remember that non-library deliverables that follow the cycle-with-intermediary release model should have an intermediary release before milestone-2. Those who haven't will be proposed to switch to the cycle-with-rc model, which is more suited to deliverables that are released only once per cycle. Next week is also the deadline to freeze the contents of the final release. All new '2023.2 Bobcat' deliverables need to have a deliverable file in https://opendev.org/openstack/releases/src/branch/master/deliverables and need to have done a release by milestone-2. Changes proposing those deliverables for inclusion in 2023.2 Bobcat have been posted, please update them with an actual release request before the milestone-2 deadline if you plan on including that deliverable in 2023.2 Bobcat, or -1 if you need one more cycle to be ready. Upcoming Deadlines & Dates -------------------------- Bobcat-2 Milestone: July 6th, 2023 Final 2023.2 Bobcat release: October 4th, 2023

1 0

Problem to login to skyline
by Alexander Luedtke 30 Jun '23

30 Jun '23

Hello Everyone, I have made an setup for testing of OpenStack Antelope on cent os. Nearly everything works fine, especially I can login to the Horizon Dashboard without problems. But when I installed skyline (over the tar package on the Openstack page, which start nginx and gunicorn.py it looks good - means the login page shows up and no errors are reported anywhere. Just when I try to login, it doesn't work - just got an Login error on the webpage, which says that the login credentials are wrong ! Funny thing: not one error in all the skyline log, even not in debug mode. The keystone doesn't show an error also. Maybe someone here had a similar problem and give me a hint - I honestly have no idea where else to look. Thanks a lot, Alex

2 1

[neutron] Neutron drivers meeting at 1400UTC
by Rodolfo Alonso Hernandez 30 Jun '23

30 Jun '23

Hello Neutrinos: This is a heads-up for today's meeting at 1400UTC. We have 3 RFE proposals today: https://wiki.openstack.org/wiki/Meetings/NeutronDrivers. See you later!

1 0

Cinder Bug Report 2023-06-28
by Sofia Enriquez 28 Jun '23

28 Jun '23

Hello Argonauts, Cinder Bug Meeting Etherpad <https://etherpad.opendev.org/p/cinder-bug-squad-meeting> Medium - test_rebuild_volume_backed_server failing 100% on ceph job <https://bugs.launchpad.net/cinder/+bug/2025096>. - Status: The "test_rebuild_volume_backed_server" test is skipped for the Ceph job <https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/887003> . Cheers, -- Sofía Enriquez she/her Software Engineer Red Hat PnT <https://www.redhat.com> IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 0

[neutron] Transition networking-odl Ussuri to EOL
by Rodolfo Alonso Hernandez 28 Jun '23

28 Jun '23

Hello Neutrinos: Yesterday we found that we are still executing some periodic jobs for networking-odl in some stable releases. For the newer branches, we have proposed the corresponding patches to remove these zuul jobs [1][2]. The problem is in the Ussuri branch [3], which is completely broken (UTs, pep8, etc). This is why I'm proposing to transition it to EOL. If you have any concerns, suggestions or questions, please let me know (via mail or IRC in the #openstack-neutron channel). Regards. [1]https://review.opendev.org/c/openstack/networking-odl/+/887080 [2]https://review.opendev.org/c/openstack/networking-odl/+/887077 [3]https://review.opendev.org/c/openstack/networking-odl/+/887076

1 0

[largescale-sig] Next meeting: June 28, 8utc
by Thierry Carrez 28 Jun '23

28 Jun '23

Hi everyone, The Large Scale SIG will be meeting for the last time before the Northern hemisphere summer this Wednesday in #openstack-operators on OFTC IRC, at 8UTC, our APAC+EU-friendly time. I will be chairing. You can doublecheck how that UTC time translates locally at: https://www.timeanddate.com/worldclock/fixedtime.html?iso=20230628T08 Feel free to add topics to the agenda: https://etherpad.opendev.org/p/large-scale-sig-meeting Regards, -- Thierry Carrez

1 1

[Neutron Community] Neutron OVN vs OVS
by Karl Kloppenborg 27 Jun '23

27 Jun '23

Hi Neutron community, Firstly, apologies if this has been asked before, a cursory search of the list rendered no information. At a high level, is OVN replacing OVS in terms of the preferred standard neutron deployment? I am seeing more and more about OVN on the mailing list and wondering if I should be investigating it to supersede OVS standalone. Again, apologies if this is vague/asked before. Kind Regards, Karl Kloppenborg. Openstack-Helm Team.

2 2

[kolla] Transitioning stable/wallaby to EOL
by Michał Nasiadka 27 Jun '23

27 Jun '23

Hello Koalas, On a weekly meeting last week - we’ve decided that due to core team size we’re unable to support that many stable branches - and with the recent release of Antelope (2023.1) we are going to transition Wallaby branch to End of Life. That means the branch will get removed and wallaby-eol tag will be available for those that are still using that code for their deployments. Best regards, Michal

1 0

[tc] July 4 TC Meeting cancelled; next meeting July 11
by Jay Faulkner 27 Jun '23

27 Jun '23

Hi all, The next Technical Committee meeting, as currently scheduled, would fall on July 4th, a US holiday. For this reason, we've cancelled the July 4th meeting. The next scheduled meeting of the TC will be July 11th. As always, if you need anything from us in the meantime, we're in #openstack-tc. Thanks, Jay Faulkner Technical Committee Vice-Chair

1 0

[qa] Cancelling office hour July 4th
by Martin Kopec 27 Jun '23

27 Jun '23

Hello everyone, I'll be on PTO starting this Thursday (June 29th) until July 10th. Therefore I'm gonna cancel our next office hour on July 4th. In case of emergencies, contact me via email - I'll be off the IRC. Thanks, -- Martin Kopec Principal Software Quality Engineer Red Hat EMEA IM: kopecmartin

1 0

Cloudkitty support for Opensearch on Yoga
by wodel youchi 27 Jun '23

27 Jun '23

Hi, Reading the The release notes about Cloudkitty I read this : "Support for using Elasticsearch as a storage backend is being deprecated in the Antelope release in favour of OpenSearch. We will try to keep CloudKitty compatible with both solutions. However, we will only test with OpenSearch." What is the state of the support of Opensearch on Cloudkitty? If it is fully supported, does Openstack Yoga offer the same support? Regards.

2 1

[neutron] Bug deputy report - week of Jun 19th
by Slawek Kaplonski 27 Jun '23

27 Jun '23

Hi, I was bug deputy last week and here is summary of new bugs opened in Neutron: ## Critical ## * https://bugs.launchpad.net/neutron/+bug/2024903 - [neutron-lib] FT "test_negative_update_floatingip_port_forwarding" failing randomly very often - already fixed by Rodolfo, * https://bugs.launchpad.net/neutron/+bug/2025126 - Functional MySQL sync test failing with oslo/sqlalchemy master branch - not assigned ## High ## * https://bugs.launchpad.net/neutron/+bug/2024674 - Unit tests fails with oslo_db.exception.DBNonExistentTable: (sqlite3.OperationalError) no such table: ml2_geneve_allocations when run with low concurrency or on loaded systems - assigned to Yatin * https://bugs.launchpad.net/neutron/+bug/2025129 - DvrLocalRouter init references namespace before it is created - not assigned ## Medium ## * https://bugs.launchpad.net/neutron/+bug/2024381 - keepalived fails to start after updating DVR-HA internal network MTU - assigned to Anton Kurbatov, fix proposed, * https://bugs.launchpad.net/neutron/+bug/2024912 - [ovn-octavia-provider] Updating status on incorrect pool when HM delete - assigned to Fernando * https://bugs.launchpad.net/neutron/+bug/2025056 - Router ports without IP addresses shouldn't be allowed to deletion using port's API directly - assigned to slaweq, fix proposed ## Whishlist ## * https://bugs.launchpad.net/neutron/+bug/2024502 - Tempest: add scenario to validate that stateless SG rules are working in presence of Load Balancer attached to the same network - assigned to slaweq, * https://bugs.launchpad.net/neutron/+bug/2024621 - tempest: add scenario for reverse DNS resolution - not assigned - I marked it as low-hanging-fruit task as I think that writing such test case is good way to start contribution to Neutron, ## RFEs ## * https://bugs.launchpad.net/neutron/+bug/2024581 - [RFE] Caching instance id in the metadata agent to have less RPC messages sent to server - not assigned * https://bugs.launchpad.net/neutron/+bug/2024921- [RFE] Formalize use of subnet service-type for draining subnets - not assigned * https://bugs.launchpad.net/neutron/+bug/2025055 - [rfe][ml2] Add a new API that supports cloning a specified security group - not assigned ## Needs triage ## * https://bugs.launchpad.net/ubuntu/+source/neutron-dynamic-routing/+bug/2024… - Address on SNAT port won't be advertised by BGP speaker - not assigned * https://bugs.launchpad.net/neutron/+bug/2024481- [ndr] neutron-bgp-dragent is racy when a service restart is made just before a speaker is added - frickler is already taking care of this one but it needs more triaging still ## Incomplete ## * https://bugs.launchpad.net/neutron/+bug/2024976 - iptable rules restoring error in l3-agent and openvswitch-agent - already taken care by Brian, it seems like it's deployment issue, -- Slawek Kaplonski Principal Software Engineer Red Hat

1 0

[nova] Last Spec review day next Tuesday !
by Sylvain Bauza 27 Jun '23

27 Jun '23

Hey folks, As a reminder [1], we will have a spec review day next Tuesday June 27th. Sharpen your pens and your Gerrit patches because it will be the last spec review day for this cycle and the Spec Approval Freeze will be on July 6th [2] ! Make sure you have everything uploaded so we can look at them during this day ! After July 6th, no new features will be accepted if they need a specific spec. -Sylvain [1] https://releases.openstack.org/bobcat/schedule.html#b-nova-spec-review-day [2] https://releases.openstack.org/bobcat/schedule.html#b-nova-spec-freeze

1 1

Neutron BGP agent advertisement and l3/openvswitch-agent problems (zed)
by Bryan Huang 26 Jun '23

26 Jun '23

Dear folks, Recently, we met some neutron networking problems in our envrionment, openstack version is zed, and kolla-ansible as the deployment tool. 1. Neutron BGP agent doesn't advertise the floating IPs to the BGP peer, in case of the floating IPs were served for port forwarding, but the floating IPs attached to VM/Container were advertised correctly. so the question is this scenario supported by BGP agent, if not when will it be supported, is it in the plan? 2. iptable rules restoring error in l3-agent and openvswitch-agent (A bug was reported in launchpad: https://bugs.launchpad.net/neutron/+bug/2024976) Bug #2024976 “iptable rules restoring error in l3-agent and open...” : Bugs : neutron<https://bugs.launchpad.net/neutron/+bug/2024976> Openstack version: zed/stable OS version: Ubuntu 22.04.2 LTS Kernel version: 5.15.0-75-generic #82-Ubuntu Deployment: kolla-ansible iptable rules restoring error in l3-agent and openvswitch-agent: openvswitch-agnet log: 2023-06-23 15:54:58.616 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [None req-4440bce1-8c07-4243-ac1b-2566b406a30a - - - - - -] Error while processing VIF ports: neutron_lib.exceptions.ProcessExecutionError: Exit code: 2; Cmd: [... bugs.launchpad.net openvswitch-agnet log: 2023-06-23 15:54:58.616 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [None req-4440bce1-8c07-4243-ac1b-2566b406a30a - - - - - -] Error while processing VIF ports: neutron_lib.exceptions.ProcessExecutionError: Exit code: 2; Cmd: ['iptables-restore', '-n']; Stdin: # Generated by iptables_manager *filter :FORWARD - [0:0] :INPUT - [0:0] :OUTPUT - [0:0] :neutron-filter-top - [0:0] :neutron-openvswi-FORWARD - [0:0] :neutron-openvswi-INPUT - [0:0] :neutron-openvswi-OUTPUT - [0:0] :neutron-openvswi-local - [0:0] :neutron-openvswi-sg-chain - [0:0] :neutron-openvswi-sg-fallback - [0:0] -I FORWARD 1 -j neutron-filter-top -I FORWARD 2 -j neutron-openvswi-FORWARD -I INPUT 1 -j neutron-openvswi-INPUT -I OUTPUT 1 -j neutron-filter-top -I OUTPUT 2 -j neutron-openvswi-OUTPUT -I neutron-filter-top 1 -j neutron-openvswi-local -I neutron-openvswi-FORWARD 1 -m physdev --physdev-out tap2fcacaf9-9d --physdev-is-bridged -m comment --comment "Accept all packets when port is trusted." -j ACCEPT -I neutron-openvswi-FORWARD 2 -m physdev --physdev-in tap2fcacaf9-9d --physdev-is-bridged -m comment --comment "Accept all packets when port is trusted." -j ACCEPT -I neutron-openvswi-FORWARD 3 -m physdev --physdev-out tap8c64cce3-ea --physdev-is-bridged -m comment --comment "Accept all packets when port is trusted." -j ACCEPT -I neutron-openvswi-FORWARD 4 -m physdev --physdev-in tap8c64cce3-ea --physdev-is-bridged -m comment --comment "Accept all packets when port is trusted." -j ACCEPT -I neutron-openvswi-sg-chain 1 -j ACCEPT -I neutron-openvswi-sg-fallback 1 -m comment --comment "Default drop rule for unmatched traffic." -j DROP COMMIT # Completed by iptables_manager # Generated by iptables_manager *raw :OUTPUT - [0:0] :PREROUTING - [0:0] :neutron-openvswi-OUTPUT - [0:0] :neutron-openvswi-PREROUTING - [0:0] -I OUTPUT 1 -j neutron-openvswi-OUTPUT -I PREROUTING 1 -j neutron-openvswi-PREROUTING COMMIT # Completed by iptables_manager ; Stdout: ; Stderr: iptables-restore v1.8.7 (nf_tables): Couldn't load match `physdev':No such file or directory Error occurred at line: 19 Try `iptables-restore -h' or 'iptables-restore --help' for more information. l3-agent log: 2023-06-23 16:15:49.545 33 ERROR neutron.agent.linux.iptables_manager [-] Failure applying iptables rules: neutron_lib.exceptions.ProcessExecutionError: Exit code: 2; Cmd: ['ip', 'netns', 'exec', 'qrouter-0f0e60d0-bf51-4361-901b-4b998201b44b', 'iptables-restore', '-n']; Stdin: # Generated by iptables_manager *filter :FORWARD - [0:0] :INPUT - [0:0] :OUTPUT - [0:0] :neutron-filter-top - [0:0] :neutron-l3-agent-FORWARD - [0:0] :neutron-l3-agent-INPUT - [0:0] :neutron-l3-agent-OUTPUT - [0:0] :neutron-l3-agent-local - [0:0] :neutron-l3-agent-scope - [0:0] -I FORWARD 1 -j neutron-filter-top -I FORWARD 2 -j neutron-l3-agent-FORWARD -I INPUT 1 -j neutron-l3-agent-INPUT -I OUTPUT 1 -j neutron-filter-top -I OUTPUT 2 -j neutron-l3-agent-OUTPUT -I neutron-filter-top 1 -j neutron-l3-agent-local -I neutron-l3-agent-FORWARD 1 -j neutron-l3-agent-scope -I neutron-l3-agent-scope 1 -m mark --mark 0x1/0xffff -j DROP COMMIT # Completed by iptables_manager # Generated by iptables_manager *mangle :FORWARD - [0:0] :INPUT - [0:0] :OUTPUT - [0:0] :POSTROUTING - [0:0] :PREROUTING - [0:0] :neutron-l3-agent-FORWARD - [0:0] :neutron-l3-agent-INPUT - [0:0] :neutron-l3-agent-OUTPUT - [0:0] :neutron-l3-agent-POSTROUTING - [0:0] :neutron-l3-agent-PREROUTING - [0:0] :neutron-l3-agent-float-snat - [0:0] :neutron-l3-agent-floatingip - [0:0] :neutron-l3-agent-mark - [0:0] :neutron-l3-agent-scope - [0:0] -I FORWARD 1 -j neutron-l3-agent-FORWARD -I INPUT 1 -j neutron-l3-agent-INPUT -I OUTPUT 1 -j neutron-l3-agent-OUTPUT -I POSTROUTING 1 -j neutron-l3-agent-POSTROUTING -I PREROUTING 1 -j neutron-l3-agent-PREROUTING -I neutron-l3-agent-PREROUTING 1 -j neutron-l3-agent-mark -I neutron-l3-agent-PREROUTING 2 -j neutron-l3-agent-scope -I neutron-l3-agent-PREROUTING 3 -m connmark ! --mark 0x0/0xffff0000 -j CONNMARK --restore-mark --nfmask 0xffff0000 --ctmask 0xffff0000 -I neutron-l3-agent-PREROUTING 4 -j neutron-l3-agent-floatingip -I neutron-l3-agent-PREROUTING 5 -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j MARK --set-xmark 0x1/0xffff -I neutron-l3-agent-float-snat 1 -m connmark --mark 0x0/0xffff0000 -j CONNMARK --save-mark --nfmask 0xffff0000 --ctmask 0xffff0000 COMMIT # Completed by iptables_manager # Generated by iptables_manager *nat :OUTPUT - [0:0] :POSTROUTING - [0:0] :PREROUTING - [0:0] :neutron-l3-agent-OUTPUT - [0:0] :neutron-l3-agent-POSTROUTING - [0:0] :neutron-l3-agent-PREROUTING - [0:0] :neutron-l3-agent-float-snat - [0:0] :neutron-l3-agent-snat - [0:0] :neutron-postrouting-bottom - [0:0] -I OUTPUT 1 -j neutron-l3-agent-OUTPUT -I POSTROUTING 1 -j neutron-l3-agent-POSTROUTING -I POSTROUTING 2 -j neutron-postrouting-bottom -I PREROUTING 1 -j neutron-l3-agent-PREROUTING -I neutron-l3-agent-POSTROUTING 1 ! -o rfp-0f0e60d0-b -m conntrack ! --ctstate DNAT -j ACCEPT -I neutron-l3-agent-PREROUTING 1 -d 137.175.31.207/32 -i rfp-0f0e60d0-b -j DNAT --to-destination 10.10.0.246 -I neutron-l3-agent-float-snat 1 -s 10.10.0.246/32 -j SNAT --to-source 137.175.31.207 --random-fully -I neutron-l3-agent-snat 1 -j neutron-l3-agent-float-snat -I neutron-postrouting-bottom 1 -m comment --comment "Perform source NAT on outgoing traffic." -j neutron-l3-agent-snat COMMIT # Completed by iptables_manager # Generated by iptables_manager *raw :OUTPUT - [0:0] :PREROUTING - [0:0] :neutron-l3-agent-OUTPUT - [0:0] :neutron-l3-agent-PREROUTING - [0:0] -I OUTPUT 1 -j neutron-l3-agent-OUTPUT -I PREROUTING 1 -j neutron-l3-agent-PREROUTING COMMIT # Completed by iptables_manager ; Stdout: ; Stderr: iptables-restore v1.8.7 (nf_tables): Couldn't load match `mark':No such file or directory Error occurred at line: 19 And we check the system the x_tables kernel module were loaded: # lsmod | grep x_tables x_tables 53248 12 xt_conntrack,nft_compat,xt_tcpudp,xt_physdev,xt_nat,xt_comment,ip6_tables,xt_connmark,xt_CT,ip_tables,xt_REDIRECT,xt_mark (neutron-l3-agent)[neutron@compute06 usr]$ find . -name "*mark.so" ./lib/x86_64-linux-gnu/xtables/libxt_connmark.so ./lib/x86_64-linux-gnu/xtables/libxt_mark.so ./lib/x86_64-linux-gnu/xtables/libebt_mark.so (neutron-l3-agent)[neutron@compute06 usr]$ find . -name "*physdev.so" ./lib/x86_64-linux-gnu/xtables/libxt_physdev.so Does someone have ever met the problems what is the solution the resovle them. Thanks in advance Sincerely， Bryan

2 1

[glance][nova] nova.exception.ImageNotAuthorized: Not authorized for image
by Satish Patel 26 Jun '23

26 Jun '23

Folks, I am running kolla-ansible on small environments with ceph. When I am getting the following error when performing a VM snapshot. This started happening after I upgraded from yoga to Zed. Any idea what changed here? 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server return self._client.call( 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/image/glance.py", line 191, in call 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server result = getattr(controller, method)(*args, **kwargs) 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/glanceclient/v2/images.py", line 503, in add_location 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server response = self._send_image_update_request(image_id, add_patch) 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/glanceclient/common/utils.py", line 670, in inner 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server return RequestIdProxy(wrapped(*args, **kwargs)) 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/glanceclient/v2/images.py", line 483, in _send_image_update_request 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server resp, body = self.http_client.patch(url, headers=hdrs, 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystoneauth1/adapter.py", line 407, in patch 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server return self.request(url, 'PATCH', **kwargs) 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/glanceclient/common/http.py", line 380, in request 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server return self._handle_response(resp) 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/glanceclient/common/http.py", line 120, in _handle_response 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server raise exc.from_response(resp, resp.content) 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server glanceclient.exc.HTTPForbidden: HTTP 403 Forbidden: It's not allowed to add locations if locations are invisible. 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred: 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/exception_wrapper.py", line 65, in wrapped 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server self.force_reraise() 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server raise self.value 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/exception_wrapper.py", line 63, in wrapped 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw) 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 164, in decorated_function 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server self.force_reraise() 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server raise self.value 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 155, in decorated_function 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/utils.py", line 1439, in decorated_function 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 211, in decorated_function 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server self.force_reraise() 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server raise self.value 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 201, in decorated_function 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 231, in decorated_function 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server self.force_reraise() 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server raise self.value 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 228, in decorated_function 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server return function(self, context, image_id, instance, 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 4219, in snapshot_instance 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server self._snapshot_instance(context, image_id, instance, 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 4252, in _snapshot_instance 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server self.driver.snapshot(context, instance, image_id, 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/driver.py", line 3116, in snapshot 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server self.force_reraise() 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server raise self.value 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/driver.py", line 3045, in snapshot 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server self._image_api.update(context, image_id, metadata, 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/image/glance.py", line 1243, in update 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server return session.update(context, image_id, image_info, data=data, 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/image/glance.py", line 693, in update 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server _reraise_translated_image_exception(image_id) 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/image/glance.py", line 1031, in _reraise_translated_image_exception 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server raise new_exc.with_traceback(exc_trace) 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/image/glance.py", line 691, in update 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server image = self._update_v2(context, sent_service_image_meta, data) 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/image/glance.py", line 705, in _update_v2 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server image = self._add_location(context, image_id, location) 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/image/glance.py", line 561, in _add_location 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server return self._client.call( 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/image/glance.py", line 191, in call 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server result = getattr(controller, method)(*args, **kwargs) 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/glanceclient/v2/images.py", line 503, in add_location 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server response = self._send_image_update_request(image_id, add_patch) 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/glanceclient/common/utils.py", line 670, in inner 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server return RequestIdProxy(wrapped(*args, **kwargs)) 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/glanceclient/v2/images.py", line 483, in _send_image_update_request 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server resp, body = self.http_client.patch(url, headers=hdrs, 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystoneauth1/adapter.py", line 407, in patch 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server return self.request(url, 'PATCH', **kwargs) 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/glanceclient/common/http.py", line 380, in request 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server return self._handle_response(resp) 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/glanceclient/common/http.py", line 120, in _handle_response 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server raise exc.from_response(resp, resp.content) 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server nova.exception.ImageNotAuthorized: Not authorized for image 6d39ead7-e543-4ab6-b54c-78ca16421242. 2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server

2 2

[manila] Vancouver PTG and Forum
by Carlos Silva 26 Jun '23

26 Jun '23

Hello, Zorillas! I would like to share with you a brief summary of the PTG and the Manila forum from last week's OpenInfra summit. *Manila Forum:* In the Manila Operator Forum session [0], we showed how the Ceph driver could support clustered NFS. We had feedback from operators in different areas. Many operators in the room were interested in VirtIOFS so they can consume Native CephFS without having to expose the cluster to user VMs/containers. The discussion extended into the Nova Forum session the next day. *PTG:* Open discussion/operator feedback: - It would be great if we had this bug [1] fixed. - Ceph Mon IP changes not reflecting on exported shares - We are targeting the fix for this bug on Bobcat-2 - Interest in share backups: - CERN is looking for a share backup approach similar to what cinder currently does. We have one blueprint open for it and the implementation is currently going on [2] - This can likely integrate with a tool they are developing internally: CBACK - CERN will also provide feedback on the implementation of this feature and help us shape it. For more details, please check the Manila PTG etherpad [3]. We joined the Nova team to discuss the manila work for supporting VirtioFS. Please check out the notes for the discussion in the nova summary [4]. [0] https://etherpad.opendev.org/p/manila-vancouver-forum-2023 [1] https://bugs.launchpad.net/manila/+bug/1996793 [2] https://review.opendev.org/c/openstack/manila/+/343980 [3] https://etherpad.opendev.org/p/manila-openinfra-2023 [4] https://lists.openstack.org/pipermail/openstack-discuss/2023-June/034152.ht… Thanks, carloss

2 2

[tc] Technical Committee next weekly meeting on June 27, 2023
by Nikolla, Kristi 26 Jun '23

26 Jun '23

Hi all, This is a reminder that the next weekly Technical Committee meeting is to be held on Tuesday, Jun 27, 2023 at 1800 UTC on #openstack-tc on OFTC IRC. Please propose items to the agenda by editing the wiki page at https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting At the end of the day I will send out an email with the finalized agenda. Thank you, Kristi Nikolla

1 0

[blazar] Cancelling next two IRC meetings
by Pierre Riteau 26 Jun '23

26 Jun '23

Hello, I am cancelling the next two Blazar IRC meetings due to being out of the office. The next meeting will be on July 27. Best wishes, Pierre Riteau (priteau)

1 0

[kolla-ansible][nova] nova upgrade failed from zed to antelope
by Satish Patel 26 Jun '23

26 Jun '23

Folks, I'am upgrading zed to antelope using kolla-ansible and encount following error TASK [nova : Upgrade status check result] ************************************************************************************************************************************************************ fatal: [kolla-infra-1]: FAILED! => {"changed": false, "msg": ["There was an upgrade status check failure!", "See the detail at https://docs.openstack.org/nova/latest/cli/nova-status.html#nova-status-che… "]} fatal: [kolla-infra-2]: FAILED! => {"changed": false, "msg": ["There was an upgrade status check failure!", "See the detail at https://docs.openstack.org/nova/latest/cli/nova-status.html#nova-status-che… "]} fatal: [kolla-infra-3]: FAILED! => {"changed": false, "msg": ["There was an upgrade status check failure!", "See the detail at https://docs.openstack.org/nova/latest/cli/nova-status.html#nova-status-che… "]} After running upgrade check command manually on nova-api container got following error (nova-api)[root@kolla-infra-2 /]# nova-status upgrade check Modules with known eventlet monkey patching issues were imported prior to eventlet monkey patching: urllib3. This warning can usually be ignored if the caller is only importing and not executing nova code. +---------------------------------------------------------------------+ | Upgrade Check Results | +---------------------------------------------------------------------+ | Check: Cells v2 | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: Placement API | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: Cinder API | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: Policy File JSON to YAML Migration | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: Older than N-1 computes | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: hw_machine_type unset | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: Service User Token Configuration | | Result: Failure | | Details: Service user token configuration is required for all Nova | | services. For more details see the following: https://docs | | .openstack.org/latest/nova/admin/configuration/service- | | user-token.html | +---------------------------------------------------------------------+ Service user token reference to the following doc [1] . Do I need to configure token users in order to upgrade nova? [1] https://docs.openstack.org/nova/latest/admin/configuration/service-user-tok…

4 6

[kolla] Weekly meetings in July 2023
by Michał Nasiadka 26 Jun '23

26 Jun '23

Hello Koalas, We’ve decided last week that due to my absence from 1st July - we’re cancelling weekly meetings for the first three weeks of July - i.e. we will meet 26th of July for the only July meeting. NOTE: This Wednesday meeting is not cancelled In case of queries of additional topics - please reach out on #openstack-kolla Best regards, Michal

1 0

[kolla-ansible][yoga] How can I modify the logrotate configuration for kolla to save some space
by wodel youchi 26 Jun '23

26 Jun '23

Hi, Is there a way to tweak kolla's logrotate configuration, to save up some space on /var/log/kolla? Regards. <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campai…> Virus-free.www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campai…> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

2 2

Kolla iser (RDMA) support
by Tomas Leypold 26 Jun '23

26 Jun '23

Hi, I have a kolla-ansible deployment, and I am trying to switch from the LVM iSCSI protocol to iSER (RDMA). On the host machine, I have installed rdma-core, which enabled kernel module loading for the NIC card we are using, the "BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller". I have verified the function of RDMA with ib_write_bw, which worked fine. The first problem I encountered with Kolla is the absence of necessary packages - ibverbs-providers, which includes userspace drivers for RDMA. Without it, cinder_volume will not attach a volume. I did a image rebuild with package ibverbs-providers included. I will fill an issue for this later. Then I just changed target_protocol = iser in cinder_volume config file. The current issue I am facing is that the instance is stuck at the "Spawning" phase. In the nova_compute log, I see the following lines: 2023-06-26 10:57:36.204 7 DEBUG os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] ==> connect_volume: call "{'self': <os_brick.initiator.connectors.iscsi.ISCSIConnector object at 0x7f4dd45adc30>, 'connection_properties': {'target_discovered': False, 'target_portal': '10.47.1.25:3260', 'target_iqn': 'iqn.2010-10.org.openstack:volume-bc382b4c-dac4-4ca6-a260-0d450d6c4e26', 'target_lun': 1, 'volume_id': 'bc382b4c-dac4-4ca6-a260-0d450d6c4e26', 'auth_method': 'CHAP', 'auth_username': '8n7WyZ2wZFNd7B33rfYw', 'auth_password': '***', 'encrypted': False, 'qos_specs': None, 'access_mode': 'rw': False}}" trace_logging_wrapper /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/utils.py:174 2023-06-26 10:57:36.208 7 DEBUG os_brick.initiator.connectors.base [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] Acquiring lock "connect_to_iscsi_portal-10.47.1.25:3260-iqn.2010-10.org.openstack:volume-bc382b4c-dac4-4ca6-a260-0d450d6c4e26" by "os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe" inner /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/base.py:68 2023-06-26 10:57:36.208 7 DEBUG os_brick.initiator.connectors.base [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] Lock "connect_to_iscsi_portal-10.47.1.25:3260-iqn.2010-10.org.openstack:volume-bc382b4c-dac4-4ca6-a260-0d450d6c4e26" acquired by "os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe" :: waited 0.001s inner /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/base.py:73 2023-06-26 10:57:36.209 7 INFO os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] Trying to connect to iSCSI portal 10.47.1.25:3260 2023-06-26 10:57:36.219 7 DEBUG os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] iscsiadm (): stdout= stderr=iscsiadm: No records found _run_iscsiadm /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py:1026 2023-06-26 10:57:36.227 7 DEBUG os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] iscsiadm ('--interface', 'iser', '--op', 'new'): stdout=New iSCSI node [iser:[hw=,ip=,net_if=,iscsi_if=iser] 10.47.1.25,3260,-1 iqn.2010-10.org.openstack:volume-bc382b4c-dac4-4ca6-a260-0d450d6c4e26] added stderr= _run_iscsiadm /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py:1026 2023-06-26 10:57:36.234 7 DEBUG os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] iscsiadm ('--op', 'update', '-n', 'node.session.scan', '-v', 'manual'): stdout= stderr= _run_iscsiadm /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py:1026 2023-06-26 10:57:36.241 7 DEBUG os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] iscsiadm ('--op', 'update', '-n', 'node.session.auth.authmethod', '-v', 'CHAP'): stdout= stderr= _run_iscsiadm /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py:1026 2023-06-26 10:57:36.247 7 DEBUG os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] iscsiadm ('--op', 'update', '-n', 'node.session.auth.username', '-v', '8n7WyZ2wZFNd7B33rfYw'): stdout= stderr= _run_iscsiadm /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py:1026 2023-06-26 10:57:36.254 7 DEBUG os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] iscsiadm ('--op', 'update', '-n', 'node.session.auth.password', '-v', '***'): stdout= stderr= _run_iscsiadm /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py:1026 2023-06-26 10:57:36.260 7 DEBUG os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] iscsiadm ('-m', 'session'): stdout= stderr=iscsiadm: No active sessions. _run_iscsiadm_bare /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py:1182 2023-06-26 10:57:36.261 7 DEBUG os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] iscsi session list stdout= stderr=iscsiadm: No active sessions. _run_iscsi_session /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py:1171 2023-06-26 10:57:36.261 7 WARNING os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] iscsiadm stderr output when getting sessions: iscsiadm: No active sessions. 2 min later it timeouts: 2023-06-26 10:59:38.377 7 WARNING os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] Failed to login iSCSI target iqn.2010-10.org.openstack:volume-bc382b4c-dac4-4ca6-a260-0d450d6c4e26 on portal 10.47.1.25:3260 (exit code 8).: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command. 2023-06-26 10:59:38.378 7 DEBUG os_brick.initiator.connectors.base [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] Lock "connect_to_iscsi_portal-10.47.1.25:3260-iqn.2010-10.org.openstack:volume-bc382b4c-dac4-4ca6-a260-0d450d6c4e26" "released" by "os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe" :: held 122.170s inner /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/base.py:87 2023-06-26 10:59:38.378 7 WARNING os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] Failed to connect to iSCSI portal 10.47.1.25:3260. 2023-06-26 10:59:38.378 7 DEBUG os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] Getting connected devices for (ips,iqns,luns)=[('10.47.1.25:3260', 'iqn.2010-10.org.openstack:volume-bc382b4c-dac4-4ca6-a260-0d450d6c4e26', 1)] _get_connection_devices /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py:819 2023-06-26 10:59:38.394 7 DEBUG os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] iscsiadm ('-m', 'session'): stdout= stderr=iscsiadm: No active sessions. _run_iscsiadm_bare /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py:1182 2023-06-26 10:59:38.394 7 DEBUG os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] iscsi session list stdout= stderr=iscsiadm: No active sessions. _run_iscsi_session /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py:1171 2023-06-26 10:59:38.394 7 WARNING os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] iscsiadm stderr output when getting sessions: iscsiadm: No active sessions. 2023-06-26 10:59:38.394 7 DEBUG os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] Resulting device map defaultdict(<function ISCSIConnector._get_connection_devices.<locals>.<lambda> at 0x7f4dd43230a0>, {('10.47.1.25:3260', 'iqn.2010-10.org.openstack:volume-bc382b4c-dac4-4ca6-a260-0d450d6c4e26'): (set(), set())}) _get_connection_devices /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py:852 2023-06-26 10:59:38.394 7 DEBUG os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] Disconnecting from: [('10.47.1.25:3260', 'iqn.2010-10.org.openstack:volume-bc382b4c-dac4-4ca6-a260-0d450d6c4e26')] _disconnect_connection /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py:1160 2023-06-26 10:59:38.401 7 DEBUG os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] iscsiadm ('--op', 'update', '-n', 'node.startup', '-v', 'manual'): stdout= stderr= _run_iscsiadm /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py:1026 2023-06-26 10:59:38.407 7 DEBUG os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] iscsiadm ('--logout',): stdout= stderr=iscsiadm: No matching sessions found _run_iscsiadm /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py:1026 2023-06-26 10:59:38.413 7 DEBUG os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] iscsiadm ('--op', 'delete'): stdout= stderr= _run_iscsiadm /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py:1026 2023-06-26 10:59:38.414 7 DEBUG os_brick.utils [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] Finished call to 'os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_single_volume' after 122.207(s), this was the 1st time calling it. log_it /var/lib/kolla/venv/lib/python3.10/site-packages/tenacity/after.py:30 2023-06-26 10:59:38.415 7 DEBUG os_brick.utils [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] Retrying os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_single_volume in 1.0 seconds as it raised VolumeDeviceNotFound: Volume device not found at .. log_it /var/lib/kolla/venv/lib/python3.10/site-packages/tenacity/before_sleep.py:40 2023-06-26 10:59:39.393 7 DEBUG oslo_service.periodic_task [None req-a325bda7-8910-43e3-90a9-6ae0dc8ae6a7 - - - - - -] Running periodic task ComputeManager._instance_usage_audit run_periodic_tasks /var/lib/kolla/venv/lib/python3.10/site-packages/oslo_service/periodic_task.py:210 2023-06-26 10:59:39.416 7 WARNING os_brick.initiator.connectors.base [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] Service needs to call os_brick.setup() before connecting volumes, if it doesn't it will break on the next release 2023-06-26 10:59:39.416 7 DEBUG os_brick.initiator.connectors.base [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] Acquiring lock "connect_to_iscsi_portal-10.47.1.25:3260-iqn.2010-10.org.openstack:volume-bc382b4c-dac4-4ca6-a260-0d450d6c4e26" by "os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe" inner /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/base.py:68 2023-06-26 10:59:39.417 7 DEBUG os_brick.initiator.connectors.base [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] Lock "connect_to_iscsi_portal-10.47.1.25:3260-iqn.2010-10.org.openstack:volume-bc382b4c-dac4-4ca6-a260-0d450d6c4e26" acquired by "os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe" :: waited 0.001s inner /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/base.py:73 2023-06-26 10:59:39.418 7 INFO os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] Trying to connect to iSCSI portal 10.47.1.25:3260 2023-06-26 10:59:39.426 7 DEBUG os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] iscsiadm (): stdout= stderr=iscsiadm: No records found _run_iscsiadm /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py:1026 2023-06-26 10:59:39.435 7 DEBUG os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] iscsiadm ('--interface', 'iser', '--op', 'new'): stdout=New iSCSI node [iser:[hw=,ip=,net_if=,iscsi_if=iser] 10.47.1.25,3260,-1 iqn.2010-10.org.openstack:volume-bc382b4c-dac4-4ca6-a260-0d450d6c4e26] added stderr= _run_iscsiadm /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py:1026 2023-06-26 10:59:39.441 7 DEBUG os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] iscsiadm ('--op', 'update', '-n', 'node.session.scan', '-v', 'manual'): stdout= stderr= _run_iscsiadm /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py:1026 2023-06-26 10:59:39.447 7 DEBUG os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] iscsiadm ('--op', 'update', '-n', 'node.session.auth.authmethod', '-v', 'CHAP'): stdout= stderr= _run_iscsiadm /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py:1026 2023-06-26 10:59:39.454 7 DEBUG os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] iscsiadm ('--op', 'update', '-n', 'node.session.auth.username', '-v', '8n7WyZ2wZFNd7B33rfYw'): stdout= stderr= _run_iscsiadm /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py:1026 2023-06-26 10:59:39.454 7 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 20 __log_wakeup /usr/lib/python3/dist-packages/ovs/poller.py:263 2023-06-26 10:59:39.461 7 DEBUG os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] iscsiadm ('--op', 'update', '-n', 'node.session.auth.password', '-v', '***'): stdout= stderr= _run_iscsiadm /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py:1026 2023-06-26 10:59:39.467 7 DEBUG os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] iscsiadm ('-m', 'session'): stdout= stderr=iscsiadm: No active sessions. _run_iscsiadm_bare /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py:1182 2023-06-26 10:59:39.467 7 DEBUG os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] iscsi session list stdout= stderr=iscsiadm: No active sessions. _run_iscsi_session /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py:1171 2023-06-26 10:59:39.468 7 WARNING os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] iscsiadm stderr output when getting sessions: iscsiadm: No active sessions. In this log, the problem is the line "Failed to login to iSCSI target", but I am not sure why this is happening. Does anyone have any ideas about what could be wrong? Thanks! --- Best regards, Tomas Leypold

1 0

Neutron BGP agent advertisement and l3/openvswitch-agent problems (zed)
by Bryan Huang 24 Jun '23

24 Jun '23

Dear folks, Recently, we met some neutron networking problems in our envrionment, openstack version is zed, and kolla-ansible as the deployment tool. 1. Neutron BGP agent doesn't advertise the floating IPs to the BGP peer, in case of the floating IPs were served for port forwarding, but the floating IPs attached to VM/Container were advertised correctly. so the question is this scenario supported by BGP agent, if not when will it be supported, is it in the plan? 2. iptable rules restoring error in l3-agent and openvswitch-agent (A bug was reported in launchpad: https://bugs.launchpad.net/neutron/+bug/2024976) Bug #2024976 “iptable rules restoring error in l3-agent and open...” : Bugs : neutron<https://bugs.launchpad.net/neutron/+bug/2024976> Openstack version: zed/stable OS version: Ubuntu 22.04.2 LTS Kernel version: 5.15.0-75-generic #82-Ubuntu Deployment: kolla-ansible iptable rules restoring error in l3-agent and openvswitch-agent: openvswitch-agnet log: 2023-06-23 15:54:58.616 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [None req-4440bce1-8c07-4243-ac1b-2566b406a30a - - - - - -] Error while processing VIF ports: neutron_lib.exceptions.ProcessExecutionError: Exit code: 2; Cmd: [... bugs.launchpad.net openvswitch-agnet log: 2023-06-23 15:54:58.616 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [None req-4440bce1-8c07-4243-ac1b-2566b406a30a - - - - - -] Error while processing VIF ports: neutron_lib.exceptions.ProcessExecutionError: Exit code: 2; Cmd: ['iptables-restore', '-n']; Stdin: # Generated by iptables_manager *filter :FORWARD - [0:0] :INPUT - [0:0] :OUTPUT - [0:0] :neutron-filter-top - [0:0] :neutron-openvswi-FORWARD - [0:0] :neutron-openvswi-INPUT - [0:0] :neutron-openvswi-OUTPUT - [0:0] :neutron-openvswi-local - [0:0] :neutron-openvswi-sg-chain - [0:0] :neutron-openvswi-sg-fallback - [0:0] -I FORWARD 1 -j neutron-filter-top -I FORWARD 2 -j neutron-openvswi-FORWARD -I INPUT 1 -j neutron-openvswi-INPUT -I OUTPUT 1 -j neutron-filter-top -I OUTPUT 2 -j neutron-openvswi-OUTPUT -I neutron-filter-top 1 -j neutron-openvswi-local -I neutron-openvswi-FORWARD 1 -m physdev --physdev-out tap2fcacaf9-9d --physdev-is-bridged -m comment --comment "Accept all packets when port is trusted." -j ACCEPT -I neutron-openvswi-FORWARD 2 -m physdev --physdev-in tap2fcacaf9-9d --physdev-is-bridged -m comment --comment "Accept all packets when port is trusted." -j ACCEPT -I neutron-openvswi-FORWARD 3 -m physdev --physdev-out tap8c64cce3-ea --physdev-is-bridged -m comment --comment "Accept all packets when port is trusted." -j ACCEPT -I neutron-openvswi-FORWARD 4 -m physdev --physdev-in tap8c64cce3-ea --physdev-is-bridged -m comment --comment "Accept all packets when port is trusted." -j ACCEPT -I neutron-openvswi-sg-chain 1 -j ACCEPT -I neutron-openvswi-sg-fallback 1 -m comment --comment "Default drop rule for unmatched traffic." -j DROP COMMIT # Completed by iptables_manager # Generated by iptables_manager *raw :OUTPUT - [0:0] :PREROUTING - [0:0] :neutron-openvswi-OUTPUT - [0:0] :neutron-openvswi-PREROUTING - [0:0] -I OUTPUT 1 -j neutron-openvswi-OUTPUT -I PREROUTING 1 -j neutron-openvswi-PREROUTING COMMIT # Completed by iptables_manager ; Stdout: ; Stderr: iptables-restore v1.8.7 (nf_tables): Couldn't load match `physdev':No such file or directory Error occurred at line: 19 Try `iptables-restore -h' or 'iptables-restore --help' for more information. l3-agent log: 2023-06-23 16:15:49.545 33 ERROR neutron.agent.linux.iptables_manager [-] Failure applying iptables rules: neutron_lib.exceptions.ProcessExecutionError: Exit code: 2; Cmd: ['ip', 'netns', 'exec', 'qrouter-0f0e60d0-bf51-4361-901b-4b998201b44b', 'iptables-restore', '-n']; Stdin: # Generated by iptables_manager *filter :FORWARD - [0:0] :INPUT - [0:0] :OUTPUT - [0:0] :neutron-filter-top - [0:0] :neutron-l3-agent-FORWARD - [0:0] :neutron-l3-agent-INPUT - [0:0] :neutron-l3-agent-OUTPUT - [0:0] :neutron-l3-agent-local - [0:0] :neutron-l3-agent-scope - [0:0] -I FORWARD 1 -j neutron-filter-top -I FORWARD 2 -j neutron-l3-agent-FORWARD -I INPUT 1 -j neutron-l3-agent-INPUT -I OUTPUT 1 -j neutron-filter-top -I OUTPUT 2 -j neutron-l3-agent-OUTPUT -I neutron-filter-top 1 -j neutron-l3-agent-local -I neutron-l3-agent-FORWARD 1 -j neutron-l3-agent-scope -I neutron-l3-agent-scope 1 -m mark --mark 0x1/0xffff -j DROP COMMIT # Completed by iptables_manager # Generated by iptables_manager *mangle :FORWARD - [0:0] :INPUT - [0:0] :OUTPUT - [0:0] :POSTROUTING - [0:0] :PREROUTING - [0:0] :neutron-l3-agent-FORWARD - [0:0] :neutron-l3-agent-INPUT - [0:0] :neutron-l3-agent-OUTPUT - [0:0] :neutron-l3-agent-POSTROUTING - [0:0] :neutron-l3-agent-PREROUTING - [0:0] :neutron-l3-agent-float-snat - [0:0] :neutron-l3-agent-floatingip - [0:0] :neutron-l3-agent-mark - [0:0] :neutron-l3-agent-scope - [0:0] -I FORWARD 1 -j neutron-l3-agent-FORWARD -I INPUT 1 -j neutron-l3-agent-INPUT -I OUTPUT 1 -j neutron-l3-agent-OUTPUT -I POSTROUTING 1 -j neutron-l3-agent-POSTROUTING -I PREROUTING 1 -j neutron-l3-agent-PREROUTING -I neutron-l3-agent-PREROUTING 1 -j neutron-l3-agent-mark -I neutron-l3-agent-PREROUTING 2 -j neutron-l3-agent-scope -I neutron-l3-agent-PREROUTING 3 -m connmark ! --mark 0x0/0xffff0000 -j CONNMARK --restore-mark --nfmask 0xffff0000 --ctmask 0xffff0000 -I neutron-l3-agent-PREROUTING 4 -j neutron-l3-agent-floatingip -I neutron-l3-agent-PREROUTING 5 -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j MARK --set-xmark 0x1/0xffff -I neutron-l3-agent-float-snat 1 -m connmark --mark 0x0/0xffff0000 -j CONNMARK --save-mark --nfmask 0xffff0000 --ctmask 0xffff0000 COMMIT # Completed by iptables_manager # Generated by iptables_manager *nat :OUTPUT - [0:0] :POSTROUTING - [0:0] :PREROUTING - [0:0] :neutron-l3-agent-OUTPUT - [0:0] :neutron-l3-agent-POSTROUTING - [0:0] :neutron-l3-agent-PREROUTING - [0:0] :neutron-l3-agent-float-snat - [0:0] :neutron-l3-agent-snat - [0:0] :neutron-postrouting-bottom - [0:0] -I OUTPUT 1 -j neutron-l3-agent-OUTPUT -I POSTROUTING 1 -j neutron-l3-agent-POSTROUTING -I POSTROUTING 2 -j neutron-postrouting-bottom -I PREROUTING 1 -j neutron-l3-agent-PREROUTING -I neutron-l3-agent-POSTROUTING 1 ! -o rfp-0f0e60d0-b -m conntrack ! --ctstate DNAT -j ACCEPT -I neutron-l3-agent-PREROUTING 1 -d 137.175.31.207/32 -i rfp-0f0e60d0-b -j DNAT --to-destination 10.10.0.246 -I neutron-l3-agent-float-snat 1 -s 10.10.0.246/32 -j SNAT --to-source 137.175.31.207 --random-fully -I neutron-l3-agent-snat 1 -j neutron-l3-agent-float-snat -I neutron-postrouting-bottom 1 -m comment --comment "Perform source NAT on outgoing traffic." -j neutron-l3-agent-snat COMMIT # Completed by iptables_manager # Generated by iptables_manager *raw :OUTPUT - [0:0] :PREROUTING - [0:0] :neutron-l3-agent-OUTPUT - [0:0] :neutron-l3-agent-PREROUTING - [0:0] -I OUTPUT 1 -j neutron-l3-agent-OUTPUT -I PREROUTING 1 -j neutron-l3-agent-PREROUTING COMMIT # Completed by iptables_manager ; Stdout: ; Stderr: iptables-restore v1.8.7 (nf_tables): Couldn't load match `mark':No such file or directory Error occurred at line: 19 And we check the system the x_tables kernel module were loaded: # lsmod | grep x_tables x_tables 53248 12 xt_conntrack,nft_compat,xt_tcpudp,xt_physdev,xt_nat,xt_comment,ip6_tables,xt_connmark,xt_CT,ip_tables,xt_REDIRECT,xt_mark (neutron-l3-agent)[neutron@compute06 usr]$ find . -name "*mark.so" ./lib/x86_64-linux-gnu/xtables/libxt_connmark.so ./lib/x86_64-linux-gnu/xtables/libxt_mark.so ./lib/x86_64-linux-gnu/xtables/libebt_mark.so (neutron-l3-agent)[neutron@compute06 usr]$ find . -name "*physdev.so" ./lib/x86_64-linux-gnu/xtables/libxt_physdev.so Does someone have ever met the problems what is the solution the resovle them. Thanks in advance Sincerely， Bryan

1 0

the nova errors (vresion queens) errors
by 无数的星球 24 Jun '23

24 Jun '23

I reported an error on the compute node after building nova。 2023-06-23 10:04:25.334 15207 ERROR nova.compute.manager [req-a85e1436-e5a4-4395-ba5c-899d3b21b7bc - - - - -] Error updating resources for node compute.: KeyError: 'allocations' 2023-06-23 10:04:25.334 15207 ERROR nova.compute.manager Traceback (most recent call last): 2023-06-23 10:04:25.334 15207 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7578, in update_available_resource_for_node 2023-06-23 10:04:25.334 15207 ERROR nova.compute.manager rt.update_available_resource(context, nodename) 2023-06-23 10:04:25.334 15207 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 720, in update_available_resource 2023-06-23 10:04:25.334 15207 ERROR nova.compute.manager self._update_available_resource(context, resources) 2023-06-23 10:04:25.334 15207 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2023-06-23 10:04:25.334 15207 ERROR nova.compute.manager return f(*args, **kwargs) 2023-06-23 10:04:25.334 15207 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 771, in _update_available_resource 2023-06-23 10:04:25.334 15207 ERROR nova.compute.manager context, self.compute_nodes[nodename], migrations) 2023-06-23 10:04:25.334 15207 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 1277, in _remove_deleted_instances_allocations 2023-06-23 10:04:25.334 15207 ERROR nova.compute.manager context, cn.uuid) or {} 2023-06-23 10:04:25.334 15207 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 37, in __run_method 2023-06-23 10:04:25.334 15207 ERROR nova.compute.manager return getattr(self.instance, __name)(*args, **kwargs) 2023-06-23 10:04:25.334 15207 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 64, in wrapper 2023-06-23 10:04:25.334 15207 ERROR nova.compute.manager return f(self, *a, **k) 2023-06-23 10:04:25.334 15207 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 1785, in get_allocations_for_resource_provider After this error occurs, after building the neutron and creating the virtual machine, it will be unable to create the virtual machine, which will be reported in the controller NoValidHost: No valid host was found. 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager Traceback (most recent call last): 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 1165, in schedule_and_build_instances 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager instance_uuids, return_alternates=True) 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 760, in _schedule_instances 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager return_alternates=return_alternates) 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/utils.py", line 793, in wrapped 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager return func(*args, **kwargs) 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/init.py", line 53, in select_destinations 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager instance_uuids, return_objects, return_alternates) 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/init.py", line 37, in __run_method 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager return getattr(self.instance, __name)(*args, **kwargs) 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/query.py", line 42, in select_destinations 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager instance_uuids, return_objects, return_alternates) 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/rpcapi.py", line 158, in select_destinations 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager return cctxt.call(ctxt, 'select_destinations', **msg_args) 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 174, in call 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager retry=self.retry) 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 131, in _send 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager timeout=timeout, retry=retry) 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 625, in send 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager retry=retry) 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 616, in _send 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager raise result 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager NoValidHost_Remote: No valid host was found. 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager Traceback (most recent call last): 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 229, in inner 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager return func(*args, **kwargs) 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py", line 139, in select_destinations 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager raise exception.NoValidHost(reason="") 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager NoValidHost: No valid host was found. 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager Note: As the logs were obtained at different times, please ignore the timing issue. The following will list my nova configuration files. Please review them. Thank you system is centos 7 2009 When a computer error is found, Error updating resources for node compute.: KeyError: 'allocations' the controller will report it 2023-06-24 10:45:51.133 96992 INFO nova.osapi_compute.wsgi.server [-] 192.168.80.20 "GET /resource_providers/6304d158-7a38-4a61-b6f6-f16c22e27b53/allocations HTTP/1.1" status: 300 len: 720 time: 0.0006239 the placement is Fix bugs Listen 8778 <VirtualHost *:8778>   WSGIProcessGroup nova-placement-api   WSGIApplicationGroup %{GLOBAL}   WSGIPassAuthorization On   WSGIDaemonProcess nova-placement-api processes=3 threads=1 user=nova group=nova   WSGIScriptAlias / /usr/bin/nova-placement-api   <IfVersion >= 2.4>     ErrorLogFormat "%M"   </IfVersion>   ErrorLog /var/log/nova/nova-placement-api.log   #SSLEngine On   #SSLCertificateFile ...   #SSLCertificateKeyFile ... </VirtualHost> Alias /nova-placement-api /usr/bin/nova-placement-api <Location /nova-placement-api>   SetHandler wsgi-script   Options +ExecCGI   WSGIProcessGroup nova-placement-api   WSGIApplicationGroup %{GLOBAL}   WSGIPassAuthorization On </Location> <Directory /usr/bin>   <IfVersion >= 2.4>         Require all granted   </IfVersion>   <IfVersion < 2.4>         Order allow,deny         Allow from all   </IfVersion> </Directory>

1 0