April 2023 - openstack-discuss - lists.openstack.org

[oslo.messaging][nova][neutron] quorum queues
by Arnaud Morin 13 Apr '23

13 Apr '23

Hey team, I am currently looking into using quorum queues (see patch [1] introduced in oslo_messaging) for openstack services (nova, neutron mostly). After switching the value to true and bootstraping a new cluster, all queues are quorum except the fanout and reply queues. I understand that transient queues are managed differently in openstack, but why is it so? I am asking this because when using rabbit in cluster mode, the quorum queues are replicated accross the cluster. But because the fanout and reply queues are not replicated, if one of the cluster is shutting down during an instance boot, the boot may endup in error (the reply queue may be on the faulty node). Is there any reason to not enable quorum on transient queues? [1] https://github.com/openstack/oslo.messaging/commit/7e8acbf87070354cd2438f3a…

1 0

Updating OS-TRAITS in placement stable/yoga and stable/zed
by Karl Kloppenborg 13 Apr '23

13 Apr '23

Hello Placement and OS-Traits teams. Karl from Openstack-Helm. After debugging a number of issues in Cyborg, it has become necessary to update the pinned OS-TRAITs version in stable/yoga and stable/zed. I have submitted the following two reviews for updating the pinned versions to 2.10.0 https://review.opendev.org/c/openstack/placement/+/880249 https://review.opendev.org/c/openstack/placement/+/880249 By merging these changes, it will allow us to continue to use vGPU capabilities in cyborg in the stable/yoga and stable/zed onwards branches. Kind Regards, -- Karl Kloppenborg, (BCompSc, CNCF-[KCNA, CKA, CKAD], LFCE, CompTIA Linux+ XK0-004) Managing Director, Invention Labs.

3 2

[Cloudkitty][ptg] CloudKitty 2023.2 Bobcat PTG outcome
by Rafael Weingärtner 13 Apr '23

13 Apr '23

Hello, Thanks to everyone who attended the CloudKitty PTG sessions, we covered a lot during the meeting. The Etherpad is available at [1]. [1] https://etherpad.opendev.org/p/march2023-ptg-cloudkitty -- Rafael Weingärtner

1 0

Does cinder-backup support multi-backend?
by wodel youchi 13 Apr '23

13 Apr '23

Hi, Could cinder-backup be configured with multiple-backends, ceph and nfs for example? Regards.

5 9

[kolla-ansible][mariadb] socat: Connection reset by peer
by Satish Patel 12 Apr '23

12 Apr '23

Folks, I am getting the following errors in logs constantly and continually. I have 3 node galera clusters. Everything looks good except this error making me nervous. I have deployed Zed release of kolla openstack. ==> /var/log/kolla/mariadb/mariadb-clustercheck.log <== 2023/04/12 02:42:16 socat[1574755] W read(7, 0x5653c4a4c000, 8192): Connection reset by peer 2023/04/12 02:42:16 socat[1574761] W read(7, 0x5653c4a4c000, 8192): Connection reset by peer 2023/04/12 02:42:16 socat[1574755] W read(6, 0x5653c4a4c000, 8192): Connection reset by peer 2023/04/12 02:42:16 socat[1574761] W read(6, 0x5653c4a4c000, 8192): Connection reset by peer 2023/04/12 02:42:18 socat[1574767] W read(7, 0x5653c4a4c000, 8192): Connection reset by peer 2023/04/12 02:42:18 socat[1574767] W read(6, 0x5653c4a4c000, 8192): Connection reset by peer

2 2

[kolla-ansible][yoga] how to configure cinder to connect to iscsi target?
by wodel youchi 12 Apr '23

12 Apr '23

Hi, I am trying to configure cinder to connect to an iscsi target. I am using targetcli as my target server. I have tested the connection from the node using iscsiadm and it works from the OS. Now I am trying to use the iscsid container to do the job. I added this to my globals.yml enable_iscsid: "yes" enable_cinder_backend_iscsi: "yes" enable_cinder_backend_lvm: "no" And I created the cinder.conf file like this [DEFAULT] enabled_backends = rbd1,nfs-1,*lvmtnas * # add follows to the end [lvmtnas] target_helper = lioadm target_protocol = iscsi target_ip_address = 20.1.0.32 (my target server) volume_backend_name = lvmtnas # volume group name created volume_group = *vg_volume01 * volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver use_chap_auth = false I did create the vg_volume01 on the LUN prior to the test. I did a deploy, the container has been created, but nothing happened, it's like I am missing something. docker logs iscsid fome one the compute nodes + sudo -E kolla_set_configs INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files INFO:__main__:Writing out command to execute ++ cat /run_command + CMD='iscsid -d 8 -f' + ARGS= + sudo kolla_copy_cacerts + [[ ! -n '' ]] + . kolla_extend_start ++ [[ ! -f /etc/iscsi/initiatorname.iscsi ]] + echo 'Running command: '\''iscsid -d 8 -f'\''' + exec iscsid -d 8 -f Running command: 'iscsid -d 8 -f' iscsid: sysfs_init: sysfs_path='/sys' iscsid: in ctldev_open iscsid: created NETLINK_ISCSI socket... iscsid: InitiatorName=iqn.1994-05.com.redhat:2a318909ad8 iscsid: InitiatorName=iqn.1994-05.com.redhat:2a318909ad8 iscsid: InitiatorAlias=computehci01 iscsid: Max file limits 1048576 1048576 iscsid: current time 258465 iscsid: nothing on pend_list, deactivating alarm Any idea? Regards.

2 4

[neutron] Deprecation of the neutronclient CLI code
by Rodolfo Alonso Hernandez 12 Apr '23

12 Apr '23

Hello all: Since this release [1], the python-neutronclient project will no longer provide CLI support. Any Neutron related CLI command should be executed using python-openstackclient (OSC). This deprecation and removal was announced in [2] and was initially planned for Zed release. We have been working during the last releases to finish the migration and provide all the commands in OSC. In case you find any missing command, argument or output, please open a bug in the Neutron site of launchpad and we'll address it. You can also install a previous (Antelope) version of neutronclient if you missed the migration window. The python-neutronclient will still be available to provide the Networking API python bindings [3]. Regards. [1]https://review.opendev.org/c/openstack/python-neutronclient/+/871711 [2] https://docs.openstack.org/python-neutronclient/latest/contributor/transiti… [3]https://docs.openstack.org/python-neutronclient/ocata/usage/library.html

1 0

CYBORG HTTP400 create trait OWNER_CYBORG
by Karl Kloppenborg 12 Apr '23

12 Apr '23

Hi Cyborg Team, Karl from the Openstack-Helm group. I’ve hit a really an issue I cannot seem to overcome, Cyborg is detecting GPU Profiles from a NVIDIA A40, and it goes to create the record but cyborg-conductor errors with ``` [cyborg-conductor-6fc477ff8b-cw5lf cyborg-conductor] 2023-04-12 06:17:37.581 1 INFO cyborg.conductor.manager [None req-600fcdcf-a620-466a-ba6c-cdfbf6bf83e2 - - - - - -] Failed to add device DriverDevice(controlpath_id=DriverControlPathID,created_at=<?>,deployable_list=[DriverDeployable],model='NVIDIA Corporation Device',std_board_info='{"product_id": "2235", "controller": "3D controller"}',stub=False,type='GPU',updated_at=<?>,vendor='10de',vendor_board_info='{"vendor_info": "gpu_vb_info"}'). Reason: Failed to create trait OWNER_CYBORG: HTTP 400: {"errors": [{"status": 400, "title": "Bad Request", "detail": "The server could not comply with the request since it is either malformed or otherwise incorrect.\n\n The trait is invalid. A valid trait must be no longer than 255 characters, start with the prefix \"CUSTOM_\" and use following characters: \"A\"-\"Z\", \"0\"-\"9\" and \"_\" ", "request_id": "req-24634fad-b312-4ad3-9482-09d6df2b26c2"}]} ``` It appears Cyborg is attempting to create the trait OWNER_CYBORG but I cannot seem to get it to stop that, can I please have some assistance from the Cyborg team to overcome this? Kind Regards, -- Karl Kloppenborg, (BCompSc, CNCF-[KCNA, CKA, CKAD], LFCE, CompTIA Linux+ XK0-004) Managing Director, Invention Labs.

1 0

In Person PTG June 2023 Teams List
by Kendall Nelson 11 Apr '23

11 Apr '23

Hello Everyone! The June 2023 Project Teams List is official! Projects + Teams - OpenInfra Foundation Teams - - Diversity & Inclusion Working Group - Environmental Sustainability Working Group - Kata Containers - StarlingX - OpenStack - - Service Teams - - Cinder - Cyborg - Designate - Heat - Ironic - Keystone - Manila - Neutron - Nova - OpenStack-Ansible - Puppet OpenStack - QA - SDK/CLI/AOC - Tacker - Telemetry - Other OpenStack Teams - - Security SIG - Technical Committee - Scientific SIG If your team was planning to meet and isn’t in this list, please contact ptg(a)openinfra.dev IMMEDIATELY. In a couple weeks, I will be contacting moderators to sign up for time via the PTGBot[1] once I have it configured for the teams that are signed up to attend the event. Otherwise, please don't forget to register[2] — prices go up May 5th! Thanks! -Kendall (diablo_rojo) [1] PTGBot Docs: https://github.com/openstack/ptgbot#open-infrastructure-ptg-bot [2] PTG Registration: https://vancouver2023.openinfra.dev/

1 0

[openstack][sharing][kolla ansible]Problems when 1 of 3 controller was be down
by Nguyễn Hữu Khôi 11 Apr '23

11 Apr '23

Hello everyone. I want to summary for who meets problems with Openstack when deploy cluster with 3 controller using Kolla Ansible Scenario: 1 of 3 controller is down 1. Login horizon and use API such as nova, cinder will be very slow fix by: nano: kolla-ansible/ansible/roles/heat/templates/heat.conf.j2 kolla-ansible/ansible/roles/nova/templates/nova.conf.j2 kolla-ansible/ansible/roles/keystone/templates/keystone.conf.j2 kolla-ansible/ansible/roles/neutron/templates/neutron.conf.j2 kolla-ansible/ansible/roles/cinder/templates/cinder.conf.j2 or which service need caches add as below [cache] backend = oslo_cache.memcache_pool enabled = True memcache_servers = {{ kolla_internal_vip_address }}:{{ memcached_port }} memcache_dead_retry = 0.25 memcache_socket_timeout = 900 https://review.opendev.org/c/openstack/kolla-ansible/+/849487 but it is not the end 2. Cannot launch instance or mapping block device(stuck at this step) nano kolla-ansible/ansible/roles/rabbitmq/templates/definitions.json.j2 "policies":[ {"vhost": "/", "name": "ha-all", "pattern": "^(?!(amq\.)|(.*_fanout_)|(reply_)).*", "apply-to": "all", "definition": {"ha-mode":"all"}, "priority":0}{% if project_name == 'outward_rabbitmq' %}, {"vhost": "{{ murano_agent_rabbitmq_vhost }}", "name": "ha-all", "pattern": ".*", "apply-to": "all", "definition": {"ha-mode":"all"}, "priority":0} {% endif %} ] nano /etc/kollla/global.conf [oslo_messaging_rabbit] kombu_reconnect_delay=0.5 https://bugs.launchpad.net/oslo.messaging/+bug/1993149 https://docs.openstack.org/large-scale/journey/configure/rabbitmq.html I used Xena 13.4 and Yoga 14.8.1. Above bugs are critical, but I see that it was not fixed. I am just an operator and I want to share what I encountered for new people who come to Openstack Nguyen Huu Khoi

6 11

Re: StackOver Flow Error: 2103 ERROR oslo.messaging._drivers.impl_rabbit Connection failed: failed to resolve broker hostname
by Dmitriy Rabotyagov 11 Apr '23

11 Apr '23

If you're going to go prod I would for sure use some deployment tool rather then manual deployment, as further operations and scaling up would be way easier. What error do you get with existing hosts record? вт, 11 апр. 2023 г., 14:57 Trey Fleishman <nfleishman(a)student.ccaeagles.org >: > Hello Dmitriy, > > That is what I thought too, in my host file I define the hostname as the > correct IP address, and if I change the "controller" hostname to some > factitious IP I get a different error. I've tried putting the IP instead of > the hostname, but to no avail. I can successfully ping the hostname from > the machine. I would also rather not use a tool as I plan to go production > once I get my two node setup functional. I am not sure if this might be a > configuration issue, or maybe a bug. > Trey Fleishman > Student and IT Intern > (704) 750 - 1740 > *Calvary Chapel Fort Lauderdale* > 2401 W. Cypress Creek Rd., Ft. Lauderdale, FL 33309 > CalvaryFTL.org > > > On Tue, Apr 11, 2023 at 12:54 AM Dmitriy Rabotyagov < > noonedeadpunk(a)gmail.com> wrote: > >> Hi Trey, >> >> Have you defined a record in your /etc/hosts for "controller", as >> according to the error service simply can not resolve this hostname to an >> IP address. >> >> Also I can suggest trying out one of the following OpenStack deployment >> tools: >> https://docs.openstack.org/zed/deploy/index.html >> >> Most of them have All-In-One guides that can spawn an opestack sandbox to >> play with. >> >> вт, 11 апр. 2023 г., 00:21 Trey Fleishman < >> nfleishman(a)student.ccaeagles.org>: >> >>> Hello all, >>> Recently I tried installing Openstack via this >>> <https://docs.openstack.org/install-guide/openstack-services.html> >>> guide. Upon completion of all the steps including the dashboard step, I was >>> greeted with this error inside the nova_schedular.log file, after I tried >>> creating an instance. >>> >>> 2103 ERROR oslo.messaging._drivers.impl_rabbit Connection failed: failed >>> to resolve broker hostname: OSError: failed to resolve broker hostname >>> >>> When I try to create an instance it hangs in the "scheduling" state and >>> throws this error. I have not configured anything in the "oslo.messaging" >>> section of the Nova config file. I believe that under the [default] section >>> this is what's causing the issue, but the password and hostname are fine. >>> >>> transport_url = >>> rabbit://openstack:BnE9LLfCtjdPvpEnZUnPytSyrfUxE8th@controller >>> >>> Thanks! >>> >>

2 1

[keystone] Weekly Meeting Cancelled (11-Apr-2023)
by Dave Wilde 11 Apr '23

11 Apr '23

Hi Folks, Unfortunately I have to cancel the weekly meeting today. Please reach out to me directly via email if you need anything. Due to equipment failure my IRC availability is spotty. Thanks! /Dave

1 0

[Kolla-ansible][Yoga] Cannot live migrate VMs
by wodel youchi 11 Apr '23

11 Apr '23

Hi, I cannot live-migrate any VM I have. All my compute nodes are enabled. I have a few VMs, I am in the testing phase. I tried to live-migrate toward an empty node, then toward a node which already had a VM in it. It is the same problem. In the log files I have : Source node : Source > ==> /var/log/kolla/nova/nova-compute.log <== > 2023-04-09 11:32:49.419 7 INFO nova.compute.resource_tracker > [req-4e91d562-d7ad-4585-a827-3db6277fd81e - - - - -] [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Updating resource usage from > migration f60eb077-8e1c-404a-aca5-64e5609ca02c > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [-] [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Pre live migration failed at > computehci02: oslo_messaging.rpc.client.RemoteError: Remote error: > ClientException Gateway Time-out (HTTP 504) > ['Traceback (most recent call last):\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", > line 165, in _process_incoming\n res = > self.dispatcher.dispatch(message)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", > line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, > args)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", > line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", > line 72, in wrapped\n context, exc, binary)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 227, in __exit__\n self.force_reraise()\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 200, in force_reraise\n raise self.value\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", > line 63, in wrapped\n return f(self, context, *args, **kw)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/utils.py", > line 1439, in decorated_function\n return function(self, context, *args, > **kwargs)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", > line 213, in decorated_function\n kwargs[\'instance\'], e, > sys.exc_info())\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 227, in __exit__\n self.force_reraise()\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 200, in force_reraise\n raise self.value\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", > line 201, in decorated_function\n return function(self, context, *args, > **kwargs)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", > line 8367, in pre_live_migration\n bdm.save()\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 227, in __exit__\n self.force_reraise()\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 200, in force_reraise\n raise self.value\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", > line 8304, in pre_live_migration\n connector=connector, > mountpoint=bdm.device_name)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", > line 395, in wrapper\n res = method(self, ctx, *args, **kwargs)\n', ' > File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", > line 433, in wrapper\n res = method(self, ctx, volume_id, *args, > **kwargs)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", > line 813, in attachment_create\n instance_uuid=instance_id)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 227, in __exit__\n self.force_reraise()\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 200, in force_reraise\n raise self.value\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", > line 801, in attachment_create\n volume_id, _connector, instance_id)\n', > ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/api_versions.py", > line 421, in substitution\n return method.func(obj, *args, **kwargs)\n', > ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/v3/attachments.py", > line 39, in create\n retval = self._create(\'/attachments\', body, > \'attachment\')\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/base.py", > line 306, in _create\n resp, body = self.api.client.post(url, > body=body)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", > line 227, in post\n return self._cs_request(url, \'POST\', **kwargs)\n', > ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", > line 215, in _cs_request\n return self.request(url, method, > **kwargs)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", > line 201, in request\n raise exceptions.from_response(resp, body)\n', > 'cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504)\n']. > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Traceback (most recent call last): > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", > line 8501, in _do_pre_live_migration_from_source > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] block_migration, disk, dest, > migrate_data) > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/rpcapi.py", > line 948, in pre_live_migration > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] **msg_args) > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/client.py", > line 192, in call > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] retry=self.retry, > transport_options=self.transport_options) > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/transport.py", > line 128, in _send > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] > transport_options=transport_options) > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", > line 691, in send > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] > transport_options=transport_options) > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", > line 681, in _send > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] raise result > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] > oslo_messaging.rpc.client.RemoteError: Remote error: ClientException > Gateway Time-out (HTTP 504) > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] ['Traceback (most recent call > last):\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", > line 165, in _process_incoming\n res = > self.dispatcher.dispatch(message)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", > line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, > args)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", > line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", > line 72, in wrapped\n context, exc, binary)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 227, in __exit__\n self.force_reraise()\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 200, in force_reraise\n raise self.value\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", > line 63, in wrapped\n return f(self, context, *args, **kw)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/utils.py", > line 1439, in decorated_function\n return function(self, context, *args, > **kwargs)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", > line 213, in decorated_function\n kwargs[\'instance\'], e, > sys.exc_info())\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 227, in __exit__\n self.force_reraise()\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 200, in force_reraise\n raise self.value\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", > line 201, in decorated_function\n return function(self, context, *args, > **kwargs)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", > line 8367, in pre_live_migration\n bdm.save()\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 227, in __exit__\n self.force_reraise()\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 200, in force_reraise\n raise self.value\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", > line 8304, in pre_live_migration\n connector=connector, > mountpoint=bdm.device_name)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", > line 395, in wrapper\n res = method(self, ctx, *args, **kwargs)\n', ' > File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", > line 433, in wrapper\n res = method(self, ctx, volume_id, *args, > **kwargs)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", > line 813, in attachment_create\n instance_uuid=instance_id)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 227, in __exit__\n self.force_reraise()\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 200, in force_reraise\n raise self.value\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", > line 801, in attachment_create\n volume_id, _connector, instance_id)\n', > ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/api_versions.py", > line 421, in substitution\n return method.func(obj, *args, **kwargs)\n', > ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/v3/attachments.py", > line 39, in create\n retval = self._create(\'/attachments\', body, > \'attachment\')\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/base.py", > line 306, in _create\n resp, body = self.api.client.post(url, > body=body)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", > line 227, in post\n return self._cs_request(url, \'POST\', **kwargs)\n', > ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", > line 215, in _cs_request\n return self.request(url, method, > **kwargs)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", > line 201, in request\n raise exceptions.from_response(resp, body)\n', > 'cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504)\n']. > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] > 2023-04-09 11:33:30.157 7 INFO nova.compute.manager [-] [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Swapping old allocation on > dict_keys(['2857ae62-3519-4b13-b700-858c827dd78d']) held by migration > f60eb077-8e1c-404a-aca5-64e5609ca02c for instance In the destination node : > nova-compute.log > > > > > *2023-04-09 11:32:30.052 7 WARNING os_brick.initiator.connectors.nvmeof > [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 > b0f76b5c6dcb457fa716762bbf954837 - default default] Process execution error > in _get_host_uuid: Unexpected error while running command.Command: blkid > overlay -s UUID -o valueExit code: 2Stdout: ''Stderr: '': > oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while > running command.* > > 2023-04-09 11:33:05.354 7 INFO nova.compute.resource_tracker > [req-9d480442-87f7-4d4e-a866-c46d645ac7f2 - - - - -] [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Updating resource usage from > migration f60eb077-8e1c-404a-aca5-64e5609ca02c > 2023-04-09 11:33:05.392 7 WARNING nova.compute.resource_tracker > [req-9d480442-87f7-4d4e-a866-c46d645ac7f2 - - - - -] Instance > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7 has been moved to another host > computehci01(computehci01). There are allocations remaining against the > source host that might need to be removed: {'resources': {'VCPU': 1, > 'MEMORY_MB': 2048}}. > 2023-04-09 11:33:30.086 7 ERROR nova.volume.cinder > [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 > b0f76b5c6dcb457fa716762bbf954837 - default default] [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Create attachment failed for volume > 3d7bd0af-5e4c-48fc-a027-f6c9dd6280bb. Error: Gateway Time-out (HTTP 504) > Code: 504: cinderclient.exceptions.ClientException: Gateway Time-out (HTTP > 504) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server > [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 > b0f76b5c6dcb457fa716762bbf954837 - default default] Exception during > message handling: cinderclient.exceptions.ClientException: Gateway Time-out > (HTTP 504) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server Traceback (most > recent call last): > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", > line 165, in _process_incoming > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server res = > self.dispatcher.dispatch(message) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", > line 309, in dispatch > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return > self._do_dispatch(endpoint, method, ctxt, args) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", > line 229, in _do_dispatch > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server result = > func(ctxt, **new_args) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", > line 72, in wrapped > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server context, > exc, binary) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 227, in __exit__ > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server > self.force_reraise() > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 200, in force_reraise > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise > self.value > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", > line 63, in wrapped > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return > f(self, context, *args, **kw) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/utils.py", > line 1439, in decorated_function > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return > function(self, context, *args, **kwargs) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", > line 213, in decorated_function > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server > kwargs['instance'], e, sys.exc_info()) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 227, in __exit__ > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server > self.force_reraise() > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 200, in force_reraise > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise > self.value > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", > line 201, in decorated_function > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return > function(self, context, *args, **kwargs) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", > line 8367, in pre_live_migration > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server bdm.save() > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 227, in __exit__ > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server > self.force_reraise() > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 200, in force_reraise > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise > self.value > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", > line 8304, in pre_live_migration > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server > connector=connector, mountpoint=bdm.device_name) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", > line 395, in wrapper > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server res = > method(self, ctx, *args, **kwargs) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", > line 433, in wrapper > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server res = > method(self, ctx, volume_id, *args, **kwargs) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", > line 813, in attachment_create > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server > instance_uuid=instance_id) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 227, in __exit__ > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server > self.force_reraise() > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 200, in force_reraise > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise > self.value > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", > line 801, in attachment_create > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server volume_id, > _connector, instance_id) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/api_versions.py", > line 421, in substitution > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return > method.func(obj, *args, **kwargs) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/v3/attachments.py", > line 39, in create > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server retval = > self._create('/attachments', body, 'attachment') > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/base.py", > line 306, in _create > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server resp, body = > self.api.client.post(url, body=body) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", > line 227, in post > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return > self._cs_request(url, 'POST', **kwargs) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", > line 215, in _cs_request > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return > self.request(url, method, **kwargs) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", > line 201, in request > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise > exceptions.from_response(resp, body) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server > cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server > 2023-04-09 11:33:30.238 7 INFO nova.virt.block_device > [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 > b0f76b5c6dcb457fa716762bbf954837 - default default] [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Attempting to driver detach volume > 3d7bd0af-5e4c-48fc-a027-f6c9dd6280bb from mountpoint /dev/vda > 2023-04-09 11:33:30.239 7 WARNING nova.virt.block_device > [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 > b0f76b5c6dcb457fa716762bbf954837 - default default] [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Detaching volume from unknown instance > 2023-04-09 11:33:30.240 7 WARNING nova.virt.libvirt.driver > [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 > b0f76b5c6dcb457fa716762bbf954837 - default default] [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] During detach_volume, instance > disappeared.: nova.exception.InstanceNotFound: Instance > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7 could not be found. > 2023-04-09 11:33:30.678 7 INFO nova.virt.libvirt.driver [-] [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Instance destroyed successfully. > 2023-04-09 11:33:30.682 7 INFO os_vif > [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 > b0f76b5c6dcb457fa716762bbf954837 - default default] Successfully unplugged > vif > VIFBridge(active=True,address=fa:16:3e:9b:92:46,bridge_name='qbr46651303-65',has_traffic_filtering=True,id=46651303-65e1-4809-95cc-7c2d97acf16b,network=Network(532470c1-dd2f-4505-8557-ba24e9d4062b),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap46651303-65') > All I can see is *b0f76b5c6dcb457fa716762bbf954837 - default default] Process execution error in _get_host_uuid: Unexpected error while running command.Command: blkid overlay -s UUID -o valueExit code: 2Stdout: ''Stderr: '': oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.* Searching the web I didn't find anything useful about this error, and I don't know if it's the origin of the problem. My default storage is ceph rbd, I have an HCI deployment, and all my VMs are for now created on Ceph only. But I have configured two other cinder backends, NFS and LVM, but they are node used for now. Thanks in advance. Regards.

3 4

[neutron] bug deputy report for week of 2023-04-03
by Bence Romsics 11 Apr '23

11 Apr '23

Hi All, We had the following new bugs reported last week: Critical: * https://bugs.launchpad.net/neutron/+bug/2015065 [tempest] "test_security_group_rules_create" unstable in "neutron-ovs-grenade-dvr-multinode" unassigned High: * https://bugs.launchpad.net/neutron/+bug/2015364 [skip-level] OVN tests constantly failing unassigned * https://bugs.launchpad.net/neutron/+bug/2015376 [OVN] min-BW rule, defined in the LSP.options field, must be accompanied with a max-BW value unassigned * https://bugs.launchpad.net/neutron/+bug/2015728 ovs/ovn source(with master branch) deployments broken workarounds proposed/merged by Yatin: https://review.opendev.org/c/openstack/neutron/+/879938 https://review.opendev.org/c/openstack/ovn-octavia-provider/+/879939 * https://bugs.launchpad.net/neutron/+bug/2015090 neutron.agent.dhcp.agent TypeError: 'bool' object is not subscriptable Technically unassigned, but I believe Sahid may be working on it. Medium: * https://bugs.launchpad.net/neutron/+bug/2015388 Deleting network does not remove network namespace fix proposed by Brian: https://review.opendev.org/c/openstack/neutron/+/880006 * https://bugs.launchpad.net/neutron/+bug/2015377 [ovn] If dhcp port is deleted from neutron, it is never recreated fix proposed by Brian: https://review.opendev.org/c/openstack/neutron/+/879913 RFE: * https://bugs.launchpad.net/neutron/+bug/2015471 [RFE] Add ERSPAN for tap-as-a-service with OVS and OVN Triage still in progress: * https://bugs.launchpad.net/neutron/+bug/2015449 Remote security groups don't allow traffic from floating IPs This issue may be undocumented behavior, the following question could be discussed on the weekly meeting: https://meetings.opendev.org/irclogs/%23openstack-neutron/%23openstack-neut… * https://bugs.launchpad.net/neutron/+bug/2015275 Trunk port update_subport_bindings RPC call not processed by RPC workers Cheers, Bence (rubasov)

1 0

[openstackclient] openstack cli broken after update to Wallaby
by Eugen Block 11 Apr '23

11 Apr '23

Hi *, today I upgraded my virtual test environment from V to W when (Ubuntu 20.04) all of a sudden cli commands didn't work anymore with this stack trace: ---snip--- root@control01:~# openstack network agent list Traceback (most recent call last): File "/usr/bin/openstack", line 6, in <module> from openstackclient.shell import main File "/usr/lib/python3/dist-packages/openstackclient/shell.py", line 23, in <module> from osc_lib import shell File "/usr/lib/python3/dist-packages/osc_lib/shell.py", line 24, in <module> from cliff import app File "/usr/lib/python3/dist-packages/cliff/app.py", line 22, in <module> import cmd2 File "/usr/lib/python3/dist-packages/cmd2.py", line 585, in <module> _ = pyperclip.paste() File "/usr/lib/python3/dist-packages/pyperclip/__init__.py", line 667, in lazy_load_stub_paste copy, paste = determine_clipboard() File "/usr/lib/python3/dist-packages/pyperclip/__init__.py", line 558, in determine_clipboard return init_gi_clipboard() File "/usr/lib/python3/dist-packages/pyperclip/__init__.py", line 167, in init_gi_clipboard gi.require_version('Gtk', '3.0') File "/usr/lib/python3/dist-packages/gi/__init__.py", line 129, in require_version raise ValueError('Namespace %s not available' % namespace) ValueError: Namespace Gtk not available ---snip--- I found this bug [1] describing the same issue but there has been no progress. I posted my comments there as well. I found one way to get the openstack shell to work by installing libgtk-3-dev (I found a hint in a search engine). Apparently, python3-cmd2 requires python3-pyperclip which requires python3-gi and so on. Is this really the desired way? I didn't notice anything in the release notes (maybe I missed it). When comparing to a different environment (Victoria on baremetal) I see that libgtk-3 is installed there (not -dev though), but even with libgtk-3-0 the error message was still present. So the question is, which dependencies are missing where? It's not really obvious to me. Could this already be fixed in Xena? If it is fixed there I could do the double upgrade, of course, especially since Wallaby is already under extended maintenance. Any comments are appreciated. Thanks, Eugen [1] https://bugs.launchpad.net/ubuntu/+source/python-openstackclient/+bug/19456…

4 9

[ironic][nova][neutron][oslo][sdk][swift][charm] netifaces has been archived
by Takashi Kajinami 11 Apr '23

11 Apr '23

Hello, I recently (actually, this morning) noticed the netifaces library[1] was archived a while ago, because the maintenor could no longer continue Open Source contributions, unfortunately[2]. [1] https://github.com/al45tair/netifaces [2] https://github.com/al45tair/netifaces/issues/78 I quickly searched the repositories and found some projects which have netifaces in their requirements.txt . We might need to look for alternatives (or offer to maintain the project, which is always the best) before we face any problems caused by that unmaintained library. Thank you, Takashi Kajinami

4 4

StackOver Flow Error: 2103 ERROR oslo.messaging._drivers.impl_rabbit Connection failed: failed to resolve broker hostname
by Trey Fleishman 10 Apr '23

10 Apr '23

Hello all, Recently I tried installing Openstack via this <https://docs.openstack.org/install-guide/openstack-services.html> guide. Upon completion of all the steps including the dashboard step, I was greeted with this error inside the nova_schedular.log file, after I tried creating an instance. 2103 ERROR oslo.messaging._drivers.impl_rabbit Connection failed: failed to resolve broker hostname: OSError: failed to resolve broker hostname When I try to create an instance it hangs in the "scheduling" state and throws this error. I have not configured anything in the "oslo.messaging" section of the Nova config file. I believe that under the [default] section this is what's causing the issue, but the password and hostname are fine. transport_url = rabbit://openstack:BnE9LLfCtjdPvpEnZUnPytSyrfUxE8th@controller Thanks!

2 1

[tc] Technical Committee next weekly meeting on 2023 Apr 11 at 1800 UTC
by Nikolla, Kristi 10 Apr '23

10 Apr '23

Hi all, This is a reminder that the next weekly Technical Committee meeting is to be held tomorrow (April 11) at 1800 UTC on #openstack-tc on OFTC IRC Note: This is the first meeting with the new meeting time, please update your calendars accordingly moving forward. Items can be proposed by editing the wiki page at https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting At the end of today I will finalize the agenda and send out another email. Thank you, Kristi Nikolla

1 1

[kolla ansible] SRIOV compute deployment question
by Satish Patel 08 Apr '23

08 Apr '23

Folks, I have deployed SRIOV compute but i found it deployed sriov-agent and openvswitch-agent together. If I am not mistaken then we don't need openvswitch agent on sriov computes nodes so how do i tell kolla to not deploy openvswitch-agent on sriov compute nodes? Here is the output on the compute node. root@kolla-comp-sriov-1-1:~# docker ps --filter name=agent CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 92af9870ee5b docker-reg:4000/kolla/neutron-sriov-agent:zed "dumb-init --single-…" 32 hours ago Up 31 hours (healthy) neutron_sriov_agent 14aea45e8b47 docker-reg:4000/kolla/neutron-openvswitch-agent:zed "dumb-init --single-…" 32 hours ago Up 3 seconds (health: starting) neutron_openvswitch_agent Configuration: (venv-kolla) root@kolla-infra-1-1:/var/lib/nova# cat /etc/kolla/config/neutron/kolla-comp-sriov-1-1/sriov_agent.ini [sriov_nic] physical_device_mappings = physnet1:eno50 (venv-kolla) root@kolla-infra-1-1:/var/lib/nova# cat /etc/kolla/config/neutron/kolla-comp-sriov-1-1/ml2_conf.ini [ml2] mechanism_drivers = sriovnicswitch [ml2_type_vlan] network_vlan_ranges = physnet1:30:50

1 0

[all] Hold your rechecks
by Dmitriy Rabotyagov 07 Apr '23

07 Apr '23

Hi everyone, Due to the recent update of gnupg2 package for CentOS Stream 9 to the version 2.3.3-3, a lot of repositories, including BaseOS and NFV, are failing to pass GPG check [1]. This affects all latest images, including container and cloud ones. At the moment, images used in Zuul are not affected, but this might change with the next periodic rebuild unless the related BZ report is fixed. Explicit execution of `rpm --import` can be used as a temporary workaround. [1] https://bugzilla.redhat.com/show_bug.cgi?id=2184640

2 3

[kolla-ansible]ask about multi region deploymet
by Nguyễn Hữu Khôi 07 Apr '23

07 Apr '23

Hello. I have read about kolla ansible multi regions. Can i ask that do we need 2 deployment host? And can we use 3 controller each region? I searched on Internet, I see that controller just sit on region one only? Please advise. Thank you.

1 0

[outreachy][cinder] Unable to generate api-ref samples for snapshot transfer
by Toheeb Oyekola 07 Apr '23

07 Apr '23

Hi, I am trying to generate api-ref doc from template by running "tox -e snaphot_transfer" but I keep getting the same error pasted in the link https://paste.openstack.org/show/bFPbWKQAKRbIds4R0wWn and I have installed the missing dependency as shown below, anyone have an Idea what the issue could have been? Thanks for your time. [image: image.png]

2 1

答复: [ptl] Need PTL volunteer for OpenStack Sahara
by Jerry Zhou (周鑫勇) 06 Apr '23

06 Apr '23

Hi Gmann, I can lead this project. -----邮件原件----- 发件人: Ghanshyam Mann <gmann(a)ghanshyammann.com> 发送时间: 2023年3月23日 2:35 收件人: Qiujunting <qiujunting(a)inspur.com>; openstack-discuss <openstack-discuss(a)lists.openstack.org> 主题: [ptl] Need PTL volunteer for OpenStack Sahara Hi Qiu, I am reaching out to you as you were PTL for OpenStack Sahara project in the last cycle. There is no PTL candidate for the next cycle (2023.2), and it is on the leaderless project list. Please check if you or anyone you know would like to lead this project. - https://etherpad.opendev.org/p/2023.2-leaderless Also, if anyone else would like to help leading this project, this is time to let TC knows. -gmann

1 0

[kolla-ansible] updating procedure
by wodel youchi 06 Apr '23

06 Apr '23

Hi, This is what documentation says about updating kolla-ansible : "Inside a series, it is usually sufficient to just update the kolla-ansible package, rebuild (if needed) and pull the images, and run kolla-ansible deploy again. Please follow release notes to check if there are any issues to be aware of." So to summarize, the procedure is like this : 1 - update kolla-ansible 2 - Download the latest containers of the used version 3 - In my case publish them in the local registry 4 - launch kolla-ansible deploy My question : Can I do the update directly , especially on the compute nodes, or do I have to empty some of them by moving VMs and execute the update on the empty nodes only, then proceed with the same logic with the rest of the nodes? Regards.

3 3

Re: multi-region deployment within datacenter scenario
by Satish Patel 06 Apr '23

06 Apr '23

Hi Piotr, Basically we use a vlan base tenant so the network node has very less usage. All my vms talk directly to the physical network and go out via the edge router. Our VM churn rate is also low because it's a private cloud. I am thinking of creating 3x API nodes and 3x RabbitMQ/DB nodes (total 6 nodes control plane and hope it works fine). One big advantage of having multiple pods (deployments) so during upgrade or any issue we have backup to work. On Thu, Apr 6, 2023 at 3:42 PM Piotr Misiak <piotrmisiak1984(a)gmail.com> wrote: > Hi Satish, > > Nova is well designed to handle hundreds of nodes. I would take a look at > Neutron instead, because it doesn’t scale very well. It depends on how your > networking configuration and use cases look like. > > BR, > Piotr > > On Wed, 5 Apr 2023 at 17:46 Satish Patel <satish.txt(a)gmail.com> wrote: > >> Folks, >> >> This is a very broad question and could have many answers but still >> wanted to ask to see what people are thinking and doing. >> >> We have DC with multiple racks and around 600 nodes so we are planning to >> create 3 private openstack and each with 200 nodes. In other data centers >> we have multiple 200 nodes openstack clusters and life is good. In the new >> datacenter I am thinking of deploying the same way openstack but shared >> keystone between multiple openstack environments to have a single place for >> identification and I am going to call it 3 regions even it's in a single >> DC. Hope shared keystone is helpful/easier for doing terrafrom work and >> selecting region easily. >> >> Other option I was thinking about to deploy cellv2 technology and put all >> 600 nodes in single cloud but always worried for operation point of view. >> Now sure how many people running cellv2 successfully without pain. (Except >> CERN) >> >> If people running multi-region openstack cloud then please share some >> thought and idea to make life easier. Not looking for best solution but >> just clue to get idea to deploy and manage multiple openstack. >> >> Thanks in advance. >> S >> >> >>

1 0

multi-region deployment within datacenter scenario
by Satish Patel 06 Apr '23

06 Apr '23

Folks, This is a very broad question and could have many answers but still wanted to ask to see what people are thinking and doing. We have DC with multiple racks and around 600 nodes so we are planning to create 3 private openstack and each with 200 nodes. In other data centers we have multiple 200 nodes openstack clusters and life is good. In the new datacenter I am thinking of deploying the same way openstack but shared keystone between multiple openstack environments to have a single place for identification and I am going to call it 3 regions even it's in a single DC. Hope shared keystone is helpful/easier for doing terrafrom work and selecting region easily. Other option I was thinking about to deploy cellv2 technology and put all 600 nodes in single cloud but always worried for operation point of view. Now sure how many people running cellv2 successfully without pain. (Except CERN) If people running multi-region openstack cloud then please share some thought and idea to make life easier. Not looking for best solution but just clue to get idea to deploy and manage multiple openstack. Thanks in advance. S

3 7

[interop][ptg] Interop Bobcat vPTG summary
by Martin Kopec 06 Apr '23

06 Apr '23

Hi everyone, here is a summary of the discussions. The full agenda can be found here [1]. == Current pain points and challenges of interop testing == The current guidelines don't contain complex enough tests - apart from api tests we should also have more functional tests which mock users' interactions with the cloud - we should test as a real user would use the cloud. People could fake the refstack results - after a discussion we came to an agreement that those people would always find a way, there is no need for coming up with sophisticated ways to prevent it. The point of Interop is to provide a check for vendors/users to see where they stand with their product based on the agreed standards. Use of Interop's tooling is complicated - there have been reports that users struggle with refstack-client and tempest. There is an ongoing effort to create a refstack container which would contain all necessary tools and could be run with just one command. Passing the Interop's guidelines means interoperability on which level? Passing the guidelines should mean being in compliance with the beforehand community agreed upon standards. Getting vendors running the interop tests more often - we only came up with the strategies which are already being used, such as expirations and guideline versioning. == Future strategy == The future strategy could be summarized in 3 points: * discuss the possibility for using sdk-based and tobiko testing * better describe to users the expectations they should have for clouds which assert interoperability * look into expiration of older standards and require that compliant providers indicate to users which version of the standard they last tested/passed [1] https://etherpad.opendev.org/p/bobcat-ptg-interop Regards, -- Martin Kopec Principal Software Quality Engineer Red Hat EMEA IM: kopecmartin

2 1

[tacker][ptg] Tacker PTG Summary
by Yasufumi Ogawa 06 Apr '23

06 Apr '23

Hi team, Thank you for joining our three days vPTG sessions for Tacker bobcat. It was three days and we had about 20 attendees each day. You can find all the result of discussions in [1] and cross project session with Keystone, Ironic and Barbican for supporting external authz for OAuth2.0 [2]. Here is a summary of our vPTG. - We shared several issues of functional tests have been suffered us for unexpected failures through the latest cycles and agree to revise the tests. Current target include refactoring messed directory structure and useless logs, reducing too much timeouts or redefining test scenarios. - For docs, there is no instruction for supported versions of k8s, helm or prometheus introduced in Tacker in the latest releases. There is also something missing instructions or too old contents. We've agreed to make a plan to update the docs and complete TODOs in bobcat. - Enable tacker to get error messages of management driver as a requirement from operators. - Share the current status of secure RBAC support in Tacker and remained tasks mainly for implementing unit and functional tests and documentation. - We agreed to introduct a new feature for supporting external key manager for authn credentials with encryption for security and custom resources for the latest k8s cluster APIs. - Discussed a requirement for supporting terraform infra-driver for some usecases deloying telco systems on cloud-based infra, and roughly agreed to include the feature into Tacker although we need to have some more surveys for implementation details. - We've had several proposals for improving our dev scheme such as introducing type annotation, plantUML for docs or so. - Shared the current status of Pypi maintainer cleanup. [1] https://etherpad.opendev.org/p/tacker-bobcat-ptg [2] https://etherpad.opendev.org/p/bobcat-ptg-ext_oauth2_server

1 0

[tc] May 2023 OpenInfra Board Sync
by Jeremy Stanley 06 Apr '23

06 Apr '23

The Open Infrastructure Foundation Board of Directors is endeavoring to engage in regular check-ins with official OpenInfra projects. The goal is for a loosely structured discussion one-hour in length, involving members of the board and the OpenStack TC, along with other interested community members. This is not intended to be a formal presentation, and no materials need to be prepared in advance. I've started an Etherpad where participants can brainstorm potential topics of conversation, time-permitting: https://etherpad.opendev.org/p/2023-05-board-openstack-sync Immediately following the February 8 discussion, I proposed to schedule the next call for 20:00 UTC on Wednesday, May 10 and heard no objections, so I've attached a calendar file which can serve as a convenient schedule hold for this, in case anyone needs it. -- Jeremy Stanley

1 0

[nova][cinder] why volumes can only be attached and dettached for running instances
by kaqiu pi 06 Apr '23

06 Apr '23

hello, I can only attached and dettached volumes for running instances in openstack. I would like to ask the reason for this design. And is there any danger if this restriction is turned on through custom modification in a private scene? Thank you so much. Yours

3 2

(no subject)
by Richard Buregeya 06 Apr '23

06 Apr '23

Hello Team, Hope you are doing well. Any one with an idea if there is a modification in COA exam, I heard it is now a multiple question exams instead of practical exam, Anyone with current information, kindly update me. Thanks a lot! Regards

3 6

[nova][cinder] future of rebuild without reimaging
by Bence Romsics 06 Apr '23

06 Apr '23

Hi All! We have users who use 'rebuild' on volume booted servers before nova microversion 2.93, relying on the behavior that it keeps the volume as is. And they would like to keep doing this even after the openstack distro moves to a(n at least) zed base (sometime in the future). As a naive user, it seems to me both behaviors make sense. I can easily imagine use cases for rebuild with and without reimaging. However since the implementation of https://specs.openstack.org/openstack/nova-specs/specs/zed/implemented/volu… rebuild without reimaging is only possible using an old microversion (<2.93). With that change merged, rebuild without reimaging seems to be a somewhat less than fully supported feature. A few examples of what I mean by that: First, there's this warning: https://opendev.org/openstack/python-openstackclient/src/commit/5eb89e4ca1c… In which it is unclear to me what exactly will become an error in a future release. Rebuild with a different image? Or any rebuild with microversion <2.93? Then old nova microversions may get dropped. Though what I heard from nova folks, this is unlikely to happen. Then there are a few hypothetical situations like: a) Rebuild gets a new api feature (in a new microversion) which can never be combined with the do-not-reimage behavior. b) Rebuild may have a bug, whose fix requires a microversion bump. This again can never be combined with the old behavior. What do you think, are these concerns purely theoretical or real? If we would like to keep having rebuild without reimaging, can we rely on the old microversion indefinitely? Alternatively shall we propose and implement a nova spec to explicitly expose the choice in the rebuild api (just to express the idea: osc server rebuild --reimage|--no-reimage)? If the topic is worth further discussion beyond the ML, I can also bring it to the nova ptg. Thanks in advance, Bence Romsics (rubasov) ps: I'll be afk for a few days, but I'll follow up next Tuesday.

7 16

[qa][ptg] QA Bobcat vPTG summary
by Martin Kopec 06 Apr '23

06 Apr '23

Hi everyone, here is a summary of the discussions. The full agenda can be found here [1]. == Antelope Retrospective == Good things: kudos to all who are still around and help with reviews, patches, bug fixes Bad things: unstable gate due to tox update, timeouts, job flakiness (mainly multinode job) == uWSGI is in the maintenance mode == This doesn't require an immediate action as it's still in the maintenance mode = any bugs should be fixed and until we need new features we don't have to rush into replacing it. We should have it on the watchlist though and potentially test some alternatives - I've put this as one of the priority items for this cycle [2]. == FIPS interop testing == We had a FIPS goal since the last PTG [3]. The patches [4] and [5] were to have FIPS on ubuntu and these [6] and [7] are to add FIPS on a centos job. The next goal is to have a FIPS job on Rocky linux as it's more stable than centos. == Retiring Patrole == We are going to retire Patrole project. 2 main facts that speak in favor of retiring: Fact 1: plugins have enough RBAC tests (and are writing more to test SRBAC) - therefore patrole is not needed, the testing coverage got by the plugins is enough. Fact 2: patrole gates have been broken for a long time and no one was affected by that. == Additional PyPi maintainer cleanup == Only 2 projects from QA left with additional maintainers: * openstack/os-performance-tools - additional maintainer SpamapS (Clint Byrum) * openstack/patrole - additional maintainer DavidPurcell If you know any of them, please, tell them to reach out to me. I've sent emails to them, but maybe the email addresses I found aren't checked regularly anymore. == Tempest/Devstack gate stability == Several precautions have been done such as reduce mysql memory [8] or reduce the number of tests executed per job, increase of a timeout etc [9]. However, those precautions are mainly workarounds. The important note is that the instability was only highlighted by our testing, not caused by it. Many problems are inefficiencies in openstack software, e.g.: * openstack-client issues a new token for every operation * python startup time - entrypoint lookup * privesp processes in aggregate use more memory than nova (every service runs their own copy) * inefficient ansible code, etc Any volunteers who would like to first analyse the situation, find which of those inefficiencies are the biggest culprits and how big effect they have on the system? [1] https://etherpad.opendev.org/p/qa-bobcat-ptg [2] https://etherpad.opendev.org/p/qa-bobcat-priority [3] https://etherpad.opendev.org/p/antelope-ptg-interop [4] https://review.opendev.org/c/zuul/zuul-jobs/+/866881 [5] https://review.opendev.org/c/zuul/zuul-jobs/+/873893 [6] https://review.opendev.org/c/openstack/devstack/+/871606 [7] https://review.opendev.org/c/openstack/tempest/+/873697 [8] https://review.opendev.org/c/openstack/devstack/+/873646 [9] https://review.opendev.org/q/topic:bug%252F2004780 Regards, -- Martin Kopec Principal Software Quality Engineer Red Hat EMEA IM: kopecmartin

1 0

[cinder] 2023.2 Bobcat PTG Summary
by Rajat Dhasmana 06 Apr '23

06 Apr '23

Hello, Please find the summary of 2023.2 Bobcat PTG in the wiki page[1]. It includes recordings, summary and action items for each topic. More details can be found in the PTG etherpad[2]. I would recommend everyone to please look for the action items assigned to you to plan it according to the cycle deadlines. [1] https://wiki.openstack.org/wiki/CinderBobcatPTGSummary [2] https://etherpad.opendev.org/p/bobcat-ptg-cinder Thanks Rajat Dhasmana

1 0

[cinder]multi backend SAN and Ceph
by Nguyễn Hữu Khôi 06 Apr '23

06 Apr '23

Hello. I have a question that could we use both SAN and Ceph for multi backend? If yes can we change volume type from SAN to Ceph and otherwise? Thanks.

3 3

[Glance] Weekly Meeting Cancelled
by Pranali Deore 06 Apr '23

06 Apr '23

Hi All, Canceling today's glance weekly meeting, I'm on sick leave today. See you next week ! Thanks, Pranali D

1 0

[cinder]cannot lauch instance with cinder volume
by Nguyễn Hữu Khôi 05 Apr '23

05 Apr '23

Hello guys. I am using kolla-ansible yoga to deploy an openstack cluster, but when I launch an instance with volume, it has errors. but I can create volume first and boot from it. I am using Powerstore 5000Tb. Is it a bug? 672 ERROR oslo_messaging.rpc.server [req-1a822ccd-1d3a-4f9a-a453-22d04f07f816 47cefefdadbe4bb584d8cf983de71eb5 708d7af8178143aab263d3942a124ec8 - - -] Exception during message handling: cinder.exception.ImageCopyFailure: Failed to copy image to volume: Bad or unexpected response from the storage volume backend API: Unable to fetch connection information from backend: Bad or unexpected response from the storage volume backend API: Failed to create PowerStore host . 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/volume/driver.py", line 1045, in _attach_volume 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server conn = self.initialize_connection(volume, properties) 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/volume/drivers/dell_emc/powerstore/driver.py", line 153, in initialize_connection 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server return self.adapter.initialize_connection(volume, connector, **kwargs) 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/volume/drivers/dell_emc/powerstore/adapter.py", line 272, in initialize_connection 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server connection_properties = self._connect_volume(volume, connector) 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/volume/drivers/dell_emc/powerstore/adapter.py", line 613, in _connect_volume 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server chap_credentials, volume_lun = self._create_host_and_attach( 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/volume/drivers/dell_emc/powerstore/adapter.py", line 602, in _create_host_and_attach 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server host, chap_credentials = self._create_host_if_not_exist(connector) 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/decorator.py", line 232, in fun 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/coordination.py", line 195, in _synchronized 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server return f(*a, **k) 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/volume/drivers/dell_emc/powerstore/adapter.py", line 446, in _create_host_if_not_exist 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server host = self.client.create_host(host_name, ports) 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/volume/drivers/dell_emc/powerstore/client.py", line 302, in create_host 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server raise exception.VolumeBackendAPIException(data=msg) 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server cinder.exception.VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Failed to create PowerStore host . Nguyen Huu Khoi

1 1

[kolla-ansible][yoga][neutron][neutron-dhcp]ERRORS.... Need to understand
by Franck VEDEL 05 Apr '23

05 Apr '23

Hello. I come back to you to try to understand my problem. My installation is based on kolla-ansible, Yoga, Centos-Stream. For some time, without understanding why, neutron_dhcp no longer works on external networks and the networks of my students' projects. This generates many error messages, which are difficult to understand. Could this be a bug in this version? Before, it worked very well, but I have the impression that when the number of network increases (i’m not sure), from a moment, dhcp does not work anymore. What is the way to find the problem? Here are examples of errors that won't stop even if no one creates more instances of them 2023-04-04 20:36:52.870 7 ERROR neutron.agent.dhcp.agent device, namespace, scope, broadcast) 2023-04-04 20:36:52.870 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 272, in _wrap 2023-04-04 20:36:52.870 7 ERROR neutron.agent.dhcp.agent r_call_timeout) 2023-04-04 20:36:52.870 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 215, in remote_call 2023-04-04 20:36:52.870 7 ERROR neutron.agent.dhcp.agent raise exc_type(*result[2]) 2023-04-04 20:36:52.870 7 ERROR neutron.agent.dhcp.agent neutron.privileged.agent.linux.ip_lib.InvalidArgument: Invalid parameter/value used on interface tap73cd30e2-35, namespace qdhcp-395f1edb-de6d-4dd8-b31b-69a53780c012. 2023-04-04 20:36:52.870 7 ERROR neutron.agent.dhcp.agent 2023-04-04 20:36:52.871 7 INFO neutron.agent.dhcp.agent [-] Finished network 395f1edb-de6d-4dd8-b31b-69a53780c012 dhcp configuration 2023-04-04 20:36:52.872 7 INFO neutron.agent.dhcp.agent [-] Starting network 8db375e0-28d2-4379-9330-397dc9b4f2b1 dhcp configuration 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent [-] Unable to enable dhcp for 8db375e0-28d2-4379-9330-397dc9b4f2b1.: neutron.privileged.agent.linux.ip_lib.InvalidArgument: Invalid parameter/value used on interface tap1c220d25-b9, namespace qdhcp-8db375e0-28d2-4379-9330-397dc9b4f2b1. 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent Traceback (most recent call last): 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/dhcp/agent.py", line 218, in call_driver 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent rv = getattr(driver, action)(**action_kwargs) 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/dhcp.py", line 274, in enable 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent common_utils.wait_until_true(self._enable, timeout=300) 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/common/utils.py", line 717, in wait_until_true 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent while not predicate(): 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/dhcp.py", line 286, in _enable 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent interface_name = self.device_manager.setup(self.network) 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/dhcp.py", line 1779, in setup 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent namespace=network.namespace) 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/interface.py", line 152, in init_l3 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent device.addr.add(ip_cidr) 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 537, in add 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent add_broadcast) 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 823, in add_ip_address 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent device, namespace, scope, broadcast) 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 272, in _wrap 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent r_call_timeout) 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 215, in remote_call 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent raise exc_type(*result[2]) 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent neutron.privileged.agent.linux.ip_lib.InvalidArgument: Invalid parameter/value used on interface tap1c220d25-b9, namespace qdhcp-8db375e0-28d2-4379-9330-397dc9b4f2b1. 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent Thanks a lot Franck

2 4

[ironic][ptg] 2023.2 Work Items
by Jay Faulkner 05 Apr '23

05 Apr '23

Hey all, As usual, I documented items we talked about working on in Bobcat in a spec. Please go have a look and review it: https://review.opendev.org/c/openstack/ironic-specs/+/879381 This will hopefully be published very soon at https://specs.openstack.org/openstack/ironic-specs/#themes as our 2023.2 work items. Thanks everyone for participating in our vPTG this cycle! -- Jay Faulkner Ironic PTL

1 0

[tripleo] hold rechecks content provider gate blocker
by Marios Andreou 05 Apr '23

05 Apr '23

Hello tripleo we have a gate blocker at [1] hitting the centos-9-content-provider jobs. The tripleo CI team is tracking this - please hold rechecks until it we have a resolution or workaround regards, marios [1] https://bugs.launchpad.net/tripleo/+bug/2015309

1 1

[kolla] weekly meeting cancelled today
by Michał Nasiadka 05 Apr '23

05 Apr '23

Hello Koalas, Weekly meeting cancelled today. Please note that our weekly meeting time for next week and onwards has changed to 13:00 UTC due to DST changes. Best regards, Michal -- Michał Nasiadka mnasiadka(a)gmail.com

1 0

[neutron][OpenStack-ansible] Performance issues with trunk ports
by John Bartelme 05 Apr '23

05 Apr '23

Hello, I'm currently experiencing some pretty severe performance issues with my openstack-ansible deployed cluster(yoga) while deploying trunk ports and I'm looking for some help determining what might be the cause of this poor performance. In my simplest case I'm deploying 2 servers each with one trunk port each. The first trunk has 2 subports and the second 6 subports. Both servers also have 3 other regular ports. When deploying the first trunk port its subports are often provisioned quickly and the second trunk port takes anywhere from 30 seconds to 18 minutes. This happens even when I isolate neutron-server to a single physical machine with 44(88 threads) and 256GB ram. Further diagnosis has shown me some things i didn't quite understand. My deployment with OpenStack-ansible deploys neutron-server with 16 uWSGI processes and neutron-rpc-server with 16 rpc workers. However the way that the trunk RPC server is implemented it is only run on the parent RPC thread and instead runs in all of the uWSGI processes as well. This means that most of my trunk RPC calls are being handled by the uWSGI instead of the RPC workers. When the parent RPC thread handles the trunk port creation calls I constantly see creation times of 1-1.5 seconds. I've isolated it so that this thread does all of the trunk RPC calls and this works quite well but this doesn't seem ideal. What could be causing such poor performance in the uWSGI side of the house? I'm having a really hard time getting a good feeling for what might be slowing it down so much. I'm wondering if it could be green thread preemption but I really don't know. I've tried setting 'enable-threads' false for uWSGI but I don't think that is improving performance. Putting the profiled decorator on update_subport_bindings shows different places taking longer every time, but in general a lot of time(tottime, i.e. not subfunction time) spent in webob/dec.py(__call__), paste/urlmap.py(__call__), webob/request.py(call_application),webob/request.py(send). What else can I do to try and look for why this is taking so long? As a side question it seems counterintuitive that the uWSGI handles most of the trunk RPC calls and not the RPC workers? A couple other notes about my environment that could indicate my challenges: I had to disable rabbitmq heartbeats for neutron as they kept not getting sent reliably and connections were terminated. I tried with heartbeat_in_pthread both true and false but still had issues. It looks like nova also sometimes experiences this but not near as often. I was overzealous with my vxlan ranges in my first configuration and gave it a range of 10,000,000 not realizing that would create that many rows in the database. Looking into that I saw that pymysql in my cluster takes 3.5 minutes to retrieve those rows. mysql CLI only takes 4 seconds. Perhaps that is just the overhead of pymysql? I've greatly scaled down the vxlan range now. I'm provisioning the 2 servers with a heat template that contains around 200 custom resources. For 198 of the resources they are set to conditionally not create any OpenStack native resources. Deploying this template of mostly no-op resources still takes about 3 minutes. Horizon works but almost every page load take a few seconds to load. I'm not sure if that is normal or not. Thanks for any help anyone can provide. john

4 8

Does Openstack support the notion of cinder backend in maintenance mode?
by wodel youchi 05 Apr '23

05 Apr '23

Hi, When you have multiple cinder backends, and you need to take down one of them for maintenance purposes, what is the best way to proceed? Does Openstack have the notion of : cinder backend in maintenance mode? Regards.

3 6

[cinder] 2023.2 Bobcat Meeting Etherpad
by Rajat Dhasmana 04 Apr '23

04 Apr '23

Hello Argonauts, As we're done with the 2023.2 Bobcat PTG, we will be moving to a new etherpad for cinder meetings for the 2023.2 Bobcat development cycle. Please add topics (if you have any) to the etherpad for the meeting on 05 April, 2023 (Today). Etherpad: https://etherpad.opendev.org/p/cinder-bobcat-meetings Thanks Rajat Dhasmana

1 0

[manila] Bobcat vPTG summary
by Carlos Silva 04 Apr '23

04 Apr '23

Hello, Zorillas and interested stackers! Thank you for attending last week's PTG. We had productive discussions and came up with good plans for this cycle! Here is a summary of the topics we spoke about during the PTG. The etherpad with all the notes is available in [0]. If you would like to see the recordings for those discussions, they are available on the Manila Youtube channel [1]. - *Share Transfer Future development* - Share transfers feature was introduced in Antelope but it has a couple of possible enhancements, such as: transferring shares that have snapshots, replicated shares, share networks alongside its share servers and shares. - Snapshots shouldn't be very difficult to cover but we would need to keep in mind quota allocations while doing so. - Transferring shares with replicas is the bigger challenge, considering we need to work with modifying the access rules. - The next step would be to work on transferring share networks. - AIs: Haixin plans to work on this feature during Bobcat. - *S-RBAC changes in Bobcat* - For the Bobcat cycle, we will focus on: - Ensure we are landing even more tests and enabling the oslo policy defaults. - Modify the testing we are doing to use the reader role to test read-only API, opposed to the member role. - Audit all cross service interactions and isolate actions performed by the "service" user. - Push for the completion of unit tests for Secure RBAC - AIs: Manila team will continue promoting events to get tests reviewed and implemented - *Cross project discussion with Nova for VirtIO FS * - Manila and Nova team discussed approaches for avoiding shares to be deleted while they are attached to instances. - We have agreed that an admin/service level API for locking/unlocking shares deletion would be the best solution to avoid shares being deleted while mounted. This API would ideally be a 'locked by <consumer>' action instead of a simple boolean - As part of this cross-project effort, the Manila team has also been focusing on getting more APIs available in OpenStack SDK. We have been making great progress with it and we plan to have all APIs nova is using merged in the OpenStack SDK by the bobcat cycle. - AIs: - Manila team to write a spec describing the behavior of the locking/unlocking API for share. - Manila team to focus on having all APIs nova is using in OpenStack SDK by the Bobcat cycle. - *Scalable NFS Ganesha for CephFS - state of the kitchen* - We have shared a status of what was our focus during the previous release and some important bugs that have been fixed. - Updates on moving from a standalone ganesha to a cephadm deployed: - There are three changes being worked on at the moment, after past discussions: - Ensure that the shares access rules will be applied when moving from the old to the new server - Ensure that the share exists in both old and new ganesha servers and to reapply all access rules in both - Change to represent export locations from both old and new export locations while both cephfs_nfs_cluster_id and ganesha_server_ips are present in the backend. - With these three changes we would cover all the scenarios considering release upgrades. - The changes are being tested and need feedback. - A job for automated testing is in progress. We are targeting to have a multi-node CI job that will be capable of deploying ganesha using cephadm and run tests using it. - AI: carloss and gouthamr will pursue closure for the three changes related to the upgrades and ashrodri will continue to work on the multi-node testing job. - *SQLAlchemy 2.0 and other DB challenges/changes* - DB testing is failing sporadically in the CI and we were discussing possible changes to avoid rechecks caused by sporadic failures. - We intend to work on missing indexes for queries, so we can gain performance in the database. - Implementing __repr__ to manila models - Other people are already doing this and this could be of huge help for debugging, as one of the operators suggested in the session - We will investigate and apply this in Manila - AIs: gouthamr will open bugs for tracking the work on the joinedloads and a blueprint for the missing indexes. - Sqlalchemy 2.0 changes have already started and we have a list of changes yet to merge. stephenfin is doing a great work on the changes. - AIs: manila team to review the changes proposed by stephenfin - *Create share from replicated snapshot / keep source share on replica promote* - Users of this feature are looking for a way to test the inactive replicas before actually promoting them and if everything is correct, they continue with the promotion. - One suggestion to get this working would be having a replication pause API, which would make the inactive instance able to be mounted - We do think this feature could be relevant and some suggestions from the Manila team are: - Letting users know if they can pause/unpause replication even before they create their share replicas - Making users able to know about errors in the API (something similar to the check mechanism on security-service-update) - AIs: carthaca and team will work on a spec and will discuss it further with the Manila team. - *Tech debt:* - We brought up the current issues with our CI, mostly caused by tests being very resource intensive. - We discussed approaches for scenario testing and thought it is worth it trying to create containers to mount share during scenario tests instead of spawning VMs. - Also, the audience has mentioned that it is difficult to keep track of all of these issues and it would be better if we had them documented somewhere, and that we could bring these up more often. - Migration to Ubuntu 22 is yet to complete for some jobs due to failures in package installation and the usage of quagga. - AIs: - carloss will work on a wiki with tech debt items and will bring this up more often in the manila weekly meetings, so if people are available in that time frame to work on them, they will have more chances to raise their hands. - carloss and gouthamr will work to fix the issues with jobs yet to migrate to Ubuntu 22 - *Scheduler data placement based on vendor specific tool* - NetApp has a tool called Active IQ that uses machine learning to identify what is the best pool of disks to place a request based on custom parameters (latency, iops and so on) - They are thinking of trying to have this tool interact with Manila, so it would also benefit from their tool when NetApp is the only storage vendor in the cloud. - The suggestion is to have a new weigher that will make calls to the storage based on the request and use Active IQ response to identify where the share should be placed. - Testing this feature could be a challenge, considering that it's not straightforward to foresee what a weigher would return in a given moment - AIs: felipe_rodriguess alongside the NetApp team will work on a spec so this can be further discussed. [0] https://etherpad.opendev.org/p/manila-bobcat-ptg [1] https://www.youtube.com/watch?v=gefbYrzRD04&list=PLnpzT0InFrqDW_djitgCiFzwh… Thank you! carloss

1 0

[neutron] bug deputy report March 27th - April 2nd
by Miguel Lavalle 04 Apr '23

04 Apr '23

Here's last week's bugs deputy report: Medium ====== https://bugs.launchpad.net/neutron/+bug/2012944 neutron-remove-duplicated-port-bindings does not write to stdout. Valid bug. Assigned to mlavalle https://bugs.launchpad.net/neutron/+bug/2013326 Trying to activate port binding as regular user causes error 500. Assigned to slaweq Low: ========== https://bugs.launchpad.net/neutron/+bug/2012912 [neutron-lib][stable] Wallaby CI n-t-p, test "test_qos_dscp_create_and_update" failing. This bug is addressed by https://review.opendev.org/c/openstack/neutron-lib/+/879441. No further action seems likely RFEs ==== https://bugs.launchpad.net/neutron/+bug/2013228 [rfe] Non-admin users unable to create SR-IOV switch dev ports. Discussed during the PTG during the Neutron / Nova session, pending further discussion in drivers meeting https://bugs.launchpad.net/neutron/+bug/2013540 [RFE] Add support for Napatech LinkVirt SmartNICs. Discussed during the PTG, pending further discussion in drivers meeting

1 0

OpenInfra Live - April 6 at 9 a.m. CT / 14:00 UTC
by Kristin Barrientos 04 Apr '23

04 Apr '23

Hi everyone, This week’s OpenInfra Live episode is brought to you by the OpenStack Large Scale SIG. Episode: Large Scale Ops Deep Dive: Société Générale In the "Large Scale Ops Deep Dive" series, a panel of OpenStack operators invites special guests to talk about their deployment and discuss their operations. For this episode, our guest will be Guillaume Allard from Société Générale, one of France’s largest banks. Speakers: Guillaume Allard, Arnaud Morin, Thierry Carrez, Belmiro Moreira, Mohammed Naser and Kristin Barrientos Date and time: April 6, 2023, at 9 a.m. CT (14:00 UTC) You can watch us live on: YouTube: https://www.youtube.com/live/ZIjpEZSu1gw?feature=share LinkedIn: https://www.linkedin.com/video/event/urn:li:ugcPost:7043641363350908928/ WeChat: recording will be posted on OpenStack WeChat after the live stream Have an idea for a future episode? Share it now at ideas.openinfra.live <https://openinfrafoundation.formstack.com/forms/openinfralive>. Thanks, Kristin Barrientos Marketing Coordinator OpenInfra Foundation

1 0

[horizon][domain][yoga][kolla-ansible] horizon display failed to display domain tab
by James Leong 04 Apr '23

04 Apr '23

Hi all, I am playing around with the domain. I have used kolla-ansible as a deployment tool for deploying the yoga version of OpenStack. However, after deploying a fresh OpenStack, I was not able to view the domain tab on my dashboard. Every time when I want to create a new domain, I have to go to my terminal and create it manually without any GUI. Is there any other configuration I have to enable in the globals.yml file other than setting "horizon_keystone_multidomain: "True"? Thanks for your help James

3 4

[policy][rbac] RBAC 2023.2 Bobcat cycle vPTG discussions summary
by Ghanshyam Mann 04 Apr '23

04 Apr '23

Hello Everyone, We discussed the RBAC goal on Tuesday. I am summarizing the discussion here. Goal document: https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rb… Tracking: https://etherpad.opendev.org/p/rbac-goal-tracking Current progress: ============= Phase-1 (project personas and drop system scope): ------------------------------------------------------------ Projects completed: * Nova * Neutron * Glance * Manila * Ironic (no change needed) * Octavia * Placement * Service that completed Phase 1 in Zed and enabled scope and new defaults by default (enforce_scope=True & enforce_new_defaults=True by default): ** Nova ** Glance Projects in progress: * Cinder (almost completed. not using system scope, but have not added scope=['project'] to the default rules (currently, no scope is specified) * Magnum ** https://review.opendev.org/c/openstack/magnum/+/874945 * Tacker ** https://review.opendev.org/q/topic:bp%252Fimplement-project-personas Pending work (for phase-2|3): * Keystone implements a new default role called manager: ** https://review.opendev.org/c/openstack/keystone/+/822601 * Keystone implements a new default role called service: ** https://review.opendev.org/c/openstack/keystone/+/863420 Phase-2 (service role): -------------------------- In-progress: * Keystone: ** bootstrap support for servicerole: *** https://review.opendev.org/c/openstack/keystone/+/863420 ** bootstrap support for manager role: *** https://review.opendev.org/c/openstack/keystone/+/822601 *Nova ** https://review.opendev.org/c/openstack/nova/+/864594 Other discussion: ============= *Service role We discussed the service role and how the policy will add the service role. It is correct to add service as well as user role as default if that API is supposed to be called by the service as well as the user role. For example, if Manila is talking to Nova, Cinder, or Neutron via APIs, it needs to use a service role to interact, and Nova, Cinder, and Neutron can update such API rules to allow for service roles also. * Update the goal timeline for removing deprecated rules as per the SLURP release. ** Need at least 1 SLURP release between enabling the new default and removal. ** Action: gmann to update this in the goal document. *Testing: ** Manila has a lot of tests and running on the stable release ** Tempest and devstack ready to implement the test ** The current job with nova, cinder, neutron, glance with the scope and new default enable *** https://zuul.openstack.org/builds?job_name=tempest-full-enforce-scope-new-d… * Tempest now has a project reader/member same project_id * Related sessions: ** Tacker *** Title: Secure RBAC: Implement support of project-personas in Tacker [Continue from Antelope release] (manpreetk) *** Etherpad: https://etherpad.opendev.org/p/tacker-bobcat-ptg#L136 ** Glance *** Title: Secure RBAC *** Etherpad: https://etherpad.opendev.org/p/glance-bobcat-ptg#L53 ** Neutron *** Title: (slaweq) Secure RBAC - phase 2 description and review of the existing API calls *** Etherpad: https://etherpad.opendev.org/p/neutron-bobcat-ptg#L358 <feel free to add here if I missed any related sessions> I will continue holding the biweekly meeting to discuss progress and any query on RBAC. - https://wiki.openstack.org/wiki/Consistent_and_Secure_Default_Policies_Popu… -gmann

1 0

[keystone] Weekly Meeting Cancelled (04-Apr 2023)
by Dave Wilde 04 Apr '23

04 Apr '23

Hi folks, I'm cancelling the Keystone weekly meeting this week (04-Apr 2023) due to a conflict that I have. Please reach out to me on IRC or via email if you need anything. Thanks! /Dave

1 0

[neutron][ovn-bgp-agent] Make ovn-bgp-agent a Neutron stadium project
by Rodolfo Alonso Hernandez 04 Apr '23

04 Apr '23

Hello all: During the last PTG, it was discussed and accepted to migrate the ovn-bgp-agent project [1] to the Neutron stadium umbrella. This mail is a heads-up for anyone using it. In the next days (maybe today), Lucas will send the corresponding patches for "governance" and "project-config" repositories in order to start this migration. If you have any questions, please ask them in this email or via IRC in the #openstack-neutron channel. Regards. [1]https://opendev.org/x/ovn-bgp-agent

1 0

[nova][cinder] Providing ephemeral storage to instances - Cinder or Nova
by Christian Rohmann 04 Apr '23

04 Apr '23

Hello OpenStack-discuss, I am currently looking into how one can provide fast ephemeral storage (backed by local NVME drives) to instances. There seem to be two approaches and I would love to double-check my thoughts and assumptions. 1) *Via Nova* instance storage and the configurable "ephemeral" volume for a flavor a) We currently use Ceph RBD als image_type (https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.im…) so instance images are stored in Ceph, not locally on disk. I believe this setting will also cause ephemeral volumes (destination_local) to be placed on a RBD and not /var/lib/nova/instances? Or is there a setting to set a different backend for local block devices providing "ephemeral" storage? So RBD for the root disk and a local LVM VG for ephemeral? b) Will an ephemeral volume also be migrated when the instance is shutoff as with live-migration? Or will there be an new volume created on the target host? I am asking because I want to avoid syncing 500G or 1T when it's only "ephemeral" and the instance will not expect any data on it on the next boot. c) Is the size of the ephemeral storage for flavors a fixed size or just the upper bound for users? So if I limit this to 1T, will such a flavor always provision a block device with his size? I suppose using LVM this will be thin provisioned anyways? 2) *Via Cinder*, running cinder-volume on each compute node to provide a volume type "ephemeral", using e.g. the LVM driver a) While not really "ephemeral" and bound to the instance lifecycle, this would allow users to provision ephemeral volume just as they need them. I suppose I could use backend specific quotas (https://docs.openstack.org/cinder/latest/cli/cli-cinder-quotas.html#view-bl…) to limit the number of size of such volumes? b) Do I need to use the instance locality filter (https://docs.openstack.org/cinder/latest/contributor/api/cinder.scheduler.f…) then? c) Since a volume will always be bound to a certain host, I suppose this will cause side-effects to instance scheduling? With the volume remaining after an instance has been destroyed (beating the purpose of it being "ephemeral") I suppose any other instance attaching this volume will be scheduling on this very machine? Is there any way around this? Maybe a driver setting to have such volumes "self-destroy" if they are not attached anymore? d) Same question as with Nova: What happens when an instance is live-migrated? Maybe others also have this use case and you can share your solution(s)? Thanks and with regards Christian

3 4

[Neutron] CI meeting 04.04.2023 cancelled
by Slawek Kaplonski 04 Apr '23

04 Apr '23

Hi, I have overlapping internal meeting this week and will not be able to run Neutron CI meeting. So lets cancel it and meet next week. See You there. -- Slawek Kaplonski Principal Software Engineer Red Hat

1 0

[neutron] Neutron drivers meeting
by Rodolfo Alonso Hernandez 04 Apr '23

04 Apr '23

Hello Neutrinos: Next Neutron drivers meeting (April 7) will be cancelled. This is Easter week and most people will have Friday as a holiday. See you next week.

1 0

[nova] 2023.2 Bobcat PTG summary
by Sylvain Bauza 04 Apr '23

04 Apr '23

Hey folks, that's a wrap ! I'm happy to say we had a very productive PTG week with around more than 12 contributors each day. It was nice to see new faces. As a reminder, please don't open the main etherpad link [1] if you use a Web browser with an automatic translation modification feature. If you do it, then it would internally translate all the phrases directly in the etherpad as if you were directly modifying the etherpad by yourself. Please also make sure you don't accidentally remove or modify lines . In order to prevent any misusage, this etherpad is a read-only copy of the main one : https://etherpad.opendev.org/p/r.22d95e20f7133350dcad573c015ed7da Last point but not the least, I'm a human who usually writes bugs [2], including when it's about writing a report of the past week's discussions and the actions taken. Don't feel offended or irritated if you read silly notes of the very beloved chat you had, please rather fix it the opensource way by correcting my errors with a reply on that thread. Enough chat, please grab a coffee [3] (as it's a quite verbose email) and jump straight on below. ### Operator hour ### (specific and distinct read-only etherpad here : https://etherpad.opendev.org/p/r.f94e6dda608450b6a4a5c9afc76e8a88 ) Yet again, we missed a few more operators in this room (at least we had 3 of them, thanks for those that were there !). I'm still wondering why they are not joining a free session where they can discuss their concerns or some wanted features. Anyway, that's what we discussed : - for the moment, no operators have tried to use the Yoga unified limits feature. - operators are happy with the new default RBAC scopes. We could create a small and quick admin-guide section for explaining that and pointing to the oslo.policy documentation. - operators would love to use the (possibly) new virtiofs feature from Bobcat for Manila shares (if we are able to merge it). That being said, they think they couldn't provide it for their environments until Nova can live-migrate the instances using the shares (which is not possible as for the moment, even QEMU doesn't support it) - we also discussed a pain point from an operator and then we asked him to provide a bug report (related to the vm state continuing to be ACTIVE after shelving). For the moment, we can't reproduce the problem. ### Cross-project discussions ### ## Manila/Nova ### # Prevent shares deletion while it's attached to an instance - Nova could add some traits to reflect the share type. - the Manila team will create a manila spec for adding a service-level (or admin-level) lock API with a 'locked-by' parameter (or using service tokens if set) ### Neutron/Nova ### Read Neutron summary email [4] if you want more details. - We agreed on a delete_on_termination feature for ports. We need to see if we can use some already existing Neutron API (tags) or if we need a whole new API. - We are quite OK with the Napatech LinkVirt SmartNics feature, but we need to review the spec and we also want some third-party CI support for it. - the Neutron team accepted to return a HTTP409 Conflict return code if a duplicated port binding action is done by Nova and the Nova team agreed on handling it. - About the SRIOV port tracking in Placement, the Nova team needs to help the spec owner on how to work on it as it's a large and difficult design (but we agreed on the first part of the design). ### Glance/Nova ### Glance wants to add a new API for Image Direct URL access. Nova was quite OK with this new API, but the Nova team was having some concerns about upgrades and client calls to the new API. The Manila team agreed to create a nova spec for discussing those. ### Cinder/Nova ### - The Nova team is quite OK with the usecase for NFV encryption. Since the design is quite simple, maybe this could be a specless feature, just using traits for telling to Placement which computes support NFV encryption. We're OK to test this with cinder NFV jobs and maybe also testing it with a Nova periodic job depending on the implementation. - The Cinder team agreed on using its volume metadata API for tracking the cinder volumes hardware models. ### Horizon/Nova ### Horizon wanted to call the Placement API. We agreed on using the openstackSDK for this as Placement doesn't have python bindings directly. Some new methods may be added against the SDK for calling specific Placement APIs if we think we may need those (instead of directly calling Placement with the standard HTTP verbs and the API URIs) ### Nova-specific topics ### # Antelope retrospective and process-related discussions Shorter cycle than Zed, 6 features landed (same as Zed), 46 contributors (down by 8 compared to Zed), 27 bugfixes (vs. 54 for Zed). We were happy to not have a RC2 for the first time in the Nova history. The bug triage backlog kept small. - We want to discuss on a better review cadence to not look at all the implementations by the last weeks. - We want to get rid of Storyboard stories in Placement. We found and agreed on a way forward. - We will set 5-years-old-and-more bug reports as Incomplete, so bug reporters have 90 days to confirm that the bug is still present on master. - We will add more contrib documentation for explaining more how to find Gerrit changes for one-off reviewers and how to create some changes - We agreed on no longer automatically accepting an already-approved spec from a previous cycle if no code is still not existing. - All efforts on porting our client calls to the SDK should be tagged in Launchpad with the 'sdk' tag. - A Bobcat schedule will be proposed on next Tuesday weekly meeting with feature and spec deadlines + some review days. - We also discussed what we should discuss in the Vancouver PTG. # CI failures continuous improvement - We want to test some alternative image but cirros (alpine as a potential alternative) with a small footprint in some specific nova jobs - We need to continue investigating on the volume detach/attach failures. We may create a specific job that would do serialized volume checks as a canary job for forcing the job to fail more frequently in order to dig down further. - (related) Sylvain will propose a Vancouver PTG cross-project session for a CI debugging war room experience. # The new release cadence We had long arguments whether we should hold deprecations and removals this cycle, given the releasenotes tooling isn't in place yet. As a reminder, since Bobcat is a non-SLURP release, operators are able to skip it and just look at the C releasenotes so we want to make sure we forward-port upgrade notes for them. For the moment, no consensus in place, we defer the outcome into a subsequent nova meeting and in the meantime, we raised the point to the TC for guidance needs. # The problem with hard affinity group policies - We agreed on the fact that hard affinity/anti-affinity policies aren't ideal (operators, if you read me, please prefer soft affinity/anti-affinity policies for various reasons I won't explain here) but since we support those policies and the use case is quite understandable, we need to find a way to unblock instances that can't be easily-migrated from those kind of groups (eg. try to move your instances off a host with hard Affinity between them, good luck) - A consensus has been reached to propose a new parameter for live-migrate, evacuate and migrate that would only skip the hard affinity checks. A backlog spec will be written capturing this consensus. - Since a group policy may be violated, we'll also propose a new server group API modification that would show if a group has its policy violated. # Unified limits next steps - We agreed on the fact we should move on and enable by default unified limits in a close future, but we first need to provide a bit of tooling to help the migration. - tool could be a nova-manage command that would provide the existing quota limits as a yaml file so we could inject those into keystone (to be further defined during Bobcat) - a nova-status upgrade check may be nice for yelling if operators forgot to define their limits in keystone before upgrading to the release (potentially C) that would default to unified limits - testing in the grenade job is a feat. # Frequently written objects may lead to exhaustion of DB primary keys - We agreed on having all new primary keys to have BigInteger as data type. Existing keys should use BigInt but there is a huge upgrade impact to mitigate. - Operators will be sollicited to know their preferences between a costly-at-once data migration vs. a rolling data online migration for existing tables. - We will document which DB tables are safe to reindex (since their PKs isn't normalized as a FK somewhere else) - A backlog spec will capture all of the above # Leftover volumes relationship when you directly call Cinder instead of Nova for detaching - Yup, this is a known issue and we should discuss this with the Cinder team (unfortunately, we didn't did it this week) - We maybe should have a nova-manage script that would cleanup the BDMs but we're afraid of leaking some volume residues in the compute OS - Anyway, we need to understand the main scope and we also need to discuss about the preconditions ('when' should we call this tool and 'what' this tool should be doing ?). Probably a spec. # Misc but not the leasc (heh) - we will fill a rfe bug for lazy-loading instance name from system metadata or update the nested field in the instance object (no additional DB write) - we will add a new policy for cold-migrate specifically when a host is passed as a parameter (admin-only as a default) - We agreed on continuing the effort on compute node hostname robustification (the Bobcat spec seems not controversial) - We agreed on a few things about the server show command and the use of the SDK. Some existing OSC patch may require a bit of rework but it would cover most of the concerns we discussed. - we should fix the post-copy issue when you want to live-migrate a paused instance by removing the post-copy migration flag if so. - we could enable virtio-blk trim support by default. It's both a bugfix (we want to remove an exception) and a new feature (we want to enable it by default) so we'll discuss whether we need a specless blueprint by a next meeting. - we also discuss about a Generic VDPA support feature for Nova, and we asked the owner to provide us a spec for explaining the usecase. ### That's it. ### Yet again, I'm impressed. If you read that phrase, you're done with reading. I just hope your coffee was good and the time you took reading that email was worth it. If you have questions or want to reach me directly on IRC, that's simple : I'm bauzas on #openstack-nova. HTH, -Sylvain (on behalf of the whole Nova community) [1] https://etherpad.opendev.org/p/nova-bobcat-pt g (please remove the empty char between 'pt' and 'g') [2] Unfortunately, I'm very good at creating bugs as I discovered that based on my 10-year unfortunate contributions to OpenStack. [3] Or a tea, or whatever you prefer. [4] https://lists.openstack.org/pipermail/openstack-discuss/2023-April/033115.h…

1 1

[kolla-ansible][zed][15.1.0] Horizon failed to load trunk and QoS
by Stefan Bryzga 04 Apr '23

04 Apr '23

Hi all, Some time ago I deployed a small cluster with kolla-ansible v15.1.0. Before deployment I enabled neutron_qos and neutron_trunk flags in /etc/kolla/globals.yml. Both functions work fine via cli and I can create new entries via dashboard however horizon won't display created trunks/qos on the page (It shows "No items to display"). In web browser console I get this error while loading page: TypeError: Cannot read properties of undefined (reading 'data') at addTrackBy (output.0a9483334bbd.js:1115:59) at processQueue (output.43a09e34a303.js:1581:786) at output.43a09e34a303.js:1585:110 at Scope.$digest (output.43a09e34a303.js:1636:200) at Scope.$apply (output.43a09e34a303.js:1643:269) at done (output.43a09e34a303.js:1356:123) at completeRequest (output.43a09e34a303.js:1375:20) at XMLHttpRequest.requestLoaded (output.43a09e34a303.js:1368:1) 'Possibly unhandled rejection: {}' Does anyone knows how to fix this? Should I modify any configs to display new entries in horizon? Best Regards, Stefan

1 0

Opensearch service upgrade
by Daniel Pawlik 04 Apr '23

04 Apr '23

Hello, We would like to notify you that the Opensearch service [1] would be updated to a newer version on 03.04.2023 at 12:00 PM UTC. This procedure might take a while, depending of the cluster size. During that time, the Opensearch service would not be available. If anyone has any doubts, please reply to the email. Dan [1] - https://opensearch.logs.openstack.org/_dashboards/app/home

1 1

Propose Dale Smith as Magnum Caore
by Jake Yip 04 Apr '23

04 Apr '23

Hi all, I would like to propose Dale Smith as a Magnum Core Reviewer. Dale is a contributor to Magnum, and also takes care of Magnum at his place of employment - Catalyst Cloud NZ. As many Magnum contributors are ourselves operators, I think it is important that we have representation from different cloud providers to help guide and shape Magnum going forward. He will replace Feilong, who is the former PTL and used to work in Catalyst Cloud, in the Core Team. We thank Feilong for his contributions and welcome Dale. Regards, Jake -- Jake Yip Technical Lead, ARDC Nectar Research Cloud

1 0

[nova][heat] The next steps to "fix" libvirt problems in Ubuntu Jammy
by Takashi Kajinami 03 Apr '23

03 Apr '23

Hello, Since we migrated our jobs from Ubuntu Focal to Ubuntu Jammy, heat gate jobs have become very flaky. Further investigation revealed that the issue is related to something in libvirt from Ubuntu Jammy and that prevents detaching devices from instances[1]. The same problem appears in different jobs[2] and we workaround the problem by disabling some affected jobs. In heat we also disabled some flaky tests but because of this we no longer run basic scenario tests which deploys instance/volume/network in a single stack, which means we lost the quite basic test coverage. My question is, is there anyone in the Nova team working on "fixing" this problem ? We might be able to implement some workaround (like checking status of the instances before attempting to delete it) but this should be fixed in libvirt side IMO, as this looks like a "regression" in Ubuntu Jammy. Probably we should report a bug against the libvirt package in Ubuntu but I'd like to hear some thoughts from the nova team because they are more directly affected by this problem. I'm now trying to set up a centos stream 9 job in Heat repo to see whether this can be reproduced if we use centos stream 9. I've been running that specific scenario test in centos stream 9 jobs in puppet repos but I've never seen this issue, so I suspect the issue is really specific to libvirt in Jammy. [1] https://bugs.launchpad.net/nova/+bug/1998274 [2] https://bugs.launchpad.net/nova/+bug/1998148 Thank you, Takashi

3 5

[horizon]display quota volume type on horizon
by Nguyễn Hữu Khôi 03 Apr '23

03 Apr '23

Hello. I see that blueprint is very real case but it has not deployed yet. https://blueprints.launchpad.net/horizon/+spec/quotas-per-cinder-volume-type I am curious why it is abanddoned .

2 2

[designate][ptg] Designate Bobcat (2023.2) PTG summary
by Michael Johnson 03 Apr '23

03 Apr '23

We accomplished some big tasks in Antelope: Finishing the shared zones feature, adding support for SQL Alchemy 2.x, and hard zone deletes. For the Bobcat series, Designate had a pretty straightforward PTG session with good discussion. The focus areas for Bobcat are going to be on the batch serial number update patch[1] (deferred from Antelope) and improving the multiple pools feature[2][3]. Multiple DNS server pools have been a feature in Designate for a number of years now, but we need to improve the test coverage and documentation. Other minor work anticipated for Bobcat is to update the RBAC testing to account for the RBAC "Direction Change"[4] in support of enabling the new Keystone default roles as the default RBAC configuration. We also discussed the new specification that merged during Antelope[5] to add support for catalog zones. This specification is merged and ready for development, but it was not clear that there are developers planning to work on it during Bobcat. We hope this feature can at least get started during the Bobcat series. Thank you to everyone that participated! Michael [1] https://review.opendev.org/c/openstack/designate/+/871255 [2] https://review.opendev.org/c/openstack/designate/+/857978 [3] https://review.opendev.org/c/openstack/designate-tempest-plugin/+/871184 [4] https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rb… [5] https://specs.openstack.org/openstack/designate-specs/specs/antelope/catalo…

1 0

[ptls] PyPI maintainer cleanup - Action needed: Contact extra maintainers
by Jay Faulkner 03 Apr '23

03 Apr '23

Hi PTLs, The TC recently voted[1] to require humans be removed from PyPI access for OpenStack-managed projects. This helps ensure all releases are created via releases team tooling and makes it less likely for a user account compromise to impact OpenStack packages. Many projects have already updated https://etherpad.opendev.org/p/openstack-pypi-maintainers-cleanup#L33 with a list of packages that contain extra maintainers. We'd like to request that PTLs, or their designate, reach out to any extra maintainers listed for projects you are responsible for and request they remove their access in accordance with policy. An example email, and detailed steps to follow have been provided at https://etherpad.opendev.org/p/openstack-pypi-maintainers-cleanup-email-tem… . Thank you for your cooperation as we work to improve our security posture and harden against supply chain attacks. Thank you, Jay Faulkner TC Vice-Chair 1: https://opendev.org/openstack/governance/commit/979e339f899ef62d2a6871a99c9…

3 6

[barbican] database is growing and can not be purged
by Pavlo Shchelokovskyy 03 Apr '23

03 Apr '23

Hi all, we are observing the following behavior in Barbican: - OpenStack environment is using both encrypted Cinder volumes and encrypted local storage (lvm) for Nova instances - over the time, the secrets and orders tables are growing - many soft-deleted entries in secrets DB can not be purged by the db cleanup script As I understand what is happening - both Cinder and Nova create secrets in Barbican on behalf of the user when creating an encrypted volume or booting an instance with encrypted local storage. They both do it via castellan library, that under the hood creates orders in Barbican, waits for them to become active and returns to the caller only the ID of the generated secret. When time comes to delete the thing (volume or instance) Cinder/Nova again use castellan, but only delete the secret, not the order (they are not aware that there was any 'order' created anyway). As a result, the orders are left in place, and DB cleanup procedure does not delete soft-deleted secrets when there's an ACTIVE order referencing such secret. This is troublesomes on many levels - users who use Cinder or Nova may not even be aware that they are creating something in Barbican. Orders accumulating like that may eventually result in cryptic errors when e.g. when you run out of quota for orders. And what's more, default Barbican policies do allow 'normal' (creator) users to create an order, but not delete it (only project admin can do it), so even if the users are aware of Barbican involvement, they can not delete those orders manually anyway. Plus there's no good way in API to determine outright which orders are referencing deleted secrets. I see several ways of dealing with that and would like to ask for your opinion on what would be the best one: 1. Amend Barbican API to allow filtering orders by the secrets, when castellan deletes a secret - search for corresponding order and delete it as well, change default policy to actually allow order deletion by the same users who can create them. 2. Cascade-delete orders when deleting secrets - this is easy but probably violates that very policy that disallowed normal users to delete orders. 3. improve the database cleanup so it first marks any order that references a deleted secret also as deleted, so later when time comes both could be purged (or something like that). This also has a similar downside to the previous option by not being explicit enough. I've filed a bug for that https://storyboard.openstack.org/#!/story/2010625 and proposed a patch for option 2 (cascade delete), but would like to ask what would you see as the most appropriate way or may be there's something else that I've missed. Btw, the problem is probably even more pronounced with keypairs - when castellan is used to create those, under the hood both order and container are created besides the actual secrets, and again only the secret ids are returned to the caller. When time comes to delete things, the caller only knows about secret IDs, and can only delete them, leaving both container and order behind. Luckily, I did not find any place across OpenStack that actually creates keypairs using castellan... but the problem is definitely there. Best regards, -- Dr. Pavlo Shchelokovskyy Principal Software Engineer Mirantis Inc www.mirantis.com

2 2

[SWIFT] ssync receiver crash
by Reid Guyett 03 Apr '23

03 Apr '23

Hello, We recently upgraded our swift servers to 20.04 from 18.04. After the upgrades were completed we are seeing 1 server where the ssync receiver keeps crashing. The crash is preventing hand offs from being cleanup up in the cluster (50 servers). Looking for some advice to fix/workaround the issue. We have 20 clusters but are only seeing this issue in 1 cluster. It is strange to me that we only see this ssync receiver error in one server out of the ~1000 we are running. OS: Ubuntu 20.04.5 Swift version: 2.25.2 On the receiver side we are seeing the following trace: object-server: senderIp/d1/2166 EXCEPTION in ssync.Receiver: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/eventlet/wsgi.py", line 209, in _chunked_read self.chunk_length = int(rfile.readline().split(b";",1)[0], 16) ValueError: invalid literal for int() with base 16: b'' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/swift/obj/ssync_receiver.py", line 166, in __call__ for data in self.missing_check(): File "/usr/lib/python3/dist-packages/swift/obj/ssync_receiver.py", line 340, in missing_check line = self.fp.readline(self.app.network_chunk_size) File "/usr/lib/python3/dist-packages/eventlet/wsgi.py", line 226, in readline return self._chunked_read(self.rfile, size, True) File "/usr/lib/python3/dist-packages/eventlet/wsgi.py", line 211, in _chunked_read raise ChunkReadError(err) eventlet.wsgi.ChunkReadError: invalid literal for int() with base 16: b'' On the sender side we are seeing this message when I try to run the reconstructor manually with debug logging: root@use1-saas-p6-paco-9:~# swift-object-reconstructor object-server.conf -p 27999 -v -o object-reconstructor: Starting 271723 object-reconstructor: Spawned worker 271755 with {'override_partitions': [27999], 'override_devices': ['d10', 'd17', 'd19', 'd9', 'd29', 'd15', 'd2', 'd24', ' d32', 'd1', 'd8', 'd14', 'd11', 'd28', 'd27', 'd6', 'd7', 'd16', 'd13', 'd25', 'd26', 'd20', 'd18', 'd12', 'd4', 'd0', 'd22', 'd3', 'd5', 'd21', 'd33', 'd23', 'd30', 'd34', 'd31'], 'multiprocess_worker_index': 0} object-reconstructor: [worker 1/1 pid=271755] Running object reconstructor in script mode. object-reconstructor: [worker 1/1 pid=271755] Run listdir on /srv/node/d22/objects-4/27999 object-reconstructor: [worker 1/1 pid=271755] recieverIP:6200/d31/27999 10.0 seconds: connect receive object-reconstructor: [worker 1/1 pid=271755] 1/26820 (0.00%) partitions reconstructed in 10.14s (0.10/sec, 75h remaining) object-reconstructor: [worker 1/1 pid=271755] Object reconstruction complete (once). (0.17 minutes) object-reconstructor: Forked worker 271755 finished object-reconstructor: Worker 271755 exited object-reconstructor: Finished 271723 object-reconstructor: Exited 271723 Thanks! Reid Guyett Staff Storage Engineer Datto, A Kaseya Company www.datto.com<http://www.datto.com/> Important Notice: This email is intended to be received only by persons entitled to receive the confidential and legally privileged information it presumptively contains, and this notice constitutes identification as such. Any reading, disclosure, copying, distribution or use of this information by or to someone who is not the intended recipient, is prohibited. If you received this email in error, please notify us immediately at legal(a)kaseya.com, and then delete it. To opt-out of receiving emails Please click here<https://info.kaseya.com/email-subscription-center.html>. The term 'this e-mail' includes any and all attachments.

1 0

[nova][dev] image to local block device mapping not supported
by Nemanja Miletic 03 Apr '23

03 Apr '23

Hello, Nova does not support image to local block device mapping since issue in this bug is fixed: https://bugs.launchpad.net/nova/+bug/1377958 In particular I am interested in scenario best described with this command: openstack server create --flavor m1.small --network mgmt --image XXX \ --block-device uuid=YYY,source_type=image,\ destination_type=local,disk_bus=ide,device_type=cdrom,boot_index=1 testvm300 For above command nova gives error: Block Device Mapping is Invalid: Mapping image to local is not supported. Libvirt does not have issues with having two block devices which have file source. My question is what am I missing here? Why Nova does not support image to local block device mapping? Thank you for your help. Nemanja

2 2

[kolla] Image building question
by Satish Patel 03 Apr '23

03 Apr '23

Folks, I am playing with kolla image building to understand how it works. I am using the following command to build images and wanted to check with you folks if that is the correct way to do it. $ kolla-build -b ubuntu -t source keystone nova neutron glance Does the above command compile code from source or just download images from remote repositories and re-compile them? because in command output I've not noticed anything related to the compiling process going on. Here is the output of all images produced by kolla-build command. Do I need anything else or is this enough to deploy kolla? root@docker-reg:~# docker images REPOSITORY TAG IMAGE ID CREATED SIZE kolla/mariadb-server 15.1.0 2a497eee8269 26 minutes ago 595MB kolla/cron 15.1.0 342877f26a8a 30 minutes ago 250MB kolla/memcached 15.1.0 0d19a4902644 31 minutes ago 250MB kolla/mariadb-clustercheck 15.1.0 d84427d3c639 31 minutes ago 314MB kolla/mariadb-base 15.1.0 34447e3e59b6 31 minutes ago 314MB kolla/keepalived 15.1.0 82133b09fbf0 31 minutes ago 260MB kolla/prometheus-memcached-exporter 15.1.0 6c2d605f70ee 31 minutes ago 262MB <none> <none> e66b228c2a07 31 minutes ago 248MB kolla/rabbitmq 15.1.0 8de5c39379d3 32 minutes ago 309MB kolla/fluentd 15.1.0 adfd19027862 33 minutes ago 519MB kolla/haproxy-ssh 15.1.0 514357ac4d36 36 minutes ago 255MB kolla/haproxy 15.1.0 e5b9cfdf6dfc 37 minutes ago 257MB kolla/prometheus-haproxy-exporter 15.1.0 a679f65fd735 37 minutes ago 263MB kolla/prometheus-base 15.1.0 afeff3ed5dce 37 minutes ago 248MB kolla/glance-api 15.1.0 a2241f68f23a 38 minutes ago 1.04GB kolla/glance-base 15.1.0 7286772a03a4 About an hour ago 1.03GB kolla/neutron-infoblox-ipam-agent 15.1.0 f90ffc1a3326 About an hour ago 1.05GB kolla/neutron-server 15.1.0 69c844a2e3a9 About an hour ago 1.05GB kolla/neutron-l3-agent 15.1.0 4d87e6963c96 About an hour ago 1.05GB <none> <none> 486da9a6562e About an hour ago 1.05GB kolla/neutron-linuxbridge-agent 15.1.0 e5b3ca7e099c About an hour ago 1.04GB kolla/neutron-bgp-dragent 15.1.0 ac37377820c6 About an hour ago 1.04GB kolla/ironic-neutron-agent 15.1.0 90993adcd74b About an hour ago 1.04GB kolla/neutron-metadata-agent 15.1.0 8522f147f88d About an hour ago 1.04GB kolla/neutron-sriov-agent 15.1.0 8a92ce7d13c0 About an hour ago 1.04GB kolla/neutron-dhcp-agent 15.1.0 5c214b0171f5 About an hour ago 1.04GB kolla/neutron-metering-agent 15.1.0 7b3b91ecd77b About an hour ago 1.04GB kolla/neutron-openvswitch-agent 15.1.0 1f8807308814 About an hour ago 1.04GB kolla/neutron-base 15.1.0 f85b6a2e2725 About an hour ago 1.04GB kolla/nova-libvirt 15.1.0 0f3ecefe4752 About an hour ago 987MB kolla/nova-compute 15.1.0 241b7e7fafbe About an hour ago 1.47GB kolla/nova-spicehtml5proxy 15.1.0 b740820a7ad1 About an hour ago 1.15GB kolla/nova-novncproxy 15.1.0 1ba2f443d5c3 About an hour ago 1.22GB kolla/nova-compute-ironic 15.1.0 716612107532 About an hour ago 1.12GB kolla/nova-ssh 15.1.0 ae2397f4e1c1 About an hour ago 1.11GB kolla/nova-api 15.1.0 2aef02667ff8 About an hour ago 1.11GB kolla/nova-conductor 15.1.0 6f1da3400901 About an hour ago 1.11GB kolla/nova-scheduler 15.1.0 628326776b1d About an hour ago 1.11GB kolla/nova-serialproxy 15.1.0 28eb7a4a13f8 About an hour ago 1.11GB kolla/nova-base 15.1.0 e47420013283 About an hour ago 1.11GB kolla/keystone 15.1.0 e5530d829d5f 2 hours ago 947MB kolla/keystone-ssh 15.1.0 eaa7e3f3985a 2 hours ago 953MB kolla/keystone-fernet 15.1.0 8a4fa24853a8 2 hours ago 951MB kolla/keystone-base 15.1.0 b6f9562364a9 2 hours ago 945MB kolla/barbican-base 15.1.0 b2fdef1afb44 2 hours ago 915MB kolla/barbican-keystone-listener 15.1.0 58bd59de2c63 2 hours ago 915MB kolla/openstack-base 15.1.0 c805b4b3b1c1 2 hours ago 893MB kolla/base 15.1.0 f68e9ef3dd30 2 hours ago 248MB registry 2 8db46f9d7550 19 hours ago 24.2MB ubuntu 22.04 08d22c0ceb15 3 weeks ago 77.8MB

4 8

[neutron][ptg] Neutron PTG summary
by Rodolfo Alonso Hernandez 03 Apr '23

03 Apr '23

Hello Neutrinos: This is a quick summary of the PTG sessions last week. This is the etherpad used: https://etherpad.opendev.org/p/neutron-bobcat-ptg *Retrospective.* - Good: - Speed on RFE spec reviewal. - Continue the deprecation of non-supported code: Linux Bridge, neutronclient - Good CI tracking of external library dependencies: pyroute2, sqlalchemy - Good progress on TC goals, in particular sRBAC (now supported in Zed). - Close of stale bugs - Bad: - The use of devstack for newcomers, in particular because of Neutron. It was proposed to implement vagrant files that will work with ant IDE and will allow to deploy OpenStack easily. - Scenario CI jobs still running in Ubuntu Focal due to the nested virt provider issues: https://bugs.launchpad.net/neutron/+bug/1999249 - To be improved: - Engagement with students to join the Neutron community *Stadium projects.* SQLAlchemy 2.0 is the main problem right now. The CI "-sqlalchmey-master" jobs should help us to prepare these projects for the next framework version and we need to start now. Train branch is broken in most of the CIs. With https://review.opendev.org/c/openstack/releases/+/878202, all projects will be migrated to EOL. networking-odl will be deprecated: it is still testing against ODL 11, while the latest version is 16. Due to the lack of maintainers, we are no longer delivering a new release. *Release cadence.* New meeting the first Tuesday of each month (Neutron team meeting) to decide when to release a new project version. Transition Train to EOL: https://review.opendev.org/c/openstack/releases/+/878202 *python-neutronclient deprecation.* Both the CLI and the python bindings code: https://etherpad.opendev.org/p/python-neutronclient_deprecation We decided not to spend too much time on this goal, as long as there are other priorities. But we are going to mark this project as deprecated (README and mail); that means we'll only push patches for bugs but no new feature will be accepted. If any project is still using the CLI or the bindings and needs a new feature, it will need to migrate to SDK (that will help us on this common effort). *Port hints feature.* https://review.opendev.org/q/topic:port-hints It was considered to use the "tags" resource attribute but this functionality is very specific to the backend and the property is by default admin-only. We'll keep the current implementation (under review right now). *Stateless security groups.* This issue was found during the implementation of stateless SG in ML2/OVN. Main problems: - Semantics and mechanics defined in the documentation and the API. - DHCP and DHCPv6: as long as other FWs (ML2/OVS native and iptables) support both by default with stateless SGs, it will be implicitly supported too in ML2/OVN. That will be documented. Rationale: DHCP is a networking service and we should be able to provide IP in any case. - Metadata: no support. Users will need to explicitly define the egress/ingress rules to allow the metadata traffic (as is now in ML2/OVS FW drivers). *IPv6 prefix delegation.* https://bugs.launchpad.net/neutron/+bug/1895972 It was decided not to implement this feature during this release. At the same time it was decided to define L3 agent PD feature as "experimental" because the client used, dibbler, ended the support: https://bugs.launchpad.net/neutron/+bug/1916428 *Neutron agent status report.* https://bugs.launchpad.net/neutron/+bug/2011422 It was accepted to have a better report from the agent side of the current status, including if it was during the initial transient period, if the agent was doing a full sync, if it was restarted, etc. This functionality should also provide the ability of reporting customized status for particular agents. *DHCP agents with IPv6 metadata services.* https://bugs.launchpad.net/neutron/+bug/1953165 Several options were considered. The approved one was (3): to schedule a single IPv6 metadata agent per network, regardless of the HA number of DHCP agents. That will prevent the IPv6 DAD error now present; the DHCP agent will use that to stop spawning a new metadata agent (it will consider another DHCP agent has a running metadata server). No DB/API changes will be needed. *sRBAC, phase 2.* A new role called "service" will be created. It will identify the server to service calls during the normal operation. For example: Nova to Neutron port binding activation. During this cycle we need to identify the server-2-server calls, performed by other projects against the Neutron API and update these actions with the new role. *HWOL API for ports.* Same topic as commented during the last PTG: the HWOL functionality implies modifying the port binding, that is an admin-only property (and should always be like that). Normal users cannot create and request a port with this functionality if that port was not created before by an admin. It was also suggested (but at least rejected by me: config driven API is something now recommended) to have a configuration option to allow the creation of HWOL ports by default. *Delete on termination for Neutron ports.* The goal of this new feature is to mark the port in order to know if it needs to be deleted after the VM is deleted or not. This functionality is valid for user created ports, not Nova created ports (the last ones are always deleted when the VM is). Nova folks are investigating if "tags" API is valid or not for this RFE. If not, a new API will be proposed for Neutron ports. *Allow to specify the hardware model in the Neutron port.* The goal of this feature is to tint the port with a HW model. For example "hw_vif_model=virtio" or "hw_vif_model=e1000". Same as before, investigate if "tags" is valid or not. We are also considering improving "tags" to be a key-value model, instead of a list of strings only. *Add support for Napatech LinkVirt SmartNICs* https://review.opendev.org/c/openstack/nova-specs/+/859290 The OVS service is running on the NIC HW. It uses DPDK port representors (similar to devlink port representors but not present in kernel). There is a POC in gerrit to be reviewed (patches in the etherpad). The VNIC type was added some releases ago. The main issue of this RFE: the lack of CI support. This is why the main request for this company is to implement an external CI for zuul, adding some basic tempest tests. *Port binding activation duplication.* https://bugs.launchpad.net/neutron/+bug/1986003 It was decided to implement the current solution ( https://review.opendev.org/c/openstack/neutron/+/853281/) if a duplicated call is received by Neutron, it will raise a Conflict (409) exception, handled by Nova. *SR-IOV port tracking in Placement.* Still under definition. Several alternatives to the current ML2/SRIOV agent-resource reporting (for QoS) are presented, some of them exclusive to the current model. There is a etherpad to track the spec definition: https://etherpad.opendev.org/p/track-sriov-nics-in-placement *"skip-level" jobs this cycle (non SLURP).* It was decided: - To continue testing it but testing master against Zed. - To move the jobs to experimental and periodic queues. *ovn-bgp-agent roadmap.* It was decided to add ovn-bgp-agent to the Neutron stadium. The migration will be done during this release. The current API, provided by networking-bgpvpn, is not enough. It is under consideration to improve it or create a new one, just for OVN. It is also needed to improve the current documentation: installation, current uses, drivers available, etc. *Extend OVN db sync tool.* Now the OVN DB sync tool removes anything in the OVN NB DB that is not present in the Neutron DB. That clashes with some deployments, for example when using the OVN-IC to connect several nodes. E.g.: http://dani.foroselectronica.es/ovn-cluster-interconnection-567/ It will be proposed a pluggable interface to make the sync tool aware of these NB objects that have no counterpart in the Neutron DB but in other OVN DB (ovn-ic-nb, for example). *FIPS jobs, OS support.* Currently this functionality is only tested in CentOS8. CentOS9 has an issue with the conntrack module: https://bugs.launchpad.net/neutron/+bug/1976323 These jobs are not tested in Ubuntu in the CI because this is a paid feature. *Vancouver PTG.* We have registered the Neutron team for the Vancouver PTG. The topics to be discussed will be tracked in https://etherpad.opendev.org/p/neutron-vancouver-2023. Regards.

1 0

[openstack-ansible] Bobcat vPTG results
by Dmitriy Rabotyagov 03 Apr '23

03 Apr '23

Hi everyone, This year we didn't have many attendees, but I want to thank everyone who managed to join the discussion regarding further project plans and goals. 1. Things we'd love to work on to release 2023.1 (Antelope): * Documentation improvements. That includes: ** Since Zed we're using OVN as a default network driver, while most docs still refer to LXB in examples. We'd need to update architecture and most networking docs with regards to this change, as they might be misleading at the moment. ** Update Ironic documentation with regards to the latest changes to the role. ** Describe multi-arch deployments * Finalize internal TLS - encryption between haproxy and uwsgi backends that happens through the management network. We have couple of topics here ** Split haproxy configuration so that backends were configured per-service playbooks rather then while running haproxy one. https://review.opendev.org/q/topic:separated-haproxy-service-config ** Actually patching services to support TLS for backends https://review.opendev.org/q/topic:tls-backend ** Ensure we test in CI both TLS and non-TLS usecases for internal traffic. * Bump MariaDB version to the next LTS release - 10.11 * Adding support for leap upgrades to our upgrade script and CI * After switching to cloudsmith repo to be source for RabbitMQ and Erlang we struggle a lot with packages being obsoleted and removed randomly from the repo. With that we're going to execute following plan: ** Switch rabbitmq back to packagecloud - it contains packages for both deb and rpm ** Use minor releases of erlang rather than specific bugfix releases that will still come from cloudsmith. Hopefully that will fail less. ** File an issue to rabbitmq-erlang-deb to publish releases in github/packagecloud alike to rpms ** Track down state of ELS repo that could be used as alternative to cloudsmith given it's working reliably. * At the moment systemd services don't restart if only init overrides are applied. POC patch was proposed and this topic should track progress: https://review.opendev.org/q/topic:osa%252Fsystemd_restart_on_unit_change We will also attempt to backport this to stable branches with just the single vote as patches are quite trivial. 2. Bobcat goals * Pretty endpoint names. It's great to have a supported way of having pretty names in your endpoints, like compute.example.com. While this is completely possible to achieve with current codebase, we have ways to improve and ease the process of doing so. ** haproxy maps we've added should simplify the process a lot ** Add a boolean variable, that instead of concatenating vip_address + port, do service_type + vip_address in roles with ability of global override ** Fix certificates to include SAN or wildcard for all endpoint names ** document a way of doing that * There is a long-going confusion about variables external/internal_lb_vip_address as they could be either IP or an FQDN and then needing other vars for the IP in haproxy/keepalived. We can attempt to replace this variable with a more meaningful or intuitive one. ** With that change we also have a chance of fixing inconsistent service names (url vs. uri) across roles * At the moment PKI keys/certificates can't be stored in a trusted storage, as community.crypto.openssl_privatekey does require keys to be files on filesystem. With that we can replace such module with openssl_privatekey_pipe as then it can use different lookups. ** With that it would be also great to re-think/document a way of storing user-secrets outside of using ansible-vault (that's trivial). * With RabbitMQ 4.0 HA queues that are used now by default are going to be removed. Quorum queues can be used instead to provide HA for queues. Though they have quite different concepts. In order to migrate to them, the exchange needs to be re-created with the enabled persistence option. Easiest way to do that would be re-creating vhosts, which will result in significant downtime for services like nova or neutron, but that's the price that should be paid for the migration/upgrade. Topic can be tracked here: https://review.opendev.org/q/topic:osa%252Fquorum_queues * ansible-core 2.14 requires >=py39, so Ubuntu 20.04 support will be dropped early in the cycle. * We also discussed resuming efforts of fucntional testing with molecule. We came up with following requirements/pain ponts: ** avoid duplicated boilerplate in all repos ** a way to run tests locally, document that ** clean up current tox/test for projects to avoid confusion * We agreed on having a common role for managing openstack resources, like images, networks, flavors, etc. This is valuable not only for operators, but also for our service roles, like octavia or tempest, where we're creating specific resources in pretty different ways each time. Work has started in the plugins repo with ambition to move that to openstack collection in the future: https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/878794 * We're not sure about current IPv6-only deployments at the moment, when talking about public networks. While private networks is not smth we want to focus on right now, we need to ensure that IPv6-only public networks are supported and documented. Volunteers are highly appreciated for this work. * Releasing skyline role. While we have a POC role, Skyline doesn't meet internal requirements of OSA maintainers. So while we'd love to see role being released, it's unlikely to be a priority for majority of current team. So we're calling for volunteers to make Skyline role ready for releasing in Bobcat. * We also agreed to work on Core reviewers promotion/demotion policies and process, that should be published on Core reviewers page: https://docs.openstack.org/openstack-ansible/latest/contributor/core-review…

1 0

[ptl][monasca] Need PTL volunteer for OpenStack Monasca
by Dmitriy Rabotyagov 03 Apr '23

03 Apr '23

Hi everyone, I am reaching out to find interested parties to maintain the OpenStack Monasca project. At the moment there is no PTL candidate for the next cycle (2023.2), and it is on the leaderless project list [1]. In case there are no interested parties to lead this project, TC might take the decision to retire it. If you're interested to help out with the project maintenance to keep it alive, please kindly reach out to TC either by replying to this email or joining us on IRC OFTC network in #openstack-tc channel. [1] https://etherpad.opendev.org/p/2023.2-leaderless

1 0

[tripleo] Removal of TripleO Master Integration and Component Lines
by Ronelle Landy 03 Apr '23

03 Apr '23

Hello All, Removal of TripleO Master Integration and Component Lines Per the decision to not maintain TripleO after the Zed release [1], the master/main integration and component lines are being removed in the following patches: https://review.rdoproject.org/r/c/config/+/48073 https://review.rdoproject.org/r/c/config/+/48074 https://review.rdoproject.org/r/c/rdo-jobs/+/48075 The last promoted release of master through TripleO is: https://trunk.rdoproject.org/centos9-master/current-tripleo/delorean.repo (hash: ddce25bad764dde7e0515094b4d40471), which was promoted on 03/28/2023. Check/gate testing for the master branch is in process of being removed as well. [1] https://review.opendev.org/c/openstack/governance/+/878799

3 2

[kolla] horizon image build failed
by Satish Patel 02 Apr '23

02 Apr '23

Folks, All other images build successfully but when i am trying to build horizon which failed with following error: $ kolla-build --registry docker-reg:4000 -b ubuntu -t source --tag zed horizon INFO:kolla.common.utils.horizon:Collecting XStatic-Dagre>=0.6.4.0 INFO:kolla.common.utils.horizon: Downloading XStatic_Dagre-0.6.4.1-py2.py3-none-any.whl (140 kB) INFO:kolla.common.utils.horizon: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 140.0/140.0 kB 14.4 MB/s eta 0:00:00 INFO:kolla.common.utils.horizon:Collecting XStatic-Dagre-D3>=0.4.17.0 INFO:kolla.common.utils.horizon: Downloading XStatic_Dagre_D3-0.4.17.0-py2.py3-none-any.whl (357 kB) INFO:kolla.common.utils.horizon: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 357.4/357.4 kB 13.5 MB/s eta 0:00:00 INFO:kolla.common.utils.horizon:Requirement already satisfied: XStatic-Font-Awesome>=4.7.0.0 in /var/lib/kolla/venv/lib/python3.10/site-packages (from vitrage-dashboard==3.6.1.dev2) (4.7.0.0) INFO:kolla.common.utils.horizon:Collecting XStatic-Graphlib>=2.1.7.0 INFO:kolla.common.utils.horizon: Downloading XStatic_Graphlib-2.1.7.0-py2.py3-none-any.whl (51 kB) INFO:kolla.common.utils.horizon: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 51.5/51.5 kB 114.3 MB/s eta 0:00:00 INFO:kolla.common.utils.horizon:Requirement already satisfied: XStatic-jQuery>=1.8.2.1 in /var/lib/kolla/venv/lib/python3.10/site-packages (from vitrage-dashboard==3.6.1.dev2) (1.12.4.1) INFO:kolla.common.utils.horizon:Collecting XStatic-lodash>=4.16.4.1 INFO:kolla.common.utils.horizon: Downloading XStatic_lodash-4.16.4.2-py3-none-any.whl (167 kB) INFO:kolla.common.utils.horizon: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 167.9/167.9 kB 12.4 MB/s eta 0:00:00 INFO:kolla.common.utils.horizon:Collecting XStatic-moment>=2.8.4.1 INFO:kolla.common.utils.horizon: Downloading XStatic_moment-2.8.4.3-py3-none-any.whl (58 kB) INFO:kolla.common.utils.horizon: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 58.0/58.0 kB 66.7 MB/s eta 0:00:00 INFO:kolla.common.utils.horizon:Collecting XStatic-Moment-Timezone>=0.5.22.0 INFO:kolla.common.utils.horizon: Downloading XStatic-Moment-Timezone-0.5.22.0.tar.gz (99 kB) INFO:kolla.common.utils.horizon: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 99.7/99.7 kB 45.1 MB/s eta 0:00:00 INFO:kolla.common.utils.horizon: Preparing metadata (setup.py): started INFO:kolla.common.utils.horizon: Preparing metadata (setup.py): finished with status 'error' INFO:kolla.common.utils.horizon: error: subprocess-exited-with-error INFO:kolla.common.utils.horizon: INFO:kolla.common.utils.horizon: × python setup.py egg_info did not run successfully. INFO:kolla.common.utils.horizon: │ exit code: 1 INFO:kolla.common.utils.horizon: ╰─> [6 lines of output] INFO:kolla.common.utils.horizon: Traceback (most recent call last): INFO:kolla.common.utils.horizon: File "<string>", line 2, in <module> INFO:kolla.common.utils.horizon: File "<pip-setuptools-caller>", line 34, in <module> INFO:kolla.common.utils.horizon: File "/tmp/pip-install-dqag1zef/xstatic-moment-timezone_60eeadc1dfb9492781fe3ca90e3b95c2/setup.py", line 2, in <module> INFO:kolla.common.utils.horizon: from xstatic.pkg import moment_timezone as xs INFO:kolla.common.utils.horizon: ImportError: cannot import name 'moment_timezone' from 'xstatic.pkg' (unknown location) INFO:kolla.common.utils.horizon: [end of output] INFO:kolla.common.utils.horizon: INFO:kolla.common.utils.horizon: note: This error originates from a subprocess, and is likely not a problem with pip. INFO:kolla.common.utils.horizon: INFO:kolla.common.utils.horizon:error: metadata-generation-failed INFO:kolla.common.utils.horizon:× Encountered error while generating package metadata. INFO:kolla.common.utils.horizon:╰─> See above for output. INFO:kolla.common.utils.horizon:note: This is an issue with the package mentioned above, not pip. INFO:kolla.common.utils.horizon:hint: See above for details. INFO:kolla.common.utils.horizon: INFO:kolla.common.utils.horizon:Removing intermediate container e6cd437ba529 ERROR:kolla.common.utils.horizon:Error'd with the following message ERROR:kolla.common.utils.horizon:The command '/bin/sh -c ln -s horizon-source/* horizon && sed -i /^horizon=/d /requirements/upper-constraints.txt && SETUPTOOLS_USE_DISTUTILS=stdlib python3 -m pip --no-cache-dir install --upgrade -c /requirements/upper-constraints.txt /horizon && mkdir -p /etc/openstack-dashboard && cp -r /horizon/openstack_dashboard/conf/* /etc/openstack-dashboard/ && cp /horizon/openstack_dashboard/local/local_settings.py.example /etc/openstack-dashboard/local_settings && cp /horizon/manage.py /var/lib/kolla/venv/bin/manage.py && if [ "$(ls /plugins)" ]; then SETUPTOOLS_USE_DISTUTILS=stdlib python3 -m pip --no-cache-dir install --upgrade -c /requirements/upper-constraints.txt /plugins/*; fi && for locale in /var/lib/kolla/venv/lib/python3.10/site-packages/*/locale; do (cd ${locale%/*} && /var/lib/kolla/venv/bin/django-admin compilemessages) done && chmod 644 /usr/local/bin/kolla_extend_start' returned a non-zero code: 1 INFO:kolla.common.utils:========================= INFO:kolla.common.utils:Successfully built images INFO:kolla.common.utils:========================= INFO:kolla.common.utils:base INFO:kolla.common.utils:openstack-base INFO:kolla.common.utils:=========================== INFO:kolla.common.utils:Images that failed to build INFO:kolla.common.utils:===========================

4 8

[Horizon][policy][keystone][yoga] allow __member__ role to create user
by James Leong 02 Apr '23

02 Apr '23

Hi all, I have deploy openstack in yoga version using kolla-ansible. I noticed that only admin role can create a new user. Would it be possible to also allow user with __member__ role to create user in the respective project. Best, James

2 3

Unable to create external vlan network | openstack wallaby
by Swogat Pradhan 01 Apr '23

01 Apr '23

Hi, I have an openstack setup which already has vm's running on it and other external and internal networks. I configured DCN recently adding 2 additional sites and while testing the services, I found I am unable to create external vlan networks for some reason, whereas other networks like geneve are getting created. Horizon is throwing 1 error and that is Internal server error, i am not sure why this happened, or how to solve this issue. Can someone please suggest? With regards, Swogat Pradhan

1 3