April 2023 - openstack-discuss - lists.openstack.org

[oslo.messaging][nova][neutron] quorum queues
by Arnaud Morin 13 Apr '23

13 Apr '23

Hey team, I am currently looking into using quorum queues (see patch [1] introduced in oslo_messaging) for openstack services (nova, neutron mostly). After switching the value to true and bootstraping a new cluster, all queues are quorum except the fanout and reply queues. I understand that transient queues are managed differently in openstack, but why is it so? I am asking this because when using rabbit in cluster mode, the quorum queues are replicated accross the cluster. But because the fanout and reply queues are not replicated, if one of the cluster is shutting down during an instance boot, the boot may endup in error (the reply queue may be on the faulty node). Is there any reason to not enable quorum on transient queues? [1] https://github.com/openstack/oslo.messaging/commit/7e8acbf87070354cd2438f3a…

1 0

Updating OS-TRAITS in placement stable/yoga and stable/zed
by Karl Kloppenborg 13 Apr '23

13 Apr '23

Hello Placement and OS-Traits teams. Karl from Openstack-Helm. After debugging a number of issues in Cyborg, it has become necessary to update the pinned OS-TRAITs version in stable/yoga and stable/zed. I have submitted the following two reviews for updating the pinned versions to 2.10.0 https://review.opendev.org/c/openstack/placement/+/880249 https://review.opendev.org/c/openstack/placement/+/880249 By merging these changes, it will allow us to continue to use vGPU capabilities in cyborg in the stable/yoga and stable/zed onwards branches. Kind Regards, -- Karl Kloppenborg, (BCompSc, CNCF-[KCNA, CKA, CKAD], LFCE, CompTIA Linux+ XK0-004) Managing Director, Invention Labs.

3 2

[Cloudkitty][ptg] CloudKitty 2023.2 Bobcat PTG outcome
by Rafael Weingärtner 13 Apr '23

13 Apr '23

Hello, Thanks to everyone who attended the CloudKitty PTG sessions, we covered a lot during the meeting. The Etherpad is available at [1]. [1] https://etherpad.opendev.org/p/march2023-ptg-cloudkitty -- Rafael Weingärtner

1 0

Does cinder-backup support multi-backend?
by wodel youchi 13 Apr '23

13 Apr '23

Hi, Could cinder-backup be configured with multiple-backends, ceph and nfs for example? Regards.

5 9

[kolla-ansible][mariadb] socat: Connection reset by peer
by Satish Patel 12 Apr '23

12 Apr '23

Folks, I am getting the following errors in logs constantly and continually. I have 3 node galera clusters. Everything looks good except this error making me nervous. I have deployed Zed release of kolla openstack. ==> /var/log/kolla/mariadb/mariadb-clustercheck.log <== 2023/04/12 02:42:16 socat[1574755] W read(7, 0x5653c4a4c000, 8192): Connection reset by peer 2023/04/12 02:42:16 socat[1574761] W read(7, 0x5653c4a4c000, 8192): Connection reset by peer 2023/04/12 02:42:16 socat[1574755] W read(6, 0x5653c4a4c000, 8192): Connection reset by peer 2023/04/12 02:42:16 socat[1574761] W read(6, 0x5653c4a4c000, 8192): Connection reset by peer 2023/04/12 02:42:18 socat[1574767] W read(7, 0x5653c4a4c000, 8192): Connection reset by peer 2023/04/12 02:42:18 socat[1574767] W read(6, 0x5653c4a4c000, 8192): Connection reset by peer

2 2

[kolla-ansible][yoga] how to configure cinder to connect to iscsi target?
by wodel youchi 12 Apr '23

12 Apr '23

Hi, I am trying to configure cinder to connect to an iscsi target. I am using targetcli as my target server. I have tested the connection from the node using iscsiadm and it works from the OS. Now I am trying to use the iscsid container to do the job. I added this to my globals.yml enable_iscsid: "yes" enable_cinder_backend_iscsi: "yes" enable_cinder_backend_lvm: "no" And I created the cinder.conf file like this [DEFAULT] enabled_backends = rbd1,nfs-1,*lvmtnas * # add follows to the end [lvmtnas] target_helper = lioadm target_protocol = iscsi target_ip_address = 20.1.0.32 (my target server) volume_backend_name = lvmtnas # volume group name created volume_group = *vg_volume01 * volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver use_chap_auth = false I did create the vg_volume01 on the LUN prior to the test. I did a deploy, the container has been created, but nothing happened, it's like I am missing something. docker logs iscsid fome one the compute nodes + sudo -E kolla_set_configs INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files INFO:__main__:Writing out command to execute ++ cat /run_command + CMD='iscsid -d 8 -f' + ARGS= + sudo kolla_copy_cacerts + [[ ! -n '' ]] + . kolla_extend_start ++ [[ ! -f /etc/iscsi/initiatorname.iscsi ]] + echo 'Running command: '\''iscsid -d 8 -f'\''' + exec iscsid -d 8 -f Running command: 'iscsid -d 8 -f' iscsid: sysfs_init: sysfs_path='/sys' iscsid: in ctldev_open iscsid: created NETLINK_ISCSI socket... iscsid: InitiatorName=iqn.1994-05.com.redhat:2a318909ad8 iscsid: InitiatorName=iqn.1994-05.com.redhat:2a318909ad8 iscsid: InitiatorAlias=computehci01 iscsid: Max file limits 1048576 1048576 iscsid: current time 258465 iscsid: nothing on pend_list, deactivating alarm Any idea? Regards.

2 4

[neutron] Deprecation of the neutronclient CLI code
by Rodolfo Alonso Hernandez 12 Apr '23

12 Apr '23

Hello all: Since this release [1], the python-neutronclient project will no longer provide CLI support. Any Neutron related CLI command should be executed using python-openstackclient (OSC). This deprecation and removal was announced in [2] and was initially planned for Zed release. We have been working during the last releases to finish the migration and provide all the commands in OSC. In case you find any missing command, argument or output, please open a bug in the Neutron site of launchpad and we'll address it. You can also install a previous (Antelope) version of neutronclient if you missed the migration window. The python-neutronclient will still be available to provide the Networking API python bindings [3]. Regards. [1]https://review.opendev.org/c/openstack/python-neutronclient/+/871711 [2] https://docs.openstack.org/python-neutronclient/latest/contributor/transiti… [3]https://docs.openstack.org/python-neutronclient/ocata/usage/library.html

1 0

CYBORG HTTP400 create trait OWNER_CYBORG
by Karl Kloppenborg 12 Apr '23

12 Apr '23

Hi Cyborg Team, Karl from the Openstack-Helm group. I’ve hit a really an issue I cannot seem to overcome, Cyborg is detecting GPU Profiles from a NVIDIA A40, and it goes to create the record but cyborg-conductor errors with ``` [cyborg-conductor-6fc477ff8b-cw5lf cyborg-conductor] 2023-04-12 06:17:37.581 1 INFO cyborg.conductor.manager [None req-600fcdcf-a620-466a-ba6c-cdfbf6bf83e2 - - - - - -] Failed to add device DriverDevice(controlpath_id=DriverControlPathID,created_at=<?>,deployable_list=[DriverDeployable],model='NVIDIA Corporation Device',std_board_info='{"product_id": "2235", "controller": "3D controller"}',stub=False,type='GPU',updated_at=<?>,vendor='10de',vendor_board_info='{"vendor_info": "gpu_vb_info"}'). Reason: Failed to create trait OWNER_CYBORG: HTTP 400: {"errors": [{"status": 400, "title": "Bad Request", "detail": "The server could not comply with the request since it is either malformed or otherwise incorrect.\n\n The trait is invalid. A valid trait must be no longer than 255 characters, start with the prefix \"CUSTOM_\" and use following characters: \"A\"-\"Z\", \"0\"-\"9\" and \"_\" ", "request_id": "req-24634fad-b312-4ad3-9482-09d6df2b26c2"}]} ``` It appears Cyborg is attempting to create the trait OWNER_CYBORG but I cannot seem to get it to stop that, can I please have some assistance from the Cyborg team to overcome this? Kind Regards, -- Karl Kloppenborg, (BCompSc, CNCF-[KCNA, CKA, CKAD], LFCE, CompTIA Linux+ XK0-004) Managing Director, Invention Labs.

1 0

In Person PTG June 2023 Teams List
by Kendall Nelson 11 Apr '23

11 Apr '23

Hello Everyone! The June 2023 Project Teams List is official! Projects + Teams - OpenInfra Foundation Teams - - Diversity & Inclusion Working Group - Environmental Sustainability Working Group - Kata Containers - StarlingX - OpenStack - - Service Teams - - Cinder - Cyborg - Designate - Heat - Ironic - Keystone - Manila - Neutron - Nova - OpenStack-Ansible - Puppet OpenStack - QA - SDK/CLI/AOC - Tacker - Telemetry - Other OpenStack Teams - - Security SIG - Technical Committee - Scientific SIG If your team was planning to meet and isn’t in this list, please contact ptg(a)openinfra.dev IMMEDIATELY. In a couple weeks, I will be contacting moderators to sign up for time via the PTGBot[1] once I have it configured for the teams that are signed up to attend the event. Otherwise, please don't forget to register[2] — prices go up May 5th! Thanks! -Kendall (diablo_rojo) [1] PTGBot Docs: https://github.com/openstack/ptgbot#open-infrastructure-ptg-bot [2] PTG Registration: https://vancouver2023.openinfra.dev/

1 0

[openstack][sharing][kolla ansible]Problems when 1 of 3 controller was be down
by Nguyễn Hữu Khôi 11 Apr '23

11 Apr '23

Hello everyone. I want to summary for who meets problems with Openstack when deploy cluster with 3 controller using Kolla Ansible Scenario: 1 of 3 controller is down 1. Login horizon and use API such as nova, cinder will be very slow fix by: nano: kolla-ansible/ansible/roles/heat/templates/heat.conf.j2 kolla-ansible/ansible/roles/nova/templates/nova.conf.j2 kolla-ansible/ansible/roles/keystone/templates/keystone.conf.j2 kolla-ansible/ansible/roles/neutron/templates/neutron.conf.j2 kolla-ansible/ansible/roles/cinder/templates/cinder.conf.j2 or which service need caches add as below [cache] backend = oslo_cache.memcache_pool enabled = True memcache_servers = {{ kolla_internal_vip_address }}:{{ memcached_port }} memcache_dead_retry = 0.25 memcache_socket_timeout = 900 https://review.opendev.org/c/openstack/kolla-ansible/+/849487 but it is not the end 2. Cannot launch instance or mapping block device(stuck at this step) nano kolla-ansible/ansible/roles/rabbitmq/templates/definitions.json.j2 "policies":[ {"vhost": "/", "name": "ha-all", "pattern": "^(?!(amq\.)|(.*_fanout_)|(reply_)).*", "apply-to": "all", "definition": {"ha-mode":"all"}, "priority":0}{% if project_name == 'outward_rabbitmq' %}, {"vhost": "{{ murano_agent_rabbitmq_vhost }}", "name": "ha-all", "pattern": ".*", "apply-to": "all", "definition": {"ha-mode":"all"}, "priority":0} {% endif %} ] nano /etc/kollla/global.conf [oslo_messaging_rabbit] kombu_reconnect_delay=0.5 https://bugs.launchpad.net/oslo.messaging/+bug/1993149 https://docs.openstack.org/large-scale/journey/configure/rabbitmq.html I used Xena 13.4 and Yoga 14.8.1. Above bugs are critical, but I see that it was not fixed. I am just an operator and I want to share what I encountered for new people who come to Openstack Nguyen Huu Khoi

6 11

Re: StackOver Flow Error: 2103 ERROR oslo.messaging._drivers.impl_rabbit Connection failed: failed to resolve broker hostname
by Dmitriy Rabotyagov 11 Apr '23

11 Apr '23

If you're going to go prod I would for sure use some deployment tool rather then manual deployment, as further operations and scaling up would be way easier. What error do you get with existing hosts record? вт, 11 апр. 2023 г., 14:57 Trey Fleishman <nfleishman(a)student.ccaeagles.org >: > Hello Dmitriy, > > That is what I thought too, in my host file I define the hostname as the > correct IP address, and if I change the "controller" hostname to some > factitious IP I get a different error. I've tried putting the IP instead of > the hostname, but to no avail. I can successfully ping the hostname from > the machine. I would also rather not use a tool as I plan to go production > once I get my two node setup functional. I am not sure if this might be a > configuration issue, or maybe a bug. > Trey Fleishman > Student and IT Intern > (704) 750 - 1740 > *Calvary Chapel Fort Lauderdale* > 2401 W. Cypress Creek Rd., Ft. Lauderdale, FL 33309 > CalvaryFTL.org > > > On Tue, Apr 11, 2023 at 12:54 AM Dmitriy Rabotyagov < > noonedeadpunk(a)gmail.com> wrote: > >> Hi Trey, >> >> Have you defined a record in your /etc/hosts for "controller", as >> according to the error service simply can not resolve this hostname to an >> IP address. >> >> Also I can suggest trying out one of the following OpenStack deployment >> tools: >> https://docs.openstack.org/zed/deploy/index.html >> >> Most of them have All-In-One guides that can spawn an opestack sandbox to >> play with. >> >> вт, 11 апр. 2023 г., 00:21 Trey Fleishman < >> nfleishman(a)student.ccaeagles.org>: >> >>> Hello all, >>> Recently I tried installing Openstack via this >>> <https://docs.openstack.org/install-guide/openstack-services.html> >>> guide. Upon completion of all the steps including the dashboard step, I was >>> greeted with this error inside the nova_schedular.log file, after I tried >>> creating an instance. >>> >>> 2103 ERROR oslo.messaging._drivers.impl_rabbit Connection failed: failed >>> to resolve broker hostname: OSError: failed to resolve broker hostname >>> >>> When I try to create an instance it hangs in the "scheduling" state and >>> throws this error. I have not configured anything in the "oslo.messaging" >>> section of the Nova config file. I believe that under the [default] section >>> this is what's causing the issue, but the password and hostname are fine. >>> >>> transport_url = >>> rabbit://openstack:BnE9LLfCtjdPvpEnZUnPytSyrfUxE8th@controller >>> >>> Thanks! >>> >>

2 1

[keystone] Weekly Meeting Cancelled (11-Apr-2023)
by Dave Wilde 11 Apr '23

11 Apr '23

Hi Folks, Unfortunately I have to cancel the weekly meeting today. Please reach out to me directly via email if you need anything. Due to equipment failure my IRC availability is spotty. Thanks! /Dave

1 0

[Kolla-ansible][Yoga] Cannot live migrate VMs
by wodel youchi 11 Apr '23

11 Apr '23

Hi, I cannot live-migrate any VM I have. All my compute nodes are enabled. I have a few VMs, I am in the testing phase. I tried to live-migrate toward an empty node, then toward a node which already had a VM in it. It is the same problem. In the log files I have : Source node : Source > ==> /var/log/kolla/nova/nova-compute.log <== > 2023-04-09 11:32:49.419 7 INFO nova.compute.resource_tracker > [req-4e91d562-d7ad-4585-a827-3db6277fd81e - - - - -] [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Updating resource usage from > migration f60eb077-8e1c-404a-aca5-64e5609ca02c > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [-] [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Pre live migration failed at > computehci02: oslo_messaging.rpc.client.RemoteError: Remote error: > ClientException Gateway Time-out (HTTP 504) > ['Traceback (most recent call last):\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", > line 165, in _process_incoming\n res = > self.dispatcher.dispatch(message)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", > line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, > args)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", > line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", > line 72, in wrapped\n context, exc, binary)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 227, in __exit__\n self.force_reraise()\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 200, in force_reraise\n raise self.value\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", > line 63, in wrapped\n return f(self, context, *args, **kw)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/utils.py", > line 1439, in decorated_function\n return function(self, context, *args, > **kwargs)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", > line 213, in decorated_function\n kwargs[\'instance\'], e, > sys.exc_info())\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 227, in __exit__\n self.force_reraise()\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 200, in force_reraise\n raise self.value\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", > line 201, in decorated_function\n return function(self, context, *args, > **kwargs)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", > line 8367, in pre_live_migration\n bdm.save()\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 227, in __exit__\n self.force_reraise()\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 200, in force_reraise\n raise self.value\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", > line 8304, in pre_live_migration\n connector=connector, > mountpoint=bdm.device_name)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", > line 395, in wrapper\n res = method(self, ctx, *args, **kwargs)\n', ' > File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", > line 433, in wrapper\n res = method(self, ctx, volume_id, *args, > **kwargs)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", > line 813, in attachment_create\n instance_uuid=instance_id)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 227, in __exit__\n self.force_reraise()\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 200, in force_reraise\n raise self.value\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", > line 801, in attachment_create\n volume_id, _connector, instance_id)\n', > ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/api_versions.py", > line 421, in substitution\n return method.func(obj, *args, **kwargs)\n', > ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/v3/attachments.py", > line 39, in create\n retval = self._create(\'/attachments\', body, > \'attachment\')\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/base.py", > line 306, in _create\n resp, body = self.api.client.post(url, > body=body)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", > line 227, in post\n return self._cs_request(url, \'POST\', **kwargs)\n', > ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", > line 215, in _cs_request\n return self.request(url, method, > **kwargs)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", > line 201, in request\n raise exceptions.from_response(resp, body)\n', > 'cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504)\n']. > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Traceback (most recent call last): > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", > line 8501, in _do_pre_live_migration_from_source > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] block_migration, disk, dest, > migrate_data) > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/rpcapi.py", > line 948, in pre_live_migration > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] **msg_args) > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/client.py", > line 192, in call > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] retry=self.retry, > transport_options=self.transport_options) > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/transport.py", > line 128, in _send > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] > transport_options=transport_options) > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", > line 691, in send > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] > transport_options=transport_options) > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", > line 681, in _send > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] raise result > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] > oslo_messaging.rpc.client.RemoteError: Remote error: ClientException > Gateway Time-out (HTTP 504) > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] ['Traceback (most recent call > last):\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", > line 165, in _process_incoming\n res = > self.dispatcher.dispatch(message)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", > line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, > args)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", > line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", > line 72, in wrapped\n context, exc, binary)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 227, in __exit__\n self.force_reraise()\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 200, in force_reraise\n raise self.value\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", > line 63, in wrapped\n return f(self, context, *args, **kw)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/utils.py", > line 1439, in decorated_function\n return function(self, context, *args, > **kwargs)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", > line 213, in decorated_function\n kwargs[\'instance\'], e, > sys.exc_info())\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 227, in __exit__\n self.force_reraise()\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 200, in force_reraise\n raise self.value\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", > line 201, in decorated_function\n return function(self, context, *args, > **kwargs)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", > line 8367, in pre_live_migration\n bdm.save()\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 227, in __exit__\n self.force_reraise()\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 200, in force_reraise\n raise self.value\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", > line 8304, in pre_live_migration\n connector=connector, > mountpoint=bdm.device_name)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", > line 395, in wrapper\n res = method(self, ctx, *args, **kwargs)\n', ' > File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", > line 433, in wrapper\n res = method(self, ctx, volume_id, *args, > **kwargs)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", > line 813, in attachment_create\n instance_uuid=instance_id)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 227, in __exit__\n self.force_reraise()\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 200, in force_reraise\n raise self.value\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", > line 801, in attachment_create\n volume_id, _connector, instance_id)\n', > ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/api_versions.py", > line 421, in substitution\n return method.func(obj, *args, **kwargs)\n', > ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/v3/attachments.py", > line 39, in create\n retval = self._create(\'/attachments\', body, > \'attachment\')\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/base.py", > line 306, in _create\n resp, body = self.api.client.post(url, > body=body)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", > line 227, in post\n return self._cs_request(url, \'POST\', **kwargs)\n', > ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", > line 215, in _cs_request\n return self.request(url, method, > **kwargs)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", > line 201, in request\n raise exceptions.from_response(resp, body)\n', > 'cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504)\n']. > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] > 2023-04-09 11:33:30.157 7 INFO nova.compute.manager [-] [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Swapping old allocation on > dict_keys(['2857ae62-3519-4b13-b700-858c827dd78d']) held by migration > f60eb077-8e1c-404a-aca5-64e5609ca02c for instance In the destination node : > nova-compute.log > > > > > *2023-04-09 11:32:30.052 7 WARNING os_brick.initiator.connectors.nvmeof > [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 > b0f76b5c6dcb457fa716762bbf954837 - default default] Process execution error > in _get_host_uuid: Unexpected error while running command.Command: blkid > overlay -s UUID -o valueExit code: 2Stdout: ''Stderr: '': > oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while > running command.* > > 2023-04-09 11:33:05.354 7 INFO nova.compute.resource_tracker > [req-9d480442-87f7-4d4e-a866-c46d645ac7f2 - - - - -] [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Updating resource usage from > migration f60eb077-8e1c-404a-aca5-64e5609ca02c > 2023-04-09 11:33:05.392 7 WARNING nova.compute.resource_tracker > [req-9d480442-87f7-4d4e-a866-c46d645ac7f2 - - - - -] Instance > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7 has been moved to another host > computehci01(computehci01). There are allocations remaining against the > source host that might need to be removed: {'resources': {'VCPU': 1, > 'MEMORY_MB': 2048}}. > 2023-04-09 11:33:30.086 7 ERROR nova.volume.cinder > [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 > b0f76b5c6dcb457fa716762bbf954837 - default default] [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Create attachment failed for volume > 3d7bd0af-5e4c-48fc-a027-f6c9dd6280bb. Error: Gateway Time-out (HTTP 504) > Code: 504: cinderclient.exceptions.ClientException: Gateway Time-out (HTTP > 504) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server > [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 > b0f76b5c6dcb457fa716762bbf954837 - default default] Exception during > message handling: cinderclient.exceptions.ClientException: Gateway Time-out > (HTTP 504) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server Traceback (most > recent call last): > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", > line 165, in _process_incoming > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server res = > self.dispatcher.dispatch(message) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", > line 309, in dispatch > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return > self._do_dispatch(endpoint, method, ctxt, args) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", > line 229, in _do_dispatch > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server result = > func(ctxt, **new_args) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", > line 72, in wrapped > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server context, > exc, binary) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 227, in __exit__ > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server > self.force_reraise() > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 200, in force_reraise > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise > self.value > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", > line 63, in wrapped > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return > f(self, context, *args, **kw) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/utils.py", > line 1439, in decorated_function > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return > function(self, context, *args, **kwargs) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", > line 213, in decorated_function > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server > kwargs['instance'], e, sys.exc_info()) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 227, in __exit__ > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server > self.force_reraise() > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 200, in force_reraise > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise > self.value > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", > line 201, in decorated_function > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return > function(self, context, *args, **kwargs) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", > line 8367, in pre_live_migration > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server bdm.save() > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 227, in __exit__ > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server > self.force_reraise() > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 200, in force_reraise > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise > self.value > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", > line 8304, in pre_live_migration > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server > connector=connector, mountpoint=bdm.device_name) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", > line 395, in wrapper > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server res = > method(self, ctx, *args, **kwargs) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", > line 433, in wrapper > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server res = > method(self, ctx, volume_id, *args, **kwargs) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", > line 813, in attachment_create > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server > instance_uuid=instance_id) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 227, in __exit__ > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server > self.force_reraise() > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 200, in force_reraise > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise > self.value > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", > line 801, in attachment_create > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server volume_id, > _connector, instance_id) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/api_versions.py", > line 421, in substitution > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return > method.func(obj, *args, **kwargs) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/v3/attachments.py", > line 39, in create > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server retval = > self._create('/attachments', body, 'attachment') > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/base.py", > line 306, in _create > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server resp, body = > self.api.client.post(url, body=body) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", > line 227, in post > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return > self._cs_request(url, 'POST', **kwargs) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", > line 215, in _cs_request > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return > self.request(url, method, **kwargs) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", > line 201, in request > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise > exceptions.from_response(resp, body) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server > cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server > 2023-04-09 11:33:30.238 7 INFO nova.virt.block_device > [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 > b0f76b5c6dcb457fa716762bbf954837 - default default] [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Attempting to driver detach volume > 3d7bd0af-5e4c-48fc-a027-f6c9dd6280bb from mountpoint /dev/vda > 2023-04-09 11:33:30.239 7 WARNING nova.virt.block_device > [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 > b0f76b5c6dcb457fa716762bbf954837 - default default] [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Detaching volume from unknown instance > 2023-04-09 11:33:30.240 7 WARNING nova.virt.libvirt.driver > [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 > b0f76b5c6dcb457fa716762bbf954837 - default default] [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] During detach_volume, instance > disappeared.: nova.exception.InstanceNotFound: Instance > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7 could not be found. > 2023-04-09 11:33:30.678 7 INFO nova.virt.libvirt.driver [-] [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Instance destroyed successfully. > 2023-04-09 11:33:30.682 7 INFO os_vif > [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 > b0f76b5c6dcb457fa716762bbf954837 - default default] Successfully unplugged > vif > VIFBridge(active=True,address=fa:16:3e:9b:92:46,bridge_name='qbr46651303-65',has_traffic_filtering=True,id=46651303-65e1-4809-95cc-7c2d97acf16b,network=Network(532470c1-dd2f-4505-8557-ba24e9d4062b),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap46651303-65') > All I can see is *b0f76b5c6dcb457fa716762bbf954837 - default default] Process execution error in _get_host_uuid: Unexpected error while running command.Command: blkid overlay -s UUID -o valueExit code: 2Stdout: ''Stderr: '': oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.* Searching the web I didn't find anything useful about this error, and I don't know if it's the origin of the problem. My default storage is ceph rbd, I have an HCI deployment, and all my VMs are for now created on Ceph only. But I have configured two other cinder backends, NFS and LVM, but they are node used for now. Thanks in advance. Regards.

3 4

[neutron] bug deputy report for week of 2023-04-03
by Bence Romsics 11 Apr '23

11 Apr '23

Hi All, We had the following new bugs reported last week: Critical: * https://bugs.launchpad.net/neutron/+bug/2015065 [tempest] "test_security_group_rules_create" unstable in "neutron-ovs-grenade-dvr-multinode" unassigned High: * https://bugs.launchpad.net/neutron/+bug/2015364 [skip-level] OVN tests constantly failing unassigned * https://bugs.launchpad.net/neutron/+bug/2015376 [OVN] min-BW rule, defined in the LSP.options field, must be accompanied with a max-BW value unassigned * https://bugs.launchpad.net/neutron/+bug/2015728 ovs/ovn source(with master branch) deployments broken workarounds proposed/merged by Yatin: https://review.opendev.org/c/openstack/neutron/+/879938 https://review.opendev.org/c/openstack/ovn-octavia-provider/+/879939 * https://bugs.launchpad.net/neutron/+bug/2015090 neutron.agent.dhcp.agent TypeError: 'bool' object is not subscriptable Technically unassigned, but I believe Sahid may be working on it. Medium: * https://bugs.launchpad.net/neutron/+bug/2015388 Deleting network does not remove network namespace fix proposed by Brian: https://review.opendev.org/c/openstack/neutron/+/880006 * https://bugs.launchpad.net/neutron/+bug/2015377 [ovn] If dhcp port is deleted from neutron, it is never recreated fix proposed by Brian: https://review.opendev.org/c/openstack/neutron/+/879913 RFE: * https://bugs.launchpad.net/neutron/+bug/2015471 [RFE] Add ERSPAN for tap-as-a-service with OVS and OVN Triage still in progress: * https://bugs.launchpad.net/neutron/+bug/2015449 Remote security groups don't allow traffic from floating IPs This issue may be undocumented behavior, the following question could be discussed on the weekly meeting: https://meetings.opendev.org/irclogs/%23openstack-neutron/%23openstack-neut… * https://bugs.launchpad.net/neutron/+bug/2015275 Trunk port update_subport_bindings RPC call not processed by RPC workers Cheers, Bence (rubasov)

1 0

[openstackclient] openstack cli broken after update to Wallaby
by Eugen Block 11 Apr '23

11 Apr '23

Hi *, today I upgraded my virtual test environment from V to W when (Ubuntu 20.04) all of a sudden cli commands didn't work anymore with this stack trace: ---snip--- root@control01:~# openstack network agent list Traceback (most recent call last): File "/usr/bin/openstack", line 6, in <module> from openstackclient.shell import main File "/usr/lib/python3/dist-packages/openstackclient/shell.py", line 23, in <module> from osc_lib import shell File "/usr/lib/python3/dist-packages/osc_lib/shell.py", line 24, in <module> from cliff import app File "/usr/lib/python3/dist-packages/cliff/app.py", line 22, in <module> import cmd2 File "/usr/lib/python3/dist-packages/cmd2.py", line 585, in <module> _ = pyperclip.paste() File "/usr/lib/python3/dist-packages/pyperclip/__init__.py", line 667, in lazy_load_stub_paste copy, paste = determine_clipboard() File "/usr/lib/python3/dist-packages/pyperclip/__init__.py", line 558, in determine_clipboard return init_gi_clipboard() File "/usr/lib/python3/dist-packages/pyperclip/__init__.py", line 167, in init_gi_clipboard gi.require_version('Gtk', '3.0') File "/usr/lib/python3/dist-packages/gi/__init__.py", line 129, in require_version raise ValueError('Namespace %s not available' % namespace) ValueError: Namespace Gtk not available ---snip--- I found this bug [1] describing the same issue but there has been no progress. I posted my comments there as well. I found one way to get the openstack shell to work by installing libgtk-3-dev (I found a hint in a search engine). Apparently, python3-cmd2 requires python3-pyperclip which requires python3-gi and so on. Is this really the desired way? I didn't notice anything in the release notes (maybe I missed it). When comparing to a different environment (Victoria on baremetal) I see that libgtk-3 is installed there (not -dev though), but even with libgtk-3-0 the error message was still present. So the question is, which dependencies are missing where? It's not really obvious to me. Could this already be fixed in Xena? If it is fixed there I could do the double upgrade, of course, especially since Wallaby is already under extended maintenance. Any comments are appreciated. Thanks, Eugen [1] https://bugs.launchpad.net/ubuntu/+source/python-openstackclient/+bug/19456…

4 9

[ironic][nova][neutron][oslo][sdk][swift][charm] netifaces has been archived
by Takashi Kajinami 11 Apr '23

11 Apr '23

Hello, I recently (actually, this morning) noticed the netifaces library[1] was archived a while ago, because the maintenor could no longer continue Open Source contributions, unfortunately[2]. [1] https://github.com/al45tair/netifaces [2] https://github.com/al45tair/netifaces/issues/78 I quickly searched the repositories and found some projects which have netifaces in their requirements.txt . We might need to look for alternatives (or offer to maintain the project, which is always the best) before we face any problems caused by that unmaintained library. Thank you, Takashi Kajinami

4 4

StackOver Flow Error: 2103 ERROR oslo.messaging._drivers.impl_rabbit Connection failed: failed to resolve broker hostname
by Trey Fleishman 11 Apr '23

11 Apr '23

Hello all, Recently I tried installing Openstack via this <https://docs.openstack.org/install-guide/openstack-services.html> guide. Upon completion of all the steps including the dashboard step, I was greeted with this error inside the nova_schedular.log file, after I tried creating an instance. 2103 ERROR oslo.messaging._drivers.impl_rabbit Connection failed: failed to resolve broker hostname: OSError: failed to resolve broker hostname When I try to create an instance it hangs in the "scheduling" state and throws this error. I have not configured anything in the "oslo.messaging" section of the Nova config file. I believe that under the [default] section this is what's causing the issue, but the password and hostname are fine. transport_url = rabbit://openstack:BnE9LLfCtjdPvpEnZUnPytSyrfUxE8th@controller Thanks!

2 1

[tc] Technical Committee next weekly meeting on 2023 Apr 11 at 1800 UTC
by Nikolla, Kristi 11 Apr '23

11 Apr '23

Hi all, This is a reminder that the next weekly Technical Committee meeting is to be held tomorrow (April 11) at 1800 UTC on #openstack-tc on OFTC IRC Note: This is the first meeting with the new meeting time, please update your calendars accordingly moving forward. Items can be proposed by editing the wiki page at https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting At the end of today I will finalize the agenda and send out another email. Thank you, Kristi Nikolla

1 1

[kolla ansible] SRIOV compute deployment question
by Satish Patel 09 Apr '23

09 Apr '23

Folks, I have deployed SRIOV compute but i found it deployed sriov-agent and openvswitch-agent together. If I am not mistaken then we don't need openvswitch agent on sriov computes nodes so how do i tell kolla to not deploy openvswitch-agent on sriov compute nodes? Here is the output on the compute node. root@kolla-comp-sriov-1-1:~# docker ps --filter name=agent CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 92af9870ee5b docker-reg:4000/kolla/neutron-sriov-agent:zed "dumb-init --single-…" 32 hours ago Up 31 hours (healthy) neutron_sriov_agent 14aea45e8b47 docker-reg:4000/kolla/neutron-openvswitch-agent:zed "dumb-init --single-…" 32 hours ago Up 3 seconds (health: starting) neutron_openvswitch_agent Configuration: (venv-kolla) root@kolla-infra-1-1:/var/lib/nova# cat /etc/kolla/config/neutron/kolla-comp-sriov-1-1/sriov_agent.ini [sriov_nic] physical_device_mappings = physnet1:eno50 (venv-kolla) root@kolla-infra-1-1:/var/lib/nova# cat /etc/kolla/config/neutron/kolla-comp-sriov-1-1/ml2_conf.ini [ml2] mechanism_drivers = sriovnicswitch [ml2_type_vlan] network_vlan_ranges = physnet1:30:50

1 0

[all] Hold your rechecks
by Dmitriy Rabotyagov 07 Apr '23

07 Apr '23

Hi everyone, Due to the recent update of gnupg2 package for CentOS Stream 9 to the version 2.3.3-3, a lot of repositories, including BaseOS and NFV, are failing to pass GPG check [1]. This affects all latest images, including container and cloud ones. At the moment, images used in Zuul are not affected, but this might change with the next periodic rebuild unless the related BZ report is fixed. Explicit execution of `rpm --import` can be used as a temporary workaround. [1] https://bugzilla.redhat.com/show_bug.cgi?id=2184640

2 3

[kolla-ansible]ask about multi region deploymet
by Nguyễn Hữu Khôi 07 Apr '23

07 Apr '23

Hello. I have read about kolla ansible multi regions. Can i ask that do we need 2 deployment host? And can we use 3 controller each region? I searched on Internet, I see that controller just sit on region one only? Please advise. Thank you.

1 0

[outreachy][cinder] Unable to generate api-ref samples for snapshot transfer
by Toheeb Oyekola 07 Apr '23

07 Apr '23

Hi, I am trying to generate api-ref doc from template by running "tox -e snaphot_transfer" but I keep getting the same error pasted in the link https://paste.openstack.org/show/bFPbWKQAKRbIds4R0wWn and I have installed the missing dependency as shown below, anyone have an Idea what the issue could have been? Thanks for your time. [image: image.png]

2 1

答复: [ptl] Need PTL volunteer for OpenStack Sahara
by Jerry Zhou (周鑫勇) 07 Apr '23

07 Apr '23

Hi Gmann, I can lead this project. -----邮件原件----- 发件人: Ghanshyam Mann <gmann(a)ghanshyammann.com> 发送时间: 2023年3月23日 2:35 收件人: Qiujunting <qiujunting(a)inspur.com>; openstack-discuss <openstack-discuss(a)lists.openstack.org> 主题: [ptl] Need PTL volunteer for OpenStack Sahara Hi Qiu, I am reaching out to you as you were PTL for OpenStack Sahara project in the last cycle. There is no PTL candidate for the next cycle (2023.2), and it is on the leaderless project list. Please check if you or anyone you know would like to lead this project. - https://etherpad.opendev.org/p/2023.2-leaderless Also, if anyone else would like to help leading this project, this is time to let TC knows. -gmann

1 0

[kolla-ansible] updating procedure
by wodel youchi 07 Apr '23

07 Apr '23

Hi, This is what documentation says about updating kolla-ansible : "Inside a series, it is usually sufficient to just update the kolla-ansible package, rebuild (if needed) and pull the images, and run kolla-ansible deploy again. Please follow release notes to check if there are any issues to be aware of." So to summarize, the procedure is like this : 1 - update kolla-ansible 2 - Download the latest containers of the used version 3 - In my case publish them in the local registry 4 - launch kolla-ansible deploy My question : Can I do the update directly , especially on the compute nodes, or do I have to empty some of them by moving VMs and execute the update on the empty nodes only, then proceed with the same logic with the rest of the nodes? Regards.

3 3

Re: multi-region deployment within datacenter scenario
by Satish Patel 06 Apr '23

06 Apr '23

Hi Piotr, Basically we use a vlan base tenant so the network node has very less usage. All my vms talk directly to the physical network and go out via the edge router. Our VM churn rate is also low because it's a private cloud. I am thinking of creating 3x API nodes and 3x RabbitMQ/DB nodes (total 6 nodes control plane and hope it works fine). One big advantage of having multiple pods (deployments) so during upgrade or any issue we have backup to work. On Thu, Apr 6, 2023 at 3:42 PM Piotr Misiak <piotrmisiak1984(a)gmail.com> wrote: > Hi Satish, > > Nova is well designed to handle hundreds of nodes. I would take a look at > Neutron instead, because it doesn’t scale very well. It depends on how your > networking configuration and use cases look like. > > BR, > Piotr > > On Wed, 5 Apr 2023 at 17:46 Satish Patel <satish.txt(a)gmail.com> wrote: > >> Folks, >> >> This is a very broad question and could have many answers but still >> wanted to ask to see what people are thinking and doing. >> >> We have DC with multiple racks and around 600 nodes so we are planning to >> create 3 private openstack and each with 200 nodes. In other data centers >> we have multiple 200 nodes openstack clusters and life is good. In the new >> datacenter I am thinking of deploying the same way openstack but shared >> keystone between multiple openstack environments to have a single place for >> identification and I am going to call it 3 regions even it's in a single >> DC. Hope shared keystone is helpful/easier for doing terrafrom work and >> selecting region easily. >> >> Other option I was thinking about to deploy cellv2 technology and put all >> 600 nodes in single cloud but always worried for operation point of view. >> Now sure how many people running cellv2 successfully without pain. (Except >> CERN) >> >> If people running multi-region openstack cloud then please share some >> thought and idea to make life easier. Not looking for best solution but >> just clue to get idea to deploy and manage multiple openstack. >> >> Thanks in advance. >> S >> >> >>

1 0

multi-region deployment within datacenter scenario
by Satish Patel 06 Apr '23

06 Apr '23

Folks, This is a very broad question and could have many answers but still wanted to ask to see what people are thinking and doing. We have DC with multiple racks and around 600 nodes so we are planning to create 3 private openstack and each with 200 nodes. In other data centers we have multiple 200 nodes openstack clusters and life is good. In the new datacenter I am thinking of deploying the same way openstack but shared keystone between multiple openstack environments to have a single place for identification and I am going to call it 3 regions even it's in a single DC. Hope shared keystone is helpful/easier for doing terrafrom work and selecting region easily. Other option I was thinking about to deploy cellv2 technology and put all 600 nodes in single cloud but always worried for operation point of view. Now sure how many people running cellv2 successfully without pain. (Except CERN) If people running multi-region openstack cloud then please share some thought and idea to make life easier. Not looking for best solution but just clue to get idea to deploy and manage multiple openstack. Thanks in advance. S

3 7

[interop][ptg] Interop Bobcat vPTG summary
by Martin Kopec 06 Apr '23

06 Apr '23

Hi everyone, here is a summary of the discussions. The full agenda can be found here [1]. == Current pain points and challenges of interop testing == The current guidelines don't contain complex enough tests - apart from api tests we should also have more functional tests which mock users' interactions with the cloud - we should test as a real user would use the cloud. People could fake the refstack results - after a discussion we came to an agreement that those people would always find a way, there is no need for coming up with sophisticated ways to prevent it. The point of Interop is to provide a check for vendors/users to see where they stand with their product based on the agreed standards. Use of Interop's tooling is complicated - there have been reports that users struggle with refstack-client and tempest. There is an ongoing effort to create a refstack container which would contain all necessary tools and could be run with just one command. Passing the Interop's guidelines means interoperability on which level? Passing the guidelines should mean being in compliance with the beforehand community agreed upon standards. Getting vendors running the interop tests more often - we only came up with the strategies which are already being used, such as expirations and guideline versioning. == Future strategy == The future strategy could be summarized in 3 points: * discuss the possibility for using sdk-based and tobiko testing * better describe to users the expectations they should have for clouds which assert interoperability * look into expiration of older standards and require that compliant providers indicate to users which version of the standard they last tested/passed [1] https://etherpad.opendev.org/p/bobcat-ptg-interop Regards, -- Martin Kopec Principal Software Quality Engineer Red Hat EMEA IM: kopecmartin

2 1

[tacker][ptg] Tacker PTG Summary
by Yasufumi Ogawa 06 Apr '23

06 Apr '23

Hi team, Thank you for joining our three days vPTG sessions for Tacker bobcat. It was three days and we had about 20 attendees each day. You can find all the result of discussions in [1] and cross project session with Keystone, Ironic and Barbican for supporting external authz for OAuth2.0 [2]. Here is a summary of our vPTG. - We shared several issues of functional tests have been suffered us for unexpected failures through the latest cycles and agree to revise the tests. Current target include refactoring messed directory structure and useless logs, reducing too much timeouts or redefining test scenarios. - For docs, there is no instruction for supported versions of k8s, helm or prometheus introduced in Tacker in the latest releases. There is also something missing instructions or too old contents. We've agreed to make a plan to update the docs and complete TODOs in bobcat. - Enable tacker to get error messages of management driver as a requirement from operators. - Share the current status of secure RBAC support in Tacker and remained tasks mainly for implementing unit and functional tests and documentation. - We agreed to introduct a new feature for supporting external key manager for authn credentials with encryption for security and custom resources for the latest k8s cluster APIs. - Discussed a requirement for supporting terraform infra-driver for some usecases deloying telco systems on cloud-based infra, and roughly agreed to include the feature into Tacker although we need to have some more surveys for implementation details. - We've had several proposals for improving our dev scheme such as introducing type annotation, plantUML for docs or so. - Shared the current status of Pypi maintainer cleanup. [1] https://etherpad.opendev.org/p/tacker-bobcat-ptg [2] https://etherpad.opendev.org/p/bobcat-ptg-ext_oauth2_server

1 0

[tc] May 2023 OpenInfra Board Sync
by Jeremy Stanley 06 Apr '23

06 Apr '23

The Open Infrastructure Foundation Board of Directors is endeavoring to engage in regular check-ins with official OpenInfra projects. The goal is for a loosely structured discussion one-hour in length, involving members of the board and the OpenStack TC, along with other interested community members. This is not intended to be a formal presentation, and no materials need to be prepared in advance. I've started an Etherpad where participants can brainstorm potential topics of conversation, time-permitting: https://etherpad.opendev.org/p/2023-05-board-openstack-sync Immediately following the February 8 discussion, I proposed to schedule the next call for 20:00 UTC on Wednesday, May 10 and heard no objections, so I've attached a calendar file which can serve as a convenient schedule hold for this, in case anyone needs it. -- Jeremy Stanley

1 0

[nova][cinder] why volumes can only be attached and dettached for running instances
by kaqiu pi 06 Apr '23

06 Apr '23

hello, I can only attached and dettached volumes for running instances in openstack. I would like to ask the reason for this design. And is there any danger if this restriction is turned on through custom modification in a private scene? Thank you so much. Yours

3 2

(no subject)
by Richard Buregeya 06 Apr '23

06 Apr '23

Hello Team, Hope you are doing well. Any one with an idea if there is a modification in COA exam, I heard it is now a multiple question exams instead of practical exam, Anyone with current information, kindly update me. Thanks a lot! Regards

3 6

[nova][cinder] future of rebuild without reimaging
by Bence Romsics 06 Apr '23

06 Apr '23

Hi All! We have users who use 'rebuild' on volume booted servers before nova microversion 2.93, relying on the behavior that it keeps the volume as is. And they would like to keep doing this even after the openstack distro moves to a(n at least) zed base (sometime in the future). As a naive user, it seems to me both behaviors make sense. I can easily imagine use cases for rebuild with and without reimaging. However since the implementation of https://specs.openstack.org/openstack/nova-specs/specs/zed/implemented/volu… rebuild without reimaging is only possible using an old microversion (<2.93). With that change merged, rebuild without reimaging seems to be a somewhat less than fully supported feature. A few examples of what I mean by that: First, there's this warning: https://opendev.org/openstack/python-openstackclient/src/commit/5eb89e4ca1c… In which it is unclear to me what exactly will become an error in a future release. Rebuild with a different image? Or any rebuild with microversion <2.93? Then old nova microversions may get dropped. Though what I heard from nova folks, this is unlikely to happen. Then there are a few hypothetical situations like: a) Rebuild gets a new api feature (in a new microversion) which can never be combined with the do-not-reimage behavior. b) Rebuild may have a bug, whose fix requires a microversion bump. This again can never be combined with the old behavior. What do you think, are these concerns purely theoretical or real? If we would like to keep having rebuild without reimaging, can we rely on the old microversion indefinitely? Alternatively shall we propose and implement a nova spec to explicitly expose the choice in the rebuild api (just to express the idea: osc server rebuild --reimage|--no-reimage)? If the topic is worth further discussion beyond the ML, I can also bring it to the nova ptg. Thanks in advance, Bence Romsics (rubasov) ps: I'll be afk for a few days, but I'll follow up next Tuesday.

7 16

[qa][ptg] QA Bobcat vPTG summary
by Martin Kopec 06 Apr '23

06 Apr '23

Hi everyone, here is a summary of the discussions. The full agenda can be found here [1]. == Antelope Retrospective == Good things: kudos to all who are still around and help with reviews, patches, bug fixes Bad things: unstable gate due to tox update, timeouts, job flakiness (mainly multinode job) == uWSGI is in the maintenance mode == This doesn't require an immediate action as it's still in the maintenance mode = any bugs should be fixed and until we need new features we don't have to rush into replacing it. We should have it on the watchlist though and potentially test some alternatives - I've put this as one of the priority items for this cycle [2]. == FIPS interop testing == We had a FIPS goal since the last PTG [3]. The patches [4] and [5] were to have FIPS on ubuntu and these [6] and [7] are to add FIPS on a centos job. The next goal is to have a FIPS job on Rocky linux as it's more stable than centos. == Retiring Patrole == We are going to retire Patrole project. 2 main facts that speak in favor of retiring: Fact 1: plugins have enough RBAC tests (and are writing more to test SRBAC) - therefore patrole is not needed, the testing coverage got by the plugins is enough. Fact 2: patrole gates have been broken for a long time and no one was affected by that. == Additional PyPi maintainer cleanup == Only 2 projects from QA left with additional maintainers: * openstack/os-performance-tools - additional maintainer SpamapS (Clint Byrum) * openstack/patrole - additional maintainer DavidPurcell If you know any of them, please, tell them to reach out to me. I've sent emails to them, but maybe the email addresses I found aren't checked regularly anymore. == Tempest/Devstack gate stability == Several precautions have been done such as reduce mysql memory [8] or reduce the number of tests executed per job, increase of a timeout etc [9]. However, those precautions are mainly workarounds. The important note is that the instability was only highlighted by our testing, not caused by it. Many problems are inefficiencies in openstack software, e.g.: * openstack-client issues a new token for every operation * python startup time - entrypoint lookup * privesp processes in aggregate use more memory than nova (every service runs their own copy) * inefficient ansible code, etc Any volunteers who would like to first analyse the situation, find which of those inefficiencies are the biggest culprits and how big effect they have on the system? [1] https://etherpad.opendev.org/p/qa-bobcat-ptg [2] https://etherpad.opendev.org/p/qa-bobcat-priority [3] https://etherpad.opendev.org/p/antelope-ptg-interop [4] https://review.opendev.org/c/zuul/zuul-jobs/+/866881 [5] https://review.opendev.org/c/zuul/zuul-jobs/+/873893 [6] https://review.opendev.org/c/openstack/devstack/+/871606 [7] https://review.opendev.org/c/openstack/tempest/+/873697 [8] https://review.opendev.org/c/openstack/devstack/+/873646 [9] https://review.opendev.org/q/topic:bug%252F2004780 Regards, -- Martin Kopec Principal Software Quality Engineer Red Hat EMEA IM: kopecmartin

1 0

[cinder] 2023.2 Bobcat PTG Summary
by Rajat Dhasmana 06 Apr '23

06 Apr '23

Hello, Please find the summary of 2023.2 Bobcat PTG in the wiki page[1]. It includes recordings, summary and action items for each topic. More details can be found in the PTG etherpad[2]. I would recommend everyone to please look for the action items assigned to you to plan it according to the cycle deadlines. [1] https://wiki.openstack.org/wiki/CinderBobcatPTGSummary [2] https://etherpad.opendev.org/p/bobcat-ptg-cinder Thanks Rajat Dhasmana

1 0

[cinder]multi backend SAN and Ceph
by Nguyễn Hữu Khôi 06 Apr '23

06 Apr '23

Hello. I have a question that could we use both SAN and Ceph for multi backend? If yes can we change volume type from SAN to Ceph and otherwise? Thanks.

3 3

[Glance] Weekly Meeting Cancelled
by Pranali Deore 06 Apr '23

06 Apr '23

Hi All, Canceling today's glance weekly meeting, I'm on sick leave today. See you next week ! Thanks, Pranali D

1 0

[cinder]cannot lauch instance with cinder volume
by Nguyễn Hữu Khôi 06 Apr '23

06 Apr '23

Hello guys. I am using kolla-ansible yoga to deploy an openstack cluster, but when I launch an instance with volume, it has errors. but I can create volume first and boot from it. I am using Powerstore 5000Tb. Is it a bug? 672 ERROR oslo_messaging.rpc.server [req-1a822ccd-1d3a-4f9a-a453-22d04f07f816 47cefefdadbe4bb584d8cf983de71eb5 708d7af8178143aab263d3942a124ec8 - - -] Exception during message handling: cinder.exception.ImageCopyFailure: Failed to copy image to volume: Bad or unexpected response from the storage volume backend API: Unable to fetch connection information from backend: Bad or unexpected response from the storage volume backend API: Failed to create PowerStore host . 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/volume/driver.py", line 1045, in _attach_volume 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server conn = self.initialize_connection(volume, properties) 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/volume/drivers/dell_emc/powerstore/driver.py", line 153, in initialize_connection 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server return self.adapter.initialize_connection(volume, connector, **kwargs) 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/volume/drivers/dell_emc/powerstore/adapter.py", line 272, in initialize_connection 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server connection_properties = self._connect_volume(volume, connector) 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/volume/drivers/dell_emc/powerstore/adapter.py", line 613, in _connect_volume 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server chap_credentials, volume_lun = self._create_host_and_attach( 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/volume/drivers/dell_emc/powerstore/adapter.py", line 602, in _create_host_and_attach 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server host, chap_credentials = self._create_host_if_not_exist(connector) 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/decorator.py", line 232, in fun 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/coordination.py", line 195, in _synchronized 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server return f(*a, **k) 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/volume/drivers/dell_emc/powerstore/adapter.py", line 446, in _create_host_if_not_exist 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server host = self.client.create_host(host_name, ports) 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/volume/drivers/dell_emc/powerstore/client.py", line 302, in create_host 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server raise exception.VolumeBackendAPIException(data=msg) 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server cinder.exception.VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Failed to create PowerStore host . Nguyen Huu Khoi

1 1

[kolla-ansible][yoga][neutron][neutron-dhcp]ERRORS.... Need to understand
by Franck VEDEL 05 Apr '23

05 Apr '23

Hello. I come back to you to try to understand my problem. My installation is based on kolla-ansible, Yoga, Centos-Stream. For some time, without understanding why, neutron_dhcp no longer works on external networks and the networks of my students' projects. This generates many error messages, which are difficult to understand. Could this be a bug in this version? Before, it worked very well, but I have the impression that when the number of network increases (i’m not sure), from a moment, dhcp does not work anymore. What is the way to find the problem? Here are examples of errors that won't stop even if no one creates more instances of them 2023-04-04 20:36:52.870 7 ERROR neutron.agent.dhcp.agent device, namespace, scope, broadcast) 2023-04-04 20:36:52.870 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 272, in _wrap 2023-04-04 20:36:52.870 7 ERROR neutron.agent.dhcp.agent r_call_timeout) 2023-04-04 20:36:52.870 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 215, in remote_call 2023-04-04 20:36:52.870 7 ERROR neutron.agent.dhcp.agent raise exc_type(*result[2]) 2023-04-04 20:36:52.870 7 ERROR neutron.agent.dhcp.agent neutron.privileged.agent.linux.ip_lib.InvalidArgument: Invalid parameter/value used on interface tap73cd30e2-35, namespace qdhcp-395f1edb-de6d-4dd8-b31b-69a53780c012. 2023-04-04 20:36:52.870 7 ERROR neutron.agent.dhcp.agent 2023-04-04 20:36:52.871 7 INFO neutron.agent.dhcp.agent [-] Finished network 395f1edb-de6d-4dd8-b31b-69a53780c012 dhcp configuration 2023-04-04 20:36:52.872 7 INFO neutron.agent.dhcp.agent [-] Starting network 8db375e0-28d2-4379-9330-397dc9b4f2b1 dhcp configuration 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent [-] Unable to enable dhcp for 8db375e0-28d2-4379-9330-397dc9b4f2b1.: neutron.privileged.agent.linux.ip_lib.InvalidArgument: Invalid parameter/value used on interface tap1c220d25-b9, namespace qdhcp-8db375e0-28d2-4379-9330-397dc9b4f2b1. 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent Traceback (most recent call last): 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/dhcp/agent.py", line 218, in call_driver 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent rv = getattr(driver, action)(**action_kwargs) 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/dhcp.py", line 274, in enable 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent common_utils.wait_until_true(self._enable, timeout=300) 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/common/utils.py", line 717, in wait_until_true 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent while not predicate(): 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/dhcp.py", line 286, in _enable 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent interface_name = self.device_manager.setup(self.network) 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/dhcp.py", line 1779, in setup 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent namespace=network.namespace) 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/interface.py", line 152, in init_l3 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent device.addr.add(ip_cidr) 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 537, in add 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent add_broadcast) 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 823, in add_ip_address 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent device, namespace, scope, broadcast) 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 272, in _wrap 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent r_call_timeout) 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 215, in remote_call 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent raise exc_type(*result[2]) 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent neutron.privileged.agent.linux.ip_lib.InvalidArgument: Invalid parameter/value used on interface tap1c220d25-b9, namespace qdhcp-8db375e0-28d2-4379-9330-397dc9b4f2b1. 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent Thanks a lot Franck

2 4

[ironic][ptg] 2023.2 Work Items
by Jay Faulkner 05 Apr '23

05 Apr '23

Hey all, As usual, I documented items we talked about working on in Bobcat in a spec. Please go have a look and review it: https://review.opendev.org/c/openstack/ironic-specs/+/879381 This will hopefully be published very soon at https://specs.openstack.org/openstack/ironic-specs/#themes as our 2023.2 work items. Thanks everyone for participating in our vPTG this cycle! -- Jay Faulkner Ironic PTL

1 0

[tripleo] hold rechecks content provider gate blocker
by Marios Andreou 05 Apr '23

05 Apr '23

Hello tripleo we have a gate blocker at [1] hitting the centos-9-content-provider jobs. The tripleo CI team is tracking this - please hold rechecks until it we have a resolution or workaround regards, marios [1] https://bugs.launchpad.net/tripleo/+bug/2015309

1 1

[kolla] weekly meeting cancelled today
by Michał Nasiadka 05 Apr '23

05 Apr '23

Hello Koalas, Weekly meeting cancelled today. Please note that our weekly meeting time for next week and onwards has changed to 13:00 UTC due to DST changes. Best regards, Michal -- Michał Nasiadka mnasiadka(a)gmail.com

1 0

[neutron][OpenStack-ansible] Performance issues with trunk ports
by John Bartelme 05 Apr '23

05 Apr '23

Hello, I'm currently experiencing some pretty severe performance issues with my openstack-ansible deployed cluster(yoga) while deploying trunk ports and I'm looking for some help determining what might be the cause of this poor performance. In my simplest case I'm deploying 2 servers each with one trunk port each. The first trunk has 2 subports and the second 6 subports. Both servers also have 3 other regular ports. When deploying the first trunk port its subports are often provisioned quickly and the second trunk port takes anywhere from 30 seconds to 18 minutes. This happens even when I isolate neutron-server to a single physical machine with 44(88 threads) and 256GB ram. Further diagnosis has shown me some things i didn't quite understand. My deployment with OpenStack-ansible deploys neutron-server with 16 uWSGI processes and neutron-rpc-server with 16 rpc workers. However the way that the trunk RPC server is implemented it is only run on the parent RPC thread and instead runs in all of the uWSGI processes as well. This means that most of my trunk RPC calls are being handled by the uWSGI instead of the RPC workers. When the parent RPC thread handles the trunk port creation calls I constantly see creation times of 1-1.5 seconds. I've isolated it so that this thread does all of the trunk RPC calls and this works quite well but this doesn't seem ideal. What could be causing such poor performance in the uWSGI side of the house? I'm having a really hard time getting a good feeling for what might be slowing it down so much. I'm wondering if it could be green thread preemption but I really don't know. I've tried setting 'enable-threads' false for uWSGI but I don't think that is improving performance. Putting the profiled decorator on update_subport_bindings shows different places taking longer every time, but in general a lot of time(tottime, i.e. not subfunction time) spent in webob/dec.py(__call__), paste/urlmap.py(__call__), webob/request.py(call_application),webob/request.py(send). What else can I do to try and look for why this is taking so long? As a side question it seems counterintuitive that the uWSGI handles most of the trunk RPC calls and not the RPC workers? A couple other notes about my environment that could indicate my challenges: I had to disable rabbitmq heartbeats for neutron as they kept not getting sent reliably and connections were terminated. I tried with heartbeat_in_pthread both true and false but still had issues. It looks like nova also sometimes experiences this but not near as often. I was overzealous with my vxlan ranges in my first configuration and gave it a range of 10,000,000 not realizing that would create that many rows in the database. Looking into that I saw that pymysql in my cluster takes 3.5 minutes to retrieve those rows. mysql CLI only takes 4 seconds. Perhaps that is just the overhead of pymysql? I've greatly scaled down the vxlan range now. I'm provisioning the 2 servers with a heat template that contains around 200 custom resources. For 198 of the resources they are set to conditionally not create any OpenStack native resources. Deploying this template of mostly no-op resources still takes about 3 minutes. Horizon works but almost every page load take a few seconds to load. I'm not sure if that is normal or not. Thanks for any help anyone can provide. john

4 8

Does Openstack support the notion of cinder backend in maintenance mode?
by wodel youchi 05 Apr '23

05 Apr '23

Hi, When you have multiple cinder backends, and you need to take down one of them for maintenance purposes, what is the best way to proceed? Does Openstack have the notion of : cinder backend in maintenance mode? Regards.

3 6

[cinder] 2023.2 Bobcat Meeting Etherpad
by Rajat Dhasmana 05 Apr '23

05 Apr '23

Hello Argonauts, As we're done with the 2023.2 Bobcat PTG, we will be moving to a new etherpad for cinder meetings for the 2023.2 Bobcat development cycle. Please add topics (if you have any) to the etherpad for the meeting on 05 April, 2023 (Today). Etherpad: https://etherpad.opendev.org/p/cinder-bobcat-meetings Thanks Rajat Dhasmana

1 0

[manila] Bobcat vPTG summary
by Carlos Silva 05 Apr '23

05 Apr '23

Hello, Zorillas and interested stackers! Thank you for attending last week's PTG. We had productive discussions and came up with good plans for this cycle! Here is a summary of the topics we spoke about during the PTG. The etherpad with all the notes is available in [0]. If you would like to see the recordings for those discussions, they are available on the Manila Youtube channel [1]. - *Share Transfer Future development* - Share transfers feature was introduced in Antelope but it has a couple of possible enhancements, such as: transferring shares that have snapshots, replicated shares, share networks alongside its share servers and shares. - Snapshots shouldn't be very difficult to cover but we would need to keep in mind quota allocations while doing so. - Transferring shares with replicas is the bigger challenge, considering we need to work with modifying the access rules. - The next step would be to work on transferring share networks. - AIs: Haixin plans to work on this feature during Bobcat. - *S-RBAC changes in Bobcat* - For the Bobcat cycle, we will focus on: - Ensure we are landing even more tests and enabling the oslo policy defaults. - Modify the testing we are doing to use the reader role to test read-only API, opposed to the member role. - Audit all cross service interactions and isolate actions performed by the "service" user. - Push for the completion of unit tests for Secure RBAC - AIs: Manila team will continue promoting events to get tests reviewed and implemented - *Cross project discussion with Nova for VirtIO FS * - Manila and Nova team discussed approaches for avoiding shares to be deleted while they are attached to instances. - We have agreed that an admin/service level API for locking/unlocking shares deletion would be the best solution to avoid shares being deleted while mounted. This API would ideally be a 'locked by <consumer>' action instead of a simple boolean - As part of this cross-project effort, the Manila team has also been focusing on getting more APIs available in OpenStack SDK. We have been making great progress with it and we plan to have all APIs nova is using merged in the OpenStack SDK by the bobcat cycle. - AIs: - Manila team to write a spec describing the behavior of the locking/unlocking API for share. - Manila team to focus on having all APIs nova is using in OpenStack SDK by the Bobcat cycle. - *Scalable NFS Ganesha for CephFS - state of the kitchen* - We have shared a status of what was our focus during the previous release and some important bugs that have been fixed. - Updates on moving from a standalone ganesha to a cephadm deployed: - There are three changes being worked on at the moment, after past discussions: - Ensure that the shares access rules will be applied when moving from the old to the new server - Ensure that the share exists in both old and new ganesha servers and to reapply all access rules in both - Change to represent export locations from both old and new export locations while both cephfs_nfs_cluster_id and ganesha_server_ips are present in the backend. - With these three changes we would cover all the scenarios considering release upgrades. - The changes are being tested and need feedback. - A job for automated testing is in progress. We are targeting to have a multi-node CI job that will be capable of deploying ganesha using cephadm and run tests using it. - AI: carloss and gouthamr will pursue closure for the three changes related to the upgrades and ashrodri will continue to work on the multi-node testing job. - *SQLAlchemy 2.0 and other DB challenges/changes* - DB testing is failing sporadically in the CI and we were discussing possible changes to avoid rechecks caused by sporadic failures. - We intend to work on missing indexes for queries, so we can gain performance in the database. - Implementing __repr__ to manila models - Other people are already doing this and this could be of huge help for debugging, as one of the operators suggested in the session - We will investigate and apply this in Manila - AIs: gouthamr will open bugs for tracking the work on the joinedloads and a blueprint for the missing indexes. - Sqlalchemy 2.0 changes have already started and we have a list of changes yet to merge. stephenfin is doing a great work on the changes. - AIs: manila team to review the changes proposed by stephenfin - *Create share from replicated snapshot / keep source share on replica promote* - Users of this feature are looking for a way to test the inactive replicas before actually promoting them and if everything is correct, they continue with the promotion. - One suggestion to get this working would be having a replication pause API, which would make the inactive instance able to be mounted - We do think this feature could be relevant and some suggestions from the Manila team are: - Letting users know if they can pause/unpause replication even before they create their share replicas - Making users able to know about errors in the API (something similar to the check mechanism on security-service-update) - AIs: carthaca and team will work on a spec and will discuss it further with the Manila team. - *Tech debt:* - We brought up the current issues with our CI, mostly caused by tests being very resource intensive. - We discussed approaches for scenario testing and thought it is worth it trying to create containers to mount share during scenario tests instead of spawning VMs. - Also, the audience has mentioned that it is difficult to keep track of all of these issues and it would be better if we had them documented somewhere, and that we could bring these up more often. - Migration to Ubuntu 22 is yet to complete for some jobs due to failures in package installation and the usage of quagga. - AIs: - carloss will work on a wiki with tech debt items and will bring this up more often in the manila weekly meetings, so if people are available in that time frame to work on them, they will have more chances to raise their hands. - carloss and gouthamr will work to fix the issues with jobs yet to migrate to Ubuntu 22 - *Scheduler data placement based on vendor specific tool* - NetApp has a tool called Active IQ that uses machine learning to identify what is the best pool of disks to place a request based on custom parameters (latency, iops and so on) - They are thinking of trying to have this tool interact with Manila, so it would also benefit from their tool when NetApp is the only storage vendor in the cloud. - The suggestion is to have a new weigher that will make calls to the storage based on the request and use Active IQ response to identify where the share should be placed. - Testing this feature could be a challenge, considering that it's not straightforward to foresee what a weigher would return in a given moment - AIs: felipe_rodriguess alongside the NetApp team will work on a spec so this can be further discussed. [0] https://etherpad.opendev.org/p/manila-bobcat-ptg [1] https://www.youtube.com/watch?v=gefbYrzRD04&list=PLnpzT0InFrqDW_djitgCiFzwh… Thank you! carloss

1 0

[neutron] bug deputy report March 27th - April 2nd
by Miguel Lavalle 05 Apr '23

05 Apr '23

Here's last week's bugs deputy report: Medium ====== https://bugs.launchpad.net/neutron/+bug/2012944 neutron-remove-duplicated-port-bindings does not write to stdout. Valid bug. Assigned to mlavalle https://bugs.launchpad.net/neutron/+bug/2013326 Trying to activate port binding as regular user causes error 500. Assigned to slaweq Low: ========== https://bugs.launchpad.net/neutron/+bug/2012912 [neutron-lib][stable] Wallaby CI n-t-p, test "test_qos_dscp_create_and_update" failing. This bug is addressed by https://review.opendev.org/c/openstack/neutron-lib/+/879441. No further action seems likely RFEs ==== https://bugs.launchpad.net/neutron/+bug/2013228 [rfe] Non-admin users unable to create SR-IOV switch dev ports. Discussed during the PTG during the Neutron / Nova session, pending further discussion in drivers meeting https://bugs.launchpad.net/neutron/+bug/2013540 [RFE] Add support for Napatech LinkVirt SmartNICs. Discussed during the PTG, pending further discussion in drivers meeting

1 0

OpenInfra Live - April 6 at 9 a.m. CT / 14:00 UTC
by Kristin Barrientos 04 Apr '23

04 Apr '23

Hi everyone, This week’s OpenInfra Live episode is brought to you by the OpenStack Large Scale SIG. Episode: Large Scale Ops Deep Dive: Société Générale In the "Large Scale Ops Deep Dive" series, a panel of OpenStack operators invites special guests to talk about their deployment and discuss their operations. For this episode, our guest will be Guillaume Allard from Société Générale, one of France’s largest banks. Speakers: Guillaume Allard, Arnaud Morin, Thierry Carrez, Belmiro Moreira, Mohammed Naser and Kristin Barrientos Date and time: April 6, 2023, at 9 a.m. CT (14:00 UTC) You can watch us live on: YouTube: https://www.youtube.com/live/ZIjpEZSu1gw?feature=share LinkedIn: https://www.linkedin.com/video/event/urn:li:ugcPost:7043641363350908928/ WeChat: recording will be posted on OpenStack WeChat after the live stream Have an idea for a future episode? Share it now at ideas.openinfra.live <https://openinfrafoundation.formstack.com/forms/openinfralive>. Thanks, Kristin Barrientos Marketing Coordinator OpenInfra Foundation

1 0

[horizon][domain][yoga][kolla-ansible] horizon display failed to display domain tab
by James Leong 04 Apr '23

04 Apr '23

Hi all, I am playing around with the domain. I have used kolla-ansible as a deployment tool for deploying the yoga version of OpenStack. However, after deploying a fresh OpenStack, I was not able to view the domain tab on my dashboard. Every time when I want to create a new domain, I have to go to my terminal and create it manually without any GUI. Is there any other configuration I have to enable in the globals.yml file other than setting "horizon_keystone_multidomain: "True"? Thanks for your help James

3 4

[policy][rbac] RBAC 2023.2 Bobcat cycle vPTG discussions summary
by Ghanshyam Mann 04 Apr '23

04 Apr '23

Hello Everyone, We discussed the RBAC goal on Tuesday. I am summarizing the discussion here. Goal document: https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rb… Tracking: https://etherpad.opendev.org/p/rbac-goal-tracking Current progress: ============= Phase-1 (project personas and drop system scope): ------------------------------------------------------------ Projects completed: * Nova * Neutron * Glance * Manila * Ironic (no change needed) * Octavia * Placement * Service that completed Phase 1 in Zed and enabled scope and new defaults by default (enforce_scope=True & enforce_new_defaults=True by default): ** Nova ** Glance Projects in progress: * Cinder (almost completed. not using system scope, but have not added scope=['project'] to the default rules (currently, no scope is specified) * Magnum ** https://review.opendev.org/c/openstack/magnum/+/874945 * Tacker ** https://review.opendev.org/q/topic:bp%252Fimplement-project-personas Pending work (for phase-2|3): * Keystone implements a new default role called manager: ** https://review.opendev.org/c/openstack/keystone/+/822601 * Keystone implements a new default role called service: ** https://review.opendev.org/c/openstack/keystone/+/863420 Phase-2 (service role): -------------------------- In-progress: * Keystone: ** bootstrap support for servicerole: *** https://review.opendev.org/c/openstack/keystone/+/863420 ** bootstrap support for manager role: *** https://review.opendev.org/c/openstack/keystone/+/822601 *Nova ** https://review.opendev.org/c/openstack/nova/+/864594 Other discussion: ============= *Service role We discussed the service role and how the policy will add the service role. It is correct to add service as well as user role as default if that API is supposed to be called by the service as well as the user role. For example, if Manila is talking to Nova, Cinder, or Neutron via APIs, it needs to use a service role to interact, and Nova, Cinder, and Neutron can update such API rules to allow for service roles also. * Update the goal timeline for removing deprecated rules as per the SLURP release. ** Need at least 1 SLURP release between enabling the new default and removal. ** Action: gmann to update this in the goal document. *Testing: ** Manila has a lot of tests and running on the stable release ** Tempest and devstack ready to implement the test ** The current job with nova, cinder, neutron, glance with the scope and new default enable *** https://zuul.openstack.org/builds?job_name=tempest-full-enforce-scope-new-d… * Tempest now has a project reader/member same project_id * Related sessions: ** Tacker *** Title: Secure RBAC: Implement support of project-personas in Tacker [Continue from Antelope release] (manpreetk) *** Etherpad: https://etherpad.opendev.org/p/tacker-bobcat-ptg#L136 ** Glance *** Title: Secure RBAC *** Etherpad: https://etherpad.opendev.org/p/glance-bobcat-ptg#L53 ** Neutron *** Title: (slaweq) Secure RBAC - phase 2 description and review of the existing API calls *** Etherpad: https://etherpad.opendev.org/p/neutron-bobcat-ptg#L358 <feel free to add here if I missed any related sessions> I will continue holding the biweekly meeting to discuss progress and any query on RBAC. - https://wiki.openstack.org/wiki/Consistent_and_Secure_Default_Policies_Popu… -gmann

1 0

[keystone] Weekly Meeting Cancelled (04-Apr 2023)
by Dave Wilde 04 Apr '23

04 Apr '23

Hi folks, I'm cancelling the Keystone weekly meeting this week (04-Apr 2023) due to a conflict that I have. Please reach out to me on IRC or via email if you need anything. Thanks! /Dave

1 0