April 2023 - openstack-discuss - lists.openstack.org

[openstack][sharing][kolla ansible]Problems when 1 of 3 controller was be down
by Nguyễn Hữu Khôi 11 Apr '23

11 Apr '23

Hello everyone. I want to summary for who meets problems with Openstack when deploy cluster with 3 controller using Kolla Ansible Scenario: 1 of 3 controller is down 1. Login horizon and use API such as nova, cinder will be very slow fix by: nano: kolla-ansible/ansible/roles/heat/templates/heat.conf.j2 kolla-ansible/ansible/roles/nova/templates/nova.conf.j2 kolla-ansible/ansible/roles/keystone/templates/keystone.conf.j2 kolla-ansible/ansible/roles/neutron/templates/neutron.conf.j2 kolla-ansible/ansible/roles/cinder/templates/cinder.conf.j2 or which service need caches add as below [cache] backend = oslo_cache.memcache_pool enabled = True memcache_servers = {{ kolla_internal_vip_address }}:{{ memcached_port }} memcache_dead_retry = 0.25 memcache_socket_timeout = 900 https://review.opendev.org/c/openstack/kolla-ansible/+/849487 but it is not the end 2. Cannot launch instance or mapping block device(stuck at this step) nano kolla-ansible/ansible/roles/rabbitmq/templates/definitions.json.j2 "policies":[ {"vhost": "/", "name": "ha-all", "pattern": "^(?!(amq\.)|(.*_fanout_)|(reply_)).*", "apply-to": "all", "definition": {"ha-mode":"all"}, "priority":0}{% if project_name == 'outward_rabbitmq' %}, {"vhost": "{{ murano_agent_rabbitmq_vhost }}", "name": "ha-all", "pattern": ".*", "apply-to": "all", "definition": {"ha-mode":"all"}, "priority":0} {% endif %} ] nano /etc/kollla/global.conf [oslo_messaging_rabbit] kombu_reconnect_delay=0.5 https://bugs.launchpad.net/oslo.messaging/+bug/1993149 https://docs.openstack.org/large-scale/journey/configure/rabbitmq.html I used Xena 13.4 and Yoga 14.8.1. Above bugs are critical, but I see that it was not fixed. I am just an operator and I want to share what I encountered for new people who come to Openstack Nguyen Huu Khoi

6 11

Re: StackOver Flow Error: 2103 ERROR oslo.messaging._drivers.impl_rabbit Connection failed: failed to resolve broker hostname
by Dmitriy Rabotyagov 11 Apr '23

11 Apr '23

If you're going to go prod I would for sure use some deployment tool rather then manual deployment, as further operations and scaling up would be way easier. What error do you get with existing hosts record? вт, 11 апр. 2023 г., 14:57 Trey Fleishman <nfleishman(a)student.ccaeagles.org >: > Hello Dmitriy, > > That is what I thought too, in my host file I define the hostname as the > correct IP address, and if I change the "controller" hostname to some > factitious IP I get a different error. I've tried putting the IP instead of > the hostname, but to no avail. I can successfully ping the hostname from > the machine. I would also rather not use a tool as I plan to go production > once I get my two node setup functional. I am not sure if this might be a > configuration issue, or maybe a bug. > Trey Fleishman > Student and IT Intern > (704) 750 - 1740 > *Calvary Chapel Fort Lauderdale* > 2401 W. Cypress Creek Rd., Ft. Lauderdale, FL 33309 > CalvaryFTL.org > > > On Tue, Apr 11, 2023 at 12:54 AM Dmitriy Rabotyagov < > noonedeadpunk(a)gmail.com> wrote: > >> Hi Trey, >> >> Have you defined a record in your /etc/hosts for "controller", as >> according to the error service simply can not resolve this hostname to an >> IP address. >> >> Also I can suggest trying out one of the following OpenStack deployment >> tools: >> https://docs.openstack.org/zed/deploy/index.html >> >> Most of them have All-In-One guides that can spawn an opestack sandbox to >> play with. >> >> вт, 11 апр. 2023 г., 00:21 Trey Fleishman < >> nfleishman(a)student.ccaeagles.org>: >> >>> Hello all, >>> Recently I tried installing Openstack via this >>> <https://docs.openstack.org/install-guide/openstack-services.html> >>> guide. Upon completion of all the steps including the dashboard step, I was >>> greeted with this error inside the nova_schedular.log file, after I tried >>> creating an instance. >>> >>> 2103 ERROR oslo.messaging._drivers.impl_rabbit Connection failed: failed >>> to resolve broker hostname: OSError: failed to resolve broker hostname >>> >>> When I try to create an instance it hangs in the "scheduling" state and >>> throws this error. I have not configured anything in the "oslo.messaging" >>> section of the Nova config file. I believe that under the [default] section >>> this is what's causing the issue, but the password and hostname are fine. >>> >>> transport_url = >>> rabbit://openstack:BnE9LLfCtjdPvpEnZUnPytSyrfUxE8th@controller >>> >>> Thanks! >>> >>

2 1

[keystone] Weekly Meeting Cancelled (11-Apr-2023)
by Dave Wilde 11 Apr '23

11 Apr '23

Hi Folks, Unfortunately I have to cancel the weekly meeting today. Please reach out to me directly via email if you need anything. Due to equipment failure my IRC availability is spotty. Thanks! /Dave

1 0

[Kolla-ansible][Yoga] Cannot live migrate VMs
by wodel youchi 11 Apr '23

11 Apr '23

Hi, I cannot live-migrate any VM I have. All my compute nodes are enabled. I have a few VMs, I am in the testing phase. I tried to live-migrate toward an empty node, then toward a node which already had a VM in it. It is the same problem. In the log files I have : Source node : Source > ==> /var/log/kolla/nova/nova-compute.log <== > 2023-04-09 11:32:49.419 7 INFO nova.compute.resource_tracker > [req-4e91d562-d7ad-4585-a827-3db6277fd81e - - - - -] [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Updating resource usage from > migration f60eb077-8e1c-404a-aca5-64e5609ca02c > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [-] [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Pre live migration failed at > computehci02: oslo_messaging.rpc.client.RemoteError: Remote error: > ClientException Gateway Time-out (HTTP 504) > ['Traceback (most recent call last):\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", > line 165, in _process_incoming\n res = > self.dispatcher.dispatch(message)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", > line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, > args)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", > line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", > line 72, in wrapped\n context, exc, binary)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 227, in __exit__\n self.force_reraise()\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 200, in force_reraise\n raise self.value\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", > line 63, in wrapped\n return f(self, context, *args, **kw)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/utils.py", > line 1439, in decorated_function\n return function(self, context, *args, > **kwargs)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", > line 213, in decorated_function\n kwargs[\'instance\'], e, > sys.exc_info())\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 227, in __exit__\n self.force_reraise()\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 200, in force_reraise\n raise self.value\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", > line 201, in decorated_function\n return function(self, context, *args, > **kwargs)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", > line 8367, in pre_live_migration\n bdm.save()\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 227, in __exit__\n self.force_reraise()\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 200, in force_reraise\n raise self.value\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", > line 8304, in pre_live_migration\n connector=connector, > mountpoint=bdm.device_name)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", > line 395, in wrapper\n res = method(self, ctx, *args, **kwargs)\n', ' > File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", > line 433, in wrapper\n res = method(self, ctx, volume_id, *args, > **kwargs)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", > line 813, in attachment_create\n instance_uuid=instance_id)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 227, in __exit__\n self.force_reraise()\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 200, in force_reraise\n raise self.value\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", > line 801, in attachment_create\n volume_id, _connector, instance_id)\n', > ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/api_versions.py", > line 421, in substitution\n return method.func(obj, *args, **kwargs)\n', > ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/v3/attachments.py", > line 39, in create\n retval = self._create(\'/attachments\', body, > \'attachment\')\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/base.py", > line 306, in _create\n resp, body = self.api.client.post(url, > body=body)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", > line 227, in post\n return self._cs_request(url, \'POST\', **kwargs)\n', > ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", > line 215, in _cs_request\n return self.request(url, method, > **kwargs)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", > line 201, in request\n raise exceptions.from_response(resp, body)\n', > 'cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504)\n']. > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Traceback (most recent call last): > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", > line 8501, in _do_pre_live_migration_from_source > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] block_migration, disk, dest, > migrate_data) > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/rpcapi.py", > line 948, in pre_live_migration > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] **msg_args) > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/client.py", > line 192, in call > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] retry=self.retry, > transport_options=self.transport_options) > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/transport.py", > line 128, in _send > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] > transport_options=transport_options) > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", > line 691, in send > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] > transport_options=transport_options) > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", > line 681, in _send > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] raise result > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] > oslo_messaging.rpc.client.RemoteError: Remote error: ClientException > Gateway Time-out (HTTP 504) > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] ['Traceback (most recent call > last):\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", > line 165, in _process_incoming\n res = > self.dispatcher.dispatch(message)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", > line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, > args)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", > line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", > line 72, in wrapped\n context, exc, binary)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 227, in __exit__\n self.force_reraise()\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 200, in force_reraise\n raise self.value\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", > line 63, in wrapped\n return f(self, context, *args, **kw)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/utils.py", > line 1439, in decorated_function\n return function(self, context, *args, > **kwargs)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", > line 213, in decorated_function\n kwargs[\'instance\'], e, > sys.exc_info())\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 227, in __exit__\n self.force_reraise()\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 200, in force_reraise\n raise self.value\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", > line 201, in decorated_function\n return function(self, context, *args, > **kwargs)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", > line 8367, in pre_live_migration\n bdm.save()\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 227, in __exit__\n self.force_reraise()\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 200, in force_reraise\n raise self.value\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", > line 8304, in pre_live_migration\n connector=connector, > mountpoint=bdm.device_name)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", > line 395, in wrapper\n res = method(self, ctx, *args, **kwargs)\n', ' > File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", > line 433, in wrapper\n res = method(self, ctx, volume_id, *args, > **kwargs)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", > line 813, in attachment_create\n instance_uuid=instance_id)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 227, in __exit__\n self.force_reraise()\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 200, in force_reraise\n raise self.value\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", > line 801, in attachment_create\n volume_id, _connector, instance_id)\n', > ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/api_versions.py", > line 421, in substitution\n return method.func(obj, *args, **kwargs)\n', > ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/v3/attachments.py", > line 39, in create\n retval = self._create(\'/attachments\', body, > \'attachment\')\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/base.py", > line 306, in _create\n resp, body = self.api.client.post(url, > body=body)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", > line 227, in post\n return self._cs_request(url, \'POST\', **kwargs)\n', > ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", > line 215, in _cs_request\n return self.request(url, method, > **kwargs)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", > line 201, in request\n raise exceptions.from_response(resp, body)\n', > 'cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504)\n']. > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] > 2023-04-09 11:33:30.157 7 INFO nova.compute.manager [-] [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Swapping old allocation on > dict_keys(['2857ae62-3519-4b13-b700-858c827dd78d']) held by migration > f60eb077-8e1c-404a-aca5-64e5609ca02c for instance In the destination node : > nova-compute.log > > > > > *2023-04-09 11:32:30.052 7 WARNING os_brick.initiator.connectors.nvmeof > [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 > b0f76b5c6dcb457fa716762bbf954837 - default default] Process execution error > in _get_host_uuid: Unexpected error while running command.Command: blkid > overlay -s UUID -o valueExit code: 2Stdout: ''Stderr: '': > oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while > running command.* > > 2023-04-09 11:33:05.354 7 INFO nova.compute.resource_tracker > [req-9d480442-87f7-4d4e-a866-c46d645ac7f2 - - - - -] [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Updating resource usage from > migration f60eb077-8e1c-404a-aca5-64e5609ca02c > 2023-04-09 11:33:05.392 7 WARNING nova.compute.resource_tracker > [req-9d480442-87f7-4d4e-a866-c46d645ac7f2 - - - - -] Instance > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7 has been moved to another host > computehci01(computehci01). There are allocations remaining against the > source host that might need to be removed: {'resources': {'VCPU': 1, > 'MEMORY_MB': 2048}}. > 2023-04-09 11:33:30.086 7 ERROR nova.volume.cinder > [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 > b0f76b5c6dcb457fa716762bbf954837 - default default] [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Create attachment failed for volume > 3d7bd0af-5e4c-48fc-a027-f6c9dd6280bb. Error: Gateway Time-out (HTTP 504) > Code: 504: cinderclient.exceptions.ClientException: Gateway Time-out (HTTP > 504) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server > [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 > b0f76b5c6dcb457fa716762bbf954837 - default default] Exception during > message handling: cinderclient.exceptions.ClientException: Gateway Time-out > (HTTP 504) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server Traceback (most > recent call last): > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", > line 165, in _process_incoming > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server res = > self.dispatcher.dispatch(message) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", > line 309, in dispatch > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return > self._do_dispatch(endpoint, method, ctxt, args) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", > line 229, in _do_dispatch > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server result = > func(ctxt, **new_args) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", > line 72, in wrapped > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server context, > exc, binary) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 227, in __exit__ > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server > self.force_reraise() > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 200, in force_reraise > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise > self.value > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", > line 63, in wrapped > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return > f(self, context, *args, **kw) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/utils.py", > line 1439, in decorated_function > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return > function(self, context, *args, **kwargs) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", > line 213, in decorated_function > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server > kwargs['instance'], e, sys.exc_info()) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 227, in __exit__ > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server > self.force_reraise() > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 200, in force_reraise > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise > self.value > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", > line 201, in decorated_function > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return > function(self, context, *args, **kwargs) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", > line 8367, in pre_live_migration > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server bdm.save() > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 227, in __exit__ > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server > self.force_reraise() > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 200, in force_reraise > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise > self.value > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", > line 8304, in pre_live_migration > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server > connector=connector, mountpoint=bdm.device_name) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", > line 395, in wrapper > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server res = > method(self, ctx, *args, **kwargs) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", > line 433, in wrapper > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server res = > method(self, ctx, volume_id, *args, **kwargs) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", > line 813, in attachment_create > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server > instance_uuid=instance_id) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 227, in __exit__ > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server > self.force_reraise() > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 200, in force_reraise > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise > self.value > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", > line 801, in attachment_create > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server volume_id, > _connector, instance_id) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/api_versions.py", > line 421, in substitution > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return > method.func(obj, *args, **kwargs) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/v3/attachments.py", > line 39, in create > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server retval = > self._create('/attachments', body, 'attachment') > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/base.py", > line 306, in _create > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server resp, body = > self.api.client.post(url, body=body) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", > line 227, in post > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return > self._cs_request(url, 'POST', **kwargs) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", > line 215, in _cs_request > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return > self.request(url, method, **kwargs) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", > line 201, in request > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise > exceptions.from_response(resp, body) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server > cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server > 2023-04-09 11:33:30.238 7 INFO nova.virt.block_device > [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 > b0f76b5c6dcb457fa716762bbf954837 - default default] [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Attempting to driver detach volume > 3d7bd0af-5e4c-48fc-a027-f6c9dd6280bb from mountpoint /dev/vda > 2023-04-09 11:33:30.239 7 WARNING nova.virt.block_device > [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 > b0f76b5c6dcb457fa716762bbf954837 - default default] [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Detaching volume from unknown instance > 2023-04-09 11:33:30.240 7 WARNING nova.virt.libvirt.driver > [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 > b0f76b5c6dcb457fa716762bbf954837 - default default] [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] During detach_volume, instance > disappeared.: nova.exception.InstanceNotFound: Instance > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7 could not be found. > 2023-04-09 11:33:30.678 7 INFO nova.virt.libvirt.driver [-] [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Instance destroyed successfully. > 2023-04-09 11:33:30.682 7 INFO os_vif > [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 > b0f76b5c6dcb457fa716762bbf954837 - default default] Successfully unplugged > vif > VIFBridge(active=True,address=fa:16:3e:9b:92:46,bridge_name='qbr46651303-65',has_traffic_filtering=True,id=46651303-65e1-4809-95cc-7c2d97acf16b,network=Network(532470c1-dd2f-4505-8557-ba24e9d4062b),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap46651303-65') > All I can see is *b0f76b5c6dcb457fa716762bbf954837 - default default] Process execution error in _get_host_uuid: Unexpected error while running command.Command: blkid overlay -s UUID -o valueExit code: 2Stdout: ''Stderr: '': oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.* Searching the web I didn't find anything useful about this error, and I don't know if it's the origin of the problem. My default storage is ceph rbd, I have an HCI deployment, and all my VMs are for now created on Ceph only. But I have configured two other cinder backends, NFS and LVM, but they are node used for now. Thanks in advance. Regards.

3 4

[neutron] bug deputy report for week of 2023-04-03
by Bence Romsics 11 Apr '23

11 Apr '23

Hi All, We had the following new bugs reported last week: Critical: * https://bugs.launchpad.net/neutron/+bug/2015065 [tempest] "test_security_group_rules_create" unstable in "neutron-ovs-grenade-dvr-multinode" unassigned High: * https://bugs.launchpad.net/neutron/+bug/2015364 [skip-level] OVN tests constantly failing unassigned * https://bugs.launchpad.net/neutron/+bug/2015376 [OVN] min-BW rule, defined in the LSP.options field, must be accompanied with a max-BW value unassigned * https://bugs.launchpad.net/neutron/+bug/2015728 ovs/ovn source(with master branch) deployments broken workarounds proposed/merged by Yatin: https://review.opendev.org/c/openstack/neutron/+/879938 https://review.opendev.org/c/openstack/ovn-octavia-provider/+/879939 * https://bugs.launchpad.net/neutron/+bug/2015090 neutron.agent.dhcp.agent TypeError: 'bool' object is not subscriptable Technically unassigned, but I believe Sahid may be working on it. Medium: * https://bugs.launchpad.net/neutron/+bug/2015388 Deleting network does not remove network namespace fix proposed by Brian: https://review.opendev.org/c/openstack/neutron/+/880006 * https://bugs.launchpad.net/neutron/+bug/2015377 [ovn] If dhcp port is deleted from neutron, it is never recreated fix proposed by Brian: https://review.opendev.org/c/openstack/neutron/+/879913 RFE: * https://bugs.launchpad.net/neutron/+bug/2015471 [RFE] Add ERSPAN for tap-as-a-service with OVS and OVN Triage still in progress: * https://bugs.launchpad.net/neutron/+bug/2015449 Remote security groups don't allow traffic from floating IPs This issue may be undocumented behavior, the following question could be discussed on the weekly meeting: https://meetings.opendev.org/irclogs/%23openstack-neutron/%23openstack-neut… * https://bugs.launchpad.net/neutron/+bug/2015275 Trunk port update_subport_bindings RPC call not processed by RPC workers Cheers, Bence (rubasov)

1 0

[openstackclient] openstack cli broken after update to Wallaby
by Eugen Block 11 Apr '23

11 Apr '23

Hi *, today I upgraded my virtual test environment from V to W when (Ubuntu 20.04) all of a sudden cli commands didn't work anymore with this stack trace: ---snip--- root@control01:~# openstack network agent list Traceback (most recent call last): File "/usr/bin/openstack", line 6, in <module> from openstackclient.shell import main File "/usr/lib/python3/dist-packages/openstackclient/shell.py", line 23, in <module> from osc_lib import shell File "/usr/lib/python3/dist-packages/osc_lib/shell.py", line 24, in <module> from cliff import app File "/usr/lib/python3/dist-packages/cliff/app.py", line 22, in <module> import cmd2 File "/usr/lib/python3/dist-packages/cmd2.py", line 585, in <module> _ = pyperclip.paste() File "/usr/lib/python3/dist-packages/pyperclip/__init__.py", line 667, in lazy_load_stub_paste copy, paste = determine_clipboard() File "/usr/lib/python3/dist-packages/pyperclip/__init__.py", line 558, in determine_clipboard return init_gi_clipboard() File "/usr/lib/python3/dist-packages/pyperclip/__init__.py", line 167, in init_gi_clipboard gi.require_version('Gtk', '3.0') File "/usr/lib/python3/dist-packages/gi/__init__.py", line 129, in require_version raise ValueError('Namespace %s not available' % namespace) ValueError: Namespace Gtk not available ---snip--- I found this bug [1] describing the same issue but there has been no progress. I posted my comments there as well. I found one way to get the openstack shell to work by installing libgtk-3-dev (I found a hint in a search engine). Apparently, python3-cmd2 requires python3-pyperclip which requires python3-gi and so on. Is this really the desired way? I didn't notice anything in the release notes (maybe I missed it). When comparing to a different environment (Victoria on baremetal) I see that libgtk-3 is installed there (not -dev though), but even with libgtk-3-0 the error message was still present. So the question is, which dependencies are missing where? It's not really obvious to me. Could this already be fixed in Xena? If it is fixed there I could do the double upgrade, of course, especially since Wallaby is already under extended maintenance. Any comments are appreciated. Thanks, Eugen [1] https://bugs.launchpad.net/ubuntu/+source/python-openstackclient/+bug/19456…

4 9

[ironic][nova][neutron][oslo][sdk][swift][charm] netifaces has been archived
by Takashi Kajinami 11 Apr '23

11 Apr '23

Hello, I recently (actually, this morning) noticed the netifaces library[1] was archived a while ago, because the maintenor could no longer continue Open Source contributions, unfortunately[2]. [1] https://github.com/al45tair/netifaces [2] https://github.com/al45tair/netifaces/issues/78 I quickly searched the repositories and found some projects which have netifaces in their requirements.txt . We might need to look for alternatives (or offer to maintain the project, which is always the best) before we face any problems caused by that unmaintained library. Thank you, Takashi Kajinami

4 4

StackOver Flow Error: 2103 ERROR oslo.messaging._drivers.impl_rabbit Connection failed: failed to resolve broker hostname
by Trey Fleishman 10 Apr '23

10 Apr '23

Hello all, Recently I tried installing Openstack via this <https://docs.openstack.org/install-guide/openstack-services.html> guide. Upon completion of all the steps including the dashboard step, I was greeted with this error inside the nova_schedular.log file, after I tried creating an instance. 2103 ERROR oslo.messaging._drivers.impl_rabbit Connection failed: failed to resolve broker hostname: OSError: failed to resolve broker hostname When I try to create an instance it hangs in the "scheduling" state and throws this error. I have not configured anything in the "oslo.messaging" section of the Nova config file. I believe that under the [default] section this is what's causing the issue, but the password and hostname are fine. transport_url = rabbit://openstack:BnE9LLfCtjdPvpEnZUnPytSyrfUxE8th@controller Thanks!

2 1

[tc] Technical Committee next weekly meeting on 2023 Apr 11 at 1800 UTC
by Nikolla, Kristi 10 Apr '23

10 Apr '23

Hi all, This is a reminder that the next weekly Technical Committee meeting is to be held tomorrow (April 11) at 1800 UTC on #openstack-tc on OFTC IRC Note: This is the first meeting with the new meeting time, please update your calendars accordingly moving forward. Items can be proposed by editing the wiki page at https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting At the end of today I will finalize the agenda and send out another email. Thank you, Kristi Nikolla

1 1

[kolla ansible] SRIOV compute deployment question
by Satish Patel 08 Apr '23

08 Apr '23

Folks, I have deployed SRIOV compute but i found it deployed sriov-agent and openvswitch-agent together. If I am not mistaken then we don't need openvswitch agent on sriov computes nodes so how do i tell kolla to not deploy openvswitch-agent on sriov compute nodes? Here is the output on the compute node. root@kolla-comp-sriov-1-1:~# docker ps --filter name=agent CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 92af9870ee5b docker-reg:4000/kolla/neutron-sriov-agent:zed "dumb-init --single-…" 32 hours ago Up 31 hours (healthy) neutron_sriov_agent 14aea45e8b47 docker-reg:4000/kolla/neutron-openvswitch-agent:zed "dumb-init --single-…" 32 hours ago Up 3 seconds (health: starting) neutron_openvswitch_agent Configuration: (venv-kolla) root@kolla-infra-1-1:/var/lib/nova# cat /etc/kolla/config/neutron/kolla-comp-sriov-1-1/sriov_agent.ini [sriov_nic] physical_device_mappings = physnet1:eno50 (venv-kolla) root@kolla-infra-1-1:/var/lib/nova# cat /etc/kolla/config/neutron/kolla-comp-sriov-1-1/ml2_conf.ini [ml2] mechanism_drivers = sriovnicswitch [ml2_type_vlan] network_vlan_ranges = physnet1:30:50

1 0