April 2023 - openstack-discuss - lists.openstack.org

Does cinder-backup support multi-backend?
by wodel youchi 13 Apr '23

13 Apr '23

Hi, Could cinder-backup be configured with multiple-backends, ceph and nfs for example? Regards.

5 9

[kolla-ansible][mariadb] socat: Connection reset by peer
by Satish Patel 12 Apr '23

12 Apr '23

Folks, I am getting the following errors in logs constantly and continually. I have 3 node galera clusters. Everything looks good except this error making me nervous. I have deployed Zed release of kolla openstack. ==> /var/log/kolla/mariadb/mariadb-clustercheck.log <== 2023/04/12 02:42:16 socat[1574755] W read(7, 0x5653c4a4c000, 8192): Connection reset by peer 2023/04/12 02:42:16 socat[1574761] W read(7, 0x5653c4a4c000, 8192): Connection reset by peer 2023/04/12 02:42:16 socat[1574755] W read(6, 0x5653c4a4c000, 8192): Connection reset by peer 2023/04/12 02:42:16 socat[1574761] W read(6, 0x5653c4a4c000, 8192): Connection reset by peer 2023/04/12 02:42:18 socat[1574767] W read(7, 0x5653c4a4c000, 8192): Connection reset by peer 2023/04/12 02:42:18 socat[1574767] W read(6, 0x5653c4a4c000, 8192): Connection reset by peer

2 2

[kolla-ansible][yoga] how to configure cinder to connect to iscsi target?
by wodel youchi 12 Apr '23

12 Apr '23

Hi, I am trying to configure cinder to connect to an iscsi target. I am using targetcli as my target server. I have tested the connection from the node using iscsiadm and it works from the OS. Now I am trying to use the iscsid container to do the job. I added this to my globals.yml enable_iscsid: "yes" enable_cinder_backend_iscsi: "yes" enable_cinder_backend_lvm: "no" And I created the cinder.conf file like this [DEFAULT] enabled_backends = rbd1,nfs-1,*lvmtnas * # add follows to the end [lvmtnas] target_helper = lioadm target_protocol = iscsi target_ip_address = 20.1.0.32 (my target server) volume_backend_name = lvmtnas # volume group name created volume_group = *vg_volume01 * volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver use_chap_auth = false I did create the vg_volume01 on the LUN prior to the test. I did a deploy, the container has been created, but nothing happened, it's like I am missing something. docker logs iscsid fome one the compute nodes + sudo -E kolla_set_configs INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files INFO:__main__:Writing out command to execute ++ cat /run_command + CMD='iscsid -d 8 -f' + ARGS= + sudo kolla_copy_cacerts + [[ ! -n '' ]] + . kolla_extend_start ++ [[ ! -f /etc/iscsi/initiatorname.iscsi ]] + echo 'Running command: '\''iscsid -d 8 -f'\''' + exec iscsid -d 8 -f Running command: 'iscsid -d 8 -f' iscsid: sysfs_init: sysfs_path='/sys' iscsid: in ctldev_open iscsid: created NETLINK_ISCSI socket... iscsid: InitiatorName=iqn.1994-05.com.redhat:2a318909ad8 iscsid: InitiatorName=iqn.1994-05.com.redhat:2a318909ad8 iscsid: InitiatorAlias=computehci01 iscsid: Max file limits 1048576 1048576 iscsid: current time 258465 iscsid: nothing on pend_list, deactivating alarm Any idea? Regards.

2 4

[neutron] Deprecation of the neutronclient CLI code
by Rodolfo Alonso Hernandez 12 Apr '23

12 Apr '23

Hello all: Since this release [1], the python-neutronclient project will no longer provide CLI support. Any Neutron related CLI command should be executed using python-openstackclient (OSC). This deprecation and removal was announced in [2] and was initially planned for Zed release. We have been working during the last releases to finish the migration and provide all the commands in OSC. In case you find any missing command, argument or output, please open a bug in the Neutron site of launchpad and we'll address it. You can also install a previous (Antelope) version of neutronclient if you missed the migration window. The python-neutronclient will still be available to provide the Networking API python bindings [3]. Regards. [1]https://review.opendev.org/c/openstack/python-neutronclient/+/871711 [2] https://docs.openstack.org/python-neutronclient/latest/contributor/transiti… [3]https://docs.openstack.org/python-neutronclient/ocata/usage/library.html

1 0

CYBORG HTTP400 create trait OWNER_CYBORG
by Karl Kloppenborg 12 Apr '23

12 Apr '23

Hi Cyborg Team, Karl from the Openstack-Helm group. I’ve hit a really an issue I cannot seem to overcome, Cyborg is detecting GPU Profiles from a NVIDIA A40, and it goes to create the record but cyborg-conductor errors with ``` [cyborg-conductor-6fc477ff8b-cw5lf cyborg-conductor] 2023-04-12 06:17:37.581 1 INFO cyborg.conductor.manager [None req-600fcdcf-a620-466a-ba6c-cdfbf6bf83e2 - - - - - -] Failed to add device DriverDevice(controlpath_id=DriverControlPathID,created_at=<?>,deployable_list=[DriverDeployable],model='NVIDIA Corporation Device',std_board_info='{"product_id": "2235", "controller": "3D controller"}',stub=False,type='GPU',updated_at=<?>,vendor='10de',vendor_board_info='{"vendor_info": "gpu_vb_info"}'). Reason: Failed to create trait OWNER_CYBORG: HTTP 400: {"errors": [{"status": 400, "title": "Bad Request", "detail": "The server could not comply with the request since it is either malformed or otherwise incorrect.\n\n The trait is invalid. A valid trait must be no longer than 255 characters, start with the prefix \"CUSTOM_\" and use following characters: \"A\"-\"Z\", \"0\"-\"9\" and \"_\" ", "request_id": "req-24634fad-b312-4ad3-9482-09d6df2b26c2"}]} ``` It appears Cyborg is attempting to create the trait OWNER_CYBORG but I cannot seem to get it to stop that, can I please have some assistance from the Cyborg team to overcome this? Kind Regards, -- Karl Kloppenborg, (BCompSc, CNCF-[KCNA, CKA, CKAD], LFCE, CompTIA Linux+ XK0-004) Managing Director, Invention Labs.

1 0

In Person PTG June 2023 Teams List
by Kendall Nelson 11 Apr '23

11 Apr '23

Hello Everyone! The June 2023 Project Teams List is official! Projects + Teams - OpenInfra Foundation Teams - - Diversity & Inclusion Working Group - Environmental Sustainability Working Group - Kata Containers - StarlingX - OpenStack - - Service Teams - - Cinder - Cyborg - Designate - Heat - Ironic - Keystone - Manila - Neutron - Nova - OpenStack-Ansible - Puppet OpenStack - QA - SDK/CLI/AOC - Tacker - Telemetry - Other OpenStack Teams - - Security SIG - Technical Committee - Scientific SIG If your team was planning to meet and isn’t in this list, please contact ptg(a)openinfra.dev IMMEDIATELY. In a couple weeks, I will be contacting moderators to sign up for time via the PTGBot[1] once I have it configured for the teams that are signed up to attend the event. Otherwise, please don't forget to register[2] — prices go up May 5th! Thanks! -Kendall (diablo_rojo) [1] PTGBot Docs: https://github.com/openstack/ptgbot#open-infrastructure-ptg-bot [2] PTG Registration: https://vancouver2023.openinfra.dev/

1 0

[openstack][sharing][kolla ansible]Problems when 1 of 3 controller was be down
by Nguyễn Hữu Khôi 11 Apr '23

11 Apr '23

Hello everyone. I want to summary for who meets problems with Openstack when deploy cluster with 3 controller using Kolla Ansible Scenario: 1 of 3 controller is down 1. Login horizon and use API such as nova, cinder will be very slow fix by: nano: kolla-ansible/ansible/roles/heat/templates/heat.conf.j2 kolla-ansible/ansible/roles/nova/templates/nova.conf.j2 kolla-ansible/ansible/roles/keystone/templates/keystone.conf.j2 kolla-ansible/ansible/roles/neutron/templates/neutron.conf.j2 kolla-ansible/ansible/roles/cinder/templates/cinder.conf.j2 or which service need caches add as below [cache] backend = oslo_cache.memcache_pool enabled = True memcache_servers = {{ kolla_internal_vip_address }}:{{ memcached_port }} memcache_dead_retry = 0.25 memcache_socket_timeout = 900 https://review.opendev.org/c/openstack/kolla-ansible/+/849487 but it is not the end 2. Cannot launch instance or mapping block device(stuck at this step) nano kolla-ansible/ansible/roles/rabbitmq/templates/definitions.json.j2 "policies":[ {"vhost": "/", "name": "ha-all", "pattern": "^(?!(amq\.)|(.*_fanout_)|(reply_)).*", "apply-to": "all", "definition": {"ha-mode":"all"}, "priority":0}{% if project_name == 'outward_rabbitmq' %}, {"vhost": "{{ murano_agent_rabbitmq_vhost }}", "name": "ha-all", "pattern": ".*", "apply-to": "all", "definition": {"ha-mode":"all"}, "priority":0} {% endif %} ] nano /etc/kollla/global.conf [oslo_messaging_rabbit] kombu_reconnect_delay=0.5 https://bugs.launchpad.net/oslo.messaging/+bug/1993149 https://docs.openstack.org/large-scale/journey/configure/rabbitmq.html I used Xena 13.4 and Yoga 14.8.1. Above bugs are critical, but I see that it was not fixed. I am just an operator and I want to share what I encountered for new people who come to Openstack Nguyen Huu Khoi

6 11

Re: StackOver Flow Error: 2103 ERROR oslo.messaging._drivers.impl_rabbit Connection failed: failed to resolve broker hostname
by Dmitriy Rabotyagov 11 Apr '23

11 Apr '23

If you're going to go prod I would for sure use some deployment tool rather then manual deployment, as further operations and scaling up would be way easier. What error do you get with existing hosts record? вт, 11 апр. 2023 г., 14:57 Trey Fleishman <nfleishman(a)student.ccaeagles.org >: > Hello Dmitriy, > > That is what I thought too, in my host file I define the hostname as the > correct IP address, and if I change the "controller" hostname to some > factitious IP I get a different error. I've tried putting the IP instead of > the hostname, but to no avail. I can successfully ping the hostname from > the machine. I would also rather not use a tool as I plan to go production > once I get my two node setup functional. I am not sure if this might be a > configuration issue, or maybe a bug. > Trey Fleishman > Student and IT Intern > (704) 750 - 1740 > *Calvary Chapel Fort Lauderdale* > 2401 W. Cypress Creek Rd., Ft. Lauderdale, FL 33309 > CalvaryFTL.org > > > On Tue, Apr 11, 2023 at 12:54 AM Dmitriy Rabotyagov < > noonedeadpunk(a)gmail.com> wrote: > >> Hi Trey, >> >> Have you defined a record in your /etc/hosts for "controller", as >> according to the error service simply can not resolve this hostname to an >> IP address. >> >> Also I can suggest trying out one of the following OpenStack deployment >> tools: >> https://docs.openstack.org/zed/deploy/index.html >> >> Most of them have All-In-One guides that can spawn an opestack sandbox to >> play with. >> >> вт, 11 апр. 2023 г., 00:21 Trey Fleishman < >> nfleishman(a)student.ccaeagles.org>: >> >>> Hello all, >>> Recently I tried installing Openstack via this >>> <https://docs.openstack.org/install-guide/openstack-services.html> >>> guide. Upon completion of all the steps including the dashboard step, I was >>> greeted with this error inside the nova_schedular.log file, after I tried >>> creating an instance. >>> >>> 2103 ERROR oslo.messaging._drivers.impl_rabbit Connection failed: failed >>> to resolve broker hostname: OSError: failed to resolve broker hostname >>> >>> When I try to create an instance it hangs in the "scheduling" state and >>> throws this error. I have not configured anything in the "oslo.messaging" >>> section of the Nova config file. I believe that under the [default] section >>> this is what's causing the issue, but the password and hostname are fine. >>> >>> transport_url = >>> rabbit://openstack:BnE9LLfCtjdPvpEnZUnPytSyrfUxE8th@controller >>> >>> Thanks! >>> >>

2 1

[keystone] Weekly Meeting Cancelled (11-Apr-2023)
by Dave Wilde 11 Apr '23

11 Apr '23

Hi Folks, Unfortunately I have to cancel the weekly meeting today. Please reach out to me directly via email if you need anything. Due to equipment failure my IRC availability is spotty. Thanks! /Dave

1 0

[Kolla-ansible][Yoga] Cannot live migrate VMs
by wodel youchi 11 Apr '23

11 Apr '23

Hi, I cannot live-migrate any VM I have. All my compute nodes are enabled. I have a few VMs, I am in the testing phase. I tried to live-migrate toward an empty node, then toward a node which already had a VM in it. It is the same problem. In the log files I have : Source node : Source > ==> /var/log/kolla/nova/nova-compute.log <== > 2023-04-09 11:32:49.419 7 INFO nova.compute.resource_tracker > [req-4e91d562-d7ad-4585-a827-3db6277fd81e - - - - -] [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Updating resource usage from > migration f60eb077-8e1c-404a-aca5-64e5609ca02c > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [-] [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Pre live migration failed at > computehci02: oslo_messaging.rpc.client.RemoteError: Remote error: > ClientException Gateway Time-out (HTTP 504) > ['Traceback (most recent call last):\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", > line 165, in _process_incoming\n res = > self.dispatcher.dispatch(message)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", > line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, > args)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", > line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", > line 72, in wrapped\n context, exc, binary)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 227, in __exit__\n self.force_reraise()\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 200, in force_reraise\n raise self.value\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", > line 63, in wrapped\n return f(self, context, *args, **kw)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/utils.py", > line 1439, in decorated_function\n return function(self, context, *args, > **kwargs)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", > line 213, in decorated_function\n kwargs[\'instance\'], e, > sys.exc_info())\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 227, in __exit__\n self.force_reraise()\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 200, in force_reraise\n raise self.value\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", > line 201, in decorated_function\n return function(self, context, *args, > **kwargs)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", > line 8367, in pre_live_migration\n bdm.save()\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 227, in __exit__\n self.force_reraise()\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 200, in force_reraise\n raise self.value\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", > line 8304, in pre_live_migration\n connector=connector, > mountpoint=bdm.device_name)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", > line 395, in wrapper\n res = method(self, ctx, *args, **kwargs)\n', ' > File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", > line 433, in wrapper\n res = method(self, ctx, volume_id, *args, > **kwargs)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", > line 813, in attachment_create\n instance_uuid=instance_id)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 227, in __exit__\n self.force_reraise()\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 200, in force_reraise\n raise self.value\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", > line 801, in attachment_create\n volume_id, _connector, instance_id)\n', > ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/api_versions.py", > line 421, in substitution\n return method.func(obj, *args, **kwargs)\n', > ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/v3/attachments.py", > line 39, in create\n retval = self._create(\'/attachments\', body, > \'attachment\')\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/base.py", > line 306, in _create\n resp, body = self.api.client.post(url, > body=body)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", > line 227, in post\n return self._cs_request(url, \'POST\', **kwargs)\n', > ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", > line 215, in _cs_request\n return self.request(url, method, > **kwargs)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", > line 201, in request\n raise exceptions.from_response(resp, body)\n', > 'cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504)\n']. > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Traceback (most recent call last): > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", > line 8501, in _do_pre_live_migration_from_source > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] block_migration, disk, dest, > migrate_data) > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/rpcapi.py", > line 948, in pre_live_migration > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] **msg_args) > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/client.py", > line 192, in call > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] retry=self.retry, > transport_options=self.transport_options) > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/transport.py", > line 128, in _send > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] > transport_options=transport_options) > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", > line 691, in send > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] > transport_options=transport_options) > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", > line 681, in _send > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] raise result > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] > oslo_messaging.rpc.client.RemoteError: Remote error: ClientException > Gateway Time-out (HTTP 504) > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] ['Traceback (most recent call > last):\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", > line 165, in _process_incoming\n res = > self.dispatcher.dispatch(message)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", > line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, > args)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", > line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", > line 72, in wrapped\n context, exc, binary)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 227, in __exit__\n self.force_reraise()\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 200, in force_reraise\n raise self.value\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", > line 63, in wrapped\n return f(self, context, *args, **kw)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/utils.py", > line 1439, in decorated_function\n return function(self, context, *args, > **kwargs)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", > line 213, in decorated_function\n kwargs[\'instance\'], e, > sys.exc_info())\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 227, in __exit__\n self.force_reraise()\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 200, in force_reraise\n raise self.value\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", > line 201, in decorated_function\n return function(self, context, *args, > **kwargs)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", > line 8367, in pre_live_migration\n bdm.save()\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 227, in __exit__\n self.force_reraise()\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 200, in force_reraise\n raise self.value\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", > line 8304, in pre_live_migration\n connector=connector, > mountpoint=bdm.device_name)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", > line 395, in wrapper\n res = method(self, ctx, *args, **kwargs)\n', ' > File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", > line 433, in wrapper\n res = method(self, ctx, volume_id, *args, > **kwargs)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", > line 813, in attachment_create\n instance_uuid=instance_id)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 227, in __exit__\n self.force_reraise()\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 200, in force_reraise\n raise self.value\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", > line 801, in attachment_create\n volume_id, _connector, instance_id)\n', > ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/api_versions.py", > line 421, in substitution\n return method.func(obj, *args, **kwargs)\n', > ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/v3/attachments.py", > line 39, in create\n retval = self._create(\'/attachments\', body, > \'attachment\')\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/base.py", > line 306, in _create\n resp, body = self.api.client.post(url, > body=body)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", > line 227, in post\n return self._cs_request(url, \'POST\', **kwargs)\n', > ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", > line 215, in _cs_request\n return self.request(url, method, > **kwargs)\n', ' File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", > line 201, in request\n raise exceptions.from_response(resp, body)\n', > 'cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504)\n']. > 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] > 2023-04-09 11:33:30.157 7 INFO nova.compute.manager [-] [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Swapping old allocation on > dict_keys(['2857ae62-3519-4b13-b700-858c827dd78d']) held by migration > f60eb077-8e1c-404a-aca5-64e5609ca02c for instance In the destination node : > nova-compute.log > > > > > *2023-04-09 11:32:30.052 7 WARNING os_brick.initiator.connectors.nvmeof > [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 > b0f76b5c6dcb457fa716762bbf954837 - default default] Process execution error > in _get_host_uuid: Unexpected error while running command.Command: blkid > overlay -s UUID -o valueExit code: 2Stdout: ''Stderr: '': > oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while > running command.* > > 2023-04-09 11:33:05.354 7 INFO nova.compute.resource_tracker > [req-9d480442-87f7-4d4e-a866-c46d645ac7f2 - - - - -] [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Updating resource usage from > migration f60eb077-8e1c-404a-aca5-64e5609ca02c > 2023-04-09 11:33:05.392 7 WARNING nova.compute.resource_tracker > [req-9d480442-87f7-4d4e-a866-c46d645ac7f2 - - - - -] Instance > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7 has been moved to another host > computehci01(computehci01). There are allocations remaining against the > source host that might need to be removed: {'resources': {'VCPU': 1, > 'MEMORY_MB': 2048}}. > 2023-04-09 11:33:30.086 7 ERROR nova.volume.cinder > [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 > b0f76b5c6dcb457fa716762bbf954837 - default default] [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Create attachment failed for volume > 3d7bd0af-5e4c-48fc-a027-f6c9dd6280bb. Error: Gateway Time-out (HTTP 504) > Code: 504: cinderclient.exceptions.ClientException: Gateway Time-out (HTTP > 504) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server > [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 > b0f76b5c6dcb457fa716762bbf954837 - default default] Exception during > message handling: cinderclient.exceptions.ClientException: Gateway Time-out > (HTTP 504) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server Traceback (most > recent call last): > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", > line 165, in _process_incoming > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server res = > self.dispatcher.dispatch(message) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", > line 309, in dispatch > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return > self._do_dispatch(endpoint, method, ctxt, args) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", > line 229, in _do_dispatch > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server result = > func(ctxt, **new_args) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", > line 72, in wrapped > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server context, > exc, binary) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 227, in __exit__ > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server > self.force_reraise() > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 200, in force_reraise > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise > self.value > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", > line 63, in wrapped > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return > f(self, context, *args, **kw) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/utils.py", > line 1439, in decorated_function > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return > function(self, context, *args, **kwargs) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", > line 213, in decorated_function > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server > kwargs['instance'], e, sys.exc_info()) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 227, in __exit__ > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server > self.force_reraise() > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 200, in force_reraise > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise > self.value > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", > line 201, in decorated_function > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return > function(self, context, *args, **kwargs) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", > line 8367, in pre_live_migration > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server bdm.save() > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 227, in __exit__ > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server > self.force_reraise() > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 200, in force_reraise > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise > self.value > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", > line 8304, in pre_live_migration > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server > connector=connector, mountpoint=bdm.device_name) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", > line 395, in wrapper > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server res = > method(self, ctx, *args, **kwargs) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", > line 433, in wrapper > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server res = > method(self, ctx, volume_id, *args, **kwargs) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", > line 813, in attachment_create > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server > instance_uuid=instance_id) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 227, in __exit__ > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server > self.force_reraise() > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 200, in force_reraise > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise > self.value > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", > line 801, in attachment_create > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server volume_id, > _connector, instance_id) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/api_versions.py", > line 421, in substitution > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return > method.func(obj, *args, **kwargs) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/v3/attachments.py", > line 39, in create > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server retval = > self._create('/attachments', body, 'attachment') > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/base.py", > line 306, in _create > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server resp, body = > self.api.client.post(url, body=body) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", > line 227, in post > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return > self._cs_request(url, 'POST', **kwargs) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", > line 215, in _cs_request > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return > self.request(url, method, **kwargs) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File > "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", > line 201, in request > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise > exceptions.from_response(resp, body) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server > cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504) > 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server > 2023-04-09 11:33:30.238 7 INFO nova.virt.block_device > [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 > b0f76b5c6dcb457fa716762bbf954837 - default default] [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Attempting to driver detach volume > 3d7bd0af-5e4c-48fc-a027-f6c9dd6280bb from mountpoint /dev/vda > 2023-04-09 11:33:30.239 7 WARNING nova.virt.block_device > [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 > b0f76b5c6dcb457fa716762bbf954837 - default default] [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Detaching volume from unknown instance > 2023-04-09 11:33:30.240 7 WARNING nova.virt.libvirt.driver > [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 > b0f76b5c6dcb457fa716762bbf954837 - default default] [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] During detach_volume, instance > disappeared.: nova.exception.InstanceNotFound: Instance > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7 could not be found. > 2023-04-09 11:33:30.678 7 INFO nova.virt.libvirt.driver [-] [instance: > f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Instance destroyed successfully. > 2023-04-09 11:33:30.682 7 INFO os_vif > [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 > b0f76b5c6dcb457fa716762bbf954837 - default default] Successfully unplugged > vif > VIFBridge(active=True,address=fa:16:3e:9b:92:46,bridge_name='qbr46651303-65',has_traffic_filtering=True,id=46651303-65e1-4809-95cc-7c2d97acf16b,network=Network(532470c1-dd2f-4505-8557-ba24e9d4062b),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap46651303-65') > All I can see is *b0f76b5c6dcb457fa716762bbf954837 - default default] Process execution error in _get_host_uuid: Unexpected error while running command.Command: blkid overlay -s UUID -o valueExit code: 2Stdout: ''Stderr: '': oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.* Searching the web I didn't find anything useful about this error, and I don't know if it's the origin of the problem. My default storage is ceph rbd, I have an HCI deployment, and all my VMs are for now created on Ceph only. But I have configured two other cinder backends, NFS and LVM, but they are node used for now. Thanks in advance. Regards.

3 4