DCN compute service goes down when a instance is scheduled to launch | wallaby | tripleo
Swogat Pradhan
swogatpradhan22 at gmail.com
Tue Mar 28 02:09:23 UTC 2023
Hi Alan,
Thank you so much.
The issue was with the images. Qcow2 images are not working.
I used RAW images and it now takes 2 mins to spawn the instances without
any issues.
Thank you
With regards,
Swogat Pradhan
On Thu, Mar 23, 2023 at 10:35 PM Alan Bishop <abishop at redhat.com> wrote:
>
>
> On Thu, Mar 23, 2023 at 9:01 AM Swogat Pradhan <swogatpradhan22 at gmail.com>
> wrote:
>
>> Hi,
>> Can someone please help me identify the issue here?
>> Latest cinder-volume logs from dcn02:
>> (ATTACHED)
>>
>
> It's really not possible to analyze what's happening with just one or two
> log entries. Do you have
> debug logs enabled? One thing I noticed is the glance image's disk_format
> is qcow2. You should
> use "raw" images with ceph RBD.
>
> Alan
>
>
>>
>> The volume is stuck in creating state.
>>
>> With regards,
>> Swogat Pradhan
>>
>> On Thu, Mar 23, 2023 at 6:12 PM Swogat Pradhan <swogatpradhan22 at gmail.com>
>> wrote:
>>
>>> Hi Jhon,
>>> Thank you for clarifying that.
>>> Right now the cinder volume is stuck in *creating *state when adding
>>> image as volume source.
>>> But when creating an empty volume the volumes are getting created
>>> successfully without any errors.
>>>
>>> We are getting volume creation request in cinder-volume.log as such:
>>> 2023-03-23 12:34:40.152 108 INFO
>>> cinder.volume.flows.manager.create_volume
>>> [req-18556796-a61c-4097-8fa8-b136ce9814f7 b240e3e89d99489284cd731e75f2a5db
>>> 4160ce999a31485fa643aed0936dfef0 - - -] Volume
>>> 872a2ae6-c75b-4fc0-8172-17a29d07a66c: being created as image with
>>> specification: {'status': 'creating', 'volume_name':
>>> 'volume-872a2ae6-c75b-4fc0-8172-17a29d07a66c', 'volume_size': 1,
>>> 'image_id': '131ed4e0-0474-45be-b74a-43b599a7d6c5', 'image_location':
>>> ('rbd://a5ae877c-bcba-53fe-8336-450e63014757/images/131ed4e0-0474-45be-b74a-43b599a7d6c5/snap',
>>> [{'url':
>>> 'rbd://a5ae877c-bcba-53fe-8336-450e63014757/images/131ed4e0-0474-45be-b74a-43b599a7d6c5/snap',
>>> 'metadata': {'store': 'ceph'}}, {'url':
>>> 'rbd://a8d5f1f5-48e7-5ede-89ab-8aca59b6397b/images/131ed4e0-0474-45be-b74a-43b599a7d6c5/snap',
>>> 'metadata': {'store': 'dcn02'}}]), 'image_meta': {'name': 'cirros',
>>> 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public',
>>> 'size': 16338944, 'virtual_size': 117440512, 'status': 'active',
>>> 'checksum': '1d3062cd89af34e419f7100277f38b2b', 'protected': False,
>>> 'min_ram': 0, 'min_disk': 0, 'owner': '4160ce999a31485fa643aed0936dfef0',
>>> 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value':
>>> '553d220ed58cfee7dafe003c446a9f197ab5edf8ffc09396c74187cf83873c877e7ae041cb80f3b91489acf687183adcd689b53b38e3ddd22e627e7f98a09c46',
>>> 'id': '131ed4e0-0474-45be-b74a-43b599a7d6c5', 'created_at':
>>> datetime.datetime(2023, 3, 23, 11, 41, 51, tzinfo=datetime.timezone.utc),
>>> 'updated_at': datetime.datetime(2023, 3, 23, 11, 46, 37,
>>> tzinfo=datetime.timezone.utc), 'locations': [{'url':
>>> 'rbd://a5ae877c-bcba-53fe-8336-450e63014757/images/131ed4e0-0474-45be-b74a-43b599a7d6c5/snap',
>>> 'metadata': {'store': 'ceph'}}, {'url':
>>> 'rbd://a8d5f1f5-48e7-5ede-89ab-8aca59b6397b/images/131ed4e0-0474-45be-b74a-43b599a7d6c5/snap',
>>> 'metadata': {'store': 'dcn02'}}], 'direct_url':
>>> 'rbd://a5ae877c-bcba-53fe-8336-450e63014757/images/131ed4e0-0474-45be-b74a-43b599a7d6c5/snap',
>>> 'tags': [], 'file': '/v2/images/131ed4e0-0474-45be-b74a-43b599a7d6c5/file',
>>> 'stores': 'ceph,dcn02', 'properties': {'os_glance_failed_import': '',
>>> 'os_glance_importing_to_stores': '', 'owner_specified.openstack.md5': '',
>>> 'owner_specified.openstack.object': 'images/cirros',
>>> 'owner_specified.openstack.sha256': ''}}, 'image_service':
>>> <cinder.image.glance.GlanceImageService object at 0x7f98d869ed68>}
>>>
>>> But there is nothing else after that and the volume doesn't even
>>> timeout, it just gets stuck in creating state.
>>> Can you advise what might be the issue here?
>>> All the containers are in a healthy state now.
>>>
>>> With regards,
>>> Swogat Pradhan
>>>
>>>
>>> On Thu, Mar 23, 2023 at 6:06 PM Alan Bishop <abishop at redhat.com> wrote:
>>>
>>>>
>>>>
>>>> On Thu, Mar 23, 2023 at 5:20 AM Swogat Pradhan <
>>>> swogatpradhan22 at gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>> Is this bind not required for cinder_scheduler container?
>>>>>
>>>>> "/var/lib/tripleo-config/ceph:/var/lib/kolla/config_files/src-ceph:ro,rprivate,rbind",
>>>>> I do not see this particular bind on the cinder scheduler containers
>>>>> on my controller nodes.
>>>>>
>>>>
>>>> That is correct, because the scheduler does not access the ceph
>>>> cluster.
>>>>
>>>> Alan
>>>>
>>>>
>>>>> With regards,
>>>>> Swogat Pradhan
>>>>>
>>>>> On Thu, Mar 23, 2023 at 2:46 AM Swogat Pradhan <
>>>>> swogatpradhan22 at gmail.com> wrote:
>>>>>
>>>>>> Cinder volume config:
>>>>>>
>>>>>> [tripleo_ceph]
>>>>>> volume_backend_name=tripleo_ceph
>>>>>> volume_driver=cinder.volume.drivers.rbd.RBDDriver
>>>>>> rbd_user=openstack
>>>>>> rbd_pool=volumes
>>>>>> rbd_flatten_volume_from_snapshot=False
>>>>>> rbd_secret_uuid=a8d5f1f5-48e7-5ede-89ab-8aca59b6397b
>>>>>> report_discard_supported=True
>>>>>> rbd_ceph_conf=/etc/ceph/dcn02.conf
>>>>>> rbd_cluster_name=dcn02
>>>>>>
>>>>>> Glance api config:
>>>>>>
>>>>>> [dcn02]
>>>>>> rbd_store_ceph_conf=/etc/ceph/dcn02.conf
>>>>>> rbd_store_user=openstack
>>>>>> rbd_store_pool=images
>>>>>> rbd_thin_provisioning=False
>>>>>> store_description=dcn02 rbd glance store
>>>>>> [ceph]
>>>>>> rbd_store_ceph_conf=/etc/ceph/ceph.conf
>>>>>> rbd_store_user=openstack
>>>>>> rbd_store_pool=images
>>>>>> rbd_thin_provisioning=False
>>>>>> store_description=Default glance store backend.
>>>>>>
>>>>>> On Thu, Mar 23, 2023 at 2:29 AM Swogat Pradhan <
>>>>>> swogatpradhan22 at gmail.com> wrote:
>>>>>>
>>>>>>> I still have the same issue, I'm not sure what's left to try.
>>>>>>> All the pods are now in a healthy state, I am getting log entries 3
>>>>>>> mins after I hit the create volume button in cinder-volume when I try to
>>>>>>> create a volume with an image.
>>>>>>> And the volumes are just stuck in creating state for more than 20
>>>>>>> mins now.
>>>>>>>
>>>>>>> Cinder logs:
>>>>>>> 2023-03-22 20:32:44.010 108 INFO cinder.rpc
>>>>>>> [req-0d2093a0-efbd-45a5-bd7d-cce25ddc200e b240e3e89d99489284cd731e75f2a5db
>>>>>>> 4160ce999a31485fa643aed0936dfef0 - - -] Automatically selected
>>>>>>> cinder-volume RPC version 3.17 as minimum service version.
>>>>>>> 2023-03-22 20:34:59.166 108 INFO
>>>>>>> cinder.volume.flows.manager.create_volume
>>>>>>> [req-0d2093a0-efbd-45a5-bd7d-cce25ddc200e b240e3e89d99489284cd731e75f2a5db
>>>>>>> 4160ce999a31485fa643aed0936dfef0 - - -] Volume
>>>>>>> 5743a879-090d-46db-bc7c-1c0b0669a112: being created as image with
>>>>>>> specification: {'status': 'creating', 'volume_name':
>>>>>>> 'volume-5743a879-090d-46db-bc7c-1c0b0669a112', 'volume_size': 2,
>>>>>>> 'image_id': 'acfd0a14-69e0-44d6-a6a1-aa9dc83e9d5b', 'image_location':
>>>>>>> ('rbd://a5ae877c-bcba-53fe-8336-450e63014757/images/acfd0a14-69e0-44d6-a6a1-aa9dc83e9d5b/snap',
>>>>>>> [{'url':
>>>>>>> 'rbd://a5ae877c-bcba-53fe-8336-450e63014757/images/acfd0a14-69e0-44d6-a6a1-aa9dc83e9d5b/snap',
>>>>>>> 'metadata': {'store': 'ceph'}}, {'url':
>>>>>>> 'rbd://a8d5f1f5-48e7-5ede-89ab-8aca59b6397b/images/acfd0a14-69e0-44d6-a6a1-aa9dc83e9d5b/snap',
>>>>>>> 'metadata': {'store': 'dcn02'}}]), 'image_meta': {'name': 'cirros',
>>>>>>> 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public',
>>>>>>> 'size': 16338944, 'virtual_size': 117440512, 'status': 'active',
>>>>>>> 'checksum': '1d3062cd89af34e419f7100277f38b2b', 'protected': False,
>>>>>>> 'min_ram': 0, 'min_disk': 0, 'owner': '4160ce999a31485fa643aed0936dfef0',
>>>>>>> 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value':
>>>>>>> '553d220ed58cfee7dafe003c446a9f197ab5edf8ffc09396c74187cf83873c877e7ae041cb80f3b91489acf687183adcd689b53b38e3ddd22e627e7f98a09c46',
>>>>>>> 'id': 'acfd0a14-69e0-44d6-a6a1-aa9dc83e9d5b', 'created_at':
>>>>>>> datetime.datetime(2023, 3, 22, 18, 50, 5, tzinfo=datetime.timezone.utc),
>>>>>>> 'updated_at': datetime.datetime(2023, 3, 22, 20, 3, 54,
>>>>>>> tzinfo=datetime.timezone.utc), 'locations': [{'url':
>>>>>>> 'rbd://a5ae877c-bcba-53fe-8336-450e63014757/images/acfd0a14-69e0-44d6-a6a1-aa9dc83e9d5b/snap',
>>>>>>> 'metadata': {'store': 'ceph'}}, {'url':
>>>>>>> 'rbd://a8d5f1f5-48e7-5ede-89ab-8aca59b6397b/images/acfd0a14-69e0-44d6-a6a1-aa9dc83e9d5b/snap',
>>>>>>> 'metadata': {'store': 'dcn02'}}], 'direct_url':
>>>>>>> 'rbd://a5ae877c-bcba-53fe-8336-450e63014757/images/acfd0a14-69e0-44d6-a6a1-aa9dc83e9d5b/snap',
>>>>>>> 'tags': [], 'file': '/v2/images/acfd0a14-69e0-44d6-a6a1-aa9dc83e9d5b/file',
>>>>>>> 'stores': 'ceph,dcn02', 'properties': {'os_glance_failed_import': '',
>>>>>>> 'os_glance_importing_to_stores': '', 'owner_specified.openstack.md5': '',
>>>>>>> 'owner_specified.openstack.object': 'images/cirros',
>>>>>>> 'owner_specified.openstack.sha256': ''}}, 'image_service':
>>>>>>> <cinder.image.glance.GlanceImageService object at 0x7f8147973438>}
>>>>>>>
>>>>>>> With regards,
>>>>>>> Swogat Pradhan
>>>>>>>
>>>>>>> On Wed, Mar 22, 2023 at 9:19 PM Alan Bishop <abishop at redhat.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Mar 22, 2023 at 8:38 AM Swogat Pradhan <
>>>>>>>> swogatpradhan22 at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Adam,
>>>>>>>>> The systems are in same LAN, in this case it seemed like the image
>>>>>>>>> was getting pulled from the central site which was caused due to an
>>>>>>>>> misconfiguration in ceph.conf file in /var/lib/tripleo-config/ceph/
>>>>>>>>> directory, which seems to have been resolved after the changes i made to
>>>>>>>>> fix it.
>>>>>>>>>
>>>>>>>>> Right now the glance api podman is running in unhealthy state and
>>>>>>>>> the podman logs don't show any error whatsoever and when issued the command
>>>>>>>>> netstat -nultp i do not see any entry for glance port i.e. 9292 in the dcn
>>>>>>>>> site, which is why cinder is throwing an error stating:
>>>>>>>>>
>>>>>>>>> 2023-03-22 13:32:29.786 108 ERROR oslo_messaging.rpc.server
>>>>>>>>> cinder.exception.GlanceConnectionFailed: Connection to glance failed: Error
>>>>>>>>> finding address for
>>>>>>>>> http://172.25.228.253:9292/v2/images/736d8779-07cd-4510-bab2-adcb653cc538:
>>>>>>>>> Unable to establish connection to
>>>>>>>>> http://172.25.228.253:9292/v2/images/736d8779-07cd-4510-bab2-adcb653cc538:
>>>>>>>>> HTTPConnectionPool(host='172.25.228.253', port=9292): Max retries exceeded
>>>>>>>>> with url: /v2/images/736d8779-07cd-4510-bab2-adcb653cc538 (Caused by
>>>>>>>>> NewConnectionError('<urllib3.connection.HTTPConnection object at
>>>>>>>>> 0x7f7682d2cd30>: Failed to establish a new connection: [Errno 111]
>>>>>>>>> ECONNREFUSED',))
>>>>>>>>>
>>>>>>>>> Now i need to find out why the port is not listed as the glance
>>>>>>>>> service is running, which i am not sure how to find out.
>>>>>>>>>
>>>>>>>>
>>>>>>>> One other thing to investigate is whether your deployment includes
>>>>>>>> this patch [1]. If it does, then bear in mind
>>>>>>>> the glance-api service running at the edge site will be an
>>>>>>>> "internal" (non public facing) instance that uses port 9293
>>>>>>>> instead of 9292. You should familiarize yourself with the release
>>>>>>>> note [2].
>>>>>>>>
>>>>>>>> [1]
>>>>>>>> https://opendev.org/openstack/tripleo-heat-templates/commit/3605d45e417a77a1d0f153fbeffcbb283ec85fe6
>>>>>>>> [2]
>>>>>>>> https://opendev.org/openstack/tripleo-heat-templates/src/branch/stable/wallaby/releasenotes/notes/glance-internal-service-86274f56712ffaac.yaml
>>>>>>>>
>>>>>>>> Alan
>>>>>>>>
>>>>>>>>
>>>>>>>>> With regards,
>>>>>>>>> Swogat Pradhan
>>>>>>>>>
>>>>>>>>> On Wed, Mar 22, 2023 at 8:11 PM Alan Bishop <abishop at redhat.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Mar 22, 2023 at 6:37 AM Swogat Pradhan <
>>>>>>>>>> swogatpradhan22 at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Update:
>>>>>>>>>>> Here is the log when creating a volume using cirros image:
>>>>>>>>>>>
>>>>>>>>>>> 2023-03-22 11:04:38.449 109 INFO
>>>>>>>>>>> cinder.volume.flows.manager.create_volume
>>>>>>>>>>> [req-646b9ac8-a5a7-45ac-a96d-8dd6bb45da17 b240e3e89d99489284cd731e75f2a5db
>>>>>>>>>>> 4160ce999a31485fa643aed0936dfef0 - - -] Volume
>>>>>>>>>>> bf341343-6609-4b8c-b9e0-93e2a89c8c8f: being created as image with
>>>>>>>>>>> specification: {'status': 'creating', 'volume_name':
>>>>>>>>>>> 'volume-bf341343-6609-4b8c-b9e0-93e2a89c8c8f', 'volume_size': 4,
>>>>>>>>>>> 'image_id': '736d8779-07cd-4510-bab2-adcb653cc538', 'image_location':
>>>>>>>>>>> ('rbd://a5ae877c-bcba-53fe-8336-450e63014757/images/736d8779-07cd-4510-bab2-adcb653cc538/snap',
>>>>>>>>>>> [{'url':
>>>>>>>>>>> 'rbd://a5ae877c-bcba-53fe-8336-450e63014757/images/736d8779-07cd-4510-bab2-adcb653cc538/snap',
>>>>>>>>>>> 'metadata': {'store': 'ceph'}}, {'url':
>>>>>>>>>>> 'rbd://a8d5f1f5-48e7-5ede-89ab-8aca59b6397b/images/736d8779-07cd-4510-bab2-adcb653cc538/snap',
>>>>>>>>>>> 'metadata': {'store': 'dcn02'}}]), 'image_meta': {'name': 'cirros',
>>>>>>>>>>> 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public',
>>>>>>>>>>> 'size': 16338944, 'virtual_size': 117440512, 'status': 'active',
>>>>>>>>>>> 'checksum': '1d3062cd89af34e419f7100277f38b2b', 'protected': False,
>>>>>>>>>>> 'min_ram': 0, 'min_disk': 0, 'owner': '4160ce999a31485fa643aed0936dfef0',
>>>>>>>>>>> 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value':
>>>>>>>>>>> '553d220ed58cfee7dafe003c446a9f197ab5edf8ffc09396c74187cf83873c877e7ae041cb80f3b91489acf687183adcd689b53b38e3ddd22e627e7f98a09c46',
>>>>>>>>>>> 'id': '736d8779-07cd-4510-bab2-adcb653cc538', 'created_at':
>>>>>>>>>>> datetime.datetime(2023, 3, 22, 10, 44, 12, tzinfo=datetime.timezone.utc),
>>>>>>>>>>> 'updated_at': datetime.datetime(2023, 3, 22, 10, 54, 1,
>>>>>>>>>>> tzinfo=datetime.timezone.utc), 'locations': [{'url':
>>>>>>>>>>> 'rbd://a5ae877c-bcba-53fe-8336-450e63014757/images/736d8779-07cd-4510-bab2-adcb653cc538/snap',
>>>>>>>>>>> 'metadata': {'store': 'ceph'}}, {'url':
>>>>>>>>>>> 'rbd://a8d5f1f5-48e7-5ede-89ab-8aca59b6397b/images/736d8779-07cd-4510-bab2-adcb653cc538/snap',
>>>>>>>>>>> 'metadata': {'store': 'dcn02'}}], 'direct_url':
>>>>>>>>>>> 'rbd://a5ae877c-bcba-53fe-8336-450e63014757/images/736d8779-07cd-4510-bab2-adcb653cc538/snap',
>>>>>>>>>>> 'tags': [], 'file': '/v2/images/736d8779-07cd-4510-bab2-adcb653cc538/file',
>>>>>>>>>>> 'stores': 'ceph,dcn02', 'properties': {'os_glance_failed_import': '',
>>>>>>>>>>> 'os_glance_importing_to_stores': '', 'owner_specified.openstack.md5': '',
>>>>>>>>>>> 'owner_specified.openstack.object': 'images/cirros',
>>>>>>>>>>> 'owner_specified.openstack.sha256': ''}}, 'image_service':
>>>>>>>>>>> <cinder.image.glance.GlanceImageService object at 0x7f449ded1198>}
>>>>>>>>>>> 2023-03-22 11:06:16.570 109 INFO cinder.image.image_utils
>>>>>>>>>>> [req-646b9ac8-a5a7-45ac-a96d-8dd6bb45da17 b240e3e89d99489284cd731e75f2a5db
>>>>>>>>>>> 4160ce999a31485fa643aed0936dfef0 - - -] Image download 15.58 MB at 0.16 MB/s
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> As Adam Savage would say, well there's your problem ^^ (Image
>>>>>>>>>> download 15.58 MB at 0.16 MB/s). Downloading the image takes too long, and
>>>>>>>>>> 0.16 MB/s suggests you have a network issue.
>>>>>>>>>>
>>>>>>>>>> John Fulton previously stated your cinder-volume service at the
>>>>>>>>>> edge site is not using the local ceph image store. Assuming you are
>>>>>>>>>> deploying GlanceApiEdge service [1], then the cinder-volume service should
>>>>>>>>>> be configured to use the local glance service [2]. You should check
>>>>>>>>>> cinder's glance_api_servers to confirm it's the edge site's glance service.
>>>>>>>>>>
>>>>>>>>>> [1]
>>>>>>>>>> https://github.com/openstack/tripleo-heat-templates/blob/stable/wallaby/environments/dcn.yaml#L29
>>>>>>>>>> [2]
>>>>>>>>>> https://github.com/openstack/tripleo-heat-templates/blob/stable/wallaby/deployment/glance/glance-api-edge-container-puppet.yaml#L80
>>>>>>>>>>
>>>>>>>>>> Alan
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> 2023-03-22 11:07:54.023 109 WARNING py.warnings
>>>>>>>>>>> [req-646b9ac8-a5a7-45ac-a96d-8dd6bb45da17 b240e3e89d99489284cd731e75f2a5db
>>>>>>>>>>> 4160ce999a31485fa643aed0936dfef0 - - -]
>>>>>>>>>>> /usr/lib/python3.6/site-packages/oslo_utils/imageutils.py:75:
>>>>>>>>>>> FutureWarning: The human format is deprecated and the format parameter will
>>>>>>>>>>> be removed. Use explicitly json instead in version 'xena'
>>>>>>>>>>> category=FutureWarning)
>>>>>>>>>>>
>>>>>>>>>>> 2023-03-22 11:11:12.161 109 WARNING py.warnings
>>>>>>>>>>> [req-646b9ac8-a5a7-45ac-a96d-8dd6bb45da17 b240e3e89d99489284cd731e75f2a5db
>>>>>>>>>>> 4160ce999a31485fa643aed0936dfef0 - - -]
>>>>>>>>>>> /usr/lib/python3.6/site-packages/oslo_utils/imageutils.py:75:
>>>>>>>>>>> FutureWarning: The human format is deprecated and the format parameter will
>>>>>>>>>>> be removed. Use explicitly json instead in version 'xena'
>>>>>>>>>>> category=FutureWarning)
>>>>>>>>>>>
>>>>>>>>>>> 2023-03-22 11:11:12.163 109 INFO cinder.image.image_utils
>>>>>>>>>>> [req-646b9ac8-a5a7-45ac-a96d-8dd6bb45da17 b240e3e89d99489284cd731e75f2a5db
>>>>>>>>>>> 4160ce999a31485fa643aed0936dfef0 - - -] Converted 112.00 MB image at 112.00
>>>>>>>>>>> MB/s
>>>>>>>>>>> 2023-03-22 11:11:14.998 109 INFO
>>>>>>>>>>> cinder.volume.flows.manager.create_volume
>>>>>>>>>>> [req-646b9ac8-a5a7-45ac-a96d-8dd6bb45da17 b240e3e89d99489284cd731e75f2a5db
>>>>>>>>>>> 4160ce999a31485fa643aed0936dfef0 - - -] Volume
>>>>>>>>>>> volume-bf341343-6609-4b8c-b9e0-93e2a89c8c8f
>>>>>>>>>>> (bf341343-6609-4b8c-b9e0-93e2a89c8c8f): created successfully
>>>>>>>>>>> 2023-03-22 11:11:15.195 109 INFO cinder.volume.manager
>>>>>>>>>>> [req-646b9ac8-a5a7-45ac-a96d-8dd6bb45da17 b240e3e89d99489284cd731e75f2a5db
>>>>>>>>>>> 4160ce999a31485fa643aed0936dfef0 - - -] Created volume successfully.
>>>>>>>>>>>
>>>>>>>>>>> The image is present in dcn02 store but still it downloaded the
>>>>>>>>>>> image in 0.16 MB/s and then created the volume.
>>>>>>>>>>>
>>>>>>>>>>> With regards,
>>>>>>>>>>> Swogat Pradhan
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Mar 21, 2023 at 6:10 PM Swogat Pradhan <
>>>>>>>>>>> swogatpradhan22 at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Jhon,
>>>>>>>>>>>> This seems to be an issue.
>>>>>>>>>>>> When i deployed the dcn ceph in both dcn01 and dcn02 the
>>>>>>>>>>>> --cluster parameter was specified to the respective cluster names but the
>>>>>>>>>>>> config files were created in the name of ceph.conf and keyring was
>>>>>>>>>>>> ceph.client.openstack.keyring.
>>>>>>>>>>>>
>>>>>>>>>>>> Which created issues in glance as well as the naming convention
>>>>>>>>>>>> of the files didn't match the cluster names, so i had to manually rename
>>>>>>>>>>>> the central ceph conf file as such:
>>>>>>>>>>>>
>>>>>>>>>>>> [root at dcn02-compute-0 ~]# cd /var/lib/tripleo-config/ceph/
>>>>>>>>>>>> [root at dcn02-compute-0 ceph]# ll
>>>>>>>>>>>> total 16
>>>>>>>>>>>> -rw-------. 1 root root 257 Mar 13 13:56
>>>>>>>>>>>> ceph_central.client.openstack.keyring
>>>>>>>>>>>> -rw-r--r--. 1 root root 428 Mar 13 13:56 ceph_central.conf
>>>>>>>>>>>> -rw-------. 1 root root 205 Mar 15 18:45
>>>>>>>>>>>> ceph.client.openstack.keyring
>>>>>>>>>>>> -rw-r--r--. 1 root root 362 Mar 15 18:45 ceph.conf
>>>>>>>>>>>> [root at dcn02-compute-0 ceph]#
>>>>>>>>>>>>
>>>>>>>>>>>> ceph.conf and ceph.client.openstack.keyring contain the fsid of
>>>>>>>>>>>> the respective clusters in both dcn01 and dcn02.
>>>>>>>>>>>> In the above cli output, the ceph.conf and ceph.client... are
>>>>>>>>>>>> the files used to access dcn02 ceph cluster and ceph_central* files are
>>>>>>>>>>>> used in for accessing central ceph cluster.
>>>>>>>>>>>>
>>>>>>>>>>>> glance multistore config:
>>>>>>>>>>>> [dcn02]
>>>>>>>>>>>> rbd_store_ceph_conf=/etc/ceph/ceph.conf
>>>>>>>>>>>> rbd_store_user=openstack
>>>>>>>>>>>> rbd_store_pool=images
>>>>>>>>>>>> rbd_thin_provisioning=False
>>>>>>>>>>>> store_description=dcn02 rbd glance store
>>>>>>>>>>>>
>>>>>>>>>>>> [ceph_central]
>>>>>>>>>>>> rbd_store_ceph_conf=/etc/ceph/ceph_central.conf
>>>>>>>>>>>> rbd_store_user=openstack
>>>>>>>>>>>> rbd_store_pool=images
>>>>>>>>>>>> rbd_thin_provisioning=False
>>>>>>>>>>>> store_description=Default glance store backend.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> With regards,
>>>>>>>>>>>> Swogat Pradhan
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Mar 21, 2023 at 5:52 PM John Fulton <
>>>>>>>>>>>> johfulto at redhat.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Mar 21, 2023 at 8:03 AM Swogat Pradhan
>>>>>>>>>>>>> <swogatpradhan22 at gmail.com> wrote:
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > Hi,
>>>>>>>>>>>>> > Seems like cinder is not using the local ceph.
>>>>>>>>>>>>>
>>>>>>>>>>>>> That explains the issue. It's a misconfiguration.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I hope this is not a production system since the mailing list
>>>>>>>>>>>>> now has
>>>>>>>>>>>>> the cinder.conf which contains passwords.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The section that looks like this:
>>>>>>>>>>>>>
>>>>>>>>>>>>> [tripleo_ceph]
>>>>>>>>>>>>> volume_backend_name=tripleo_ceph
>>>>>>>>>>>>> volume_driver=cinder.volume.drivers.rbd.RBDDriver
>>>>>>>>>>>>> rbd_ceph_conf=/etc/ceph/ceph.conf
>>>>>>>>>>>>> rbd_user=openstack
>>>>>>>>>>>>> rbd_pool=volumes
>>>>>>>>>>>>> rbd_flatten_volume_from_snapshot=False
>>>>>>>>>>>>> rbd_secret_uuid=<redacted>
>>>>>>>>>>>>> report_discard_supported=True
>>>>>>>>>>>>>
>>>>>>>>>>>>> Should be updated to refer to the local DCN ceph cluster and
>>>>>>>>>>>>> not the
>>>>>>>>>>>>> central one. Use the ceph conf file for that cluster and
>>>>>>>>>>>>> ensure the
>>>>>>>>>>>>> rbd_secret_uuid corresponds to that one.
>>>>>>>>>>>>>
>>>>>>>>>>>>> TripleO’s convention is to set the rbd_secret_uuid to the FSID
>>>>>>>>>>>>> of the
>>>>>>>>>>>>> Ceph cluster. The FSID should be in the ceph.conf file. The
>>>>>>>>>>>>> tripleo_nova_libvirt role will use virsh secret-* commands so
>>>>>>>>>>>>> that
>>>>>>>>>>>>> libvirt can retrieve the cephx secret using the FSID as a key.
>>>>>>>>>>>>> This
>>>>>>>>>>>>> can be confirmed with `podman exec nova_virtsecretd virsh
>>>>>>>>>>>>> secret-get-value $FSID`.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The documentation describes how to configure the central and
>>>>>>>>>>>>> DCN sites
>>>>>>>>>>>>> correctly but an error seems to have occurred while you were
>>>>>>>>>>>>> following
>>>>>>>>>>>>> it.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/features/distributed_multibackend_storage.html
>>>>>>>>>>>>>
>>>>>>>>>>>>> John
>>>>>>>>>>>>>
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > Ceph Output:
>>>>>>>>>>>>> > [ceph: root at dcn02-ceph-all-0 /]# rbd -p images ls -l
>>>>>>>>>>>>> > NAME SIZE PARENT
>>>>>>>>>>>>> FMT PROT LOCK
>>>>>>>>>>>>> > 2abfafaa-eff4-4c2e-a538-dc2e1249ab65 8 MiB
>>>>>>>>>>>>> 2 excl
>>>>>>>>>>>>> > 55f40c8a-8f79-48c5-a52a-9b679b762f19 16 MiB
>>>>>>>>>>>>> 2
>>>>>>>>>>>>> > 55f40c8a-8f79-48c5-a52a-9b679b762f19 at snap 16 MiB
>>>>>>>>>>>>> 2 yes
>>>>>>>>>>>>> > 59f6a9cd-721c-45b5-a15f-fd021b08160d 321 MiB
>>>>>>>>>>>>> 2
>>>>>>>>>>>>> > 59f6a9cd-721c-45b5-a15f-fd021b08160d at snap 321 MiB
>>>>>>>>>>>>> 2 yes
>>>>>>>>>>>>> > 5f5ddd77-35f3-45e8-9dd3-8c1cbb1f39f0 386 MiB
>>>>>>>>>>>>> 2
>>>>>>>>>>>>> > 5f5ddd77-35f3-45e8-9dd3-8c1cbb1f39f0 at snap 386 MiB
>>>>>>>>>>>>> 2 yes
>>>>>>>>>>>>> > 9b27248e-a8cf-4f00-a039-d3e3066cd26a 15 GiB
>>>>>>>>>>>>> 2
>>>>>>>>>>>>> > 9b27248e-a8cf-4f00-a039-d3e3066cd26a at snap 15 GiB
>>>>>>>>>>>>> 2 yes
>>>>>>>>>>>>> > b7356adc-bb47-4c05-968b-6d3c9ca0079b 15 GiB
>>>>>>>>>>>>> 2
>>>>>>>>>>>>> > b7356adc-bb47-4c05-968b-6d3c9ca0079b at snap 15 GiB
>>>>>>>>>>>>> 2 yes
>>>>>>>>>>>>> > e77e78ad-d369-4a1d-b758-8113621269a3 15 GiB
>>>>>>>>>>>>> 2
>>>>>>>>>>>>> > e77e78ad-d369-4a1d-b758-8113621269a3 at snap 15 GiB
>>>>>>>>>>>>> 2 yes
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > [ceph: root at dcn02-ceph-all-0 /]# rbd -p volumes ls -l
>>>>>>>>>>>>> > NAME SIZE
>>>>>>>>>>>>> PARENT FMT PROT LOCK
>>>>>>>>>>>>> > volume-c644086f-d3cf-406d-b0f1-7691bde5981d 100 GiB
>>>>>>>>>>>>> 2
>>>>>>>>>>>>> > volume-f0969935-a742-4744-9375-80bf323e4d63 10 GiB
>>>>>>>>>>>>> 2
>>>>>>>>>>>>> > [ceph: root at dcn02-ceph-all-0 /]#
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > Attached the cinder config.
>>>>>>>>>>>>> > Please let me know how I can solve this issue.
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > With regards,
>>>>>>>>>>>>> > Swogat Pradhan
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > On Tue, Mar 21, 2023 at 3:53 PM John Fulton <
>>>>>>>>>>>>> johfulto at redhat.com> wrote:
>>>>>>>>>>>>> >>
>>>>>>>>>>>>> >> in my last message under the line "On a DCN site if you run
>>>>>>>>>>>>> a command like this:" I suggested some steps you could try to confirm the
>>>>>>>>>>>>> image is a COW from the local glance as well as how to look at your cinder
>>>>>>>>>>>>> config.
>>>>>>>>>>>>> >>
>>>>>>>>>>>>> >> On Tue, Mar 21, 2023, 12:06 AM Swogat Pradhan <
>>>>>>>>>>>>> swogatpradhan22 at gmail.com> wrote:
>>>>>>>>>>>>> >>>
>>>>>>>>>>>>> >>> Update:
>>>>>>>>>>>>> >>> I uploaded an image directly to the dcn02 store, and it
>>>>>>>>>>>>> takes around 10,15 minutes to create a volume with image in dcn02.
>>>>>>>>>>>>> >>> The image size is 389 MB.
>>>>>>>>>>>>> >>>
>>>>>>>>>>>>> >>> On Mon, Mar 20, 2023 at 10:26 PM Swogat Pradhan <
>>>>>>>>>>>>> swogatpradhan22 at gmail.com> wrote:
>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>> >>>> Hi Jhon,
>>>>>>>>>>>>> >>>> I checked in the ceph od dcn02, I can see the images
>>>>>>>>>>>>> created after importing from the central site.
>>>>>>>>>>>>> >>>> But launching an instance normally fails as it takes a
>>>>>>>>>>>>> long time for the volume to get created.
>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>> >>>> When launching an instance from volume the instance is
>>>>>>>>>>>>> getting created properly without any errors.
>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>> >>>> I tried to cache images in nova using
>>>>>>>>>>>>> https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/post_deployment/pre_cache_images.html
>>>>>>>>>>>>> but getting checksum failed error.
>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>> >>>> With regards,
>>>>>>>>>>>>> >>>> Swogat Pradhan
>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>> >>>> On Thu, Mar 16, 2023 at 5:24 PM John Fulton <
>>>>>>>>>>>>> johfulto at redhat.com> wrote:
>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>> >>>>> On Wed, Mar 15, 2023 at 8:05 PM Swogat Pradhan
>>>>>>>>>>>>> >>>>> <swogatpradhan22 at gmail.com> wrote:
>>>>>>>>>>>>> >>>>> >
>>>>>>>>>>>>> >>>>> > Update: After restarting the nova services on the
>>>>>>>>>>>>> controller and running the deploy script on the edge site, I was able to
>>>>>>>>>>>>> launch the VM from volume.
>>>>>>>>>>>>> >>>>> >
>>>>>>>>>>>>> >>>>> > Right now the instance creation is failing as the
>>>>>>>>>>>>> block device creation is stuck in creating state, it is taking more than 10
>>>>>>>>>>>>> mins for the volume to be created, whereas the image has already been
>>>>>>>>>>>>> imported to the edge glance.
>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>> >>>>> Try following this document and making the same
>>>>>>>>>>>>> observations in your
>>>>>>>>>>>>> >>>>> environment for AZs and their local ceph cluster.
>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>> https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/features/distributed_multibackend_storage.html#confirm-images-may-be-copied-between-sites
>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>> >>>>> On a DCN site if you run a command like this:
>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>> >>>>> $ sudo cephadm shell --config /etc/ceph/dcn0.conf
>>>>>>>>>>>>> --keyring
>>>>>>>>>>>>> >>>>> /etc/ceph/dcn0.client.admin.keyring
>>>>>>>>>>>>> >>>>> $ rbd --cluster dcn0 -p volumes ls -l
>>>>>>>>>>>>> >>>>> NAME SIZE PARENT
>>>>>>>>>>>>> >>>>> FMT PROT LOCK
>>>>>>>>>>>>> >>>>> volume-28c6fc32-047b-4306-ad2d-de2be02716b7 8 GiB
>>>>>>>>>>>>> >>>>> images/8083c7e7-32d8-4f7a-b1da-0ed7884f1076 at snap 2
>>>>>>>>>>>>> excl
>>>>>>>>>>>>> >>>>> $
>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>> >>>>> Then, you should see the parent of the volume is the
>>>>>>>>>>>>> image which is on
>>>>>>>>>>>>> >>>>> the same local ceph cluster.
>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>> >>>>> I wonder if something is misconfigured and thus you're
>>>>>>>>>>>>> encountering
>>>>>>>>>>>>> >>>>> the streaming behavior described here:
>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>> >>>>> Ideally all images should reside in the central Glance
>>>>>>>>>>>>> and be copied
>>>>>>>>>>>>> >>>>> to DCN sites before instances of those images are booted
>>>>>>>>>>>>> on DCN sites.
>>>>>>>>>>>>> >>>>> If an image is not copied to a DCN site before it is
>>>>>>>>>>>>> booted, then the
>>>>>>>>>>>>> >>>>> image will be streamed to the DCN site and then the
>>>>>>>>>>>>> image will boot as
>>>>>>>>>>>>> >>>>> an instance. This happens because Glance at the DCN site
>>>>>>>>>>>>> has access to
>>>>>>>>>>>>> >>>>> the images store at the Central ceph cluster. Though the
>>>>>>>>>>>>> booting of
>>>>>>>>>>>>> >>>>> the image will take time because it has not been copied
>>>>>>>>>>>>> in advance,
>>>>>>>>>>>>> >>>>> this is still preferable to failing to boot the image.
>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>> >>>>> You can also exec into the cinder container at the DCN
>>>>>>>>>>>>> site and
>>>>>>>>>>>>> >>>>> confirm it's using it's local ceph cluster.
>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>> >>>>> John
>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>> >>>>> >
>>>>>>>>>>>>> >>>>> > I will try and create a new fresh image and test again
>>>>>>>>>>>>> then update.
>>>>>>>>>>>>> >>>>> >
>>>>>>>>>>>>> >>>>> > With regards,
>>>>>>>>>>>>> >>>>> > Swogat Pradhan
>>>>>>>>>>>>> >>>>> >
>>>>>>>>>>>>> >>>>> > On Wed, Mar 15, 2023 at 11:13 PM Swogat Pradhan <
>>>>>>>>>>>>> swogatpradhan22 at gmail.com> wrote:
>>>>>>>>>>>>> >>>>> >>
>>>>>>>>>>>>> >>>>> >> Update:
>>>>>>>>>>>>> >>>>> >> In the hypervisor list the compute node state is
>>>>>>>>>>>>> showing down.
>>>>>>>>>>>>> >>>>> >>
>>>>>>>>>>>>> >>>>> >>
>>>>>>>>>>>>> >>>>> >> On Wed, Mar 15, 2023 at 11:11 PM Swogat Pradhan <
>>>>>>>>>>>>> swogatpradhan22 at gmail.com> wrote:
>>>>>>>>>>>>> >>>>> >>>
>>>>>>>>>>>>> >>>>> >>> Hi Brendan,
>>>>>>>>>>>>> >>>>> >>> Now i have deployed another site where i have used 2
>>>>>>>>>>>>> linux bonds network template for both 3 compute nodes and 3 ceph nodes.
>>>>>>>>>>>>> >>>>> >>> The bonding options is set to mode=802.3ad
>>>>>>>>>>>>> (lacp=active).
>>>>>>>>>>>>> >>>>> >>> I used a cirros image to launch instance but the
>>>>>>>>>>>>> instance timed out so i waited for the volume to be created.
>>>>>>>>>>>>> >>>>> >>> Once the volume was created i tried launching the
>>>>>>>>>>>>> instance from the volume and still the instance is stuck in spawning state.
>>>>>>>>>>>>> >>>>> >>>
>>>>>>>>>>>>> >>>>> >>> Here is the nova-compute log:
>>>>>>>>>>>>> >>>>> >>>
>>>>>>>>>>>>> >>>>> >>> 2023-03-15 17:35:47.739 185437 INFO
>>>>>>>>>>>>> oslo.privsep.daemon [-] privsep daemon starting
>>>>>>>>>>>>> >>>>> >>> 2023-03-15 17:35:47.744 185437 INFO
>>>>>>>>>>>>> oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0
>>>>>>>>>>>>> >>>>> >>> 2023-03-15 17:35:47.749 185437 INFO
>>>>>>>>>>>>> oslo.privsep.daemon [-] privsep process running with capabilities
>>>>>>>>>>>>> (eff/prm/inh): CAP_SYS_ADMIN/CAP_SYS_ADMIN/none
>>>>>>>>>>>>> >>>>> >>> 2023-03-15 17:35:47.749 185437 INFO
>>>>>>>>>>>>> oslo.privsep.daemon [-] privsep daemon running as pid 185437
>>>>>>>>>>>>> >>>>> >>> 2023-03-15 17:35:47.974 8 WARNING
>>>>>>>>>>>>> os_brick.initiator.connectors.nvmeof
>>>>>>>>>>>>> [req-dbb11a9b-317e-4957-b141-f9e0bdf6a266 b240e3e89d99489284cd731e75f2a5db
>>>>>>>>>>>>> 4160ce999a31485fa643aed0936dfef0 - default default] Process execution error
>>>>>>>>>>>>> in _get_host_uuid: Unexpected error while running command.
>>>>>>>>>>>>> >>>>> >>> Command: blkid overlay -s UUID -o value
>>>>>>>>>>>>> >>>>> >>> Exit code: 2
>>>>>>>>>>>>> >>>>> >>> Stdout: ''
>>>>>>>>>>>>> >>>>> >>> Stderr: '':
>>>>>>>>>>>>> oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while
>>>>>>>>>>>>> running command.
>>>>>>>>>>>>> >>>>> >>> 2023-03-15 17:35:51.616 8 INFO
>>>>>>>>>>>>> nova.virt.libvirt.driver [req-dbb11a9b-317e-4957-b141-f9e0bdf6a266
>>>>>>>>>>>>> b240e3e89d99489284cd731e75f2a5db 4160ce999a31485fa643aed0936dfef0 - default
>>>>>>>>>>>>> default] [instance: 450b749c-a10a-4308-80a9-3b8020fee758] Creating image
>>>>>>>>>>>>> >>>>> >>>
>>>>>>>>>>>>> >>>>> >>> It is stuck in creating image, do i need to run the
>>>>>>>>>>>>> template mentioned here ?:
>>>>>>>>>>>>> https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/post_deployment/pre_cache_images.html
>>>>>>>>>>>>> >>>>> >>>
>>>>>>>>>>>>> >>>>> >>> The volume is already created and i do not
>>>>>>>>>>>>> understand why the instance is stuck in spawning state.
>>>>>>>>>>>>> >>>>> >>>
>>>>>>>>>>>>> >>>>> >>> With regards,
>>>>>>>>>>>>> >>>>> >>> Swogat Pradhan
>>>>>>>>>>>>> >>>>> >>>
>>>>>>>>>>>>> >>>>> >>>
>>>>>>>>>>>>> >>>>> >>> On Sun, Mar 5, 2023 at 4:02 PM Brendan Shephard <
>>>>>>>>>>>>> bshephar at redhat.com> wrote:
>>>>>>>>>>>>> >>>>> >>>>
>>>>>>>>>>>>> >>>>> >>>> Does your environment use different network
>>>>>>>>>>>>> interfaces for each of the networks? Or does it have a bond with everything
>>>>>>>>>>>>> on it?
>>>>>>>>>>>>> >>>>> >>>>
>>>>>>>>>>>>> >>>>> >>>> One issue I have seen before is that when launching
>>>>>>>>>>>>> instances, there is a lot of network traffic between nodes as the
>>>>>>>>>>>>> hypervisor needs to download the image from Glance. Along with various
>>>>>>>>>>>>> other services sending normal network traffic, it can be enough to cause
>>>>>>>>>>>>> issues if everything is running over a single 1Gbe interface.
>>>>>>>>>>>>> >>>>> >>>>
>>>>>>>>>>>>> >>>>> >>>> I have seen the same situation in fact when using a
>>>>>>>>>>>>> single active/backup bond on 1Gbe nics. It’s worth checking the network
>>>>>>>>>>>>> traffic while you try to spawn the instance to see if you’re dropping
>>>>>>>>>>>>> packets. In the situation I described, there were dropped packets which
>>>>>>>>>>>>> resulted in a loss of communication between nova_compute and RMQ, so the
>>>>>>>>>>>>> node appeared offline. You should also confirm that nova_compute is being
>>>>>>>>>>>>> disconnected in the nova_compute logs if you tail them on the Hypervisor
>>>>>>>>>>>>> while spawning the instance.
>>>>>>>>>>>>> >>>>> >>>>
>>>>>>>>>>>>> >>>>> >>>> In my case, changing from active/backup to LACP
>>>>>>>>>>>>> helped. So, based on that experience, from my perspective, is certainly
>>>>>>>>>>>>> sounds like some kind of network issue.
>>>>>>>>>>>>> >>>>> >>>>
>>>>>>>>>>>>> >>>>> >>>> Regards,
>>>>>>>>>>>>> >>>>> >>>>
>>>>>>>>>>>>> >>>>> >>>> Brendan Shephard
>>>>>>>>>>>>> >>>>> >>>> Senior Software Engineer
>>>>>>>>>>>>> >>>>> >>>> Red Hat Australia
>>>>>>>>>>>>> >>>>> >>>>
>>>>>>>>>>>>> >>>>> >>>>
>>>>>>>>>>>>> >>>>> >>>>
>>>>>>>>>>>>> >>>>> >>>> On 5 Mar 2023, at 6:47 am, Eugen Block <
>>>>>>>>>>>>> eblock at nde.ag> wrote:
>>>>>>>>>>>>> >>>>> >>>>
>>>>>>>>>>>>> >>>>> >>>> Hi,
>>>>>>>>>>>>> >>>>> >>>>
>>>>>>>>>>>>> >>>>> >>>> I tried to help someone with a similar issue some
>>>>>>>>>>>>> time ago in this thread:
>>>>>>>>>>>>> >>>>> >>>>
>>>>>>>>>>>>> https://serverfault.com/questions/1116771/openstack-oslo-messaging-exception-in-nova-conductor
>>>>>>>>>>>>> >>>>> >>>>
>>>>>>>>>>>>> >>>>> >>>> But apparently a neutron reinstallation fixed it
>>>>>>>>>>>>> for that user, not sure if that could apply here. But is it possible that
>>>>>>>>>>>>> your nova and neutron versions are different between central and edge site?
>>>>>>>>>>>>> Have you restarted nova and neutron services on the compute nodes after
>>>>>>>>>>>>> installation? Have you debug logs of nova-conductor and maybe nova-compute?
>>>>>>>>>>>>> Maybe they can help narrow down the issue.
>>>>>>>>>>>>> >>>>> >>>> If there isn't any additional information in the
>>>>>>>>>>>>> debug logs I probably would start "tearing down" rabbitmq. I didn't have to
>>>>>>>>>>>>> do that in a production system yet so be careful. I can think of two routes:
>>>>>>>>>>>>> >>>>> >>>>
>>>>>>>>>>>>> >>>>> >>>> - Either remove queues, exchanges etc. while rabbit
>>>>>>>>>>>>> is running, this will most likely impact client IO depending on your load.
>>>>>>>>>>>>> Check out the rabbitmqctl commands.
>>>>>>>>>>>>> >>>>> >>>> - Or stop the rabbitmq cluster, remove the mnesia
>>>>>>>>>>>>> tables from all nodes and restart rabbitmq so the exchanges, queues etc.
>>>>>>>>>>>>> rebuild.
>>>>>>>>>>>>> >>>>> >>>>
>>>>>>>>>>>>> >>>>> >>>> I can imagine that the failed reply "survives"
>>>>>>>>>>>>> while being replicated across the rabbit nodes. But I don't really know the
>>>>>>>>>>>>> rabbit internals too well, so maybe someone else can chime in here and give
>>>>>>>>>>>>> a better advice.
>>>>>>>>>>>>> >>>>> >>>>
>>>>>>>>>>>>> >>>>> >>>> Regards,
>>>>>>>>>>>>> >>>>> >>>> Eugen
>>>>>>>>>>>>> >>>>> >>>>
>>>>>>>>>>>>> >>>>> >>>> Zitat von Swogat Pradhan <swogatpradhan22 at gmail.com
>>>>>>>>>>>>> >:
>>>>>>>>>>>>> >>>>> >>>>
>>>>>>>>>>>>> >>>>> >>>> Hi,
>>>>>>>>>>>>> >>>>> >>>> Can someone please help me out on this issue?
>>>>>>>>>>>>> >>>>> >>>>
>>>>>>>>>>>>> >>>>> >>>> With regards,
>>>>>>>>>>>>> >>>>> >>>> Swogat Pradhan
>>>>>>>>>>>>> >>>>> >>>>
>>>>>>>>>>>>> >>>>> >>>> On Thu, Mar 2, 2023 at 1:24 PM Swogat Pradhan <
>>>>>>>>>>>>> swogatpradhan22 at gmail.com>
>>>>>>>>>>>>> >>>>> >>>> wrote:
>>>>>>>>>>>>> >>>>> >>>>
>>>>>>>>>>>>> >>>>> >>>> Hi
>>>>>>>>>>>>> >>>>> >>>> I don't see any major packet loss.
>>>>>>>>>>>>> >>>>> >>>> It seems the problem is somewhere in rabbitmq maybe
>>>>>>>>>>>>> but not due to packet
>>>>>>>>>>>>> >>>>> >>>> loss.
>>>>>>>>>>>>> >>>>> >>>>
>>>>>>>>>>>>> >>>>> >>>> with regards,
>>>>>>>>>>>>> >>>>> >>>> Swogat Pradhan
>>>>>>>>>>>>> >>>>> >>>>
>>>>>>>>>>>>> >>>>> >>>> On Wed, Mar 1, 2023 at 3:34 PM Swogat Pradhan <
>>>>>>>>>>>>> swogatpradhan22 at gmail.com>
>>>>>>>>>>>>> >>>>> >>>> wrote:
>>>>>>>>>>>>> >>>>> >>>>
>>>>>>>>>>>>> >>>>> >>>> Hi,
>>>>>>>>>>>>> >>>>> >>>> Yes the MTU is the same as the default '1500'.
>>>>>>>>>>>>> >>>>> >>>> Generally I haven't seen any packet loss, but never
>>>>>>>>>>>>> checked when
>>>>>>>>>>>>> >>>>> >>>> launching the instance.
>>>>>>>>>>>>> >>>>> >>>> I will check that and come back.
>>>>>>>>>>>>> >>>>> >>>> But everytime i launch an instance the instance
>>>>>>>>>>>>> gets stuck at spawning
>>>>>>>>>>>>> >>>>> >>>> state and there the hypervisor becomes down, so not
>>>>>>>>>>>>> sure if packet loss
>>>>>>>>>>>>> >>>>> >>>> causes this.
>>>>>>>>>>>>> >>>>> >>>>
>>>>>>>>>>>>> >>>>> >>>> With regards,
>>>>>>>>>>>>> >>>>> >>>> Swogat pradhan
>>>>>>>>>>>>> >>>>> >>>>
>>>>>>>>>>>>> >>>>> >>>> On Wed, Mar 1, 2023 at 3:30 PM Eugen Block <
>>>>>>>>>>>>> eblock at nde.ag> wrote:
>>>>>>>>>>>>> >>>>> >>>>
>>>>>>>>>>>>> >>>>> >>>> One more thing coming to mind is MTU size. Are they
>>>>>>>>>>>>> identical between
>>>>>>>>>>>>> >>>>> >>>> central and edge site? Do you see packet loss
>>>>>>>>>>>>> through the tunnel?
>>>>>>>>>>>>> >>>>> >>>>
>>>>>>>>>>>>> >>>>> >>>> Zitat von Swogat Pradhan <swogatpradhan22 at gmail.com
>>>>>>>>>>>>> >:
>>>>>>>>>>>>> >>>>> >>>>
>>>>>>>>>>>>> >>>>> >>>> > Hi Eugen,
>>>>>>>>>>>>> >>>>> >>>> > Request you to please add my email either on 'to'
>>>>>>>>>>>>> or 'cc' as i am not
>>>>>>>>>>>>> >>>>> >>>> > getting email's from you.
>>>>>>>>>>>>> >>>>> >>>> > Coming to the issue:
>>>>>>>>>>>>> >>>>> >>>> >
>>>>>>>>>>>>> >>>>> >>>> > [root at overcloud-controller-no-ceph-3 /]#
>>>>>>>>>>>>> rabbitmqctl list_policies -p
>>>>>>>>>>>>> >>>>> >>>> /
>>>>>>>>>>>>> >>>>> >>>> > Listing policies for vhost "/" ...
>>>>>>>>>>>>> >>>>> >>>> > vhost name pattern apply-to
>>>>>>>>>>>>> definition priority
>>>>>>>>>>>>> >>>>> >>>> > / ha-all ^(?!amq\.).* queues
>>>>>>>>>>>>> >>>>> >>>> >
>>>>>>>>>>>>> >>>>> >>>>
>>>>>>>>>>>>> {"ha-mode":"exactly","ha-params":2,"ha-promote-on-shutdown":"always"} 0
>>>>>>>>>>>>> >>>>> >>>> >
>>>>>>>>>>>>> >>>>> >>>> > I have the edge site compute nodes up, it only
>>>>>>>>>>>>> goes down when i am
>>>>>>>>>>>>> >>>>> >>>> trying
>>>>>>>>>>>>> >>>>> >>>> > to launch an instance and the instance comes to a
>>>>>>>>>>>>> spawning state and
>>>>>>>>>>>>> >>>>> >>>> then
>>>>>>>>>>>>> >>>>> >>>> > gets stuck.
>>>>>>>>>>>>> >>>>> >>>> >
>>>>>>>>>>>>> >>>>> >>>> > I have a tunnel setup between the central and the
>>>>>>>>>>>>> edge sites.
>>>>>>>>>>>>> >>>>> >>>> >
>>>>>>>>>>>>> >>>>> >>>> > With regards,
>>>>>>>>>>>>> >>>>> >>>> > Swogat Pradhan
>>>>>>>>>>>>> >>>>> >>>> >
>>>>>>>>>>>>> >>>>> >>>> > On Tue, Feb 28, 2023 at 9:11 PM Swogat Pradhan <
>>>>>>>>>>>>> >>>>> >>>> swogatpradhan22 at gmail.com>
>>>>>>>>>>>>> >>>>> >>>> > wrote:
>>>>>>>>>>>>> >>>>> >>>> >
>>>>>>>>>>>>> >>>>> >>>> >> Hi Eugen,
>>>>>>>>>>>>> >>>>> >>>> >> For some reason i am not getting your email to
>>>>>>>>>>>>> me directly, i am
>>>>>>>>>>>>> >>>>> >>>> checking
>>>>>>>>>>>>> >>>>> >>>> >> the email digest and there i am able to find
>>>>>>>>>>>>> your reply.
>>>>>>>>>>>>> >>>>> >>>> >> Here is the log for download:
>>>>>>>>>>>>> https://we.tl/t-L8FEkGZFSq
>>>>>>>>>>>>> >>>>> >>>> >> Yes, these logs are from the time when the issue
>>>>>>>>>>>>> occurred.
>>>>>>>>>>>>> >>>>> >>>> >>
>>>>>>>>>>>>> >>>>> >>>> >> *Note: i am able to create vm's and perform
>>>>>>>>>>>>> other activities in the
>>>>>>>>>>>>> >>>>> >>>> >> central site, only facing this issue in the edge
>>>>>>>>>>>>> site.*
>>>>>>>>>>>>> >>>>> >>>> >>
>>>>>>>>>>>>> >>>>> >>>> >> With regards,
>>>>>>>>>>>>> >>>>> >>>> >> Swogat Pradhan
>>>>>>>>>>>>> >>>>> >>>> >>
>>>>>>>>>>>>> >>>>> >>>> >> On Mon, Feb 27, 2023 at 5:12 PM Swogat Pradhan <
>>>>>>>>>>>>> >>>>> >>>> swogatpradhan22 at gmail.com>
>>>>>>>>>>>>> >>>>> >>>> >> wrote:
>>>>>>>>>>>>> >>>>> >>>> >>
>>>>>>>>>>>>> >>>>> >>>> >>> Hi Eugen,
>>>>>>>>>>>>> >>>>> >>>> >>> Thanks for your response.
>>>>>>>>>>>>> >>>>> >>>> >>> I have actually a 4 controller setup so here
>>>>>>>>>>>>> are the details:
>>>>>>>>>>>>> >>>>> >>>> >>>
>>>>>>>>>>>>> >>>>> >>>> >>> *PCS Status:*
>>>>>>>>>>>>> >>>>> >>>> >>> * Container bundle set: rabbitmq-bundle [
>>>>>>>>>>>>> >>>>> >>>> >>>
>>>>>>>>>>>>> 172.25.201.68:8787/tripleomaster/openstack-rabbitmq:pcmklatest
>>>>>>>>>>>>> ]:
>>>>>>>>>>>>> >>>>> >>>> >>> * rabbitmq-bundle-0
>>>>>>>>>>>>> (ocf::heartbeat:rabbitmq-cluster):
>>>>>>>>>>>>> >>>>> >>>> Started
>>>>>>>>>>>>> >>>>> >>>> >>> overcloud-controller-no-ceph-3
>>>>>>>>>>>>> >>>>> >>>> >>> * rabbitmq-bundle-1
>>>>>>>>>>>>> (ocf::heartbeat:rabbitmq-cluster):
>>>>>>>>>>>>> >>>>> >>>> Started
>>>>>>>>>>>>> >>>>> >>>> >>> overcloud-controller-2
>>>>>>>>>>>>> >>>>> >>>> >>> * rabbitmq-bundle-2
>>>>>>>>>>>>> (ocf::heartbeat:rabbitmq-cluster):
>>>>>>>>>>>>> >>>>> >>>> Started
>>>>>>>>>>>>> >>>>> >>>> >>> overcloud-controller-1
>>>>>>>>>>>>> >>>>> >>>> >>> * rabbitmq-bundle-3
>>>>>>>>>>>>> (ocf::heartbeat:rabbitmq-cluster):
>>>>>>>>>>>>> >>>>> >>>> Started
>>>>>>>>>>>>> >>>>> >>>> >>> overcloud-controller-0
>>>>>>>>>>>>> >>>>> >>>> >>>
>>>>>>>>>>>>> >>>>> >>>> >>> I have tried restarting the bundle multiple
>>>>>>>>>>>>> times but the issue is
>>>>>>>>>>>>> >>>>> >>>> still
>>>>>>>>>>>>> >>>>> >>>> >>> present.
>>>>>>>>>>>>> >>>>> >>>> >>>
>>>>>>>>>>>>> >>>>> >>>> >>> *Cluster status:*
>>>>>>>>>>>>> >>>>> >>>> >>> [root at overcloud-controller-0 /]# rabbitmqctl
>>>>>>>>>>>>> cluster_status
>>>>>>>>>>>>> >>>>> >>>> >>> Cluster status of node
>>>>>>>>>>>>> >>>>> >>>> >>>
>>>>>>>>>>>>> rabbit at overcloud-controller-0.internalapi.bdxworld.com ...
>>>>>>>>>>>>> >>>>> >>>> >>> Basics
>>>>>>>>>>>>> >>>>> >>>> >>>
>>>>>>>>>>>>> >>>>> >>>> >>> Cluster name:
>>>>>>>>>>>>> rabbit at overcloud-controller-no-ceph-3.bdxworld.com
>>>>>>>>>>>>> >>>>> >>>> >>>
>>>>>>>>>>>>> >>>>> >>>> >>> Disk Nodes
>>>>>>>>>>>>> >>>>> >>>> >>>
>>>>>>>>>>>>> >>>>> >>>> >>>
>>>>>>>>>>>>> rabbit at overcloud-controller-0.internalapi.bdxworld.com
>>>>>>>>>>>>> >>>>> >>>> >>>
>>>>>>>>>>>>> rabbit at overcloud-controller-1.internalapi.bdxworld.com
>>>>>>>>>>>>> >>>>> >>>> >>>
>>>>>>>>>>>>> rabbit at overcloud-controller-2.internalapi.bdxworld.com
>>>>>>>>>>>>> >>>>> >>>> >>>
>>>>>>>>>>>>> rabbit at overcloud-controller-no-ceph-3.internalapi.bdxworld.com
>>>>>>>>>>>>> >>>>> >>>> >>>
>>>>>>>>>>>>> >>>>> >>>> >>> Running Nodes
>>>>>>>>>>>>> >>>>> >>>> >>>
>>>>>>>>>>>>> >>>>> >>>> >>>
>>>>>>>>>>>>> rabbit at overcloud-controller-0.internalapi.bdxworld.com
>>>>>>>>>>>>> >>>>> >>>> >>>
>>>>>>>>>>>>> rabbit at overcloud-controller-1.internalapi.bdxworld.com
>>>>>>>>>>>>> >>>>> >>>> >>>
>>>>>>>>>>>>> rabbit at overcloud-controller-2.internalapi.bdxworld.com
>>>>>>>>>>>>> >>>>> >>>> >>>
>>>>>>>>>>>>> rabbit at overcloud-controller-no-ceph-3.internalapi.bdxworld.com
>>>>>>>>>>>>> >>>>> >>>> >>>
>>>>>>>>>>>>> >>>>> >>>> >>> Versions
>>>>>>>>>>>>> >>>>> >>>> >>>
>>>>>>>>>>>>> >>>>> >>>> >>>
>>>>>>>>>>>>> rabbit at overcloud-controller-0.internalapi.bdxworld.com:
>>>>>>>>>>>>> RabbitMQ
>>>>>>>>>>>>> >>>>> >>>> 3.8.3
>>>>>>>>>>>>> >>>>> >>>> >>> on Erlang 22.3.4.1
>>>>>>>>>>>>> >>>>> >>>> >>>
>>>>>>>>>>>>> rabbit at overcloud-controller-1.internalapi.bdxworld.com:
>>>>>>>>>>>>> RabbitMQ
>>>>>>>>>>>>> >>>>> >>>> 3.8.3
>>>>>>>>>>>>> >>>>> >>>> >>> on Erlang 22.3.4.1
>>>>>>>>>>>>> >>>>> >>>> >>>
>>>>>>>>>>>>> rabbit at overcloud-controller-2.internalapi.bdxworld.com:
>>>>>>>>>>>>> RabbitMQ
>>>>>>>>>>>>> >>>>> >>>> 3.8.3
>>>>>>>>>>>>> >>>>> >>>> >>> on Erlang 22.3.4.1
>>>>>>>>>>>>> >>>>> >>>> >>>
>>>>>>>>>>>>> rabbit at overcloud-controller-no-ceph-3.internalapi.bdxworld.com
>>>>>>>>>>>>> :
>>>>>>>>>>>>> >>>>> >>>> RabbitMQ
>>>>>>>>>>>>> >>>>> >>>> >>> 3.8.3 on Erlang 22.3.4.1
>>>>>>>>>>>>> >>>>> >>>> >>>
>>>>>>>>>>>>> >>>>> >>>> >>> Alarms
>>>>>>>>>>>>> >>>>> >>>> >>>
>>>>>>>>>>>>> >>>>> >>>> >>> (none)
>>>>>>>>>>>>> >>>>> >>>> >>>
>>>>>>>>>>>>> >>>>> >>>> >>> Network Partitions
>>>>>>>>>>>>> >>>>> >>>> >>>
>>>>>>>>>>>>> >>>>> >>>> >>> (none)
>>>>>>>>>>>>> >>>>> >>>> >>>
>>>>>>>>>>>>> >>>>> >>>> >>> Listeners
>>>>>>>>>>>>> >>>>> >>>> >>>
>>>>>>>>>>>>> >>>>> >>>> >>> Node:
>>>>>>>>>>>>> rabbit at overcloud-controller-0.internalapi.bdxworld.com,
>>>>>>>>>>>>> >>>>> >>>> interface:
>>>>>>>>>>>>> >>>>> >>>> >>> [::], port: 25672, protocol: clustering,
>>>>>>>>>>>>> purpose: inter-node and CLI
>>>>>>>>>>>>> >>>>> >>>> tool
>>>>>>>>>>>>> >>>>> >>>> >>> communication
>>>>>>>>>>>>> >>>>> >>>> >>> Node:
>>>>>>>>>>>>> rabbit at overcloud-controller-0.internalapi.bdxworld.com,
>>>>>>>>>>>>> >>>>> >>>> interface:
>>>>>>>>>>>>> >>>>> >>>> >>> 172.25.201.212, port: 5672, protocol: amqp,
>>>>>>>>>>>>> purpose: AMQP 0-9-1
>>>>>>>>>>>>> >>>>> >>>> >>> and AMQP 1.0
>>>>>>>>>>>>> >>>>> >>>> >>> Node:
>>>>>>>>>>>>> rabbit at overcloud-controller-0.internalapi.bdxworld.com,
>>>>>>>>>>>>> >>>>> >>>> interface:
>>>>>>>>>>>>> >>>>> >>>> >>> [::], port: 15672, protocol: http, purpose:
>>>>>>>>>>>>> HTTP API
>>>>>>>>>>>>> >>>>> >>>> >>> Node:
>>>>>>>>>>>>> rabbit at overcloud-controller-1.internalapi.bdxworld.com,
>>>>>>>>>>>>> >>>>> >>>> interface:
>>>>>>>>>>>>> >>>>> >>>> >>> [::], port: 25672, protocol: clustering,
>>>>>>>>>>>>> purpose: inter-node and CLI
>>>>>>>>>>>>> >>>>> >>>> tool
>>>>>>>>>>>>> >>>>> >>>> >>> communication
>>>>>>>>>>>>> >>>>> >>>> >>> Node:
>>>>>>>>>>>>> rabbit at overcloud-controller-1.internalapi.bdxworld.com,
>>>>>>>>>>>>> >>>>> >>>> interface:
>>>>>>>>>>>>> >>>>> >>>> >>> 172.25.201.205, port: 5672, protocol: amqp,
>>>>>>>>>>>>> purpose: AMQP 0-9-1
>>>>>>>>>>>>> >>>>> >>>> >>> and AMQP 1.0
>>>>>>>>>>>>> >>>>> >>>> >>> Node:
>>>>>>>>>>>>> rabbit at overcloud-controller-1.internalapi.bdxworld.com,
>>>>>>>>>>>>> >>>>> >>>> interface:
>>>>>>>>>>>>> >>>>> >>>> >>> [::], port: 15672, protocol: http, purpose:
>>>>>>>>>>>>> HTTP API
>>>>>>>>>>>>> >>>>> >>>> >>> Node:
>>>>>>>>>>>>> rabbit at overcloud-controller-2.internalapi.bdxworld.com,
>>>>>>>>>>>>> >>>>> >>>> interface:
>>>>>>>>>>>>> >>>>> >>>> >>> [::], port: 25672, protocol: clustering,
>>>>>>>>>>>>> purpose: inter-node and CLI
>>>>>>>>>>>>> >>>>> >>>> tool
>>>>>>>>>>>>> >>>>> >>>> >>> communication
>>>>>>>>>>>>> >>>>> >>>> >>> Node:
>>>>>>>>>>>>> rabbit at overcloud-controller-2.internalapi.bdxworld.com,
>>>>>>>>>>>>> >>>>> >>>> interface:
>>>>>>>>>>>>> >>>>> >>>> >>> 172.25.201.201, port: 5672, protocol: amqp,
>>>>>>>>>>>>> purpose: AMQP 0-9-1
>>>>>>>>>>>>> >>>>> >>>> >>> and AMQP 1.0
>>>>>>>>>>>>> >>>>> >>>> >>> Node:
>>>>>>>>>>>>> rabbit at overcloud-controller-2.internalapi.bdxworld.com,
>>>>>>>>>>>>> >>>>> >>>> interface:
>>>>>>>>>>>>> >>>>> >>>> >>> [::], port: 15672, protocol: http, purpose:
>>>>>>>>>>>>> HTTP API
>>>>>>>>>>>>> >>>>> >>>> >>> Node:
>>>>>>>>>>>>> rabbit at overcloud-controller-no-ceph-3.internalapi.bdxworld.com
>>>>>>>>>>>>> >>>>> >>>> ,
>>>>>>>>>>>>> >>>>> >>>> >>> interface: [::], port: 25672, protocol:
>>>>>>>>>>>>> clustering, purpose:
>>>>>>>>>>>>> >>>>> >>>> inter-node and
>>>>>>>>>>>>> >>>>> >>>> >>> CLI tool communication
>>>>>>>>>>>>> >>>>> >>>> >>> Node:
>>>>>>>>>>>>> rabbit at overcloud-controller-no-ceph-3.internalapi.bdxworld.com
>>>>>>>>>>>>> >>>>> >>>> ,
>>>>>>>>>>>>> >>>>> >>>> >>> interface: 172.25.201.209, port: 5672,
>>>>>>>>>>>>> protocol: amqp, purpose: AMQP
>>>>>>>>>>>>> >>>>> >>>> 0-9-1
>>>>>>>>>>>>> >>>>> >>>> >>> and AMQP 1.0
>>>>>>>>>>>>> >>>>> >>>> >>> Node:
>>>>>>>>>>>>> rabbit at overcloud-controller-no-ceph-3.internalapi.bdxworld.com
>>>>>>>>>>>>> >>>>> >>>> ,
>>>>>>>>>>>>> >>>>> >>>> >>> interface: [::], port: 15672, protocol: http,
>>>>>>>>>>>>> purpose: HTTP API
>>>>>>>>>>>>> >>>>> >>>> >>>
>>>>>>>>>>>>> >>>>> >>>> >>> Feature flags
>>>>>>>>>>>>> >>>>> >>>> >>>
>>>>>>>>>>>>> >>>>> >>>> >>> Flag: drop_unroutable_metric, state: enabled
>>>>>>>>>>>>> >>>>> >>>> >>> Flag: empty_basic_get_metric, state: enabled
>>>>>>>>>>>>> >>>>> >>>> >>> Flag: implicit_default_bindings, state: enabled
>>>>>>>>>>>>> >>>>> >>>> >>> Flag: quorum_queue, state: enabled
>>>>>>>>>>>>> >>>>> >>>> >>> Flag: virtual_host_metadata, state: enabled
>>>>>>>>>>>>> >>>>> >>>> >>>
>>>>>>>>>>>>> >>>>> >>>> >>> *Logs:*
>>>>>>>>>>>>> >>>>> >>>> >>> *(Attached)*
>>>>>>>>>>>>> >>>>> >>>> >>>
>>>>>>>>>>>>> >>>>> >>>> >>> With regards,
>>>>>>>>>>>>> >>>>> >>>> >>> Swogat Pradhan
>>>>>>>>>>>>> >>>>> >>>> >>>
>>>>>>>>>>>>> >>>>> >>>> >>> On Sun, Feb 26, 2023 at 2:34 PM Swogat Pradhan <
>>>>>>>>>>>>> >>>>> >>>> swogatpradhan22 at gmail.com>
>>>>>>>>>>>>> >>>>> >>>> >>> wrote:
>>>>>>>>>>>>> >>>>> >>>> >>>
>>>>>>>>>>>>> >>>>> >>>> >>>> Hi,
>>>>>>>>>>>>> >>>>> >>>> >>>> Please find the nova conductor as well as nova
>>>>>>>>>>>>> api log.
>>>>>>>>>>>>> >>>>> >>>> >>>>
>>>>>>>>>>>>> >>>>> >>>> >>>> nova-conuctor:
>>>>>>>>>>>>> >>>>> >>>> >>>>
>>>>>>>>>>>>> >>>>> >>>> >>>> 2023-02-26 08:45:01.108 31 WARNING
>>>>>>>>>>>>> >>>>> >>>> oslo_messaging._drivers.amqpdriver
>>>>>>>>>>>>> >>>>> >>>> >>>> [req-caefe26d-153a-4dfd-9ea6-bc5ca0d46679 - -
>>>>>>>>>>>>> - - -]
>>>>>>>>>>>>> >>>>> >>>> >>>> reply_349bcb075f8c49329435a0f884b33066 doesn't
>>>>>>>>>>>>> exist, drop reply to
>>>>>>>>>>>>> >>>>> >>>> >>>> 16152921c1eb45c2b1f562087140168b
>>>>>>>>>>>>> >>>>> >>>> >>>> 2023-02-26 08:45:02.144 26 WARNING
>>>>>>>>>>>>> >>>>> >>>> oslo_messaging._drivers.amqpdriver
>>>>>>>>>>>>> >>>>> >>>> >>>> [req-7b43c4e5-0475-4598-92c0-fcacb51d9813 - -
>>>>>>>>>>>>> - - -]
>>>>>>>>>>>>> >>>>> >>>> >>>> reply_276049ec36a84486a8a406911d9802f4 doesn't
>>>>>>>>>>>>> exist, drop reply to
>>>>>>>>>>>>> >>>>> >>>> >>>> 83dbe5f567a940b698acfe986f6194fa
>>>>>>>>>>>>> >>>>> >>>> >>>> 2023-02-26 08:45:02.314 32 WARNING
>>>>>>>>>>>>> >>>>> >>>> oslo_messaging._drivers.amqpdriver
>>>>>>>>>>>>> >>>>> >>>> >>>> [req-7b43c4e5-0475-4598-92c0-fcacb51d9813 - -
>>>>>>>>>>>>> - - -]
>>>>>>>>>>>>> >>>>> >>>> >>>> reply_276049ec36a84486a8a406911d9802f4 doesn't
>>>>>>>>>>>>> exist, drop reply to
>>>>>>>>>>>>> >>>>> >>>> >>>> f3bfd7f65bd542b18d84cea3033abb43:
>>>>>>>>>>>>> >>>>> >>>> >>>> oslo_messaging.exceptions.MessageUndeliverable
>>>>>>>>>>>>> >>>>> >>>> >>>> 2023-02-26 08:45:02.316 32 ERROR
>>>>>>>>>>>>> oslo_messaging._drivers.amqpdriver
>>>>>>>>>>>>> >>>>> >>>> >>>> [req-7b43c4e5-0475-4598-92c0-fcacb51d9813 - -
>>>>>>>>>>>>> - - -] The reply
>>>>>>>>>>>>> >>>>> >>>> >>>> f3bfd7f65bd542b18d84cea3033abb43 failed to
>>>>>>>>>>>>> send after 60 seconds
>>>>>>>>>>>>> >>>>> >>>> due to a
>>>>>>>>>>>>> >>>>> >>>> >>>> missing queue
>>>>>>>>>>>>> (reply_276049ec36a84486a8a406911d9802f4).
>>>>>>>>>>>>> >>>>> >>>> Abandoning...:
>>>>>>>>>>>>> >>>>> >>>> >>>> oslo_messaging.exceptions.MessageUndeliverable
>>>>>>>>>>>>> >>>>> >>>> >>>> 2023-02-26 08:48:01.282 35 WARNING
>>>>>>>>>>>>> >>>>> >>>> oslo_messaging._drivers.amqpdriver
>>>>>>>>>>>>> >>>>> >>>> >>>> [req-caefe26d-153a-4dfd-9ea6-bc5ca0d46679 - -
>>>>>>>>>>>>> - - -]
>>>>>>>>>>>>> >>>>> >>>> >>>> reply_349bcb075f8c49329435a0f884b33066 doesn't
>>>>>>>>>>>>> exist, drop reply to
>>>>>>>>>>>>> >>>>> >>>> >>>> d4b9180f91a94f9a82c3c9c4b7595566:
>>>>>>>>>>>>> >>>>> >>>> >>>> oslo_messaging.exceptions.MessageUndeliverable
>>>>>>>>>>>>> >>>>> >>>> >>>> 2023-02-26 08:48:01.284 35 ERROR
>>>>>>>>>>>>> oslo_messaging._drivers.amqpdriver
>>>>>>>>>>>>> >>>>> >>>> >>>> [req-caefe26d-153a-4dfd-9ea6-bc5ca0d46679 - -
>>>>>>>>>>>>> - - -] The reply
>>>>>>>>>>>>> >>>>> >>>> >>>> d4b9180f91a94f9a82c3c9c4b7595566 failed to
>>>>>>>>>>>>> send after 60 seconds
>>>>>>>>>>>>> >>>>> >>>> due to a
>>>>>>>>>>>>> >>>>> >>>> >>>> missing queue
>>>>>>>>>>>>> (reply_349bcb075f8c49329435a0f884b33066).
>>>>>>>>>>>>> >>>>> >>>> Abandoning...:
>>>>>>>>>>>>> >>>>> >>>> >>>> oslo_messaging.exceptions.MessageUndeliverable
>>>>>>>>>>>>> >>>>> >>>> >>>> 2023-02-26 08:49:01.303 33 WARNING
>>>>>>>>>>>>> >>>>> >>>> oslo_messaging._drivers.amqpdriver
>>>>>>>>>>>>> >>>>> >>>> >>>> [req-caefe26d-153a-4dfd-9ea6-bc5ca0d46679 - -
>>>>>>>>>>>>> - - -]
>>>>>>>>>>>>> >>>>> >>>> >>>> reply_349bcb075f8c49329435a0f884b33066 doesn't
>>>>>>>>>>>>> exist, drop reply to
>>>>>>>>>>>>> >>>>> >>>> >>>> 897911a234a445d8a0d8af02ece40f6f:
>>>>>>>>>>>>> >>>>> >>>> >>>> oslo_messaging.exceptions.MessageUndeliverable
>>>>>>>>>>>>> >>>>> >>>> >>>> 2023-02-26 08:49:01.304 33 ERROR
>>>>>>>>>>>>> oslo_messaging._drivers.amqpdriver
>>>>>>>>>>>>> >>>>> >>>> >>>> [req-caefe26d-153a-4dfd-9ea6-bc5ca0d46679 - -
>>>>>>>>>>>>> - - -] The reply
>>>>>>>>>>>>> >>>>> >>>> >>>> 897911a234a445d8a0d8af02ece40f6f failed to
>>>>>>>>>>>>> send after 60 seconds
>>>>>>>>>>>>> >>>>> >>>> due to a
>>>>>>>>>>>>> >>>>> >>>> >>>> missing queue
>>>>>>>>>>>>> (reply_349bcb075f8c49329435a0f884b33066).
>>>>>>>>>>>>> >>>>> >>>> Abandoning...:
>>>>>>>>>>>>> >>>>> >>>> >>>> oslo_messaging.exceptions.MessageUndeliverable
>>>>>>>>>>>>> >>>>> >>>> >>>> 2023-02-26 08:49:52.254 31 WARNING
>>>>>>>>>>>>> nova.cache_utils
>>>>>>>>>>>>> >>>>> >>>> >>>> [req-3a1547ea-326f-4dd0-9127-7f4a4bdf1e45
>>>>>>>>>>>>> >>>>> >>>> b240e3e89d99489284cd731e75f2a5db
>>>>>>>>>>>>> >>>>> >>>> >>>> 4160ce999a31485fa643aed0936dfef0 - default
>>>>>>>>>>>>> default] Cache enabled
>>>>>>>>>>>>> >>>>> >>>> with
>>>>>>>>>>>>> >>>>> >>>> >>>> backend dogpile.cache.null.
>>>>>>>>>>>>> >>>>> >>>> >>>> 2023-02-26 08:50:01.264 27 WARNING
>>>>>>>>>>>>> >>>>> >>>> oslo_messaging._drivers.amqpdriver
>>>>>>>>>>>>> >>>>> >>>> >>>> [req-caefe26d-153a-4dfd-9ea6-bc5ca0d46679 - -
>>>>>>>>>>>>> - - -]
>>>>>>>>>>>>> >>>>> >>>> >>>> reply_349bcb075f8c49329435a0f884b33066 doesn't
>>>>>>>>>>>>> exist, drop reply to
>>>>>>>>>>>>> >>>>> >>>> >>>> 8f723ceb10c3472db9a9f324861df2bb:
>>>>>>>>>>>>> >>>>> >>>> >>>> oslo_messaging.exceptions.MessageUndeliverable
>>>>>>>>>>>>> >>>>> >>>> >>>> 2023-02-26 08:50:01.266 27 ERROR
>>>>>>>>>>>>> oslo_messaging._drivers.amqpdriver
>>>>>>>>>>>>> >>>>> >>>> >>>> [req-caefe26d-153a-4dfd-9ea6-bc5ca0d46679 - -
>>>>>>>>>>>>> - - -] The reply
>>>>>>>>>>>>> >>>>> >>>> >>>> 8f723ceb10c3472db9a9f324861df2bb failed to
>>>>>>>>>>>>> send after 60 seconds
>>>>>>>>>>>>> >>>>> >>>> due to a
>>>>>>>>>>>>> >>>>> >>>> >>>> missing queue
>>>>>>>>>>>>> (reply_349bcb075f8c49329435a0f884b33066).
>>>>>>>>>>>>> >>>>> >>>> Abandoning...:
>>>>>>>>>>>>> >>>>> >>>> >>>> oslo_messaging.exceptions.MessageUndeliverable
>>>>>>>>>>>>> >>>>> >>>> >>>>
>>>>>>>>>>>>> >>>>> >>>> >>>> With regards,
>>>>>>>>>>>>> >>>>> >>>> >>>> Swogat Pradhan
>>>>>>>>>>>>> >>>>> >>>> >>>>
>>>>>>>>>>>>> >>>>> >>>> >>>> On Sun, Feb 26, 2023 at 2:26 PM Swogat Pradhan
>>>>>>>>>>>>> <
>>>>>>>>>>>>> >>>>> >>>> >>>> swogatpradhan22 at gmail.com> wrote:
>>>>>>>>>>>>> >>>>> >>>> >>>>
>>>>>>>>>>>>> >>>>> >>>> >>>>> Hi,
>>>>>>>>>>>>> >>>>> >>>> >>>>> I currently have 3 compute nodes on edge
>>>>>>>>>>>>> site1 where i am trying to
>>>>>>>>>>>>> >>>>> >>>> >>>>> launch vm's.
>>>>>>>>>>>>> >>>>> >>>> >>>>> When the VM is in spawning state the node
>>>>>>>>>>>>> goes down (openstack
>>>>>>>>>>>>> >>>>> >>>> compute
>>>>>>>>>>>>> >>>>> >>>> >>>>> service list), the node comes backup when i
>>>>>>>>>>>>> restart the nova
>>>>>>>>>>>>> >>>>> >>>> compute
>>>>>>>>>>>>> >>>>> >>>> >>>>> service but then the launch of the vm fails.
>>>>>>>>>>>>> >>>>> >>>> >>>>>
>>>>>>>>>>>>> >>>>> >>>> >>>>> nova-compute.log
>>>>>>>>>>>>> >>>>> >>>> >>>>>
>>>>>>>>>>>>> >>>>> >>>> >>>>> 2023-02-26 08:15:51.808 7 INFO
>>>>>>>>>>>>> nova.compute.manager
>>>>>>>>>>>>> >>>>> >>>> >>>>> [req-bc0f5f2e-53fc-4dae-b1da-82f1f972d617 - -
>>>>>>>>>>>>> - - -] Running
>>>>>>>>>>>>> >>>>> >>>> >>>>> instance usage
>>>>>>>>>>>>> >>>>> >>>> >>>>> audit for host dcn01-hci-0.bdxworld.com from
>>>>>>>>>>>>> 2023-02-26 07:00:00
>>>>>>>>>>>>> >>>>> >>>> to
>>>>>>>>>>>>> >>>>> >>>> >>>>> 2023-02-26 08:00:00. 0 instances.
>>>>>>>>>>>>> >>>>> >>>> >>>>> 2023-02-26 08:49:52.813 7 INFO
>>>>>>>>>>>>> nova.compute.claims
>>>>>>>>>>>>> >>>>> >>>> >>>>> [req-3a1547ea-326f-4dd0-9127-7f4a4bdf1e45
>>>>>>>>>>>>> >>>>> >>>> >>>>> b240e3e89d99489284cd731e75f2a5db
>>>>>>>>>>>>> >>>>> >>>> >>>>> 4160ce999a31485fa643aed0936dfef0 - default
>>>>>>>>>>>>> default] [instance:
>>>>>>>>>>>>> >>>>> >>>> >>>>> 0c62c1ef-9010-417d-a05f-4db77e901600] Claim
>>>>>>>>>>>>> successful on node
>>>>>>>>>>>>> >>>>> >>>> >>>>> dcn01-hci-0.bdxworld.com
>>>>>>>>>>>>> >>>>> >>>> >>>>> 2023-02-26 08:49:54.225 7 INFO
>>>>>>>>>>>>> nova.virt.libvirt.driver
>>>>>>>>>>>>> >>>>> >>>> >>>>> [req-3a1547ea-326f-4dd0-9127-7f4a4bdf1e45
>>>>>>>>>>>>> >>>>> >>>> >>>>> b240e3e89d99489284cd731e75f2a5db
>>>>>>>>>>>>> >>>>> >>>> >>>>> 4160ce999a31485fa643aed0936dfef0 - default
>>>>>>>>>>>>> default] [instance:
>>>>>>>>>>>>> >>>>> >>>> >>>>> 0c62c1ef-9010-417d-a05f-4db77e901600]
>>>>>>>>>>>>> Ignoring supplied device
>>>>>>>>>>>>> >>>>> >>>> name:
>>>>>>>>>>>>> >>>>> >>>> >>>>> /dev/vda. Libvirt can't honour user-supplied
>>>>>>>>>>>>> dev names
>>>>>>>>>>>>> >>>>> >>>> >>>>> 2023-02-26 08:49:54.398 7 INFO
>>>>>>>>>>>>> nova.virt.block_device
>>>>>>>>>>>>> >>>>> >>>> >>>>> [req-3a1547ea-326f-4dd0-9127-7f4a4bdf1e45
>>>>>>>>>>>>> >>>>> >>>> >>>>> b240e3e89d99489284cd731e75f2a5db
>>>>>>>>>>>>> >>>>> >>>> >>>>> 4160ce999a31485fa643aed0936dfef0 - default
>>>>>>>>>>>>> default] [instance:
>>>>>>>>>>>>> >>>>> >>>> >>>>> 0c62c1ef-9010-417d-a05f-4db77e901600] Booting
>>>>>>>>>>>>> with volume
>>>>>>>>>>>>> >>>>> >>>> >>>>> c4bd7885-5973-4860-bbe6-7a2f726baeee at
>>>>>>>>>>>>> /dev/vda
>>>>>>>>>>>>> >>>>> >>>> >>>>> 2023-02-26 08:49:55.216 7 WARNING
>>>>>>>>>>>>> nova.cache_utils
>>>>>>>>>>>>> >>>>> >>>> >>>>> [req-3a1547ea-326f-4dd0-9127-7f4a4bdf1e45
>>>>>>>>>>>>> >>>>> >>>> >>>>> b240e3e89d99489284cd731e75f2a5db
>>>>>>>>>>>>> >>>>> >>>> >>>>> 4160ce999a31485fa643aed0936dfef0 - default
>>>>>>>>>>>>> default] Cache enabled
>>>>>>>>>>>>> >>>>> >>>> with
>>>>>>>>>>>>> >>>>> >>>> >>>>> backend dogpile.cache.null.
>>>>>>>>>>>>> >>>>> >>>> >>>>> 2023-02-26 08:49:55.283 7 INFO
>>>>>>>>>>>>> oslo.privsep.daemon
>>>>>>>>>>>>> >>>>> >>>> >>>>> [req-3a1547ea-326f-4dd0-9127-7f4a4bdf1e45
>>>>>>>>>>>>> >>>>> >>>> >>>>> b240e3e89d99489284cd731e75f2a5db
>>>>>>>>>>>>> >>>>> >>>> >>>>> 4160ce999a31485fa643aed0936dfef0 - default
>>>>>>>>>>>>> default] Running
>>>>>>>>>>>>> >>>>> >>>> >>>>> privsep helper:
>>>>>>>>>>>>> >>>>> >>>> >>>>> ['sudo', 'nova-rootwrap',
>>>>>>>>>>>>> '/etc/nova/rootwrap.conf',
>>>>>>>>>>>>> >>>>> >>>> 'privsep-helper',
>>>>>>>>>>>>> >>>>> >>>> >>>>> '--config-file', '/etc/nova/nova.conf',
>>>>>>>>>>>>> '--config-file',
>>>>>>>>>>>>> >>>>> >>>> >>>>> '/etc/nova/nova-compute.conf',
>>>>>>>>>>>>> '--privsep_context',
>>>>>>>>>>>>> >>>>> >>>> >>>>> 'os_brick.privileged.default',
>>>>>>>>>>>>> '--privsep_sock_path',
>>>>>>>>>>>>> >>>>> >>>> >>>>> '/tmp/tmpin40tah6/privsep.sock']
>>>>>>>>>>>>> >>>>> >>>> >>>>> 2023-02-26 08:49:55.791 7 INFO
>>>>>>>>>>>>> oslo.privsep.daemon
>>>>>>>>>>>>> >>>>> >>>> >>>>> [req-3a1547ea-326f-4dd0-9127-7f4a4bdf1e45
>>>>>>>>>>>>> >>>>> >>>> >>>>> b240e3e89d99489284cd731e75f2a5db
>>>>>>>>>>>>> >>>>> >>>> >>>>> 4160ce999a31485fa643aed0936dfef0 - default
>>>>>>>>>>>>> default] Spawned new
>>>>>>>>>>>>> >>>>> >>>> privsep
>>>>>>>>>>>>> >>>>> >>>> >>>>> daemon via rootwrap
>>>>>>>>>>>>> >>>>> >>>> >>>>> 2023-02-26 08:49:55.717 2647 INFO
>>>>>>>>>>>>> oslo.privsep.daemon [-] privsep
>>>>>>>>>>>>> >>>>> >>>> >>>>> daemon starting
>>>>>>>>>>>>> >>>>> >>>> >>>>> 2023-02-26 08:49:55.722 2647 INFO
>>>>>>>>>>>>> oslo.privsep.daemon [-] privsep
>>>>>>>>>>>>> >>>>> >>>> >>>>> process running with uid/gid: 0/0
>>>>>>>>>>>>> >>>>> >>>> >>>>> 2023-02-26 08:49:55.726 2647 INFO
>>>>>>>>>>>>> oslo.privsep.daemon [-] privsep
>>>>>>>>>>>>> >>>>> >>>> >>>>> process running with capabilities
>>>>>>>>>>>>> (eff/prm/inh):
>>>>>>>>>>>>> >>>>> >>>> >>>>> CAP_SYS_ADMIN/CAP_SYS_ADMIN/none
>>>>>>>>>>>>> >>>>> >>>> >>>>> 2023-02-26 08:49:55.726 2647 INFO
>>>>>>>>>>>>> oslo.privsep.daemon [-] privsep
>>>>>>>>>>>>> >>>>> >>>> >>>>> daemon running as pid 2647
>>>>>>>>>>>>> >>>>> >>>> >>>>> 2023-02-26 08:49:55.956 7 WARNING
>>>>>>>>>>>>> >>>>> >>>> os_brick.initiator.connectors.nvmeof
>>>>>>>>>>>>> >>>>> >>>> >>>>> [req-3a1547ea-326f-4dd0-9127-7f4a4bdf1e45
>>>>>>>>>>>>> >>>>> >>>> >>>>> b240e3e89d99489284cd731e75f2a5db
>>>>>>>>>>>>> >>>>> >>>> >>>>> 4160ce999a31485fa643aed0936dfef0 - default
>>>>>>>>>>>>> default] Process
>>>>>>>>>>>>> >>>>> >>>> >>>>> execution error
>>>>>>>>>>>>> >>>>> >>>> >>>>> in _get_host_uuid: Unexpected error while
>>>>>>>>>>>>> running command.
>>>>>>>>>>>>> >>>>> >>>> >>>>> Command: blkid overlay -s UUID -o value
>>>>>>>>>>>>> >>>>> >>>> >>>>> Exit code: 2
>>>>>>>>>>>>> >>>>> >>>> >>>>> Stdout: ''
>>>>>>>>>>>>> >>>>> >>>> >>>>> Stderr: '':
>>>>>>>>>>>>> oslo_concurrency.processutils.ProcessExecutionError:
>>>>>>>>>>>>> >>>>> >>>> >>>>> Unexpected error while running command.
>>>>>>>>>>>>> >>>>> >>>> >>>>> 2023-02-26 08:49:58.247 7 INFO
>>>>>>>>>>>>> nova.virt.libvirt.driver
>>>>>>>>>>>>> >>>>> >>>> >>>>> [req-3a1547ea-326f-4dd0-9127-7f4a4bdf1e45
>>>>>>>>>>>>> >>>>> >>>> >>>>> b240e3e89d99489284cd731e75f2a5db
>>>>>>>>>>>>> >>>>> >>>> >>>>> 4160ce999a31485fa643aed0936dfef0 - default
>>>>>>>>>>>>> default] [instance:
>>>>>>>>>>>>> >>>>> >>>> >>>>> 0c62c1ef-9010-417d-a05f-4db77e901600]
>>>>>>>>>>>>> Creating image
>>>>>>>>>>>>> >>>>> >>>> >>>>>
>>>>>>>>>>>>> >>>>> >>>> >>>>> Is there a way to solve this issue?
>>>>>>>>>>>>> >>>>> >>>> >>>>>
>>>>>>>>>>>>> >>>>> >>>> >>>>>
>>>>>>>>>>>>> >>>>> >>>> >>>>> With regards,
>>>>>>>>>>>>> >>>>> >>>> >>>>>
>>>>>>>>>>>>> >>>>> >>>> >>>>> Swogat Pradhan
>>>>>>>>>>>>> >>>>> >>>> >>>>>
>>>>>>>>>>>>> >>>>> >>>> >>>>
>>>>>>>>>>>>> >>>>> >>>>
>>>>>>>>>>>>> >>>>> >>>>
>>>>>>>>>>>>> >>>>> >>>>
>>>>>>>>>>>>> >>>>> >>>>
>>>>>>>>>>>>> >>>>> >>>>
>>>>>>>>>>>>> >>>>> >>>>
>>>>>>>>>>>>> >>>>> >>>>
>>>>>>>>>>>>> >>>>> >>>>
>>>>>>>>>>>>> >>>>> >>>>
>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230328/9dc9ffac/attachment-0001.htm>
More information about the openstack-discuss
mailing list