Glance multistore config makes image service inaccessible Edit | openstack wallaby | tripleo
Hi, Openstack wallaby | tripleo I am trying to setup DCN where i want to setup a glance multistore config in both central and edge sites as per the documentation. But when ever i run stack update with the following config for multistore: [stack@hkg2director ~]$ cat workplace/glance.yaml parameter_defaults: GlanceShowMultipleLocations: true GlanceBackend: rbd GlanceEnabledImportMethods: web-download,copy-image GlanceBackendID: ceph GlanceMultistoreConfig: dcn01: GlanceBackend: rbd GlanceStoreDescription: 'dcn01 rbd glance store' CephClusterName: dcn01 The image service becomes inaccessible. Giving the following error: (overcloud) [stack@hkg2director ~]$ openstack image list Failed to contact the endpoint at https://bolt.bdxworld.com:13292 for discovery. Fallback to using that endpoint as the base url. Failed to contact the endpoint at https://bolt.bdxworld.com:13292 for discovery. Fallback to using that endpoint as the base url. The image service for :regionOne exists but does not have any supported versions. glance api.log from controller nodes: 2023-02-13 13:02:19.878 7 INFO glance.async_ [-] Threadpool model set to 'EventletThreadPoolModel' 2023-02-13 13:02:19.969 7 INFO keyring.backend [-] Loading KWallet 2023-02-13 13:02:19.977 7 INFO keyring.backend [-] Loading SecretService 2023-02-13 13:02:19.982 7 INFO keyring.backend [-] Loading Windows 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading chainer 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading macOS 2023-02-13 13:02:20.571 7 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True. 2023-02-13 13:02:20.711 7 ERROR stevedore.extension [-] Could not load 'glance.store.s3.Store': No module named 'boto3': ModuleNotFoundError: No module named 'boto3' 2023-02-13 13:02:20.719 7 ERROR stevedore.extension [-] Could not load 's3': No module named 'boto3': ModuleNotFoundError: No module named 'boto3' Is there a patch to fix this issue??
Additional INFO: The glance_api container is in reboot loop after adding multi store config. Which is why the image service has become unreachable. On Mon, Feb 13, 2023 at 6:39 PM Swogat Pradhan <swogatpradhan22@gmail.com> wrote:
Hi, Openstack wallaby | tripleo
I am trying to setup DCN where i want to setup a glance multistore config in both central and edge sites as per the documentation. But when ever i run stack update with the following config for multistore:
[stack@hkg2director ~]$ cat workplace/glance.yaml parameter_defaults: GlanceShowMultipleLocations: true GlanceBackend: rbd GlanceEnabledImportMethods: web-download,copy-image GlanceBackendID: ceph GlanceMultistoreConfig: dcn01: GlanceBackend: rbd GlanceStoreDescription: 'dcn01 rbd glance store' CephClusterName: dcn01
The image service becomes inaccessible. Giving the following error:
(overcloud) [stack@hkg2director ~]$ openstack image list Failed to contact the endpoint at https://bolt.bdxworld.com:13292 for discovery. Fallback to using that endpoint as the base url. Failed to contact the endpoint at https://bolt.bdxworld.com:13292 for discovery. Fallback to using that endpoint as the base url. The image service for :regionOne exists but does not have any supported versions.
glance api.log from controller nodes:
2023-02-13 13:02:19.878 7 INFO glance.async_ [-] Threadpool model set to 'EventletThreadPoolModel' 2023-02-13 13:02:19.969 7 INFO keyring.backend [-] Loading KWallet 2023-02-13 13:02:19.977 7 INFO keyring.backend [-] Loading SecretService 2023-02-13 13:02:19.982 7 INFO keyring.backend [-] Loading Windows 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading chainer 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading macOS 2023-02-13 13:02:20.571 7 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True. 2023-02-13 13:02:20.711 7 ERROR stevedore.extension [-] Could not load 'glance.store.s3.Store': No module named 'boto3': ModuleNotFoundError: No module named 'boto3' 2023-02-13 13:02:20.719 7 ERROR stevedore.extension [-] Could not load 's3': No module named 'boto3': ModuleNotFoundError: No module named 'boto3'
Is there a patch to fix this issue??
Hi Swogat, The glance.yaml you've shared is for central site or edge site? Is it possible for you to share your glance-api.conf just to check how configuration values are set? Thanks, Pranali On Tue, Feb 14, 2023 at 12:35 PM Swogat Pradhan <swogatpradhan22@gmail.com> wrote:
Additional INFO: The glance_api container is in reboot loop after adding multi store config. Which is why the image service has become unreachable.
On Mon, Feb 13, 2023 at 6:39 PM Swogat Pradhan <swogatpradhan22@gmail.com> wrote:
Hi, Openstack wallaby | tripleo
I am trying to setup DCN where i want to setup a glance multistore config in both central and edge sites as per the documentation. But when ever i run stack update with the following config for multistore:
[stack@hkg2director ~]$ cat workplace/glance.yaml parameter_defaults: GlanceShowMultipleLocations: true GlanceBackend: rbd GlanceEnabledImportMethods: web-download,copy-image GlanceBackendID: ceph GlanceMultistoreConfig: dcn01: GlanceBackend: rbd GlanceStoreDescription: 'dcn01 rbd glance store' CephClusterName: dcn01
The image service becomes inaccessible. Giving the following error:
(overcloud) [stack@hkg2director ~]$ openstack image list Failed to contact the endpoint at https://bolt.bdxworld.com:13292 for discovery. Fallback to using that endpoint as the base url. Failed to contact the endpoint at https://bolt.bdxworld.com:13292 for discovery. Fallback to using that endpoint as the base url. The image service for :regionOne exists but does not have any supported versions.
glance api.log from controller nodes:
2023-02-13 13:02:19.878 7 INFO glance.async_ [-] Threadpool model set to 'EventletThreadPoolModel' 2023-02-13 13:02:19.969 7 INFO keyring.backend [-] Loading KWallet 2023-02-13 13:02:19.977 7 INFO keyring.backend [-] Loading SecretService 2023-02-13 13:02:19.982 7 INFO keyring.backend [-] Loading Windows 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading chainer 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading macOS 2023-02-13 13:02:20.571 7 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True. 2023-02-13 13:02:20.711 7 ERROR stevedore.extension [-] Could not load 'glance.store.s3.Store': No module named 'boto3': ModuleNotFoundError: No module named 'boto3' 2023-02-13 13:02:20.719 7 ERROR stevedore.extension [-] Could not load 's3': No module named 'boto3': ModuleNotFoundError: No module named 'boto3'
Is there a patch to fix this issue??
Hi Pranali, Thank you for your response. The glance.yaml i had shared is for the central site. I am attaching the required files for your reference. With regards, Swogat Pradhan On Mon, Feb 20, 2023 at 4:29 PM Pranali Deore <pdeore@redhat.com> wrote:
Hi Swogat,
The glance.yaml you've shared is for central site or edge site? Is it possible for you to share your glance-api.conf just to check how configuration values are set?
Thanks, Pranali
On Tue, Feb 14, 2023 at 12:35 PM Swogat Pradhan <swogatpradhan22@gmail.com> wrote:
Additional INFO: The glance_api container is in reboot loop after adding multi store config. Which is why the image service has become unreachable.
On Mon, Feb 13, 2023 at 6:39 PM Swogat Pradhan <swogatpradhan22@gmail.com> wrote:
Hi, Openstack wallaby | tripleo
I am trying to setup DCN where i want to setup a glance multistore config in both central and edge sites as per the documentation. But when ever i run stack update with the following config for multistore:
[stack@hkg2director ~]$ cat workplace/glance.yaml parameter_defaults: GlanceShowMultipleLocations: true GlanceBackend: rbd GlanceEnabledImportMethods: web-download,copy-image GlanceBackendID: ceph GlanceMultistoreConfig: dcn01: GlanceBackend: rbd GlanceStoreDescription: 'dcn01 rbd glance store' CephClusterName: dcn01
The image service becomes inaccessible. Giving the following error:
(overcloud) [stack@hkg2director ~]$ openstack image list Failed to contact the endpoint at https://bolt.bdxworld.com:13292 for discovery. Fallback to using that endpoint as the base url. Failed to contact the endpoint at https://bolt.bdxworld.com:13292 for discovery. Fallback to using that endpoint as the base url. The image service for :regionOne exists but does not have any supported versions.
glance api.log from controller nodes:
2023-02-13 13:02:19.878 7 INFO glance.async_ [-] Threadpool model set to 'EventletThreadPoolModel' 2023-02-13 13:02:19.969 7 INFO keyring.backend [-] Loading KWallet 2023-02-13 13:02:19.977 7 INFO keyring.backend [-] Loading SecretService 2023-02-13 13:02:19.982 7 INFO keyring.backend [-] Loading Windows 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading chainer 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading macOS 2023-02-13 13:02:20.571 7 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True. 2023-02-13 13:02:20.711 7 ERROR stevedore.extension [-] Could not load 'glance.store.s3.Store': No module named 'boto3': ModuleNotFoundError: No module named 'boto3' 2023-02-13 13:02:20.719 7 ERROR stevedore.extension [-] Could not load 's3': No module named 'boto3': ModuleNotFoundError: No module named 'boto3'
Is there a patch to fix this issue??
Hey Swogat, By looking at the conf files you've shared, not sure from where the extra store sections 'central', 'cinder_store' are added since you have not added those in any of your yaml files. Could you please share glance-api logs from both the nodes to check more details? Probably the HA configuration is causing this kind of issue but need more information related to deployment & log to understand the root cause. Thanks, ~Pranali On Mon, Feb 20, 2023 at 4:45 PM Swogat Pradhan <swogatpradhan22@gmail.com> wrote:
Hi Pranali, Thank you for your response. The glance.yaml i had shared is for the central site. I am attaching the required files for your reference.
With regards, Swogat Pradhan
On Mon, Feb 20, 2023 at 4:29 PM Pranali Deore <pdeore@redhat.com> wrote:
Hi Swogat,
The glance.yaml you've shared is for central site or edge site? Is it possible for you to share your glance-api.conf just to check how configuration values are set?
Thanks, Pranali
On Tue, Feb 14, 2023 at 12:35 PM Swogat Pradhan < swogatpradhan22@gmail.com> wrote:
Additional INFO: The glance_api container is in reboot loop after adding multi store config. Which is why the image service has become unreachable.
On Mon, Feb 13, 2023 at 6:39 PM Swogat Pradhan < swogatpradhan22@gmail.com> wrote:
Hi, Openstack wallaby | tripleo
I am trying to setup DCN where i want to setup a glance multistore config in both central and edge sites as per the documentation. But when ever i run stack update with the following config for multistore:
[stack@hkg2director ~]$ cat workplace/glance.yaml parameter_defaults: GlanceShowMultipleLocations: true GlanceBackend: rbd GlanceEnabledImportMethods: web-download,copy-image GlanceBackendID: ceph GlanceMultistoreConfig: dcn01: GlanceBackend: rbd GlanceStoreDescription: 'dcn01 rbd glance store' CephClusterName: dcn01
The image service becomes inaccessible. Giving the following error:
(overcloud) [stack@hkg2director ~]$ openstack image list Failed to contact the endpoint at https://bolt.bdxworld.com:13292 for discovery. Fallback to using that endpoint as the base url. Failed to contact the endpoint at https://bolt.bdxworld.com:13292 for discovery. Fallback to using that endpoint as the base url. The image service for :regionOne exists but does not have any supported versions.
glance api.log from controller nodes:
2023-02-13 13:02:19.878 7 INFO glance.async_ [-] Threadpool model set to 'EventletThreadPoolModel' 2023-02-13 13:02:19.969 7 INFO keyring.backend [-] Loading KWallet 2023-02-13 13:02:19.977 7 INFO keyring.backend [-] Loading SecretService 2023-02-13 13:02:19.982 7 INFO keyring.backend [-] Loading Windows 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading chainer 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading macOS 2023-02-13 13:02:20.571 7 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True. 2023-02-13 13:02:20.711 7 ERROR stevedore.extension [-] Could not load 'glance.store.s3.Store': No module named 'boto3': ModuleNotFoundError: No module named 'boto3' 2023-02-13 13:02:20.719 7 ERROR stevedore.extension [-] Could not load 's3': No module named 'boto3': ModuleNotFoundError: No module named 'boto3'
Is there a patch to fix this issue??
Hi pranali, Those configs('central', 'cinder_store') were added when i had started deploying the cloud, those parameters have been removed from the yaml now, but exist in the config file. Right now i have reverted the config so i am not getting any erros in the glance api logs. But when i configure the multistore i get the following errors on both sides (central and edge) in glance_api.log: 2023-02-13 13:02:19.878 7 INFO glance.async_ [-] Threadpool model set to 'EventletThreadPoolModel' 2023-02-13 13:02:19.969 7 INFO keyring.backend [-] Loading KWallet 2023-02-13 13:02:19.977 7 INFO keyring.backend [-] Loading SecretService 2023-02-13 13:02:19.982 7 INFO keyring.backend [-] Loading Windows 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading chainer 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading macOS 2023-02-13 13:02:20.571 7 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True. 2023-02-13 13:02:20.711 7 ERROR stevedore.extension [-] Could not load 'glance.store.s3.Store': No module named 'boto3': ModuleNotFoundError: No module named 'boto3' 2023-02-13 13:02:20.719 7 ERROR stevedore.extension [-] Could not load 's3': No module named 'boto3': ModuleNotFoundError: No module named 'boto3' I get no error whatsoever other than the above. With regards, Swogat Pradhan On Tue, Feb 21, 2023 at 11:44 AM Pranali Deore <pdeore@redhat.com> wrote:
Hey Swogat,
By looking at the conf files you've shared, not sure from where the extra store sections 'central', 'cinder_store' are added since you have not added those in any of your yaml files. Could you please share glance-api logs from both the nodes to check more details? Probably the HA configuration is causing this kind of issue but need more information related to deployment & log to understand the root cause.
Thanks, ~Pranali
On Mon, Feb 20, 2023 at 4:45 PM Swogat Pradhan <swogatpradhan22@gmail.com> wrote:
Hi Pranali, Thank you for your response. The glance.yaml i had shared is for the central site. I am attaching the required files for your reference.
With regards, Swogat Pradhan
On Mon, Feb 20, 2023 at 4:29 PM Pranali Deore <pdeore@redhat.com> wrote:
Hi Swogat,
The glance.yaml you've shared is for central site or edge site? Is it possible for you to share your glance-api.conf just to check how configuration values are set?
Thanks, Pranali
On Tue, Feb 14, 2023 at 12:35 PM Swogat Pradhan < swogatpradhan22@gmail.com> wrote:
Additional INFO: The glance_api container is in reboot loop after adding multi store config. Which is why the image service has become unreachable.
On Mon, Feb 13, 2023 at 6:39 PM Swogat Pradhan < swogatpradhan22@gmail.com> wrote:
Hi, Openstack wallaby | tripleo
I am trying to setup DCN where i want to setup a glance multistore config in both central and edge sites as per the documentation. But when ever i run stack update with the following config for multistore:
[stack@hkg2director ~]$ cat workplace/glance.yaml parameter_defaults: GlanceShowMultipleLocations: true GlanceBackend: rbd GlanceEnabledImportMethods: web-download,copy-image GlanceBackendID: ceph GlanceMultistoreConfig: dcn01: GlanceBackend: rbd GlanceStoreDescription: 'dcn01 rbd glance store' CephClusterName: dcn01
The image service becomes inaccessible. Giving the following error:
(overcloud) [stack@hkg2director ~]$ openstack image list Failed to contact the endpoint at https://bolt.bdxworld.com:13292 for discovery. Fallback to using that endpoint as the base url. Failed to contact the endpoint at https://bolt.bdxworld.com:13292 for discovery. Fallback to using that endpoint as the base url. The image service for :regionOne exists but does not have any supported versions.
glance api.log from controller nodes:
2023-02-13 13:02:19.878 7 INFO glance.async_ [-] Threadpool model set to 'EventletThreadPoolModel' 2023-02-13 13:02:19.969 7 INFO keyring.backend [-] Loading KWallet 2023-02-13 13:02:19.977 7 INFO keyring.backend [-] Loading SecretService 2023-02-13 13:02:19.982 7 INFO keyring.backend [-] Loading Windows 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading chainer 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading macOS 2023-02-13 13:02:20.571 7 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True. 2023-02-13 13:02:20.711 7 ERROR stevedore.extension [-] Could not load 'glance.store.s3.Store': No module named 'boto3': ModuleNotFoundError: No module named 'boto3' 2023-02-13 13:02:20.719 7 ERROR stevedore.extension [-] Could not load 's3': No module named 'boto3': ModuleNotFoundError: No module named 'boto3'
Is there a patch to fix this issue??
Hi Swogat, I think this looks relevant, kindly go through this and confirm. https://bugs.launchpad.net/kolla/+bug/1884259 Thanks & Best Regards, Abhishek Kekane On Tue, Feb 21, 2023 at 11:58 AM Swogat Pradhan <swogatpradhan22@gmail.com> wrote:
Hi pranali, Those configs('central', 'cinder_store') were added when i had started deploying the cloud, those parameters have been removed from the yaml now, but exist in the config file. Right now i have reverted the config so i am not getting any erros in the glance api logs. But when i configure the multistore i get the following errors on both sides (central and edge) in glance_api.log: 2023-02-13 13:02:19.878 7 INFO glance.async_ [-] Threadpool model set to 'EventletThreadPoolModel' 2023-02-13 13:02:19.969 7 INFO keyring.backend [-] Loading KWallet 2023-02-13 13:02:19.977 7 INFO keyring.backend [-] Loading SecretService 2023-02-13 13:02:19.982 7 INFO keyring.backend [-] Loading Windows 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading chainer 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading macOS 2023-02-13 13:02:20.571 7 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True. 2023-02-13 13:02:20.711 7 ERROR stevedore.extension [-] Could not load 'glance.store.s3.Store': No module named 'boto3': ModuleNotFoundError: No module named 'boto3' 2023-02-13 13:02:20.719 7 ERROR stevedore.extension [-] Could not load 's3': No module named 'boto3': ModuleNotFoundError: No module named 'boto3'
I get no error whatsoever other than the above.
With regards, Swogat Pradhan
On Tue, Feb 21, 2023 at 11:44 AM Pranali Deore <pdeore@redhat.com> wrote:
Hey Swogat,
By looking at the conf files you've shared, not sure from where the extra store sections 'central', 'cinder_store' are added since you have not added those in any of your yaml files. Could you please share glance-api logs from both the nodes to check more details? Probably the HA configuration is causing this kind of issue but need more information related to deployment & log to understand the root cause.
Thanks, ~Pranali
On Mon, Feb 20, 2023 at 4:45 PM Swogat Pradhan <swogatpradhan22@gmail.com> wrote:
Hi Pranali, Thank you for your response. The glance.yaml i had shared is for the central site. I am attaching the required files for your reference.
With regards, Swogat Pradhan
On Mon, Feb 20, 2023 at 4:29 PM Pranali Deore <pdeore@redhat.com> wrote:
Hi Swogat,
The glance.yaml you've shared is for central site or edge site? Is it possible for you to share your glance-api.conf just to check how configuration values are set?
Thanks, Pranali
On Tue, Feb 14, 2023 at 12:35 PM Swogat Pradhan < swogatpradhan22@gmail.com> wrote:
Additional INFO: The glance_api container is in reboot loop after adding multi store config. Which is why the image service has become unreachable.
On Mon, Feb 13, 2023 at 6:39 PM Swogat Pradhan < swogatpradhan22@gmail.com> wrote:
Hi, Openstack wallaby | tripleo
I am trying to setup DCN where i want to setup a glance multistore config in both central and edge sites as per the documentation. But when ever i run stack update with the following config for multistore:
[stack@hkg2director ~]$ cat workplace/glance.yaml parameter_defaults: GlanceShowMultipleLocations: true GlanceBackend: rbd GlanceEnabledImportMethods: web-download,copy-image GlanceBackendID: ceph GlanceMultistoreConfig: dcn01: GlanceBackend: rbd GlanceStoreDescription: 'dcn01 rbd glance store' CephClusterName: dcn01
The image service becomes inaccessible. Giving the following error:
(overcloud) [stack@hkg2director ~]$ openstack image list Failed to contact the endpoint at https://bolt.bdxworld.com:13292 for discovery. Fallback to using that endpoint as the base url. Failed to contact the endpoint at https://bolt.bdxworld.com:13292 for discovery. Fallback to using that endpoint as the base url. The image service for :regionOne exists but does not have any supported versions.
glance api.log from controller nodes:
2023-02-13 13:02:19.878 7 INFO glance.async_ [-] Threadpool model set to 'EventletThreadPoolModel' 2023-02-13 13:02:19.969 7 INFO keyring.backend [-] Loading KWallet 2023-02-13 13:02:19.977 7 INFO keyring.backend [-] Loading SecretService 2023-02-13 13:02:19.982 7 INFO keyring.backend [-] Loading Windows 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading chainer 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading macOS 2023-02-13 13:02:20.571 7 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True. 2023-02-13 13:02:20.711 7 ERROR stevedore.extension [-] Could not load 'glance.store.s3.Store': No module named 'boto3': ModuleNotFoundError: No module named 'boto3' 2023-02-13 13:02:20.719 7 ERROR stevedore.extension [-] Could not load 's3': No module named 'boto3': ModuleNotFoundError: No module named 'boto3'
Is there a patch to fix this issue??
Hi Abhishek, Yes, this feels like the same bug, as per the comment from Lars Wiegman (lars-fuga) <https://launchpad.net/~lars-fuga> my glance api container is in reboot loop, when glance multi store is configured. So I have been searching for a patch for tripleo. With regards, Swogat Pradhan On Tue, Feb 21, 2023 at 12:09 PM Abhishek Kekane <akekane@redhat.com> wrote:
Hi Swogat,
I think this looks relevant, kindly go through this and confirm. https://bugs.launchpad.net/kolla/+bug/1884259
Thanks & Best Regards,
Abhishek Kekane
On Tue, Feb 21, 2023 at 11:58 AM Swogat Pradhan <swogatpradhan22@gmail.com> wrote:
Hi pranali, Those configs('central', 'cinder_store') were added when i had started deploying the cloud, those parameters have been removed from the yaml now, but exist in the config file. Right now i have reverted the config so i am not getting any erros in the glance api logs. But when i configure the multistore i get the following errors on both sides (central and edge) in glance_api.log: 2023-02-13 13:02:19.878 7 INFO glance.async_ [-] Threadpool model set to 'EventletThreadPoolModel' 2023-02-13 13:02:19.969 7 INFO keyring.backend [-] Loading KWallet 2023-02-13 13:02:19.977 7 INFO keyring.backend [-] Loading SecretService 2023-02-13 13:02:19.982 7 INFO keyring.backend [-] Loading Windows 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading chainer 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading macOS 2023-02-13 13:02:20.571 7 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True. 2023-02-13 13:02:20.711 7 ERROR stevedore.extension [-] Could not load 'glance.store.s3.Store': No module named 'boto3': ModuleNotFoundError: No module named 'boto3' 2023-02-13 13:02:20.719 7 ERROR stevedore.extension [-] Could not load 's3': No module named 'boto3': ModuleNotFoundError: No module named 'boto3'
I get no error whatsoever other than the above.
With regards, Swogat Pradhan
On Tue, Feb 21, 2023 at 11:44 AM Pranali Deore <pdeore@redhat.com> wrote:
Hey Swogat,
By looking at the conf files you've shared, not sure from where the extra store sections 'central', 'cinder_store' are added since you have not added those in any of your yaml files. Could you please share glance-api logs from both the nodes to check more details? Probably the HA configuration is causing this kind of issue but need more information related to deployment & log to understand the root cause.
Thanks, ~Pranali
On Mon, Feb 20, 2023 at 4:45 PM Swogat Pradhan < swogatpradhan22@gmail.com> wrote:
Hi Pranali, Thank you for your response. The glance.yaml i had shared is for the central site. I am attaching the required files for your reference.
With regards, Swogat Pradhan
On Mon, Feb 20, 2023 at 4:29 PM Pranali Deore <pdeore@redhat.com> wrote:
Hi Swogat,
The glance.yaml you've shared is for central site or edge site? Is it possible for you to share your glance-api.conf just to check how configuration values are set?
Thanks, Pranali
On Tue, Feb 14, 2023 at 12:35 PM Swogat Pradhan < swogatpradhan22@gmail.com> wrote:
Additional INFO: The glance_api container is in reboot loop after adding multi store config. Which is why the image service has become unreachable.
On Mon, Feb 13, 2023 at 6:39 PM Swogat Pradhan < swogatpradhan22@gmail.com> wrote:
> Hi, > Openstack wallaby | tripleo > > I am trying to setup DCN where i want to setup a glance multistore > config in both central and edge sites as per the documentation. > But when ever i run stack update with the following config for > multistore: > > [stack@hkg2director ~]$ cat workplace/glance.yaml > parameter_defaults: > GlanceShowMultipleLocations: true > GlanceBackend: rbd > GlanceEnabledImportMethods: web-download,copy-image > GlanceBackendID: ceph > GlanceMultistoreConfig: > dcn01: > GlanceBackend: rbd > GlanceStoreDescription: 'dcn01 rbd glance store' > CephClusterName: dcn01 > > The image service becomes inaccessible. Giving the following error: > > (overcloud) [stack@hkg2director ~]$ openstack image list > Failed to contact the endpoint at https://bolt.bdxworld.com:13292 > for discovery. Fallback to using that endpoint as the base url. > Failed to contact the endpoint at https://bolt.bdxworld.com:13292 > for discovery. Fallback to using that endpoint as the base url. > The image service for :regionOne exists but does not have any > supported versions. > > > glance api.log from controller nodes: > > 2023-02-13 13:02:19.878 7 INFO glance.async_ [-] Threadpool model > set to 'EventletThreadPoolModel' > 2023-02-13 13:02:19.969 7 INFO keyring.backend [-] Loading KWallet > 2023-02-13 13:02:19.977 7 INFO keyring.backend [-] Loading > SecretService > 2023-02-13 13:02:19.982 7 INFO keyring.backend [-] Loading Windows > 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading chainer > 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading macOS > 2023-02-13 13:02:20.571 7 WARNING keystonemiddleware.auth_token [-] > AuthToken middleware is set with > keystone_authtoken.service_token_roles_required set to False. This is > backwards compatible but deprecated behaviour. Please set this to True. > 2023-02-13 13:02:20.711 7 ERROR stevedore.extension [-] Could not > load 'glance.store.s3.Store': No module named 'boto3': ModuleNotFoundError: > No module named 'boto3' > 2023-02-13 13:02:20.719 7 ERROR stevedore.extension [-] Could not > load 's3': No module named 'boto3': ModuleNotFoundError: No module named > 'boto3' > > Is there a patch to fix this issue?? >
This doesn't seem like a fatal error, so the issue might be something else. What podman logs glance_api shows? On Tue, Feb 21, 2023 at 12:14 PM Swogat Pradhan <swogatpradhan22@gmail.com> wrote:
Hi Abhishek, Yes, this feels like the same bug, as per the comment from Lars Wiegman (lars-fuga) <https://launchpad.net/~lars-fuga> my glance api container is in reboot loop, when glance multi store is configured.
So I have been searching for a patch for tripleo.
With regards, Swogat Pradhan
On Tue, Feb 21, 2023 at 12:09 PM Abhishek Kekane <akekane@redhat.com> wrote:
Hi Swogat,
I think this looks relevant, kindly go through this and confirm. https://bugs.launchpad.net/kolla/+bug/1884259
Thanks & Best Regards,
Abhishek Kekane
On Tue, Feb 21, 2023 at 11:58 AM Swogat Pradhan < swogatpradhan22@gmail.com> wrote:
Hi pranali, Those configs('central', 'cinder_store') were added when i had started deploying the cloud, those parameters have been removed from the yaml now, but exist in the config file. Right now i have reverted the config so i am not getting any erros in the glance api logs. But when i configure the multistore i get the following errors on both sides (central and edge) in glance_api.log: 2023-02-13 13:02:19.878 7 INFO glance.async_ [-] Threadpool model set to 'EventletThreadPoolModel' 2023-02-13 13:02:19.969 7 INFO keyring.backend [-] Loading KWallet 2023-02-13 13:02:19.977 7 INFO keyring.backend [-] Loading SecretService 2023-02-13 13:02:19.982 7 INFO keyring.backend [-] Loading Windows 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading chainer 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading macOS 2023-02-13 13:02:20.571 7 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True. 2023-02-13 13:02:20.711 7 ERROR stevedore.extension [-] Could not load 'glance.store.s3.Store': No module named 'boto3': ModuleNotFoundError: No module named 'boto3' 2023-02-13 13:02:20.719 7 ERROR stevedore.extension [-] Could not load 's3': No module named 'boto3': ModuleNotFoundError: No module named 'boto3'
I get no error whatsoever other than the above.
With regards, Swogat Pradhan
On Tue, Feb 21, 2023 at 11:44 AM Pranali Deore <pdeore@redhat.com> wrote:
Hey Swogat,
By looking at the conf files you've shared, not sure from where the extra store sections 'central', 'cinder_store' are added since you have not added those in any of your yaml files. Could you please share glance-api logs from both the nodes to check more details? Probably the HA configuration is causing this kind of issue but need more information related to deployment & log to understand the root cause.
Thanks, ~Pranali
On Mon, Feb 20, 2023 at 4:45 PM Swogat Pradhan < swogatpradhan22@gmail.com> wrote:
Hi Pranali, Thank you for your response. The glance.yaml i had shared is for the central site. I am attaching the required files for your reference.
With regards, Swogat Pradhan
On Mon, Feb 20, 2023 at 4:29 PM Pranali Deore <pdeore@redhat.com> wrote:
Hi Swogat,
The glance.yaml you've shared is for central site or edge site? Is it possible for you to share your glance-api.conf just to check how configuration values are set?
Thanks, Pranali
On Tue, Feb 14, 2023 at 12:35 PM Swogat Pradhan < swogatpradhan22@gmail.com> wrote:
> Additional INFO: > The glance_api container is in reboot loop after adding multi store > config. > Which is why the image service has become unreachable. > > > On Mon, Feb 13, 2023 at 6:39 PM Swogat Pradhan < > swogatpradhan22@gmail.com> wrote: > >> Hi, >> Openstack wallaby | tripleo >> >> I am trying to setup DCN where i want to setup a glance multistore >> config in both central and edge sites as per the documentation. >> But when ever i run stack update with the following config for >> multistore: >> >> [stack@hkg2director ~]$ cat workplace/glance.yaml >> parameter_defaults: >> GlanceShowMultipleLocations: true >> GlanceBackend: rbd >> GlanceEnabledImportMethods: web-download,copy-image >> GlanceBackendID: ceph >> GlanceMultistoreConfig: >> dcn01: >> GlanceBackend: rbd >> GlanceStoreDescription: 'dcn01 rbd glance store' >> CephClusterName: dcn01 >> >> The image service becomes inaccessible. Giving the following error: >> >> (overcloud) [stack@hkg2director ~]$ openstack image list >> Failed to contact the endpoint at https://bolt.bdxworld.com:13292 >> for discovery. Fallback to using that endpoint as the base url. >> Failed to contact the endpoint at https://bolt.bdxworld.com:13292 >> for discovery. Fallback to using that endpoint as the base url. >> The image service for :regionOne exists but does not have any >> supported versions. >> >> >> glance api.log from controller nodes: >> >> 2023-02-13 13:02:19.878 7 INFO glance.async_ [-] Threadpool model >> set to 'EventletThreadPoolModel' >> 2023-02-13 13:02:19.969 7 INFO keyring.backend [-] Loading KWallet >> 2023-02-13 13:02:19.977 7 INFO keyring.backend [-] Loading >> SecretService >> 2023-02-13 13:02:19.982 7 INFO keyring.backend [-] Loading Windows >> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading chainer >> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading macOS >> 2023-02-13 13:02:20.571 7 WARNING keystonemiddleware.auth_token [-] >> AuthToken middleware is set with >> keystone_authtoken.service_token_roles_required set to False. This is >> backwards compatible but deprecated behaviour. Please set this to True. >> 2023-02-13 13:02:20.711 7 ERROR stevedore.extension [-] Could not >> load 'glance.store.s3.Store': No module named 'boto3': ModuleNotFoundError: >> No module named 'boto3' >> 2023-02-13 13:02:20.719 7 ERROR stevedore.extension [-] Could not >> load 's3': No module named 'boto3': ModuleNotFoundError: No module named >> 'boto3' >> >> Is there a patch to fix this issue?? >> >
I looked at tcib definitions but we have never added boto3 even in master. However, according to the glance_store code, I expect glance_store would be able to ignore the error even if boto3 is not enabled unless you enable s3 backend somehow. https://github.com/openstack/glance_store/blob/64e25979a904e468e1e89d8727e95... If you can find the log line "Failed to load driver ..." in glance-api.log then the cause would be different and we have to check the full logs(both /var/log/containers/glance/glance-api.log and probably /var/log/containers/sdtouts/glance_api.log) to find out what is actually causing the crash. On Tue, Feb 21, 2023 at 4:05 PM Pranali Deore <pdeore@redhat.com> wrote:
This doesn't seem like a fatal error, so the issue might be something else. What podman logs glance_api shows?
On Tue, Feb 21, 2023 at 12:14 PM Swogat Pradhan <swogatpradhan22@gmail.com> wrote:
Hi Abhishek, Yes, this feels like the same bug, as per the comment from Lars Wiegman (lars-fuga) <https://launchpad.net/~lars-fuga> my glance api container is in reboot loop, when glance multi store is configured.
So I have been searching for a patch for tripleo.
With regards, Swogat Pradhan
On Tue, Feb 21, 2023 at 12:09 PM Abhishek Kekane <akekane@redhat.com> wrote:
Hi Swogat,
I think this looks relevant, kindly go through this and confirm. https://bugs.launchpad.net/kolla/+bug/1884259
Thanks & Best Regards,
Abhishek Kekane
On Tue, Feb 21, 2023 at 11:58 AM Swogat Pradhan < swogatpradhan22@gmail.com> wrote:
Hi pranali, Those configs('central', 'cinder_store') were added when i had started deploying the cloud, those parameters have been removed from the yaml now, but exist in the config file. Right now i have reverted the config so i am not getting any erros in the glance api logs. But when i configure the multistore i get the following errors on both sides (central and edge) in glance_api.log: 2023-02-13 13:02:19.878 7 INFO glance.async_ [-] Threadpool model set to 'EventletThreadPoolModel' 2023-02-13 13:02:19.969 7 INFO keyring.backend [-] Loading KWallet 2023-02-13 13:02:19.977 7 INFO keyring.backend [-] Loading SecretService 2023-02-13 13:02:19.982 7 INFO keyring.backend [-] Loading Windows 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading chainer 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading macOS 2023-02-13 13:02:20.571 7 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True. 2023-02-13 13:02:20.711 7 ERROR stevedore.extension [-] Could not load 'glance.store.s3.Store': No module named 'boto3': ModuleNotFoundError: No module named 'boto3' 2023-02-13 13:02:20.719 7 ERROR stevedore.extension [-] Could not load 's3': No module named 'boto3': ModuleNotFoundError: No module named 'boto3'
I get no error whatsoever other than the above.
With regards, Swogat Pradhan
On Tue, Feb 21, 2023 at 11:44 AM Pranali Deore <pdeore@redhat.com> wrote:
Hey Swogat,
By looking at the conf files you've shared, not sure from where the extra store sections 'central', 'cinder_store' are added since you have not added those in any of your yaml files. Could you please share glance-api logs from both the nodes to check more details? Probably the HA configuration is causing this kind of issue but need more information related to deployment & log to understand the root cause.
Thanks, ~Pranali
On Mon, Feb 20, 2023 at 4:45 PM Swogat Pradhan < swogatpradhan22@gmail.com> wrote:
Hi Pranali, Thank you for your response. The glance.yaml i had shared is for the central site. I am attaching the required files for your reference.
With regards, Swogat Pradhan
On Mon, Feb 20, 2023 at 4:29 PM Pranali Deore <pdeore@redhat.com> wrote:
> Hi Swogat, > > The glance.yaml you've shared is for central site or edge site? > Is it possible for you to share your glance-api.conf just to check > how configuration values are set? > > > > Thanks, > Pranali > > On Tue, Feb 14, 2023 at 12:35 PM Swogat Pradhan < > swogatpradhan22@gmail.com> wrote: > >> Additional INFO: >> The glance_api container is in reboot loop after adding multi store >> config. >> Which is why the image service has become unreachable. >> >> >> On Mon, Feb 13, 2023 at 6:39 PM Swogat Pradhan < >> swogatpradhan22@gmail.com> wrote: >> >>> Hi, >>> Openstack wallaby | tripleo >>> >>> I am trying to setup DCN where i want to setup a glance multistore >>> config in both central and edge sites as per the documentation. >>> But when ever i run stack update with the following config for >>> multistore: >>> >>> [stack@hkg2director ~]$ cat workplace/glance.yaml >>> parameter_defaults: >>> GlanceShowMultipleLocations: true >>> GlanceBackend: rbd >>> GlanceEnabledImportMethods: web-download,copy-image >>> GlanceBackendID: ceph >>> GlanceMultistoreConfig: >>> dcn01: >>> GlanceBackend: rbd >>> GlanceStoreDescription: 'dcn01 rbd glance store' >>> CephClusterName: dcn01 >>> >>> The image service becomes inaccessible. Giving the following error: >>> >>> (overcloud) [stack@hkg2director ~]$ openstack image list >>> Failed to contact the endpoint at https://bolt.bdxworld.com:13292 >>> for discovery. Fallback to using that endpoint as the base url. >>> Failed to contact the endpoint at https://bolt.bdxworld.com:13292 >>> for discovery. Fallback to using that endpoint as the base url. >>> The image service for :regionOne exists but does not have any >>> supported versions. >>> >>> >>> glance api.log from controller nodes: >>> >>> 2023-02-13 13:02:19.878 7 INFO glance.async_ [-] Threadpool model >>> set to 'EventletThreadPoolModel' >>> 2023-02-13 13:02:19.969 7 INFO keyring.backend [-] Loading KWallet >>> 2023-02-13 13:02:19.977 7 INFO keyring.backend [-] Loading >>> SecretService >>> 2023-02-13 13:02:19.982 7 INFO keyring.backend [-] Loading Windows >>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading chainer >>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading macOS >>> 2023-02-13 13:02:20.571 7 WARNING keystonemiddleware.auth_token >>> [-] AuthToken middleware is set with >>> keystone_authtoken.service_token_roles_required set to False. This is >>> backwards compatible but deprecated behaviour. Please set this to True. >>> 2023-02-13 13:02:20.711 7 ERROR stevedore.extension [-] Could not >>> load 'glance.store.s3.Store': No module named 'boto3': ModuleNotFoundError: >>> No module named 'boto3' >>> 2023-02-13 13:02:20.719 7 ERROR stevedore.extension [-] Could not >>> load 's3': No module named 'boto3': ModuleNotFoundError: No module named >>> 'boto3' >>> >>> Is there a patch to fix this issue?? >>> >>
I agree with Pranali and Takashi. I have also seen "No module named 'boto3'" in my logs while setting up DCN with glance successfully. So that is not the cause of your fatal error. You should look in your logs for something else as the root cause of this issue. John On Tue, Feb 21, 2023 at 2:20 AM Takashi Kajinami <tkajinam@redhat.com> wrote:
I looked at tcib definitions but we have never added boto3 even in master.
However, according to the glance_store code, I expect glance_store would be able to ignore the error even if boto3 is not enabled unless you enable s3 backend somehow.
https://github.com/openstack/glance_store/blob/64e25979a904e468e1e89d8727e95...
If you can find the log line "Failed to load driver ..." in glance-api.log then the cause would be different and we have to check the full logs(both /var/log/containers/glance/glance-api.log and probably /var/log/containers/sdtouts/glance_api.log) to find out what is actually causing the crash.
On Tue, Feb 21, 2023 at 4:05 PM Pranali Deore <pdeore@redhat.com> wrote:
This doesn't seem like a fatal error, so the issue might be something else. What podman logs glance_api shows?
On Tue, Feb 21, 2023 at 12:14 PM Swogat Pradhan <swogatpradhan22@gmail.com> wrote:
Hi Abhishek, Yes, this feels like the same bug, as per the comment from Lars Wiegman (lars-fuga) my glance api container is in reboot loop, when glance multi store is configured.
So I have been searching for a patch for tripleo.
With regards, Swogat Pradhan
On Tue, Feb 21, 2023 at 12:09 PM Abhishek Kekane <akekane@redhat.com> wrote:
Hi Swogat,
I think this looks relevant, kindly go through this and confirm. https://bugs.launchpad.net/kolla/+bug/1884259
Thanks & Best Regards,
Abhishek Kekane
On Tue, Feb 21, 2023 at 11:58 AM Swogat Pradhan <swogatpradhan22@gmail.com> wrote:
Hi pranali, Those configs('central', 'cinder_store') were added when i had started deploying the cloud, those parameters have been removed from the yaml now, but exist in the config file. Right now i have reverted the config so i am not getting any erros in the glance api logs. But when i configure the multistore i get the following errors on both sides (central and edge) in glance_api.log: 2023-02-13 13:02:19.878 7 INFO glance.async_ [-] Threadpool model set to 'EventletThreadPoolModel' 2023-02-13 13:02:19.969 7 INFO keyring.backend [-] Loading KWallet 2023-02-13 13:02:19.977 7 INFO keyring.backend [-] Loading SecretService 2023-02-13 13:02:19.982 7 INFO keyring.backend [-] Loading Windows 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading chainer 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading macOS 2023-02-13 13:02:20.571 7 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True. 2023-02-13 13:02:20.711 7 ERROR stevedore.extension [-] Could not load 'glance.store.s3.Store': No module named 'boto3': ModuleNotFoundError: No module named 'boto3' 2023-02-13 13:02:20.719 7 ERROR stevedore.extension [-] Could not load 's3': No module named 'boto3': ModuleNotFoundError: No module named 'boto3'
I get no error whatsoever other than the above.
With regards, Swogat Pradhan
On Tue, Feb 21, 2023 at 11:44 AM Pranali Deore <pdeore@redhat.com> wrote:
Hey Swogat,
By looking at the conf files you've shared, not sure from where the extra store sections 'central', 'cinder_store' are added since you have not added those in any of your yaml files. Could you please share glance-api logs from both the nodes to check more details? Probably the HA configuration is causing this kind of issue but need more information related to deployment & log to understand the root cause.
Thanks, ~Pranali
On Mon, Feb 20, 2023 at 4:45 PM Swogat Pradhan <swogatpradhan22@gmail.com> wrote: > > Hi Pranali, > Thank you for your response. > The glance.yaml i had shared is for the central site. > I am attaching the required files for your reference. > > With regards, > Swogat Pradhan > > On Mon, Feb 20, 2023 at 4:29 PM Pranali Deore <pdeore@redhat.com> wrote: >> >> Hi Swogat, >> >> The glance.yaml you've shared is for central site or edge site? >> Is it possible for you to share your glance-api.conf just to check how configuration values are set? >> >> >> >> Thanks, >> Pranali >> >> On Tue, Feb 14, 2023 at 12:35 PM Swogat Pradhan <swogatpradhan22@gmail.com> wrote: >>> >>> Additional INFO: >>> The glance_api container is in reboot loop after adding multi store config. >>> Which is why the image service has become unreachable. >>> >>> >>> On Mon, Feb 13, 2023 at 6:39 PM Swogat Pradhan <swogatpradhan22@gmail.com> wrote: >>>> >>>> Hi, >>>> Openstack wallaby | tripleo >>>> >>>> I am trying to setup DCN where i want to setup a glance multistore config in both central and edge sites as per the documentation. >>>> But when ever i run stack update with the following config for multistore: >>>> >>>> [stack@hkg2director ~]$ cat workplace/glance.yaml >>>> parameter_defaults: >>>> GlanceShowMultipleLocations: true >>>> GlanceBackend: rbd >>>> GlanceEnabledImportMethods: web-download,copy-image >>>> GlanceBackendID: ceph >>>> GlanceMultistoreConfig: >>>> dcn01: >>>> GlanceBackend: rbd >>>> GlanceStoreDescription: 'dcn01 rbd glance store' >>>> CephClusterName: dcn01 >>>> >>>> The image service becomes inaccessible. Giving the following error: >>>> >>>> (overcloud) [stack@hkg2director ~]$ openstack image list >>>> Failed to contact the endpoint at https://bolt.bdxworld.com:13292 for discovery. Fallback to using that endpoint as the base url. >>>> Failed to contact the endpoint at https://bolt.bdxworld.com:13292 for discovery. Fallback to using that endpoint as the base url. >>>> The image service for :regionOne exists but does not have any supported versions. >>>> >>>> >>>> glance api.log from controller nodes: >>>> >>>> 2023-02-13 13:02:19.878 7 INFO glance.async_ [-] Threadpool model set to 'EventletThreadPoolModel' >>>> 2023-02-13 13:02:19.969 7 INFO keyring.backend [-] Loading KWallet >>>> 2023-02-13 13:02:19.977 7 INFO keyring.backend [-] Loading SecretService >>>> 2023-02-13 13:02:19.982 7 INFO keyring.backend [-] Loading Windows >>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading chainer >>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading macOS >>>> 2023-02-13 13:02:20.571 7 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True. >>>> 2023-02-13 13:02:20.711 7 ERROR stevedore.extension [-] Could not load 'glance.store.s3.Store': No module named 'boto3': ModuleNotFoundError: No module named 'boto3' >>>> 2023-02-13 13:02:20.719 7 ERROR stevedore.extension [-] Could not load 's3': No module named 'boto3': ModuleNotFoundError: No module named 'boto3' >>>> >>>> Is there a patch to fix this issue??
Hi, I am currently working on client approval to run the stack update. I will share the logs shortly. With regards, Swogat Pradhan On Tue, Feb 21, 2023 at 7:58 PM John Fulton <johfulto@redhat.com> wrote:
I agree with Pranali and Takashi.
I have also seen "No module named 'boto3'" in my logs while setting up DCN with glance successfully. So that is not the cause of your fatal error. You should look in your logs for something else as the root cause of this issue.
John
On Tue, Feb 21, 2023 at 2:20 AM Takashi Kajinami <tkajinam@redhat.com> wrote:
I looked at tcib definitions but we have never added boto3 even in
master.
However, according to the glance_store code, I expect glance_store would
be able to ignore the error
even if boto3 is not enabled unless you enable s3 backend somehow.
https://github.com/openstack/glance_store/blob/64e25979a904e468e1e89d8727e95...
If you can find the log line "Failed to load driver ..." in
glance-api.log then the cause would be different
and we have to check the full logs(both /var/log/containers/glance/glance-api.log and probably /var/log/containers/sdtouts/glance_api.log) to find out what is actually causing the crash.
On Tue, Feb 21, 2023 at 4:05 PM Pranali Deore <pdeore@redhat.com> wrote:
This doesn't seem like a fatal error, so the issue might be something
else.
What podman logs glance_api shows?
On Tue, Feb 21, 2023 at 12:14 PM Swogat Pradhan < swogatpradhan22@gmail.com> wrote:
Hi Abhishek, Yes, this feels like the same bug, as per the comment from Lars
Wiegman (lars-fuga) my glance api container is in reboot loop, when glance multi store is configured.
So I have been searching for a patch for tripleo.
With regards, Swogat Pradhan
On Tue, Feb 21, 2023 at 12:09 PM Abhishek Kekane <akekane@redhat.com>
wrote:
Hi Swogat,
I think this looks relevant, kindly go through this and confirm. https://bugs.launchpad.net/kolla/+bug/1884259
Thanks & Best Regards,
Abhishek Kekane
On Tue, Feb 21, 2023 at 11:58 AM Swogat Pradhan <
swogatpradhan22@gmail.com> wrote:
Hi pranali, Those configs('central', 'cinder_store') were added when i had
started deploying the cloud, those parameters have been removed from the yaml now, but exist in the config file.
Right now i have reverted the config so i am not getting any erros in the glance api logs. But when i configure the multistore i get the following errors on both sides (central and edge) in glance_api.log: 2023-02-13 13:02:19.878 7 INFO glance.async_ [-] Threadpool model set to 'EventletThreadPoolModel' 2023-02-13 13:02:19.969 7 INFO keyring.backend [-] Loading KWallet 2023-02-13 13:02:19.977 7 INFO keyring.backend [-] Loading SecretService 2023-02-13 13:02:19.982 7 INFO keyring.backend [-] Loading Windows 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading chainer 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading macOS 2023-02-13 13:02:20.571 7 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True. 2023-02-13 13:02:20.711 7 ERROR stevedore.extension [-] Could not load 'glance.store.s3.Store': No module named 'boto3': ModuleNotFoundError: No module named 'boto3' 2023-02-13 13:02:20.719 7 ERROR stevedore.extension [-] Could not load 's3': No module named 'boto3': ModuleNotFoundError: No module named 'boto3'
I get no error whatsoever other than the above.
With regards, Swogat Pradhan
On Tue, Feb 21, 2023 at 11:44 AM Pranali Deore <pdeore@redhat.com> wrote: > > Hey Swogat, > > By looking at the conf files you've shared, not sure from where the extra store sections 'central', 'cinder_store' are added since you have not added those in any of your yaml files. > Could you please share glance-api logs from both the nodes to check more details? > Probably the HA configuration is causing this kind of issue but need more information related to deployment & log to understand the root cause. > > > > Thanks, > ~Pranali > > > On Mon, Feb 20, 2023 at 4:45 PM Swogat Pradhan < swogatpradhan22@gmail.com> wrote: >> >> Hi Pranali, >> Thank you for your response. >> The glance.yaml i had shared is for the central site. >> I am attaching the required files for your reference. >> >> With regards, >> Swogat Pradhan >> >> On Mon, Feb 20, 2023 at 4:29 PM Pranali Deore <pdeore@redhat.com> wrote: >>> >>> Hi Swogat, >>> >>> The glance.yaml you've shared is for central site or edge site? >>> Is it possible for you to share your glance-api.conf just to check how configuration values are set? >>> >>> >>> >>> Thanks, >>> Pranali >>> >>> On Tue, Feb 14, 2023 at 12:35 PM Swogat Pradhan < swogatpradhan22@gmail.com> wrote: >>>> >>>> Additional INFO: >>>> The glance_api container is in reboot loop after adding multi store config. >>>> Which is why the image service has become unreachable. >>>> >>>> >>>> On Mon, Feb 13, 2023 at 6:39 PM Swogat Pradhan < swogatpradhan22@gmail.com> wrote: >>>>> >>>>> Hi, >>>>> Openstack wallaby | tripleo >>>>> >>>>> I am trying to setup DCN where i want to setup a glance multistore config in both central and edge sites as per the documentation. >>>>> But when ever i run stack update with the following config for multistore: >>>>> >>>>> [stack@hkg2director ~]$ cat workplace/glance.yaml >>>>> parameter_defaults: >>>>> GlanceShowMultipleLocations: true >>>>> GlanceBackend: rbd >>>>> GlanceEnabledImportMethods: web-download,copy-image >>>>> GlanceBackendID: ceph >>>>> GlanceMultistoreConfig: >>>>> dcn01: >>>>> GlanceBackend: rbd >>>>> GlanceStoreDescription: 'dcn01 rbd glance store' >>>>> CephClusterName: dcn01 >>>>> >>>>> The image service becomes inaccessible. Giving the following error: >>>>> >>>>> (overcloud) [stack@hkg2director ~]$ openstack image list >>>>> Failed to contact the endpoint at https://bolt.bdxworld.com:13292 for discovery. Fallback to using that endpoint as the base url. >>>>> Failed to contact the endpoint at https://bolt.bdxworld.com:13292 for discovery. Fallback to using that endpoint as the base url. >>>>> The image service for :regionOne exists but does not have any supported versions. >>>>> >>>>> >>>>> glance api.log from controller nodes: >>>>> >>>>> 2023-02-13 13:02:19.878 7 INFO glance.async_ [-] Threadpool model set to 'EventletThreadPoolModel' >>>>> 2023-02-13 13:02:19.969 7 INFO keyring.backend [-] Loading KWallet >>>>> 2023-02-13 13:02:19.977 7 INFO keyring.backend [-] Loading SecretService >>>>> 2023-02-13 13:02:19.982 7 INFO keyring.backend [-] Loading Windows >>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading chainer >>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading macOS >>>>> 2023-02-13 13:02:20.571 7 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True. >>>>> 2023-02-13 13:02:20.711 7 ERROR stevedore.extension [-] Could not load 'glance.store.s3.Store': No module named 'boto3': ModuleNotFoundError: No module named 'boto3' >>>>> 2023-02-13 13:02:20.719 7 ERROR stevedore.extension [-] Could not load 's3': No module named 'boto3': ModuleNotFoundError: No module named 'boto3' >>>>> >>>>> Is there a patch to fix this issue??
Hi All, I have set up glance multi store in the central site and the same issue is present. Attaching the logs. With regards, Swogat Pradhan On Tue, Feb 21, 2023 at 10:04 PM Swogat Pradhan <swogatpradhan22@gmail.com> wrote:
Hi, I am currently working on client approval to run the stack update. I will share the logs shortly.
With regards, Swogat Pradhan
On Tue, Feb 21, 2023 at 7:58 PM John Fulton <johfulto@redhat.com> wrote:
I agree with Pranali and Takashi.
I have also seen "No module named 'boto3'" in my logs while setting up DCN with glance successfully. So that is not the cause of your fatal error. You should look in your logs for something else as the root cause of this issue.
John
On Tue, Feb 21, 2023 at 2:20 AM Takashi Kajinami <tkajinam@redhat.com> wrote:
I looked at tcib definitions but we have never added boto3 even in
master.
However, according to the glance_store code, I expect glance_store
would be able to ignore the error
even if boto3 is not enabled unless you enable s3 backend somehow.
https://github.com/openstack/glance_store/blob/64e25979a904e468e1e89d8727e95...
If you can find the log line "Failed to load driver ..." in
and we have to check the full logs(both /var/log/containers/glance/glance-api.log and probably /var/log/containers/sdtouts/glance_api.log) to find out what is actually causing the crash.
On Tue, Feb 21, 2023 at 4:05 PM Pranali Deore <pdeore@redhat.com> wrote:
This doesn't seem like a fatal error, so the issue might be something
else.
What podman logs glance_api shows?
On Tue, Feb 21, 2023 at 12:14 PM Swogat Pradhan < swogatpradhan22@gmail.com> wrote:
Hi Abhishek, Yes, this feels like the same bug, as per the comment from Lars
Wiegman (lars-fuga) my glance api container is in reboot loop, when glance multi store is configured.
So I have been searching for a patch for tripleo.
With regards, Swogat Pradhan
On Tue, Feb 21, 2023 at 12:09 PM Abhishek Kekane <akekane@redhat.com>
wrote:
Hi Swogat,
I think this looks relevant, kindly go through this and confirm. https://bugs.launchpad.net/kolla/+bug/1884259
Thanks & Best Regards,
Abhishek Kekane
On Tue, Feb 21, 2023 at 11:58 AM Swogat Pradhan <
swogatpradhan22@gmail.com> wrote:
> > Hi pranali, > Those configs('central', 'cinder_store') were added when i had started deploying the cloud, those parameters have been removed from the yaml now, but exist in the config file. > Right now i have reverted the config so i am not getting any erros in the glance api logs. > But when i configure the multistore i get the following errors on both sides (central and edge) in glance_api.log: > 2023-02-13 13:02:19.878 7 INFO glance.async_ [-] Threadpool model set to 'EventletThreadPoolModel' > 2023-02-13 13:02:19.969 7 INFO keyring.backend [-] Loading KWallet > 2023-02-13 13:02:19.977 7 INFO keyring.backend [-] Loading SecretService > 2023-02-13 13:02:19.982 7 INFO keyring.backend [-] Loading Windows > 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading chainer > 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading macOS > 2023-02-13 13:02:20.571 7 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True. > 2023-02-13 13:02:20.711 7 ERROR stevedore.extension [-] Could not load 'glance.store.s3.Store': No module named 'boto3': ModuleNotFoundError: No module named 'boto3' > 2023-02-13 13:02:20.719 7 ERROR stevedore.extension [-] Could not load 's3': No module named 'boto3': ModuleNotFoundError: No module named 'boto3' > > I get no error whatsoever other than the above. > > With regards, > Swogat Pradhan > > On Tue, Feb 21, 2023 at 11:44 AM Pranali Deore <pdeore@redhat.com> wrote: >> >> Hey Swogat, >> >> By looking at the conf files you've shared, not sure from where
glance-api.log then the cause would be different the extra store sections 'central', 'cinder_store' are added since you have not added those in any of your yaml files.
>> Could you please share glance-api logs from both the nodes to check more details? >> Probably the HA configuration is causing this kind of issue but need more information related to deployment & log to understand the root cause. >> >> >> >> Thanks, >> ~Pranali >> >> >> On Mon, Feb 20, 2023 at 4:45 PM Swogat Pradhan < swogatpradhan22@gmail.com> wrote: >>> >>> Hi Pranali, >>> Thank you for your response. >>> The glance.yaml i had shared is for the central site. >>> I am attaching the required files for your reference. >>> >>> With regards, >>> Swogat Pradhan >>> >>> On Mon, Feb 20, 2023 at 4:29 PM Pranali Deore <pdeore@redhat.com> wrote: >>>> >>>> Hi Swogat, >>>> >>>> The glance.yaml you've shared is for central site or edge site? >>>> Is it possible for you to share your glance-api.conf just to check how configuration values are set? >>>> >>>> >>>> >>>> Thanks, >>>> Pranali >>>> >>>> On Tue, Feb 14, 2023 at 12:35 PM Swogat Pradhan < swogatpradhan22@gmail.com> wrote: >>>>> >>>>> Additional INFO: >>>>> The glance_api container is in reboot loop after adding multi store config. >>>>> Which is why the image service has become unreachable. >>>>> >>>>> >>>>> On Mon, Feb 13, 2023 at 6:39 PM Swogat Pradhan < swogatpradhan22@gmail.com> wrote: >>>>>> >>>>>> Hi, >>>>>> Openstack wallaby | tripleo >>>>>> >>>>>> I am trying to setup DCN where i want to setup a glance multistore config in both central and edge sites as per the documentation. >>>>>> But when ever i run stack update with the following config for multistore: >>>>>> >>>>>> [stack@hkg2director ~]$ cat workplace/glance.yaml >>>>>> parameter_defaults: >>>>>> GlanceShowMultipleLocations: true >>>>>> GlanceBackend: rbd >>>>>> GlanceEnabledImportMethods: web-download,copy-image >>>>>> GlanceBackendID: ceph >>>>>> GlanceMultistoreConfig: >>>>>> dcn01: >>>>>> GlanceBackend: rbd >>>>>> GlanceStoreDescription: 'dcn01 rbd glance store' >>>>>> CephClusterName: dcn01 >>>>>> >>>>>> The image service becomes inaccessible. Giving the following error: >>>>>> >>>>>> (overcloud) [stack@hkg2director ~]$ openstack image list >>>>>> Failed to contact the endpoint at https://bolt.bdxworld.com:13292 for discovery. Fallback to using that endpoint as the base url. >>>>>> Failed to contact the endpoint at https://bolt.bdxworld.com:13292 for discovery. Fallback to using that endpoint as the base url. >>>>>> The image service for :regionOne exists but does not have any supported versions. >>>>>> >>>>>> >>>>>> glance api.log from controller nodes: >>>>>> >>>>>> 2023-02-13 13:02:19.878 7 INFO glance.async_ [-] Threadpool model set to 'EventletThreadPoolModel' >>>>>> 2023-02-13 13:02:19.969 7 INFO keyring.backend [-] Loading KWallet >>>>>> 2023-02-13 13:02:19.977 7 INFO keyring.backend [-] Loading SecretService >>>>>> 2023-02-13 13:02:19.982 7 INFO keyring.backend [-] Loading Windows >>>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading chainer >>>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading macOS >>>>>> 2023-02-13 13:02:20.571 7 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True. >>>>>> 2023-02-13 13:02:20.711 7 ERROR stevedore.extension [-] Could not load 'glance.store.s3.Store': No module named 'boto3': ModuleNotFoundError: No module named 'boto3' >>>>>> 2023-02-13 13:02:20.719 7 ERROR stevedore.extension [-] Could not load 's3': No module named 'boto3': ModuleNotFoundError: No module named 'boto3' >>>>>> >>>>>> Is there a patch to fix this issue??
Attaching podman logs glance_api. With regards, Swogat Pradhan On Wed, Feb 22, 2023 at 12:46 PM Swogat Pradhan <swogatpradhan22@gmail.com> wrote:
Hi All, I have set up glance multi store in the central site and the same issue is present. Attaching the logs.
With regards, Swogat Pradhan
On Tue, Feb 21, 2023 at 10:04 PM Swogat Pradhan <swogatpradhan22@gmail.com> wrote:
Hi, I am currently working on client approval to run the stack update. I will share the logs shortly.
With regards, Swogat Pradhan
On Tue, Feb 21, 2023 at 7:58 PM John Fulton <johfulto@redhat.com> wrote:
I agree with Pranali and Takashi.
I have also seen "No module named 'boto3'" in my logs while setting up DCN with glance successfully. So that is not the cause of your fatal error. You should look in your logs for something else as the root cause of this issue.
John
On Tue, Feb 21, 2023 at 2:20 AM Takashi Kajinami <tkajinam@redhat.com> wrote:
I looked at tcib definitions but we have never added boto3 even in
master.
However, according to the glance_store code, I expect glance_store
would be able to ignore the error
even if boto3 is not enabled unless you enable s3 backend somehow.
https://github.com/openstack/glance_store/blob/64e25979a904e468e1e89d8727e95...
If you can find the log line "Failed to load driver ..." in
and we have to check the full logs(both /var/log/containers/glance/glance-api.log and probably /var/log/containers/sdtouts/glance_api.log) to find out what is actually causing the crash.
On Tue, Feb 21, 2023 at 4:05 PM Pranali Deore <pdeore@redhat.com> wrote:
This doesn't seem like a fatal error, so the issue might be something
else.
What podman logs glance_api shows?
On Tue, Feb 21, 2023 at 12:14 PM Swogat Pradhan < swogatpradhan22@gmail.com> wrote:
Hi Abhishek, Yes, this feels like the same bug, as per the comment from Lars
Wiegman (lars-fuga) my glance api container is in reboot loop, when glance multi store is configured.
So I have been searching for a patch for tripleo.
With regards, Swogat Pradhan
On Tue, Feb 21, 2023 at 12:09 PM Abhishek Kekane <akekane@redhat.com>
wrote:
> > Hi Swogat, > > I think this looks relevant, kindly go through this and confirm. > https://bugs.launchpad.net/kolla/+bug/1884259 > > > Thanks & Best Regards, > > Abhishek Kekane > > > On Tue, Feb 21, 2023 at 11:58 AM Swogat Pradhan < swogatpradhan22@gmail.com> wrote: >> >> Hi pranali, >> Those configs('central', 'cinder_store') were added when i had started deploying the cloud, those parameters have been removed from the yaml now, but exist in the config file. >> Right now i have reverted the config so i am not getting any erros in the glance api logs. >> But when i configure the multistore i get the following errors on both sides (central and edge) in glance_api.log: >> 2023-02-13 13:02:19.878 7 INFO glance.async_ [-] Threadpool model set to 'EventletThreadPoolModel' >> 2023-02-13 13:02:19.969 7 INFO keyring.backend [-] Loading KWallet >> 2023-02-13 13:02:19.977 7 INFO keyring.backend [-] Loading SecretService >> 2023-02-13 13:02:19.982 7 INFO keyring.backend [-] Loading Windows >> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading chainer >> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading macOS >> 2023-02-13 13:02:20.571 7 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True. >> 2023-02-13 13:02:20.711 7 ERROR stevedore.extension [-] Could not load 'glance.store.s3.Store': No module named 'boto3': ModuleNotFoundError: No module named 'boto3' >> 2023-02-13 13:02:20.719 7 ERROR stevedore.extension [-] Could not load 's3': No module named 'boto3': ModuleNotFoundError: No module named 'boto3' >> >> I get no error whatsoever other than the above. >> >> With regards, >> Swogat Pradhan >> >> On Tue, Feb 21, 2023 at 11:44 AM Pranali Deore <pdeore@redhat.com> wrote: >>> >>> Hey Swogat, >>> >>> By looking at the conf files you've shared, not sure from where
glance-api.log then the cause would be different the extra store sections 'central', 'cinder_store' are added since you have not added those in any of your yaml files.
>>> Could you please share glance-api logs from both the nodes to check more details? >>> Probably the HA configuration is causing this kind of issue but need more information related to deployment & log to understand the root cause. >>> >>> >>> >>> Thanks, >>> ~Pranali >>> >>> >>> On Mon, Feb 20, 2023 at 4:45 PM Swogat Pradhan < swogatpradhan22@gmail.com> wrote: >>>> >>>> Hi Pranali, >>>> Thank you for your response. >>>> The glance.yaml i had shared is for the central site. >>>> I am attaching the required files for your reference. >>>> >>>> With regards, >>>> Swogat Pradhan >>>> >>>> On Mon, Feb 20, 2023 at 4:29 PM Pranali Deore <pdeore@redhat.com> wrote: >>>>> >>>>> Hi Swogat, >>>>> >>>>> The glance.yaml you've shared is for central site or edge site? >>>>> Is it possible for you to share your glance-api.conf just to check how configuration values are set? >>>>> >>>>> >>>>> >>>>> Thanks, >>>>> Pranali >>>>> >>>>> On Tue, Feb 14, 2023 at 12:35 PM Swogat Pradhan < swogatpradhan22@gmail.com> wrote: >>>>>> >>>>>> Additional INFO: >>>>>> The glance_api container is in reboot loop after adding multi store config. >>>>>> Which is why the image service has become unreachable. >>>>>> >>>>>> >>>>>> On Mon, Feb 13, 2023 at 6:39 PM Swogat Pradhan < swogatpradhan22@gmail.com> wrote: >>>>>>> >>>>>>> Hi, >>>>>>> Openstack wallaby | tripleo >>>>>>> >>>>>>> I am trying to setup DCN where i want to setup a glance multistore config in both central and edge sites as per the documentation. >>>>>>> But when ever i run stack update with the following config for multistore: >>>>>>> >>>>>>> [stack@hkg2director ~]$ cat workplace/glance.yaml >>>>>>> parameter_defaults: >>>>>>> GlanceShowMultipleLocations: true >>>>>>> GlanceBackend: rbd >>>>>>> GlanceEnabledImportMethods: web-download,copy-image >>>>>>> GlanceBackendID: ceph >>>>>>> GlanceMultistoreConfig: >>>>>>> dcn01: >>>>>>> GlanceBackend: rbd >>>>>>> GlanceStoreDescription: 'dcn01 rbd glance store' >>>>>>> CephClusterName: dcn01 >>>>>>> >>>>>>> The image service becomes inaccessible. Giving the following error: >>>>>>> >>>>>>> (overcloud) [stack@hkg2director ~]$ openstack image list >>>>>>> Failed to contact the endpoint at https://bolt.bdxworld.com:13292 for discovery. Fallback to using that endpoint as the base url. >>>>>>> Failed to contact the endpoint at https://bolt.bdxworld.com:13292 for discovery. Fallback to using that endpoint as the base url. >>>>>>> The image service for :regionOne exists but does not have any supported versions. >>>>>>> >>>>>>> >>>>>>> glance api.log from controller nodes: >>>>>>> >>>>>>> 2023-02-13 13:02:19.878 7 INFO glance.async_ [-] Threadpool model set to 'EventletThreadPoolModel' >>>>>>> 2023-02-13 13:02:19.969 7 INFO keyring.backend [-] Loading KWallet >>>>>>> 2023-02-13 13:02:19.977 7 INFO keyring.backend [-] Loading SecretService >>>>>>> 2023-02-13 13:02:19.982 7 INFO keyring.backend [-] Loading Windows >>>>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading chainer >>>>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading macOS >>>>>>> 2023-02-13 13:02:20.571 7 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True. >>>>>>> 2023-02-13 13:02:20.711 7 ERROR stevedore.extension [-] Could not load 'glance.store.s3.Store': No module named 'boto3': ModuleNotFoundError: No module named 'boto3' >>>>>>> 2023-02-13 13:02:20.719 7 ERROR stevedore.extension [-] Could not load 's3': No module named 'boto3': ModuleNotFoundError: No module named 'boto3' >>>>>>> >>>>>>> Is there a patch to fix this issue??
I can see RADOS not found error multiple times in podman_logs. 2023-02-22T07:01:01.639404765+00:00 stderr F + echo 'Running command: '\''/usr/bin/glance-api --config-file /etc/glance/glance-api.conf --config-file /etc/glance/glance-image-import.conf'\''' 2023-02-22T07:01:01.639420550+00:00 stdout F Running command: '/usr/bin/glance-api --config-file /etc/glance/glance-api.conf --config-file /etc/glance/glance-image-import.conf' 2023-02-22T07:01:01.639464013+00:00 stderr F + exec /usr/bin/glance-api --config-file /etc/glance/glance-api.conf --config-file /etc/glance/glance-image-import.conf 2023-02-22T07:01:03.662036516+00:00 stderr F ERROR: [errno 2] RADOS object not found (error calling conf_read_file) I think this might be the reason glance-api is stuck in a recursive loop. Thanks & Best Regards, Abhishek Kekane On Wed, Feb 22, 2023 at 12:57 PM Swogat Pradhan <swogatpradhan22@gmail.com> wrote:
Attaching podman logs glance_api.
With regards, Swogat Pradhan
On Wed, Feb 22, 2023 at 12:46 PM Swogat Pradhan <swogatpradhan22@gmail.com> wrote:
Hi All, I have set up glance multi store in the central site and the same issue is present. Attaching the logs.
With regards, Swogat Pradhan
On Tue, Feb 21, 2023 at 10:04 PM Swogat Pradhan < swogatpradhan22@gmail.com> wrote:
Hi, I am currently working on client approval to run the stack update. I will share the logs shortly.
With regards, Swogat Pradhan
On Tue, Feb 21, 2023 at 7:58 PM John Fulton <johfulto@redhat.com> wrote:
I agree with Pranali and Takashi.
I have also seen "No module named 'boto3'" in my logs while setting up DCN with glance successfully. So that is not the cause of your fatal error. You should look in your logs for something else as the root cause of this issue.
John
On Tue, Feb 21, 2023 at 2:20 AM Takashi Kajinami <tkajinam@redhat.com> wrote:
I looked at tcib definitions but we have never added boto3 even in
master.
However, according to the glance_store code, I expect glance_store
would be able to ignore the error
even if boto3 is not enabled unless you enable s3 backend somehow.
https://github.com/openstack/glance_store/blob/64e25979a904e468e1e89d8727e95...
If you can find the log line "Failed to load driver ..." in
and we have to check the full logs(both /var/log/containers/glance/glance-api.log and probably /var/log/containers/sdtouts/glance_api.log) to find out what is actually causing the crash.
On Tue, Feb 21, 2023 at 4:05 PM Pranali Deore <pdeore@redhat.com> wrote:
This doesn't seem like a fatal error, so the issue might be
something else.
What podman logs glance_api shows?
On Tue, Feb 21, 2023 at 12:14 PM Swogat Pradhan < swogatpradhan22@gmail.com> wrote: > > Hi Abhishek, > Yes, this feels like the same bug, as per the comment from Lars Wiegman (lars-fuga) my glance api container is in reboot loop, when glance multi store is configured. > > So I have been searching for a patch for tripleo. > > With regards, > Swogat Pradhan > > On Tue, Feb 21, 2023 at 12:09 PM Abhishek Kekane < akekane@redhat.com> wrote: >> >> Hi Swogat, >> >> I think this looks relevant, kindly go through this and confirm. >> https://bugs.launchpad.net/kolla/+bug/1884259 >> >> >> Thanks & Best Regards, >> >> Abhishek Kekane >> >> >> On Tue, Feb 21, 2023 at 11:58 AM Swogat Pradhan < swogatpradhan22@gmail.com> wrote: >>> >>> Hi pranali, >>> Those configs('central', 'cinder_store') were added when i had started deploying the cloud, those parameters have been removed from the yaml now, but exist in the config file. >>> Right now i have reverted the config so i am not getting any erros in the glance api logs. >>> But when i configure the multistore i get the following errors on both sides (central and edge) in glance_api.log: >>> 2023-02-13 13:02:19.878 7 INFO glance.async_ [-] Threadpool model set to 'EventletThreadPoolModel' >>> 2023-02-13 13:02:19.969 7 INFO keyring.backend [-] Loading KWallet >>> 2023-02-13 13:02:19.977 7 INFO keyring.backend [-] Loading SecretService >>> 2023-02-13 13:02:19.982 7 INFO keyring.backend [-] Loading Windows >>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading chainer >>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading macOS >>> 2023-02-13 13:02:20.571 7 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True. >>> 2023-02-13 13:02:20.711 7 ERROR stevedore.extension [-] Could not load 'glance.store.s3.Store': No module named 'boto3': ModuleNotFoundError: No module named 'boto3' >>> 2023-02-13 13:02:20.719 7 ERROR stevedore.extension [-] Could not load 's3': No module named 'boto3': ModuleNotFoundError: No module named 'boto3' >>> >>> I get no error whatsoever other than the above. >>> >>> With regards, >>> Swogat Pradhan >>> >>> On Tue, Feb 21, 2023 at 11:44 AM Pranali Deore <pdeore@redhat.com> wrote: >>>> >>>> Hey Swogat, >>>> >>>> By looking at the conf files you've shared, not sure from where
>>>> Could you please share glance-api logs from both the nodes to check more details? >>>> Probably the HA configuration is causing this kind of issue but need more information related to deployment & log to understand the root cause. >>>> >>>> >>>> >>>> Thanks, >>>> ~Pranali >>>> >>>> >>>> On Mon, Feb 20, 2023 at 4:45 PM Swogat Pradhan < swogatpradhan22@gmail.com> wrote: >>>>> >>>>> Hi Pranali, >>>>> Thank you for your response. >>>>> The glance.yaml i had shared is for the central site. >>>>> I am attaching the required files for your reference. >>>>> >>>>> With regards, >>>>> Swogat Pradhan >>>>> >>>>> On Mon, Feb 20, 2023 at 4:29 PM Pranali Deore <
glance-api.log then the cause would be different the extra store sections 'central', 'cinder_store' are added since you have not added those in any of your yaml files. pdeore@redhat.com> wrote:
>>>>>> >>>>>> Hi Swogat, >>>>>> >>>>>> The glance.yaml you've shared is for central site or edge site? >>>>>> Is it possible for you to share your glance-api.conf just to check how configuration values are set? >>>>>> >>>>>> >>>>>> >>>>>> Thanks, >>>>>> Pranali >>>>>> >>>>>> On Tue, Feb 14, 2023 at 12:35 PM Swogat Pradhan < swogatpradhan22@gmail.com> wrote: >>>>>>> >>>>>>> Additional INFO: >>>>>>> The glance_api container is in reboot loop after adding multi store config. >>>>>>> Which is why the image service has become unreachable. >>>>>>> >>>>>>> >>>>>>> On Mon, Feb 13, 2023 at 6:39 PM Swogat Pradhan < swogatpradhan22@gmail.com> wrote: >>>>>>>> >>>>>>>> Hi, >>>>>>>> Openstack wallaby | tripleo >>>>>>>> >>>>>>>> I am trying to setup DCN where i want to setup a glance multistore config in both central and edge sites as per the documentation. >>>>>>>> But when ever i run stack update with the following config for multistore: >>>>>>>> >>>>>>>> [stack@hkg2director ~]$ cat workplace/glance.yaml >>>>>>>> parameter_defaults: >>>>>>>> GlanceShowMultipleLocations: true >>>>>>>> GlanceBackend: rbd >>>>>>>> GlanceEnabledImportMethods: web-download,copy-image >>>>>>>> GlanceBackendID: ceph >>>>>>>> GlanceMultistoreConfig: >>>>>>>> dcn01: >>>>>>>> GlanceBackend: rbd >>>>>>>> GlanceStoreDescription: 'dcn01 rbd glance store' >>>>>>>> CephClusterName: dcn01 >>>>>>>> >>>>>>>> The image service becomes inaccessible. Giving the following error: >>>>>>>> >>>>>>>> (overcloud) [stack@hkg2director ~]$ openstack image list >>>>>>>> Failed to contact the endpoint at https://bolt.bdxworld.com:13292 for discovery. Fallback to using that endpoint as the base url. >>>>>>>> Failed to contact the endpoint at https://bolt.bdxworld.com:13292 for discovery. Fallback to using that endpoint as the base url. >>>>>>>> The image service for :regionOne exists but does not have any supported versions. >>>>>>>> >>>>>>>> >>>>>>>> glance api.log from controller nodes: >>>>>>>> >>>>>>>> 2023-02-13 13:02:19.878 7 INFO glance.async_ [-] Threadpool model set to 'EventletThreadPoolModel' >>>>>>>> 2023-02-13 13:02:19.969 7 INFO keyring.backend [-] Loading KWallet >>>>>>>> 2023-02-13 13:02:19.977 7 INFO keyring.backend [-] Loading SecretService >>>>>>>> 2023-02-13 13:02:19.982 7 INFO keyring.backend [-] Loading Windows >>>>>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading chainer >>>>>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading macOS >>>>>>>> 2023-02-13 13:02:20.571 7 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True. >>>>>>>> 2023-02-13 13:02:20.711 7 ERROR stevedore.extension [-] Could not load 'glance.store.s3.Store': No module named 'boto3': ModuleNotFoundError: No module named 'boto3' >>>>>>>> 2023-02-13 13:02:20.719 7 ERROR stevedore.extension [-] Could not load 's3': No module named 'boto3': ModuleNotFoundError: No module named 'boto3' >>>>>>>> >>>>>>>> Is there a patch to fix this issue??
Hi Abhishek, I cannot understand why this error is coming though 'ERROR: [errno 2] RADOS object not found (error calling conf_read_file)' In controller 1 and Edge 1 nodes i have the config files of both sides along with the openstack and admin keyring files. And in other controllers and edge nodes i have config files and openstack keyring of other site and admin keyring of the same site. When i am trying to connect using cephadm shell --config ceph.conf / dcn01.conf --keyring <admin keyring> --fsid <central/edge> i am able to connect to the cluster and issue commands as well. With regards, Swogat Pradhan On Wed, Feb 22, 2023 at 1:02 PM Abhishek Kekane <akekane@redhat.com> wrote:
I can see RADOS not found error multiple times in podman_logs.
2023-02-22T07:01:01.639404765+00:00 stderr F + echo 'Running command: '\''/usr/bin/glance-api --config-file /etc/glance/glance-api.conf --config-file /etc/glance/glance-image-import.conf'\''' 2023-02-22T07:01:01.639420550+00:00 stdout F Running command: '/usr/bin/glance-api --config-file /etc/glance/glance-api.conf --config-file /etc/glance/glance-image-import.conf' 2023-02-22T07:01:01.639464013+00:00 stderr F + exec /usr/bin/glance-api --config-file /etc/glance/glance-api.conf --config-file /etc/glance/glance-image-import.conf 2023-02-22T07:01:03.662036516+00:00 stderr F ERROR: [errno 2] RADOS object not found (error calling conf_read_file)
I think this might be the reason glance-api is stuck in a recursive loop.
Thanks & Best Regards,
Abhishek Kekane
On Wed, Feb 22, 2023 at 12:57 PM Swogat Pradhan <swogatpradhan22@gmail.com> wrote:
Attaching podman logs glance_api.
With regards, Swogat Pradhan
On Wed, Feb 22, 2023 at 12:46 PM Swogat Pradhan < swogatpradhan22@gmail.com> wrote:
Hi All, I have set up glance multi store in the central site and the same issue is present. Attaching the logs.
With regards, Swogat Pradhan
On Tue, Feb 21, 2023 at 10:04 PM Swogat Pradhan < swogatpradhan22@gmail.com> wrote:
Hi, I am currently working on client approval to run the stack update. I will share the logs shortly.
With regards, Swogat Pradhan
On Tue, Feb 21, 2023 at 7:58 PM John Fulton <johfulto@redhat.com> wrote:
I agree with Pranali and Takashi.
I have also seen "No module named 'boto3'" in my logs while setting up DCN with glance successfully. So that is not the cause of your fatal error. You should look in your logs for something else as the root cause of this issue.
John
On Tue, Feb 21, 2023 at 2:20 AM Takashi Kajinami <tkajinam@redhat.com> wrote:
I looked at tcib definitions but we have never added boto3 even in
master.
However, according to the glance_store code, I expect glance_store
would be able to ignore the error
even if boto3 is not enabled unless you enable s3 backend somehow.
https://github.com/openstack/glance_store/blob/64e25979a904e468e1e89d8727e95...
If you can find the log line "Failed to load driver ..." in
and we have to check the full logs(both /var/log/containers/glance/glance-api.log and probably /var/log/containers/sdtouts/glance_api.log) to find out what is actually causing the crash.
On Tue, Feb 21, 2023 at 4:05 PM Pranali Deore <pdeore@redhat.com> wrote: > > This doesn't seem like a fatal error, so the issue might be something else. > What podman logs glance_api shows? > > On Tue, Feb 21, 2023 at 12:14 PM Swogat Pradhan < swogatpradhan22@gmail.com> wrote: >> >> Hi Abhishek, >> Yes, this feels like the same bug, as per the comment from Lars Wiegman (lars-fuga) my glance api container is in reboot loop, when glance multi store is configured. >> >> So I have been searching for a patch for tripleo. >> >> With regards, >> Swogat Pradhan >> >> On Tue, Feb 21, 2023 at 12:09 PM Abhishek Kekane < akekane@redhat.com> wrote: >>> >>> Hi Swogat, >>> >>> I think this looks relevant, kindly go through this and confirm. >>> https://bugs.launchpad.net/kolla/+bug/1884259 >>> >>> >>> Thanks & Best Regards, >>> >>> Abhishek Kekane >>> >>> >>> On Tue, Feb 21, 2023 at 11:58 AM Swogat Pradhan < swogatpradhan22@gmail.com> wrote: >>>> >>>> Hi pranali, >>>> Those configs('central', 'cinder_store') were added when i had started deploying the cloud, those parameters have been removed from the yaml now, but exist in the config file. >>>> Right now i have reverted the config so i am not getting any erros in the glance api logs. >>>> But when i configure the multistore i get the following errors on both sides (central and edge) in glance_api.log: >>>> 2023-02-13 13:02:19.878 7 INFO glance.async_ [-] Threadpool model set to 'EventletThreadPoolModel' >>>> 2023-02-13 13:02:19.969 7 INFO keyring.backend [-] Loading KWallet >>>> 2023-02-13 13:02:19.977 7 INFO keyring.backend [-] Loading SecretService >>>> 2023-02-13 13:02:19.982 7 INFO keyring.backend [-] Loading Windows >>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading chainer >>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading macOS >>>> 2023-02-13 13:02:20.571 7 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True. >>>> 2023-02-13 13:02:20.711 7 ERROR stevedore.extension [-] Could not load 'glance.store.s3.Store': No module named 'boto3': ModuleNotFoundError: No module named 'boto3' >>>> 2023-02-13 13:02:20.719 7 ERROR stevedore.extension [-] Could not load 's3': No module named 'boto3': ModuleNotFoundError: No module named 'boto3' >>>> >>>> I get no error whatsoever other than the above. >>>> >>>> With regards, >>>> Swogat Pradhan >>>> >>>> On Tue, Feb 21, 2023 at 11:44 AM Pranali Deore <
>>>>> >>>>> Hey Swogat, >>>>> >>>>> By looking at the conf files you've shared, not sure from where
>>>>> Could you please share glance-api logs from both the nodes to check more details? >>>>> Probably the HA configuration is causing this kind of issue but need more information related to deployment & log to understand the root cause. >>>>> >>>>> >>>>> >>>>> Thanks, >>>>> ~Pranali >>>>> >>>>> >>>>> On Mon, Feb 20, 2023 at 4:45 PM Swogat Pradhan < swogatpradhan22@gmail.com> wrote: >>>>>> >>>>>> Hi Pranali, >>>>>> Thank you for your response. >>>>>> The glance.yaml i had shared is for the central site. >>>>>> I am attaching the required files for your reference. >>>>>> >>>>>> With regards, >>>>>> Swogat Pradhan >>>>>> >>>>>> On Mon, Feb 20, 2023 at 4:29 PM Pranali Deore <
glance-api.log then the cause would be different pdeore@redhat.com> wrote: the extra store sections 'central', 'cinder_store' are added since you have not added those in any of your yaml files. pdeore@redhat.com> wrote:
>>>>>>> >>>>>>> Hi Swogat, >>>>>>> >>>>>>> The glance.yaml you've shared is for central site or edge site? >>>>>>> Is it possible for you to share your glance-api.conf just to check how configuration values are set? >>>>>>> >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> Pranali >>>>>>> >>>>>>> On Tue, Feb 14, 2023 at 12:35 PM Swogat Pradhan < swogatpradhan22@gmail.com> wrote: >>>>>>>> >>>>>>>> Additional INFO: >>>>>>>> The glance_api container is in reboot loop after adding multi store config. >>>>>>>> Which is why the image service has become unreachable. >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Feb 13, 2023 at 6:39 PM Swogat Pradhan < swogatpradhan22@gmail.com> wrote: >>>>>>>>> >>>>>>>>> Hi, >>>>>>>>> Openstack wallaby | tripleo >>>>>>>>> >>>>>>>>> I am trying to setup DCN where i want to setup a glance multistore config in both central and edge sites as per the documentation. >>>>>>>>> But when ever i run stack update with the following config for multistore: >>>>>>>>> >>>>>>>>> [stack@hkg2director ~]$ cat workplace/glance.yaml >>>>>>>>> parameter_defaults: >>>>>>>>> GlanceShowMultipleLocations: true >>>>>>>>> GlanceBackend: rbd >>>>>>>>> GlanceEnabledImportMethods: web-download,copy-image >>>>>>>>> GlanceBackendID: ceph >>>>>>>>> GlanceMultistoreConfig: >>>>>>>>> dcn01: >>>>>>>>> GlanceBackend: rbd >>>>>>>>> GlanceStoreDescription: 'dcn01 rbd glance store' >>>>>>>>> CephClusterName: dcn01 >>>>>>>>> >>>>>>>>> The image service becomes inaccessible. Giving the following error: >>>>>>>>> >>>>>>>>> (overcloud) [stack@hkg2director ~]$ openstack image list >>>>>>>>> Failed to contact the endpoint at https://bolt.bdxworld.com:13292 for discovery. Fallback to using that endpoint as the base url. >>>>>>>>> Failed to contact the endpoint at https://bolt.bdxworld.com:13292 for discovery. Fallback to using that endpoint as the base url. >>>>>>>>> The image service for :regionOne exists but does not have any supported versions. >>>>>>>>> >>>>>>>>> >>>>>>>>> glance api.log from controller nodes: >>>>>>>>> >>>>>>>>> 2023-02-13 13:02:19.878 7 INFO glance.async_ [-] Threadpool model set to 'EventletThreadPoolModel' >>>>>>>>> 2023-02-13 13:02:19.969 7 INFO keyring.backend [-] Loading KWallet >>>>>>>>> 2023-02-13 13:02:19.977 7 INFO keyring.backend [-] Loading SecretService >>>>>>>>> 2023-02-13 13:02:19.982 7 INFO keyring.backend [-] Loading Windows >>>>>>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading chainer >>>>>>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading macOS >>>>>>>>> 2023-02-13 13:02:20.571 7 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True. >>>>>>>>> 2023-02-13 13:02:20.711 7 ERROR stevedore.extension [-] Could not load 'glance.store.s3.Store': No module named 'boto3': ModuleNotFoundError: No module named 'boto3' >>>>>>>>> 2023-02-13 13:02:20.719 7 ERROR stevedore.extension [-] Could not load 's3': No module named 'boto3': ModuleNotFoundError: No module named 'boto3' >>>>>>>>> >>>>>>>>> Is there a patch to fix this issue??
Sounds like you've confirmed that /etc/ceph/ceph.conf and /etc/ceph/client.openstack.keyring (or similar) are on your container host and functioning correctly. Next check inside the glance container that they are mounted correctly inside of it. Note this parameter and its default: https://github.com/openstack/tripleo-heat-templates/blob/stable/wallaby/depl... Is the glance container mounting this way? /var/lib/tripleo-config/ceph:/etc/ceph If so then check the contents of /var/lib/tripleo-config/ceph on your container host. John On Wed, Feb 22, 2023 at 3:18 AM Swogat Pradhan <swogatpradhan22@gmail.com> wrote:
Hi Abhishek, I cannot understand why this error is coming though 'ERROR: [errno 2] RADOS object not found (error calling conf_read_file)'
In controller 1 and Edge 1 nodes i have the config files of both sides along with the openstack and admin keyring files. And in other controllers and edge nodes i have config files and openstack keyring of other site and admin keyring of the same site.
When i am trying to connect using cephadm shell --config ceph.conf / dcn01.conf --keyring <admin keyring> --fsid <central/edge> i am able to connect to the cluster and issue commands as well.
With regards, Swogat Pradhan
On Wed, Feb 22, 2023 at 1:02 PM Abhishek Kekane <akekane@redhat.com> wrote:
I can see RADOS not found error multiple times in podman_logs.
2023-02-22T07:01:01.639404765+00:00 stderr F + echo 'Running command: '\''/usr/bin/glance-api --config-file /etc/glance/glance-api.conf --config-file /etc/glance/glance-image-import.conf'\''' 2023-02-22T07:01:01.639420550+00:00 stdout F Running command: '/usr/bin/glance-api --config-file /etc/glance/glance-api.conf --config-file /etc/glance/glance-image-import.conf' 2023-02-22T07:01:01.639464013+00:00 stderr F + exec /usr/bin/glance-api --config-file /etc/glance/glance-api.conf --config-file /etc/glance/glance-image-import.conf 2023-02-22T07:01:03.662036516+00:00 stderr F ERROR: [errno 2] RADOS object not found (error calling conf_read_file)
I think this might be the reason glance-api is stuck in a recursive loop.
Thanks & Best Regards,
Abhishek Kekane
On Wed, Feb 22, 2023 at 12:57 PM Swogat Pradhan <swogatpradhan22@gmail.com> wrote:
Attaching podman logs glance_api.
With regards, Swogat Pradhan
On Wed, Feb 22, 2023 at 12:46 PM Swogat Pradhan <swogatpradhan22@gmail.com> wrote:
Hi All, I have set up glance multi store in the central site and the same issue is present. Attaching the logs.
With regards, Swogat Pradhan
On Tue, Feb 21, 2023 at 10:04 PM Swogat Pradhan <swogatpradhan22@gmail.com> wrote:
Hi, I am currently working on client approval to run the stack update. I will share the logs shortly.
With regards, Swogat Pradhan
On Tue, Feb 21, 2023 at 7:58 PM John Fulton <johfulto@redhat.com> wrote:
I agree with Pranali and Takashi.
I have also seen "No module named 'boto3'" in my logs while setting up DCN with glance successfully. So that is not the cause of your fatal error. You should look in your logs for something else as the root cause of this issue.
John
On Tue, Feb 21, 2023 at 2:20 AM Takashi Kajinami <tkajinam@redhat.com> wrote: > > I looked at tcib definitions but we have never added boto3 even in master. > > However, according to the glance_store code, I expect glance_store would be able to ignore the error > even if boto3 is not enabled unless you enable s3 backend somehow. > > https://github.com/openstack/glance_store/blob/64e25979a904e468e1e89d8727e95... > > If you can find the log line "Failed to load driver ..." in glance-api.log then the cause would be different > and we have to check the full logs(both /var/log/containers/glance/glance-api.log and probably > /var/log/containers/sdtouts/glance_api.log) to find out what is actually causing the crash. > > On Tue, Feb 21, 2023 at 4:05 PM Pranali Deore <pdeore@redhat.com> wrote: >> >> This doesn't seem like a fatal error, so the issue might be something else. >> What podman logs glance_api shows? >> >> On Tue, Feb 21, 2023 at 12:14 PM Swogat Pradhan <swogatpradhan22@gmail.com> wrote: >>> >>> Hi Abhishek, >>> Yes, this feels like the same bug, as per the comment from Lars Wiegman (lars-fuga) my glance api container is in reboot loop, when glance multi store is configured. >>> >>> So I have been searching for a patch for tripleo. >>> >>> With regards, >>> Swogat Pradhan >>> >>> On Tue, Feb 21, 2023 at 12:09 PM Abhishek Kekane <akekane@redhat.com> wrote: >>>> >>>> Hi Swogat, >>>> >>>> I think this looks relevant, kindly go through this and confirm. >>>> https://bugs.launchpad.net/kolla/+bug/1884259 >>>> >>>> >>>> Thanks & Best Regards, >>>> >>>> Abhishek Kekane >>>> >>>> >>>> On Tue, Feb 21, 2023 at 11:58 AM Swogat Pradhan <swogatpradhan22@gmail.com> wrote: >>>>> >>>>> Hi pranali, >>>>> Those configs('central', 'cinder_store') were added when i had started deploying the cloud, those parameters have been removed from the yaml now, but exist in the config file. >>>>> Right now i have reverted the config so i am not getting any erros in the glance api logs. >>>>> But when i configure the multistore i get the following errors on both sides (central and edge) in glance_api.log: >>>>> 2023-02-13 13:02:19.878 7 INFO glance.async_ [-] Threadpool model set to 'EventletThreadPoolModel' >>>>> 2023-02-13 13:02:19.969 7 INFO keyring.backend [-] Loading KWallet >>>>> 2023-02-13 13:02:19.977 7 INFO keyring.backend [-] Loading SecretService >>>>> 2023-02-13 13:02:19.982 7 INFO keyring.backend [-] Loading Windows >>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading chainer >>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading macOS >>>>> 2023-02-13 13:02:20.571 7 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True. >>>>> 2023-02-13 13:02:20.711 7 ERROR stevedore.extension [-] Could not load 'glance.store.s3.Store': No module named 'boto3': ModuleNotFoundError: No module named 'boto3' >>>>> 2023-02-13 13:02:20.719 7 ERROR stevedore.extension [-] Could not load 's3': No module named 'boto3': ModuleNotFoundError: No module named 'boto3' >>>>> >>>>> I get no error whatsoever other than the above. >>>>> >>>>> With regards, >>>>> Swogat Pradhan >>>>> >>>>> On Tue, Feb 21, 2023 at 11:44 AM Pranali Deore <pdeore@redhat.com> wrote: >>>>>> >>>>>> Hey Swogat, >>>>>> >>>>>> By looking at the conf files you've shared, not sure from where the extra store sections 'central', 'cinder_store' are added since you have not added those in any of your yaml files. >>>>>> Could you please share glance-api logs from both the nodes to check more details? >>>>>> Probably the HA configuration is causing this kind of issue but need more information related to deployment & log to understand the root cause. >>>>>> >>>>>> >>>>>> >>>>>> Thanks, >>>>>> ~Pranali >>>>>> >>>>>> >>>>>> On Mon, Feb 20, 2023 at 4:45 PM Swogat Pradhan <swogatpradhan22@gmail.com> wrote: >>>>>>> >>>>>>> Hi Pranali, >>>>>>> Thank you for your response. >>>>>>> The glance.yaml i had shared is for the central site. >>>>>>> I am attaching the required files for your reference. >>>>>>> >>>>>>> With regards, >>>>>>> Swogat Pradhan >>>>>>> >>>>>>> On Mon, Feb 20, 2023 at 4:29 PM Pranali Deore <pdeore@redhat.com> wrote: >>>>>>>> >>>>>>>> Hi Swogat, >>>>>>>> >>>>>>>> The glance.yaml you've shared is for central site or edge site? >>>>>>>> Is it possible for you to share your glance-api.conf just to check how configuration values are set? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Pranali >>>>>>>> >>>>>>>> On Tue, Feb 14, 2023 at 12:35 PM Swogat Pradhan <swogatpradhan22@gmail.com> wrote: >>>>>>>>> >>>>>>>>> Additional INFO: >>>>>>>>> The glance_api container is in reboot loop after adding multi store config. >>>>>>>>> Which is why the image service has become unreachable. >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, Feb 13, 2023 at 6:39 PM Swogat Pradhan <swogatpradhan22@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> Openstack wallaby | tripleo >>>>>>>>>> >>>>>>>>>> I am trying to setup DCN where i want to setup a glance multistore config in both central and edge sites as per the documentation. >>>>>>>>>> But when ever i run stack update with the following config for multistore: >>>>>>>>>> >>>>>>>>>> [stack@hkg2director ~]$ cat workplace/glance.yaml >>>>>>>>>> parameter_defaults: >>>>>>>>>> GlanceShowMultipleLocations: true >>>>>>>>>> GlanceBackend: rbd >>>>>>>>>> GlanceEnabledImportMethods: web-download,copy-image >>>>>>>>>> GlanceBackendID: ceph >>>>>>>>>> GlanceMultistoreConfig: >>>>>>>>>> dcn01: >>>>>>>>>> GlanceBackend: rbd >>>>>>>>>> GlanceStoreDescription: 'dcn01 rbd glance store' >>>>>>>>>> CephClusterName: dcn01 >>>>>>>>>> >>>>>>>>>> The image service becomes inaccessible. Giving the following error: >>>>>>>>>> >>>>>>>>>> (overcloud) [stack@hkg2director ~]$ openstack image list >>>>>>>>>> Failed to contact the endpoint at https://bolt.bdxworld.com:13292 for discovery. Fallback to using that endpoint as the base url. >>>>>>>>>> Failed to contact the endpoint at https://bolt.bdxworld.com:13292 for discovery. Fallback to using that endpoint as the base url. >>>>>>>>>> The image service for :regionOne exists but does not have any supported versions. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> glance api.log from controller nodes: >>>>>>>>>> >>>>>>>>>> 2023-02-13 13:02:19.878 7 INFO glance.async_ [-] Threadpool model set to 'EventletThreadPoolModel' >>>>>>>>>> 2023-02-13 13:02:19.969 7 INFO keyring.backend [-] Loading KWallet >>>>>>>>>> 2023-02-13 13:02:19.977 7 INFO keyring.backend [-] Loading SecretService >>>>>>>>>> 2023-02-13 13:02:19.982 7 INFO keyring.backend [-] Loading Windows >>>>>>>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading chainer >>>>>>>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading macOS >>>>>>>>>> 2023-02-13 13:02:20.571 7 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True. >>>>>>>>>> 2023-02-13 13:02:20.711 7 ERROR stevedore.extension [-] Could not load 'glance.store.s3.Store': No module named 'boto3': ModuleNotFoundError: No module named 'boto3' >>>>>>>>>> 2023-02-13 13:02:20.719 7 ERROR stevedore.extension [-] Could not load 's3': No module named 'boto3': ModuleNotFoundError: No module named 'boto3' >>>>>>>>>> >>>>>>>>>> Is there a patch to fix this issue??
Hi Jhon, Thank you for your response. I never checked the /var/lib/tripleo-config/ceph directory. I had created the ceph config file and keyring for edge in /etc/ceph/ what i needed to do was copy those config to /var/lib/tripleo-config/ceph directory. Now i am able to list the images and glance stores-info gives the desired output. Thank you everyone for helping on this issue. With regards, Swogat Pradhan On Wed, Feb 22, 2023 at 6:36 PM John Fulton <johfulto@redhat.com> wrote:
Sounds like you've confirmed that /etc/ceph/ceph.conf and /etc/ceph/client.openstack.keyring (or similar) are on your container host and functioning correctly. Next check inside the glance container that they are mounted correctly inside of it.
Note this parameter and its default:
https://github.com/openstack/tripleo-heat-templates/blob/stable/wallaby/depl...
Is the glance container mounting this way?
/var/lib/tripleo-config/ceph:/etc/ceph
If so then check the contents of /var/lib/tripleo-config/ceph on your container host.
John
On Wed, Feb 22, 2023 at 3:18 AM Swogat Pradhan <swogatpradhan22@gmail.com> wrote:
Hi Abhishek, I cannot understand why this error is coming though 'ERROR: [errno 2]
RADOS object not found (error calling conf_read_file)'
In controller 1 and Edge 1 nodes i have the config files of both sides
And in other controllers and edge nodes i have config files and openstack keyring of other site and admin keyring of the same site.
When i am trying to connect using cephadm shell --config ceph.conf / dcn01.conf --keyring <admin keyring> --fsid <central/edge> i am able to connect to the cluster and issue commands as well.
With regards, Swogat Pradhan
On Wed, Feb 22, 2023 at 1:02 PM Abhishek Kekane <akekane@redhat.com> wrote:
I can see RADOS not found error multiple times in podman_logs.
2023-02-22T07:01:01.639404765+00:00 stderr F + echo 'Running command:
'\''/usr/bin/glance-api --config-file /etc/glance/glance-api.conf --config-file /etc/glance/glance-image-import.conf'\'''
2023-02-22T07:01:01.639420550+00:00 stdout F Running command: '/usr/bin/glance-api --config-file /etc/glance/glance-api.conf --config-file /etc/glance/glance-image-import.conf' 2023-02-22T07:01:01.639464013+00:00 stderr F + exec /usr/bin/glance-api --config-file /etc/glance/glance-api.conf --config-file /etc/glance/glance-image-import.conf 2023-02-22T07:01:03.662036516+00:00 stderr F ERROR: [errno 2] RADOS object not found (error calling conf_read_file)
I think this might be the reason glance-api is stuck in a recursive loop.
Thanks & Best Regards,
Abhishek Kekane
On Wed, Feb 22, 2023 at 12:57 PM Swogat Pradhan < swogatpradhan22@gmail.com> wrote:
Attaching podman logs glance_api.
With regards, Swogat Pradhan
On Wed, Feb 22, 2023 at 12:46 PM Swogat Pradhan <
swogatpradhan22@gmail.com> wrote:
Hi All, I have set up glance multi store in the central site and the same
issue is present.
Attaching the logs.
With regards, Swogat Pradhan
On Tue, Feb 21, 2023 at 10:04 PM Swogat Pradhan < swogatpradhan22@gmail.com> wrote:
Hi, I am currently working on client approval to run the stack update. I will share the logs shortly.
With regards, Swogat Pradhan
On Tue, Feb 21, 2023 at 7:58 PM John Fulton <johfulto@redhat.com>
wrote:
> > I agree with Pranali and Takashi. > > I have also seen "No module named 'boto3'" in my logs while setting up > DCN with glance successfully. So that is not the cause of your fatal > error. You should look in your logs for something else as the root > cause of this issue. > > John > > On Tue, Feb 21, 2023 at 2:20 AM Takashi Kajinami < tkajinam@redhat.com> wrote: > > > > I looked at tcib definitions but we have never added boto3 even in master. > > > > However, according to the glance_store code, I expect glance_store would be able to ignore the error > > even if boto3 is not enabled unless you enable s3 backend somehow. > > > > https://github.com/openstack/glance_store/blob/64e25979a904e468e1e89d8727e95... > > > > If you can find the log line "Failed to load driver ..." in glance-api.log then the cause would be different > > and we have to check the full logs(both /var/log/containers/glance/glance-api.log and probably > > /var/log/containers/sdtouts/glance_api.log) to find out what is actually causing the crash. > > > > On Tue, Feb 21, 2023 at 4:05 PM Pranali Deore <pdeore@redhat.com> wrote: > >> > >> This doesn't seem like a fatal error, so the issue might be something else. > >> What podman logs glance_api shows? > >> > >> On Tue, Feb 21, 2023 at 12:14 PM Swogat Pradhan < swogatpradhan22@gmail.com> wrote: > >>> > >>> Hi Abhishek, > >>> Yes, this feels like the same bug, as per the comment from Lars Wiegman (lars-fuga) my glance api container is in reboot loop, when glance multi store is configured. > >>> > >>> So I have been searching for a patch for tripleo. > >>> > >>> With regards, > >>> Swogat Pradhan > >>> > >>> On Tue, Feb 21, 2023 at 12:09 PM Abhishek Kekane < akekane@redhat.com> wrote: > >>>> > >>>> Hi Swogat, > >>>> > >>>> I think this looks relevant, kindly go through this and confirm. > >>>> https://bugs.launchpad.net/kolla/+bug/1884259 > >>>> > >>>> > >>>> Thanks & Best Regards, > >>>> > >>>> Abhishek Kekane > >>>> > >>>> > >>>> On Tue, Feb 21, 2023 at 11:58 AM Swogat Pradhan < swogatpradhan22@gmail.com> wrote: > >>>>> > >>>>> Hi pranali, > >>>>> Those configs('central', 'cinder_store') were added when i had started deploying the cloud, those parameters have been removed from
> >>>>> Right now i have reverted the config so i am not getting any erros in the glance api logs. > >>>>> But when i configure the multistore i get the following errors on both sides (central and edge) in glance_api.log: > >>>>> 2023-02-13 13:02:19.878 7 INFO glance.async_ [-] Threadpool model set to 'EventletThreadPoolModel' > >>>>> 2023-02-13 13:02:19.969 7 INFO keyring.backend [-] Loading KWallet > >>>>> 2023-02-13 13:02:19.977 7 INFO keyring.backend [-] Loading SecretService > >>>>> 2023-02-13 13:02:19.982 7 INFO keyring.backend [-] Loading Windows > >>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading chainer > >>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading macOS > >>>>> 2023-02-13 13:02:20.571 7 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True. > >>>>> 2023-02-13 13:02:20.711 7 ERROR stevedore.extension [-] Could not load 'glance.store.s3.Store': No module named 'boto3': ModuleNotFoundError: No module named 'boto3' > >>>>> 2023-02-13 13:02:20.719 7 ERROR stevedore.extension [-] Could not load 's3': No module named 'boto3': ModuleNotFoundError: No module named 'boto3' > >>>>> > >>>>> I get no error whatsoever other than the above. > >>>>> > >>>>> With regards, > >>>>> Swogat Pradhan > >>>>> > >>>>> On Tue, Feb 21, 2023 at 11:44 AM Pranali Deore <
> >>>>>> > >>>>>> Hey Swogat, > >>>>>> > >>>>>> By looking at the conf files you've shared, not sure from where the extra store sections 'central', 'cinder_store' are added since you have not added those in any of your yaml files. > >>>>>> Could you please share glance-api logs from both the nodes to check more details? > >>>>>> Probably the HA configuration is causing this kind of issue but need more information related to deployment & log to understand the root cause. > >>>>>> > >>>>>> > >>>>>> > >>>>>> Thanks, > >>>>>> ~Pranali > >>>>>> > >>>>>> > >>>>>> On Mon, Feb 20, 2023 at 4:45 PM Swogat Pradhan < swogatpradhan22@gmail.com> wrote: > >>>>>>> > >>>>>>> Hi Pranali, > >>>>>>> Thank you for your response. > >>>>>>> The glance.yaml i had shared is for the central site. > >>>>>>> I am attaching the required files for your reference. > >>>>>>> > >>>>>>> With regards, > >>>>>>> Swogat Pradhan > >>>>>>> > >>>>>>> On Mon, Feb 20, 2023 at 4:29 PM Pranali Deore <
along with the openstack and admin keyring files. the yaml now, but exist in the config file. pdeore@redhat.com> wrote: pdeore@redhat.com> wrote:
> >>>>>>>> > >>>>>>>> Hi Swogat, > >>>>>>>> > >>>>>>>> The glance.yaml you've shared is for central site or edge site? > >>>>>>>> Is it possible for you to share your glance-api.conf just to check how configuration values are set? > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> Thanks, > >>>>>>>> Pranali > >>>>>>>> > >>>>>>>> On Tue, Feb 14, 2023 at 12:35 PM Swogat Pradhan < swogatpradhan22@gmail.com> wrote: > >>>>>>>>> > >>>>>>>>> Additional INFO: > >>>>>>>>> The glance_api container is in reboot loop after adding multi store config. > >>>>>>>>> Which is why the image service has become unreachable. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> On Mon, Feb 13, 2023 at 6:39 PM Swogat Pradhan < swogatpradhan22@gmail.com> wrote: > >>>>>>>>>> > >>>>>>>>>> Hi, > >>>>>>>>>> Openstack wallaby | tripleo > >>>>>>>>>> > >>>>>>>>>> I am trying to setup DCN where i want to setup a glance multistore config in both central and edge sites as per the documentation. > >>>>>>>>>> But when ever i run stack update with the following config for multistore: > >>>>>>>>>> > >>>>>>>>>> [stack@hkg2director ~]$ cat workplace/glance.yaml > >>>>>>>>>> parameter_defaults: > >>>>>>>>>> GlanceShowMultipleLocations: true > >>>>>>>>>> GlanceBackend: rbd > >>>>>>>>>> GlanceEnabledImportMethods: web-download,copy-image > >>>>>>>>>> GlanceBackendID: ceph > >>>>>>>>>> GlanceMultistoreConfig: > >>>>>>>>>> dcn01: > >>>>>>>>>> GlanceBackend: rbd > >>>>>>>>>> GlanceStoreDescription: 'dcn01 rbd glance store' > >>>>>>>>>> CephClusterName: dcn01 > >>>>>>>>>> > >>>>>>>>>> The image service becomes inaccessible. Giving the following error: > >>>>>>>>>> > >>>>>>>>>> (overcloud) [stack@hkg2director ~]$ openstack image list > >>>>>>>>>> Failed to contact the endpoint at https://bolt.bdxworld.com:13292 for discovery. Fallback to using that endpoint as the base url. > >>>>>>>>>> Failed to contact the endpoint at https://bolt.bdxworld.com:13292 for discovery. Fallback to using that endpoint as the base url. > >>>>>>>>>> The image service for :regionOne exists but does not have any supported versions. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> glance api.log from controller nodes: > >>>>>>>>>> > >>>>>>>>>> 2023-02-13 13:02:19.878 7 INFO glance.async_ [-] Threadpool model set to 'EventletThreadPoolModel' > >>>>>>>>>> 2023-02-13 13:02:19.969 7 INFO keyring.backend [-] Loading KWallet > >>>>>>>>>> 2023-02-13 13:02:19.977 7 INFO keyring.backend [-] Loading SecretService > >>>>>>>>>> 2023-02-13 13:02:19.982 7 INFO keyring.backend [-] Loading Windows > >>>>>>>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading chainer > >>>>>>>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading macOS > >>>>>>>>>> 2023-02-13 13:02:20.571 7 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True. > >>>>>>>>>> 2023-02-13 13:02:20.711 7 ERROR stevedore.extension [-] Could not load 'glance.store.s3.Store': No module named 'boto3': ModuleNotFoundError: No module named 'boto3' > >>>>>>>>>> 2023-02-13 13:02:20.719 7 ERROR stevedore.extension [-] Could not load 's3': No module named 'boto3': ModuleNotFoundError: No module named 'boto3' > >>>>>>>>>> > >>>>>>>>>> Is there a patch to fix this issue?? >
participants (5)
-
Abhishek Kekane
-
John Fulton
-
Pranali Deore
-
Swogat Pradhan
-
Takashi Kajinami