[kolla-ansible][Antelope][Heat] Issues with ASG and Zaqar Trigger
Hello, We are implementing the auto-healing workflow captured here - https://github.com/openstack/heat-templates/tree/master/hot/autohealing We are deploying a Heat auto-scaling group that scales a template that contains the needed services and the Nova resource. The template contains a Zaqar queue, Mistral trigger and Mistral workflow. We are also using a scaling policy to scale up and down the template resources. This is working fine, but we are experiencing an issue with scaling up the auto-scaling group via 'openstack stack resource signal <stack> <scaling_policy>. If the Heat stack is update with a larger value, the resources are added correctly. It is only when using the scaling policy. Logs show a forbidden error when attempting to create the Zaqer Mistral trigger. Any ideas? 2024-05-09 15:10:41Z [stack_1-DmMTV.asg.gxvscbt3q3ga]: UPDATE_COMPLETE Stack UPDATE completed successfully 2024-05-09 15:10:41Z [stack_1-DmMTV.asg.5rjz74hujhxh.alarm_subscription]: CREATE_FAILED ServiceUnavailableError: resources.alarm_subscription: Error response from Zaqar. Code: 503. Title: Service temporarily unavailable. Description: Subscription could not be created. Please try again in a few seconds.. 2024-05-09 15:10:41Z [stack_1-DmMTV.asg.5rjz74hujhxh]: CREATE_FAILED Resource CREATE failed: ServiceUnavailableError: resources.alarm_subscription: Error response from Zaqar. Code: 503. Title: Service temporarily unavailable. Description: Subscription could not be created. Please try again in a few seconds.. 2024-05-09 15:10:41Z [stack_1-DmMTV.asg.5rjz74hujhxh]: CREATE_FAILED ServiceUnavailableError: resources.5rjz74hujhxh.resources.alarm_subscription: Error response from Zaqar. Code: 503. Title: Service temporarily unavailable. Description: Subscription could not be created. Please try again in a few seconds.. 2024-05-09 15:10:41Z [stack_1-DmMTV.asg]: UPDATE_FAILED Resource CREATE failed: ServiceUnavailableError: resources.5rjz74hujhxh.resources.alarm_subscription: Error response from Zaqar. Code: 503. Title: Service temporarily unavailable. Description: Subscription could not be created. Please try again in a few s 2024-05-09 15:10:41Z [stack_1-DmMTV.asg.gxvscbt3q3ga]: UPDATE_COMPLETE state changed 2024-05-09 15:10:42Z [stack_1-DmMTV.asg.nzixbsxmb5mt]: UPDATE_COMPLETE Stack UPDATE completed successfully 2024-05-09 15:10:42Z [stack_1-DmMTV.asg.nzixbsxmb5mt]: UPDATE_COMPLETE state changed 2024-05-09 14:43:47.510 731 ERROR zaqar.transport.wsgi.v2_0.subscriptions [] Subscription could not be created.: keystoneauth1.exceptions.http.Forbidden: Remaining redelegation depth of 0 out of allowed range of [0..10] (Disable insecure_debug mode to suppress these details.) (HTTP 403) (Request-ID: ) 2024-05-09 14:43:47.510 731 ERROR zaqar.transport.wsgi.v2_0.subscriptions Traceback (most recent call last): 2024-05-09 14:43:47.510 731 ERROR zaqar.transport.wsgi.v2_0.subscriptions File "/var/lib/kolla/venv/lib/python3.10/site-packages/zaqar/transport/wsgi/v2_0/subscriptions.py", line 193, in on_post 2024-05-09 14:43:47.510 731 ERROR zaqar.transport.wsgi.v2_0.subscriptions mgr.driver.register(subscriber, options, ttl, project_id, req_data) 2024-05-09 14:43:47.510 731 ERROR zaqar.transport.wsgi.v2_0.subscriptions File "/var/lib/kolla/venv/lib/python3.10/site-packages/zaqar/notification/tasks/trust.py", line 60, in register 2024-05-09 14:43:47.510 731 ERROR zaqar.transport.wsgi.v2_0.subscriptions trust_id = auth.create_trust_id( 2024-05-09 14:43:47.510 731 ERROR zaqar.transport.wsgi.v2_0.subscriptions File "/var/lib/kolla/venv/lib/python3.10/site-packages/zaqar/common/auth.py", line 67, in create_trust_id 2024-05-09 14:43:47.510 731 ERROR zaqar.transport.wsgi.v2_0.subscriptions trust = client.trusts.create(trustor_user=trustor_user_id, 2024-05-09 14:43:47.510 731 ERROR zaqar.transport.wsgi.v2_0.subscriptions File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystoneclient/v3/contrib/trusts.py", line 74, in create 2024-05-09 14:43:47.510 731 ERROR zaqar.transport.wsgi.v2_0.subscriptions return super(TrustManager, self).create( 2024-05-09 14:43:47.510 731 ERROR zaqar.transport.wsgi.v2_0.subscriptions File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystoneclient/base.py", line 86, in func 2024-05-09 14:43:47.510 731 ERROR zaqar.transport.wsgi.v2_0.subscriptions return f(*args, **new_kwargs) 2024-05-09 14:43:47.510 731 ERROR zaqar.transport.wsgi.v2_0.subscriptions File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystoneclient/base.py", line 383, in create 2024-05-09 14:43:47.510 731 ERROR zaqar.transport.wsgi.v2_0.subscriptions return self._post( 2024-05-09 14:43:47.510 731 ERROR zaqar.transport.wsgi.v2_0.subscriptions File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystoneclient/base.py", line 192, in _post 2024-05-09 14:43:47.510 731 ERROR zaqar.transport.wsgi.v2_0.subscriptions resp, body = self.client.post(url, body=body, **kwargs) 2024-05-09 14:43:47.510 731 ERROR zaqar.transport.wsgi.v2_0.subscriptions File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystoneauth1/adapter.py", line 401, in post 2024-05-09 14:43:47.510 731 ERROR zaqar.transport.wsgi.v2_0.subscriptions return self.request(url, 'POST', **kwargs) 2024-05-09 14:43:47.510 731 ERROR zaqar.transport.wsgi.v2_0.subscriptions File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystoneauth1/adapter.py", line 554, in request 2024-05-09 14:43:47.510 731 ERROR zaqar.transport.wsgi.v2_0.subscriptions resp = super(LegacyJsonAdapter, self).request(*args, **kwargs) 2024-05-09 14:43:47.510 731 ERROR zaqar.transport.wsgi.v2_0.subscriptions File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystoneauth1/adapter.py", line 257, in request 2024-05-09 14:43:47.510 731 ERROR zaqar.transport.wsgi.v2_0.subscriptions return self.session.request(url, method, **kwargs) 2024-05-09 14:43:47.510 731 ERROR zaqar.transport.wsgi.v2_0.subscriptions File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystoneauth1/session.py", line 986, in request 2024-05-09 14:43:47.510 731 ERROR zaqar.transport.wsgi.v2_0.subscriptions raise exceptions.from_response(resp, method, url) 2024-05-09 14:43:47.510 731 ERROR zaqar.transport.wsgi.v2_0.subscriptions keystoneauth1.exceptions.http.Forbidden: Remaining redelegation depth of 0 out of allowed range of [0..10] (Disable insecure_debug mode to suppress these details.) (HTTP 403) (Request-ID: ) 2024-05-09 14:43:47.510 731 ERROR zaqar.transport.wsgi.v2_0.subscriptions Regards, Devin
participants (1)
-
Devin Patterson