Hello,

We are implementing the auto-healing workflow captured here - https://github.com/openstack/heat-templates/tree/master/hot/autohealing

We are deploying a Heat auto-scaling group that scales a template that contains the needed services and the Nova resource. The template contains a Zaqar queue, Mistral trigger and Mistral workflow. We are also using a scaling policy to scale up and down the template resources.

This is working fine, but we are experiencing an issue with scaling up the auto-scaling group via 'openstack stack resource signal <stack> <scaling_policy>. If the Heat stack is update with a larger value, the resources are added correctly. It is only when using the scaling policy. Logs show a forbidden error when attempting to create the Zaqer Mistral trigger. Any ideas?

2024-05-09 15:10:41Z [stack_1-DmMTV.asg.gxvscbt3q3ga]: UPDATE_COMPLETE Stack UPDATE completed successfully

2024-05-09 15:10:41Z [stack_1-DmMTV.asg.5rjz74hujhxh.alarm_subscription]: CREATE_FAILED ServiceUnavailableError: resources.alarm_subscription: Error response from Zaqar. Code: 503. Title: Service temporarily unavailable. Description: Subscription could not be created. Please try again in a few seconds..

2024-05-09 15:10:41Z [stack_1-DmMTV.asg.5rjz74hujhxh]: CREATE_FAILED Resource CREATE failed: ServiceUnavailableError: resources.alarm_subscription: Error response from Zaqar. Code: 503. Title: Service temporarily unavailable. Description: Subscription could not be created. Please try again in a few seconds..

2024-05-09 15:10:41Z [stack_1-DmMTV.asg.5rjz74hujhxh]: CREATE_FAILED ServiceUnavailableError: resources.5rjz74hujhxh.resources.alarm_subscription: Error response from Zaqar. Code: 503. Title: Service temporarily unavailable. Description: Subscription could not be created. Please try again in a few seconds..

2024-05-09 15:10:41Z [stack_1-DmMTV.asg]: UPDATE_FAILED Resource CREATE failed: ServiceUnavailableError: resources.5rjz74hujhxh.resources.alarm_subscription: Error response from Zaqar. Code: 503. Title: Service temporarily unavailable. Description: Subscription could not be created. Please try again in a few s

2024-05-09 15:10:41Z [stack_1-DmMTV.asg.gxvscbt3q3ga]: UPDATE_COMPLETE state changed

2024-05-09 15:10:42Z [stack_1-DmMTV.asg.nzixbsxmb5mt]: UPDATE_COMPLETE Stack UPDATE completed successfully

2024-05-09 15:10:42Z [stack_1-DmMTV.asg.nzixbsxmb5mt]: UPDATE_COMPLETE state changed

2024-05-09 14:43:47.510 731 ERROR zaqar.transport.wsgi.v2_0.subscriptions [] Subscription could not be created.: keystoneauth1.exceptions.http.Forbidden: Remaining redelegation depth of 0 out of allowed range of [0..10] (Disable insecure_debug mode to suppress these details.) (HTTP 403) (Request-ID: )

2024-05-09 14:43:47.510 731 ERROR zaqar.transport.wsgi.v2_0.subscriptions Traceback (most recent call last):

2024-05-09 14:43:47.510 731 ERROR zaqar.transport.wsgi.v2_0.subscriptions File "/var/lib/kolla/venv/lib/python3.10/site-packages/zaqar/transport/wsgi/v2_0/subscriptions.py", line 193, in on_post

2024-05-09 14:43:47.510 731 ERROR zaqar.transport.wsgi.v2_0.subscriptions mgr.driver.register(subscriber, options, ttl, project_id, req_data)

2024-05-09 14:43:47.510 731 ERROR zaqar.transport.wsgi.v2_0.subscriptions File "/var/lib/kolla/venv/lib/python3.10/site-packages/zaqar/notification/tasks/trust.py", line 60, in register

2024-05-09 14:43:47.510 731 ERROR zaqar.transport.wsgi.v2_0.subscriptions trust_id = auth.create_trust_id(

2024-05-09 14:43:47.510 731 ERROR zaqar.transport.wsgi.v2_0.subscriptions File "/var/lib/kolla/venv/lib/python3.10/site-packages/zaqar/common/auth.py", line 67, in create_trust_id

2024-05-09 14:43:47.510 731 ERROR zaqar.transport.wsgi.v2_0.subscriptions trust = client.trusts.create(trustor_user=trustor_user_id,

2024-05-09 14:43:47.510 731 ERROR zaqar.transport.wsgi.v2_0.subscriptions File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystoneclient/v3/contrib/trusts.py", line 74, in create

2024-05-09 14:43:47.510 731 ERROR zaqar.transport.wsgi.v2_0.subscriptions return super(TrustManager, self).create(

2024-05-09 14:43:47.510 731 ERROR zaqar.transport.wsgi.v2_0.subscriptions File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystoneclient/base.py", line 86, in func

2024-05-09 14:43:47.510 731 ERROR zaqar.transport.wsgi.v2_0.subscriptions return f(*args, **new_kwargs)

2024-05-09 14:43:47.510 731 ERROR zaqar.transport.wsgi.v2_0.subscriptions File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystoneclient/base.py", line 383, in create

2024-05-09 14:43:47.510 731 ERROR zaqar.transport.wsgi.v2_0.subscriptions return self._post(

2024-05-09 14:43:47.510 731 ERROR zaqar.transport.wsgi.v2_0.subscriptions File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystoneclient/base.py", line 192, in _post

2024-05-09 14:43:47.510 731 ERROR zaqar.transport.wsgi.v2_0.subscriptions resp, body = self.client.post(url, body=body, **kwargs)

2024-05-09 14:43:47.510 731 ERROR zaqar.transport.wsgi.v2_0.subscriptions File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystoneauth1/adapter.py", line 401, in post

2024-05-09 14:43:47.510 731 ERROR zaqar.transport.wsgi.v2_0.subscriptions return self.request(url, 'POST', **kwargs)

2024-05-09 14:43:47.510 731 ERROR zaqar.transport.wsgi.v2_0.subscriptions File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystoneauth1/adapter.py", line 554, in request

2024-05-09 14:43:47.510 731 ERROR zaqar.transport.wsgi.v2_0.subscriptions resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)

2024-05-09 14:43:47.510 731 ERROR zaqar.transport.wsgi.v2_0.subscriptions File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystoneauth1/adapter.py", line 257, in request

2024-05-09 14:43:47.510 731 ERROR zaqar.transport.wsgi.v2_0.subscriptions return self.session.request(url, method, **kwargs)

2024-05-09 14:43:47.510 731 ERROR zaqar.transport.wsgi.v2_0.subscriptions File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystoneauth1/session.py", line 986, in request

2024-05-09 14:43:47.510 731 ERROR zaqar.transport.wsgi.v2_0.subscriptions raise exceptions.from_response(resp, method, url)

2024-05-09 14:43:47.510 731 ERROR zaqar.transport.wsgi.v2_0.subscriptions keystoneauth1.exceptions.http.Forbidden: Remaining redelegation depth of 0 out of allowed range of [0..10] (Disable insecure_debug mode to suppress these details.) (HTTP 403) (Request-ID: )

2024-05-09 14:43:47.510 731 ERROR zaqar.transport.wsgi.v2_0.subscriptions

Regards,

Devin