Hey team,

Thanks for those who replied off list.

Thankfully we figured it out, it was due to underlying resolver config mysterious mulching AAAA records instead of requesting A records…

Probably goes to say, Openstack needs to have some seriously heavier logging 😃 haha.

Thanks!
Karl.

From: Dmitriy Rabotyagov <noonedeadpunk@gmail.com>
Date: Tuesday, 12 December 2023 at 5:35 pm
To: Karl Kloppenborg <kkloppenborg@resetdata.com.au>
Subject: Re: [URGENT] - Help needed with open stack services hung

Hey,

> stopping services from processing requests

I assume this means that services are started and do not report any errors in logs?

What happens with requests themselves? Are they answered with error or just hang?

Also how environment was deployed? Some deployment tooling or manually with packages, k8s?

On Tue, Dec 12, 2023, 05:30 Karl Kloppenborg <kkloppenborg@resetdata.com.au> wrote:

Hi Openstack Community,

Unfortunately we’ve had a weird situation arise where one of our regions has just stopped dead in it’s tracks, all management systems have been rebooted but there seems to be something stopping services from processing requests.

Database, RabbitMQ Memcached have all been individually tested and I cannot find fault with them.

Could any openstack consultants who are available for immediate emergency work please contact me off list.

Thanks,
Karl.

Kind Regards,

Karl Kloppenborg
BCompSc, CNCF-[KCNA, CKA, CKAD], LFCE, CompTIA Linux+ XK0-004

Chief Technology Officer

m: +61 437 239 565 | e: kkloppenborg@resetdata.com.au | w: https://resetdata.com

This email transmission is intended only for the addressee / person responsible for delivery of the message to such person and may contain confidential or privileged information. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you, nor may you use, review, disclose, disseminate or copy any information contained in or attached to it. Whilst this email has been checked for viruses, the sender does not warrant that any attachments are free from viruses or other defects. You assume all liability for any loss, damage or other consequences which may arise from opening or using the attachments. If you received this e-mail in error please delete it and any attachments and kindly notify us by immediately sending an email to contact@resetdata.com.au