December 2023 - openstack-discuss

After a power outage - Cinder Scheduler reports Volume Service as down on only 1 node
by Abhijit Anand 23 Dec '23

23 Dec '23

Hi, Problem Statement: - “openstack volume service list” shows “volume service” down on 1 node. - Cinder scheduler logs show this. 2023-12-22 09:00:19.779 7 WARNING cinder.scheduler.host_manager [None req-40e35d10-9441-43c2-9647-6630cba21e1c - - - - - -] volume service is down. (host: compute2@rbd-1) - Why is the host_manager of Cinder-Scheduler showing “compute2” as down? Versions: - Openstack version 2023.1 Installed using kolla-ansible on ubuntu 22.04 - Ceph version 18.2.0 More info: - I’ve got a 3 node openstack deployment with one of them doubling up as both compute as well as controller node. - Backend storage for cinder volumes and backups is a 3 node Ceph cluster deployed using cephadm. - a power outage resulted in all nodes shutting down. - once back up, the “openstack volume service list” command shows that cinder-volume and cinder-backup services are down on “compute2”. Refer “Output #1” below. - I enabled “debug=True” on both the “good” i.e. compute1, as well as “bad” i.e. “compute2” nodes. I dont see any errors or issues in the cinder-volume and cinder-backup logs of either. Refer outputs #2 and #3 below. - I don’t see any errors related to compute2 in ceph or cephadm logs. - No config has been changed. The only “change” was the forced shutdown. - I’ve restarted the docker containers (cinder volume, cinder backup, cinder-scheduler on controller). I’ve even rebooted the controller and the “bad” node. - All containers come back as healthy - Please see output #4, it shows, what appear to be “All Good!” updates from the “bad” node. Then why is the scheduler considering this node as down? Outputs: #1: openstack volume service list abanand@Abhijits-MacBook-Air ~ % openstack volume service list +------------------+------------------+------+---------+-------+----------------------------+ | Binary | Host | Zone | Status | State | Updated At | +------------------+------------------+------+---------+-------+----------------------------+ | cinder-scheduler | controller | nova | enabled | up | 2023-12-22T03:44:22.000000 | | cinder-volume | compute1@rbd-1 | nova | enabled | up | 2023-12-22T03:44:21.000000 | | cinder-volume | compute2@rbd-1 | nova | enabled | down | 2023-12-22T03:42:04.000000 | | cinder-volume | controller@rbd-1 | nova | enabled | up | 2023-12-22T03:44:20.000000 | | cinder-backup | compute2 | nova | enabled | down | 2023-12-22T03:42:02.000000 | | cinder-backup | compute1 | nova | enabled | up | 2023-12-22T03:44:20.000000 | | cinder-backup | controller | nova | enabled | up | 2023-12-22T03:44:19.000000 | +------------------+------------------+------+---------+-------+——————————————+ #2: compute2 - “Bad” node logs after enabling debug=true 2023-12-22 09:25:28.041 23 DEBUG oslo_service.periodic_task [None req-8388d960-ae3a-46d3-838a-24c112929683 - - - - - -] Running periodic task VolumeManager.publish_service_capabilities run_periodic_tasks /var/lib/kolla/venv/lib/python3.10/site-packages/oslo_service/periodic_task.py:210 2023-12-22 09:25:28.043 23 DEBUG cinder.volume.drivers.rbd [None req-8388d960-ae3a-46d3-838a-24c112929683 - - - - - -] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=15). _do_conn /var/lib/kolla/venv/lib/python3.10/site-packages/cinder/volume/drivers/rbd.py:597 2023-12-22 09:25:28.074 23 DEBUG cinder.volume.drivers.rbd [None req-8388d960-ae3a-46d3-838a-24c112929683 - - - - - -] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=15). _do_conn /var/lib/kolla/venv/lib/python3.10/site-packages/cinder/volume/drivers/rbd.py:597 2023-12-22 09:25:28.103 23 DEBUG cinder.manager [None req-8388d960-ae3a-46d3-838a-24c112929683 - - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /var/lib/kolla/venv/lib/python3.10/site-packages/cinder/manager.py:202 #3: compute1 - “Good” node logs after enabling debug=true 2023-12-22 09:25:28.041 23 DEBUG oslo_service.periodic_task [None req-8388d960-ae3a-46d3-838a-24c112929683 - - - - - -] Running periodic task VolumeManager.publish_service_capabilities run_periodic_tasks /var/lib/kolla/venv/lib/python3.10/site-packages/oslo_service/periodic_task.py:210 2023-12-22 09:25:28.043 23 DEBUG cinder.volume.drivers.rbd [None req-8388d960-ae3a-46d3-838a-24c112929683 - - - - - -] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=15). _do_conn /var/lib/kolla/venv/lib/python3.10/site-packages/cinder/volume/drivers/rbd.py:597 2023-12-22 09:25:28.074 23 DEBUG cinder.volume.drivers.rbd [None req-8388d960-ae3a-46d3-838a-24c112929683 - - - - - -] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=15). _do_conn /var/lib/kolla/venv/lib/python3.10/site-packages/cinder/volume/drivers/rbd.py:597 2023-12-22 09:25:28.103 23 DEBUG cinder.manager [None req-8388d960-ae3a-46d3-838a-24c112929683 - - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /var/lib/kolla/venv/lib/python3.10/site-packages/cinder/manager.py:202 #4: Output from Cinder Scheduler logs 2023-12-22 09:35:18.861 7 DEBUG cinder.scheduler.host_manager [None req-09c91b17-3861-4c95-b85e-e4d39c17923c - - - - - -] Received backup service update from compute2: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} update_service_capabilities /var/lib/kolla/venv/lib/python3.10/site-packages/cinder/scheduler/host_manager.py:597 2023-12-22 09:35:29.659 7 DEBUG cinder.scheduler.host_manager [None req-ba4bd917-52a8-4b02-92a3-5f989c5d70ae - - - - - -] Received volume service update from compute2@rbd-1: {'vendor_name': 'Open Source', 'driver_version': '1.3.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 14649.52, 'free_capacity_gb': 14216.63, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:d524a89e-60eb-11ee-a0c3-7db08f4d9091:cinder:volumes', 'backend_state': 'up', 'qos_support': True, 'volume_backend_name': 'rbd-1', 'replication_enabled': False, 'allocated_capacity_gb': 1031, 'filter_function': None, 'goodness_function': None} update_service_capabilities /var/lib/kolla/venv/lib/python3.10/site-packages/cinder/scheduler/host_manager.py:628

3 4

[all] Posting results from generative "AI" (LLM) queries
by Jeremy Stanley 22 Dec '23

22 Dec '23

I'm refraining from calling out any particular participant on this (for now), but we've gotten a lot of recent and very enthusiastic answers to people's questions which look very much like the results of blindly pasting them into an LLM (e.g. ChatGPT) and then passing the response off as the sender's own experience. This quickly falls apart when there are multiple open source projects with the same name and the LLM gives a response about the "wrong" project with that name, revealing that the person posting it really doesn't even know the difference between them. Users post their questions to this list because they've exhausted other avenues and are hoping to hear responses derived from (human) subscribers' own personal/professional experiences. If you've legitimately run the software and actually know what the question means, then please do answer. But if users wanted to see the results of pasting their question into ChatGPT then they'd just do that (and likely already have). I don't want to institute "AI" oriented policies for this list like other forums have been forced to in recent months, so please (everyone!) just make it clear when the information you're posting is the result of consulting another source, and state what source it came from. Thanks! -- Jeremy Stanley

1 0

Re: [swift][osprofiler][telemetry] What is the current state of tracing for Swift?
by Matheus Victorello 22 Dec '23

22 Dec '23

While reading your response, I realized that my question was a bit unclear. I was referring to OpenStack Object Storage 'Swift', which is why I asked about OSProfiler. I mention this because it got confusing since I mentioned OpenTelemetry. I came across the association between OpenTelemetry and the Swift programming language while researching and ended up including that in my question. Just to clarify, I want to do tracing of OpenStack Swift. Sorry for the confusion. -- _‘Esta mensagem é direcionada apenas para os endereços constantes no cabeçalho inicial. Se você não está listado nos endereços constantes no cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão imediatamente anuladas e proibidas’._ * **‘Apesar do Magazine Luiza tomar todas as precauções razoáveis para assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.*

1 0

[openstack][kolla-ansible] kolla-ansible stucked when reconfigure
by Nguyễn Hữu Khôi 22 Dec '23

22 Dec '23

Hello guys. I am operating an openstack cluster with kolla but sometimes I cannot reconfigure. It stucked at some hosts(gather fact) and I need to restart docker service on the stucked host to continue. I tried to google but I cannot find the root cause. Would you share some experiences on it because my solution causes instances on host to timeout some seconds. Thank you. Nguyen Huu Khoi

2 7

Devstack, Cinder, Nova versus memcached
by Pete Zaitcev 21 Dec '23

21 Dec '23

Hello: We all know how Cinder and Nova use memcached to save Keystone tokens (technically, authtoken middleware does it for them). That is expected. However, devstack leaves a config for Cinder that looks like this: [nova] region_name = RegionOne memcached_servers = localhost:11211 Does anyone remember how that is used? I looked at Cinder and python-novaclient, and neither use memcached for anything there. Thanks in advance, -- Pete

3 8

[neutron] Next week CI meeting is cancelled
by Yatin Karel 21 Dec '23

21 Dec '23

Hello Neutron Team !! With the holiday season, next week's CI meeting is cancelled. Wish you happy holidays. Next CI meeting will be now on 02nd January, 2024. -- Thanks and Regards Yatin Karel

1 0

[kolla-ansible][zun] Intention to drop support for Zun for the 2023.1 and path forward
by Jan Gutter 21 Dec '23

21 Dec '23

Hi folks, In yesterday's Kolla IRC meeting we discussed the path forward for Zun [1] and came to the conclusion that it might be better for all involved to drop support for Zun for the 2023.1 release until it can be updated to support the necessary dependencies. More specific details at the end of the message. With this in mind, the plan is to start this process in a week from now, unless a better solution is presented. There is still some time for discussion, please don't hesitate! There are two paths forward after this: If Zun can be updated to support the dependencies during the maintenance timeframe of 2023.1, then support would be restored. If no support is available at the time Caracal releases, however, it will be considered for removal in kolla and kolla-ansible. What this means: Kolla-ansible operators currently running Zun should hold off migration to 2023.1. Any development help would of course be appreciated. Please coordinate with the Zun project to help with development. Details on the dependency problems: * Currently, Zun has an external dependency on Docker 20.x, and uses a feature that has been removed in later versions of Docker. * This Docker feature also relies on an older version of etcd (3.3) * The mitigation [2] is to pin to the old versions of Docker and etcd, but this is not possible with Debian Bookworm hosts. Why this is a problem in Kolla-ansible (and Kolla): * This can't easily be solved by vendoring in an older version of etcd: the migration path it introduces to operators would incur a significant amount of risk. * Host level involvement is also particularly painful, the host version of Docker needs to connect to the etcd service running in the containers. * Even though the containers can be kept buildable and the configuration and inventory settings could be maintained, a working setup is not testable in CI. Thanks very much for everyone's patience, contributions and time! [1]: https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-09-13-13.00.log… [2]: https://bugs.launchpad.net/zun/+bug/2007142

2 2

[neutron] Neutron drivers meeting cancelled
by Rodolfo Alonso Hernandez 21 Dec '23

21 Dec '23

Hello Neutrinos: Due to the lack of agenda [1], tomorrow's meeting is cancelled. Have a nice weekend and happy holidays. [1]https://wiki.openstack.org/wiki/Meetings/NeutronDrivers

1 0

[nova] Next two weekly meetings CANCELLED
by Sylvain Bauza 21 Dec '23

21 Dec '23

That's it. See you back on Jan 9th :)

1 0

[Glance] Cancelling Today's Weekly Meeting
by Pranali Deore 21 Dec '23

21 Dec '23

Hello, Since most of the team members are not around and there is nothing much on agenda too, cancelling today's (21st Dec) weekly meeting. Thanks & Regards, Pranali Deore

1 0