Hi, 1. When you remove definition from openstack_user_config, items from inventory won't be dropped automatically. For that you would need to use scripts/inventory-manage.py -r <container_name>. You should indeed destroy containers first. 2. In order to remove cider-volume definitions from API, you would need to run cinder-manage service remove <service> <host> https://docs.openstack.org/cinder/rocky/man/cinder-manage.html#cinder-servic... cinder-manage binary can be found on cinder-api containers inside virtualenv, ie /openstack/venvs/cinder-<venv_tag>/bin/cinder-manage вс, 7 авг. 2022 г., 16:22 Dave Hall <kdhall@binghamton.edu>:

Hello,

Please pardon the repost - I noticed this morning that I didn't finish the subject line.

Problem summary: I have a bunch of lingering non-functional cinder definitions and I'm looking for guidance on how to clean them up.

Thanks.

-Dave

-- Dave Hall Binghamton University kdhall@binghamton.edu

On Sat, Aug 6, 2022 at 2:52 PM Dave Hall <kdhall@binghamton.edu> wrote:

...

Hello,

I seem to have gotten myself in a bit of a mess trying to set up Cinder with an NFS back-end. After working with Glance and NFS, I started on Cinder. I noticed immediately that there weren't any NFS mounts in the Cinder-API containers like there were in the Glance-API containers. Also that there were no NFS packages in the Cinder-API containers.

In reading some Cinder documentation, I also got the impression that each Cinder host/container needs to have its own NFS store.

Pawing through the playbooks and documentation I saw that unlike Glance, Cinder is split into two pieces - Cinder-API and Cinder-Volume. I found cinder-volume.yml.example in env.d, activated it, and created Cinder-Volume containers on my 3 infra hosts. I also created 3 separate NFS shares and changed the storage-hosts section of my openstack_user_config.yml accordingly.

After this I found that while I was able to create volumes, the prep_volume part of launching an instance was failing.

Digging in, I found:

# openstack volume service list

+------------------+-------------------------------------------------------+------+---------+-------+----------------------------+ | Binary | Host | Zone | Status | State | Updated At |

+------------------+-------------------------------------------------------+------+---------+-------+----------------------------+ | cinder-volume | C6220-9@nfs_volume | nova | enabled | down | 2022-07-23T02:46:13.000000 | | cinder-volume | C6220-10@nfs_volume | nova | enabled | down | 2022-07-23T02:46:14.000000 | | cinder-volume | C6220-11@nfs_volume | nova | enabled | down | 2022-07-23T02:46:14.000000 | | cinder-scheduler | infra36-cinder-api-container-da8e100f | nova | enabled | up | 2022-08-06T13:29:10.000000 | | cinder-scheduler | infra38-cinder-api-container-27219f93 | nova | enabled | up | 2022-08-06T13:29:10.000000 | | cinder-scheduler | infra37-cinder-api-container-ea7f847b | nova | enabled | up | 2022-08-06T13:29:10.000000 | | cinder-volume | C6220-9@nfs_volume1 | nova | enabled | up | 2022-08-06T13:29:10.000000 | | cinder-volume | infra37-cinder-volumes-container-5b9635ad@nfs_volume | nova | enabled | down | 2022-08-04T18:32:53.000000 | | cinder-volume | infra36-cinder-volumes-container-77190057@nfs_volume1 | nova | enabled | down | 2022-08-06T13:03:03.000000 | | cinder-volume | infra38-cinder-volumes-container-a7bcfc9b@nfs_volume | nova | enabled | down | 2022-08-04T18:32:53.000000 | | cinder-volume | infra37-cinder-volumes-container-5b9635ad@nfs_volume2 | nova | enabled | down | 2022-08-06T13:03:05.000000 | | cinder-volume | C6220-10@nfs_volume2 | nova | enabled | up | 2022-08-06T13:29:10.000000 | | cinder-volume | C6220-11@nfs_volume3 | nova | enabled | up | 2022-08-06T13:29:10.000000 | | cinder-volume | infra38-cinder-volumes-container-a7bcfc9b@nfs_volume3 | nova | enabled | down | 2022-08-06T13:03:03.000000 |

+------------------+-------------------------------------------------------+------+---------+-------+----------------------------+

Thinking I could save this, I used containers-lxc-destroy.yml to destroy my cinder-volumes containers and deactivated cinder-volume.yml.example. Then I ran setup-hosts.yml, which has restored the cinder-volumes containers even though is_metal: false has been removed.

Clearly a stronger intervention will be required. I would like to fully get rid of the cinder-volumes containers and go back to an is_metal: true scenario. I also need to get rid of the unnumbered nfs_volume referenes, which I assume are in some cinder config file somewhere.

Below is a clip from my openstack_user_config.yml:

storage_hosts: infra36: ip: 172.29.236.36 container_vars: cinder_backends: nfs_volume1: volume_backend_name: NFS_VOLUME1 volume_driver: cinder.volume.drivers.nfs.NfsDriver nfs_mount_options: "rsize=65535,wsize=65535,timeo=1200,actimeo=120" nfs_shares_config: /etc/cinder/nfs_shares_volume1 shares: - { ip: "172.29.244.27", share: "/NFS_VOLUME1" } infra37: ip: 172.29.236.37 container_vars: cinder_backends: nfs_volume2: volume_backend_name: NFS_VOLUME2 volume_driver: cinder.volume.drivers.nfs.NfsDriver nfs_mount_options: "rsize=65535,wsize=65535,timeo=1200,actimeo=120" nfs_shares_config: /etc/cinder/nfs_shares_volume2 shares: - { ip: "172.29.244.27", share: "/NFS_VOLUME2" } infra38: ip: 172.29.236.38 container_vars: cinder_backends: nfs_volume3: volume_backend_name: NFS_VOLUME3 volume_driver: cinder.volume.drivers.nfs.NfsDriver nfs_mount_options: "rsize=65535,wsize=65535,timeo=1200,actimeo=120" nfs_shares_config: /etc/cinder/nfs_shares_volume3 shares: - { ip: "172.29.244.27", share: "/NFS_VOLUME3" }

Any advice would be greatly appreciated.

Thanks.

-Dave

-- Dave Hall Binghamton University kdhall@binghamton.edu

I have actually a question on unrelated topic, but related to the active/standby of cinder-volume. Is there any way to explicitly tell to the backend that it should be active? As well as would it restore active mode if at some point become unreachable and then restore? Recently I tried to look through docs and config options, but was not able to find an answer. вс, 7 авг. 2022 г., 17:09 Takashi Kajinami <tkajinam@redhat.com>:

- Usually you need to deploy cinder-volume as an act/sby process. Your cinder-volume would run on only one of the controller nodes managed by a cluster technology like pacemaker. You can split cinder-volume per backend but in such case you'd run each c-vol in act/sby mode.

On Sun, Aug 7, 2022 at 8:56 AM Dmitriy Rabotyagov <noonedeadpunk@gmail.com> wrote:

Another question - are features that tooz support with etcd enough for cinder active/active to work?

Here's some general info about running cinder-volume in A/A mode: - Very few drivers support A/A. Drivers need to explicitly declare their support, and there's no conformance test to ensure A/A mode actually works (drivers merely claim it should work). The RBD driver definitely supports A/A, and NFS is currently *not* supported. - You need to configure two things for a driver to run A/A; 1) configure the 'cluster' setting in cinder.conf, 2) you need to configure a suitable DLM for the [coordination]backend_url in cinder.conf. From a DLM perspective, your best option is tooz's etcd3gw driver. Bear in mind the etcd3 driver is deprecated [1]. [1] https://review.opendev.org/c/openstack/tooz/+/833107

IIRC, for NFS openstack-ansible deploy in active/standby mode by default. Only ceph goes "active/active" but we don't define coordination url for that - only define cluster_name...

As I mentioned above, you definitely need to configure the coordination url with a suitable DLM for A/A to work properly. Things may seem to work when doing some light testing, but the cinder-volume services will eventually start stepping all over each other once the activity load increases. Alan

Though, we do provide a role/way to get etcd, so no blocker to get coordination_url for cinder if etcd covers all cinder needs

вс, 7 авг. 2022 г., 17:36 Dmitriy Rabotyagov <noonedeadpunk@gmail.com>:

...

I have actually a question on unrelated topic, but related to the active/standby of cinder-volume.

Is there any way to explicitly tell to the backend that it should be active? As well as would it restore active mode if at some point become unreachable and then restore? Recently I tried to look through docs and config options, but was not able to find an answer.

вс, 7 авг. 2022 г., 17:09 Takashi Kajinami <tkajinam@redhat.com>:

...

- Usually you need to deploy cinder-volume as an act/sby process. Your cinder-volume would run on only one of the controller nodes managed by a cluster technology like pacemaker. You can split cinder-volume per backend but in such case you'd run each c-vol in act/sby mode.

Thank you both for your responses and insight. It is very much appreciated. Dmitriy, Is there any sort of document/webpage about OpenStack-Ansible Internals? I didn't know anything about the scripts/inventory-manage.py script. I will study it further and get a full understanding. I'd also be interested to know how OpenStack-Ansible is put together - how the inventory is generated and managed, how the playbooks in /etc/ansible are managed and interpreted, etc. I'm also going to be wondering how to clean up the cinder-volume processes running on metal - until I get this active/active vs. active/standby sorted out I may need to drop to a single cinder-volume process. What's important for me right now is to be able to deploy instances ASAP and sort out the resilience issues later. Takashi, Thank you for clearing up my misconceptions and helping me to understand cinder's logical structure. In the long run I will definitely aim for a 3-way active/active deployment. -Dave -- Dave Hall Binghamton University kdhall@binghamton.edu On Sun, Aug 7, 2022 at 11:10 AM Takashi Kajinami <tkajinam@redhat.com> wrote:

I'm not familiar with the setup created by openstack-ansible but let me leave a few comments from cinder's perspective.

- cinder api does not require access to backend storage. Only cinder-volume communicates with backend storage. Thus missing nfs mount/tools in cinder-api is what we definitely expect.

- Usually you need to deploy cinder-volume as an act/sby process. Your cinder-volume would run on only one of the controller nodes managed by a cluster technology like pacemaker. You can split cinder-volume per backend but in such case you'd run each c-vol in act/sby mode.

- Recent cinder supports deploying cinder-volume in active/active mode with lock management by tooz. I'm not sure whether this is supported by openstack-ansible as well as it is well tested with NFS backend.

- When you deploy cinder-volume in active/active mode, cinder-volume should run in all controller nodes and all processes should have access to the same storage backends. If you use different backend setting for each controller then you'd lost control about some of your volumes when the associated controller node goes down.

I'd recommend checking whether you expect deploying cinder-volume in active/standby or in active/active. In case you want active/active then you should have multiple cinder-volume processes with the same backend definition.

On Sun, Aug 7, 2022 at 11:55 PM Dmitriy Rabotyagov < noonedeadpunk@gmail.com> wrote:

...

Hi,

1. When you remove definition from openstack_user_config, items from inventory won't be dropped automatically. For that you would need to use scripts/inventory-manage.py -r <container_name>. You should indeed destroy containers first.

2. In order to remove cider-volume definitions from API, you would need to run cinder-manage service remove <service> <host>

https://docs.openstack.org/cinder/rocky/man/cinder-manage.html#cinder-servic...

cinder-manage binary can be found on cinder-api containers inside virtualenv, ie /openstack/venvs/cinder-<venv_tag>/bin/cinder-manage

вс, 7 авг. 2022 г., 16:22 Dave Hall <kdhall@binghamton.edu>:

...

Hello,

Please pardon the repost - I noticed this morning that I didn't finish the subject line.

Problem summary: I have a bunch of lingering non-functional cinder definitions and I'm looking for guidance on how to clean them up.

Thanks.

-Dave

-- Dave Hall Binghamton University kdhall@binghamton.edu

On Sat, Aug 6, 2022 at 2:52 PM Dave Hall <kdhall@binghamton.edu> wrote:

...

Hello,

I seem to have gotten myself in a bit of a mess trying to set up Cinder with an NFS back-end. After working with Glance and NFS, I started on Cinder. I noticed immediately that there weren't any NFS mounts in the Cinder-API containers like there were in the Glance-API containers. Also that there were no NFS packages in the Cinder-API containers.

In reading some Cinder documentation, I also got the impression that each Cinder host/container needs to have its own NFS store.

Pawing through the playbooks and documentation I saw that unlike Glance, Cinder is split into two pieces - Cinder-API and Cinder-Volume. I found cinder-volume.yml.example in env.d, activated it, and created Cinder-Volume containers on my 3 infra hosts. I also created 3 separate NFS shares and changed the storage-hosts section of my openstack_user_config.yml accordingly.

After this I found that while I was able to create volumes, the prep_volume part of launching an instance was failing.

Digging in, I found:

# openstack volume service list

+------------------+-------------------------------------------------------+------+---------+-------+----------------------------+ | Binary | Host | Zone | Status | State | Updated At |

+------------------+-------------------------------------------------------+------+---------+-------+----------------------------+ | cinder-volume | C6220-9@nfs_volume | nova | enabled | down | 2022-07-23T02:46:13.000000 | | cinder-volume | C6220-10@nfs_volume | nova | enabled | down | 2022-07-23T02:46:14.000000 | | cinder-volume | C6220-11@nfs_volume | nova | enabled | down | 2022-07-23T02:46:14.000000 | | cinder-scheduler | infra36-cinder-api-container-da8e100f | nova | enabled | up | 2022-08-06T13:29:10.000000 | | cinder-scheduler | infra38-cinder-api-container-27219f93 | nova | enabled | up | 2022-08-06T13:29:10.000000 | | cinder-scheduler | infra37-cinder-api-container-ea7f847b | nova | enabled | up | 2022-08-06T13:29:10.000000 | | cinder-volume | C6220-9@nfs_volume1 | nova | enabled | up | 2022-08-06T13:29:10.000000 | | cinder-volume | infra37-cinder-volumes-container-5b9635ad@nfs_volume | nova | enabled | down | 2022-08-04T18:32:53.000000 | | cinder-volume | infra36-cinder-volumes-container-77190057@nfs_volume1 | nova | enabled | down | 2022-08-06T13:03:03.000000 | | cinder-volume | infra38-cinder-volumes-container-a7bcfc9b@nfs_volume | nova | enabled | down | 2022-08-04T18:32:53.000000 | | cinder-volume | infra37-cinder-volumes-container-5b9635ad@nfs_volume2 | nova | enabled | down | 2022-08-06T13:03:05.000000 | | cinder-volume | C6220-10@nfs_volume2 | nova | enabled | up | 2022-08-06T13:29:10.000000 | | cinder-volume | C6220-11@nfs_volume3 | nova | enabled | up | 2022-08-06T13:29:10.000000 | | cinder-volume | infra38-cinder-volumes-container-a7bcfc9b@nfs_volume3 | nova | enabled | down | 2022-08-06T13:03:03.000000 |

+------------------+-------------------------------------------------------+------+---------+-------+----------------------------+

Thinking I could save this, I used containers-lxc-destroy.yml to destroy my cinder-volumes containers and deactivated cinder-volume.yml.example. Then I ran setup-hosts.yml, which has restored the cinder-volumes containers even though is_metal: false has been removed.

Clearly a stronger intervention will be required. I would like to fully get rid of the cinder-volumes containers and go back to an is_metal: true scenario. I also need to get rid of the unnumbered nfs_volume referenes, which I assume are in some cinder config file somewhere.

Below is a clip from my openstack_user_config.yml:

storage_hosts: infra36: ip: 172.29.236.36 container_vars: cinder_backends: nfs_volume1: volume_backend_name: NFS_VOLUME1 volume_driver: cinder.volume.drivers.nfs.NfsDriver nfs_mount_options: "rsize=65535,wsize=65535,timeo=1200,actimeo=120" nfs_shares_config: /etc/cinder/nfs_shares_volume1 shares: - { ip: "172.29.244.27", share: "/NFS_VOLUME1" } infra37: ip: 172.29.236.37 container_vars: cinder_backends: nfs_volume2: volume_backend_name: NFS_VOLUME2 volume_driver: cinder.volume.drivers.nfs.NfsDriver nfs_mount_options: "rsize=65535,wsize=65535,timeo=1200,actimeo=120" nfs_shares_config: /etc/cinder/nfs_shares_volume2 shares: - { ip: "172.29.244.27", share: "/NFS_VOLUME2" } infra38: ip: 172.29.236.38 container_vars: cinder_backends: nfs_volume3: volume_backend_name: NFS_VOLUME3 volume_driver: cinder.volume.drivers.nfs.NfsDriver nfs_mount_options: "rsize=65535,wsize=65535,timeo=1200,actimeo=120" nfs_shares_config: /etc/cinder/nfs_shares_volume3 shares: - { ip: "172.29.244.27", share: "/NFS_VOLUME3" }

Any advice would be greatly appreciated.

Thanks.

-Dave

-- Dave Hall Binghamton University kdhall@binghamton.edu