I'm not familiar with the setup created by openstack-ansible but let me leave a few comments
from cinder's perspective.

- cinder api does not require access to backend storage. Only cinder-volume communicates with
  backend storage. Thus missing nfs mount/tools in cinder-api is what we definitely expect.

- Usually you need to deploy cinder-volume as an act/sby process. Your cinder-volume would  run
  on only one of the controller nodes managed by a cluster technology like pacemaker.
  You can split cinder-volume per backend but in such case you'd run each c-vol in act/sby mode.

- Recent cinder supports deploying cinder-volume in active/active mode with lock management by
  tooz. I'm not sure whether this is supported by openstack-ansible as well as it is well tested with
  NFS backend.

- When you deploy cinder-volume in active/active mode, cinder-volume should run in all controller nodes
  and all processes should have access to the same storage backends. If you use different backend setting
  for each controller then you'd lost control about some of your volumes when the associated controller node
  goes down.

I'd recommend checking whether you expect deploying cinder-volume in active/standby or in
active/active. In case you want active/active then you should have multiple cinder-volume processes
with the same backend definition.

On Sun, Aug 7, 2022 at 11:55 PM Dmitriy Rabotyagov <noonedeadpunk@gmail.com> wrote:
Hi,

1. When you remove definition from openstack_user_config, items from inventory won't be dropped automatically. For that you would need to use scripts/inventory-manage.py -r <container_name>. You should indeed destroy containers first.

2. In order to remove cider-volume definitions from API, you would need to run cinder-manage service remove <service> <host>

https://docs.openstack.org/cinder/rocky/man/cinder-manage.html#cinder-service

cinder-manage binary can be found on cinder-api containers inside virtualenv, ie /openstack/venvs/cinder-<venv_tag>/bin/cinder-manage


вс, 7 авг. 2022 г., 16:22 Dave Hall <kdhall@binghamton.edu>:
Hello,
 
Please pardon the repost - I noticed this morning that I didn't finish the subject line.  

Problem summary:  I have a bunch of lingering non-functional cinder definitions and I'm looking for guidance on how to clean them up.

Thanks.

-Dave

--
Dave Hall
Binghamton University
kdhall@binghamton.edu



On Sat, Aug 6, 2022 at 2:52 PM Dave Hall <kdhall@binghamton.edu> wrote:
Hello,

I seem to have gotten myself in a bit of a mess trying to set up Cinder with an NFS back-end.  After working with Glance and NFS, I started on Cinder.  I noticed immediately that there weren't any NFS mounts in the Cinder-API containers like there were in the Glance-API containers.  Also that there were no NFS packages in the Cinder-API containers. 

In reading some Cinder documentation, I also got the impression that each Cinder host/container needs to have its own NFS store. 

Pawing through the playbooks and documentation I saw that unlike Glance, Cinder is split into two pieces - Cinder-API and Cinder-Volume.  I found cinder-volume.yml.example in env.d, activated it, and created Cinder-Volume containers on my 3 infra hosts.  I also created 3 separate NFS shares and changed the storage-hosts section of my openstack_user_config.yml accordingly.

After this I found that while I was able to create volumes, the prep_volume part of launching an instance was failing. 

Digging in, I found:

# openstack volume service list
+------------------+-------------------------------------------------------+------+---------+-------+----------------------------+
| Binary           | Host                                                  | Zone | Status  | State | Updated At                 |
+------------------+-------------------------------------------------------+------+---------+-------+----------------------------+
| cinder-volume    | C6220-9@nfs_volume                                    | nova | enabled | down  | 2022-07-23T02:46:13.000000 |
| cinder-volume    | C6220-10@nfs_volume                                   | nova | enabled | down  | 2022-07-23T02:46:14.000000 |
| cinder-volume    | C6220-11@nfs_volume                                   | nova | enabled | down  | 2022-07-23T02:46:14.000000 |
| cinder-scheduler | infra36-cinder-api-container-da8e100f                 | nova | enabled | up    | 2022-08-06T13:29:10.000000 |
| cinder-scheduler | infra38-cinder-api-container-27219f93                 | nova | enabled | up    | 2022-08-06T13:29:10.000000 |
| cinder-scheduler | infra37-cinder-api-container-ea7f847b                 | nova | enabled | up    | 2022-08-06T13:29:10.000000 |
| cinder-volume    | C6220-9@nfs_volume1                                   | nova | enabled | up    | 2022-08-06T13:29:10.000000 |
| cinder-volume    | infra37-cinder-volumes-container-5b9635ad@nfs_volume  | nova | enabled | down  | 2022-08-04T18:32:53.000000 |
| cinder-volume    | infra36-cinder-volumes-container-77190057@nfs_volume1 | nova | enabled | down  | 2022-08-06T13:03:03.000000 |
| cinder-volume    | infra38-cinder-volumes-container-a7bcfc9b@nfs_volume  | nova | enabled | down  | 2022-08-04T18:32:53.000000 |
| cinder-volume    | infra37-cinder-volumes-container-5b9635ad@nfs_volume2 | nova | enabled | down  | 2022-08-06T13:03:05.000000 |
| cinder-volume    | C6220-10@nfs_volume2                                  | nova | enabled | up    | 2022-08-06T13:29:10.000000 |
| cinder-volume    | C6220-11@nfs_volume3                                  | nova | enabled | up    | 2022-08-06T13:29:10.000000 |
| cinder-volume    | infra38-cinder-volumes-container-a7bcfc9b@nfs_volume3 | nova | enabled | down  | 2022-08-06T13:03:03.000000 |
+------------------+-------------------------------------------------------+------+---------+-------+----------------------------+

Thinking I could save this, I used containers-lxc-destroy.yml to destroy my cinder-volumes containers and deactivated cinder-volume.yml.example.  Then I ran setup-hosts.yml, which has restored the cinder-volumes containers even though is_metal: false has been removed.

Clearly a stronger intervention will be required.  I would like to fully get rid of the cinder-volumes containers and go back to an is_metal: true scenario.  I also need to get rid of  the unnumbered nfs_volume referenes, which I assume are in some cinder config file somewhere. 

Below is a clip from my openstack_user_config.yml:

storage_hosts:
  infra36:
    ip: 172.29.236.36
    container_vars:
      cinder_backends:
        nfs_volume1:
          volume_backend_name: NFS_VOLUME1
          volume_driver: cinder.volume.drivers.nfs.NfsDriver
          nfs_mount_options: "rsize=65535,wsize=65535,timeo=1200,actimeo=120"
          nfs_shares_config: /etc/cinder/nfs_shares_volume1
          shares:
          - { ip: "172.29.244.27", share: "/NFS_VOLUME1" }
  infra37:
    ip: 172.29.236.37
    container_vars:
      cinder_backends:
        nfs_volume2:
          volume_backend_name: NFS_VOLUME2
          volume_driver: cinder.volume.drivers.nfs.NfsDriver
          nfs_mount_options: "rsize=65535,wsize=65535,timeo=1200,actimeo=120"
          nfs_shares_config: /etc/cinder/nfs_shares_volume2
          shares:
          - { ip: "172.29.244.27", share: "/NFS_VOLUME2" }
  infra38:
    ip: 172.29.236.38
    container_vars:
      cinder_backends:
        nfs_volume3:
          volume_backend_name: NFS_VOLUME3
          volume_driver: cinder.volume.drivers.nfs.NfsDriver
          nfs_mount_options: "rsize=65535,wsize=65535,timeo=1200,actimeo=120"
          nfs_shares_config: /etc/cinder/nfs_shares_volume3
          shares:
          - { ip: "172.29.244.27", share: "/NFS_VOLUME3" }

Any advice would be greatly appreciated. 

Thanks.

-Dave

--
Dave Hall
Binghamton University
kdhall@binghamton.edu