<div dir="ltr"><div>Thank you both for your responses and insight. It is very much appreciated.</div><div><br></div><div>Dmitriy,</div><div><br></div><div>Is there any sort of document/webpage about OpenStack-Ansible Internals? I didn't know anything about the scripts/inventory-manage.py script. I will study it further and get a full understanding. I'd also be interested to know how OpenStack-Ansible is put together - how the inventory is generated and managed, how the playbooks in /etc/ansible are managed and interpreted, etc. </div><div><br></div><div>I'm also going to be wondering how to clean up the cinder-volume processes running on metal - until I get this active/active vs. active/standby sorted out I may need to drop to a single cinder-volume process. What's important for me right now is to be able to deploy instances ASAP and sort out the resilience issues later.</div><div><br></div><div>Takashi,</div><div><br></div><div>Thank you for clearing up my misconceptions and helping me to understand cinder's logical structure. In the long run I will definitely aim for a 3-way active/active deployment. </div><div dir="ltr"><br></div><div>-Dave</div><div dir="ltr"><br clear="all"><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>--</div><div>Dave Hall<br>Binghamton University<br><a href="mailto:kdhall@binghamton.edu" target="_blank">kdhall@binghamton.edu</a><br></div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Aug 7, 2022 at 11:10 AM Takashi Kajinami <<a href="mailto:tkajinam@redhat.com">tkajinam@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>I'm not familiar with the setup created by openstack-ansible but let me leave a few comments</div><div>from cinder's perspective.</div><div><br></div><div>- cinder api does not require access to backend storage. Only cinder-volume communicates with</div><div> backend storage. Thus missing nfs mount/tools in cinder-api is what we definitely expect.</div><div><br></div><div>- Usually you need to deploy cinder-volume as an act/sby process. Your cinder-volume would run</div><div> on only one of the controller nodes managed by a cluster technology like pacemaker.<br></div><div> You can split cinder-volume per backend but in such case you'd run each c-vol in act/sby mode.</div><div><br></div><div>- Recent cinder supports deploying cinder-volume in active/active mode with lock management by</div><div> tooz. I'm not sure whether this is supported by openstack-ansible as well as it is well tested with</div><div> NFS backend.</div><div><br></div><div>- When you deploy cinder-volume in active/active mode, cinder-volume should run in all controller nodes</div><div> and all processes should have access to the same storage backends. If you use different backend setting</div><div> for each controller then you'd lost control about some of your volumes when the associated controller node</div><div> goes down.<br></div><div><br></div><div>I'd recommend checking whether you expect deploying cinder-volume in active/standby or in</div><div>active/active. In case you want active/active then you should have multiple cinder-volume processes</div><div>with the same backend definition.<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Aug 7, 2022 at 11:55 PM Dmitriy Rabotyagov <<a href="mailto:noonedeadpunk@gmail.com" target="_blank">noonedeadpunk@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto"><div>Hi,<div dir="auto"><br></div><div dir="auto">1. When you remove definition from openstack_user_config, items from inventory won't be dropped automatically. For that you would need to use scripts/inventory-manage.py -r <container_name>. You should indeed destroy containers first.</div><div dir="auto"><br></div><div dir="auto">2. In order to remove cider-volume definitions from API, you would need to run <span style="font-family:menlo,monaco,consolas,"courier new",monospace;font-size:14px;font-weight:700;background-color:rgb(255,255,255)">cinder-manage</span><span style="font-family:menlo,monaco,consolas,"courier new",monospace;font-size:14px;font-weight:700;background-color:rgb(255,255,255)"> </span><span style="font-family:menlo,monaco,consolas,"courier new",monospace;font-size:14px;font-weight:700;background-color:rgb(255,255,255)">service</span><span style="font-family:menlo,monaco,consolas,"courier new",monospace;font-size:14px;font-weight:700;background-color:rgb(255,255,255)"> </span><span style="font-family:menlo,monaco,consolas,"courier new",monospace;font-size:14px;font-weight:700;background-color:rgb(255,255,255)">remove</span><span style="font-family:menlo,monaco,consolas,"courier new",monospace;font-size:14px;font-weight:700;background-color:rgb(255,255,255)"> </span><span style="font-family:menlo,monaco,consolas,"courier new",monospace;font-size:14px;font-weight:700;background-color:rgb(255,255,255)"><service></span><span style="font-family:menlo,monaco,consolas,"courier new",monospace;font-size:14px;font-weight:700;background-color:rgb(255,255,255)"> </span><span style="font-family:menlo,monaco,consolas,"courier new",monospace;font-size:14px;font-weight:700;background-color:rgb(255,255,255)"><host></span></div><div dir="auto"><span style="font-family:menlo,monaco,consolas,"courier new",monospace;font-size:14px;font-weight:700;background-color:rgb(255,255,255)"><br></span></div><div dir="auto"><a href="https://docs.openstack.org/cinder/rocky/man/cinder-manage.html#cinder-service" target="_blank">https://docs.openstack.org/cinder/rocky/man/cinder-manage.html#cinder-service</a></div><br>cinder-manage binary can be found on cinder-api containers inside virtualenv, ie /openstack/venvs/cinder-<venv_tag>/bin/cinder-manage</div><div dir="auto"><br></div><div dir="auto"><br><div class="gmail_quote" dir="auto"><div dir="ltr" class="gmail_attr">вс, 7 авг. 2022 г., 16:22 Dave Hall <<a href="mailto:kdhall@binghamton.edu" target="_blank">kdhall@binghamton.edu</a>>:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">Hello,<br></div><div dir="ltr"> </div><div>Please pardon the repost - I noticed this morning that I didn't finish the subject line. </div><div><br></div><div>Problem summary: I have a bunch of lingering non-functional cinder definitions and I'm looking for guidance on how to clean them up.</div><div><br></div><div>Thanks.</div><div><br></div><div>-Dave</div><div dir="ltr"><br clear="all"><div><div dir="ltr"><div dir="ltr"><div>--</div><div>Dave Hall<br>Binghamton University<br><a href="mailto:kdhall@binghamton.edu" rel="noreferrer" target="_blank">kdhall@binghamton.edu</a><br><br></div></div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Aug 6, 2022 at 2:52 PM Dave Hall <<a href="mailto:kdhall@binghamton.edu" rel="noreferrer" target="_blank">kdhall@binghamton.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hello,</div><div><br></div><div>I seem to have gotten myself in a bit of a mess trying to set up Cinder with an NFS back-end. After working with Glance and NFS, I started on Cinder. I noticed immediately that there weren't any NFS mounts in the Cinder-API containers like there were in the Glance-API containers. Also that there were no NFS packages in the Cinder-API containers. <br></div><div><br></div><div>In reading some Cinder documentation, I also got the impression that each Cinder host/container needs to have its own NFS store. <br></div><div><br></div><div>Pawing through the playbooks and documentation I saw that unlike Glance, Cinder is split into two pieces - Cinder-API and Cinder-Volume. I found cinder-volume.yml.example in env.d, activated it, and created Cinder-Volume containers on my 3 infra hosts. I also created 3 separate NFS shares and changed the storage-hosts section of my openstack_user_config.yml accordingly.</div><div><br></div><div>After this I found that while I was able to create volumes, the prep_volume part of launching an instance was failing. <br></div><div><br></div><div>Digging in, I found:</div><div style="margin-left:40px"><br></div><div style="margin-left:40px"><span style="font-family:monospace"># openstack volume service list<br>+------------------+-------------------------------------------------------+------+---------+-------+----------------------------+<br>| Binary | Host | Zone | Status | State | Updated At |<br>+------------------+-------------------------------------------------------+------+---------+-------+----------------------------+<br>| cinder-volume | C6220-9@nfs_volume | nova | enabled | down | 2022-07-23T02:46:13.000000 |<br>| cinder-volume | C6220-10@nfs_volume | nova | enabled | down | 2022-07-23T02:46:14.000000 |<br>| cinder-volume | C6220-11@nfs_volume | nova | enabled | down | 2022-07-23T02:46:14.000000 |<br>| cinder-scheduler | infra36-cinder-api-container-da8e100f | nova | enabled | up | 2022-08-06T13:29:10.000000 |<br>| cinder-scheduler | infra38-cinder-api-container-27219f93 | nova | enabled | up | 2022-08-06T13:29:10.000000 |<br>| cinder-scheduler | infra37-cinder-api-container-ea7f847b | nova | enabled | up | 2022-08-06T13:29:10.000000 |<br>| cinder-volume | C6220-9@nfs_volume1 | nova | enabled | up | 2022-08-06T13:29:10.000000 |<br>| cinder-volume | infra37-cinder-volumes-container-5b9635ad@nfs_volume | nova | enabled | down | 2022-08-04T18:32:53.000000 |<br>| cinder-volume | infra36-cinder-volumes-container-77190057@nfs_volume1 | nova | enabled | down | 2022-08-06T13:03:03.000000 |<br>| cinder-volume | infra38-cinder-volumes-container-a7bcfc9b@nfs_volume | nova | enabled | down | 2022-08-04T18:32:53.000000 |<br>| cinder-volume | infra37-cinder-volumes-container-5b9635ad@nfs_volume2 | nova | enabled | down | 2022-08-06T13:03:05.000000 |<br>| cinder-volume | C6220-10@nfs_volume2 | nova | enabled | up | 2022-08-06T13:29:10.000000 |<br>| cinder-volume | C6220-11@nfs_volume3 | nova | enabled | up | 2022-08-06T13:29:10.000000 |<br>| cinder-volume | infra38-cinder-volumes-container-a7bcfc9b@nfs_volume3 | nova | enabled | down | 2022-08-06T13:03:03.000000 |<br>+------------------+-------------------------------------------------------+------+---------+-------+----------------------------+</span><br></div><div><br></div><div>Thinking I could save this, I used containers-lxc-destroy.yml to destroy my cinder-volumes containers and deactivated cinder-volume.yml.example. Then I ran setup-hosts.yml, which has restored the cinder-volumes containers even though is_metal: false has been removed.</div><div><br></div><div>Clearly a stronger intervention will be required. I would like to fully get rid of the cinder-volumes containers and go back to an is_metal: true scenario. I also need to get rid of the unnumbered nfs_volume referenes, which I assume are in some cinder config file somewhere. <br></div><div><br></div><div>Below is a clip from my openstack_user_config.yml:</div><div><br></div><div style="margin-left:40px"><span style="font-family:monospace">storage_hosts:<br> infra36:<br> ip: 172.29.236.36<br> container_vars:<br> cinder_backends:<br> nfs_volume1:<br> volume_backend_name: NFS_VOLUME1<br> volume_driver: cinder.volume.drivers.nfs.NfsDriver<br> nfs_mount_options: "rsize=65535,wsize=65535,timeo=1200,actimeo=120"<br> nfs_shares_config: /etc/cinder/nfs_shares_volume1<br> shares:<br> - { ip: "172.29.244.27", share: "/NFS_VOLUME1" }<br> infra37:<br> ip: 172.29.236.37<br> container_vars:<br> cinder_backends:<br> nfs_volume2:<br> volume_backend_name: NFS_VOLUME2<br> volume_driver: cinder.volume.drivers.nfs.NfsDriver<br> nfs_mount_options: "rsize=65535,wsize=65535,timeo=1200,actimeo=120"<br> nfs_shares_config: /etc/cinder/nfs_shares_volume2<br> shares:<br> - { ip: "172.29.244.27", share: "/NFS_VOLUME2" }<br> infra38:<br> ip: 172.29.236.38<br> container_vars:<br> cinder_backends:<br> nfs_volume3:<br> volume_backend_name: NFS_VOLUME3<br> volume_driver: cinder.volume.drivers.nfs.NfsDriver<br> nfs_mount_options: "rsize=65535,wsize=65535,timeo=1200,actimeo=120"<br> nfs_shares_config: /etc/cinder/nfs_shares_volume3<br> shares:<br> - { ip: "172.29.244.27", share: "/NFS_VOLUME3" }<br></span></div><div><br></div><div>Any advice would be greatly appreciated. <br></div><div><br></div><div>Thanks.</div><div><br></div><div>-Dave<br></div><div><br></div><div><div dir="ltr"><div dir="ltr"><div>--</div><div>Dave Hall<br>Binghamton University<br><a href="mailto:kdhall@binghamton.edu" rel="noreferrer" target="_blank">kdhall@binghamton.edu</a><br><br></div></div></div></div></div>
</blockquote></div></div>
</blockquote></div></div></div>
</blockquote></div>
</blockquote></div></div>