Cinder / Nova - Select cinder backend based on nova availability zone?
Apologies in advance if this seems trivial but I am looking for some direction on this and I may have found a bug while testing also. Some background - I have 2 physical hosts which I am testing with (host1, host2) and I have 2 separate cinder backends (backend1, backend2). Backend1 can only be utilised by host1. Same for backend2 - it can only be utilised by host2. So they are paired together like: host1:backend1 host2:backend2 So I wanted to select a Cinder storage back-end based on nova availability zone and to do this when creating an instance through horizon (not creating a volume directly). Also I wanted to avoid the use of metadata input on each instance create or by using metadata from images (such as cinder_img_volume_type) [2] . Because I can foresee a necessity to then have a number of images which reference each AZ or backend individually. - Is it possible to select a backend based on nova AZ? If so, could anyone share any resources to me that could help me understand how to achieve it? Because I failed at achieving the above, I then decided to use one way which had worked for me in the past, which was to use the image metadata "cinder_img_volume_type". However I find that this is not working. The “default” volume type is selected (if cinder.conf has it) or if no default, then `__DEFAULT__` is being selected. The link at [2] states that first, a volume type is used based on the volume type selected and if not chosen/set then the 2nd method is "cinder_img_volume_type" from the image metadata and then the 3rd and final is the default from cinder.conf. I have tested with fresh deployment using Kayobe as well as RDO’s packstack. Openstack version is *Train* Steps to reproduce: 1. Install packstack 2. Update cinder.conf with enabled_backends and the [backend] 3. Add the volume type to reference the backend (for reference, I call this volume type `number-1`) 4. Upload an image and add metadata `cinder_img_volume_type` and the name as mentioned in step 3: number-1 5. Try and create an instance using horizon. Source = image and create new volume 6. Result = volume type / backend as chosen in the image metadata is not used and instance goes into error status. After fresh-deploying the RDO Packstack, I enabled debug logs and tested again. In the cinder-api.log I see “"volume_type": null,” and then the next debug log immediately after logged as “Create volume request body:” has “volume_type': None”. I was searching for a list of the supported image metadata, in case it had changed but the pages seem empty one rocky/stein/train [3] or not yet updated. Selecting backend based on nova AZ:: I was searching how to achieve this and I came across this video on the subject of AZs [1]. Although it seems only in the context of creating volumes (not with creating instances with volume from an image, for example). I have tried creating a host aggregate in nova, with AZ name `host1az`. I've also created a backend in Cinder (cinder.conf) with `backend_availability_zone = host1az`. But this does not appear to achieve the desired result, either and the cinder api logs are showing “"availability_zone": null” during the volume create part of the launch instance from Horizon. I also tried setting RESKEY [3] in the volume type, but again similar situation seen; although I dont think this option is the correct context for what I am attempting. Could anyone please nudge me in the right direction on this? Any pointers appreciated at this point. Thanks in advance. References: [1] https://www.youtube.com/watch?v=a5332_Ew9JA [2] https://docs.openstack.org/cinder/latest/cli/cli-manage-volumes.html [3] https://docs.openstack.org/cinder/train/contributor/api/cinder.api.schemas.v... [4] https://docs.openstack.org/cinder/rocky/admin/blockstorage-availability-zone... Regards, Tony Pearce
Hi Rui, thank you for the reply. I did find that link previously but I was unable to get it working. Because I am unable to get it working, I tried to use the `cinder_img_volume_type` in the glance image metadata to select a backend (which is also not working - bug: https://bugs.launchpad.net/cinder/+bug/1864616) When I try and use the RESKEY:availability_zones in the volume type it does not work for me and what is happening is that the volume type of `__DEFAULT__` is being selected. I peformed these steps to test, are these steps correct? 1. create the 2 cinder backends 2. create 2 volume types (one for each backend). 3. add the RESKEY:availability_zones extra specs to the volume type, like: RESKEY:availability_zones="host1" for the first volume type and RESKEY:availability_zones="host2" for the 2nd volume type 4. create 2 host aggrigates with AZ "host1" and "host2" for each host1 / host2 and assign a host to host1 5. try to launch an instance and choose the availability zone "host1" from the dropdown 6. result = instance goes error state volume goes into in-use volume shows "host" as storage@backend and the "backend" is incorrectly chosen In addition to that, the "volume type" selected (which I expected to be host1) is showing as __DEFAUT__ At this point I am not sure if I am doing something incorrect and this is the reason why it is not working, or, I have correct config and this is a bug experience. I'd be grateful if you or anyone else that reads this could confirm. Thanks and regards *Tony Pearce* | *Senior Network Engineer / Infrastructure Lead**Cinglevue International <https://www.cinglevue.com>* Email: tony.pearce@cinglevue.com Web: http://www.cinglevue.com *Australia* 1 Walsh Loop, Joondalup, WA 6027 Australia. Direct: +61 8 6202 0036 | Main: +61 8 6202 0024 Note: This email and all attachments are the sole property of Cinglevue International Pty Ltd. (or any of its subsidiary entities), and the information contained herein must be considered confidential, unless specified otherwise. If you are not the intended recipient, you must not use or forward the information contained in these documents. If you have received this message in error, please delete the email and notify the sender. On Tue, 25 Feb 2020 at 10:24, rui zang <rui.zang@yandex.com> wrote:
I don't think there is auto-alignment between nova az and cinder az. Probably you may want to look at this https://docs.openstack.org/cinder/rocky/admin/blockstorage-availability-zone... Cinder AZ can be associated with volume types. When you create a VM, you can specify the VM nova az (eg. host1), volume type (cinder az, eg, also host1), thus by aligning nova az and cinder az manually, maybe your goal can be achieved. I believe there are also other similar configuration ways to make it work like this.
Thanks, Zang, Rui
24.02.2020, 16:53, "Tony Pearce" <tony.pearce@cinglevue.com>:
Apologies in advance if this seems trivial but I am looking for some direction on this and I may have found a bug while testing also.
Some background - I have 2 physical hosts which I am testing with (host1, host2) and I have 2 separate cinder backends (backend1, backend2). Backend1 can only be utilised by host1. Same for backend2 - it can only be utilised by host2. So they are paired together like: host1:backend1 host2:backend2
So I wanted to select a Cinder storage back-end based on nova availability zone and to do this when creating an instance through horizon (not creating a volume directly). Also I wanted to avoid the use of metadata input on each instance create or by using metadata from images (such as cinder_img_volume_type) [2] . Because I can foresee a necessity to then have a number of images which reference each AZ or backend individually.
- Is it possible to select a backend based on nova AZ? If so, could anyone share any resources to me that could help me understand how to achieve it?
Because I failed at achieving the above, I then decided to use one way which had worked for me in the past, which was to use the image metadata "cinder_img_volume_type". However I find that this is not working. The “default” volume type is selected (if cinder.conf has it) or if no default, then `__DEFAULT__` is being selected. The link at [2] states that first, a volume type is used based on the volume type selected and if not chosen/set then the 2nd method is "cinder_img_volume_type" from the image metadata and then the 3rd and final is the default from cinder.conf.
I have tested with fresh deployment using Kayobe as well as RDO’s packstack.
Openstack version is *Train*
Steps to reproduce: 1. Install packstack
2. Update cinder.conf with enabled_backends and the [backend]
3. Add the volume type to reference the backend (for reference, I call this volume type `number-1`)
4. Upload an image and add metadata `cinder_img_volume_type` and the name as mentioned in step 3: number-1
5. Try and create an instance using horizon. Source = image and create new volume
6. Result = volume type / backend as chosen in the image metadata is not used and instance goes into error status.
After fresh-deploying the RDO Packstack, I enabled debug logs and tested again. In the cinder-api.log I see “"volume_type": null,” and then the next debug log immediately after logged as “Create volume request body:” has “volume_type': None”.
I was searching for a list of the supported image metadata, in case it had changed but the pages seem empty one rocky/stein/train [3] or not yet updated.
Selecting backend based on nova AZ::
I was searching how to achieve this and I came across this video on the subject of AZs [1]. Although it seems only in the context of creating volumes (not with creating instances with volume from an image, for example).
I have tried creating a host aggregate in nova, with AZ name `host1az`. I've also created a backend in Cinder (cinder.conf) with `backend_availability_zone = host1az`. But this does not appear to achieve the desired result, either and the cinder api logs are showing “"availability_zone": null” during the volume create part of the launch instance from Horizon.
I also tried setting RESKEY [3] in the volume type, but again similar situation seen; although I dont think this option is the correct context for what I am attempting.
Could anyone please nudge me in the right direction on this? Any pointers appreciated at this point. Thanks in advance.
References:
[1] *https://www.youtube.com/watch?v=a5332_Ew9JA* <https://www.youtube.com/watch?v=a5332_Ew9JA>
[2] *https://docs.openstack.org/cinder/latest/cli/cli-manage-volumes.html* <https://docs.openstack.org/cinder/latest/cli/cli-manage-volumes.html>
[3] *https://docs.openstack.org/cinder/train/contributor/api/cinder.api.schemas.v... <https://docs.openstack.org/cinder/train/contributor/api/cinder.api.schemas.volume_image_metadata.html>
[4] *https://docs.openstack.org/cinder/rocky/admin/blockstorage-availability-zone... <https://docs.openstack.org/cinder/rocky/admin/blockstorage-availability-zone-type.html>
Regards,
*Tony Pearce*
Hi Rui, Many thanks for sending over that information to me. It looks like the missing piece of the puzzle was nova.conf `cross_az_attach`. I also reached out to the IRC groups and want to say thank you for the help in the #openstack-cinder. The Mirantis blog was very helpful (and also mentions my exact use-case) but also those docs.openstack links mention some bugs with the `cross_az_attach`. I've applied the config, and tests show that it works great and the desired outcome is achieved. But I'm also going to look into and try the other suggestions from the mirantis blog page. Thanks again for your reply on this :) Regards, *Tony Pearce* On Wed, 26 Feb 2020 at 10:57, rui zang <rui.zang@yandex.com> wrote:
Hi Tony,
I do not have an environment at hand to verify. So I was replying based on what I understood. However I dig a little further and found out there is indeed some alignment between nova AZ and cinder AZ.
https://docs.openstack.org/nova/latest/admin/availability-zones.html#resourc... More specifically https://docs.openstack.org/nova/latest/configuration/config.html#cinder.cros...
And also this helped me a lot in understanding the whole picture : https://www.mirantis.com/blog/the-first-and-final-word-on-openstack-availabi...
Hope it helps.
Thanks, Zang, Rui
25.02.2020, 16:56, "Tony Pearce" <tony.pearce@cinglevue.com>:
Hi Rui, thank you for the reply. I did find that link previously but I was unable to get it working. Because I am unable to get it working, I tried to use the `cinder_img_volume_type` in the glance image metadata to select a backend (which is also not working - bug: https://bugs.launchpad.net/cinder/+bug/1864616)
When I try and use the RESKEY:availability_zones in the volume type it does not work for me and what is happening is that the volume type of `__DEFAULT__` is being selected.
I peformed these steps to test, are these steps correct?
1. create the 2 cinder backends 2. create 2 volume types (one for each backend). 3. add the RESKEY:availability_zones extra specs to the volume type, like: RESKEY:availability_zones="host1" for the first volume type and RESKEY:availability_zones="host2" for the 2nd volume type 4. create 2 host aggrigates with AZ "host1" and "host2" for each host1 / host2 and assign a host to host1 5. try to launch an instance and choose the availability zone "host1" from the dropdown
6. result = instance goes error state volume goes into in-use volume shows "host" as storage@backend and the "backend" is incorrectly chosen In addition to that, the "volume type" selected (which I expected to be host1) is showing as __DEFAUT__
At this point I am not sure if I am doing something incorrect and this is the reason why it is not working, or, I have correct config and this is a bug experience.
I'd be grateful if you or anyone else that reads this could confirm.
Thanks and regards
*Tony Pearce* | *Senior Network Engineer / Infrastructure LeadCinglevue International <https://www.cinglevue.com/>*
Email: tony.pearce@cinglevue.com Web: http://www.cinglevue.com
*Australia* 1 Walsh Loop, Joondalup, WA 6027 Australia.
Direct: +61 8 6202 0036 | Main: +61 8 6202 0024
Note: This email and all attachments are the sole property of Cinglevue International Pty Ltd. (or any of its subsidiary entities), and the information contained herein must be considered confidential, unless specified otherwise. If you are not the intended recipient, you must not use or forward the information contained in these documents. If you have received this message in error, please delete the email and notify the sender.
On Tue, 25 Feb 2020 at 10:24, rui zang <rui.zang@yandex.com> wrote:
I don't think there is auto-alignment between nova az and cinder az. Probably you may want to look at this https://docs.openstack.org/cinder/rocky/admin/blockstorage-availability-zone... Cinder AZ can be associated with volume types. When you create a VM, you can specify the VM nova az (eg. host1), volume type (cinder az, eg, also host1), thus by aligning nova az and cinder az manually, maybe your goal can be achieved. I believe there are also other similar configuration ways to make it work like this.
Thanks, Zang, Rui
24.02.2020, 16:53, "Tony Pearce" <tony.pearce@cinglevue.com>:
Apologies in advance if this seems trivial but I am looking for some direction on this and I may have found a bug while testing also.
Some background - I have 2 physical hosts which I am testing with (host1, host2) and I have 2 separate cinder backends (backend1, backend2). Backend1 can only be utilised by host1. Same for backend2 - it can only be utilised by host2. So they are paired together like: host1:backend1 host2:backend2
So I wanted to select a Cinder storage back-end based on nova availability zone and to do this when creating an instance through horizon (not creating a volume directly). Also I wanted to avoid the use of metadata input on each instance create or by using metadata from images (such as cinder_img_volume_type) [2] . Because I can foresee a necessity to then have a number of images which reference each AZ or backend individually.
- Is it possible to select a backend based on nova AZ? If so, could anyone share any resources to me that could help me understand how to achieve it?
Because I failed at achieving the above, I then decided to use one way which had worked for me in the past, which was to use the image metadata "cinder_img_volume_type". However I find that this is not working. The “default” volume type is selected (if cinder.conf has it) or if no default, then `__DEFAULT__` is being selected. The link at [2] states that first, a volume type is used based on the volume type selected and if not chosen/set then the 2nd method is "cinder_img_volume_type" from the image metadata and then the 3rd and final is the default from cinder.conf.
I have tested with fresh deployment using Kayobe as well as RDO’s packstack.
Openstack version is *Train*
Steps to reproduce: 1. Install packstack
2. Update cinder.conf with enabled_backends and the [backend]
3. Add the volume type to reference the backend (for reference, I call this volume type `number-1`)
4. Upload an image and add metadata `cinder_img_volume_type` and the name as mentioned in step 3: number-1
5. Try and create an instance using horizon. Source = image and create new volume
6. Result = volume type / backend as chosen in the image metadata is not used and instance goes into error status.
After fresh-deploying the RDO Packstack, I enabled debug logs and tested again. In the cinder-api.log I see “"volume_type": null,” and then the next debug log immediately after logged as “Create volume request body:” has “volume_type': None”.
I was searching for a list of the supported image metadata, in case it had changed but the pages seem empty one rocky/stein/train [3] or not yet updated.
Selecting backend based on nova AZ::
I was searching how to achieve this and I came across this video on the subject of AZs [1]. Although it seems only in the context of creating volumes (not with creating instances with volume from an image, for example).
I have tried creating a host aggregate in nova, with AZ name `host1az`. I've also created a backend in Cinder (cinder.conf) with `backend_availability_zone = host1az`. But this does not appear to achieve the desired result, either and the cinder api logs are showing “"availability_zone": null” during the volume create part of the launch instance from Horizon.
I also tried setting RESKEY [3] in the volume type, but again similar situation seen; although I dont think this option is the correct context for what I am attempting.
Could anyone please nudge me in the right direction on this? Any pointers appreciated at this point. Thanks in advance.
References:
[1] *https://www.youtube.com/watch?v=a5332_Ew9JA* <https://www.youtube.com/watch?v=a5332_Ew9JA>
[2] *https://docs.openstack.org/cinder/latest/cli/cli-manage-volumes.html* <https://docs.openstack.org/cinder/latest/cli/cli-manage-volumes.html>
[3] *https://docs.openstack.org/cinder/train/contributor/api/cinder.api.schemas.v... <https://docs.openstack.org/cinder/train/contributor/api/cinder.api.schemas.volume_image_metadata.html>
[4] *https://docs.openstack.org/cinder/rocky/admin/blockstorage-availability-zone... <https://docs.openstack.org/cinder/rocky/admin/blockstorage-availability-zone-type.html>
Regards,
*Tony Pearce*
participants (2)
-
rui zang
-
Tony Pearce