Open Stack

Hello,

First time posting here.
We have been running a production openstack environment at my office since the kilo release. We are currently on train, and I'm trying to get up to a more recent version. To make it more difficult, we are on centos7, so having to switch to ubuntu as we update versions.

The problem that I am having after updaing to victoria, is that when I delete a vm via horizon, the instance disappears but the cinder volume doesn't delete the attachment.
It appears this is due to the following error in /var/log/apache2/cinder_error.log

ERROR cinder.api.middleware.fault novaclient.exceptions.NotAcceptable: Version 2.89 is not supported by the API. Minimum is 2.1 and maximum is 2.87. (HTTP 406)

When I look at the /usr/lib/python3/dist-packages/cinder/compute/nova.py I can see it's using 2.89 in get_server_volume

 def get_server_volume(context, server_id, volume_id):
        # Use microversion that includes attachment_id
        nova = novaclient(context, api_version='2.89')
        return nova.volumes.get_server_volume(server_id, volume_id)

I am not sure why cinder and nova are in disagreement on the api_version.
I have verified that they are both upgraded to the victoria release.

Anyone have any ideas as to why I would be getting this error or a possible fix? I haven't been able to find any information on this error.


Here are the nova package versions:
nova-api/focal-updates,now 2:22.4.0-0ubuntu1~cloud3 all [installed]
nova-common/focal-updates,now 2:22.4.0-0ubuntu1~cloud3 all [installed]
nova-conductor/focal-updates,now 2:22.4.0-0ubuntu1~cloud3 all [installed]
nova-novncproxy/focal-updates,now 2:22.4.0-0ubuntu1~cloud3 all [installed]
nova-scheduler/focal-updates,now 2:22.4.0-0ubuntu1~cloud3 all [installed]
python3-nova/focal-updates,now 2:22.4.0-0ubuntu1~cloud3 all [installed]
python3-novaclient/focal-updates,now 2:17.2.1-0ubuntu1~cloud0 all [installed,automatic]

Here are the cinder package versions:
cinder-api/focal-updates,now 2:17.4.0-0ubuntu1~cloud3 all [installed]
cinder-common/focal-updates,now 2:17.4.0-0ubuntu1~cloud3 all [installed,automatic]
cinder-scheduler/focal-updates,now 2:17.4.0-0ubuntu1~cloud3 all [installed]
cinder-volume/focal-updates,now 2:17.4.0-0ubuntu1~cloud3 all [installed]
python3-cinder/focal-updates,now 2:17.4.0-0ubuntu1~cloud3 all [installed,automatic]
python3-cinderclient/focal-updates,now 1:7.2.0-0ubuntu1~cloud0 all [installed]


Thanks in advance for any ideas!