Well, thanks for pointing me to the fact that code wasnt in victoria. I went ahead and removed and purged the cinder packages, and reinstalled from victoria. Now it shows the api verision of 2.51, like it should. Maybe at some point the wrong repo was added and removed, causing a package update that shouldnt have happened. This issue is resolved. ________________________________________ From: Jim Kilborn Sent: Tuesday, May 23, 2023 11:28 AM To: Masayuki Igawa; openstack-discuss@lists.openstack.org Subject: RE: upgrade issue with nova/cinder and api version error Well, that is very confusing. When I check that file, it comes from python3-cinder, and that package appears to be from victoria, as I dont have a zena repo installed. This is from the ubuntu repos, so maybe it got backported too far. # grep 2.89 /usr/lib/python3/dist-packages/cinder/compute/nova.py nova = novaclient(context, api_version='2.89') # dpkg -S /usr/lib/python3/dist-packages/cinder/compute/nova.py python3-cinder: /usr/lib/python3/dist-packages/cinder/compute/nova.py # apt-cache showpkg python3-cinder Package: python3-cinder Versions: 2:17.4.0-0ubuntu1~cloud4 (/var/lib/apt/lists/ubuntu-cloud.archive.canonical.com_ubuntu_dists_focal-updates_victoria_main_binary-amd64_Packages) # ls /etc/apt/sources.list.d cloudarchive-stein.list cloudarchive-stein.list.distUpgrade cloudarchive-stein.list.save cloudarchive-train.list cloudarchive-train.list.save cloudarchive-ussuri.list cloudarchive-ussuri.list.distUpgrade cloudarchive-ussuri.list.save cloudarchive-victoria.list cloudarchive-victoria.list.save gluster-ubuntu-glusterfs-6-bionic.list gluster-ubuntu-glusterfs-6-bionic.list.distUpgrade gluster-ubuntu-glusterfs-6-bionic.list.save gluster-ubuntu-glusterfs-6-focal.list gluster-ubuntu-glusterfs-6-focal.list.save mariadb.list mariadb.list.distUpgrade mariadb.list.save ________________________________________ From: Masayuki Igawa [masayuki.igawa@gmail.com] Sent: Tuesday, May 23, 2023 12:41 AM To: openstack-discuss@lists.openstack.org Subject: Re: upgrade issue with nova/cinder and api version error Hi,

When I look at the /usr/lib/python3/dist-packages/cinder/compute/nova.py I can see it's using 2.89 in get_server_volume

def get_server_volume(context, server_id, volume_id): # Use microversion that includes attachment_id nova = novaclient(context, api_version='2.89') return nova.volumes.get_server_volume(server_id, volume_id)

It's weird for me. That function was introduced this patch[1] but it was backported till xena not victoria. So, I wondering if you are using mixed versioned openstack somehow. [1] https://review.opendev.org/q/I612905a1bf4a1706cce913c0d8a6df7a240d599a -- Masayuki Igawa On Tue, May 23, 2023, at 01:13, Jim Kilborn wrote:

Hello,

First time posting here. We have been running a production openstack environment at my office since the kilo release. We are currently on train, and I'm trying to get up to a more recent version. To make it more difficult, we are on centos7, so having to switch to ubuntu as we update versions.

The problem that I am having after updaing to victoria, is that when I delete a vm via horizon, the instance disappears but the cinder volume doesn't delete the attachment. It appears this is due to the following error in /var/log/apache2/cinder_error.log

ERROR cinder.api.middleware.fault novaclient.exceptions.NotAcceptable: Version 2.89 is not supported by the API. Minimum is 2.1 and maximum is 2.87. (HTTP 406)

When I look at the /usr/lib/python3/dist-packages/cinder/compute/nova.py I can see it's using 2.89 in get_server_volume

def get_server_volume(context, server_id, volume_id): # Use microversion that includes attachment_id nova = novaclient(context, api_version='2.89') return nova.volumes.get_server_volume(server_id, volume_id)

I am not sure why cinder and nova are in disagreement on the api_version. I have verified that they are both upgraded to the victoria release.

Anyone have any ideas as to why I would be getting this error or a possible fix? I haven't been able to find any information on this error.

Here are the nova package versions: nova-api/focal-updates,now 2:22.4.0-0ubuntu1~cloud3 all [installed] nova-common/focal-updates,now 2:22.4.0-0ubuntu1~cloud3 all [installed] nova-conductor/focal-updates,now 2:22.4.0-0ubuntu1~cloud3 all [installed] nova-novncproxy/focal-updates,now 2:22.4.0-0ubuntu1~cloud3 all [installed] nova-scheduler/focal-updates,now 2:22.4.0-0ubuntu1~cloud3 all [installed] python3-nova/focal-updates,now 2:22.4.0-0ubuntu1~cloud3 all [installed] python3-novaclient/focal-updates,now 2:17.2.1-0ubuntu1~cloud0 all [installed,automatic]

Here are the cinder package versions: cinder-api/focal-updates,now 2:17.4.0-0ubuntu1~cloud3 all [installed] cinder-common/focal-updates,now 2:17.4.0-0ubuntu1~cloud3 all [installed,automatic] cinder-scheduler/focal-updates,now 2:17.4.0-0ubuntu1~cloud3 all [installed] cinder-volume/focal-updates,now 2:17.4.0-0ubuntu1~cloud3 all [installed] python3-cinder/focal-updates,now 2:17.4.0-0ubuntu1~cloud3 all [installed,automatic] python3-cinderclient/focal-updates,now 1:7.2.0-0ubuntu1~cloud0 all [installed]

Thanks in advance for any ideas!