Nova not updating to new size of an extended in-use / attached cinder volume (Ceph RBD) to guest
christian.rohmann at inovex.de
Wed May 5 15:34:12 UTC 2021
On 16/02/2021 22:32, Christian Rohmann wrote:
> I have just been trying to reproduce the issue, but in all my new
> attempts it just worked as expected:
>> [162262.926512] sd 0:0:0:1: Capacity data has changed
>> [162262.932868] sd 0:0:0:1: [sdb] 6291456 512-byte logical blocks:
>> (3.22 GB/3.00 GiB)
>> [162262.933061] sdb: detected capacity change from 2147483648 to
On 17/02/2021 09:43, Tobias Urdin wrote:
> Regarding the actual extending of in-use volumes, we had an issue
> where cinder could not talk to os-server-external-events endpoint for
> nova because it used the wrong endpoint when looking up in keystone.
> We saw the error in cinder-volume.log except for that I can't remember
> we did anything special.
> Had to use newer microversion for cinder when using CLI.
> cinder --os-volume-api-version 3.42 extend <volume ID or name> <new
> size in GB>
I am very sorry for the delay, but I was now finally able to reproduce
the issue and with a VERY strange finding:
* When using the cinder client with a Volume API version >= 3.42 like
you suggested Tobias, it works just fine using cloud admin credentials.
* The volume is attached / in-use, but it is resized just fine,
including the notification of the kernel on the VM.
* When attempting the same thing using the project users credentials the
resize also works just fine, volume still attached and in-use, but then
the VM is NOT notified
* Also this does not seems to be related to nova or QEMU, but rather it
appears there is no extend_volume event triggered or at least logged:
--- cut ---
Request ID -- Action -- Start Time -- User ID -- Message
req-a4065b2d-1b77-4c5a-bf53-a2967b574fa0 extend_volume May 5, 2021,
1:22 p.m. 784bd2a5b82c3b31eb56ee -
2021, 1:09 p.m. 4b2abc14e511a7c0b10c -
req-75ef5bd3-75d3-4146-84eb-1809789b6586 Create May 5, 2021,
1:09 p.m. 4b2abc14e511a7c0b10c -
--- cut ---
UserID "784bd2a5b82c3b31eb56ee" is the regular user creating and
attaching the volume. But that user also did an extend_volume, which is
not logged as an event.
There also was no API errors reported back to the client, the resize did
happen - just not propagated to the VM - so a stop and restart was required.
But the admin user with id "4b2abc14e511a7c0b10c" doing a resize attempt
caused an extend_volume and consequently did trigger a notification of
just as expected and documented in regards to this feature.
Does anybody have any idea what could cause this or where to look for
More information about the openstack-discuss