[openstack-nova] Attached volume is not available in the VM
Lee Yarwood
lyarwood at redhat.com
Thu Jul 29 16:11:33 UTC 2021
On 29-07-21 12:51:26, Aleksander Wojtal wrote:
> Hi Lee
>
> Let me clarify the steps, I might have mislead you. The reboot of the compute is done before issuing the server add volume command.
> Steps:
> 1. Create VM
> 2. Create volume
> 3. Reboot compute on which is the VM
> 4. About 1 min later issue "openstack server add volume" command
> 5. Observe volume being attached in DB but not in Libvirt.
What was the state of the instance during the attach request, had it
restarted after the host reboot?
> -----Original Message-----
> From: Lee Yarwood <lyarwood at redhat.com>
> Sent: Tuesday, July 27, 2021 10:25 PM
> To: Aleksander Wojtal <aleksander.wojtal at tietoevry.com>
> Cc: openstack-discuss at lists.openstack.org; Balázs Gibizer <balazs.gibizer at ericsson.com>; Piotr Mossakowski <Piotr.Mossakowski at tietoevry.com>
> Subject: Re: [openstack-nova] Attached volume is not available in the VM
>
> On 26-07-21 08:14:46, Aleksander Wojtal wrote:
> > Hello,
> >
> > During our testing we perform reboot of compute during volume attached task. Result is as follows.
>
> I was about to ask for more details when I saw this initial line. This is almost expected if you restart n-cpu while it's servicing a request like attaching a volume.
>
> The compute service isn't able to pick these tasks back up again and complete them once it restarts. That said given the `in-use` volume state below we should've basically completed the attach before the host reboot so I don't understand why it didn't show up in the instance.
>
> > Volume is marked as attached to VM (also in DB) ceeinfra at lcm1:~>
> > openstack volume list
> > +--------------------------------------+-----------------------------------------------------------------------------+-----------+------+--------------------------------------------------------------------------------------------------------------+
> > | ID | Name | Status | Size | Attached to |
> > +--------------------------------------+-----------------------------------------------------------------------------+-----------+------+--------------------------------------------------------------------------------------------------------------+
> > | c52d406c-5587-445d-9da1-436ffbbe3541 | vol_neXt-488 | available | 10 | |
> > | f3e45efc-35eb-4f95-8e71-f50b5cb69028 | RebootOfComputeHostWhilePerformingVolumeOperations-Volume-0618_19_45_12_669 | in-use | 20 | Attached to neXt-377_VM1--RebootOfComputeHostWhilePerformingVolumeOperations--0618_19_44_16_775 on /dev/vdb |
> > | 1bd81392-e2c9-4725-93c8-baa484924f21 | vol_neXt-488 | available | 10 | |
> > | 919db17c-0729-409c-bf59-74539edcea47 | vol_neXt-488 | available | 10 | |
> > +--------------------------------------+-----------------------------------------------------------------------------+-----------+------+--------------------------------------------------------------------------------------------------------------+
> >
> > From VM perspective too.
> > ceeinfra at lcm1:~> openstack server show
> > neXt-377_VM1--RebootOfComputeHostWhilePerformingVolumeOperations--0618
> > _19_44_16_775
> > +-------------------------------------+-------------------------------------------------------------------------------------+
> > | Field | Value |
> > +-------------------------------------+-------------------------------------------------------------------------------------+
> > | OS-DCF:diskConfig | MANUAL |
> > | OS-EXT-AZ:availability_zone | nova |
> > | OS-EXT-SRV-ATTR:host | compute-0-11.k2.ericsson.se |
> > | OS-EXT-SRV-ATTR:hypervisor_hostname | compute-0-11.k2.ericsson.se |
> > | OS-EXT-SRV-ATTR:instance_name | instance-00001edc |
> > | OS-EXT-STS:power_state | Running |
> > | OS-EXT-STS:task_state | None |
> > | OS-EXT-STS:vm_state | active |
> > | OS-SRV-USG:launched_at | 2021-06-18T14:14:58.000000 |
> > | OS-SRV-USG:terminated_at | None |
> > | accessIPv4 | |
> > | accessIPv6 | |
> > | addresses | NovaController-Network-0618_19_44_16_852=10.0.0.5 |
> > | config_drive | True |
> > | created | 2021-06-18T14:14:19Z |
> > | flavor | m1.small (2) |
> > | hostId | 5078bfd4fabd8b2e7c9f4554b3ac08d536ceebb5f393db4d38c8ff60 |
> > | id | 95ed49ed-b3f8-46ae-80a0-544b939c50b7 |
> > | image | JCAT_Common_CirrOS_i386 (b71f3fbd-59f1-4c29-a296-5065fe0406f7) |
> > | key_name | None |
> > | name | neXt-377_VM1--RebootOfComputeHostWhilePerformingVolumeOperations--0618_19_44_16_775 |
> > | progress | 0 |
> > | project_id | b63725c8ebaf43ebabbed497f41eb71c |
> > | properties | ha-policy='managed-on-host' |
> > | scheduler_hints | {} |
> > | security_groups | name='default' |
> > | status | ACTIVE |
> > | updated | 2021-06-18T14:22:07Z |
> > | user_id | f490f07b813748698e56ca7641d46a72 |
> > | volumes_attached | id='f3e45efc-35eb-4f95-8e71-f50b5cb69028' |
> > +-------------------------------------+-------------------------------------------------------------------------------------+
>
> What about `openstack server volume list 95ed49ed-b3f8-46ae-80a0-544b939c50b7`?
>
> Did you restart the instance as I assuming a hard reboot would resolve the issue here.
>
> > From Libvirt perspective, volume is not attached to VM
> >
> > compute-0-11:/var/log # virsh dumpxml instance-00001edc <domain
> > type='kvm' id='1'>
> > <name>instance-00001edc</name>
> > <uuid>95ed49ed-b3f8-46ae-80a0-544b939c50b7</uuid>
> > (...)
> > <devices>
> > <emulator>/usr/bin/qemu-system-x86_64</emulator>
> > <disk type='file' device='disk'>
> > <driver name='qemu' type='qcow2' cache='directsync'/>
> > <source file='/var/lib/nova/instances/95ed49ed-b3f8-46ae-80a0-544b939c50b7/disk' index='2'/>
> > <backingStore type='file' index='3'>
> > <format type='raw'/>
> > <source file='/var/lib/nova/instances/_base/af82cd8216adb002916bb9e422a3fbb637022ec6'/>
> > <backingStore/>
> > </backingStore>
> > <target dev='vda' bus='virtio'/>
> > <alias name='virtio-disk0'/>
> > <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
> > </disk>
> > <disk type='file' device='cdrom'>
> > <driver name='qemu' type='raw' cache='directsync'/>
> > <source file='/var/lib/nova/instances/95ed49ed-b3f8-46ae-80a0-544b939c50b7/disk.config' index='1'/>
> > <backingStore/>
> > <target dev='hdd' bus='ide'/>
> > <readonly/>
> > <alias name='ide0-1-1'/>
> > <address type='drive' controller='0' bus='1' target='0' unit='1'/>
> > </disk>
> > <controller type='usb' index='0' model='piix3-uhci'>
> > (...)
> > </domain>
>
> It's easier to use `virsh domblklist 95ed49ed-b3f8-46ae-80a0-544b939c50b7` to list the attached disks FWIW.
>
> > System does not provide any indication of the problem. There should be some kind of indication to user that volume attachment was not completed.
>
> Yeah there's no real way for n-cpu to handle this, the event log for the action isn't going to be complete but there's no state machine for volume attachments that we can rely on here.
>
> Would you mind opening a bug with the above and the following event details for the attach request:
>
> $ openstack server event list 95ed49ed-b3f8-46ae-80a0-544b939c50b7
>
> $ openstack server event show 95ed49ed-b3f8-46ae-80a0-544b939c50b7
> $volume-attach-request-id
>
> https://docs.openstack.org/api-guide/compute/faults.html
>
> Cheers,
>
> --
> Lee Yarwood A5D1 9385 88CB 7E5F BE64 6618 BCA6 6E33 F672 2D76
--
Lee Yarwood A5D1 9385 88CB 7E5F BE64 6618 BCA6 6E33 F672 2D76
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20210729/69c03c94/attachment.sig>
More information about the openstack-discuss
mailing list