[openstack-nova] Attached volume is not available in the VM

Lee Yarwood lyarwood at redhat.com
Thu Jul 29 16:11:33 UTC 2021


On 29-07-21 12:51:26, Aleksander Wojtal wrote:
> Hi Lee
> 
> Let me clarify the steps, I might have mislead you. The reboot of the compute is done before issuing the server add volume command.
> Steps:
> 1. Create VM
> 2. Create volume
> 3. Reboot compute on which is the VM
> 4. About 1 min later issue "openstack server add volume" command
> 5. Observe volume being attached in DB but not in Libvirt.

What was the state of the instance during the attach request, had it
restarted after the host reboot?

> -----Original Message-----
> From: Lee Yarwood <lyarwood at redhat.com> 
> Sent: Tuesday, July 27, 2021 10:25 PM
> To: Aleksander Wojtal <aleksander.wojtal at tietoevry.com>
> Cc: openstack-discuss at lists.openstack.org; Balázs Gibizer <balazs.gibizer at ericsson.com>; Piotr Mossakowski <Piotr.Mossakowski at tietoevry.com>
> Subject: Re: [openstack-nova] Attached volume is not available in the VM
> 
> On 26-07-21 08:14:46, Aleksander Wojtal wrote:
> > Hello,
> > 
> > During our testing we perform reboot of compute during volume attached task. Result is as follows.
> 
> I was about to ask for more details when I saw this initial line. This is almost expected if you restart n-cpu while it's servicing a request like attaching a volume.
> 
> The compute service isn't able to pick these tasks back up again and complete them once it restarts. That said given the `in-use` volume state below we should've basically completed the attach before the host reboot so I don't understand why it didn't show up in the instance.
>  
> > Volume is marked as attached to VM (also in DB) ceeinfra at lcm1:~> 
> > openstack volume list
> > +--------------------------------------+-----------------------------------------------------------------------------+-----------+------+--------------------------------------------------------------------------------------------------------------+
> > | ID                                   | Name                                                                        | Status    | Size | Attached to                                                                                                  |
> > +--------------------------------------+-----------------------------------------------------------------------------+-----------+------+--------------------------------------------------------------------------------------------------------------+
> > | c52d406c-5587-445d-9da1-436ffbbe3541 | vol_neXt-488                                                                | available |   10 |                                                                                                              |
> > | f3e45efc-35eb-4f95-8e71-f50b5cb69028 | RebootOfComputeHostWhilePerformingVolumeOperations-Volume-0618_19_45_12_669 | in-use    |   20 | Attached to neXt-377_VM1--RebootOfComputeHostWhilePerformingVolumeOperations--0618_19_44_16_775 on /dev/vdb  |
> > | 1bd81392-e2c9-4725-93c8-baa484924f21 | vol_neXt-488                                                                | available |   10 |                                                                                                              |
> > | 919db17c-0729-409c-bf59-74539edcea47 | vol_neXt-488                                                                | available |   10 |                                                                                                              |
> > +--------------------------------------+-----------------------------------------------------------------------------+-----------+------+--------------------------------------------------------------------------------------------------------------+
> > 
> > From VM perspective too.
> > ceeinfra at lcm1:~> openstack server show 
> > neXt-377_VM1--RebootOfComputeHostWhilePerformingVolumeOperations--0618
> > _19_44_16_775
> > +-------------------------------------+-------------------------------------------------------------------------------------+
> > | Field                               | Value                                                                               |
> > +-------------------------------------+-------------------------------------------------------------------------------------+
> > | OS-DCF:diskConfig                   | MANUAL                                                                              |
> > | OS-EXT-AZ:availability_zone         | nova                                                                                |
> > | OS-EXT-SRV-ATTR:host                | compute-0-11.k2.ericsson.se                                                         |
> > | OS-EXT-SRV-ATTR:hypervisor_hostname | compute-0-11.k2.ericsson.se                                                         |
> > | OS-EXT-SRV-ATTR:instance_name       | instance-00001edc                                                                   |
> > | OS-EXT-STS:power_state              | Running                                                                             |
> > | OS-EXT-STS:task_state               | None                                                                                |
> > | OS-EXT-STS:vm_state                 | active                                                                              |
> > | OS-SRV-USG:launched_at              | 2021-06-18T14:14:58.000000                                                          |
> > | OS-SRV-USG:terminated_at            | None                                                                                |
> > | accessIPv4                          |                                                                                     |
> > | accessIPv6                          |                                                                                     |
> > | addresses                           | NovaController-Network-0618_19_44_16_852=10.0.0.5                                   |
> > | config_drive                        | True                                                                                |
> > | created                             | 2021-06-18T14:14:19Z                                                                |
> > | flavor                              | m1.small (2)                                                                        |
> > | hostId                              | 5078bfd4fabd8b2e7c9f4554b3ac08d536ceebb5f393db4d38c8ff60                            |
> > | id                                  | 95ed49ed-b3f8-46ae-80a0-544b939c50b7                                                |
> > | image                               | JCAT_Common_CirrOS_i386 (b71f3fbd-59f1-4c29-a296-5065fe0406f7)                      |
> > | key_name                            | None                                                                                |
> > | name                                | neXt-377_VM1--RebootOfComputeHostWhilePerformingVolumeOperations--0618_19_44_16_775 |
> > | progress                            | 0                                                                                   |
> > | project_id                          | b63725c8ebaf43ebabbed497f41eb71c                                                    |
> > | properties                          | ha-policy='managed-on-host'                                                         |
> > | scheduler_hints                     | {}                                                                                  |
> > | security_groups                     | name='default'                                                                      |
> > | status                              | ACTIVE                                                                              |
> > | updated                             | 2021-06-18T14:22:07Z                                                                |
> > | user_id                             | f490f07b813748698e56ca7641d46a72                                                    |
> > | volumes_attached                    | id='f3e45efc-35eb-4f95-8e71-f50b5cb69028'                                           |
> > +-------------------------------------+-------------------------------------------------------------------------------------+
> 
> What about `openstack server volume list 95ed49ed-b3f8-46ae-80a0-544b939c50b7`?
> 
> Did you restart the instance as I assuming a hard reboot would resolve the issue here.
> 
> > From Libvirt perspective, volume is not attached to VM
> > 
> > compute-0-11:/var/log # virsh dumpxml instance-00001edc <domain 
> > type='kvm' id='1'>
> >   <name>instance-00001edc</name>
> >   <uuid>95ed49ed-b3f8-46ae-80a0-544b939c50b7</uuid>
> > (...)
> >   <devices>
> >     <emulator>/usr/bin/qemu-system-x86_64</emulator>
> >     <disk type='file' device='disk'>
> >       <driver name='qemu' type='qcow2' cache='directsync'/>
> >       <source file='/var/lib/nova/instances/95ed49ed-b3f8-46ae-80a0-544b939c50b7/disk' index='2'/>
> >       <backingStore type='file' index='3'>
> >         <format type='raw'/>
> >         <source file='/var/lib/nova/instances/_base/af82cd8216adb002916bb9e422a3fbb637022ec6'/>
> >         <backingStore/>
> >       </backingStore>
> >       <target dev='vda' bus='virtio'/>
> >       <alias name='virtio-disk0'/>
> >       <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
> >     </disk>
> >     <disk type='file' device='cdrom'>
> >       <driver name='qemu' type='raw' cache='directsync'/>
> >       <source file='/var/lib/nova/instances/95ed49ed-b3f8-46ae-80a0-544b939c50b7/disk.config' index='1'/>
> >       <backingStore/>
> >       <target dev='hdd' bus='ide'/>
> >       <readonly/>
> >       <alias name='ide0-1-1'/>
> >       <address type='drive' controller='0' bus='1' target='0' unit='1'/>
> >     </disk>
> >     <controller type='usb' index='0' model='piix3-uhci'>
> > (...)
> > </domain>
> 
> It's easier to use `virsh domblklist 95ed49ed-b3f8-46ae-80a0-544b939c50b7` to list the attached disks FWIW.
>  
> > System does not provide any indication of the problem. There should be some kind of indication to user that volume attachment was not completed.
> 
> Yeah there's no real way for n-cpu to handle this, the event log for the action isn't going to be complete but there's no state machine for volume attachments that we can rely on here.
> 
> Would you mind opening a bug with the above and the following event details for the attach request:
> 
> $ openstack server event list 95ed49ed-b3f8-46ae-80a0-544b939c50b7
> 
> $ openstack server event show 95ed49ed-b3f8-46ae-80a0-544b939c50b7
> $volume-attach-request-id
> 
> https://docs.openstack.org/api-guide/compute/faults.html
> 
> Cheers,
> 
> -- 
> Lee Yarwood                 A5D1 9385 88CB 7E5F BE64  6618 BCA6 6E33 F672 2D76

-- 
Lee Yarwood                 A5D1 9385 88CB 7E5F BE64  6618 BCA6 6E33 F672 2D76
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20210729/69c03c94/attachment.sig>


More information about the openstack-discuss mailing list