[nova] Rescue booting on wrong disk
Hey all,
From time to time we are facing an issue when puting instance in rescue with the same image as the one the instance was booted.
E.G. I booted an instance using Debian 10, disk are: debian@testarnaud:~$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sr0 11:0 1 486K 0 rom vda 254:0 0 10G 0 disk └─vda1 254:1 0 10G 0 part / debian@testarnaud:~$ cat /etc/fstab # /etc/fstab: static file system information. UUID=5605171d-d590-46d5-85e2-60096b533a18 / ext4 errors=remount-ro 0 1 I rescued the instance: $ openstack server rescue --image bc73a901-6366-4a69-8ddc-00479b4d647f testarnaud Then, back in the instance: debian@testarnaud:~$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sr0 11:0 1 486K 0 rom vda 254:0 0 2G 0 disk └─vda1 254:1 0 2G 0 part vdb 254:16 0 10G 0 disk └─vdb1 254:17 0 10G 0 part / Instance booted on /dev/vdb1 instead of /dev/vda1 Is there anything we can configure on nova side to avoid this situation? Thanks -- Arnaud Morin
On Tue, 2021-02-09 at 11:21 +0000, Arnaud Morin wrote:
Hey all,
From time to time we are facing an issue when puting instance in rescue with the same image as the one the instance was booted.
E.G. I booted an instance using Debian 10, disk are:
debian@testarnaud:~$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sr0 11:0 1 486K 0 rom vda 254:0 0 10G 0 disk └─vda1 254:1 0 10G 0 part / debian@testarnaud:~$ cat /etc/fstab # /etc/fstab: static file system information. UUID=5605171d-d590-46d5-85e2-60096b533a18 / ext4 errors=remount-ro 0 1
I rescued the instance: $ openstack server rescue --image bc73a901-6366-4a69-8ddc-00479b4d647f testarnaud
Then, back in the instance:
debian@testarnaud:~$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sr0 11:0 1 486K 0 rom vda 254:0 0 2G 0 disk └─vda1 254:1 0 2G 0 part vdb 254:16 0 10G 0 disk └─vdb1 254:17 0 10G 0 part /
Instance booted on /dev/vdb1 instead of /dev/vda1
Is there anything we can configure on nova side to avoid this situation? in ussuri lee yarwood added https://specs.openstack.org/openstack/nova-specs/specs/ussuri/implemented/vi... to nova which i belive will resolve the issue
from ocata+ i think you can add hw_rescue_bus=usb to get a similar effect but from ussuri we cahgne the layout so thtat the rescue disk is always used. lee is that right?
Thanks
On 09-02-21 12:23:58, Sean Mooney wrote:
On Tue, 2021-02-09 at 11:21 +0000, Arnaud Morin wrote:
Hey all,
From time to time we are facing an issue when puting instance in rescue with the same image as the one the instance was booted.
E.G. I booted an instance using Debian 10, disk are:
debian@testarnaud:~$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sr0 11:0 1 486K 0 rom vda 254:0 0 10G 0 disk └─vda1 254:1 0 10G 0 part / debian@testarnaud:~$ cat /etc/fstab # /etc/fstab: static file system information. UUID=5605171d-d590-46d5-85e2-60096b533a18 / ext4 errors=remount-ro 0 1
I rescued the instance: $ openstack server rescue --image bc73a901-6366-4a69-8ddc-00479b4d647f testarnaud
Then, back in the instance:
debian@testarnaud:~$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sr0 11:0 1 486K 0 rom vda 254:0 0 2G 0 disk └─vda1 254:1 0 2G 0 part vdb 254:16 0 10G 0 disk └─vdb1 254:17 0 10G 0 part /
Instance booted on /dev/vdb1 instead of /dev/vda1
Is there anything we can configure on nova side to avoid this situation?
in ussuri lee yarwood added https://specs.openstack.org/openstack/nova-specs/specs/ussuri/implemented/vi... to nova which i belive will resolve the issue
from ocata+ i think you can add hw_rescue_bus=usb to get a similar effect but from ussuri we cahgne the layout so thtat the rescue disk is always used. lee is that right?
Yeah that's correct, from Ussuri (21.0.0) with the libvirt driver the instance should continue to show all disks connected as normal with the rescue disk appended last and always used as the boot device. In your case, switching the rescue bus should change the default boot device type n-cpu tells libvirt to use working around your problem. Another obvious thing you can try is just using a different image so the two MBRs don't conflict? Cheers, -- Lee Yarwood A5D1 9385 88CB 7E5F BE64 6618 BCA6 6E33 F672 2D76
Hello, Thanks, we will check these params. Cheers Le 9 février 2021 17:50:23 GMT+01:00, Lee Yarwood <lyarwood@redhat.com> a écrit :
On 09-02-21 12:23:58, Sean Mooney wrote:
On Tue, 2021-02-09 at 11:21 +0000, Arnaud Morin wrote:
Hey all,
From time to time we are facing an issue when puting instance in rescue with the same image as the one the instance was booted.
E.G. I booted an instance using Debian 10, disk are:
debian@testarnaud:~$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sr0 11:0 1 486K 0 rom vda 254:0 0 10G 0 disk └─vda1 254:1 0 10G 0 part / debian@testarnaud:~$ cat /etc/fstab # /etc/fstab: static file system information. UUID=5605171d-d590-46d5-85e2-60096b533a18 / ext4 errors=remount-ro 0 1
I rescued the instance: $ openstack server rescue --image bc73a901-6366-4a69-8ddc-00479b4d647f testarnaud
Then, back in the instance:
debian@testarnaud:~$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sr0 11:0 1 486K 0 rom vda 254:0 0 2G 0 disk └─vda1 254:1 0 2G 0 part vdb 254:16 0 10G 0 disk └─vdb1 254:17 0 10G 0 part /
Instance booted on /dev/vdb1 instead of /dev/vda1
Is there anything we can configure on nova side to avoid this situation?
in ussuri lee yarwood added
https://specs.openstack.org/openstack/nova-specs/specs/ussuri/implemented/vi...
to nova which i belive will resolve the issue
from ocata+ i think you can add hw_rescue_bus=usb to get a similar effect but from ussuri we cahgne the layout so thtat the rescue disk is always used. lee is that right?
Yeah that's correct, from Ussuri (21.0.0) with the libvirt driver the instance should continue to show all disks connected as normal with the rescue disk appended last and always used as the boot device.
In your case, switching the rescue bus should change the default boot device type n-cpu tells libvirt to use working around your problem. Another obvious thing you can try is just using a different image so the two MBRs don't conflict?
Cheers,
-- Lee Yarwood A5D1 9385 88CB 7E5F BE64 6618 BCA6 6E33 F672 2D76
participants (4)
-
Arnaud
-
Arnaud Morin
-
Lee Yarwood
-
Sean Mooney