Open Stack

Hi,

We are running Openstack Rocky version. Our instances are getting volume
from CEPH (mimic version).

Suddenly one of our compute node got shutdown due to RAM issue. So all
instances running on that compute node (linux/windows) got effected. After
starting the compute node all services up & running. But instances on that
compute node fail to boot. Instances are getting volume from CEPH storage &
CEPH is in Healthy state. Please see below log.

Requesting if anyone can help me to solve.

 

 

````````````````````````````````````````````````````````````````````````````
````````````````````````````````````````````````````````````````````````````
````````````````````````````````````````````````````````

````````````````````````````````````````````````````````````````````````````
````````````````````````````````````````````````````````````````````````````
````````````````````````````````````````````````````````

 

 

Welcome to [0;34mCentOS Linux 7 (Core) dracut-033-554.el7 (Initramfs)[0m!

 

[    2.543762] systemd[1]: Set hostname to <localhost.localdomain>.

[    2.601779] usb 1-1: new full-speed USB device number 2 using uhci_hcd

[    2.609813] random: systemd: uninitialized urandom read (16 bytes read)

[    2.616574] random: systemd: uninitialized urandom read (16 bytes read)

[    2.622966] random: systemd: uninitialized urandom read (16 bytes read)

[    2.628863] random: systemd: uninitialized urandom read (16 bytes read)

[    2.635771] random: systemd: uninitialized urandom read (16 bytes read)

[    2.641509] random: systemd: uninitialized urandom read (16 bytes read)

[    2.647362] random: systemd: uninitialized urandom read (16 bytes read)

[    2.660222] systemd[1]: Reached target Local File Systems.

[[32m  OK  [0m] Reached target Local File Systems.

[    2.668273] systemd[1]: Created slice Root Slice.

[[32m  OK  [0m] Created slice Root Slice.

[    2.674835] systemd[1]: Created slice System Slice.

[[32m  OK  [0m] Created slice System Slice.

[    2.681170] systemd[1]: Listening on udev Kernel Socket.

[[32m  OK  [0m] Listening on udev Kernel Socket.

[    2.687811] systemd[1]: Listening on Journal Socket.

[[32m  OK  [0m] Listening on Journal Socket.

[    2.694604] systemd[1]: Starting Create list of required static device
nodes for the current kernel...

         Starting Create list of required st... nodes for the current
kernel...

[    2.705069] systemd[1]: Starting Journal Service...

         Starting Journal Service...

[    2.710298] systemd[1]: Listening on udev Control Socket.

[[32m  OK  [0m] Listening on udev Control Socket.

[    2.715399] systemd[1]: Reached target Sockets.

[[32m  OK  [0m] Reached target Sockets.

[    2.720550] systemd[1]: Starting dracut cmdline hook...

         Starting dracut cmdline hook...

[    2.725277] systemd[1]: Reached target Slices.

[[32m  OK  [0m] Reached target Slices.

[    2.731319] systemd[1]: Starting Apply Kernel Variables...

         Starting Apply Kernel Variables...

[    2.738718] systemd[1]: Starting Setup Virtual Console...

         Starting Setup Virtual Console...

[    2.744446] systemd[1]: Reached target Timers.

[[32m  OK  [0m] Reached target Timers.

[    2.749668] systemd[1]: Reached target Swap.

[    2.781213] usb 1-1: New USB device found, idVendor=0627, idProduct=0001

[    2.781217] usb 1-1: New USB device strings: Mfr=1, Product=3,
SerialNumber=5

[    2.781220] usb 1-1: Product: QEMU USB Tablet

[    2.781222] usb 1-1: Manufacturer: QEMU

[    2.781225] usb 1-1: SerialNumber: 42

[    2.789965] input: QEMU QEMU USB Tablet as
/devices/pci0000:00/0000:00:01.2/usb1/1-1/1-1:1.0/input/input4

[    2.790189] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v0.01
Mouse [QEMU QEMU USB Tablet] on usb-0000:00:01.2-1/input0

[[32m  OK  [0m] Reached target Swap.

[    2.887336] systemd[1]: Started Journal Service.

[[32m  OK  [0m] Started Journal Service.

[[32m  OK  [0m] Started Create list of required sta...ce nodes for the
current kernel.

[[32m  OK  [0m] Started Apply Kernel Variables.

[    2.903653] random: fast init done

         Starting Create Static Device Nodes in /dev...

[[32m  OK  [0m] Started Create Static Device Nodes in /dev.

[[32m  OK  [0m] Started dracut cmdline hook.

         Starting dracut pre-udev hook...

[[32m  OK  [0m] Started Setup Virtual Console.

[    2.962744] device-mapper: uevent: version 1.0.3

[    2.966701] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised:
dm-devel at redhat.com <mailto:dm-devel at redhat.com> 

[[32m  OK  [0m] Started dracut pre-udev hook.

         Starting udev Kernel Device Manager...

[[32m  OK  [0m] Started udev Kernel Device Manager.

         Starting udev Coldplug all Devices...

[[32m  OK  [0m] Started udev Coldplug all Devices.

[[32m  OK  [0m] Reached target System Initialization.

        Starting Show Plymouth Boot Screen...

         Starting dracut initqueue hook...

[[32m  OK  [0m] Started Show Plymouth Boot Screen.

[[32m  OK  [0m] Reached target Paths.

[[32m  OK  [0m] Started Forward Password Requests to Plymouth Directory
Watch.

[[32m  OK  [0m] Reached target Basic System.

         Mounting Configuration File System...

[[32m  OK  [0m] Mounted Configuration File System.

[    3.278679] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 10

[    3.284823] FDC 0 is a S82078B

[    3.346628] scsi host0: ata_piix

[    3.358340] scsi host1: ata_piix

[    3.363061] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc0c0 irq 14

[    3.369572] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc0c8 irq 15

[    3.413844] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10

%G[    3.489243] virtio_blk virtio1: [vda] 629145600 512-byte logical blocks
(322 GB/300 GiB)

[    3.489830]  vda: vda1 vda2 vda3

[    3.527229] ata1.00: ATAPI: QEMU DVD-ROM, 2.5+, max UDMA/100

[    3.529577] ata1.00: configured for MWDMA2

[    3.542907] scsi 0:0:0:0: CD-ROM            QEMU     QEMU DVD-ROM
2.5+ PQ: 0 ANSI: 5

[    3.622851] sr 0:0:0:0: [sr0] scsi3-mmc drive: 4x/4x cd/rw xa/form2 tray

[    3.626938] cdrom: Uniform CD-ROM driver Revision: 3.20

[[32m  OK  [0m] Found device /dev/mapper/brilliant--centos--vg-root.

         Starting File System Check on
/dev/...er/brilliant--centos--vg-root...

[[32m  OK  [0m] Started File System Check on
/dev/mapper/brilliant--centos--vg-root.

[[32m  OK  [0m] Started dracut initqueue hook.

         Mounting /sysroot...

[[32m  OK  [0m] Reached target Remote File Systems (Pre).

[[32m  OK  [0m] Reached target Remote File Systems.

[    4.029336] SGI XFS with ACLs, security attributes, no debug enabled

[    4.035798] XFS (dm-0): Mounting V5 Filesystem

[    5.917747] blk_update_request: I/O error, dev vda, sector 13098937

[    5.922604] blk_update_request: I/O error, dev vda, sector 13099961

[    5.927008] blk_update_request: I/O error, dev vda, sector 13100985

[    5.931339] blk_update_request: I/O error, dev vda, sector 13102009

[    5.935231] XFS (dm-0): xfs_do_force_shutdown(0x1) called from line 1266
of file fs/xfs/xfs_buf.c.  Return address = 0xffffffffc02738cc

[    5.942478] XFS (dm-0): I/O Error Detected. Shutting down filesystem

[    5.946134] XFS (dm-0): Please umount the filesystem and rectify the
problem(s)

[    5.952283] XFS (dm-0): metadata I/O error: block 0x782fb9
("xlog_bwrite") error 5 numblks 8192

[    5.959663] XFS (dm-0): failed to locate log tail

[    5.963221] XFS (dm-0): log mount/recovery failed: error -5

[    5.967272] XFS (dm-0): log mount failed

[    5.979786] blk_update_request: I/O error, dev vda, sector 0

[    5.992687] blk_update_request: I/O error, dev vda, sector 0

[[1;31mFAILED[0m] Failed to mount /sysroot.

See 'systemctl status sysroot.mount' for details.

[[1;33mDEPEND[0m] Dependency failed for Initrd Root File System.

[[1;33mDEPEND[0m] Dependency failed for Reload Configuration from the Real
Root.

[[32m  OK  [0m] Stopped dracut pre-udev hook.

[[32m  OK  [0m] Stopped dracut cmdline hook.

[[32m  OK  [0m] Stopped target Basic System.

[[32m  OK  [0m] Stopped target System Initialization.

[[32m  OK  [0m] Started Emergency Shell.

[[32m  OK  [0m] Reached target Emergency Mode.

[[32m  OK  [0m] Stopped dracut initqueue hook.

[[32m  OK  [0m] Reached target Initrd File Systems.

 

Generating "/run/initramfs/rdsosreport.txt"

 

 

Entering emergency mode. Exit the shell to continue.

Type "journalctl" to view system logs.

You might want to save "/run/initramfs/rdsosreport.txt" to a USB stick or
/boot

after mounting them and attach it to a bug report.

 

 

:/# [   12.851915] random: crng init done

 

````````````````````````````````````````````````````````````````````````````
````````````````````````````````````````````````````````````````````````````
````````````````````````````````````````````````````````

````````````````````````````````````````````````````````````````````````````
````````````````````````````````````````````````````````````````````````````
````````````````````````````````````````````````````````

 

 

Thanks & B'Rgds,

Rony

 

 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20190425/163017db/attachment-0001.html>