Open Stack

Hello,

Did an upgrade from liberty to mitaka and we are experiencing an issue where the audit of locally available compute resource usage is not seeing instances that were deployed prior to the upgrade. Has anyone experienced this before? We hard power cycled all the vms (and hypervisor) to make sure they are running on the latest nova version. This is impacting scheduling of resources/quota usage as you could imagine ☺

Example from one box.

2017-03-02 19:37:52.189 17586 INFO nova.compute.resource_tracker [req-da808a51-f9e3-4762-b46b-ee313183d2d2 - - - - -] Auditing locally available compute resources for node usw1comp024.openstack.local
2017-03-02 19:37:54.413 17586 INFO nova.compute.resource_tracker [req-da808a51-f9e3-4762-b46b-ee313183d2d2 - - - - -] Total usable vcpus: 56, total allocated vcpus: 0
2017-03-02 19:37:54.413 17586 INFO nova.compute.resource_tracker [req-da808a51-f9e3-4762-b46b-ee313183d2d2 - - - - -] Final resource view: name=hostname.openstack.local phys_ram=257725MB used_ram=2048MB phys_disk=4433GB used_disk=2GB total_vcpus=56 used_vcpus=0 pci_stats=[]

virsh list --all
Id    Name                           State
----------------------------------------------------
2     instance-000018ed              running
3     instance-000018ea              running
4     instance-000018e4              running
5     instance-000018de              running
6     instance-0000076f              running
7     instance-00000420              running

libvirt+  7995  1.6  0.2 13401648 653416 ?     Sl   19:07   0:40 /usr/bin/qemu-system-x86_64 -name instance-000018ed -S -machine pc-i440fx-trusty,accel=kvm,usb=off -cpu SandyBridge,+invpcid,+erms,+bmi2,+smep,+avx2,+bmi1,+fsgsbase,+abm,+pdpe1gb,+rdrand,+f16c,+osxsave,+movbe,+dca,+pcid,+pdcm,+xtpr,+fma,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme -m 8192 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid d306091c-e1a6-44ef-b5ff-bc412c4b2341 -smbios type=1,manufacturer=OpenStack Foundation,product=OpenStack Nova,version=13.1.3,serial=03000200-0400-0500-0006-000700080009,uuid=d306091c-e1a6-44ef-b5ff-bc412c4b2341,family=Virtual Machine -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-000018ed.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive file=/var/lib/nova/instances/d306091c-e1a6-44ef-b5ff-bc412c4b2341/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none,discard=ignore -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=25,id=hostnet0,vhost=on,vhostfd=27 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:bc:3c:ee,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/d306091c-e1a6-44ef-b5ff-bc412c4b2341/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -chardev pty,id=charchannel0 -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -spice port=5900,addr=10.60.16.73,disable-ticketing,seamless-migration=on -k en-us -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6
libvirt+  8516  1.6  0.2 13401672 594804 ?     Sl   19:07   0:39 /usr/bin/qemu-system-x86_64 -name instance-000018ea -S -machine pc-i440fx-trusty,accel=kvm,usb=off -cpu SandyBridge,+invpcid,+erms,+bmi2,+smep,+avx2,+bmi1,+fsgsbase,+abm,+pdpe1gb,+rdrand,+f16c,+osxsave,+movbe,+dca,+pcid,+pdcm,+xtpr,+fma,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme -m 8192 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid 2e3cdf87-5022-45df-8a47-150afee1ca83 -smbios type=1,manufacturer=OpenStack Foundation,product=OpenStack Nova,version=13.1.3,serial=03000200-0400-0500-0006-000700080009,uuid=2e3cdf87-5022-45df-8a47-150afee1ca83,family=Virtual Machine -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-000018ea.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive file=/var/lib/nova/instances/2e3cdf87-5022-45df-8a47-150afee1ca83/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none,discard=ignore -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=25,id=hostnet0,vhost=on,vhostfd=28 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:44:46:a9,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/2e3cdf87-5022-45df-8a47-150afee1ca83/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -chardev pty,id=charchannel0 -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -spice port=5901,addr=10.60.16.73,disable-ticketing,seamless-migration=on -k en-us -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6

and on..
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20170302/6bbdae22/attachment.html>