Open Stack

Hi All,

I'm running Ubuntu 12.04 cloudarchive Havana

I just installed a few new hypervisors.  Instances launch, run far
enough to show "booting from disk" in the VNC console, then fall into
a paused state and will not resume.

I have identical hardware running identically configured 2013.2.2 and
all is well (I double checked /proc/cpu info to be sure they are
really the same cpus with the same options enabled)

the libvirt log for the instance says:

KVM internal error. Suberror: 1
emulation failure
EAX=2ad1dcff EBX=022dd200 ECX=a644bc82 EDX=ffffffff
ESI=00007c05 EDI=d8918c96 EBP=0007fff0 ESP=0007ffec
EIP=0000e34f EFL=00010013 [----A-C] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
CS =0008 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
SS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
DS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
FS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
GS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
LDT=0000 00000000 0000ffff 00008200 DPL=0 LDT
TR =0000 00000000 0000ffff 00008b00 DPL=0 TSS32-busy
GDT=     00008280 00000027
IDT=     00000000 00000000
CR0=00000011 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=2d 02 b9 82 bc 44 a6 d6 86 9d 78 82 28 ad 68 63 de 5b 46 54 <de>
87 d4 a9 71 43 04 21 5a 48 a0 dd a2 b8 e9 80 70 00 d3 37 1d 61 ac 11
63 76 d3 33 ca f1

The nova-compute log does show the instance attempting to be resumed
and failing:

2014-05-20 21:32:59.165 2289 TRACE nova.openstack.common.rpc.amqp
libvirtError: internal error: unable to execute QEMU command 'cont':
Resetting the Virtual Machine is required

using 'virsh reset' then 'virsh resume' on the instance I can very
briefly get it back into running state but it almost immediately fails
back to paused state with another failure and register dump in the
libvirt log.

Obviously my day is now filled with diffing installed package versions
on the working and broken systems and selectively force downgrading
them to see what is responsible, wonder if anyonne has seen this and
could possibly shorten my search?

Thanks,
-Jon