[Openstack] [devstack][Nova] Unable to spawn VMs
Deepak Shetty
dpkshetty at gmail.com
Mon Jun 9 10:28:58 UTC 2014
Hi sergey
Don't see the .log in qemu/instnace-XXX at all
But there is more...
[stack at devstack-large-vm ~]$ [ready] nova show
*bc598be6-999b-4ab9-8a0a-1c4d29dd811a*
+--------------------------------------+----------------------------------------------------------------+
| Property |
Value |
+--------------------------------------+----------------------------------------------------------------+
| OS-DCF:diskConfig |
MANUAL |
| OS-EXT-AZ:availability_zone |
nova |
| OS-EXT-SRV-ATTR:host |
devstack-large-vm.localdomain |
| OS-EXT-SRV-ATTR:hypervisor_hostname |
devstack-large-vm.localdomain |
*| OS-EXT-SRV-ATTR:instance_name | instance-0000000a *
|
| OS-EXT-STS:power_state |
0 |
| OS-EXT-STS:task_state |
spawning |
| OS-EXT-STS:vm_state |
building |
| OS-SRV-USG:launched_at |
- |
| OS-SRV-USG:terminated_at |
- |
| accessIPv4
| |
| accessIPv6
| |
| config_drive
| |
| created |
2014-06-09T10:18:53Z |
| flavor | m1.tiny
(1) |
| hostId |
393d36ca7cc7c30957286e775b5808c8103b4b7be1af4f2163cc2fe4 |
| id |
bc598be6-999b-4ab9-8a0a-1c4d29dd811a |
| image | cirros-0.3.2-x86_64-uec
(99471623-273c-46ed-b1b0-5a58605aab76) |
| key_name |
- |
| metadata |
{} |
| name |
mynewvm |
| os-extended-volumes:volumes_attached |
[] |
| private network |
10.0.0.2 |
| progress |
0 |
| security_groups |
default |
| status |
BUILD |
| tenant_id |
1d0dedb6e6f344258a1ab44df3fcd4ee |
| updated |
2014-06-09T10:18:53Z |
| user_id |
730ccd3aaf7140789f4929edd71d3d31 |
+--------------------------------------+----------------------------------------------------------------+
[stack at devstack-large-vm ~]$ [ready] ps aux| grep qemu| grep instance
stack 11388 99.1 0.9 1054588 36884 ? Sl 10:18 4:03
/usr/bin/qemu-system-x86_64 -machine accel=kvm *-name
guestfs-tgx65m4jvhgmu8la* -S -machine pc-i440fx-1.6,accel=kvm,usb=off -cpu
host -m 500 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid
6bd85030-c2a0-4576-82d2-ff30a9e44024 -nographic -no-user-config -nodefaults
-chardev
socket,id=charmonitor,path=/opt/stack/.config/libvirt/qemu/lib/guestfs-tgx65m4jvhgmu8la.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc
base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet
-no-reboot -no-acpi -kernel /var/tmp/.guestfs-1001/appliance.d/kernel
-initrd /var/tmp/.guestfs-1001/appliance.d/initrd -append panic=1
console=ttyS0 udevtimeout=600 no_timer_check acpi=off printk.time=1
cgroup_disable=memory root=/dev/sdb selinux=0 TERM=screen -device
piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device
virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x2 -device
virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x3 *-drive
file=/opt/stack/data/nova/instances/bc598be6-999b-4ab9-8a0a-1c4d29dd811a/disk*,if=none,id=drive-scsi0-0-0-0,format=qcow2,cache=writeback
-device
scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1
-drive
file=/tmp/libguestfsagFFSQ/overlay1,if=none,id=drive-scsi0-0-1-0,format=qcow2,cache=unsafe
-device
scsi-hd,bus=scsi0.0,channel=0,scsi-id=1,lun=0,drive=drive-scsi0-0-1-0,id=scsi0-0-1-0
-chardev socket,id=charserial0,path=/tmp/libguestfsagFFSQ/console.sock
-device isa-serial,chardev=charserial0,id=serial0 -chardev
socket,id=charchannel0,path=/tmp/libguestfsagFFSQ/guestfsd.sock -device
virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.libguestfs.channel.0
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4
[root at devstack-large-vm qemu]# ls -lt
total 84
-rw-------. 1 root root 8279 Jun 9 09:59 instance-00000001.log
-rw-------. 1 root root 2070 Jun 9 09:29 instance-00000007.log
-rw-------. 1 root root 63444 Jun 6 09:20 instance-00000004.log
-rw-------. 1 root root 2071 Jun 6 07:12 instance-00000002.log
[root at devstack-large-vm qemu]# date
Mon Jun 9 10:24:32 UTC 2014
^^ No instance with 0a name above ^^
So the qemu process above is using the disk from my instance but its -name
is not instance-XXXXX tho'
*OS-EXT-SRV-ATTR:instance_name | instance-0000000a *
show 0a as instance name
virsh shows it as running...
[stack at devstack-large-vm ~]$ [ready] virsh list
Id Name State
----------------------------------------------------
* 2 guestfs-tgx65m4jvhgmu8la running*
thanx,
deepak
On Mon, Jun 9, 2014 at 3:50 PM, Sergey Kolekonov <skolekonov at mirantis.com>
wrote:
> Hi,
>
> have you inspected /var/log/libvirt/qemu/instance-xxxxxxxx.log files?
> Such problem can be connected with your hypervisor. Especially if you use
> nested kvm virtualization on your host machine.
>
>
> On Mon, Jun 9, 2014 at 2:06 PM, Deepak Shetty <dpkshetty at gmail.com> wrote:
>
>> (Hit send by mistake.. continuing below)
>>
>>
>> On Mon, Jun 9, 2014 at 3:32 PM, Deepak Shetty <dpkshetty at gmail.com>
>> wrote:
>>
>>> Hi All,
>>> The last time I sent this issue, I didn't had the right tags in the
>>> subject so sending with the right tags, in the hope that the right folks
>>> might help provide some clues.
>>>
>>> I am usign latest devstack on F20 and I see VMs stuck in 'spawning'
>>> state.
>>> No errors in n-sch, n-cpu and I checked other n-* screen logs too, no
>>> errors whatsoever
>>>
>>> There is enuf memory and disk space available for the tiny and small
>>> flavour VMs I am trying to run
>>>
>>>
>> 2014-06-09 09:57:19.935 AUDIT nova.compute.resource_tracker [-] Free ram
>> (MB): 1905
>> 2014-06-09 09:57:19.935 AUDIT nova.compute.resource_tracker [-] Free disk
>> (GB): 46
>> 2014-06-09 09:57:19.936 AUDIT nova.compute.resource_tracker [-] Free
>> VCPUS: 1
>> 2014-06-09 09:57:19.936 AUDIT nova.compute.resource_tracker [-] PCI
>> stats: []
>>
>> The only thing that i see other than INFO/DEbug is the below
>> 2014-06-09 09:57:42.700 WARNING nova.compute.manager [-] Found 3 in the
>> database and 0 on the hypervisor.
>>
>> What does that mean ? Does that say that Nova sees 3 VMs in its DB but a
>> query to the hyp (via libvirt I assume) returns none ?
>>
>> Interestingly I can see the qemu process for my VM that nova says is
>> stuck in Spawning state and virsh list also lists' it
>> But i don't see the libvirt.xml for this stuck VM (from nova's
>> perspective) in the /opt/stack/data/nova/instances/<uuid> folder.
>>
>> I was trying to figure what this means ? Where is nova stuck ? Screen
>> logs just doesn't point to anything useful to debug further.
>>
>> From the n-cpu logs i see the "Creating image" info log from
>> nova.virt.libvirt and thats the last
>> I don't see the info log "Creating isntance" from nova.virt.. so it looks
>> like its stuck somewhere between creating image and spawning instance.. but
>> virsh list and qemu process says otherwise
>>
>> Looking for some debug hints here
>>
>> thanx,
>> deepak
>>
>> _______________________________________________
>> Mailing list:
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>> Post to : openstack at lists.openstack.org
>> Unsubscribe :
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20140609/5412e9a6/attachment.html>
More information about the Openstack
mailing list