[Openstack] Troubles on creating an Xen instance via libvirt
Bernd Bausch
berndbausch at gmail.com
Fri Nov 2 12:11:46 UTC 2018
This is the part that determines the hypervisor:
# /etc/nova/nova-compute.conf
[DEFAULT]
compute_driver=libvirt.LibvirtDriver
[libvirt]
virt_type=xen
Nova uses libvirt to manage VMs, and libvirt uses Xen. According to the
Nova/Xen configuration guide
<https://docs.openstack.org/nova/latest/admin/configuration/hypervisor-xen-libvirt.html>,
it is enough to set these variables. However, if you have not read that
config page, do it now. Are you using correct image formats? Do you find
anything in the other logfiles mentioned on that page?
The references to /qemu-img /in the nova-compute log don't mean that
qemu is used. Nova uses /qemu-img/ as a general tool for converting VM
images, no matter which hypervisor is configured.
However, something prevents Nova to access one of the disk images,
namely /home/caslab/xenguest1/domains/guest1/disk.img. Is /guest1/ the
VM you started manually?
I see:
Unexpected error while running command.
Command: /usr/bin/python2 -m oslo_concurrency.prlimit --as=1073741824
--cpu=30 -- env LC_ALL=C LANG=C *qemu-img info
/home/caslab/xenguest1/domains/guest1/disk.img*
and
Could not open '/home/caslab/xenguest1/domains/guest1/disk.img':
Permission denied
I don't know, though, whether these errors are related to your problem.
Why is Libvirt or Xen unable to start your instances? I suggest you look
for other errors in the nova-compute.log files.
Bernd
On 11/2/2018 6:30 PM, Minjun Hong wrote:
> On Fri, Nov 2, 2018 at 9:05 AM Bernd Bausch <berndbausch at gmail.com
> <mailto:berndbausch at gmail.com>> wrote:
>
> Welcome to OpenStack! I think the joy of troubleshooting it is one
> of its main selling points.
>
> The conductor log says "/libxenlight failed to create new domain/"
> on various nodes. You should check the nova-compute logs on those
> nodes for relevant messages. I suspect an ill-configured interface
> between Nova Compute and libvirt/Xen. In other words, double-check
> the libvirt and xen sections of nova.conf on those nodes.
>
> Then there is this message "/Invalid //input for field
> 'identity/password/user/password': None is not of type 'string'
> ..../". This is often caused by incompatible software. I.e. you
> installed Openstack components that don't fit together. Might also
> be caused by a configuration error.
> Or it's an ordinary bug, but it seems to occur on node 2 only.
>
> On 11/1/2018 11:01 PM, Minjun Hong wrote:
>> Hi everyone.
>> I'm really new to OpenStack.
>> After I install essential components of OpenStack, such as Nova,
>> Keystone, etc, I attempted creating an instance through Openstack
>> command in the terminal.
>> But a trouble has occurred. It was logged in "nova-conductor.log":
>>
>> 2018-11-01 22:18:59.831 2570 ERROR nova.scheduler.utils
>> [req-12aae2ff-4186-4ab0-964c-35b335c3188a
>> cc22ec575cb44e53aced9ddf58d9e8d7
>> 965ff1c2002d4c278b5f838dbdbbb780 - default default]
>> [instance: 684b0a7d-22b9-4c87-88f8-b1474d3f9cee] Error from
>> last host: node2 (node node2): [u'Traceback (most recent call
>> last):\n', u' File
>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py",
>> line 1840, in _do_build_and_run_instance\n
>> filter_properties, request_spec)\n', u' File
>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py",
>> line 2120, in _build_and_run_instance\n
>> instance_uuid=instance.uuid, reason=six.text_type(e))\n',
>> u"RescheduledException: Build of instance
>> 684b0a7d-22b9-4c87-88f8-b1474d3f9cee was re-scheduled:
>> Invalid input for field 'identity/password/user/password':
>> None is not of type 'string'\n\nFailed validating 'type' in
>> schema['properties']['identity']['properties']['password']['properties']['user']['properties']['password']:\n
>> {'type': 'string'}\n\nOn
>> instance['identity']['password']['user']['password']:\n
>> None (HTTP 400) (Request-ID:
>> req-350beee4-2fed-4645-9e21-79806d7ebfe7)\n"]
>> 2018-11-01 22:18:59.833 2570 WARNING oslo_config.cfg
>> [req-12aae2ff-4186-4ab0-964c-35b335c3188a
>> cc22ec575cb44e53aced9ddf58d9e8d7
>> 965ff1c2002d4c278b5f838dbdbbb780 - default default] Option
>> "os_region_name" from group "placement" is deprecated. Use
>> option "region-name" from group "placement".
>> 2018-11-01 22:19:10.936 2571 ERROR nova.scheduler.utils
>> [req-12aae2ff-4186-4ab0-964c-35b335c3188a
>> cc22ec575cb44e53aced9ddf58d9e8d7
>> 965ff1c2002d4c278b5f838dbdbbb780 - default default]
>> [instance: 684b0a7d-22b9-4c87-88f8-b1474d3f9cee] Error from
>> last host: node4 (node node4): [u'Traceback (most recent call
>> last):\n', u' File
>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py",
>> line 1840, in _do_build_and_run_instance\n
>> filter_properties, request_spec)\n', u' File
>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py",
>> line 2120, in _build_and_run_instance\n
>> instance_uuid=instance.uuid, reason=six.text_type(e))\n',
>> u"RescheduledException: Build of instance
>> 684b0a7d-22b9-4c87-88f8-b1474d3f9cee was re-scheduled:
>> internal error: libxenlight failed to create new domain
>> 'instance-00000005'\n"]
>> 2018-11-01 22:19:21.783 2567 ERROR nova.scheduler.utils
>> [req-12aae2ff-4186-4ab0-964c-35b335c3188a
>> cc22ec575cb44e53aced9ddf58d9e8d7
>> 965ff1c2002d4c278b5f838dbdbbb780 - default default]
>> [instance: 684b0a7d-22b9-4c87-88f8-b1474d3f9cee] Error from
>> last host: node5 (node node5): [u'Traceback (most recent call
>> last):\n', u' File
>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py",
>> line 1840, in _do_build_and_run_instance\n
>> filter_properties, request_spec)\n', u' File
>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py",
>> line 2120, in _build_and_run_instance\n
>> instance_uuid=instance.uuid, reason=six.text_type(e))\n',
>> u"RescheduledException: Build of instance
>> 684b0a7d-22b9-4c87-88f8-b1474d3f9cee was re-scheduled:
>> internal error: libxenlight failed to create new domain
>> 'instance-00000005'\n"]
>> 2018-11-01 22:19:21.783 2567 WARNING nova.scheduler.utils
>> [req-12aae2ff-4186-4ab0-964c-35b335c3188a
>> cc22ec575cb44e53aced9ddf58d9e8d7
>> 965ff1c2002d4c278b5f838dbdbbb780 - default default] Failed to
>> compute_task_build_instances: Exceeded maximum number of
>> retries. Exceeded max scheduling attempts 3 for instance
>> 684b0a7d-22b9-4c87-88f8-b1474d3f9cee. Last exception:
>> internal error: libxenlight failed to create new domain
>> 'instance-00000005': MaxRetriesExceeded: Exceeded maximum
>> number of retries. Exceeded max scheduling attempts 3 for
>> instance 684b0a7d-22b9-4c87-88f8-b1474d3f9cee. Last
>> exception: internal error: libxenlight failed to create new
>> domain 'instance-00000005'
>> 2018-11-01 22:19:21.784 2567 WARNING nova.scheduler.utils
>> [req-12aae2ff-4186-4ab0-964c-35b335c3188a
>> cc22ec575cb44e53aced9ddf58d9e8d7
>> 965ff1c2002d4c278b5f838dbdbbb780 - default default]
>> [instance: 684b0a7d-22b9-4c87-88f8-b1474d3f9cee] Setting
>> instance to ERROR state.: MaxRetriesExceeded: Exceeded
>> maximum number of retries. Exceeded max scheduling attempts 3
>> for instance 684b0a7d-22b9-4c87-88f8-b1474d3f9cee. Last
>> exception: internal error: libxenlight failed to create new
>> domain 'instance-00000005'
>>
>>
>> I'm not sure which component is involved in this trouble.
>> And libvirt and Xen have been successfully installed on all
>> compute node I have without any problem.
>>
>> nickeys at node2:~$ virsh create ./test.xml
>> Domain guest1 created from ./test.xml
>> nickeys at node2:~$ virsh list
>> Id Name State
>> --------------------------
>> 0 Domain-0 running
>> 1 guest1 running
>> nickeys at node2:~$
>>
>>
>> What should I check first for that issue ?
>> Your hint would be big help for me.
>>
>> Thanks!
>>
>>
>> _______________________________________________
>> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>> Post to : openstack at lists.openstack.org <mailto:openstack at lists.openstack.org>
>> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>
>
>
> Thanks for your kind answer, Bernd.
> I checked the configure files which you mentioned like it can cause
> the trouble. following files are my nova.conf and nova-compute.conf of
> the compute node ('node2').
>
> # /etc/nova/nova.conf
>
> [DEFAULT]
> dhcpbridge_flagfile=/etc/nova/nova.conf
> dhcpbridge=/usr/bin/nova-dhcpbridge
> state_path=/var/lib/nova
> lock_path=/var/lock/nova
> force_dhcp_release=True
> libvirt_use_virtio_for_bridges=True
> verbose=True
> ec2_private_dns_show_ip=True
> api_paste_config=/etc/nova/api-paste.ini
> enabled_apis=osapi_compute,metadata
> transport_url = rabbit://admin:1234@10.150.3.88
> <mailto:admin%3A1234 at 10.150.3.88>
> my_ip = 10.150.21.182
> use_neutron = True
> firewall_driver = nova.virt.firewall.NoopFirewallDriver
> [api]
> auth_strategy = keystone
> [vnc]
> enabled = True
> server_listen = 0.0.0.0
> vncserver_proxyclient_address = 10.150.21.182
> novncproxy_base_url = http://10.150.3.88:6080/vnc_auth.html
> [glance]
> api_servers = http://10.150.3.88:9292
> [oslo_concurrency]
> lock_path = /var/lib/nova/tmp
> [keystone_authtoken]
> auth_url = http://10.150.3.88:5000/v3
> memcached_servers = 10.150.3.88:11211 <http://10.150.3.88:11211>
> auth_type = password
> project_domain_name = default
> user_domain_name = default
> project_name = service
> username = nova
> password = <KEYSTONE_PASS>
> [placement]
> os_region_name = RegionOne
> project_domain_name = Default
> project_name = service
> auth_type = password
> user_domain_name = Default
> auth_url = http://10.150.3.88:5000/v3
> username = placement
> password = <PLACEMENT_PASS>
> [database]
> connection=sqlite:////var/lib/nova/nova.sqlite
> [api_database]
> connection=sqlite:////var/lib/nova/nova.sqlite
> [neutron]
> url = http://10.150.3.88:9696
> auth_url = http://10.150.3.88:5000
> auth_type = password
> project_domain_name = default
> user_domain_name = default
> region_name = RegionOne
> project_name = service
> username = neutron
> passsword = <NEUTRON_PASS>
>
>
> # /etc/nova/nova-compute.conf
> [DEFAULT]
> compute_driver=libvirt.LibvirtDriver
> [libvirt]
> virt_type=xen
>
>
> And I set all the compute nodes I'm using with the same configure file
> above.
> Also, I installed the OpenStack components, which belong to Queens
> version, following the official installation guide
> (https://docs.openstack.org/install-guide/).
>
> By the way, I found something weird in
> '/var/log/nova/nova-compute.log' of the 'node2'.
> After I manually created a VM by 'virsh' command (i.g. $ virsh create
> test.xml), nova-compute of 'node2' started to complain.
> And on an other node which have not created a virtual machine by the
> 'virsh' command, there is not below error.
> I guess that the nova-compute component was configured to qemu as
> virtualization type, but I have no idea where the setting value is.
>
> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager
> [req-ec324126-6dfc-4011-9f45-41d014e6f900 - - - - -] Error
> updating resources for node node2.: InvalidDiskInfo: Disk info
> file is invalid: qemu-img failed to execute on
> /home/caslab/xenguest1/domains/guest1/disk.img : Unexpected error
> while running command.
> Command: /usr/bin/python2 -m oslo_concurrency.prlimit
> --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info
> /home/caslab/xenguest1/domains/guest1/disk.img
> Exit code: 1
> Stdout: u''
> Stderr: u"qemu-img: Could not open
> '/home/caslab/xenguest1/domains/guest1/disk.img': Could not open
> '/home/caslab/xenguest1/domains/guest1/disk.img': Permission denied\n"
> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager Traceback
> (most recent call last):
> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager File
> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line
> 7284, in update_available_resource_for_node
> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager
> rt.update_available_resource(context, nodename)
> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager File
> "/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py",
> line 664, in update_available_resource
> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager
> resources = self.driver.get_available_resource(nodename)
> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager File
> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py",
> line 6440, in get_available_resource
> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager
> disk_over_committed = self._get_disk_over_committed_size_total()
> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager File
> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py",
> line 8019, in _get_disk_over_committed_size_total
> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager
> config, block_device_info)
> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager File
> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py",
> line 7918, in _get_instance_disk_info_from_config
> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager
> dk_size = disk_api.get_allocated_disk_size(path)
> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager File
> "/usr/lib/python2.7/dist-packages/nova/virt/disk/api.py", line
> 147, in get_allocated_disk_size
> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager
> return images.qemu_img_info(path).disk_size
> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager File
> "/usr/lib/python2.7/dist-packages/nova/virt/images.py", line 87,
> in qemu_img_info
> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager raise
> exception.InvalidDiskInfo(reason=msg)
> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager
> InvalidDiskInfo: Disk info file is invalid: qemu-img failed to
> execute on /home/caslab/xenguest1/domains/guest1/disk.img :
> Unexpected error while running command.
> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager Command:
> /usr/bin/python2 -m oslo_concurrency.prlimit --as=1073741824
> --cpu=30 -- env LC_ALL=C LANG=C qemu-img info
> /home/caslab/xenguest1/domains/guest1/disk.img
> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager Exit code: 1
> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager Stdout: u''
> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager Stderr:
> u"qemu-img: Could not open
> '/home/caslab/xenguest1/domains/guest1/disk.img': Could not open
> '/home/caslab/xenguest1/domains/guest1/disk.img': Permission denied\n"
> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager
>
>
> Thanks!
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20181102/eb1482b5/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20181102/eb1482b5/attachment.sig>
More information about the Openstack
mailing list