<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div class="gmail_quote"><div dir="ltr">On Fri, Nov 2, 2018 at 9:05 AM Bernd Bausch <<a href="mailto:berndbausch@gmail.com">berndbausch@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Welcome to OpenStack! I think the joy of troubleshooting it is
one of its main selling points. <br>
</p>
<p>The conductor log says "<i>libxenlight failed to create new
domain</i>" on various nodes. You should check the nova-compute
logs on those nodes for relevant messages. I suspect an
ill-configured interface between Nova Compute and libvirt/Xen. In
other words, double-check the libvirt and xen sections of
nova.conf on those nodes.<br>
</p>
<p>Then there is this message "<i>Invalid </i><i>input for field
'identity/password/user/password': None is not of type 'string'
....</i>". This is often caused by incompatible software. I.e.
you installed Openstack components that don't fit together. Might
also be caused by a configuration error.<br>
Or it's an ordinary bug, but it seems to occur on node 2 only.<br>
</p>
On 11/1/2018 11:01 PM, Minjun Hong wrote:<br>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">Hi everyone.
<div>I'm really new to OpenStack.</div>
<div>After I install essential components of OpenStack,
such as Nova, Keystone, etc, I attempted creating an
instance through Openstack command in the terminal.</div>
<div>But a trouble has occurred. It was logged
in "nova-conductor.log":</div>
<div><br>
</div>
<div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">2018-11-01
22:18:59.831 2570 ERROR nova.scheduler.utils
[req-12aae2ff-4186-4ab0-964c-35b335c3188a
cc22ec575cb44e53aced9ddf58d9e8d7
965ff1c2002d4c278b5f838dbdbbb780 - default default]
[instance: 684b0a7d-22b9-4c87-88f8-b1474d3f9cee] Error
from last host: node2 (node node2): [u'Traceback (most
recent call last):\n', u' File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py",
line 1840, in _do_build_and_run_instance\n
filter_properties, request_spec)\n', u' File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py",
line 2120, in _build_and_run_instance\n
instance_uuid=instance.uuid,
reason=six.text_type(e))\n', u"RescheduledException:
Build of instance 684b0a7d-22b9-4c87-88f8-b1474d3f9cee
was re-scheduled: Invalid input for field
'identity/password/user/password': None is not of type
'string'\n\nFailed validating 'type' in
schema['properties']['identity']['properties']['password']['properties']['user']['properties']['password']:\n
{'type': 'string'}\n\nOn
instance['identity']['password']['user']['password']:\n
None (HTTP 400) (Request-ID:
req-350beee4-2fed-4645-9e21-79806d7ebfe7)\n"]<br>
2018-11-01 22:18:59.833 2570 WARNING oslo_config.cfg
[req-12aae2ff-4186-4ab0-964c-35b335c3188a
cc22ec575cb44e53aced9ddf58d9e8d7
965ff1c2002d4c278b5f838dbdbbb780 - default default]
Option "os_region_name" from group "placement" is
deprecated. Use option "region-name" from group
"placement".<br>
2018-11-01 22:19:10.936 2571 ERROR
nova.scheduler.utils
[req-12aae2ff-4186-4ab0-964c-35b335c3188a
cc22ec575cb44e53aced9ddf58d9e8d7
965ff1c2002d4c278b5f838dbdbbb780 - default default]
[instance: 684b0a7d-22b9-4c87-88f8-b1474d3f9cee] Error
from last host: node4 (node node4): [u'Traceback (most
recent call last):\n', u' File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py",
line 1840, in _do_build_and_run_instance\n
filter_properties, request_spec)\n', u' File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py",
line 2120, in _build_and_run_instance\n
instance_uuid=instance.uuid,
reason=six.text_type(e))\n', u"RescheduledException:
Build of instance 684b0a7d-22b9-4c87-88f8-b1474d3f9cee
was re-scheduled: internal error: libxenlight failed
to create new domain 'instance-00000005'\n"]<br>
2018-11-01 22:19:21.783 2567 ERROR
nova.scheduler.utils
[req-12aae2ff-4186-4ab0-964c-35b335c3188a
cc22ec575cb44e53aced9ddf58d9e8d7
965ff1c2002d4c278b5f838dbdbbb780 - default default]
[instance: 684b0a7d-22b9-4c87-88f8-b1474d3f9cee] Error
from last host: node5 (node node5): [u'Traceback (most
recent call last):\n', u' File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py",
line 1840, in _do_build_and_run_instance\n
filter_properties, request_spec)\n', u' File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py",
line 2120, in _build_and_run_instance\n
instance_uuid=instance.uuid,
reason=six.text_type(e))\n', u"RescheduledException:
Build of instance 684b0a7d-22b9-4c87-88f8-b1474d3f9cee
was re-scheduled: internal error: libxenlight failed
to create new domain 'instance-00000005'\n"]<br>
2018-11-01 22:19:21.783 2567 WARNING
nova.scheduler.utils
[req-12aae2ff-4186-4ab0-964c-35b335c3188a
cc22ec575cb44e53aced9ddf58d9e8d7
965ff1c2002d4c278b5f838dbdbbb780 - default default]
Failed to compute_task_build_instances: Exceeded
maximum number of retries. Exceeded max scheduling
attempts 3 for instance
684b0a7d-22b9-4c87-88f8-b1474d3f9cee. Last exception:
internal error: libxenlight failed to create new
domain 'instance-00000005': MaxRetriesExceeded:
Exceeded maximum number of retries. Exceeded max
scheduling attempts 3 for instance
684b0a7d-22b9-4c87-88f8-b1474d3f9cee. Last exception:
internal error: libxenlight failed to create new
domain 'instance-00000005'<br>
2018-11-01 22:19:21.784 2567 WARNING
nova.scheduler.utils
[req-12aae2ff-4186-4ab0-964c-35b335c3188a
cc22ec575cb44e53aced9ddf58d9e8d7
965ff1c2002d4c278b5f838dbdbbb780 - default default]
[instance: 684b0a7d-22b9-4c87-88f8-b1474d3f9cee]
Setting instance to ERROR state.: MaxRetriesExceeded:
Exceeded maximum number of retries. Exceeded max
scheduling attempts 3 for instance
684b0a7d-22b9-4c87-88f8-b1474d3f9cee. Last exception:
internal error: libxenlight failed to create new
domain 'instance-00000005'</blockquote>
<div><br>
</div>
<div> I'm not sure which component is involved in this
trouble.</div>
</div>
<div>And libvirt and Xen have been successfully installed
on all compute node I have without any problem.</div>
<div><br>
</div>
<div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">nickeys@node2:~$
virsh create ./test.xml<br>
Domain guest1 created from ./test.xml<br>
nickeys@node2:~$ virsh list<br>
Id Name State<br>
--------------------------<br>
0 Domain-0 running<br>
1 guest1 running<br>
nickeys@node2:~$</blockquote>
<div><br>
</div>
<div>What should I check first for that issue ? </div>
</div>
<div>Your hint would be big help for me.</div>
<div><br>
</div>
<div>Thanks!</div>
</div>
</div>
</div>
</div>
<br>
<fieldset class="gmail-m_-6848492476500160991mimeAttachmentHeader"></fieldset>
<br>
<pre>_______________________________________________
Mailing list: <a class="gmail-m_-6848492476500160991moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a>
Post to : <a class="gmail-m_-6848492476500160991moz-txt-link-abbreviated" href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a>
Unsubscribe : <a class="gmail-m_-6848492476500160991moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a>
</pre>
</blockquote>
<br></div></blockquote><div><h3 class="gmail-iw" style="overflow:hidden;white-space:nowrap;max-width:92%;font-size:0.75rem;font-weight:inherit;margin:inherit;font-family:Roboto,RobotoDraft,Helvetica,Arial,sans-serif;letter-spacing:0.3px;color:rgb(95,99,104);line-height:20px"><br></h3></div><div>Thanks for your kind answer, Bernd.</div><div>I checked the configure files which you mentioned like it can cause the trouble. following files are my nova.conf and nova-compute.conf of the compute node ('node2').</div><div><br></div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"># /etc/nova/nova.conf </blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">[DEFAULT]<br>dhcpbridge_flagfile=/etc/nova/nova.conf<br>dhcpbridge=/usr/bin/nova-dhcpbridge<br>state_path=/var/lib/nova<br>lock_path=/var/lock/nova<br>force_dhcp_release=True<br>libvirt_use_virtio_for_bridges=True<br>verbose=True<br>ec2_private_dns_show_ip=True<br>api_paste_config=/etc/nova/api-paste.ini<br>enabled_apis=osapi_compute,metadata<br>transport_url = rabbit://<a href="mailto:admin%3A1234@10.150.3.88">admin:1234@10.150.3.88</a><br>my_ip = 10.150.21.182<br>use_neutron = True<br>firewall_driver = nova.virt.firewall.NoopFirewallDriver<br>[api]<br>auth_strategy = keystone<br>[vnc]<br>enabled = True<br>server_listen = 0.0.0.0<br>vncserver_proxyclient_address = 10.150.21.182<br>novncproxy_base_url = <a href="http://10.150.3.88:6080/vnc_auth.html">http://10.150.3.88:6080/vnc_auth.html</a><br>[glance]<br>api_servers = <a href="http://10.150.3.88:9292">http://10.150.3.88:9292</a><br>[oslo_concurrency]<br>lock_path = /var/lib/nova/tmp<br>[keystone_authtoken]<br>auth_url = <a href="http://10.150.3.88:5000/v3">http://10.150.3.88:5000/v3</a><br>memcached_servers = <a href="http://10.150.3.88:11211">10.150.3.88:11211</a><br>auth_type = password<br>project_domain_name = default<br>user_domain_name = default<br>project_name = service<br>username = nova<br>password = <KEYSTONE_PASS><br>[placement]<br>os_region_name = RegionOne<br>project_domain_name = Default<br>project_name = service<br>auth_type = password<br>user_domain_name = Default<br>auth_url = <a href="http://10.150.3.88:5000/v3">http://10.150.3.88:5000/v3</a><br>username = placement<br>password = <PLACEMENT_PASS><br>[database]<br>connection=sqlite:////var/lib/nova/nova.sqlite<br>[api_database]<br>connection=sqlite:////var/lib/nova/nova.sqlite<br>[neutron]<br>url = <a href="http://10.150.3.88:9696">http://10.150.3.88:9696</a><br>auth_url = <a href="http://10.150.3.88:5000">http://10.150.3.88:5000</a><br>auth_type = password<br>project_domain_name = default<br>user_domain_name = default<br>region_name = RegionOne<br>project_name = service<br>username = neutron<br>passsword = <NEUTRON_PASS></blockquote></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"># /etc/nova/nova-compute.conf<br>[DEFAULT]<br>compute_driver=libvirt.LibvirtDriver<br>[libvirt]<br>virt_type=xen</blockquote><div><br></div><div>And I set all the compute nodes I'm using with the same configure file above.</div><div>Also, I installed the OpenStack components, which belong to Queens version, following the official installation guide (<a href="https://docs.openstack.org/install-guide/">https://docs.openstack.org/install-guide/</a>).</div><div><br></div><div>By the way, I found something weird in '/var/log/nova/nova-compute.log' of the 'node2'.</div><div>After I manually created a VM by 'virsh' command (i.g. $ virsh create test.xml), nova-compute of 'node2' started to complain.</div><div>And on an other node which have not created a virtual machine by the 'virsh' command, there is not below error.</div><div>I guess that the nova-compute component was configured to qemu as virtualization type, but I have no idea where the setting value is.</div><div><br></div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager [req-ec324126-6dfc-4011-9f45-41d014e6f900 - - - - -] Error updating resources for node node2.: InvalidDiskInfo: Disk info file is invalid: qemu-img failed to execute on /home/caslab/xenguest1/domains/guest1/disk.img : Unexpected error while running command.<br>Command: /usr/bin/python2 -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info /home/caslab/xenguest1/domains/guest1/disk.img<br>Exit code: 1<br>Stdout: u''<br>Stderr: u"qemu-img: Could not open '/home/caslab/xenguest1/domains/guest1/disk.img': Could not open '/home/caslab/xenguest1/domains/guest1/disk.img': Permission denied\n"<br>2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager Traceback (most recent call last):<br>2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 7284, in update_available_resource_for_node<br>2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager rt.update_available_resource(context, nodename)<br>2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py", line 664, in update_available_resource<br>2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager resources = self.driver.get_available_resource(nodename)<br>2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 6440, in get_available_resource<br>2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager disk_over_committed = self._get_disk_over_committed_size_total()<br>2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 8019, in _get_disk_over_committed_size_total<br>2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager config, block_device_info)<br>2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 7918, in _get_instance_disk_info_from_config<br>2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager dk_size = disk_api.get_allocated_disk_size(path)<br>2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/virt/disk/api.py", line 147, in get_allocated_disk_size<br>2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager return images.qemu_img_info(path).disk_size<br>2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/virt/images.py", line 87, in qemu_img_info<br>2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager raise exception.InvalidDiskInfo(reason=msg)<br>2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager InvalidDiskInfo: Disk info file is invalid: qemu-img failed to execute on /home/caslab/xenguest1/domains/guest1/disk.img : Unexpected error while running command.<br>2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager Command: /usr/bin/python2 -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info /home/caslab/xenguest1/domains/guest1/disk.img<br>2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager Exit code: 1<br>2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager Stdout: u''<br>2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager Stderr: u"qemu-img: Could not open '/home/caslab/xenguest1/domains/guest1/disk.img': Could not open '/home/caslab/xenguest1/domains/guest1/disk.img': Permission denied\n"<br>2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager</blockquote></div><div><br></div><div>Thanks!</div><div><br></div></div></div></div></div></div></div></div>