- openstack-discuss - lists.openstack.org

[Openstack] [Nova] Instance creation problem
by Minjun Hong 28 Nov '18

28 Nov '18

Hi. I'm setting up a Openstack system on the servers of my laboratory. While I try to create an instance, a problem has occurred! Instance creation was failed and it seems that libvirt failed to attaching the vif to the instance. When I create a virtual machine by using virsh tool (libvirt) manually, there was no problem. I add the logs as follows: 1. controller node > "/var/log/nova/nova-conductor.log" > 2018-11-28 21:18:13.033 2657 ERROR nova.scheduler.utils > [req-291fdb2d-fa94-461c-9f5f-68d340791c77 3367829d9c004653bdc9102443bd4736 > 47270e4fb58045dc88b6f0f736286ffc - default default] [instance: > 9c2d08f3-0680-4709-a64d-ae1729a11304] Error from last host: node1 (node > node1): [u'Traceback (most recent call last):\n', u' File > "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1840, in > _do_build_and_run_instance\n filter_properties, request_spec)\n', u' > File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2120, > in _build_and_run_instance\n instance_uuid=instance.uuid, > reason=six.text_type(e))\n', u"RescheduledException: Build of instance > 9c2d08f3-0680-4709-a64d-ae1729a11304 was re-scheduled: internal error: > libxenlight failed to create new domain 'instance-00000008'\n"] > 2018-11-28 21:18:13.033 2657 WARNING nova.scheduler.utils > [req-291fdb2d-fa94-461c-9f5f-68d340791c77 3367829d9c004653bdc9102443bd4736 > 47270e4fb58045dc88b6f0f736286ffc - default default] Failed to > compute_task_build_instances: Exceeded maximum number of retries. Exceeded > max scheduling attempts 3 for instance > 9c2d08f3-0680-4709-a64d-ae1729a11304. Last exception: internal error: > libxenlight failed to create new domain 'instance-00000008': > MaxRetriesExceeded: Exceeded maximum number of retries. Exceeded max > scheduling attempts 3 for instance 9c2d08f3-0680-4709-a64d-ae1729a11304. > Last exception: internal error: libxenlight failed to create new domain > 'instance-00000008' > 2018-11-28 21:18:13.034 2657 WARNING nova.scheduler.utils > [req-291fdb2d-fa94-461c-9f5f-68d340791c77 3367829d9c004653bdc9102443bd4736 > 47270e4fb58045dc88b6f0f736286ffc - default default] [instance: > 9c2d08f3-0680-4709-a64d-ae1729a11304] Setting instance to ERROR state.: > MaxRetriesExceeded: Exceeded maximum number of retries. Exceeded max > scheduling attempts 3 for instance 9c2d08f3-0680-4709-a64d-ae1729a11304. > Last exception: internal error: libxenlight failed to create new domain > 'instance-00000008' > 2018-11-28 21:18:13.067 2657 WARNING oslo_config.cfg > [req-291fdb2d-fa94-461c-9f5f-68d340791c77 3367829d9c004653bdc9102443bd4736 > 47270e4fb58045dc88b6f0f736286ffc - default default] Option "url" from group > "neutron" is deprecated for removal (Endpoint lookup uses the service > catalog via common keystoneauth1 Adapter configuration options. In the > current release, "url" will override this behavior, but will be ignored > and/or removed in a future release. To achieve the same result, use the > endpoint_override option instead.). Its value may be silently ignored in > the future. > "/var/log/neutron/neutron-linuxbridge-agent.log" > 2018-11-28 17:41:45.593 2476 INFO > neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] > Interface mappings: {'provider': 'enp1s0f1'} > 2018-11-28 17:41:45.593 2476 INFO > neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] > Bridge mappings: {} > 2018-11-28 17:41:45.624 2476 INFO > neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] > Agent initialized successfully, now running... > 2018-11-28 17:41:45.901 2476 INFO > neutron.plugins.ml2.drivers.agent._common_agent > [req-c447894c-9013-4bec-82e4-8b501414184d - - - - -] RPC agent_id: > lba0369fa2714a > 2018-11-28 17:41:45.907 2476 INFO neutron.agent.agent_extensions_manager > [req-c447894c-9013-4bec-82e4-8b501414184d - - - - -] Loaded agent > extensions: [] > 2018-11-28 17:41:46.121 2476 INFO > neutron.plugins.ml2.drivers.agent._common_agent > [req-c447894c-9013-4bec-82e4-8b501414184d - - - - -] Linux bridge agent > Agent RPC Daemon Started! > 2018-11-28 17:41:46.122 2476 INFO > neutron.plugins.ml2.drivers.agent._common_agent > [req-c447894c-9013-4bec-82e4-8b501414184d - - - - -] Linux bridge agent > Agent out of sync with plugin! > 2018-11-28 17:41:46.512 2476 INFO > neutron.plugins.ml2.drivers.linuxbridge.agent.arp_protect > [req-c447894c-9013-4bec-82e4-8b501414184d - - - - -] Clearing orphaned ARP > spoofing entries for devices [] > 2018-11-28 17:41:47.020 2476 INFO > neutron.plugins.ml2.drivers.linuxbridge.agent.arp_protect > [req-c447894c-9013-4bec-82e4-8b501414184d - - - - -] Clearing orphaned ARP > spoofing entries for devices [] > 2018-11-28 17:42:45.981 2476 ERROR > neutron.plugins.ml2.drivers.agent._common_agent [-] Failed reporting > state!: MessagingTimeout: Timed out waiting for a reply to message ID > 20ef587240864120b878559ab821adbf > 2018-11-28 17:42:45.981 2476 ERROR > neutron.plugins.ml2.drivers.agent._common_agent Traceback (most recent call > last): > 2018-11-28 17:42:45.981 2476 ERROR > neutron.plugins.ml2.drivers.agent._common_agent File > "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/agent/_common_agent.py", > line 128, in _report_state > 2018-11-28 17:42:45.981 2476 ERROR > neutron.plugins.ml2.drivers.agent._common_agent True) > 2018-11-28 17:42:45.981 2476 ERROR > neutron.plugins.ml2.drivers.agent._common_agent File > "/usr/lib/python2.7/dist-packages/neutron/agent/rpc.py", line 93, in > report_state > 2018-11-28 17:42:45.981 2476 ERROR > neutron.plugins.ml2.drivers.agent._common_agent return method(context, > 'report_state', **kwargs) > 2018-11-28 17:42:45.981 2476 ERROR > neutron.plugins.ml2.drivers.agent._common_agent File > "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 174, > in call > 2018-11-28 17:42:45.981 2476 ERROR > neutron.plugins.ml2.drivers.agent._common_agent retry=self.retry) > 2018-11-28 17:42:45.981 2476 ERROR > neutron.plugins.ml2.drivers.agent._common_agent File > "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 131, > in _send > 2018-11-28 17:42:45.981 2476 ERROR > neutron.plugins.ml2.drivers.agent._common_agent timeout=timeout, > retry=retry) > 2018-11-28 17:42:45.981 2476 ERROR > neutron.plugins.ml2.drivers.agent._common_agent File > "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", > line 559, in send > 2018-11-28 17:42:45.981 2476 ERROR > neutron.plugins.ml2.drivers.agent._common_agent retry=retry) > 2018-11-28 17:42:45.981 2476 ERROR > neutron.plugins.ml2.drivers.agent._common_agent File > "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", > line 548, in _send > 2018-11-28 17:42:45.981 2476 ERROR > neutron.plugins.ml2.drivers.agent._common_agent result = > self._waiter.wait(msg_id, timeout) > 2018-11-28 17:42:45.981 2476 ERROR > neutron.plugins.ml2.drivers.agent._common_agent File > "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", > line 440, in wait > 2018-11-28 17:42:45.981 2476 ERROR > neutron.plugins.ml2.drivers.agent._common_agent message = > self.waiters.get(msg_id, timeout=timeout) > 2018-11-28 17:42:45.981 2476 ERROR > neutron.plugins.ml2.drivers.agent._common_agent File > "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", > line 328, in get > 2018-11-28 17:42:45.981 2476 ERROR > neutron.plugins.ml2.drivers.agent._common_agent 'to message ID %s' % > msg_id) > 2018-11-28 17:42:45.981 2476 ERROR > neutron.plugins.ml2.drivers.agent._common_agent MessagingTimeout: Timed out > waiting for a reply to message ID 20ef587240864120b878559ab821adbf > 2018-11-28 17:42:45.981 2476 ERROR > neutron.plugins.ml2.drivers.agent._common_agent > 2018-11-28 17:42:45.986 2476 WARNING oslo.service.loopingcall [-] Function > 'neutron.plugins.ml2.drivers.agent._common_agent.CommonAgentLoop._report_state' > run outlasted interval by 30.07 sec > 2018-11-28 17:42:46.055 2476 INFO > neutron.plugins.ml2.drivers.agent._common_agent [-] Linux bridge agent > Agent has just been revived. Doing a full sync. > 2018-11-28 17:42:46.156 2476 INFO > neutron.plugins.ml2.drivers.agent._common_agent > [req-c447894c-9013-4bec-82e4-8b501414184d - - - - -] Linux bridge agent > Agent out of sync with plugin! > 2018-11-28 17:43:40.189 2476 INFO neutron.agent.securitygroups_rpc > [req-c447894c-9013-4bec-82e4-8b501414184d - - - - -] Preparing filters for > devices set(['tap4a09374a-7f']) > 2018-11-28 17:43:40.935 2476 INFO > neutron.plugins.ml2.drivers.agent._common_agent > [req-c447894c-9013-4bec-82e4-8b501414184d - - - - -] Port tap4a09374a-7f > updated. Details: {u'profile': {}, u'network_qos_policy_id': None, > u'qos_policy_id': None, u'allowed_address_pairs': [], u'admin_state_up': > True, u'network_id': u'87fa0d9c-5ed3-4332-8782-0d4139eed7f3', > u'segmentation_id': None, u'mtu': 1500, u'device_owner': u'network:dhcp', > u'physical_network': u'provider', u'mac_address': u'fa:16:3e:ab:0e:84', > u'device': u'tap4a09374a-7f', u'port_security_enabled': False, u'port_id': > u'4a09374a-7fa5-42c2-9430-67a0cd65336c', u'fixed_ips': [{u'subnet_id': > u'e95946a8-070c-42c4-877e-279e6e7acc7e', u'ip_address': u'192.0.10.4'}], > u'network_type': u'flat'} > 2018-11-28 17:43:41.124 2476 INFO > neutron.plugins.ml2.drivers.linuxbridge.agent.arp_protect > [req-c447894c-9013-4bec-82e4-8b501414184d - - - - -] Skipping ARP spoofing > rules for port 'tap4a09374a-7f' because it has port security disabled "/var/log/neutron/neutron-server.log" > 2018-11-28 17:30:02.130 15995 INFO neutron.pecan_wsgi.hooks.translation > [req-e7554d70-c84d-46d5-8ff6-a536fb35664c 29a3a16fd2484ee9bed834a3835545af > 5ebe3484974848b182a381127cb35a22 - default default] GET failed (client > error): The resource could not be found. > 2018-11-28 17:30:02.130 15995 INFO neutron.wsgi > [req-e7554d70-c84d-46d5-8ff6-a536fb35664c 29a3a16fd2484ee9bed834a3835545af > 5ebe3484974848b182a381127cb35a22 - default default] 10.150.21.183 "GET > /v2.0/floatingips?fixed_ip_address=192.0.10.10&port_id=8ab0d544-ec5b-4e69-95f4-1f06f7b53bb4 > HTTP/1.1" status: 404 len: 309 time: 0.0072770 > 2018-11-28 17:30:02.167 15995 INFO neutron.wsgi > [req-a2b5f53b-8992-4178-b0b6-55be9d1a0f32 29a3a16fd2484ee9bed834a3835545af > 5ebe3484974848b182a381127cb35a22 - default default] 10.150.21.183 "GET > /v2.0/subnets?id=e95946a8-070c-42c4-877e-279e6e7acc7e HTTP/1.1" status: > 200 len: 822 time: 0.0341990 > 2018-11-28 17:30:02.199 15995 INFO neutron.wsgi > [req-9e1a51e9-9ccc-4226-a78f-0420ff95c147 29a3a16fd2484ee9bed834a3835545af > 5ebe3484974848b182a381127cb35a22 - default default] 10.150.21.183 "GET > /v2.0/ports?network_id=87fa0d9c-5ed3-4332-8782-0d4139eed7f3&device_owner=network%3Adhcp > HTTP/1.1" status: 200 len: 1080 time: 0.0300300 > 2018-11-28 17:30:02.584 15995 INFO neutron.notifiers.nova [-] Nova event > response: {u'status': u'completed', u'tag': > u'8ab0d544-ec5b-4e69-95f4-1f06f7b53bb4', u'name': u'network-changed', > u'server_uuid': u'a9afc2d4-f4c9-429b-9773-4de8a3eaefa5', u'code': 200} > 2018-11-28 17:30:02.628 15995 INFO neutron.wsgi > [req-73265cf5-5f0d-4217-b716-caa2fb906abf 29a3a16fd2484ee9bed834a3835545af > 5ebe3484974848b182a381127cb35a22 - default default] 10.150.21.183 "GET > /v2.0/ports?tenant_id=47270e4fb58045dc88b6f0f736286ffc&device_id=a9afc2d4-f4c9-429b-9773-4de8a3eaefa5 > HTTP/1.1" status: 200 len: 1062 time: 0.0316660 > 2018-11-28 17:30:02.696 15995 INFO neutron.wsgi > [req-ed53b92c-3033-4b4a-ade4-fdc5a3463e8c 29a3a16fd2484ee9bed834a3835545af > 5ebe3484974848b182a381127cb35a22 - default default] 10.150.21.183 "GET > /v2.0/networks?id=87fa0d9c-5ed3-4332-8782-0d4139eed7f3 HTTP/1.1" status: > 200 len: 872 time: 0.0655539 > 2018-11-28 17:30:02.702 15995 WARNING neutron.pecan_wsgi.controllers.root > [req-ccd8c9b8-d2cf-40f7-b53b-5936bd0c9a6d 29a3a16fd2484ee9bed834a3835545af > 5ebe3484974848b182a381127cb35a22 - default default] No controller found > for: floatingips - returning response code 404: PecanNotFound 2. compute node > "/var/log/libvirt/libxl/libxl-driver.log" > 2018-11-28 08:40:31.920+0000: libxl: > libxl_event.c:681:libxl__ev_xswatch_deregister: remove watch for path > @releaseDomain: Bad file descriptor > 2018-11-28 09:57:01.707+0000: libxl: > libxl_exec.c:118:libxl_report_child_exitstatus: /etc/xen/scripts/vif-bridge > online [2536] exited with error status 1 > 2018-11-28 09:57:01.708+0000: libxl: > libxl_device.c:1286:device_hotplug_child_death_cb: script: ip link set > vif1.0 name tape5a239a8-6e failed > 2018-11-28 09:57:01.708+0000: libxl: > libxl_create.c:1522:domcreate_attach_devices: Domain 1:unable to add vif > devices "/var/log/xen/xen-hotplug.log" > RTNETLINK answers: Device or resource busy "/var/log/nova/nova-compute.log" > : libvirtError: internal error: libxenlight failed to create new domain > 'instance-00000008' > 2018-11-28 21:18:11.350 2384 ERROR nova.virt.libvirt.driver > [req-b3e761b7-00fa-4930-bd9e-4330f8440c03 48086750fa13420888601964bb6a9d0d > 5ebe3484974848b182a381127cb35a22 - default default] [instance: > 9c2d08f3-0680-4709-a64d-ae1729a11304] Failed to start libvirt guest: > libvirtError: internal error: libxenlight failed to create new domain > 'instance-00000008' > 2018-11-28 21:18:11.352 2384 INFO os_vif > [req-b3e761b7-00fa-4930-bd9e-4330f8440c03 48086750fa13420888601964bb6a9d0d > 5ebe3484974848b182a381127cb35a22 - default default] Successfully unplugged > vif > VIFBridge(active=False,address=fa:16:3e:6b:e4:b7,bridge_name='brq87fa0d9c-5e',has_traffic_filtering=True,id=484807ca-8c7c-4509-a5f5-ed7e5fd2078f,network=Network(87fa0d9c-5ed3-4332-8782-0d4139eed7f3),plugin='linux_bridge',port_profile=<?>,preserve_on_delete=False,vif_name='tap484807ca-8c') > 2018-11-28 21:18:11.554 2384 INFO nova.virt.libvirt.driver > [req-b3e761b7-00fa-4930-bd9e-4330f8440c03 48086750fa13420888601964bb6a9d0d > 5ebe3484974848b182a381127cb35a22 - default default] [instance: > 9c2d08f3-0680-4709-a64d-ae1729a11304] Deleting instance files > /var/lib/nova/instances/9c2d08f3-0680-4709-a64d-ae1729a11304_del > 2018-11-28 21:18:11.556 2384 INFO nova.virt.libvirt.driver > [req-b3e761b7-00fa-4930-bd9e-4330f8440c03 48086750fa13420888601964bb6a9d0d > 5ebe3484974848b182a381127cb35a22 - default default] [instance: > 9c2d08f3-0680-4709-a64d-ae1729a11304] Deletion of > /var/lib/nova/instances/9c2d08f3-0680-4709-a64d-ae1729a11304_del complete > 2018-11-28 21:18:11.614 2384 ERROR nova.compute.manager > [req-b3e761b7-00fa-4930-bd9e-4330f8440c03 48086750fa13420888601964bb6a9d0d > 5ebe3484974848b182a381127cb35a22 - default default] [instance: > 9c2d08f3-0680-4709-a64d-ae1729a11304] Instance failed to spawn: > libvirtError: internal error: libxenlight failed to create new domain > 'instance-00000008' > 2018-11-28 21:18:11.614 2384 ERROR nova.compute.manager [instance: > 9c2d08f3-0680-4709-a64d-ae1729a11304] Traceback (most recent call last): > 2018-11-28 21:18:11.614 2384 ERROR nova.compute.manager [instance: > 9c2d08f3-0680-4709-a64d-ae1729a11304] File > "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2251, in > _build_resources > 2018-11-28 21:18:11.614 2384 ERROR nova.compute.manager [instance: > 9c2d08f3-0680-4709-a64d-ae1729a11304] yield resources > 2018-11-28 21:18:11.614 2384 ERROR nova.compute.manager [instance: > 9c2d08f3-0680-4709-a64d-ae1729a11304] File > "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2031, in > _build_and_run_instance > 2018-11-28 21:18:11.614 2384 ERROR nova.compute.manager [instance: > 9c2d08f3-0680-4709-a64d-ae1729a11304] > block_device_info=block_device_info) > 2018-11-28 21:18:11.614 2384 ERROR nova.compute.manager [instance: > 9c2d08f3-0680-4709-a64d-ae1729a11304] File > "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3089, > in spawn > 2018-11-28 21:18:11.614 2384 ERROR nova.compute.manager [instance: > 9c2d08f3-0680-4709-a64d-ae1729a11304] destroy_disks_on_failure=True) > 2018-11-28 21:18:11.614 2384 ERROR nova.compute.manager [instance: > 9c2d08f3-0680-4709-a64d-ae1729a11304] File > "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5614, > in _create_domain_and_network > 2018-11-28 21:18:11.614 2384 ERROR nova.compute.manager [instance: > 9c2d08f3-0680-4709-a64d-ae1729a11304] destroy_disks_on_failure) > 2018-11-28 21:18:11.614 2384 ERROR nova.compute.manager [instance: > 9c2d08f3-0680-4709-a64d-ae1729a11304] File > "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in > __exit__ > 2018-11-28 21:18:11.614 2384 ERROR nova.compute.manager [instance: > 9c2d08f3-0680-4709-a64d-ae1729a11304] self.force_reraise() > 2018-11-28 21:18:11.614 2384 ERROR nova.compute.manager [instance: > 9c2d08f3-0680-4709-a64d-ae1729a11304] File > "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in > force_reraise > 2018-11-28 21:18:11.614 2384 ERROR nova.compute.manager [instance: > 9c2d08f3-0680-4709-a64d-ae1729a11304] six.reraise(self.type_, > self.value, self.tb) > 2018-11-28 21:18:11.614 2384 ERROR nova.compute.manager [instance: > 9c2d08f3-0680-4709-a64d-ae1729a11304] File > "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5583, > in _create_domain_and_network > 2018-11-28 21:18:11.614 2384 ERROR nova.compute.manager [instance: > 9c2d08f3-0680-4709-a64d-ae1729a11304] > post_xml_callback=post_xml_callback) > 2018-11-28 21:18:11.614 2384 ERROR nova.compute.manager [instance: > 9c2d08f3-0680-4709-a64d-ae1729a11304] File > "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5502, > in _create_domain > 2018-11-28 21:18:11.614 2384 ERROR nova.compute.manager [instance: > 9c2d08f3-0680-4709-a64d-ae1729a11304] guest.launch(pause=pause) > 2018-11-28 21:18:11.614 2384 ERROR nova.compute.manager [instance: > 9c2d08f3-0680-4709-a64d-ae1729a11304] File > "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 144, in > launch > 2018-11-28 21:18:11.614 2384 ERROR nova.compute.manager [instance: > 9c2d08f3-0680-4709-a64d-ae1729a11304] self._encoded_xml, > errors='ignore') > 2018-11-28 21:18:11.614 2384 ERROR nova.compute.manager [instance: > 9c2d08f3-0680-4709-a64d-ae1729a11304] File > "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in > __exit__ > 2018-11-28 21:18:11.614 2384 ERROR nova.compute.manager [instance: > 9c2d08f3-0680-4709-a64d-ae1729a11304] self.force_reraise() > 2018-11-28 21:18:11.614 2384 ERROR nova.compute.manager [instance: > 9c2d08f3-0680-4709-a64d-ae1729a11304] File > "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in > force_reraise > 2018-11-28 21:18:11.614 2384 ERROR nova.compute.manager [instance: > 9c2d08f3-0680-4709-a64d-ae1729a11304] six.reraise(self.type_, > self.value, self.tb) > 2018-11-28 21:18:11.614 2384 ERROR nova.compute.manager [instance: > 9c2d08f3-0680-4709-a64d-ae1729a11304] File > "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 139, in > launch > 2018-11-28 21:18:11.614 2384 ERROR nova.compute.manager [instance: > 9c2d08f3-0680-4709-a64d-ae1729a11304] return > self._domain.createWithFlags(flags) > 2018-11-28 21:18:11.614 2384 ERROR nova.compute.manager [instance: > 9c2d08f3-0680-4709-a64d-ae1729a11304] File > "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit > 2018-11-28 21:18:11.614 2384 ERROR nova.compute.manager [instance: > 9c2d08f3-0680-4709-a64d-ae1729a11304] result = > proxy_call(self._autowrap, f, *args, **kwargs) > 2018-11-28 21:18:11.614 2384 ERROR nova.compute.manager [instance: > 9c2d08f3-0680-4709-a64d-ae1729a11304] File > "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in > proxy_call > 2018-11-28 21:18:11.614 2384 ERROR nova.compute.manager [instance: > 9c2d08f3-0680-4709-a64d-ae1729a11304] rv = execute(f, *args, **kwargs) > 2018-11-28 21:18:11.614 2384 ERROR nova.compute.manager [instance: > 9c2d08f3-0680-4709-a64d-ae1729a11304] File > "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute > 2018-11-28 21:18:11.614 2384 ERROR nova.compute.manager [instance: > 9c2d08f3-0680-4709-a64d-ae1729a11304] six.reraise(c, e, tb) > 2018-11-28 21:18:11.614 2384 ERROR nova.compute.manager [instance: > 9c2d08f3-0680-4709-a64d-ae1729a11304] File > "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker > 2018-11-28 21:18:11.614 2384 ERROR nova.compute.manager [instance: > 9c2d08f3-0680-4709-a64d-ae1729a11304] rv = meth(*args, **kwargs) > 2018-11-28 21:18:11.614 2384 ERROR nova.compute.manager [instance: > 9c2d08f3-0680-4709-a64d-ae1729a11304] File > "/usr/lib/python2.7/dist-packages/libvirt.py", line 1092, in createWithFlags > 2018-11-28 21:18:11.614 2384 ERROR nova.compute.manager [instance: > 9c2d08f3-0680-4709-a64d-ae1729a11304] if ret == -1: raise libvirtError > ('virDomainCreateWithFlags() failed', dom=self) > 2018-11-28 21:18:11.614 2384 ERROR nova.compute.manager [instance: > 9c2d08f3-0680-4709-a64d-ae1729a11304] libvirtError: internal error: > libxenlight failed to create new domain 'instance-00000008' Anyone help me please. _______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack(a)lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

2 1

[TripleO][Edge] Reduce base layer of containers for security and size of images (maintenance) sakes
by Bogdan Dobrelya 28 Nov '18

28 Nov '18

Changing the topic to follow the subject. [tl;dr] it's time to rearchitect container images to stop incluiding config-time only (puppet et al) bits, which are not needed runtime and pose security issues, like CVEs, to maintain daily. Background: 1) For the Distributed Compute Node edge case, there is potentially tens of thousands of a single-compute-node remote edge sites connected over WAN to a single control plane, which is having high latency, like a 100ms or so, and limited bandwith. Reducing the base layer size becomes a decent goal there. See the security background below. 2) For a generic security (Day 2, maintenance) case, when puppet/ruby/systemd/name-it gets a CVE fixed, the base layer has to be updated and all layers on top - to be rebuild, and all of those layers, to be re-fetched for cloud hosts and all containers to be restarted... And all of that because of some fixes that have nothing to OpenStack. By the remote edge sites as well, remember of "tens of thousands", high latency and limited bandwith?.. 3) TripleO CI updates (including puppet*) packages in containers, not in a common base layer of those. So each a CI job has to update puppet* and its dependencies - ruby/systemd as well. Reducing numbers of packages to update for each container makes sense for CI as well. Implementation related: WIP patches [0],[1] for early review, uses a config "pod" approach, does not require to maintain a two sets of config vs runtime images. Future work: a) cronie requires systemd, we'd want to fix that also off the base layer. b) rework to podman pods for docker-puppet.py instead of --volumes-from a side car container (can't be backported for Queens then, which is still nice to have a support for the Edge DCN case, at least downstream only perhaps). Some questions raised on IRC: Q: is having a service be able to configure itself really need to involve a separate pod? A: Highly likely yes, removing not-runtime things is a good idea and pods is an established PaaS paradigm already. That will require some changes in the architecture though (see the topic with WIP patches). Q: that's (fetching a config container) actually more data that about to download otherwise A: It's not, if thinking of Day 2, when have to re-fetch the base layer and top layers, when some unrelated to openstack CVEs got fixed there for ruby/puppet/systemd. Avoid the need to restart service containers because of those minor updates puched is also a nice thing. Q: the best solution here would be using packages on the host, generating the config files on the host. And then having an all-in-one container for all the services which lets them run in an isolated mannner. A: I think for Edge cases, that's a no go as we might want to consider tiny low footprint OS distros like former known Container Linux or Atomic. Also, an all-in-one container looks like an anti-pattern from the world of VMs. [0] https://review.openstack.org/#/q/topic:base-container-reduction [1] https://review.rdoproject.org/r/#/q/topic:base-container-reduction > Here is a related bug [1] and implementation [1] for that. PTAL folks! > > [0] https://bugs.launchpad.net/tripleo/+bug/1804822 > [1] https://review.openstack.org/#/q/topic:base-container-reduction > >> Let's also think of removing puppet-tripleo from the base container. >> It really brings the world-in (and yum updates in CI!) each job and each >> container! >> So if we did so, we should then either install puppet-tripleo and co on >> the host and bind-mount it for the docker-puppet deployment task steps >> (bad idea IMO), OR use the magical --volumes-from <a-side-car-container> >> option to mount volumes from some "puppet-config" sidecar container >> inside each of the containers being launched by docker-puppet tooling. > > On Wed, Oct 31, 2018 at 11:16 AM Harald Jensås <hjensas at redhat.com> > wrote: >> We add this to all images: >> >> https://github.com/openstack/tripleo-common/blob/d35af75b0d8c4683a677660646… >> >> /bin/sh -c yum -y install iproute iscsi-initiator-utils lvm2 python >> socat sudo which openstack-tripleo-common-container-base rsync cronie >> crudini openstack-selinux ansible python-shade puppet-tripleo python2- >> kubernetes && yum clean all && rm -rf /var/cache/yum 276 MB >> >> Is the additional 276 MB reasonable here? >> openstack-selinux <- This package run relabling, does that kind of >> touching the filesystem impact the size due to docker layers? >> >> Also: python2-kubernetes is a fairly large package (18007990) do we use >> that in every image? I don't see any tripleo related repos importing >> from that when searching on Hound? The original commit message[1] >> adding it states it is for future convenience. >> >> On my undercloud we have 101 images, if we are downloading every 18 MB >> per image thats almost 1.8 GB for a package we don't use? (I hope it's >> not like this? With docker layers, we only download that 276 MB >> transaction once? Or?) >> >> >> [1] https://review.openstack.org/527927 > > > > -- > Best regards, > Bogdan Dobrelya, > Irc #bogdando -- Best regards, Bogdan Dobrelya, Irc #bogdando

4 5

[Openstack] [OpenStack][cinder]Does cinder allocate disk to the volumes dynamically?
by Soheil Pourbafrani 28 Nov '18

28 Nov '18

Hi, I was wondering if the Cinder allocates disk to volumes statically or dynamically? _______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack(a)lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

2 2

[tripleo] Let's improve upstream docs
by Marios Andreou 28 Nov '18

28 Nov '18

Hi folks, as just mentioned in the tripleo weekly irc meeting [1] some of us are trying to make small weekly improvements to the tripleo docs [2]. We are using this bug [3] for tracking and this effort is a result of some feedback during the recent Berlin summit. The general idea is 1 per week (or more if you can and want) - improvement/removal of stale content/identifying missing sections, or anything else you might care to propose. Please join us if you can, just add "Related-Bug: #1804642" to your commit message thanks [1] https://wiki.openstack.org/wiki/Meetings/TripleO [2] https://docs.openstack.org/tripleo-docs/latest/ [3] https://bugs.launchpad.net/tripleo/+bug/1804642

2 3

[kolla] Meeting 28-nov-2018 canceled
by Eduardo Gonzalez 28 Nov '18

28 Nov '18

Dear Kolla Team, Due to many people wont be able to attend today meeting, the Kolla weekly meeting on Wednesday 28th is canceled. Best regards

1 0

[dev] [nova] API updates week 18-48
by Ghanshyam Mann 28 Nov '18

28 Nov '18

Hi All, Please find the Nova API highlights of this week. Weekly Office Hour: =============== No Office hour (Alex was not online) - I did bug triage during that time. Triaged 5 bugs and updated the API bug report etherpad. Planned Features : ============== Below are the API related features for Stein. Ref - https://etherpad.openstack.org/p/stein-nova-subteam-tracking (feel free to add API item there if you are working or found any). NOTE: sequence order are not the priority, they are listed as per their start date. 1. Handling a down cell - https://blueprints.launchpad.net/nova/+spec/handling-down-cell - https://review.openstack.org/#/q/topic:bp/handling-down-cell+(status:open+O… - Weekly Progress: tssurya has progress on this and discussin with matt and melanie . 2. Servers Ips non-unique network names : - https://blueprints.launchpad.net/nova/+spec/servers-ips-non-unique-network-… - Spec Merged - https://review.openstack.org/#/q/topic:bp/servers-ips-non-unique-network-na… - Weekly Progress: No progress. I will take care of this after api policy improvement spec update. 3. Volume multiattach enhancements: - https://blueprints.launchpad.net/nova/+spec/volume-multiattach-enhancements - https://review.openstack.org/#/q/topic:bp/volume-multiattach-enhancements+(… - Weekly Progress: No progress. 4. Add API ref guideline for body text (takashin) - https://review.openstack.org/#/c/605628/ - Weekly Progress: It has one +2 and need one more +2. 5. Detach and attach boot volumes: - https://blueprints.launchpad.net/nova/+spec/detach-boot-volume - https://review.openstack.org/#/q/topic:bp/detach-boot-volume+(status:open+O… - Weekly Progress: No progress. Specs: 6. Nova API policy updates https://blueprints.launchpad.net/nova/+spec/granular-api-policy Spec: https://review.openstack.org/#/c/547850/ - Weekly Progress: No progress in this week. I will update the spec next week. 7. Nova API cleanup https://blueprints.launchpad.net/nova/+spec/api-consistency-cleanup Spec: https://review.openstack.org/#/c/603969/ - Weekly Progress: Updated the spec with review comment and added 2 more cleanup listed in etherpad 8. Support deleting data volume when destroy instance(Brin Zhang) - https://review.openstack.org/#/c/580336/ - Weekly Progress: No Progress this week. Waiting for spec review. COMPLETED: 1. API Extensions merge work - https://blueprints.launchpad.net/nova/+spec/api-extensions-merge-stein - https://review.openstack.org/#/q/project:openstack/nova+branch:master+topic… - Weekly Progress: This was COMPLETED few weeks back itself. 2. Boot instance specific storage backend - https://blueprints.launchpad.net/nova/+spec/boot-instance-specific-storage-… - https://review.openstack.org/#/q/topic:bp/boot-instance-specific-storage-ba… - Weekly Progress: COMPLETED Bugs: ==== This week Bug Progress: https://etherpad.openstack.org/p/nova-api-weekly-bug-report Critical: 0->0 High importance: 1->1 By Status: New: 5->2 Confirmed/Triage: 32-> 32 In-progress: 36->35 Incomplete: 5->6 ===== Total: 78->75 NOTE- There might be some bug which are not tagged as 'api' or 'api-ref', those are not in above list. Tag such bugs so that we can keep our eyes. -gmann

1 0

[openstack-dev] [Cyborg] IRC meeting still happens at usual time this Wednesday
by Li Liu 27 Nov '18

27 Nov '18

Hi Team, I know there is a polling on the new IRC meeting time, but before we gather more inputs, let's keep using the old time slot for this week (10PM Beijing Time/9AM EST/6AM PST) Agenda for this week's meeting: 1. Status update on NOVA interaction spec 2. Status update on DB scheme spec 3. Status tracking on all the open patches -- Thank you Regards Li __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

1 0

Re: [Openstack] [PackStack][Cinder]Save Volumes in The Compute nodes
by Soheil Pourbafrani 27 Nov '18

27 Nov '18

Thanks! and Does it need to cinder module to be installed on compute nodes? On Sat, Nov 17, 2018 at 1:13 PM Bernd Bausch <berndbausch(a)gmail.com> wrote: > Yes. You need to enable Cinder’s InstanceLocalityFilter, see > > https://docs.openstack.org/cinder/latest/configuration/block-storage/schedu… > . > > Here a tip: > https://ask.openstack.org/en/question/92001/cinder-lvm-volume-local-to-inst… > > Bernd > > On Nov 17, 2018, at 8:38, Soheil Pourbafrani <soheil.ir08(a)gmail.com> > wrote: > > Hi, > I have 8 servers with just local HDD disk. I want to use one server as the > controller and network node and the other (7 servers) as the compute node. > > I was wondering if it's possible to install PackStack that every compute > node to store its volumes in it's HDD local disk? (I guess the Cinder > should be installed on every Compute node alongside other settings) > > Thanks > > _______________________________________________ > Mailing list: > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > Post to : openstack(a)lists.openstack.org > Unsubscribe : > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > > _______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack(a)lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

6 8

[dev] [tc][all] TC office hours is started now on #openstack-tc
by Ghanshyam Mann 27 Nov '18

27 Nov '18

Hi All, TC office hour is started on #openstack-tc channel. Feel free to reach to us for anything you want discuss/input/feedback/help from TC. - TC

1 0

Any way to reply to archived messages?
by Ben Nemec 27 Nov '18

27 Nov '18

Let's say hypothetically someone procrastinated too long on the mailing list switchover and missed some messages on openstack-discuss, but now wants to reply to them. Is there any way to do that without breaking the threading in people's clients? Perhaps some way to request a message from the archive to be sent to me (err, this hypothetical person ;-). -Ben

2 2