[Openstack] vm not starting

Manuel Sopena Ballesteros manuel.sb at garvan.org.au
Mon Oct 8 05:24:45 UTC 2018


Dear Openstack community,

I had to reboot my server and now I can't start one of the vms (danrod-server) which boots from volume.

vm details

[root at openstack ~(keystone_admin)]# openstack server show danrod-server
+--------------------------------------+----------------------------------------------------------+
| Field                                | Value                                                    |
+--------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                                   |
| OS-EXT-AZ:availability_zone          | nova                                                     |
| OS-EXT-SRV-ATTR:host                 | openstack.localdomain                                    |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | openstack.localdomain                                    |
| OS-EXT-SRV-ATTR:instance_name        | instance-00000069                                        |
| OS-EXT-STS:power_state               | NOSTATE                                                  |
| OS-EXT-STS:task_state                | None                                                     |
| OS-EXT-STS:vm_state                  | stopped                                                  |
| OS-SRV-USG:launched_at               | 2018-05-29T11:22:17.000000                               |
| OS-SRV-USG:terminated_at             | None                                                     |
| accessIPv4                           |                                                          |
| accessIPv6                           |                                                          |
| addresses                            | privatenetwork=192.168.1.105, 129.94.14.238              |
| config_drive                         |                                                          |
| created                              | 2018-05-29T11:22:09Z                                     |
| flavor                               | xlarge.cpu.xxxlarge.ram (6)                              |
| hostId                               | ecef276660cd714fe626073a18c11fe1c00bec91c15516178fb6ac28 |
| id                                   | 1e59f329-072e-48ae-abf1-266eba437508                     |
| image                                |                                                          |
| key_name                             | None                                                     |
| name                                 | danrod-server                                            |
| os-extended-volumes:volumes_attached | [{u'id': u'f1ac2e94-b0ed-4089-898f-5b6467fb51e3'}]       |
| project_id                           | d58cf22d960e4de49b71658aee642e94                         |
| properties                           |                                                          |
| security_groups                      | [{u'name': u'admin'}, {u'name': u'R-Studio Server'}]     |
| status                               | SHUTOFF                                                  |
| updated                              | 2018-10-08T02:52:41Z                                     |
| user_id                              | c412f34c353244eabecd4b6dc4d36392                         |
+--------------------------------------+----------------------------------------------------------+


List of volumes

[root at openstack ~(keystone_admin)]# openstack volume list --all
+--------------------------------------+--------------+--------+------+----------------------------------------+
| ID                                   | Display Name | Status | Size | Attached to                            |
+--------------------------------------+--------------+--------+------+----------------------------------------+
| f1ac2e94-b0ed-4089-898f-5b6467fb51e3 |              | in-use |  700 | Attached to danrod-server on /dev/vda  |
+--------------------------------------+--------------+--------+------+----------------------------------------+


nova-compute.log

2018-10-08 13:51:46.476 4015 INFO os_brick.initiator.connectors.iscsi [req-d9ed864b-9a4f-4182-b91b-9a2f1e38fb02 c412f34c353244eabecd4b6dc4d36392 d58cf22d960e4de49b71658aee642e94 - - -] Multipath discovery for iSCSI not enabled.
2018-10-08 13:51:46.476 4015 INFO os_brick.initiator.connectors.iscsi [req-d9ed864b-9a4f-4182-b91b-9a2f1e38fb02 c412f34c353244eabecd4b6dc4d36392 d58cf22d960e4de49b71658aee642e94 - - -] Trying to connect to iSCSI portal 129.94.14.254:3260
2018-10-08 13:51:46.504 4015 WARNING os_brick.initiator.connectors.iscsi [req-d9ed864b-9a4f-4182-b91b-9a2f1e38fb02 c412f34c353244eabecd4b6dc4d36392 d58cf22d960e4de49b71658aee642e94 - - -] ISCSI volume not yet found at: [u'/dev/disk/by-path/ip-129.94.14.254:3260-iscsi-iqn.2010-10.org.openstack:volume-f1ac2e94-b0ed-4089-898f-5b6467fb51e3-lun-0']. Will rescan & retry.  Try number: 0.
2018-10-08 13:51:47.510 4015 WARNING os_brick.initiator.connectors.iscsi [req-d9ed864b-9a4f-4182-b91b-9a2f1e38fb02 c412f34c353244eabecd4b6dc4d36392 d58cf22d960e4de49b71658aee642e94 - - -] ISCSI volume not yet found at: [u'/dev/disk/by-path/ip-129.94.14.254:3260-iscsi-iqn.2010-10.org.openstack:volume-f1ac2e94-b0ed-4089-898f-5b6467fb51e3-lun-0']. Will rescan & retry.  Try number: 1.
2018-10-08 13:51:51.519 4015 WARNING os_brick.initiator.connectors.iscsi [req-d9ed864b-9a4f-4182-b91b-9a2f1e38fb02 c412f34c353244eabecd4b6dc4d36392 d58cf22d960e4de49b71658aee642e94 - - -] ISCSI volume not yet found at: [u'/dev/disk/by-path/ip-129.94.14.254:3260-iscsi-iqn.2010-10.org.openstack:volume-f1ac2e94-b0ed-4089-898f-5b6467fb51e3-lun-0']. Will rescan & retry.  Try number: 2.
2018-10-08 13:52:00.527 4015 WARNING os_brick.initiator.connectors.iscsi [req-d9ed864b-9a4f-4182-b91b-9a2f1e38fb02 c412f34c353244eabecd4b6dc4d36392 d58cf22d960e4de49b71658aee642e94 - - -] ISCSI volume not yet found at: [u'/dev/disk/by-path/ip-129.94.14.254:3260-iscsi-iqn.2010-10.org.openstack:volume-f1ac2e94-b0ed-4089-898f-5b6467fb51e3-lun-0']. Will rescan & retry.  Try number: 3.
2018-10-08 13:52:16.535 4015 WARNING os_brick.initiator.connectors.iscsi [req-d9ed864b-9a4f-4182-b91b-9a2f1e38fb02 c412f34c353244eabecd4b6dc4d36392 d58cf22d960e4de49b71658aee642e94 - - -] ISCSI volume not yet found at: [u'/dev/disk/by-path/ip-129.94.14.254:3260-iscsi-iqn.2010-10.org.openstack:volume-f1ac2e94-b0ed-4089-898f-5b6467fb51e3-lun-0']. Will rescan & retry.  Try number: 4.
2018-10-08 13:52:23.357 4015 INFO nova.compute.resource_tracker [req-359039dc-29dc-4f06-805f-0ca943bb600f - - - - -] Auditing locally available compute resources for node openstack.localdomain
2018-10-08 13:52:23.713 4015 INFO nova.compute.resource_tracker [req-359039dc-29dc-4f06-805f-0ca943bb600f - - - - -] Total usable vcpus: 56, total allocated vcpus: 52
2018-10-08 13:52:23.713 4015 INFO nova.compute.resource_tracker [req-359039dc-29dc-4f06-805f-0ca943bb600f - - - - -] Final resource view: name=openstack.localdomain phys_ram=524173MB used_ram=454608MB phys_disk=9312GB used_disk=1540GB total_vcpus=56 used_vcpus=52 pci_stats=[]
2018-10-08 13:52:23.733 4015 INFO nova.compute.resource_tracker [req-359039dc-29dc-4f06-805f-0ca943bb600f - - - - -] Compute_service record updated for openstack.localdomain:openstack.localdomain
2018-10-08 13:52:41.665 4015 INFO nova.compute.manager [req-d9ed864b-9a4f-4182-b91b-9a2f1e38fb02 c412f34c353244eabecd4b6dc4d36392 d58cf22d960e4de49b71658aee642e94 - - -] [instance: 1e59f329-072e-48ae-abf1-266eba437508] Successfully reverted task state from powering-on on failure for instance.
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server [req-d9ed864b-9a4f-4182-b91b-9a2f1e38fb02 c412f34c353244eabecd4b6dc4d36392 d58cf22d960e4de49b71658aee642e94 - - -] Exception during message handling
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 75, in wrapped
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server     self.force_reraise()
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 188, in decorated_function
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server     LOG.warning(msg, e, instance=instance)
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server     self.force_reraise()
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 157, in decorated_function
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 613, in decorated_function
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_function
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server     self.force_reraise()
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2475, in start_instance
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server     self._power_on(context, instance)
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2445, in _power_on
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server     block_device_info)
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2434, in power_on
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server     self._hard_reboot(context, instance, network_info, block_device_info)
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2307, in _hard_reboot
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server     block_device_info=block_device_info)
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4658, in _get_guest_xml
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server     context)
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4483, in _get_guest_config
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server     flavor, guest.os_type)
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3523, in _get_guest_storage_config
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server     self._connect_volume(connection_info, info)
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1099, in _connect_volume
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server     vol_driver.connect_volume(connection_info, disk_info)
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume/iscsi.py", line 64, in connect_volume
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server     device_info = self.connector.connect_volume(connection_info['data'])
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/os_brick/utils.py", line 137, in trace_logging_wrapper
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py", line 404, in connect_volume
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server     raise exception.VolumeDeviceNotFound(device=host_devices)
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server VolumeDeviceNotFound: Volume device not found at [u'/dev/disk/by-path/ip-129.94.14.254:3260-iscsi-iqn.2010-10.org.openstack:volume-f1ac2e94-b0ed-4089-898f-5b6467fb51e3-lun-0'].
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server



Cinder logs

2018-10-08 08:52:58.368 7069 ERROR oslo.messaging._drivers.impl_rabbit [-] [3dce0b7f-7eea-45e9-9fa5-220e783c0401] AMQP server 129.94.14.254:5672 closed the connection. Check login credentials: Socket closed
2018-10-08 08:52:59.395 7069 ERROR oslo.messaging._drivers.impl_rabbit [-] [3dce0b7f-7eea-45e9-9fa5-220e783c0401] AMQP server on 129.94.14.254:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: 60428
2018-10-08 08:53:01.404 7069 ERROR oslo.messaging._drivers.impl_rabbit [-] [3dce0b7f-7eea-45e9-9fa5-220e783c0401] AMQP server on 129.94.14.254:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: 60428
2018-10-08 08:53:04.131 7069 ERROR cinder.service [req-f6138d6c-a09f-4ed4-881f-ed095d172461 - - - - -] model server went away
2018-10-08 08:53:04.131 7069 ERROR cinder.service Traceback (most recent call last):
2018-10-08 08:53:04.131 7069 ERROR cinder.service   File "/usr/lib/python2.7/site-packages/cinder/service.py", line 445, in report_state
2018-10-08 08:53:04.131 7069 ERROR cinder.service     service_ref = objects.Service.get_by_id(ctxt, self.service_id)
2018-10-08 08:53:04.131 7069 ERROR cinder.service   File "/usr/lib/python2.7/site-packages/cinder/objects/base.py", line 287, in get_by_id
2018-10-08 08:53:04.131 7069 ERROR cinder.service     orm_obj = db.get_by_id(context, cls.model, id, *args, **kwargs)
2018-10-08 08:53:04.131 7069 ERROR cinder.service   File "/usr/lib/python2.7/site-packages/cinder/db/api.py", line 1624, in get_by_id
2018-10-08 08:53:04.131 7069 ERROR cinder.service     return IMPL.get_by_id(context, model, id, *args, **kwargs)
2018-10-08 08:53:04.131 7069 ERROR cinder.service   File "/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 196, in wrapper
2018-10-08 08:53:04.131 7069 ERROR cinder.service     return f(*args, **kwargs)
2018-10-08 08:53:04.131 7069 ERROR cinder.service   File "/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 6204, in get_by_id
2018-10-08 08:53:04.131 7069 ERROR cinder.service     return _GET_METHODS[model](context, id, *args, **kwargs)
2018-10-08 08:53:04.131 7069 ERROR cinder.service   File "/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 178, in wrapper
2018-10-08 08:53:04.131 7069 ERROR cinder.service     return f(*args, **kwargs)
2018-10-08 08:53:04.131 7069 ERROR cinder.service   File "/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 483, in service_get
2018-10-08 08:53:04.131 7069 ERROR cinder.service     service = None if not query else query.first()
...

However the connection to the rabbitmq server is fine?

[root at openstack ~(keystone_admin)]# sudo ss -nolpt | grep 5672
LISTEN     0      128          *:25672                    *:*                   users:(("beam.smp",pid=2465,fd=43))
LISTEN     0      128         :::5672                    :::*                   users:(("beam.smp",pid=2465,fd=52))
[root at openstack ~(keystone_admin)]# telnet 129.94.14.254 5672
Trying 129.94.14.254...
Connected to 129.94.14.254.
Escape character is '^]'.

Any advice in regards how to fix this issue and be able to bring the vm back online would be highly appreciated?

Thank you very much

Manuel Sopena Ballesteros | Big data Engineer
Garvan Institute of Medical Research
The Kinghorn Cancer Centre, 370 Victoria Street, Darlinghurst, NSW 2010
T: + 61 (0)2 9355 5760 | F: +61 (0)2 9295 8507 | E: manuel.sb at garvan.org.au<mailto:manuel.sb at garvan.org.au>

NOTICE
Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20181008/28dc2d1c/attachment.html>


More information about the Openstack mailing list