Open Stack

Hi,

I have created a development setup with Devstack on 2 machines, where
one is a controller + compute node (IP: 172.23.29.96) and other is a storage
node(IP: 172.23.29.118). I am running all the services on the
controller, except the c-vol service which runs on the storage node.

Whenever I try to create a new instance, the volume gets created but
the instance is stuck with "Spawning" status, and in some time gets
errored out. This volume is then not attached to the VM. These volumes
are LVM based, SCSI backed volumes.

During this time, I can see the following 2 errors of interest in
stack traces/logs. It would be really helpful if anyone here can take
a look and point me in the right direction.

I've also attached the stack traces in case the formatting gets messed up. 

====== START OF STACK TRACE 1 =========

DEBUG oslo.privsep.daemon [-] privsep: Exception during request[140390878014992]: Unexpected error while running command.
Command: iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9ae59d83-ec09-4fd5-aa2c-5049a29bed5e -p 172.23.29.118:3260 --login
Exit code: 8
Stdout: u'Logging in to [iface: default, target: iqn.2010-10.org.openstack:volume-9ae59d83-ec09-4fd5-aa2c-5049a29bed5e, portal: 172.23.29.118,3260] (multiple)\n'
Stderr: u'iscsiadm: Could not login to [iface: default, target: iqn.2010-10.org.openstack:volume-9ae59d83-ec09-4fd5-aa2c-5049a29bed5e, portal: 172.23.29.118,3260].\nis\
csiadm: initiator reported error (8 - connection timed out)\niscsiadm: Could not log into all portals\n' {{(pid=14097) loop /usr/lib/python2.7/site-packages/oslo_privs\
ep/daemon.py:449}}
ERROR oslo.privsep.daemon Traceback (most recent call last):
ERROR oslo.privsep.daemon   File "/usr/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 445, in loop
ERROR oslo.privsep.daemon     reply = self._process_cmd(*msg)
ERROR oslo.privsep.daemon   File "/usr/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 428, in _process_cmd
ERROR oslo.privsep.daemon     ret = func(*f_args, **f_kwargs)
ERROR oslo.privsep.daemon   File "/usr/lib/python2.7/site-packages/oslo_privsep/priv_context.py", line 209, in _wrap
ERROR oslo.privsep.daemon     return func(*args, **kwargs)
ERROR oslo.privsep.daemon   File "/usr/lib/python2.7/site-packages/os_brick/privileged/rootwrap.py", line 194, in execute_root
ERROR oslo.privsep.daemon     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
ERROR oslo.privsep.daemon   File "/usr/lib/python2.7/site-packages/os_brick/privileged/rootwrap.py", line 143, in custom_execute
ERROR oslo.privsep.daemon     on_completion=on_completion, *cmd, **kwargs)
ERROR oslo.privsep.daemon   File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 424, in execute
ERROR oslo.privsep.daemon     cmd=sanitized_cmd)
ERROR oslo.privsep.daemon ProcessExecutionError: Unexpected error while running command.
ERROR oslo.privsep.daemon Command: iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9ae59d83-ec09-4fd5-aa2c-5049a29bed5e -p 172.23.29.118:3260 --login
ERROR oslo.privsep.daemon Exit code: 8
ERROR oslo.privsep.daemon Stdout: u'Logging in to [iface: default, target: iqn.2010-10.org.openstack:volume-9ae59d83-ec09-4fd5-aa2c-5049a29bed5e, portal: 172.23.29.118\
,3260] (multiple)\n'
ERROR oslo.privsep.daemon Stderr: u'iscsiadm: Could not login to [iface: default, target: iqn.2010-10.org.openstack:volume-9ae59d83-ec09-4fd5-aa2c-5049a29bed5e, portal\
: 172.23.29.118,3260].\niscsiadm: initiator reported error (8 - connection timed out)\niscsiadm: Could not log into all portals\n'
ERROR oslo.privsep.daemon

====== END OF STACK TRACE 1   ===========


====== START OF STACK TRACE 2 ===========


[instance: d2a44cc2-c367-4d6e-b572-0d174e44d817] Instance failed to spawn: VolumeDeviceNotFound: Volume device not found at .
Error:  Traceback (most recent call last):
Error:    File "/opt/stack/nova/nova/compute/manager.py", line 2357, in _build_resources
Error:      yield resources
Error:    File "/opt/stack/nova/nova/compute/manager.py", line 2121, in _build_and_run_instance
Error:      block_device_info=block_device_info)
Error:    File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3075, in spawn
Error:      mdevs=mdevs)
Error:    File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5430, in _get_guest_xml
Error:      context, mdevs)
Error:    File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5216, in _get_guest_config
Error:      flavor, guest.os_type)
Error:    File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4030, in _get_guest_storage_config
Error:      self._connect_volume(context, connection_info, instance)
Error:    File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1222, in _connect_volume
Error:      vol_driver.connect_volume(connection_info, instance)
Error:    File "/opt/stack/nova/nova/virt/libvirt/volume/iscsi.py", line 64, in connect_volume
Error:      device_info = self.connector.connect_volume(connection_info['data'])
Error:    File "/usr/lib/python2.7/site-packages/os_brick/utils.py", line 150, in trace_logging_wrapper
Error:      result = f(*args, **kwargs)
Error:    File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
Error:      return f(*args, **kwargs)
Error:    File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py", line 516, in connect_volume
Error:      self._cleanup_connection(connection_properties, force=True)
Error:    File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
Error:      self.force_reraise()
Error:    File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
Error:      six.reraise(self.type_, self.value, self.tb)
Error:    File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py", line 510, in connect_volume
Error:      return self._connect_single_volume(connection_properties)
Error:    File "/usr/lib/python2.7/site-packages/os_brick/utils.py", line 61, in _wrapper
Error:      return r.call(f, *args, **kwargs)
Error:    File "/usr/lib/python2.7/site-packages/retrying.py", line 212, in call
Error:      raise attempt.get()
Error:    File "/usr/lib/python2.7/site-packages/retrying.py", line 247, in get
Error:      six.reraise(self.value[0], self.value[1], self.value[2])
Error:    File "/usr/lib/python2.7/site-packages/retrying.py", line 200, in call
Error:      attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
Error:    File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py", line 585, in _connect_single_volume
Error:      raise exception.VolumeDeviceNotFound(device='')
Error:  VolumeDeviceNotFound: Volume device not found at .

====== END OF STACK TRACE 2 ===========



-- 
  Tushar Tyagi
  ttml at fastmail.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: stack_trace_1.log
Type: application/octet-stream
Size: 2890 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20181114/32b9db35/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: stack_trace_2.log
Type: application/octet-stream
Size: 3168 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20181114/32b9db35/attachment-0001.obj>