Hello Openstackers, I have used Cinder stein release (v14.0.2) source base and deployed in All-In-One setup on ubuntu16.04 using Kolla-Ansible. The cinder image was built using python 2.7 somewhere which i am trying to deploy in my new setup having python3.5. All services are deployed and running fine. However when I try to attach one volume to an instance, it shows below error in /var/log/kolla/cinder/cinder-volume.log. 2020-05-26 08:10:07.026 35 INFO oslo.privsep.daemon [req-db287360-876c-425e-b8df-a4f3b0db3163 bc13e1a279854a6fa1d553c7b7a0214c 9bdfc4ffefaa4edb823ff9ba5383ab42 - default default] Running privsep helper: ['sudo', 'cinder-rootwrap', '/etc/cinder/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/cinder/cinder.conf', '--privsep_context', 'cinder.privsep.sys_admin_pctxt', '--privsep_sock_path', '/tmp/tmpan8c8r/privsep.sock'] 2020-05-26 08:10:10.952 35 INFO oslo.privsep.daemon [req-db287360-876c-425e-b8df-a4f3b0db3163 bc13e1a279854a6fa1d553c7b7a0214c 9bdfc4ffefaa4edb823ff9ba5383ab42 - default default] Spawned new privsep daemon via rootwrap 2020-05-26 08:10:10.647 1071 INFO oslo.privsep.daemon [-] privsep daemon starting 2020-05-26 08:10:10.721 1071 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0 2020-05-26 08:10:10.725 1071 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_CHOWN|CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_FOWNER|CAP_NET_ADMIN|CAP_SYS_ADMIN/CAP_CHOWN|CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_FOWNER|CAP_NET_ADMIN|CAP_SYS_ADMIN/none 2020-05-26 08:10:10.726 1071 INFO oslo.privsep.daemon [-] privsep daemon running as pid 1071 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server [req-db287360-876c-425e-b8df-a4f3b0db3163 bc13e1a279854a6fa1d553c7b7a0214c 9bdfc4ffefaa4edb823ff9ba5383ab42 - default default] Exception during message handling: ProcessExecutionError: Unexpected error while running command. Command: tgtadm --lld iscsi --op show --mode target Exit code: 107 Stdout: u'' Stderr: u'tgtadm: failed to send request hdr to tgt daemon, Transport endpoint is not connected\n' 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/manager.py", line 4517, in attachment_update 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server connector) 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/manager.py", line 4445, in _connection_create 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server volume, connector) 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/drivers/lvm.py", line 830, in create_export 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server volume_path) 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/targets/iscsi.py", line 210, in create_export 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server **portals_config) 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/utils.py", line 818, in _wrapper 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server return r.call(f, *args, **kwargs) 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/retrying.py", line 206, in call 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server return attempt.get(self._wrap_exception) 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/retrying.py", line 247, in get 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server six.reraise(self.value[0], self.value[1], self.value[2]) 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/retrying.py", line 200, in call 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server attempt = Attempt(fn(*args, **kwargs), attempt_number, False) 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/targets/tgt.py", line 126, in create_iscsi_target 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server (out, err) = cinder.privsep.targets.tgt.tgtadm_show() 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_privsep/priv_context.py", line 244, in _wrap 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server return self.channel.remote_call(name, args, kwargs) 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 203, in remote_call 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server raise exc_type(*result[2]) 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server ProcessExecutionError: Unexpected error while running command. 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server Command: tgtadm --lld iscsi --op show --mode target 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server Exit code: 107 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server Stdout: u'' 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server Stderr: u'tgtadm: failed to send request hdr to tgt daemon, Transport endpoint is not connected\n' 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server When I check inside cinder-volume container, the tgtd is not running. SO I started tgtd from root user. Now the volume attachment happens for few seconds and then it is detached. 2020-05-26 08:14:39.221 35 INFO cinder.volume.manager [req-5da81388-33ce-4994-9114-52442a08ac2a bc13e1a279854a6fa1d553c7b7a0214c 9bdfc4ffefaa4edb823ff9ba5383ab42 - default default] attachment_update completed successfully. 2020-05-26 08:14:52.622 35 INFO cinder.volume.manager [req-cbaa1b6c-4fa8-4ece-b08b-8ca09f78ecaf bc13e1a279854a6fa1d553c7b7a0214c 9bdfc4ffefaa4edb823ff9ba5383ab42 - default default] Terminate volume connection completed successfully. 2020-05-26 08:14:52.840 35 INFO cinder.volume.targets.tgt [req-cbaa1b6c-4fa8-4ece-b08b-8ca09f78ecaf bc13e1a279854a6fa1d553c7b7a0214c 9bdfc4ffefaa4edb823ff9ba5383ab42 - default default] Removing iscsi_target for Volume ID: d713ce55-1fa6-4d96-a69e-69368494f8f9 2020-05-26 08:14:53.739 35 WARNING py.warnings [req-cbaa1b6c-4fa8-4ece-b08b-8ca09f78ecaf bc13e1a279854a6fa1d553c7b7a0214c 9bdfc4ffefaa4edb823ff9ba5383ab42 - default default] /var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/evaluator.py:99: SAWarning: Evaluating non-mapped column expression 'updated_at' onto ORM instances; this is a deprecated use case. Please make use of the actual mapped columns in ORM-evaluated UPDATE / DELETE expressions. "UPDATE / DELETE expressions." % clause I also used the cinder 14.0.4 source tag to build the new image, but the same error persists. Can you please point me what is the issue in my test case or setup. Appreciate your help. best regards, Sanjay