On 10/01, Jay S. Bryant wrote:

Grant,

So, the copy is failing because it can't find the volume to copy the image into.

I would check the host and container for any iSCSI errors as well as the backend.  It appears that something is going wrong when attempting to temporarily attach the volume to write the image into it.

Jay

Hi, I've also seen this error when the initiator name in /etc/iscsi/initiatorname.iscsi inside the container does not match the one in use by the iscsid initiator daemon. This can happen because the initiator name was changed after the daemon started or because it is not shared between the container and the host. I've also seen this happen (thought this is not the case) on VM migrations when the driver has a bug and doesn't return the right connection information (returns the first one). I would recommend setting the log level to debug to see additional info from OS-Brick. I've debugged these type of issues many times, and if it's not a production env I usually go with: - Setting a breakpoint in the OS-Brick code: I stop at the right place and check the state of the system and how the volume has been exported and mapped in the backend. - Installing cinderlib on a virtualenv (with --system-site-packages) in the cinder node, then using cinderlib to create a volume and debug an attach operation same as in previous step, like this: * Prepare the env: $ virtualenv --system-site-packages venv $ source venv/bin/activate (venv) $ pip install cinderlib * Run python interpreter (venv) $ python * Initialize cinderlib to store volumes in ./cl3.sqlite >>> import cinderlib as cl >>> db_connection = 'sqlite:///cl3.sqlite' >>> persistence_config = {'storage': 'db', 'connection': db_connection} >>> cl.setup(persistence_config=persistence_config, disable_logs=False, debug=True) * Setup the backend. You'll have to use your own configuration here: >>> sf = cl.Backend( ... volume_backend_name='solidfire', ... volume_driver='cinder.volume.drivers.solidfire.SolidFireDriver', ... san_ip='192.168.1.4', ... san_login='admin', ... san_password='admin_password', ... sf_allow_template_caching=False) * Create a 1GB empty volume: >>> vol = sf.create_volume(1) * Debug the attachment: >>> import pdb >>> pdb.run('att = vol.attach()') - If it's a container I usually execute a bash terminal interactively and pip install cinderlib and do the debugging like in the step above. Cheers, Gorka.

On 1/10/2019 7:16 AM, Grant Morley wrote:

...

Hi all,

We are in the process of trying to add a SolidFire storage solution to our existing OpenStack setup and seem to have hit a snag with cinder / iscsi.

We are trying to create a bootable volume to allow us to launch an instance from it, but we are getting some errors in our cinder-volumes containers that seem to suggest they can't connect to iscsi although the volume seems to create fine on the SolidFire node.

The command we are running is:

openstack volume create --image $image-id --size 20 --bootable --type solidfire sf-volume-v12

The volume seems to create on SolidFire but I then see these errors in the "cinder-volume.log"

https://pastebin.com/LyjLUhfk

The volume containers can talk to the iscsi VIP on the SolidFire so I am a bit stuck and wondered if anyone had come across any issues before?

Kind Regards,

-- Grant Morley Cloud Lead Absolute DevOps Ltd Units H, J & K, Gateway 1000, Whittle Way, Stevenage, Herts, SG1 2FP www.absolutedevops.io <http://www.absolutedevops.io/> grant@absolutedevops.io <mailto:grant@absolutedevops.io> 0845 874 0580