Hello Everyone,
I have a really strange problem with iSCSI, Cinder, Kolla setup.
1. What I want to do:
I did setup with kolla 1 Controller, 4 Compute nodes and 2 storage nodes
(iSCSI, Cinder, LVM).
Kolla Deploying works fine, all Machines can find all others.
2. Problem: happens when I try to create an instance
* Im creating an new instance what fails because the cinder volume
couldn't be "mounted" on the compute node
* What I checked then:
o storage2 (the 2. cinder iscsi storage node) is available (by
ping ) from the compute node
o Port 3260 on storage2 is not reachable from compute node
o so checking storage2
+ the volume was created on LVM (checked with lvs)
+ Port 3260 is not available local too
+ it looks like that cinder creates an iSCSI target and
deletes it after some seconds ... no reason shown (I
activated debug mode in Kolla in the hope to see more but no
...)
Did anyone have some similar behavior or can give me a hint where else
to look ?
Thanks a lot for your help !
best
Alex
Logfile of the storage2 node /var/log/kolla/cinder/cinder-volume.log:
the following lines came 5-6 times and then nothing more.
2025-03-03 08:17:02.625 42 DEBUG cinder.volume.drivers.lvm [None
req-c3be57e8-eeac-4b14-99c7-f3f10474a3cf - - - - - -] Updating volume
stats _update_volume_stats
/var/lib/kolla/venv/lib/python3.12/site-packages/cinder/volume/drivers/lvm.py:227
2025-03-03 08:17:02.626 42 DEBUG oslo_concurrency.processutils [None
req-c3be57e8-eeac-4b14-99c7-f3f10474a3cf - - - - - -] Running cmd
(subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf env
LC_ALL=C vgs --noheadings --unit=g -o name,size,free,lv_count,uuid
--separator :
--nosuffix cinder-volumes execute
/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349
2025-03-03 08:17:03.057 42 DEBUG oslo_concurrency.processutils [None
req-c3be57e8-eeac-4b14-99c7-f3f10474a3cf - - - - - -] CMD "sudo
cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C vgs --noheadings
--unit=g -o name,size,free,lv_count,uuid --separator : --nosuffix cinder-vo
lumes" returned: 0 in 0.432s execute
/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372
2025-03-03 08:17:03.059 42 DEBUG oslo_concurrency.processutils [None
req-c3be57e8-eeac-4b14-99c7-f3f10474a3cf - - - - - -] Running cmd
(subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf env
LC_ALL=C lvs --noheadings --unit=g -o vg_name,name,size --nosuffix
--readonly ci
nder-volumes execute
/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349
2025-03-03 08:17:03.483 42 DEBUG oslo_concurrency.processutils [None
req-c3be57e8-eeac-4b14-99c7-f3f10474a3cf - - - - - -] CMD "sudo
cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C lvs --noheadings
--unit=g -o vg_name,name,size --nosuffix --readonly cinder-volumes" returne
d: 0 in 0.425s execute
/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372
2025-03-03 08:17:03.485 42 DEBUG oslo_concurrency.processutils [None
req-c3be57e8-eeac-4b14-99c7-f3f10474a3cf - - - - - -] Running cmd
(subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf env
LC_ALL=C lvs --noheadings --unit=g -o size,data_percent --separator :
--nosuffix
/dev/cinder-volumes/cinder-volumes-pool execute
/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349
2025-03-03 08:17:03.899 42 DEBUG oslo_concurrency.processutils [None
req-c3be57e8-eeac-4b14-99c7-f3f10474a3cf - - - - - -] CMD "sudo
cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C lvs --noheadings
--unit=g -o size,data_percent --separator : --nosuffix /dev/cinder-volumes/
cinder-volumes-pool" returned: 0 in 0.414s execute
/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372
2025-03-03 08:17:03.900 42 DEBUG oslo_concurrency.processutils [None
req-c3be57e8-eeac-4b14-99c7-f3f10474a3cf - - - - - -] Running cmd
(subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf env
LC_ALL=C lvs --noheadings --unit=g -o vg_name,name,size --nosuffix
--readonly ci
nder-volumes execute
/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349
2025-03-03 08:17:04.319 42 DEBUG oslo_concurrency.processutils [None
req-c3be57e8-eeac-4b14-99c7-f3f10474a3cf - - - - - -] CMD "sudo
cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C lvs --noheadings
--unit=g -o vg_name,name,size --nosuffix --readonly cinder-volumes" returne
d: 0 in 0.418s execute
/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372
2025-03-03 08:17:04.320 42 DEBUG cinder.manager [None
req-c3be57e8-eeac-4b14-99c7-f3f10474a3cf - - - - - -] Notifying
Schedulers of capabilities ... _publish_service_capabilities
/var/lib/kolla/venv/lib/python3.12/site-packages/cinder/manager.py:202
2025-03-03 08:18:02.626 42 DEBUG oslo_service.periodic_task [None
req-c3be57e8-eeac-4b14-99c7-f3f10474a3cf - - - - - -] Running periodic
task VolumeManager.publish_service_capabilities run_periodic_tasks
/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_service/periodic_task.
py:210