[Cinder] 인스턴스 생성 시 volume 연결 문제
안녕하세요. Openstack을 학교 서버에 수동으로 설치 중인데, Cinder 관련 해서 문제가 발생해서 도움 요청 합니다. Queens 버전 최소 설치 가이드 ( https://docs.openstack.org/cinder/queens/install/index-ubuntu.html)를 참고 해서 설치 중인데, 인스턴스를 생성 할 때, 해당 인스턴스가 사용할 volume을 storage server에서 못받아 오는 것 처럼 보입니다. iSCSI를 위한 tgt 서비스를 보면, 뭔가 커널 모듈이 추가적으로 로드 되어야 하는거 같은데, 이게 스토리지 서버에 있는 RAID controller 와 관련이 있을까요? 아래는 tgt 서비스의 상태입니다. ● tgt.service - (i)SCSI target daemon
Loaded: loaded (/lib/systemd/system/tgt.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2018-12-19 22:05:55 KST; 26min ago Docs: man:tgtd(8) Process: 2249 ExecStartPost=/usr/sbin/tgtadm --op update --mode sys --name State -v ready (code=exited, status=0/SUCCESS) Process: 2230 ExecStartPost=/usr/sbin/tgt-admin -e -c /etc/tgt/targets.conf (code=exited, status=0/SUCCESS) Process: 2206 ExecStartPost=/usr/sbin/tgtadm --op update --mode sys --name State -v offline (code=exited, status=0/SUCCESS) Main PID: 2170 (tgtd) Status: "Starting event loop..." Tasks: 1 Memory: 2.7M CPU: 87ms CGroup: /system.slice/tgt.service └─2170 /usr/sbin/tgtd -f Dec 19 22:05:55 node0 systemd[1]: Starting (i)SCSI target daemon... Dec 19 22:05:55 node0 tgtd[2170]: tgtd: iser_ib_init(3434) Failed to initialize RDMA; load kernel modules? Dec 19 22:05:55 node0 tgtd[2170]: tgtd: work_timer_start(150) use signal based scheduler Dec 19 22:05:55 node0 tgtd[2170]: tgtd: bs_init(387) use signalfd notification Dec 19 22:05:55 node0 systemd[1]: Started (i)SCSI target daemon.
그리고 아래는 cinder 서비스의 상태 입니다. ● cinder-volume.service - OpenStack Cinder Volume
Loaded: loaded (/lib/systemd/system/cinder-volume.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2018-12-19 22:05:55 KST; 45min ago Main PID: 2110 (cinder-volume) Tasks: 2 Memory: 210.2M CPU: 1min 23.649s CGroup: /system.slice/cinder-volume.service ├─2110 /usr/bin/python2 /usr/bin/cinder-volume --config-file=/etc/cinder/cinder.conf --log-file=/var/log/cinder/cinder-volume.log └─2657 /usr/bin/python2 /usr/bin/cinder-volume --config-file=/etc/cinder/cinder.conf --log-file=/var/log/cinder/cinder-volume.log Dec 19 22:05:55 node0 systemd[1]: Started OpenStack Cinder Volume. Dec 19 22:06:00 node0 sudo[2658]: pam_unix(sudo:auth): auth could not identify password for [cinder]
Cinder-volume의 로그는 계속해서 cinder-volume 서비스가 통신을 못하고 있는 것 처럼 보이는데, 아래와 같이 volume에 접근 하려고 할 때 (e.g. instance 생성), 에러 메시지가 출력 되고, 그 이후에는 계속해서 동일한 메시지가 매초 반복되고 있습니다. 2018-12-19 22:04:00.747 35996 INFO oslo_service.service
[req-bcc39e60-531e-4484-ab4b-bab085d112c4 - - - - -] Caught SIGTERM, stopping children 2018-12-19 22:04:00.784 35996 INFO oslo_service.service [req-bcc39e60-531e-4484-ab4b-bab085d112c4 - - - - -] Waiting on 1 children to exit 2018-12-19 22:04:00.809 35996 INFO oslo_service.service [req-bcc39e60-531e-4484-ab4b-bab085d112c4 - - - - -] Child 36039 killed by signal 15 2018-12-19 22:05:58.378 2110 INFO root [-] Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt 2018-12-19 22:05:58.403 2110 INFO root [-] Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt 2018-12-19 22:05:59.820 2110 INFO cinder.rpc [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Automatically selected cinder-scheduler objects version 1.35 as minimum service version. 2018-12-19 22:05:59.826 2110 INFO cinder.rpc [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Automatically selected cinder-scheduler RPC version 3.10 as minimum service version. 2018-12-19 22:05:59.875 2110 INFO cinder.volume.manager [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Determined volume DB was empty at startup. 2018-12-19 22:05:59.908 2110 WARNING oslo_config.cfg [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Option "iscsi_helper" from group "lvm" is deprecated. Use option "target_helper" from group "lvm". 2018-12-19 22:05:59.914 2110 WARNING oslo_config.cfg [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Option "iscsi_protocol" from group "lvm" is deprecated. Use option "target_protocol" from group "lvm". 2018-12-19 22:05:59.916 2110 INFO cinder.volume.manager [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Image-volume cache disabled for host node0@lvm. 2018-12-19 22:05:59.932 2110 INFO oslo_service.service [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Starting 1 workers 2018-12-19 22:05:59.940 2657 INFO cinder.service [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Starting cinder-volume node (version 13.0.0) 2018-12-19 22:05:59.942 2110 WARNING oslo_config.cfg [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Option "iscsi_helper" from group "DEFAULT" is deprecated. Use option "target_helper" from group "DEFAULT". 2018-12-19 22:05:59.952 2110 WARNING oslo_config.cfg [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Option "auth_uri" from group "keystone_authtoken" is deprecated for removal (The auth_uri option is deprecated in favor of www_authenticate_uri and will be removed in the S release.). Its value may be silently ignored in the future. 2018-12-19 22:05:59.953 2110 WARNING oslo_config.cfg [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Option "auth_uri" from group "keystone_authtoken" is deprecated. Use option "www_authenticate_uri" from group "keystone_authtoken". 2018-12-19 22:05:59.961 2657 INFO cinder.volume.manager [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Starting volume driver LVMVolumeDriver (3.0.0) 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Failed to initialize driver.: ProcessExecutionError: Unexpected error while running command. Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C vgs --version Exit code: 1 Stdout: u'' Stderr: u'sudo: no tty present and no askpass program specified\n' 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager Traceback (most recent call last): 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/manager.py", line 433, in init_host 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager self.driver.check_for_setup_error() 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/lvm.py", line 286, in check_for_setup_error 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager if volutils.supports_thin_provisioning(): 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/utils.py", line 628, in supports_thin_provisioning 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager utils.get_root_helper()) 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/brick/local_dev/lvm.py", line 225, in supports_thin_provisioning 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager return LVM.get_lvm_version(root_helper) >= (2, 2, 95) 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/brick/local_dev/lvm.py", line 202, in get_lvm_version 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager run_as_root=True) 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 424, in execute 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager cmd=sanitized_cmd) 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager ProcessExecutionError: Unexpected error while running command. 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C vgs --version 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager Exit code: 1 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager Stdout: u'' 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager Stderr: u'sudo: no tty present and no askpass program specified\n' 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager 2018-12-19 22:06:00.075 2657 INFO cinder.volume.manager [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Initializing RPC dependent components of volume driver LVMVolumeDriver (3.0.0) 2018-12-19 22:06:00.076 2657 ERROR cinder.utils [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Volume driver LVMVolumeDriver not initialized 2018-12-19 22:06:00.076 2657 ERROR cinder.volume.manager [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Cannot complete RPC initialization because driver isn't initialized properly.: DriverNotInitialized: Volume driver not ready. 2018-12-19 22:06:10.077 2657 ERROR cinder.service [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Manager for service cinder-volume node0@lvm is reporting problems, not sending heartbeat. Service will appear "down". 2018-12-19 22:06:20.077 2657 ERROR cinder.service [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Manager for service cinder-volume node0@lvm is reporting problems, not sending heartbeat. Service will appear "down". 2018-12-19 22:06:30.078 2657 ERROR cinder.service [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Manager for service cinder-volume node0@lvm is reporting problems, not sending heartbeat. Service will appear "down". 2018-12-19 22:06:40.078 2657 ERROR cinder.service [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Manager for service cinder-volume node0@lvm is reporting problems, not sending heartbeat. Service will appear "down". 2018-12-19 22:06:50.079 2657 ERROR cinder.service [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Manager for service cinder-volume node0@lvm is reporting problems, not sending heartbeat. Service will appear "down". 2018-12-19 22:07:00.079 2657 ERROR cinder.service [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Manager for service cinder-volume node0@lvm is reporting problems, not sending heartbeat. Service will appear "down". 2018-12-19 22:07:10.078 2657 WARNING cinder.volume.manager [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Update driver status failed: (config name lvm) is uninitialized.
위의 로그에서 'Manage for service cinder -volume node0@lvm is ...' 이 부분이 계속 반복 되는 부분 입니다. 혹시, 이에 대해서 경험 해 보신 분 있으시면, 조언 부탁 드립니다. 감사합니다.
기본 설치 가이드를 따라서 설치하셨다면 lvm-cinder 로 설치하셨을 텐데, 기억으로는 아마 그 가이드에 cinder용 lvm 볼륨을 생성하는 부분이 빠져있어서 cinder 데몬이 제대로 작동하지 않았을 수 있습니다. 이 경우라면 해결 방법은 cinder lvm 볼륨명을 cinder.conf 에 있는 것으로 수작업으로 만들어주고 cinder-volume 서비스를 재시작하면 됩니다. 보낸 사람: Minjun Hong 보낸 날짜: 2018년 12월 19일 수요일 오후 11:28 받는 사람: openstack-ko@lists.openstack.org 제목: [openstack-ko] [Cinder] 인스턴스 생성 시 volume 연결 문제 안녕하세요. Openstack을 학교 서버에 수동으로 설치 중인데, Cinder 관련 해서 문제가 발생해서 도움 요청 합니다. Queens 버전 최소 설치 가이드 (https://docs.openstack.org/cinder/queens/install/index-ubuntu.html)를 참고 해서 설치 중인데, 인스턴스를 생성 할 때, 해당 인스턴스가 사용할 volume을 storage server에서 못받아 오는 것 처럼 보입니다. iSCSI를 위한 tgt 서비스를 보면, 뭔가 커널 모듈이 추가적으로 로드 되어야 하는거 같은데, 이게 스토리지 서버에 있는 RAID controller 와 관련이 있을까요? 아래는 tgt 서비스의 상태입니다. ● tgt.service - (i)SCSI target daemon Loaded: loaded (/lib/systemd/system/tgt.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2018-12-19 22:05:55 KST; 26min ago Docs: man:tgtd(8) Process: 2249 ExecStartPost=/usr/sbin/tgtadm --op update --mode sys --name State -v ready (code=exited, status=0/SUCCESS) Process: 2230 ExecStartPost=/usr/sbin/tgt-admin -e -c /etc/tgt/targets.conf (code=exited, status=0/SUCCESS) Process: 2206 ExecStartPost=/usr/sbin/tgtadm --op update --mode sys --name State -v offline (code=exited, status=0/SUCCESS) Main PID: 2170 (tgtd) Status: "Starting event loop..." Tasks: 1 Memory: 2.7M CPU: 87ms CGroup: /system.slice/tgt.service └─2170 /usr/sbin/tgtd -f Dec 19 22:05:55 node0 systemd[1]: Starting (i)SCSI target daemon... Dec 19 22:05:55 node0 tgtd[2170]: tgtd: iser_ib_init(3434) Failed to initialize RDMA; load kernel modules? Dec 19 22:05:55 node0 tgtd[2170]: tgtd: work_timer_start(150) use signal based scheduler Dec 19 22:05:55 node0 tgtd[2170]: tgtd: bs_init(387) use signalfd notification Dec 19 22:05:55 node0 systemd[1]: Started (i)SCSI target daemon. 그리고 아래는 cinder 서비스의 상태 입니다. ● cinder-volume.service - OpenStack Cinder Volume Loaded: loaded (/lib/systemd/system/cinder-volume.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2018-12-19 22:05:55 KST; 45min ago Main PID: 2110 (cinder-volume) Tasks: 2 Memory: 210.2M CPU: 1min 23.649s CGroup: /system.slice/cinder-volume.service ├─2110 /usr/bin/python2 /usr/bin/cinder-volume --config-file=/etc/cinder/cinder.conf --log-file=/var/log/cinder/cinder-volume.log └─2657 /usr/bin/python2 /usr/bin/cinder-volume --config-file=/etc/cinder/cinder.conf --log-file=/var/log/cinder/cinder-volume.log Dec 19 22:05:55 node0 systemd[1]: Started OpenStack Cinder Volume. Dec 19 22:06:00 node0 sudo[2658]: pam_unix(sudo:auth): auth could not identify password for [cinder] Cinder-volume의 로그는 계속해서 cinder-volume 서비스가 통신을 못하고 있는 것 처럼 보이는데, 아래와 같이 volume에 접근 하려고 할 때 (e.g. instance 생성), 에러 메시지가 출력 되고, 그 이후에는 계속해서 동일한 메시지가 매초 반복되고 있습니다. 2018-12-19 22:04:00.747 35996 INFO oslo_service.service [req-bcc39e60-531e-4484-ab4b-bab085d112c4 - - - - -] Caught SIGTERM, stopping children 2018-12-19 22:04:00.784 35996 INFO oslo_service.service [req-bcc39e60-531e-4484-ab4b-bab085d112c4 - - - - -] Waiting on 1 children to exit 2018-12-19 22:04:00.809 35996 INFO oslo_service.service [req-bcc39e60-531e-4484-ab4b-bab085d112c4 - - - - -] Child 36039 killed by signal 15 2018-12-19 22:05:58.378 2110 INFO root [-] Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt 2018-12-19 22:05:58.403 2110 INFO root [-] Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt 2018-12-19 22:05:59.820 2110 INFO cinder.rpc [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Automatically selected cinder-scheduler objects version 1.35 as minimum service version. 2018-12-19 22:05:59.826 2110 INFO cinder.rpc [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Automatically selected cinder-scheduler RPC version 3.10 as minimum service version. 2018-12-19 22:05:59.875 2110 INFO cinder.volume.manager [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Determined volume DB was empty at startup. 2018-12-19 22:05:59.908 2110 WARNING oslo_config.cfg [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Option "iscsi_helper" from group "lvm" is deprecated. Use option "target_helper" from group "lvm". 2018-12-19 22:05:59.914 2110 WARNING oslo_config.cfg [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Option "iscsi_protocol" from group "lvm" is deprecated. Use option "target_protocol" from group "lvm". 2018-12-19 22:05:59.916 2110 INFO cinder.volume.manager [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Image-volume cache disabled for host node0@lvm. 2018-12-19 22:05:59.932 2110 INFO oslo_service.service [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Starting 1 workers 2018-12-19 22:05:59.940 2657 INFO cinder.service [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Starting cinder-volume node (version 13.0.0) 2018-12-19 22:05:59.942 2110 WARNING oslo_config.cfg [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Option "iscsi_helper" from group "DEFAULT" is deprecated. Use option "target_helper" from group "DEFAULT". 2018-12-19 22:05:59.952 2110 WARNING oslo_config.cfg [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Option "auth_uri" from group "keystone_authtoken" is deprecated for removal (The auth_uri option is deprecated in favor of www_authenticate_uri and will be removed in the S release.). Its value may be silently ignored in the future. 2018-12-19 22:05:59.953 2110 WARNING oslo_config.cfg [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Option "auth_uri" from group "keystone_authtoken" is deprecated. Use option "www_authenticate_uri" from group "keystone_authtoken". 2018-12-19 22:05:59.961 2657 INFO cinder.volume.manager [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Starting volume driver LVMVolumeDriver (3.0.0) 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Failed to initialize driver.: ProcessExecutionError: Unexpected error while running command. Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C vgs --version Exit code: 1 Stdout: u'' Stderr: u'sudo: no tty present and no askpass program specified\n' 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager Traceback (most recent call last): 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/manager.py", line 433, in init_host 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager self.driver.check_for_setup_error() 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/lvm.py", line 286, in check_for_setup_error 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager if volutils.supports_thin_provisioning(): 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/utils.py", line 628, in supports_thin_provisioning 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager utils.get_root_helper()) 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/brick/local_dev/lvm.py", line 225, in supports_thin_provisioning 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager return LVM.get_lvm_version(root_helper) >= (2, 2, 95) 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/brick/local_dev/lvm.py", line 202, in get_lvm_version 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager run_as_root=True) 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 424, in execute 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager cmd=sanitized_cmd) 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager ProcessExecutionError: Unexpected error while running command. 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C vgs --version 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager Exit code: 1 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager Stdout: u'' 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager Stderr: u'sudo: no tty present and no askpass program specified\n' 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager 2018-12-19 22:06:00.075 2657 INFO cinder.volume.manager [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Initializing RPC dependent components of volume driver LVMVolumeDriver (3.0.0) 2018-12-19 22:06:00.076 2657 ERROR cinder.utils [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Volume driver LVMVolumeDriver not initialized 2018-12-19 22:06:00.076 2657 ERROR cinder.volume.manager [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Cannot complete RPC initialization because driver isn't initialized properly.: DriverNotInitialized: Volume driver not ready. 2018-12-19 22:06:10.077 2657 ERROR cinder.service [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Manager for service cinder-volume node0@lvm is reporting problems, not sending heartbeat. Service will appear "down". 2018-12-19 22:06:20.077 2657 ERROR cinder.service [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Manager for service cinder-volume node0@lvm is reporting problems, not sending heartbeat. Service will appear "down". 2018-12-19 22:06:30.078 2657 ERROR cinder.service [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Manager for service cinder-volume node0@lvm is reporting problems, not sending heartbeat. Service will appear "down". 2018-12-19 22:06:40.078 2657 ERROR cinder.service [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Manager for service cinder-volume node0@lvm is reporting problems, not sending heartbeat. Service will appear "down". 2018-12-19 22:06:50.079 2657 ERROR cinder.service [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Manager for service cinder-volume node0@lvm is reporting problems, not sending heartbeat. Service will appear "down". 2018-12-19 22:07:00.079 2657 ERROR cinder.service [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Manager for service cinder-volume node0@lvm is reporting problems, not sending heartbeat. Service will appear "down". 2018-12-19 22:07:10.078 2657 WARNING cinder.volume.manager [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Update driver status failed: (config name lvm) is uninitialized. 위의 로그에서 'Manage for service cinder -volume node0@lvm is ...' 이 부분이 계속 반복 되는 부분 입니다. 혹시, 이에 대해서 경험 해 보신 분 있으시면, 조언 부탁 드립니다. 감사합니다.
직접적인 원인은 "sudo: no tty....." 문구로 보이네요 아래처럼 해보시겠어요? # vi /etc/sudoers 1. 아래부분 주석처리 #Defaults requiretty 2. cinder 유저권한 확인 cinder ALL=(ALL) NOPASSWD: ALL 3. cinder-volume 서비스 재시작 2018년 12월 20일 (목) 오후 4:19에 Jongwoo Han <jongwoohan@gmail.com>님이 작성:
기본 설치 가이드를 따라서 설치하셨다면 lvm-cinder 로 설치하셨을 텐데, 기억으로는 아마 그 가이드에 cinder용 lvm 볼륨을 생성하는 부분이 빠져있어서 cinder 데몬이 제대로 작동하지 않았을 수 있습니다.
이 경우라면 해결 방법은 cinder lvm 볼륨명을 cinder.conf 에 있는 것으로 수작업으로 만들어주고 cinder-volume 서비스를 재시작하면 됩니다.
*보낸 사람: *Minjun Hong <nickeysgo@gmail.com> *보낸 날짜: *2018년 12월 19일 수요일 오후 11:28 *받는 사람: *openstack-ko@lists.openstack.org *제목: *[openstack-ko] [Cinder] 인스턴스 생성 시 volume 연결 문제
안녕하세요. Openstack을 학교 서버에 수동으로 설치 중인데, Cinder 관련 해서 문제가 발생해서 도움 요청 합니다.
Queens 버전 최소 설치 가이드 ( https://docs.openstack.org/cinder/queens/install/index-ubuntu.html)를 참고 해서 설치 중인데, 인스턴스를 생성 할 때, 해당 인스턴스가 사용할 volume을 storage server에서 못받아 오는 것 처럼 보입니다.
iSCSI를 위한 tgt 서비스를 보면, 뭔가 커널 모듈이 추가적으로 로드 되어야 하는거 같은데, 이게 스토리지 서버에 있는 RAID controller 와 관련이 있을까요?
아래는 tgt 서비스의 상태입니다.
● tgt.service - (i)SCSI target daemon Loaded: loaded (/lib/systemd/system/tgt.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2018-12-19 22:05:55 KST; 26min ago Docs: man:tgtd(8) Process: 2249 ExecStartPost=/usr/sbin/tgtadm --op update --mode sys --name State -v ready (code=exited, status=0/SUCCESS) Process: 2230 ExecStartPost=/usr/sbin/tgt-admin -e -c /etc/tgt/targets.conf (code=exited, status=0/SUCCESS) Process: 2206 ExecStartPost=/usr/sbin/tgtadm --op update --mode sys --name State -v offline (code=exited, status=0/SUCCESS) Main PID: 2170 (tgtd) Status: "Starting event loop..." Tasks: 1 Memory: 2.7M CPU: 87ms CGroup: /system.slice/tgt.service └─2170 /usr/sbin/tgtd -f Dec 19 22:05:55 node0 systemd[1]: Starting (i)SCSI target daemon... Dec 19 22:05:55 node0 tgtd[2170]: tgtd: iser_ib_init(3434) Failed to initialize RDMA; load kernel modules? Dec 19 22:05:55 node0 tgtd[2170]: tgtd: work_timer_start(150) use signal based scheduler Dec 19 22:05:55 node0 tgtd[2170]: tgtd: bs_init(387) use signalfd notification Dec 19 22:05:55 node0 systemd[1]: Started (i)SCSI target daemon.
그리고 아래는 cinder 서비스의 상태 입니다.
● cinder-volume.service - OpenStack Cinder Volume Loaded: loaded (/lib/systemd/system/cinder-volume.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2018-12-19 22:05:55 KST; 45min ago Main PID: 2110 (cinder-volume) Tasks: 2 Memory: 210.2M CPU: 1min 23.649s CGroup: /system.slice/cinder-volume.service ├─2110 /usr/bin/python2 /usr/bin/cinder-volume --config-file=/etc/cinder/cinder.conf --log-file=/var/log/cinder/cinder-volume.log └─2657 /usr/bin/python2 /usr/bin/cinder-volume --config-file=/etc/cinder/cinder.conf --log-file=/var/log/cinder/cinder-volume.log Dec 19 22:05:55 node0 systemd[1]: Started OpenStack Cinder Volume. Dec 19 22:06:00 node0 sudo[2658]: pam_unix(sudo:auth): auth could not identify password for [cinder]
Cinder-volume의 로그는 계속해서 cinder-volume 서비스가 통신을 못하고 있는 것 처럼 보이는데, 아래와 같이 volume에 접근 하려고 할 때 (e.g. instance 생성), 에러 메시지가 출력 되고, 그 이후에는 계속해서 동일한 메시지가 매초 반복되고 있습니다.
2018-12-19 22:04:00.747 35996 INFO oslo_service.service [req-bcc39e60-531e-4484-ab4b-bab085d112c4 - - - - -] Caught SIGTERM, stopping children 2018-12-19 22:04:00.784 35996 INFO oslo_service.service [req-bcc39e60-531e-4484-ab4b-bab085d112c4 - - - - -] Waiting on 1 children to exit 2018-12-19 22:04:00.809 35996 INFO oslo_service.service [req-bcc39e60-531e-4484-ab4b-bab085d112c4 - - - - -] Child 36039 killed by signal 15 2018-12-19 22:05:58.378 2110 INFO root [-] Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt 2018-12-19 22:05:58.403 2110 INFO root [-] Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt 2018-12-19 22:05:59.820 2110 INFO cinder.rpc [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Automatically selected cinder-scheduler objects version 1.35 as minimum service version. 2018-12-19 22:05:59.826 2110 INFO cinder.rpc [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Automatically selected cinder-scheduler RPC version 3.10 as minimum service version. 2018-12-19 22:05:59.875 2110 INFO cinder.volume.manager [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Determined volume DB was empty at startup. 2018-12-19 22:05:59.908 2110 WARNING oslo_config.cfg [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Option "iscsi_helper" from group "lvm" is deprecated. Use option "target_helper" from group "lvm". 2018-12-19 22:05:59.914 2110 WARNING oslo_config.cfg [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Option "iscsi_protocol" from group "lvm" is deprecated. Use option "target_protocol" from group "lvm". 2018-12-19 22:05:59.916 2110 INFO cinder.volume.manager [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Image-volume cache disabled for host node0@lvm. 2018-12-19 22:05:59.932 2110 INFO oslo_service.service [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Starting 1 workers 2018-12-19 22:05:59.940 2657 INFO cinder.service [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Starting cinder-volume node (version 13.0.0) 2018-12-19 22:05:59.942 2110 WARNING oslo_config.cfg [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Option "iscsi_helper" from group "DEFAULT" is deprecated. Use option "target_helper" from group "DEFAULT". 2018-12-19 22:05:59.952 2110 WARNING oslo_config.cfg [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Option "auth_uri" from group "keystone_authtoken" is deprecated for removal (The auth_uri option is deprecated in favor of www_authenticate_uri and will be removed in the S release.). Its value may be silently ignored in the future. 2018-12-19 22:05:59.953 2110 WARNING oslo_config.cfg [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Option "auth_uri" from group "keystone_authtoken" is deprecated. Use option "www_authenticate_uri" from group "keystone_authtoken". 2018-12-19 22:05:59.961 2657 INFO cinder.volume.manager [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Starting volume driver LVMVolumeDriver (3.0.0) 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Failed to initialize driver.: ProcessExecutionError: Unexpected error while running command. Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C vgs --version Exit code: 1 Stdout: u'' Stderr: u'sudo: no tty present and no askpass program specified\n' 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager Traceback (most recent call last): 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/manager.py", line 433, in init_host 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager self.driver.check_for_setup_error() 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/lvm.py", line 286, in check_for_setup_error 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager if volutils.supports_thin_provisioning(): 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/utils.py", line 628, in supports_thin_provisioning 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager utils.get_root_helper()) 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/brick/local_dev/lvm.py", line 225, in supports_thin_provisioning 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager return LVM.get_lvm_version(root_helper) >= (2, 2, 95) 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/brick/local_dev/lvm.py", line 202, in get_lvm_version 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager run_as_root=True) 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 424, in execute 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager cmd=sanitized_cmd) 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager ProcessExecutionError: Unexpected error while running command. 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C vgs --version 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager Exit code: 1 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager Stdout: u'' 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager Stderr: u'sudo: no tty present and no askpass program specified\n' 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager 2018-12-19 22:06:00.075 2657 INFO cinder.volume.manager [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Initializing RPC dependent components of volume driver LVMVolumeDriver (3.0.0) 2018-12-19 22:06:00.076 2657 ERROR cinder.utils [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Volume driver LVMVolumeDriver not initialized 2018-12-19 22:06:00.076 2657 ERROR cinder.volume.manager [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Cannot complete RPC initialization because driver isn't initialized properly.: DriverNotInitialized: Volume driver not ready. 2018-12-19 22:06:10.077 2657 ERROR cinder.service [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Manager for service cinder-volume node0@lvm is reporting problems, not sending heartbeat. Service will appear "down". 2018-12-19 22:06:20.077 2657 ERROR cinder.service [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Manager for service cinder-volume node0@lvm is reporting problems, not sending heartbeat. Service will appear "down". 2018-12-19 22:06:30.078 2657 ERROR cinder.service [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Manager for service cinder-volume node0@lvm is reporting problems, not sending heartbeat. Service will appear "down". 2018-12-19 22:06:40.078 2657 ERROR cinder.service [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Manager for service cinder-volume node0@lvm is reporting problems, not sending heartbeat. Service will appear "down". 2018-12-19 22:06:50.079 2657 ERROR cinder.service [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Manager for service cinder-volume node0@lvm is reporting problems, not sending heartbeat. Service will appear "down". 2018-12-19 22:07:00.079 2657 ERROR cinder.service [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Manager for service cinder-volume node0@lvm is reporting problems, not sending heartbeat. Service will appear "down". 2018-12-19 22:07:10.078 2657 WARNING cinder.volume.manager [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Update driver status failed: (config name lvm) is uninitialized.
위의 로그에서 'Manage for service cinder -volume node0@lvm is ...' 이 부분이 계속 반복 되는 부분 입니다.
혹시, 이에 대해서 경험 해 보신 분 있으시면, 조언 부탁 드립니다.
감사합니다.
_______________________________________________ openstack-ko mailing list openstack-ko@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-ko
participants (3)
-
Jongwoo Han
-
Minjun Hong
-
한승진