- openstack-discuss - lists.openstack.org

[Cinder] cinder-volume connection problem when creating the instance
by Minjun Hong 20 Dec '18

20 Dec '18

Hi. I'm installing Cinder service in an storage server of my laboratory and I need you guys' help about Cinder. I'm installing Cinder by referring 'Cinder Installation Guide for Ubuntu' ( https://docs.openstack.org/cinder/queens/install/index-ubuntu.html) However, when I request creating an instance, it seems that Openstack cannot receive the volume which the instance needs from the storage server. And seeing status message of tgt service for iSCSI, it seems that the service needs certain kernel module. Status of tgt service is following: > ● tgt.service - (i)SCSI target daemon > Loaded: loaded (/lib/systemd/system/tgt.service; enabled; vendor > preset: enabled) > Active: active (running) since Wed 2018-12-19 22:05:55 KST; 26min ago > Docs: man:tgtd(8) > Process: 2249 ExecStartPost=/usr/sbin/tgtadm --op update --mode sys > --name State -v ready (code=exited, status=0/SUCCESS) > Process: 2230 ExecStartPost=/usr/sbin/tgt-admin -e -c > /etc/tgt/targets.conf (code=exited, status=0/SUCCESS) > Process: 2206 ExecStartPost=/usr/sbin/tgtadm --op update --mode sys > --name State -v offline (code=exited, status=0/SUCCESS) > Main PID: 2170 (tgtd) > Status: "Starting event loop..." > Tasks: 1 > Memory: 2.7M > CPU: 87ms > CGroup: /system.slice/tgt.service > └─2170 /usr/sbin/tgtd -f > Dec 19 22:05:55 node0 systemd[1]: Starting (i)SCSI target daemon... > Dec 19 22:05:55 node0 tgtd[2170]: tgtd: iser_ib_init(3434) Failed to > initialize RDMA; load kernel modules? > Dec 19 22:05:55 node0 tgtd[2170]: tgtd: work_timer_start(150) use signal > based scheduler > Dec 19 22:05:55 node0 tgtd[2170]: tgtd: bs_init(387) use signalfd > notification > Dec 19 22:05:55 node0 systemd[1]: Started (i)SCSI target daemon. And the status of Cinder-volume service as follows: > ● cinder-volume.service - OpenStack Cinder Volume > Loaded: loaded (/lib/systemd/system/cinder-volume.service; enabled; > vendor preset: enabled) > Active: active (running) since Wed 2018-12-19 22:05:55 KST; 45min ago > Main PID: 2110 (cinder-volume) > Tasks: 2 > Memory: 210.2M > CPU: 1min 23.649s > CGroup: /system.slice/cinder-volume.service > ├─2110 /usr/bin/python2 /usr/bin/cinder-volume > --config-file=/etc/cinder/cinder.conf > --log-file=/var/log/cinder/cinder-volume.log > └─2657 /usr/bin/python2 /usr/bin/cinder-volume > --config-file=/etc/cinder/cinder.conf > --log-file=/var/log/cinder/cinder-volume.log > Dec 19 22:05:55 node0 systemd[1]: Started OpenStack Cinder Volume. > Dec 19 22:06:00 node0 sudo[2658]: pam_unix(sudo:auth): auth could not > identify password for [cinder] Log message of Cinder-volume service continuously complains that the service cannot communicate with the internal driver (maybe, LVM?). In other words, when Cinder-volume is accessed (e.g. instance creation), an error message is printed and after that, the same message is repeated every second as follows: 2018-12-19 22:04:00.747 35996 INFO oslo_service.service > [req-bcc39e60-531e-4484-ab4b-bab085d112c4 - - - - -] Caught SIGTERM, > stopping children > 2018-12-19 22:04:00.784 35996 INFO oslo_service.service > [req-bcc39e60-531e-4484-ab4b-bab085d112c4 - - - - -] Waiting on 1 children > to exit > 2018-12-19 22:04:00.809 35996 INFO oslo_service.service > [req-bcc39e60-531e-4484-ab4b-bab085d112c4 - - - - -] Child 36039 killed by > signal 15 > 2018-12-19 22:05:58.378 2110 INFO root [-] Generating grammar tables from > /usr/lib/python2.7/lib2to3/Grammar.txt > 2018-12-19 22:05:58.403 2110 INFO root [-] Generating grammar tables from > /usr/lib/python2.7/lib2to3/PatternGrammar.txt > 2018-12-19 22:05:59.820 2110 INFO cinder.rpc > [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Automatically selected > cinder-scheduler objects version 1.35 as minimum service version. > 2018-12-19 22:05:59.826 2110 INFO cinder.rpc > [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Automatically selected > cinder-scheduler RPC version 3.10 as minimum service version. > 2018-12-19 22:05:59.875 2110 INFO cinder.volume.manager > [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Determined volume DB > was empty at startup. > 2018-12-19 22:05:59.908 2110 WARNING oslo_config.cfg > [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Option "iscsi_helper" > from group "lvm" is deprecated. Use option "target_helper" from group "lvm". > 2018-12-19 22:05:59.914 2110 WARNING oslo_config.cfg > [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Option > "iscsi_protocol" from group "lvm" is deprecated. Use option > "target_protocol" from group "lvm". > 2018-12-19 22:05:59.916 2110 INFO cinder.volume.manager > [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Image-volume cache > disabled for host node0@lvm. > 2018-12-19 22:05:59.932 2110 INFO oslo_service.service > [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Starting 1 workers > 2018-12-19 22:05:59.940 2657 INFO cinder.service > [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Starting cinder-volume > node (version 13.0.0) > 2018-12-19 22:05:59.942 2110 WARNING oslo_config.cfg > [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Option "iscsi_helper" > from group "DEFAULT" is deprecated. Use option "target_helper" from group > "DEFAULT". > 2018-12-19 22:05:59.952 2110 WARNING oslo_config.cfg > [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Option "auth_uri" from > group "keystone_authtoken" is deprecated for removal (The auth_uri option > is deprecated in favor of www_authenticate_uri and will be removed in the > S release.). Its value may be silently ignored in the future. > 2018-12-19 22:05:59.953 2110 WARNING oslo_config.cfg > [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Option "auth_uri" from > group "keystone_authtoken" is deprecated. Use option "www_authenticate_uri" > from group "keystone_authtoken". > 2018-12-19 22:05:59.961 2657 INFO cinder.volume.manager > [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Starting volume driver > LVMVolumeDriver (3.0.0) > 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager > [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Failed to initialize > driver.: ProcessExecutionError: Unexpected error while running command. > Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C vgs > --version > Exit code: 1 > Stdout: u'' > Stderr: u'sudo: no tty present and no askpass program specified\n' > 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager Traceback (most > recent call last): > 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager File > "/opt/stack/cinder/cinder/volume/manager.py", line 433, in init_host > 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager > self.driver.check_for_setup_error() > 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager File > "/opt/stack/cinder/cinder/volume/drivers/lvm.py", line 286, in > check_for_setup_error > 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager if > volutils.supports_thin_provisioning(): > 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager File > "/opt/stack/cinder/cinder/volume/utils.py", line 628, in > supports_thin_provisioning > 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager > utils.get_root_helper()) > 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager File > "/opt/stack/cinder/cinder/brick/local_dev/lvm.py", line 225, in > supports_thin_provisioning > 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager return > LVM.get_lvm_version(root_helper) >= (2, 2, 95) > 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager File > "/opt/stack/cinder/cinder/brick/local_dev/lvm.py", line 202, in > get_lvm_version > 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager > run_as_root=True) > 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager File > "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", > line 424, in execute > 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager > cmd=sanitized_cmd) > 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager > ProcessExecutionError: Unexpected error while running command. > 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager Command: sudo > cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C vgs --version > 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager Exit code: 1 > 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager Stdout: u'' > 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager Stderr: u'sudo: > no tty present and no askpass program specified\n' > 2018-12-19 22:06:00.007 2657 ERROR cinder.volume.manager > 2018-12-19 22:06:00.075 2657 INFO cinder.volume.manager > [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Initializing RPC > dependent components of volume driver LVMVolumeDriver (3.0.0) > 2018-12-19 22:06:00.076 2657 ERROR cinder.utils > [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Volume driver > LVMVolumeDriver not initialized > 2018-12-19 22:06:00.076 2657 ERROR cinder.volume.manager > [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Cannot complete RPC > initialization because driver isn't initialized properly.: > DriverNotInitialized: Volume driver not ready. > 2018-12-19 22:06:10.077 2657 ERROR cinder.service > [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Manager for service > cinder-volume node0@lvm is reporting problems, not sending heartbeat. > Service will appear "down". > 2018-12-19 22:06:20.077 2657 ERROR cinder.service > [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Manager for service > cinder-volume node0@lvm is reporting problems, not sending heartbeat. > Service will appear "down". > 2018-12-19 22:06:30.078 2657 ERROR cinder.service > [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Manager for service > cinder-volume node0@lvm is reporting problems, not sending heartbeat. > Service will appear "down". > 2018-12-19 22:06:40.078 2657 ERROR cinder.service > [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Manager for service > cinder-volume node0@lvm is reporting problems, not sending heartbeat. > Service will appear "down". > 2018-12-19 22:06:50.079 2657 ERROR cinder.service > [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Manager for service > cinder-volume node0@lvm is reporting problems, not sending heartbeat. > Service will appear "down". > 2018-12-19 22:07:00.079 2657 ERROR cinder.service > [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Manager for service > cinder-volume node0@lvm is reporting problems, not sending heartbeat. > Service will appear "down". > 2018-12-19 22:07:10.078 2657 WARNING cinder.volume.manager > [req-b3178448-4e17-489b-8267-2fd7992bc876 - - - - -] Update driver status > failed: (config name lvm) is uninitialized. In above log, 'Manage for service cinder -volume node0@lvm is ...' is the repeated part. If there is anyone who has experienced this kind of problem, please give me any advice. Thanks!

2 2

[neutron] [upgrade] No meeting on Dec. 27th
by Lujin Luo 20 Dec '18

20 Dec '18

Hi everyone, We will not have the next subteam meeting on Dec. 27th. Thank you for all of your work this year, and let's resume in next year, Jan. 3rd. Happy Holidays! Best regards, Lujin

1 0

[nova][vitrage] Nova extended server attributes
by Ifat Afek 20 Dec '18

20 Dec '18

Hi, Is there a way to get Nova extended server attributes (and specifically, OS-EXT-SRV-ATTR:instance_name) as part of the notification that is sent upon instance creation? In Vitrage we are using the immediate notifications from Nova, but this specific information is missing. Thanks, Ifat

2 2

[watcher] core reviewers update
by Чадин Александр Сергеевич 20 Dec '18

20 Dec '18

Hello Watcher team, Some of core contributors are inactive for a year or more and their priorities have been changed. I suggest to remove the following contributors from core reviewers of Watcher: - Joe Cropper - Prashanth Hari - Prudhvi Rao Shedimbi - Susanne Balle - Vincent Francoise - David Tardivel If some of them would like to get back to work on Watcher, I see no barriers for that. Best Regards, ____ Alex

1 0

[storyboard] it's all gone quiet
by Mark Goddard 20 Dec '18

20 Dec '18

I haven't received an email from StoryBoard for a few weeks now. While I'm quite enjoying the silence, it's probably not productive... I'm subscribed to a few projects and have email notifications checked in preferences. Is it just me? Mark

2 1

[all] Please help document dependencies between services
by Thierry Carrez 20 Dec '18

20 Dec '18

Hi everyone, As Jimmy announced[1] a couple of weeks ago, we need your help in compiling accurate dependency information between OpenStack services, to display in future versions of the openstack.org/software pages[2]. Each OpenStack service may have hard dependencies (can't work without) or soft dependencies (benefits of the presence of) on other services, and that is very useful information for users looking into various OpenStack components. You can help accurately document those, by proposing changes to the openstack/openstack-map repository[3], looking up your service in the openstack_components.yaml file and adding the following keys: dependencies: (for hard dependencies, when the service can't work without those other services) see-also: (for soft dependencies, when the service benefits of the presence of those other services) As an example, you can look at Octavia's entry[4]. Once we have that information compiled, we'll be able to publish it on the website, adding links between related services. Thanks in advance, [1] http://lists.openstack.org/pipermail/openstack-discuss/2018-December/000568… [2] https://www.openstack.org/software/project-navigator/openstack-components#o… [3] https://git.openstack.org/cgit/openstack/openstack-map/ [4] https://git.openstack.org/cgit/openstack/openstack-map/tree/openstack_compo… -- Thierry Carrez (ttx)

1 0

[dev][qa] Cancelling QA Office Hour for next two weeks
by Ghanshyam Mann 20 Dec '18

20 Dec '18

Hello Everyone, I will be on vacation from 24th Dec till 9th Jan, so cancelling the QA office hour for the next two weeks (27th Dec and 3rd Jan). We will resume it from 10th Jan. -gmann

1 0

[ironic][neutron] Baremetal node cannot get back IP address from neutron
by Dave.Chen＠Dell.com 19 Dec '18

19 Dec '18

Hi all, I have setup ironic with devstack, initially, there are three fake nodes and I could successfully create a server, access to the server with the IP assigned from neutron as well, so I guess the configuration should be good, then I move to a real baremetal node, and want to do the same thing, so I issued the below command, $ openstack server create --flavor baremetal --image ${IMAGE_NAME} --key-name local realnode The node could power on and start to boot with PXE, two MAC address has been bound to the node, and seen from the BOOT screen the NIC was up but was pending on configuration and then exit, seem like it cannot get the IP address back from the Neutron. On the Ironic server, I can see the DHCP request from the baremetal node, but there is no ACK no IP offering here. $ sudo tcpdump -i enp0s3 -n port 67 and port 68 ... 18:21:08.112923 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 50:6b:4b:df:35:87, length 394 18:21:12.143679 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 50:6b:4b:df:35:87, length 394 ... Also tried to capture the traffic on the switch, it shows the request has been sent as well but no IP offering. dnsmasq is running, $ ps -ef | grep dnsm dnsmasq --no-hosts --strict-order --except-interface=lo --pid-file=/opt/stack/data/neutron/dhcp/ca541e23-629b-440d-9ef4-d42914af536c/pid --dhcp-hostsfile=/opt/stack/data/neutron/dhcp/ca541e23-629b-440d-9ef4-d42914af536c/host --addn-hosts=/opt/stack/data/neutron/dhcp/ca541e23-629b-440d-9ef4-d42914af536c/addn_hosts --dhcp-optsfile=/opt/stack/data/neutron/dhcp/ca541e23-629b-440d-9ef4-d42914af536c/opts --dhcp-leasefile=/opt/stack/data/neutron/dhcp/ca541e23-629b-440d-9ef4-d42914af536c/leases --dhcp-match=set:ipxe,175 --bind-interfaces --interface=tap4c76340a-7f --dhcp-range=set:tag1,10.0.0.0,static,255.255.255.192,86400s --dhcp-option-force=option:mtu,1450 --dhcp-lease-max=64 --conf-file= --domain=openstacklocal The data in neutron looks like below, $ /opt/stack/data/neutron/dhcp/ca541e23-629b-440d-9ef4-d42914af536c/host fa:16:3e:43:85:35,host-10-0-0-1.openstacklocal,10.0.0.1 fa:16:3e:19:60:19,host-10-0-0-2.openstacklocal,10.0.0.2 52:54:00:72:0e:a8,host-10-0-0-9.openstacklocal,10.0.0.9,set:348f921b-62c3-451f-884c-d630ae8f60b9 IP seems has been assigned to the baremetal node, but after the node changed its status to "Maintenance", the entry removed from this file, and the entry never get into the "lease" file, $ /opt/stack/data/neutron/dhcp/ca541e23-629b-440d-9ef4-d42914af536c/leases 1545344481 52:54:00:72:0e:a8 10.0.0.9 host-10-0-0-9 01:52:54:00:72:0e:a8 The only entry listed here is for the fake node. Could anyone kindly give me any suggestion where to looking into? How can I identify the root cause? Many thanks! Best Regards, Dave Chen

1 1

[opensetack-discuss][Openstack-Helm] Patches for upgrade rally
by Jae Sang Lee 19 Dec '18

19 Dec '18

Hi, I recently watched IRC talk about revert rally commit. I posted a patch[1] to upgrade rally last October. This is because openstack-helm is using a very old rally, which caused a dependency error when creating another image. I also changed the helm-toolkit of openstack-helm-infra because we need to change the rally init script.[2] After patch # 1 has been applied first, patch # 2 has to be merged. Only patch # 2 has been merged. Failure of all chart helm test or patch # 2 has been reverted again. So I updated the patch to upgrade rally today[3][4]. I hope [3] merges first. After that, [4] should be merged. Thanks. Jaesang [1] https://review.openstack.org/#/c/586783/1 [2] https://review.openstack.org/#/c/607822/ [3] https://review.openstack.org/#/c/586783 [4] https://review.openstack.org/626490

1 0

[dev] How to develop changes in a series
by Nadathur, Sundar 19 Dec '18

19 Dec '18

On Dec 5, 2018, Eric fried wrote: > This started off as a response to some questions Sundar asked, but Ithought it might be interesting/useful for new[er] OpenStack developers at large, so broadcasting. Most of this is familiar material for those used to github process. The 'git restack' part is interesting, though it seems a lot like 'git rebase -i'. Eric answered far more than what I asked. Thanks, Eric. Regards, Sundar

2 1