On 10/05, RAI, SNEHA wrote:
Thanks Gorka for your response.
I have changed the version of libvirt and qemu on my host and I am able to move past the previous error mentioned in my last email.
Current versions of libvirt and qemu: root@CSSOSBE04-B09:/etc# libvirtd --version libvirtd (libvirt) 1.3.1 root@CSSOSBE04-B09:/etc# kvm --version QEMU emulator version 2.5.0 (Debian 1:2.5+dfsg-5ubuntu10.36), Copyright (c) 2003-2008 Fabrice Bellard
Also, I made a change in /etc/nova/nova.conf and set virt_type=qemu. Earlier it was set to kvm. I restarted all nova services post the changes but I can see one nova service was disabled and state was down.
root@CSSOSBE04-B09:/etc# nova service-list +--------------------------------------+------------------+---------------+----------+----------+-------+----------------------------+-------------------------------------+-------------+ | Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason | Forced down | +--------------------------------------+------------------+---------------+----------+----------+-------+----------------------------+-------------------------------------+-------------+ | 1ebcd1f6-b7dc-40ce-8d7b-95d60503c0ff | nova-scheduler | CSSOSBE04-B09 | internal | enabled | up | 2019-05-10T05:48:59.000000 | - | False | | ed82277c-d2e0-4a1a-adf6-9bcdcc50ba29 | nova-consoleauth | CSSOSBE04-B09 | internal | enabled | up | 2019-05-10T05:48:49.000000 | - | False | | bc2b6703-7a1e-4f07-96b9-35cbb14398d5 | nova-conductor | CSSOSBE04-B09 | internal | enabled | up | 2019-05-10T05:48:59.000000 | - | False | | 72ecbc1d-1b47-4f55-a18d-de2fbf1771e9 | nova-conductor | CSSOSBE04-B09 | internal | enabled | up | 2019-05-10T05:48:54.000000 | - | False | | 9c700ee1-1694-479b-afc0-1fd37c1a5561 | nova-compute | CSSOSBE04-B09 | nova | disabled | down | 2019-05-07T22:11:06.000000 | AUTO: Connection to libvirt lost: 1 | False | +--------------------------------------+------------------+---------------+----------+----------+-------+----------------------------+-------------------------------------+-------------+
So, I manually enabled the service, but the state was still down. root@CSSOSBE04-B09:/etc# nova service-enable 9c700ee1-1694-479b-afc0-1fd37c1a5561 +--------------------------------------+---------------+--------------+---------+ | ID | Host | Binary | Status | +--------------------------------------+---------------+--------------+---------+ | 9c700ee1-1694-479b-afc0-1fd37c1a5561 | CSSOSBE04-B09 | nova-compute | enabled | +--------------------------------------+---------------+--------------+---------+
root@CSSOSBE04-B09:/etc# nova service-list +--------------------------------------+------------------+---------------+----------+---------+-------+----------------------------+-----------------+-------------+ | Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason | Forced down | +--------------------------------------+------------------+---------------+----------+---------+-------+----------------------------+-----------------+-------------+ | 1ebcd1f6-b7dc-40ce-8d7b-95d60503c0ff | nova-scheduler | CSSOSBE04-B09 | internal | enabled | up | 2019-05-10T05:49:19.000000 | - | False | | ed82277c-d2e0-4a1a-adf6-9bcdcc50ba29 | nova-consoleauth | CSSOSBE04-B09 | internal | enabled | up | 2019-05-10T05:49:19.000000 | - | False | | bc2b6703-7a1e-4f07-96b9-35cbb14398d5 | nova-conductor | CSSOSBE04-B09 | internal | enabled | up | 2019-05-10T05:49:19.000000 | - | False | | 72ecbc1d-1b47-4f55-a18d-de2fbf1771e9 | nova-conductor | CSSOSBE04-B09 | internal | enabled | up | 2019-05-10T05:49:14.000000 | - | False | | 9c700ee1-1694-479b-afc0-1fd37c1a5561 | nova-compute | CSSOSBE04-B09 | nova | enabled | down | 2019-05-10T05:49:14.000000 | - | False | +--------------------------------------+------------------+---------------+----------+---------+-------+----------------------------+-----------------+-------------+
Hi, If it appears as down it's probably because there is an issue during the service's start procedure. You can look in the logs to see what messages appeared during the start or tail the logs and restart the service to see what error appears there. Cheers, Gorka.
So, now when I try to attach a volume to nova instance, I get the below error. As one of the service is down it fails in filter validation for nova-compute and gives us "No host" error.
May 10 10:43:00 CSSOSBE04-B09 nova-scheduler[21775]: #033[00;32mDEBUG nova.filters [#033[01;36mNone req-b0ca81b3-a2b6-492e-9036-249644b94349 #033[00;36mdemo admin#033[00;32m] #033[01;35m#033[00;32mFilter RetryFilter returned 1 host(s)#033[00m #033[00;33m{{(pid=21775) get_filtered_objects /opt/stack/nova/nova/filters.py:104}}#033[00m May 10 10:43:00 CSSOSBE04-B09 nova-scheduler[21775]: #033[00;32mDEBUG nova.filters [#033[01;36mNone req-b0ca81b3-a2b6-492e-9036-249644b94349 #033[00;36mdemo admin#033[00;32m] #033[01;35m#033[00;32mFilter AvailabilityZoneFilter returned 1 host(s)#033[00m #033[00;33m{{(pid=21775) get_filtered_objects /opt/stack/nova/nova/filters.py:104}}#033[00m May 10 10:43:00 CSSOSBE04-B09 nova-scheduler[21775]: #033[00;32mDEBUG nova.scheduler.filters.compute_filter [#033[01;36mNone req-b0ca81b3-a2b6-492e-9036-249644b94349 #033[00;36mdemo admin#033[00;32m] #033[01;35m#033[00;32m(CSSOSBE04-B09, CSSOSBE04-B09) ram: 30810MB disk: 1737728MB io_ops: 0 instances: 1 is disabled, reason: AUTO: Connection to libvirt lost: 1#033[00m #033[00;33m{{(pid=21775) host_passes /opt/stack/nova/nova/scheduler/filters/compute_filter.py:42}}#033[00m May 10 10:43:00 CSSOSBE04-B09 nova-scheduler[21775]: #033[00;36mINFO nova.filters [#033[01;36mNone req-b0ca81b3-a2b6-492e-9036-249644b94349 #033[00;36mdemo admin#033[00;36m] #033[01;35m#033[00;36mFilter ComputeFilter returned 0 hosts#033[00m May 10 10:43:00 CSSOSBE04-B09 nova-scheduler[21775]: #033[00;32mDEBUG nova.filters [#033[01;36mNone req-b0ca81b3-a2b6-492e-9036-249644b94349 #033[00;36mdemo admin#033[00;32m] #033[01;35m#033[00;32mFiltering removed all hosts for the request with instance ID '1735ece5-d187-454a-aab1-12650646a2ec'. Filter results: [('RetryFilter', [(u'CSSOSBE04-B09', u'CSSOSBE04-B09')]), ('AvailabilityZoneFilter', [(u'CSSOSBE04-B09', u'CSSOSBE04-B09')]), ('ComputeFilter', None)]#033[00m #033[00;33m{{(pid=21775) get_filtered_objects /opt/stack/nova/nova/filters.py:129}}#033[00m May 10 10:43:00 CSSOSBE04-B09 nova-scheduler[21775]: #033[00;36mINFO nova.filters [#033[01;36mNone req-b0ca81b3-a2b6-492e-9036-249644b94349 #033[00;36mdemo admin#033[00;36m] #033[01;35m#033[00;36mFiltering removed all hosts for the request with instance ID '1735ece5-d187-454a-aab1-12650646a2ec'. Filter results: ['RetryFilter: (start: 1, end: 1)', 'AvailabilityZoneFilter: (start: 1, end: 1)', 'ComputeFilter: (start: 1, end: 0)']#033[00m May 10 10:43:00 CSSOSBE04-B09 nova-scheduler[21775]: #033[00;32mDEBUG nova.scheduler.filter_scheduler [#033[01;36mNone req-b0ca81b3-a2b6-492e-9036-249644b94349 #033[00;36mdemo admin#033[00;32m] #033[01;35m#033[00;32mFiltered []#033[00m #033[00;33m{{(pid=21775) _get_sorted_hosts /opt/stack/nova/nova/scheduler/filter_scheduler.py:404}}#033[00m May 10 10:43:00 CSSOSBE04-B09 nova-scheduler[21775]: #033[00;32mDEBUG nova.scheduler.filter_scheduler [#033[01;36mNone req-b0ca81b3-a2b6-492e-9036-249644b94349 #033[00;36mdemo admin#033[00;32m] #033[01;35m#033[00;32mThere are 0 hosts available but 1 instances requested to build.#033[00m #033[00;33m{{(pid=21775) _ensure_sufficient_hosts /opt/stack/nova/nova/scheduler/filter_scheduler.py:279}}#033[00m May 10 10:43:00 CSSOSBE04-B09 nova-conductor[21789]: #033[01;31mERROR nova.conductor.manager [#033[01;36mNone req-b0ca81b3-a2b6-492e-9036-249644b94349 #033[00;36mdemo admin#033[01;31m] #033[01;35m#033[01;31mFailed to schedule instances#033[00m: NoValidHost_Remote: No valid host was found. There are not enough hosts available. May 10 10:43:00 CSSOSBE04-B09 nova-conductor[21789]: Traceback (most recent call last): May 10 10:43:00 CSSOSBE04-B09 nova-conductor[21789]: File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 226, in inner May 10 10:43:00 CSSOSBE04-B09 nova-conductor[21789]: return func(*args, **kwargs) May 10 10:43:00 CSSOSBE04-B09 nova-conductor[21789]: File "/opt/stack/nova/nova/scheduler/manager.py", line 154, in select_destinations May 10 10:43:00 CSSOSBE04-B09 nova-conductor[21789]: allocation_request_version, return_alternates) May 10 10:43:00 CSSOSBE04-B09 nova-conductor[21789]: File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 91, in select_destinations May 10 10:43:00 CSSOSBE04-B09 nova-conductor[21789]: allocation_request_version, return_alternates) May 10 10:43:00 CSSOSBE04-B09 nova-conductor[21789]: File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 244, in _schedule May 10 10:43:00 CSSOSBE04-B09 nova-conductor[21789]: claimed_instance_uuids) May 10 10:43:00 CSSOSBE04-B09 nova-conductor[21789]: File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 281, in _ensure_sufficient_hosts May 10 10:43:00 CSSOSBE04-B09 nova-conductor[21789]: raise exception.NoValidHost(reason=reason) May 10 10:43:00 CSSOSBE04-B09 nova-conductor[21789]: NoValidHost: No valid host was found. There are not enough hosts available.
Need help in understanding on how to fix this error. For detailed logs, please refer the attached syslog.
Thanks & Regards, Sneha Rai
-----Original Message----- From: Gorka Eguileor [mailto:geguileo@redhat.com] Sent: Friday, May 10, 2019 2:56 PM To: RAI, SNEHA <sneha.rai@hpe.com> Cc: openstack-dev@lists.openstack.org Subject: Re: Help needed to Support Multi-attach feature
On 02/05, RAI, SNEHA wrote:
Hi Team,
I am currently working on multiattach feature for HPE 3PAR cinder driver.
For this, while setting up devstack(on stable/queens) I made below
change in the local.conf [[local|localrc]]
ENABLE_VOLUME_MULTIATTACH=True ENABLE_UBUNTU_CLOUD_ARCHIVE=False
/etc/cinder/cinder.conf:
[3pariscsi_1]
hpe3par_api_url =
https://urldefense.proofpoint.com/v2/url?u=https-3A__192.168.1.7-3A808
0_api_v1&d=DwIBAg&c=C5b8zRQO1miGmBeVZ2LFWg&r=8drU3i56Z5sQ_Ltpya89LTNn3
xDSwtigjYbGrSY1lM8&m=zTRvI4nj8MoP0_z5MmxTYwKiNNW6addwP4L5VFG4wkg&s=a2D
HbzzRtbbBPz0_kfodZv5X1HxbN_hFxte5rEZabAg&e=
hpe3par_username = user
hpe3par_password = password
san_ip = 192.168.1.7
san_login = user
san_password = password
volume_backend_name = 3pariscsi_1
hpe3par_cpg = my_cpg
hpe3par_iscsi_ips = 192.168.11.2,192.168.11.3 volume_driver =
cinder.volume.drivers.hpe.hpe_3par_iscsi.HPE3PARISCSIDriver
hpe3par_iscsi_chap_enabled = True
hpe3par_debug = True
image_volume_cache_enabled = True
/etc/cinder/policy.json:
'volume:multiattach': 'rule:admin_or_owner'
But I am getting below error in the nova log:
Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [None req-2cda6e90-fd45-4bfe-960a-7fca9ba4abab demo admin] [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa] Instance failed block device setup: MultiattachNotSupportedByVirtDriver: Volume dc25f09a-6ae1-4b06-a814-73a8afaba62f has 'multiattach' set, which is not supported for this instance.
Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa] Traceback (most recent call last):
Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa] File "/opt/stack/nova/nova/compute/manager.py", line 1615, in _prep_block_device
Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa] wait_func=self._await_block_device_map_created)
Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa] File "/opt/stack/nova/nova/virt/block_device.py", line 840, in attach_block_devices
Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa] _log_and_attach(device)
Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa] File "/opt/stack/nova/nova/virt/block_device.py", line 837, in _log_and_attach
Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa] bdm.attach(*attach_args, **attach_kwargs)
Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa] File "/opt/stack/nova/nova/virt/block_device.py", line 46, in wrapped
Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa] ret_val = method(obj, context, *args, **kwargs)
Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa] File "/opt/stack/nova/nova/virt/block_device.py", line 620, in attach
Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa] virt_driver, do_driver_attach)
Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa] File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 274, in inner
Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa] return f(*args, **kwargs)
Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa] File "/opt/stack/nova/nova/virt/block_device.py", line 617, in _do_locked_attach
Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa] self._do_attach(*args, **_kwargs)
Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa] File "/opt/stack/nova/nova/virt/block_device.py", line 602, in _do_attach
Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa] do_driver_attach)
Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa] File "/opt/stack/nova/nova/virt/block_device.py", line 509, in _volume_attach
Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa] volume_id=volume_id)
Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa] MultiattachNotSupportedByVirtDriver: Volume dc25f09a-6ae1-4b06-a814-73a8afaba62f has 'multiattach' set, which is not supported for this instance.
Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR
nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa]
Apr 29 05:41:20 CSSOSBE04-B09 nova-compute[20455]: DEBUG
nova.virt.libvirt.driver [-] Volume multiattach is not supported based
on current versions of QEMU and libvirt. QEMU must be less than 2.10
or libvirt must be greater than or equal to 3.10. {{(pid=20455)
_set_multiattach_support
/opt/stack/nova/nova/virt/libvirt/driver.py:619}}
stack@CSSOSBE04-B09:/tmp$ virsh --version
3.6.0
stack@CSSOSBE04-B09:/tmp$ kvm --version QEMU emulator version
2.10.1(Debian 1:2.10+dfsg-0ubuntu3.8~cloud1) Copyright (c) 2003-2017
Fabrice Bellard and the QEMU Project developers
Hi Sneha,
I don't know much about this side of Nova, but reading the log error I would say that you either need to update your libvirt version from 3.6.0 to 3.10, or you need to downgrade your QEMU version to something prior to 2.10.
The later is probably easier.
I don't use Ubuntu, but according to the Internet you can list available versions with "apt-cache policy qemu" and then install or downgrade to the specific version with "sudo apt-get install qemu=2.5\*" if you wanted to install version 2.5
I hope this helps.
Cheers,
Gorka.
openstack volume show -c multiattach -c status sneha1
+-------------+-----------+
| Field | Value |
+-------------+-----------+
| multiattach | True |
| status | available |
+-------------+-----------+
cinder extra-specs-list
+--------------------------------------+-------------+--------------------------------------------------------------------+
| ID | Name | extra_specs |
+--------------------------------------+-------------+--------------------------------------------------------------------+
| bd077fde-51c3-4581-80d5-5855e8ab2f6b | 3pariscsi_1 |
| {'volume_backend_name': '3pariscsi_1', 'multiattach': '<is> True'}|
+--------------------------------------+-------------+--------------------------------------------------------------------+
echo $OS_COMPUTE_API_VERSION
2.60
pip list | grep python-novaclient
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
python-novaclient 13.0.0
How do I fix this version issue on my setup to proceed? Please help.
Thanks & Regards,
Sneha Rai