Open Stack

I have a small testbed OpenStack cloud (running Ocata) where I am trying to
debug a problem with Nova scheduling.


In short: I see different behaviors when I create a new VM and when I try
to migrate a VM


Since I want to partition the Cloud so that each project uses only certain
compute nodes, I created one host aggregate per project (see also this
thread:
http://lists.openstack.org/pipermail/openstack-operators/2018-February/014831.html
)


The host-aggregate for my project is:

# nova  aggregate-show 52
+----+-----------+-------------------+--------------------------------------------------------------+----------------------------------------------------------------------------------------------+--------------------------------------+
| Id | Name      | Availability Zone | Hosts
                        | Metadata
                                           | UUID
       |
+----+-----------+-------------------+--------------------------------------------------------------+----------------------------------------------------------------------------------------------+--------------------------------------+
| 52 | SgaraPrj1 | nova              | 'compute-01.cloud.pd.infn.it', '
compute-02.cloud.pd.infn.it' | 'availability_zone=nova',
'filter_tenant_id=ee1865a76440481cbcff08544c7d580a', 'size=normal' |
675f6291-6997-470d-87e1-e9ea199a379f |
+----+-----------+-------------------+--------------------------------------------------------------+----------------------------------------------------------------------------------------------+--------------------------------------+

The same compute nodes are shared by other projects  (for which specific
host-aggregates, as this one, have been created)
The other compute node (I have only 3 compute nodes in this small testbed)
is targeted to other projects (for which specific host-aggregates exist)


This is what I have in nova.conf wrt scheduling filters:

enabled_filters =
AggregateInstanceExtraSpecsFilter,AggregateMultiTenancyIsolation,RetryFilter,AvailabilityZoneFilter,RamFilter,CoreFilter,AggregateRamFilter,AggregateCo
reFilter,DiskFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter



If I try to create a VM, I see from the scheduler log [*] that
the AggregateMultiTenancyIsolation selects only 2 compute nodes, as
expected.


But if I then try to migrate the very same VM, it reports that no valid
host was found:

# nova migrate afaf2a2d-7ff8-4e52-a89a-031ee079a9ba
ERROR (BadRequest): No valid host was found. No valid host found for cold
migrate (HTTP 400) (Request-ID: req-45b8afd5-9683-40a6-8416-295563e37e34)


And according to the scheduler log the problem is with the
AggregateMultiTenancyIsolation which returned 0 hosts (while I would have
expected one):

2018-05-29 11:12:56.375 19428 INFO nova.scheduler.host_manager
[req-45b8afd5-9683-40a6-8416-295563e37e34 9bd03f63fa9d4beb8de31e6c2f2c8d12
56c3f5c047e74a78a714\
38c4412e6e13 - - -\
] Host filter ignoring hosts: compute-02.cloud.pd.infn.it
2018-05-29 11:12:56.375 19428 DEBUG nova.filters
[req-45b8afd5-9683-40a6-8416-295563e37e34 9bd03f63fa9d4beb8de31e6c2f2c8d12
56c3f5c047e74a78a71438c4412e6e13 -\
 - -] Starting wit\
h 2 host(s) get_filtered_objects
/usr/lib/python2.7/site-packages/nova/filters.py:70
2018-05-29 11:12:56.376 19428 DEBUG nova.filters
[req-45b8afd5-9683-40a6-8416-295563e37e34 9bd03f63fa9d4beb8de31e6c2f2c8d12
56c3f5c047e74a78a71438c4412e6e13 -\
 - -] Filter Aggre\
gateInstanceExtraSpecsFilter returned 2 host(s) get_filtered_objects
/usr/lib/python2.7/site-packages/nova/filters.py:104
2018-05-29 11:12:56.377 19428 DEBUG
nova.scheduler.filters.aggregate_multitenancy_isolation
[req-45b8afd5-9683-40a6-8416-295563e37e34 9bd03f63fa9d4beb8de31e6c\
2f2c8d12 56c3f5c04\
7e74a78a71438c4412e6e13 - - -] (compute-01.cloud.pd.infn.it,
compute-01.cloud.pd.infn.it) ram: 12797MB disk: 48128MB io_ops: 0
instances: 0 fails tenant id on\
 aggregate host_pa\
sses
/usr/lib/python2.7/site-packages/nova/scheduler/filters/aggregate_multitenancy_isolation.py:50
2018-05-29 11:12:56.378 19428 DEBUG
nova.scheduler.filters.aggregate_multitenancy_isolation
[req-45b8afd5-9683-40a6-8416-295563e37e34 9bd03f63fa9d4beb8de31e6c\
2f2c8d12 56c3f5c04\
7e74a78a71438c4412e6e13 - - -] (compute-03.cloud.pd.infn.it,
compute-03.cloud.pd.infn.it) ram: 8701MB disk: -4096MB io_ops: 0 instances:
0 fails tenant id on \
aggregate host_pas\
ses
/usr/lib/python2.7/site-packages/nova/scheduler/filters/aggregate_multitenancy_isolation.py:50
2018-05-29 11:12:56.378 19428 INFO nova.filters
[req-45b8afd5-9683-40a6-8416-295563e37e34 9bd03f63fa9d4beb8de31e6c2f2c8d12
56c3f5c047e74a78a71438c4412e6e13 - \
- -] Filter Aggreg\
ateMultiTenancyIsolation returned 0 hosts



I am confused ...
Any hints ?

Thanks, Massimo

[*]


2018-05-29 11:09:54.328 19428 DEBUG nova.filters
[req-1a838e77-8042-4550-b157-4943445119a2 ab573ba3ea014b778193b6922ffffe6d
ee1865a76440481cbcff08544c7d580a -\
 - -] Filter AggregateInstanceExtraSpecsFilter returned 3 host(s)
get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:104
2018-05-29 11:09:54.330 19428 DEBUG nova.filters
[req-1a838e77-8042-4550-b157-4943445119a2 ab573ba3ea014b778193b6922ffffe6d
ee1865a76440481cbcff08544c7d580a -\
 - -] Filter AggregateMultiTenancyIsolation returned 2 host(s)
get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:104
2018-05-29 11:09:54.332 19428 DEBUG nova.filters
[req-1a838e77-8042-4550-b157-4943445119a2 ab573ba3ea014b778193b6922ffffe6d
ee1865a76440481cbcff08544c7d580a -\
 - -] Filter RetryFilter returned 2 host(s) get_filtered_objects
/usr/lib/python2.7/site-packages/nova/filters.py:104
2018-05-29 11:09:54.332 19428 DEBUG nova.filters
[req-1a838e77-8042-4550-b157-4943445119a2 ab573ba3ea014b778193b6922ffffe6d
ee1865a76440481cbcff08544c7d580a -\
 - -] Filter AvailabilityZoneFilter returned 2 host(s) get_filtered_objects
/usr/lib/python2.7/site-packages/nova/filters.py:104
2018-05-29 11:09:54.333 19428 DEBUG nova.filters
[req-1a838e77-8042-4550-b157-4943445119a2 ab573ba3ea014b778193b6922ffffe6d
ee1865a76440481cbcff08544c7d580a -\
 - -] Filter RamFilter returned 2 host(s) get_filtered_objects
/usr/lib/python2.7/site-packages/nova/filters.py:104
2018-05-29 11:09:54.334 19428 DEBUG nova.filters
[req-1a838e77-8042-4550-b157-4943445119a2 ab573ba3ea014b778193b6922ffffe6d
ee1865a76440481cbcff08544c7d580a -\
 - -] Filter CoreFilter returned 2 host(s) get_filtered_objects
/usr/lib/python2.7/site-packages/nova/filters.py:104
2018-05-29 11:09:54.334 19428 DEBUG nova.filters
[req-1a838e77-8042-4550-b157-4943445119a2 ab573ba3ea014b778193b6922ffffe6d
ee1865a76440481cbcff08544c7d580a -\
 - -] Filter AggregateRamFilter returned 2 host(s) get_filtered_objects
/usr/lib/python2.7/site-packages/nova/filters.py:104
2018-05-29 11:09:54.335 19428 DEBUG nova.filters
[req-1a838e77-8042-4550-b157-4943445119a2 ab573ba3ea014b778193b6922ffffe6d
ee1865a76440481cbcff08544c7d580a -\
 - -] Filter AggregateCoreFilter returned 2 host(s) get_filtered_objects
/usr/lib/python2.7/site-packages/nova/filters.py:104
2018-05-29 11:09:54.335 19428 DEBUG nova.filters
[req-1a838e77-8042-4550-b157-4943445119a2 ab573ba3ea014b778193b6922ffffe6d
ee1865a76440481cbcff08544c7d580a -\
 - -] Filter DiskFilter returned 2 host(s) get_filtered_objects
/usr/lib/python2.7/site-packages/nova/filters.py:104
2018-05-29 11:09:54.336 19428 DEBUG nova.filters
[req-1a838e77-8042-4550-b157-4943445119a2 ab573ba3ea014b778193b6922ffffe6d
ee1865a76440481cbcff08544c7d580a -\
 - -] Filter ComputeFilter returned 2 host(s) get_filtered_objects
/usr/lib/python2.7/site-packages/nova/filters.py:104
2018-05-29 11:09:54.337 19428 DEBUG nova.filters
[req-1a838e77-8042-4550-b157-4943445119a2 ab573ba3ea014b778193b6922ffffe6d
ee1865a76440481cbcff08544c7d580a -\
 - -] Filter ComputeCapabilitiesFilter returned 2 host(s)
get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:104
2018-05-29 11:09:54.338 19428 DEBUG nova.filters
[req-1a838e77-8042-4550-b157-4943445119a2 ab573ba3ea014b778193b6922ffffe6d
ee1865a76440481cbcff08544c7d580a -\
 - -] Filter ImagePropertiesFilter returned 2 host(s) get_filtered_objects
/usr/lib/python2.7/site-packages/nova/filters.py:104
2018-05-29 11:09:54.339 19428 DEBUG nova.filters
[req-1a838e77-8042-4550-b157-4943445119a2 ab573ba3ea014b778193b6922ffffe6d
ee1865a76440481cbcff08544c7d580a -\
 - -] Filter ServerGroupAntiAffinityFilter returned 2 host(s)
get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:104
2018-05-29 11:09:54.339 19428 DEBUG nova.filters
[req-1a838e77-8042-4550-b157-4943445119a2 ab573ba3ea014b778193b6922ffffe6d
ee1865a76440481cbcff08544c7d580a -\
 - -] Filter ServerGroupAffinityFilter returned 2 host(s)
get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:104
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20180529/bfb85979/attachment.html>