Hi one and all!
Currently, we have an ussuri openstack installation via kolla-ansible.
The installation consists of three control nodes and two availability
zones (az-1, az-2), each with three compute nodes:
- os-cpt-10[1-3] > az-1
- os-cpt-20[1-3] > az-2
az-1 is the default compute availability zone.
The compute nodes are hardwarewise equipped exactly the same.
As storage system a ceph cluster is used.
We wanted to also use Octavia LBaaS and got it up and running after some
experimentation.
We also wanted to be able to choose the availibility zones when starting
a loadbalancer. We added the az info with:
openstack --os-cloud service_octavia loadbalancer
availabilityzoneprofile create --name az-1 --provider amphora
--availability-zone-data '{"compute_zone": "az-1"}'
openstack --os-cloud service_octavia loadbalancer
availabilityzoneprofile create --name az-2 --provider amphora
--availability-zone-data '{"compute_zone": "az-2"}'
openstack --os-cloud service_octavia loadbalancer availabilityzone
create --name az-1 --availabilityzoneprofile az-1
openstack --os-cloud service_octavia loadbalancer availabilityzone
create --name az-2 --availabilityzoneprofile az-2
Creating a loadbalancer via cli:
openstack --os-cloud $LB_PROJECT loadbalancer create --flavor $LB_FLAVOR
--name $LB_NAME --vip-subnet-id $(openstack --os-cloud $LB_PROJECT
subnet list --name $LB_SUBNET -f value -c ID) --availability-zone $LB_AZ
works, as long as $LB_AZ == az-1.
If we want to start the loadbalancer in az-2, this fails with an error
in octavia-worker.log:
...
2020-09-30 07:21:01.482 34 ERROR oslo_messaging.rpc.server
octavia.common.exceptions.ComputeBuildException: Failed to build compute
instance due to: {'code': 500, 'created': '2020-09-30T07:20:55Z',
'message': 'No valid host was found. There are not enough hosts
available.', 'details': 'Traceback (most recent call last):\n File
"/usr/lib/python3.6/site-packages/nova/conductor/manager.py", line 1463,
in schedule_and_build_instances\n instance_uuids,
return_alternates=True)\n File
"/usr/lib/python3.6/site-packages/nova/conductor/manager.py", line 870,
in _schedule_instances\n return_alternates=return_alternates)\n File
"/usr/lib/python3.6/site-packages/nova/scheduler/client/query.py", line
42, in select_destinations\n instance_uuids, return_objects,
return_alternates)\n File
"/usr/lib/python3.6/site-packages/nova/scheduler/rpcapi.py", line 160,
in select_destinations\n return cctxt.call(ctxt,
\'select_destinations\', **msg_args)\n File
"/usr/lib/python3.6/site-packages/oslo_messaging/rpc/client.py", line
181, in call\n transport_options=self.transport_options)\n File
"/usr/lib/python3.6/site-packages/oslo_messaging/transport.py", line
129, in _send\n transport_options=transport_options)\n File
"/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 654, in send\n transport_options=transport_options)\n File
"/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 644, in _send\n raise
result\nnova.exception_Remote.NoValidHost_Remote: No valid host was
found. There are not enough hosts available.\nTraceback (most recent
call last):\n\n File
"/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line
241, in inner\n return func(*args, **kwargs)\n\n File
"/usr/lib/python3.6/site-packages/nova/scheduler/manager.py", line 215,
in select_destinations\n allocation_request_version,
return_alternates)\n\n File
"/usr/lib/python3.6/site-packages/nova/scheduler/filter_scheduler.py",
line 96, in select_destinations\n allocation_request_version,
return_alternates)\n\n File
"/usr/lib/python3.6/site-packages/nova/scheduler/filter_scheduler.py",
line 265, in _schedule\n claimed_instance_uuids)\n\n File
"/usr/lib/python3.6/site-packages/nova/scheduler/filter_scheduler.py",
line 302, in _ensure_sufficient_hosts\n raise
exception.NoValidHost(reason=reason)\n\nnova.exception.NoValidHost: No
valid host was found. There are not enough hosts available.\n\n'}
...
We enabled debug logging for nova and found:
...
2020-10-13 13:50:49.193 33 DEBUG oslo_concurrency.lockutils
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
default default] Lock "7b9846a9-f8be-4d60-92ad-0fb531e48e64" acquired by
"nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections"
:: waited 0.000s inner
/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:358
2020-10-13 13:50:49.196 33 DEBUG oslo_concurrency.lockutils
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
default default] Lock "7b9846a9-f8be-4d60-92ad-0fb531e48e64" released by
"nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections"
:: held 0.003s inner
/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:370
2020-10-13 13:50:49.217 33 DEBUG oslo_db.sqlalchemy.engines
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
default default] MySQL server mode set to
STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
_check_effective_sql_mode
/usr/lib/python3.6/site-packages/oslo_db/sqlalchemy/engines.py:304
2020-10-13 13:50:49.245 33 DEBUG oslo_concurrency.lockutils
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
default default] Lock "('os-cpt-103', 'os-cpt-103')" acquired by
"nova.scheduler.host_manager.HostState.update.<locals>._locked_update"
:: waited 0.000s inner
/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:358
2020-10-13 13:50:49.246 33 DEBUG nova.scheduler.host_manager
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
default default] Update host state from compute node:
ComputeNode(cpu_allocation_ratio=4.0,cpu_info='{"arch": "x86_64",
"model": "EPYC-IBPB", "vendor": "AMD", "topology": {"cells": 1,
"sockets": 1, "cores": 24, "threads": 2}, "features": ["bmi1", "smep",
"sha-ni", "fsgsbase", "adx", "cmov", "invtsc", "fpu", "mmx", "sep",
"abm", "pni", "msr", "xsavec", "f16c", "fma", "nx", "pat", "sse4.1",
"rdrand", "wbnoinvd", "vme", "lahf_lm", "cr8legacy", "xsave", "bmi2",
"clzero", "mtrr", "arat", "amd-ssbd", "aes", "avx", "avx2", "cx8",
"umip", "de", "ibpb", "misalignsse", "osvw", "perfctr_core", "pse36",
"mce", "skinit", "syscall", "sse2", "apic", "fxsr_opt", "pse", "ht",
"pdpe1gb", "rdtscp", "xsaves", "clflushopt", "pclmuldq", "sse4.2",
"movbe", "smap", "ibs", "clwb", "xgetbv1", "sse4a", "cx16", "extapic",
"wdt", "perfctr_nb", "tsc", "mca", "topoext", "pae", "fxsr", "lm",
"cmp_legacy", "monitor", "3dnowprefetch", "ssse3", "pge", "popcnt",
"rdseed", "mmxext", "tce", "clflush", "xsaveopt", "svm",
"sse"]}',created_at=2020-09-08T09:22:00Z,current_workload=0,deleted=False,deleted_at=None,disk_allocation_ratio=1.0,disk_available_least=94,free_disk_gb=99,free_ram_mb=206453,host='os-cpt-103',host_ip=172.20.1.13,hypervisor_hostname='os-cpt-103',hypervisor_type='QEMU',hypervisor_version=4002000,id=9,local_gb=99,local_gb_used=0,mapped=0,memory_mb=257653,memory_mb_used=51200,metrics='[{"name":
"cpu.user.percent", "timestamp": "2020-10-13T13:50:07.846401", "source":
"libvirt.LibvirtDriver", "value": 0.0}, {"name": "cpu.kernel.percent",
"timestamp": "2020-10-13T13:50:07.846401", "source":
"libvirt.LibvirtDriver", "value": 0.01}, {"name": "cpu.iowait.percent",
"timestamp": "2020-10-13T13:50:07.846401", "source":
"libvirt.LibvirtDriver", "value": 0.0}, {"name": "cpu.kernel.time",
"timestamp": "2020-10-13T13:50:07.846401", "source":
"libvirt.LibvirtDriver", "value": 1373295770000000}, {"name":
"cpu.percent", "timestamp": "2020-10-13T13:50:07.846401", "source":
"libvirt.LibvirtDriver", "value": 0.01}, {"name": "cpu.frequency",
"timestamp": "2020-10-13T13:50:07.846401", "source":
"libvirt.LibvirtDriver", "value": 1796}, {"name": "cpu.user.time",
"timestamp": "2020-10-13T13:50:07.846401", "source":
"libvirt.LibvirtDriver", "value": 386827280000000}, {"name":
"cpu.idle.time", "timestamp": "2020-10-13T13:50:07.846401", "source":
"libvirt.LibvirtDriver", "value": 139696145240000000}, {"name":
"cpu.idle.percent", "timestamp": "2020-10-13T13:50:07.846401", "source":
"libvirt.LibvirtDriver", "value": 0.98}, {"name": "cpu.iowait.time",
"timestamp": "2020-10-13T13:50:07.846401", "source":
"libvirt.LibvirtDriver", "value":
3390650000000}]',numa_topology='{"nova_object.name": "NUMATopology",
"nova_object.namespace": "nova", "nova_object.version": "1.2",
"nova_object.data": {"cells": [{"nova_object.name": "NUMACell",
"nova_object.namespace": "nova", "nova_object.version": "1.4",
"nova_object.data": {"id": 0, "cpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9,
10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27,
28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45,
46, 47], "pcpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32,
33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "memory":
257653, "cpu_usage": 0, "memory_usage": 0, "pinned_cpus": [],
"siblings": [[12, 36], [10, 34], [2, 26], [1, 25], [9, 33], [0, 24], [8,
32], [11, 35], [13, 37], [19, 43], [21, 45], [18, 42], [38, 14], [46,
22], [4, 28], [20, 44], [15, 39], [17, 41], [23, 47], [16, 40], [6, 30],
[7, 31], [3, 27], [5, 29]], "mempages": [{"nova_object.name":
"NUMAPagesTopology", "nova_object.namespace": "nova",
"nova_object.version": "1.1", "nova_object.data": {"size_kb": 4,
"total": 65959381, "used": 0, "reserved": 0}, "nova_object.changes":
["used", "reserved", "size_kb", "total"]}, {"nova_object.name":
"NUMAPagesTopology", "nova_object.namespace": "nova",
"nova_object.version": "1.1", "nova_object.data": {"size_kb": 2048,
"total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["used",
"reserved", "size_kb", "total"]}, {"nova_object.name":
"NUMAPagesTopology", "nova_object.namespace": "nova",
"nova_object.version": "1.1", "nova_object.data": {"size_kb": 1048576,
"total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["used",
"reserved", "size_kb", "total"]}], "network_metadata":
{"nova_object.name": "NetworkMetadata", "nova_object.namespace": "nova",
"nova_object.version": "1.0", "nova_object.data": {"physnets": [],
"tunneled": false}, "nova_object.changes": ["physnets", "tunneled"]}},
"nova_object.changes": ["mempages", "cpu_usage", "memory",
"memory_usage", "id", "pinned_cpus", "pcpuset", "network_metadata",
"siblings", "cpuset"]}]}, "nova_object.changes":
["cells"]}',pci_device_pools=PciDevicePoolList,ram_allocation_ratio=1.2,running_vms=3,service_id=None,stats={failed_builds='0',io_workload='0',num_instances='3',num_os_type_None='3',num_proj_e06101e290f24563a836e16909146ea0='3',num_task_None='3',num_vm_active='3'},supported_hv_specs=[HVSpec,HVSpec,HVSpec,HVSpec],updated_at=2020-10-13T13:50:07Z,uuid=34993097-d180-49a0-ae3c-9111e6aa8968,vcpus=48,vcpus_used=7)
_locked_update
/usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:172
2020-10-13 13:50:49.250 33 DEBUG nova.scheduler.host_manager
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
default default] Update host state with aggregates:
[Aggregate(created_at=2020-09-08T14:01:05Z,deleted=False,deleted_at=None,hosts=['os-cpt-101','os-cpt-102','os-cpt-103'],id=3,metadata={availability_zone='az-1'},name='az-1',updated_at=None,uuid=8f37e9f3-506d-40da-8997-38915f1dfe67)]
_locked_update
/usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:175
2020-10-13 13:50:49.251 33 DEBUG nova.scheduler.host_manager
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
default default] Update host state with service dict: {'id': 27, 'uuid':
'62ea00b1-e61d-4f84-8b35-c1d3d7a333e7', 'host': 'os-cpt-103', 'binary':
'nova-compute', 'topic': 'compute', 'report_count': 294508, 'disabled':
False, 'disabled_reason': None, 'last_seen_up': datetime.datetime(2020,
10, 13, 13, 50, 45, tzinfo=<iso8601.Utc>), 'forced_down': False,
'version': 51, 'created_at': datetime.datetime(2020, 9, 8, 9, 22,
tzinfo=<iso8601.Utc>), 'updated_at': datetime.datetime(2020, 10, 13, 13,
50, 45, tzinfo=<iso8601.Utc>), 'deleted_at': None, 'deleted': False}
_locked_update
/usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:178
2020-10-13 13:50:49.251 33 DEBUG nova.scheduler.host_manager
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
default default] Update host state with instances:
['c88b77f7-2c59-497b-9633-139e62ba2bb1',
'635fbd95-dd64-4e0a-9767-a0e0a6958610',
'a02369ab-9002-4bf8-b65c-03e93cbc7df5'] _locked_update
/usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:182
2020-10-13 13:50:49.252 33 DEBUG oslo_concurrency.lockutils
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
default default] Lock "('os-cpt-103', 'os-cpt-103')" released by
"nova.scheduler.host_manager.HostState.update.<locals>._locked_update"
:: held 0.007s inner
/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:370
2020-10-13 13:50:49.253 33 DEBUG oslo_concurrency.lockutils
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
default default] Lock "('os-cpt-101', 'os-cpt-101')" acquired by
"nova.scheduler.host_manager.HostState.update.<locals>._locked_update"
:: waited 0.000s inner
/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:358
2020-10-13 13:50:49.253 33 DEBUG nova.scheduler.host_manager
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
default default] Update host state from compute node:
ComputeNode(cpu_allocation_ratio=4.0,cpu_info='{"arch": "x86_64",
"model": "EPYC-IBPB", "vendor": "AMD", "topology": {"cells": 1,
"sockets": 1, "cores": 24, "threads": 2}, "features": ["3dnowprefetch",
"fpu", "sse4.1", "fma", "arat", "avx2", "bmi1", "sse2", "extapic",
"adx", "aes", "sha-ni", "tce", "osvw", "invtsc", "sse", "xsave", "de",
"movbe", "pse", "pdpe1gb", "clflush", "mmx", "wdt", "cmov",
"perfctr_core", "skinit", "umip", "bmi2", "cx8", "amd-ssbd",
"cmp_legacy", "perfctr_nb", "rdrand", "ibpb", "monitor", "mtrr",
"clflushopt", "smap", "msr", "sep", "f16c", "pat", "avx", "xsavec",
"mca", "apic", "pni", "xsaves", "cr8legacy", "popcnt", "svm", "clzero",
"pae", "lm", "pclmuldq", "pge", "rdseed", "xgetbv1", "sse4.2", "ht",
"rdtscp", "fxsr", "lahf_lm", "vme", "sse4a", "tsc", "misalignsse",
"abm", "fxsr_opt", "mce", "syscall", "ssse3", "cx16", "ibs", "smep",
"fsgsbase", "topoext", "wbnoinvd", "xsaveopt", "mmxext", "nx", "pse36",
"clwb"]}',created_at=2020-09-08T09:22:00Z,current_workload=0,deleted=False,deleted_at=None,disk_allocation_ratio=1.0,disk_available_least=64,free_disk_gb=79,free_ram_mb=207477,host='os-cpt-101',host_ip=172.20.1.11,hypervisor_hostname='os-cpt-101',hypervisor_type='QEMU',hypervisor_version=4002000,id=6,local_gb=99,local_gb_used=20,mapped=0,memory_mb=257653,memory_mb_used=50176,metrics='[{"name":
"cpu.iowait.percent", "timestamp": "2020-10-13T13:50:21.645565",
"source": "libvirt.LibvirtDriver", "value": 0.0}, {"name":
"cpu.percent", "timestamp": "2020-10-13T13:50:21.645565", "source":
"libvirt.LibvirtDriver", "value": 0.01}, {"name": "cpu.idle.percent",
"timestamp": "2020-10-13T13:50:21.645565", "source":
"libvirt.LibvirtDriver", "value": 0.98}, {"name": "cpu.frequency",
"timestamp": "2020-10-13T13:50:21.645565", "source":
"libvirt.LibvirtDriver", "value": 1796}, {"name": "cpu.idle.time",
"timestamp": "2020-10-13T13:50:21.645565", "source":
"libvirt.LibvirtDriver", "value": 139958595980000000}, {"name":
"cpu.user.percent", "timestamp": "2020-10-13T13:50:21.645565", "source":
"libvirt.LibvirtDriver", "value": 0.0}, {"name": "cpu.kernel.percent",
"timestamp": "2020-10-13T13:50:21.645565", "source":
"libvirt.LibvirtDriver", "value": 0.01}, {"name": "cpu.user.time",
"timestamp": "2020-10-13T13:50:21.645565", "source":
"libvirt.LibvirtDriver", "value": 388866090000000}, {"name":
"cpu.kernel.time", "timestamp": "2020-10-13T13:50:21.645565", "source":
"libvirt.LibvirtDriver", "value": 1118743070000000}, {"name":
"cpu.iowait.time", "timestamp": "2020-10-13T13:50:21.645565", "source":
"libvirt.LibvirtDriver", "value":
3637460000000}]',numa_topology='{"nova_object.name": "NUMATopology",
"nova_object.namespace": "nova", "nova_object.version": "1.2",
"nova_object.data": {"cells": [{"nova_object.name": "NUMACell",
"nova_object.namespace": "nova", "nova_object.version": "1.4",
"nova_object.data": {"id": 0, "cpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9,
10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27,
28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45,
46, 47], "pcpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32,
33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "memory":
257653, "cpu_usage": 0, "memory_usage": 0, "pinned_cpus": [],
"siblings": [[12, 36], [10, 34], [2, 26], [1, 25], [9, 33], [0, 24], [8,
32], [11, 35], [13, 37], [19, 43], [21, 45], [18, 42], [38, 14], [46,
22], [4, 28], [20, 44], [15, 39], [17, 41], [23, 47], [16, 40], [6, 30],
[7, 31], [3, 27], [5, 29]], "mempages": [{"nova_object.name":
"NUMAPagesTopology", "nova_object.namespace": "nova",
"nova_object.version": "1.1", "nova_object.data": {"size_kb": 4,
"total": 65959383, "used": 0, "reserved": 0}, "nova_object.changes":
["reserved", "used", "total", "size_kb"]}, {"nova_object.name":
"NUMAPagesTopology", "nova_object.namespace": "nova",
"nova_object.version": "1.1", "nova_object.data": {"size_kb": 2048,
"total": 0, "used": 0, "reserved": 0}, "nova_object.changes":
["reserved", "used", "total", "size_kb"]}, {"nova_object.name":
"NUMAPagesTopology", "nova_object.namespace": "nova",
"nova_object.version": "1.1", "nova_object.data": {"size_kb": 1048576,
"total": 0, "used": 0, "reserved": 0}, "nova_object.changes":
["reserved", "used", "total", "size_kb"]}], "network_metadata":
{"nova_object.name": "NetworkMetadata", "nova_object.namespace": "nova",
"nova_object.version": "1.0", "nova_object.data": {"physnets": [],
"tunneled": false}, "nova_object.changes": ["physnets", "tunneled"]}},
"nova_object.changes": ["id", "cpu_usage", "pcpuset", "memory",
"cpuset", "siblings", "mempages", "network_metadata", "memory_usage",
"pinned_cpus"]}]}, "nova_object.changes":
["cells"]}',pci_device_pools=PciDevicePoolList,ram_allocation_ratio=1.2,running_vms=4,service_id=None,stats={failed_builds='0',io_workload='0',num_instances='4',num_os_type_None='4',num_proj_05af2e78169748f69938d01b5238fc8b='1',num_proj_882ea7b2a43a41819ef796f797cdbe82='1',num_proj_d52e2dccbc7e46d8b518120fa7c8753a='1',num_proj_e06101e290f24563a836e16909146ea0='1',num_task_None='4',num_vm_active='4'},supported_hv_specs=[HVSpec,HVSpec,HVSpec,HVSpec],updated_at=2020-10-13T13:50:21Z,uuid=5ab97e91-6b03-48f3-a320-c8cbb032cd3a,vcpus=48,vcpus_used=7)
_locked_update
/usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:172
2020-10-13 13:50:49.257 33 DEBUG nova.scheduler.host_manager
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
default default] Update host state with aggregates:
[Aggregate(created_at=2020-09-08T14:01:05Z,deleted=False,deleted_at=None,hosts=['os-cpt-101','os-cpt-102','os-cpt-103'],id=3,metadata={availability_zone='az-1'},name='az-1',updated_at=None,uuid=8f37e9f3-506d-40da-8997-38915f1dfe67)]
_locked_update
/usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:175
2020-10-13 13:50:49.257 33 DEBUG nova.scheduler.host_manager
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
default default] Update host state with service dict: {'id': 21, 'uuid':
'c178b502-9b8c-43f3-9053-e3d0e2405c89', 'host': 'os-cpt-101', 'binary':
'nova-compute', 'topic': 'compute', 'report_count': 294504, 'disabled':
False, 'disabled_reason': None, 'last_seen_up': datetime.datetime(2020,
10, 13, 13, 50, 43, tzinfo=<iso8601.Utc>), 'forced_down': False,
'version': 51, 'created_at': datetime.datetime(2020, 9, 8, 9, 22,
tzinfo=<iso8601.Utc>), 'updated_at': datetime.datetime(2020, 10, 13, 13,
50, 43, tzinfo=<iso8601.Utc>), 'deleted_at': None, 'deleted': False}
_locked_update
/usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:178
2020-10-13 13:50:49.258 33 DEBUG nova.scheduler.host_manager
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
default default] Update host state with instances:
['96f4c101-d75b-44b6-8af9-dd75068d207d',
'121c5df5-2d8b-48e0-a270-9f6f79193dd6',
'e235063d-f7a8-4f38-8294-d7a289167348',
'425b53c3-f6f5-43e9-beeb-25c9b6ead1f9'] _locked_update
/usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:182
2020-10-13 13:50:49.258 33 DEBUG oslo_concurrency.lockutils
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
default default] Lock "('os-cpt-101', 'os-cpt-101')" released by
"nova.scheduler.host_manager.HostState.update.<locals>._locked_update"
:: held 0.005s inner
/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:370
2020-10-13 13:50:49.259 33 DEBUG oslo_concurrency.lockutils
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
default default] Lock "('os-cpt-102', 'os-cpt-102')" acquired by
"nova.scheduler.host_manager.HostState.update.<locals>._locked_update"
:: waited 0.000s inner
/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:358
2020-10-13 13:50:49.260 33 DEBUG nova.scheduler.host_manager
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
default default] Update host state from compute node:
ComputeNode(cpu_allocation_ratio=4.0,cpu_info='{"arch": "x86_64",
"model": "EPYC-IBPB", "vendor": "AMD", "topology": {"cells": 1,
"sockets": 1, "cores": 24, "threads": 2}, "features": ["lahf_lm",
"clflush", "osvw", "extapic", "vme", "wdt", "monitor", "msr", "adx",
"pse36", "sse4a", "fma", "pat", "mce", "sse2", "nx", "f16c", "mca",
"xsaveopt", "avx", "syscall", "rdrand", "clwb", "ssse3", "xsavec",
"invtsc", "bmi2", "fpu", "movbe", "aes", "bmi1", "cr8legacy", "mmxext",
"ibpb", "amd-ssbd", "skinit", "topoext", "umip", "cmp_legacy", "arat",
"lm", "svm", "fxsr", "ibs", "pae", "misalignsse", "mtrr", "sep", "ht",
"smap", "xgetbv1", "clzero", "pdpe1gb", "apic", "abm", "pge", "pni",
"tsc", "xsaves", "wbnoinvd", "sse4.2", "cmov", "cx8", "pse", "rdtscp",
"cx16", "sse", "sse4.1", "fxsr_opt", "popcnt", "sha-ni", "perfctr_core",
"fsgsbase", "avx2", "mmx", "rdseed", "clflushopt", "pclmuldq",
"perfctr_nb", "smep", "de", "3dnowprefetch", "xsave",
"tce"]}',created_at=2020-09-08T09:21:58Z,current_workload=0,deleted=False,deleted_at=None,disk_allocation_ratio=1.0,disk_available_least=93,free_disk_gb=99,free_ram_mb=216693,host='os-cpt-102',host_ip=172.20.1.12,hypervisor_hostname='os-cpt-102',hypervisor_type='QEMU',hypervisor_version=4002000,id=3,local_gb=99,local_gb_used=0,mapped=0,memory_mb=257653,memory_mb_used=40960,metrics='[{"name":
"cpu.user.time", "timestamp": "2020-10-13T13:50:03.450583", "source":
"libvirt.LibvirtDriver", "value": 161246250000000}, {"name":
"cpu.kernel.percent", "timestamp": "2020-10-13T13:50:03.450583",
"source": "libvirt.LibvirtDriver", "value": 0.0}, {"name":
"cpu.user.percent", "timestamp": "2020-10-13T13:50:03.450583", "source":
"libvirt.LibvirtDriver", "value": 0.0}, {"name": "cpu.idle.percent",
"timestamp": "2020-10-13T13:50:03.450583", "source":
"libvirt.LibvirtDriver", "value": 0.99}, {"name": "cpu.percent",
"timestamp": "2020-10-13T13:50:03.450583", "source":
"libvirt.LibvirtDriver", "value": 0.0}, {"name": "cpu.idle.time",
"timestamp": "2020-10-13T13:50:03.450583", "source":
"libvirt.LibvirtDriver", "value": 140551238730000000}, {"name":
"cpu.frequency", "timestamp": "2020-10-13T13:50:03.450583", "source":
"libvirt.LibvirtDriver", "value": 3343}, {"name": "cpu.kernel.time",
"timestamp": "2020-10-13T13:50:03.450583", "source":
"libvirt.LibvirtDriver", "value": 366274340000000}, {"name":
"cpu.iowait.time", "timestamp": "2020-10-13T13:50:03.450583", "source":
"libvirt.LibvirtDriver", "value": 3829690000000}, {"name":
"cpu.iowait.percent", "timestamp": "2020-10-13T13:50:03.450583",
"source": "libvirt.LibvirtDriver", "value":
0.0}]',numa_topology='{"nova_object.name": "NUMATopology",
"nova_object.namespace": "nova", "nova_object.version": "1.2",
"nova_object.data": {"cells": [{"nova_object.name": "NUMACell",
"nova_object.namespace": "nova", "nova_object.version": "1.4",
"nova_object.data": {"id": 0, "cpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9,
10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27,
28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45,
46, 47], "pcpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32,
33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "memory":
257653, "cpu_usage": 0, "memory_usage": 0, "pinned_cpus": [],
"siblings": [[12, 36], [10, 34], [2, 26], [1, 25], [9, 33], [0, 24], [8,
32], [11, 35], [13, 37], [19, 43], [21, 45], [18, 42], [38, 14], [46,
22], [4, 28], [20, 44], [15, 39], [17, 41], [23, 47], [16, 40], [6, 30],
[7, 31], [3, 27], [5, 29]], "mempages": [{"nova_object.name":
"NUMAPagesTopology", "nova_object.namespace": "nova",
"nova_object.version": "1.1", "nova_object.data": {"size_kb": 4,
"total": 65959382, "used": 0, "reserved": 0}, "nova_object.changes":
["total", "used", "reserved", "size_kb"]}, {"nova_object.name":
"NUMAPagesTopology", "nova_object.namespace": "nova",
"nova_object.version": "1.1", "nova_object.data": {"size_kb": 2048,
"total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["total",
"used", "reserved", "size_kb"]}, {"nova_object.name":
"NUMAPagesTopology", "nova_object.namespace": "nova",
"nova_object.version": "1.1", "nova_object.data": {"size_kb": 1048576,
"total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["total",
"used", "reserved", "size_kb"]}], "network_metadata":
{"nova_object.name": "NetworkMetadata", "nova_object.namespace": "nova",
"nova_object.version": "1.0", "nova_object.data": {"physnets": [],
"tunneled": false}, "nova_object.changes": ["physnets", "tunneled"]}},
"nova_object.changes": ["cpu_usage", "pinned_cpus", "mempages",
"memory", "siblings", "network_metadata", "cpuset", "memory_usage",
"id", "pcpuset"]}]}, "nova_object.changes":
["cells"]}',pci_device_pools=PciDevicePoolList,ram_allocation_ratio=1.2,running_vms=0,service_id=None,stats={failed_builds='0'},supported_hv_specs=[HVSpec,HVSpec,HVSpec,HVSpec],updated_at=2020-10-13T13:50:03Z,uuid=81c2e888-56e4-4469-ab93-f204fa85a5c5,vcpus=48,vcpus_used=2)
_locked_update
/usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:172
2020-10-13 13:50:49.263 33 DEBUG nova.scheduler.host_manager
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
default default] Update host state with aggregates:
[Aggregate(created_at=2020-09-08T14:01:05Z,deleted=False,deleted_at=None,hosts=['os-cpt-101','os-cpt-102','os-cpt-103'],id=3,metadata={availability_zone='az-1'},name='az-1',updated_at=None,uuid=8f37e9f3-506d-40da-8997-38915f1dfe67)]
_locked_update
/usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:175
2020-10-13 13:50:49.264 33 DEBUG nova.scheduler.host_manager
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
default default] Update host state with service dict: {'id': 18, 'uuid':
'342a2d9b-4b32-45a2-ba16-b808343b3b8f', 'host': 'os-cpt-102', 'binary':
'nova-compute', 'topic': 'compute', 'report_count': 294494, 'disabled':
False, 'disabled_reason': None, 'last_seen_up': datetime.datetime(2020,
10, 13, 13, 50, 49, tzinfo=<iso8601.Utc>), 'forced_down': False,
'version': 51, 'created_at': datetime.datetime(2020, 9, 8, 9, 21, 58,
tzinfo=<iso8601.Utc>), 'updated_at': datetime.datetime(2020, 10, 13, 13,
50, 49, tzinfo=<iso8601.Utc>), 'deleted_at': None, 'deleted': False}
_locked_update
/usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:178
2020-10-13 13:50:49.264 33 DEBUG nova.scheduler.host_manager
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
default default] Update host state with instances: [] _locked_update
/usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:182
2020-10-13 13:50:49.265 33 DEBUG oslo_concurrency.lockutils
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
default default] Lock "('os-cpt-102', 'os-cpt-102')" released by
"nova.scheduler.host_manager.HostState.update.<locals>._locked_update"
:: held 0.005s inner
/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:370
2020-10-13 13:50:49.266 33 DEBUG nova.filters
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
default default] Starting with 3 host(s) get_filtered_objects
/usr/lib/python3.6/site-packages/nova/filters.py:70
2020-10-13 13:50:49.266 33 DEBUG
nova.scheduler.filters.availability_zone_filter
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
default default] Availability Zone 'az-2' requested. (os-cpt-101,
os-cpt-101) ram: 207477MB disk: 65536MB io_ops: 0 instances: 4 has AZs:
{'az-1'} host_passes
/usr/lib/python3.6/site-packages/nova/scheduler/filters/availability_zone_filter.py:61
2020-10-13 13:50:49.267 33 DEBUG
nova.scheduler.filters.availability_zone_filter
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
default default] Availability Zone 'az-2' requested. (os-cpt-102,
os-cpt-102) ram: 216693MB disk: 95232MB io_ops: 0 instances: 0 has AZs:
{'az-1'} host_passes
/usr/lib/python3.6/site-packages/nova/scheduler/filters/availability_zone_filter.py:61
2020-10-13 13:50:49.267 33 DEBUG
nova.scheduler.filters.availability_zone_filter
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
default default] Availability Zone 'az-2' requested. (os-cpt-103,
os-cpt-103) ram: 206453MB disk: 96256MB io_ops: 0 instances: 3 has AZs:
{'az-1'} host_passes
/usr/lib/python3.6/site-packages/nova/scheduler/filters/availability_zone_filter.py:61
2020-10-13 13:50:49.268 33 INFO nova.filters
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
default default] Filter AvailabilityZoneFilter returned 0 hosts
2020-10-13 13:50:49.269 33 DEBUG nova.filters
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
default default] Filtering removed all hosts for the request with
instance ID 'd18bf174-9ccb-4721-b7d8-36cd31834f62'. Filter results:
[('AvailabilityZoneFilter', None)] get_filtered_objects
/usr/lib/python3.6/site-packages/nova/filters.py:115
2020-10-13 13:50:49.269 33 INFO nova.filters
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
default default] Filtering removed all hosts for the request with
instance ID 'd18bf174-9ccb-4721-b7d8-36cd31834f62'. Filter results:
['AvailabilityZoneFilter: (start: 3, end: 0)']
2020-10-13 13:50:49.270 33 DEBUG nova.scheduler.filter_scheduler
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
default default] Filtered [] _get_sorted_hosts
/usr/lib/python3.6/site-packages/nova/scheduler/filter_scheduler.py:443
2020-10-13 13:50:49.270 33 DEBUG nova.scheduler.filter_scheduler
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
default default] There are 0 hosts available but 1 instances requested
to build. _ensure_sufficient_hosts
/usr/lib/python3.6/site-packages/nova/scheduler/filter_scheduler.py:300
...
So nova seems to only consider compute nodes in az-1.
The amphora image is available in both availability zones. Starting an
instance from the image directly in az-2 works.
Does anybody have any idea on how to debug this any further? Any hints,
tips or ideas would be greatly appreciated!
With kind regards,
Björn Puttmann