Hi one and all!
Currently, we have an ussuri openstack installation via kolla-ansible. The installation consists of three control nodes and two availability zones (az-1, az-2), each with three compute nodes: - os-cpt-10[1-3] > az-1 - os-cpt-20[1-3] > az-2
az-1 is the default compute availability zone.
The compute nodes are hardwarewise equipped exactly the same. As storage system a ceph cluster is used.
We wanted to also use Octavia LBaaS and got it up and running after some experimentation. We also wanted to be able to choose the availibility zones when starting a loadbalancer. We added the az info with:
openstack --os-cloud service_octavia loadbalancer availabilityzoneprofile create --name az-1 --provider amphora --availability-zone-data '{"compute_zone": "az-1"}' openstack --os-cloud service_octavia loadbalancer availabilityzoneprofile create --name az-2 --provider amphora --availability-zone-data '{"compute_zone": "az-2"}' openstack --os-cloud service_octavia loadbalancer availabilityzone create --name az-1 --availabilityzoneprofile az-1 openstack --os-cloud service_octavia loadbalancer availabilityzone create --name az-2 --availabilityzoneprofile az-2
Creating a loadbalancer via cli:
openstack --os-cloud $LB_PROJECT loadbalancer create --flavor $LB_FLAVOR --name $LB_NAME --vip-subnet-id $(openstack --os-cloud $LB_PROJECT subnet list --name $LB_SUBNET -f value -c ID) --availability-zone $LB_AZ
works, as long as $LB_AZ == az-1.
If we want to start the loadbalancer in az-2, this fails with an error in octavia-worker.log: ... 2020-09-30 07:21:01.482 34 ERROR oslo_messaging.rpc.server octavia.common.exceptions.ComputeBuildException: Failed to build compute instance due to: {'code': 500, 'created': '2020-09-30T07:20:55Z', 'message': 'No valid host was found. There are not enough hosts available.', 'details': 'Traceback (most recent call last):\n File "/usr/lib/python3.6/site-packages/nova/conductor/manager.py", line 1463, in schedule_and_build_instances\n instance_uuids, return_alternates=True)\n File "/usr/lib/python3.6/site-packages/nova/conductor/manager.py", line 870, in _schedule_instances\n return_alternates=return_alternates)\n File "/usr/lib/python3.6/site-packages/nova/scheduler/client/query.py", line 42, in select_destinations\n instance_uuids, return_objects, return_alternates)\n File "/usr/lib/python3.6/site-packages/nova/scheduler/rpcapi.py", line 160, in select_destinations\n return cctxt.call(ctxt, 'select_destinations', **msg_args)\n File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/client.py", line 181, in call\n transport_options=self.transport_options)\n File "/usr/lib/python3.6/site-packages/oslo_messaging/transport.py", line 129, in _send\n transport_options=transport_options)\n File "/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 654, in send\n transport_options=transport_options)\n File "/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 644, in _send\n raise result\nnova.exception_Remote.NoValidHost_Remote: No valid host was found. There are not enough hosts available.\nTraceback (most recent call last):\n\n File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 241, in inner\n return func(*args, **kwargs)\n\n File "/usr/lib/python3.6/site-packages/nova/scheduler/manager.py", line 215, in select_destinations\n allocation_request_version, return_alternates)\n\n File "/usr/lib/python3.6/site-packages/nova/scheduler/filter_scheduler.py", line 96, in select_destinations\n allocation_request_version, return_alternates)\n\n File "/usr/lib/python3.6/site-packages/nova/scheduler/filter_scheduler.py", line 265, in _schedule\n claimed_instance_uuids)\n\n File "/usr/lib/python3.6/site-packages/nova/scheduler/filter_scheduler.py", line 302, in _ensure_sufficient_hosts\n raise exception.NoValidHost(reason=reason)\n\nnova.exception.NoValidHost: No valid host was found. There are not enough hosts available.\n\n'} ...
We enabled debug logging for nova and found:
... 2020-10-13 13:50:49.193 33 DEBUG oslo_concurrency.lockutils [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Lock "7b9846a9-f8be-4d60-92ad-0fb531e48e64" acquired by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:358 2020-10-13 13:50:49.196 33 DEBUG oslo_concurrency.lockutils [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Lock "7b9846a9-f8be-4d60-92ad-0fb531e48e64" released by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: held 0.003s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:370 2020-10-13 13:50:49.217 33 DEBUG oslo_db.sqlalchemy.engines [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python3.6/site-packages/oslo_db/sqlalchemy/engines.py:304 2020-10-13 13:50:49.245 33 DEBUG oslo_concurrency.lockutils [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Lock "('os-cpt-103', 'os-cpt-103')" acquired by "nova.scheduler.host_manager.HostState.update.<locals>._locked_update" :: waited 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:358 2020-10-13 13:50:49.246 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state from compute node: ComputeNode(cpu_allocation_ratio=4.0,cpu_info='{"arch": "x86_64", "model": "EPYC-IBPB", "vendor": "AMD", "topology": {"cells": 1, "sockets": 1, "cores": 24, "threads": 2}, "features": ["bmi1", "smep", "sha-ni", "fsgsbase", "adx", "cmov", "invtsc", "fpu", "mmx", "sep", "abm", "pni", "msr", "xsavec", "f16c", "fma", "nx", "pat", "sse4.1", "rdrand", "wbnoinvd", "vme", "lahf_lm", "cr8legacy", "xsave", "bmi2", "clzero", "mtrr", "arat", "amd-ssbd", "aes", "avx", "avx2", "cx8", "umip", "de", "ibpb", "misalignsse", "osvw", "perfctr_core", "pse36", "mce", "skinit", "syscall", "sse2", "apic", "fxsr_opt", "pse", "ht", "pdpe1gb", "rdtscp", "xsaves", "clflushopt", "pclmuldq", "sse4.2", "movbe", "smap", "ibs", "clwb", "xgetbv1", "sse4a", "cx16", "extapic", "wdt", "perfctr_nb", "tsc", "mca", "topoext", "pae", "fxsr", "lm", "cmp_legacy", "monitor", "3dnowprefetch", "ssse3", "pge", "popcnt", "rdseed", "mmxext", "tce", "clflush", "xsaveopt", "svm", "sse"]}',created_at=2020-09-08T09:22:00Z,current_workload=0,deleted=False,deleted_at=None,disk_allocation_ratio=1.0,disk_available_least=94,free_disk_gb=99,free_ram_mb=206453,host='os-cpt-103',host_ip=172.20.1.13,hypervisor_hostname='os-cpt-103',hypervisor_type='QEMU',hypervisor_version=4002000,id=9,local_gb=99,local_gb_used=0,mapped=0,memory_mb=257653,memory_mb_used=51200,metrics='[{"name": "cpu.user.percent", "timestamp": "2020-10-13T13:50:07.846401", "source": "libvirt.LibvirtDriver", "value": 0.0}, {"name": "cpu.kernel.percent", "timestamp": "2020-10-13T13:50:07.846401", "source": "libvirt.LibvirtDriver", "value": 0.01}, {"name": "cpu.iowait.percent", "timestamp": "2020-10-13T13:50:07.846401", "source": "libvirt.LibvirtDriver", "value": 0.0}, {"name": "cpu.kernel.time", "timestamp": "2020-10-13T13:50:07.846401", "source": "libvirt.LibvirtDriver", "value": 1373295770000000}, {"name": "cpu.percent", "timestamp": "2020-10-13T13:50:07.846401", "source": "libvirt.LibvirtDriver", "value": 0.01}, {"name": "cpu.frequency", "timestamp": "2020-10-13T13:50:07.846401", "source": "libvirt.LibvirtDriver", "value": 1796}, {"name": "cpu.user.time", "timestamp": "2020-10-13T13:50:07.846401", "source": "libvirt.LibvirtDriver", "value": 386827280000000}, {"name": "cpu.idle.time", "timestamp": "2020-10-13T13:50:07.846401", "source": "libvirt.LibvirtDriver", "value": 139696145240000000}, {"name": "cpu.idle.percent", "timestamp": "2020-10-13T13:50:07.846401", "source": "libvirt.LibvirtDriver", "value": 0.98}, {"name": "cpu.iowait.time", "timestamp": "2020-10-13T13:50:07.846401", "source": "libvirt.LibvirtDriver", "value": 3390650000000}]',numa_topology='{"nova_object.name": "NUMATopology", "nova_object.namespace": "nova", "nova_object.version": "1.2", "nova_object.data": {"cells": [{"nova_object.name": "NUMACell", "nova_object.namespace": "nova", "nova_object.version": "1.4", "nova_object.data": {"id": 0, "cpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "pcpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "memory": 257653, "cpu_usage": 0, "memory_usage": 0, "pinned_cpus": [], "siblings": [[12, 36], [10, 34], [2, 26], [1, 25], [9, 33], [0, 24], [8, 32], [11, 35], [13, 37], [19, 43], [21, 45], [18, 42], [38, 14], [46, 22], [4, 28], [20, 44], [15, 39], [17, 41], [23, 47], [16, 40], [6, 30], [7, 31], [3, 27], [5, 29]], "mempages": [{"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 4, "total": 65959381, "used": 0, "reserved": 0}, "nova_object.changes": ["used", "reserved", "size_kb", "total"]}, {"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 2048, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["used", "reserved", "size_kb", "total"]}, {"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 1048576, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["used", "reserved", "size_kb", "total"]}], "network_metadata": {"nova_object.name": "NetworkMetadata", "nova_object.namespace": "nova", "nova_object.version": "1.0", "nova_object.data": {"physnets": [], "tunneled": false}, "nova_object.changes": ["physnets", "tunneled"]}}, "nova_object.changes": ["mempages", "cpu_usage", "memory", "memory_usage", "id", "pinned_cpus", "pcpuset", "network_metadata", "siblings", "cpuset"]}]}, "nova_object.changes": ["cells"]}',pci_device_pools=PciDevicePoolList,ram_allocation_ratio=1.2,running_vms=3,service_id=None,stats={failed_builds='0',io_workload='0',num_instances='3',num_os_type_None='3',num_proj_e06101e290f24563a836e16909146ea0='3',num_task_None='3',num_vm_active='3'},supported_hv_specs=[HVSpec,HVSpec,HVSpec,HVSpec],updated_at=2020-10-13T13:50:07Z,uuid=34993097-d180-49a0-ae3c-9111e6aa8968,vcpus=48,vcpus_used=7) _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:172 2020-10-13 13:50:49.250 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state with aggregates: [Aggregate(created_at=2020-09-08T14:01:05Z,deleted=False,deleted_at=None,hosts=['os-cpt-101','os-cpt-102','os-cpt-103'],id=3,metadata={availability_zone='az-1'},name='az-1',updated_at=None,uuid=8f37e9f3-506d-40da-8997-38915f1dfe67)] _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:175 2020-10-13 13:50:49.251 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state with service dict: {'id': 27, 'uuid': '62ea00b1-e61d-4f84-8b35-c1d3d7a333e7', 'host': 'os-cpt-103', 'binary': 'nova-compute', 'topic': 'compute', 'report_count': 294508, 'disabled': False, 'disabled_reason': None, 'last_seen_up': datetime.datetime(2020, 10, 13, 13, 50, 45, tzinfo=<iso8601.Utc>), 'forced_down': False, 'version': 51, 'created_at': datetime.datetime(2020, 9, 8, 9, 22, tzinfo=<iso8601.Utc>), 'updated_at': datetime.datetime(2020, 10, 13, 13, 50, 45, tzinfo=<iso8601.Utc>), 'deleted_at': None, 'deleted': False} _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:178 2020-10-13 13:50:49.251 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state with instances: ['c88b77f7-2c59-497b-9633-139e62ba2bb1', '635fbd95-dd64-4e0a-9767-a0e0a6958610', 'a02369ab-9002-4bf8-b65c-03e93cbc7df5'] _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:182 2020-10-13 13:50:49.252 33 DEBUG oslo_concurrency.lockutils [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Lock "('os-cpt-103', 'os-cpt-103')" released by "nova.scheduler.host_manager.HostState.update.<locals>._locked_update" :: held 0.007s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:370 2020-10-13 13:50:49.253 33 DEBUG oslo_concurrency.lockutils [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Lock "('os-cpt-101', 'os-cpt-101')" acquired by "nova.scheduler.host_manager.HostState.update.<locals>._locked_update" :: waited 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:358 2020-10-13 13:50:49.253 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state from compute node: ComputeNode(cpu_allocation_ratio=4.0,cpu_info='{"arch": "x86_64", "model": "EPYC-IBPB", "vendor": "AMD", "topology": {"cells": 1, "sockets": 1, "cores": 24, "threads": 2}, "features": ["3dnowprefetch", "fpu", "sse4.1", "fma", "arat", "avx2", "bmi1", "sse2", "extapic", "adx", "aes", "sha-ni", "tce", "osvw", "invtsc", "sse", "xsave", "de", "movbe", "pse", "pdpe1gb", "clflush", "mmx", "wdt", "cmov", "perfctr_core", "skinit", "umip", "bmi2", "cx8", "amd-ssbd", "cmp_legacy", "perfctr_nb", "rdrand", "ibpb", "monitor", "mtrr", "clflushopt", "smap", "msr", "sep", "f16c", "pat", "avx", "xsavec", "mca", "apic", "pni", "xsaves", "cr8legacy", "popcnt", "svm", "clzero", "pae", "lm", "pclmuldq", "pge", "rdseed", "xgetbv1", "sse4.2", "ht", "rdtscp", "fxsr", "lahf_lm", "vme", "sse4a", "tsc", "misalignsse", "abm", "fxsr_opt", "mce", "syscall", "ssse3", "cx16", "ibs", "smep", "fsgsbase", "topoext", "wbnoinvd", "xsaveopt", "mmxext", "nx", "pse36", "clwb"]}',created_at=2020-09-08T09:22:00Z,current_workload=0,deleted=False,deleted_at=None,disk_allocation_ratio=1.0,disk_available_least=64,free_disk_gb=79,free_ram_mb=207477,host='os-cpt-101',host_ip=172.20.1.11,hypervisor_hostname='os-cpt-101',hypervisor_type='QEMU',hypervisor_version=4002000,id=6,local_gb=99,local_gb_used=20,mapped=0,memory_mb=257653,memory_mb_used=50176,metrics='[{"name": "cpu.iowait.percent", "timestamp": "2020-10-13T13:50:21.645565", "source": "libvirt.LibvirtDriver", "value": 0.0}, {"name": "cpu.percent", "timestamp": "2020-10-13T13:50:21.645565", "source": "libvirt.LibvirtDriver", "value": 0.01}, {"name": "cpu.idle.percent", "timestamp": "2020-10-13T13:50:21.645565", "source": "libvirt.LibvirtDriver", "value": 0.98}, {"name": "cpu.frequency", "timestamp": "2020-10-13T13:50:21.645565", "source": "libvirt.LibvirtDriver", "value": 1796}, {"name": "cpu.idle.time", "timestamp": "2020-10-13T13:50:21.645565", "source": "libvirt.LibvirtDriver", "value": 139958595980000000}, {"name": "cpu.user.percent", "timestamp": "2020-10-13T13:50:21.645565", "source": "libvirt.LibvirtDriver", "value": 0.0}, {"name": "cpu.kernel.percent", "timestamp": "2020-10-13T13:50:21.645565", "source": "libvirt.LibvirtDriver", "value": 0.01}, {"name": "cpu.user.time", "timestamp": "2020-10-13T13:50:21.645565", "source": "libvirt.LibvirtDriver", "value": 388866090000000}, {"name": "cpu.kernel.time", "timestamp": "2020-10-13T13:50:21.645565", "source": "libvirt.LibvirtDriver", "value": 1118743070000000}, {"name": "cpu.iowait.time", "timestamp": "2020-10-13T13:50:21.645565", "source": "libvirt.LibvirtDriver", "value": 3637460000000}]',numa_topology='{"nova_object.name": "NUMATopology", "nova_object.namespace": "nova", "nova_object.version": "1.2", "nova_object.data": {"cells": [{"nova_object.name": "NUMACell", "nova_object.namespace": "nova", "nova_object.version": "1.4", "nova_object.data": {"id": 0, "cpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "pcpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "memory": 257653, "cpu_usage": 0, "memory_usage": 0, "pinned_cpus": [], "siblings": [[12, 36], [10, 34], [2, 26], [1, 25], [9, 33], [0, 24], [8, 32], [11, 35], [13, 37], [19, 43], [21, 45], [18, 42], [38, 14], [46, 22], [4, 28], [20, 44], [15, 39], [17, 41], [23, 47], [16, 40], [6, 30], [7, 31], [3, 27], [5, 29]], "mempages": [{"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 4, "total": 65959383, "used": 0, "reserved": 0}, "nova_object.changes": ["reserved", "used", "total", "size_kb"]}, {"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 2048, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["reserved", "used", "total", "size_kb"]}, {"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 1048576, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["reserved", "used", "total", "size_kb"]}], "network_metadata": {"nova_object.name": "NetworkMetadata", "nova_object.namespace": "nova", "nova_object.version": "1.0", "nova_object.data": {"physnets": [], "tunneled": false}, "nova_object.changes": ["physnets", "tunneled"]}}, "nova_object.changes": ["id", "cpu_usage", "pcpuset", "memory", "cpuset", "siblings", "mempages", "network_metadata", "memory_usage", "pinned_cpus"]}]}, "nova_object.changes": ["cells"]}',pci_device_pools=PciDevicePoolList,ram_allocation_ratio=1.2,running_vms=4,service_id=None,stats={failed_builds='0',io_workload='0',num_instances='4',num_os_type_None='4',num_proj_05af2e78169748f69938d01b5238fc8b='1',num_proj_882ea7b2a43a41819ef796f797cdbe82='1',num_proj_d52e2dccbc7e46d8b518120fa7c8753a='1',num_proj_e06101e290f24563a836e16909146ea0='1',num_task_None='4',num_vm_active='4'},supported_hv_specs=[HVSpec,HVSpec,HVSpec,HVSpec],updated_at=2020-10-13T13:50:21Z,uuid=5ab97e91-6b03-48f3-a320-c8cbb032cd3a,vcpus=48,vcpus_used=7) _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:172 2020-10-13 13:50:49.257 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state with aggregates: [Aggregate(created_at=2020-09-08T14:01:05Z,deleted=False,deleted_at=None,hosts=['os-cpt-101','os-cpt-102','os-cpt-103'],id=3,metadata={availability_zone='az-1'},name='az-1',updated_at=None,uuid=8f37e9f3-506d-40da-8997-38915f1dfe67)] _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:175 2020-10-13 13:50:49.257 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state with service dict: {'id': 21, 'uuid': 'c178b502-9b8c-43f3-9053-e3d0e2405c89', 'host': 'os-cpt-101', 'binary': 'nova-compute', 'topic': 'compute', 'report_count': 294504, 'disabled': False, 'disabled_reason': None, 'last_seen_up': datetime.datetime(2020, 10, 13, 13, 50, 43, tzinfo=<iso8601.Utc>), 'forced_down': False, 'version': 51, 'created_at': datetime.datetime(2020, 9, 8, 9, 22, tzinfo=<iso8601.Utc>), 'updated_at': datetime.datetime(2020, 10, 13, 13, 50, 43, tzinfo=<iso8601.Utc>), 'deleted_at': None, 'deleted': False} _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:178 2020-10-13 13:50:49.258 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state with instances: ['96f4c101-d75b-44b6-8af9-dd75068d207d', '121c5df5-2d8b-48e0-a270-9f6f79193dd6', 'e235063d-f7a8-4f38-8294-d7a289167348', '425b53c3-f6f5-43e9-beeb-25c9b6ead1f9'] _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:182 2020-10-13 13:50:49.258 33 DEBUG oslo_concurrency.lockutils [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Lock "('os-cpt-101', 'os-cpt-101')" released by "nova.scheduler.host_manager.HostState.update.<locals>._locked_update" :: held 0.005s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:370 2020-10-13 13:50:49.259 33 DEBUG oslo_concurrency.lockutils [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Lock "('os-cpt-102', 'os-cpt-102')" acquired by "nova.scheduler.host_manager.HostState.update.<locals>._locked_update" :: waited 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:358 2020-10-13 13:50:49.260 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state from compute node: ComputeNode(cpu_allocation_ratio=4.0,cpu_info='{"arch": "x86_64", "model": "EPYC-IBPB", "vendor": "AMD", "topology": {"cells": 1, "sockets": 1, "cores": 24, "threads": 2}, "features": ["lahf_lm", "clflush", "osvw", "extapic", "vme", "wdt", "monitor", "msr", "adx", "pse36", "sse4a", "fma", "pat", "mce", "sse2", "nx", "f16c", "mca", "xsaveopt", "avx", "syscall", "rdrand", "clwb", "ssse3", "xsavec", "invtsc", "bmi2", "fpu", "movbe", "aes", "bmi1", "cr8legacy", "mmxext", "ibpb", "amd-ssbd", "skinit", "topoext", "umip", "cmp_legacy", "arat", "lm", "svm", "fxsr", "ibs", "pae", "misalignsse", "mtrr", "sep", "ht", "smap", "xgetbv1", "clzero", "pdpe1gb", "apic", "abm", "pge", "pni", "tsc", "xsaves", "wbnoinvd", "sse4.2", "cmov", "cx8", "pse", "rdtscp", "cx16", "sse", "sse4.1", "fxsr_opt", "popcnt", "sha-ni", "perfctr_core", "fsgsbase", "avx2", "mmx", "rdseed", "clflushopt", "pclmuldq", "perfctr_nb", "smep", "de", "3dnowprefetch", "xsave", "tce"]}',created_at=2020-09-08T09:21:58Z,current_workload=0,deleted=False,deleted_at=None,disk_allocation_ratio=1.0,disk_available_least=93,free_disk_gb=99,free_ram_mb=216693,host='os-cpt-102',host_ip=172.20.1.12,hypervisor_hostname='os-cpt-102',hypervisor_type='QEMU',hypervisor_version=4002000,id=3,local_gb=99,local_gb_used=0,mapped=0,memory_mb=257653,memory_mb_used=40960,metrics='[{"name": "cpu.user.time", "timestamp": "2020-10-13T13:50:03.450583", "source": "libvirt.LibvirtDriver", "value": 161246250000000}, {"name": "cpu.kernel.percent", "timestamp": "2020-10-13T13:50:03.450583", "source": "libvirt.LibvirtDriver", "value": 0.0}, {"name": "cpu.user.percent", "timestamp": "2020-10-13T13:50:03.450583", "source": "libvirt.LibvirtDriver", "value": 0.0}, {"name": "cpu.idle.percent", "timestamp": "2020-10-13T13:50:03.450583", "source": "libvirt.LibvirtDriver", "value": 0.99}, {"name": "cpu.percent", "timestamp": "2020-10-13T13:50:03.450583", "source": "libvirt.LibvirtDriver", "value": 0.0}, {"name": "cpu.idle.time", "timestamp": "2020-10-13T13:50:03.450583", "source": "libvirt.LibvirtDriver", "value": 140551238730000000}, {"name": "cpu.frequency", "timestamp": "2020-10-13T13:50:03.450583", "source": "libvirt.LibvirtDriver", "value": 3343}, {"name": "cpu.kernel.time", "timestamp": "2020-10-13T13:50:03.450583", "source": "libvirt.LibvirtDriver", "value": 366274340000000}, {"name": "cpu.iowait.time", "timestamp": "2020-10-13T13:50:03.450583", "source": "libvirt.LibvirtDriver", "value": 3829690000000}, {"name": "cpu.iowait.percent", "timestamp": "2020-10-13T13:50:03.450583", "source": "libvirt.LibvirtDriver", "value": 0.0}]',numa_topology='{"nova_object.name": "NUMATopology", "nova_object.namespace": "nova", "nova_object.version": "1.2", "nova_object.data": {"cells": [{"nova_object.name": "NUMACell", "nova_object.namespace": "nova", "nova_object.version": "1.4", "nova_object.data": {"id": 0, "cpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "pcpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "memory": 257653, "cpu_usage": 0, "memory_usage": 0, "pinned_cpus": [], "siblings": [[12, 36], [10, 34], [2, 26], [1, 25], [9, 33], [0, 24], [8, 32], [11, 35], [13, 37], [19, 43], [21, 45], [18, 42], [38, 14], [46, 22], [4, 28], [20, 44], [15, 39], [17, 41], [23, 47], [16, 40], [6, 30], [7, 31], [3, 27], [5, 29]], "mempages": [{"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 4, "total": 65959382, "used": 0, "reserved": 0}, "nova_object.changes": ["total", "used", "reserved", "size_kb"]}, {"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 2048, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["total", "used", "reserved", "size_kb"]}, {"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 1048576, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["total", "used", "reserved", "size_kb"]}], "network_metadata": {"nova_object.name": "NetworkMetadata", "nova_object.namespace": "nova", "nova_object.version": "1.0", "nova_object.data": {"physnets": [], "tunneled": false}, "nova_object.changes": ["physnets", "tunneled"]}}, "nova_object.changes": ["cpu_usage", "pinned_cpus", "mempages", "memory", "siblings", "network_metadata", "cpuset", "memory_usage", "id", "pcpuset"]}]}, "nova_object.changes": ["cells"]}',pci_device_pools=PciDevicePoolList,ram_allocation_ratio=1.2,running_vms=0,service_id=None,stats={failed_builds='0'},supported_hv_specs=[HVSpec,HVSpec,HVSpec,HVSpec],updated_at=2020-10-13T13:50:03Z,uuid=81c2e888-56e4-4469-ab93-f204fa85a5c5,vcpus=48,vcpus_used=2) _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:172 2020-10-13 13:50:49.263 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state with aggregates: [Aggregate(created_at=2020-09-08T14:01:05Z,deleted=False,deleted_at=None,hosts=['os-cpt-101','os-cpt-102','os-cpt-103'],id=3,metadata={availability_zone='az-1'},name='az-1',updated_at=None,uuid=8f37e9f3-506d-40da-8997-38915f1dfe67)] _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:175 2020-10-13 13:50:49.264 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state with service dict: {'id': 18, 'uuid': '342a2d9b-4b32-45a2-ba16-b808343b3b8f', 'host': 'os-cpt-102', 'binary': 'nova-compute', 'topic': 'compute', 'report_count': 294494, 'disabled': False, 'disabled_reason': None, 'last_seen_up': datetime.datetime(2020, 10, 13, 13, 50, 49, tzinfo=<iso8601.Utc>), 'forced_down': False, 'version': 51, 'created_at': datetime.datetime(2020, 9, 8, 9, 21, 58, tzinfo=<iso8601.Utc>), 'updated_at': datetime.datetime(2020, 10, 13, 13, 50, 49, tzinfo=<iso8601.Utc>), 'deleted_at': None, 'deleted': False} _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:178 2020-10-13 13:50:49.264 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state with instances: [] _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:182 2020-10-13 13:50:49.265 33 DEBUG oslo_concurrency.lockutils [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Lock "('os-cpt-102', 'os-cpt-102')" released by "nova.scheduler.host_manager.HostState.update.<locals>._locked_update" :: held 0.005s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:370 2020-10-13 13:50:49.266 33 DEBUG nova.filters [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Starting with 3 host(s) get_filtered_objects /usr/lib/python3.6/site-packages/nova/filters.py:70 2020-10-13 13:50:49.266 33 DEBUG nova.scheduler.filters.availability_zone_filter [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Availability Zone 'az-2' requested. (os-cpt-101, os-cpt-101) ram: 207477MB disk: 65536MB io_ops: 0 instances: 4 has AZs: {'az-1'} host_passes /usr/lib/python3.6/site-packages/nova/scheduler/filters/availability_zone_filter.py:61 2020-10-13 13:50:49.267 33 DEBUG nova.scheduler.filters.availability_zone_filter [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Availability Zone 'az-2' requested. (os-cpt-102, os-cpt-102) ram: 216693MB disk: 95232MB io_ops: 0 instances: 0 has AZs: {'az-1'} host_passes /usr/lib/python3.6/site-packages/nova/scheduler/filters/availability_zone_filter.py:61 2020-10-13 13:50:49.267 33 DEBUG nova.scheduler.filters.availability_zone_filter [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Availability Zone 'az-2' requested. (os-cpt-103, os-cpt-103) ram: 206453MB disk: 96256MB io_ops: 0 instances: 3 has AZs: {'az-1'} host_passes /usr/lib/python3.6/site-packages/nova/scheduler/filters/availability_zone_filter.py:61 2020-10-13 13:50:49.268 33 INFO nova.filters [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Filter AvailabilityZoneFilter returned 0 hosts 2020-10-13 13:50:49.269 33 DEBUG nova.filters [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Filtering removed all hosts for the request with instance ID 'd18bf174-9ccb-4721-b7d8-36cd31834f62'. Filter results: [('AvailabilityZoneFilter', None)] get_filtered_objects /usr/lib/python3.6/site-packages/nova/filters.py:115 2020-10-13 13:50:49.269 33 INFO nova.filters [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Filtering removed all hosts for the request with instance ID 'd18bf174-9ccb-4721-b7d8-36cd31834f62'. Filter results: ['AvailabilityZoneFilter: (start: 3, end: 0)'] 2020-10-13 13:50:49.270 33 DEBUG nova.scheduler.filter_scheduler [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Filtered [] _get_sorted_hosts /usr/lib/python3.6/site-packages/nova/scheduler/filter_scheduler.py:443 2020-10-13 13:50:49.270 33 DEBUG nova.scheduler.filter_scheduler [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] There are 0 hosts available but 1 instances requested to build. _ensure_sufficient_hosts /usr/lib/python3.6/site-packages/nova/scheduler/filter_scheduler.py:300 ...
So nova seems to only consider compute nodes in az-1.
The amphora image is available in both availability zones. Starting an instance from the image directly in az-2 works.
Does anybody have any idea on how to debug this any further? Any hints, tips or ideas would be greatly appreciated!
With kind regards, Björn Puttmann