Open Stack

Hi Björn,

Yeah, I don't see a reason in that nova log snippet, so I can 't point
to the exact cause. It might be higher in the logs than the snippet
included.

That said, it might be that the AZ definition in the AZ profile does
not include the appropriate lb-mgmt-net ID for az-2. Without that
defined, Octavia will use the default lb-mgmt-net ID from the
configuration file, which is likely only available in az-1. I would
try defining the management_network and valid_vip_networks for az-2.

Michael

On Tue, Oct 13, 2020 at 8:48 AM <bjoernputtmann at netprojects.de> wrote:
>
> Hi one and all!
>
> Currently, we have an ussuri openstack installation via kolla-ansible.
> The installation consists of three control nodes and two availability
> zones (az-1, az-2), each with three compute nodes:
> - os-cpt-10[1-3] > az-1
> - os-cpt-20[1-3] > az-2
>
> az-1 is the default compute availability zone.
>
> The compute nodes are hardwarewise equipped exactly the same.
> As storage system a ceph cluster is used.
>
> We wanted to also use Octavia LBaaS and got it up and running after some
> experimentation.
> We also wanted to be able to choose the availibility zones when starting
> a loadbalancer. We added the az info with:
>
> openstack --os-cloud service_octavia loadbalancer
> availabilityzoneprofile create --name az-1 --provider amphora
> --availability-zone-data '{"compute_zone": "az-1"}'
> openstack --os-cloud service_octavia loadbalancer
> availabilityzoneprofile create --name az-2 --provider amphora
> --availability-zone-data '{"compute_zone": "az-2"}'
> openstack --os-cloud service_octavia loadbalancer availabilityzone
> create --name az-1 --availabilityzoneprofile az-1
> openstack --os-cloud service_octavia loadbalancer availabilityzone
> create --name az-2 --availabilityzoneprofile az-2
>
> Creating a loadbalancer via cli:
>
> openstack --os-cloud $LB_PROJECT loadbalancer create --flavor $LB_FLAVOR
> --name $LB_NAME --vip-subnet-id $(openstack --os-cloud $LB_PROJECT
> subnet list --name $LB_SUBNET -f value -c ID) --availability-zone $LB_AZ
>
> works, as long as $LB_AZ == az-1.
>
> If we want to start the loadbalancer in az-2, this fails with an error
> in octavia-worker.log:
> ...
> 2020-09-30 07:21:01.482 34 ERROR oslo_messaging.rpc.server
> octavia.common.exceptions.ComputeBuildException: Failed to build compute
> instance due to: {'code': 500, 'created': '2020-09-30T07:20:55Z',
> 'message': 'No valid host was found. There are not enough hosts
> available.', 'details': 'Traceback (most recent call last):\n  File
> "/usr/lib/python3.6/site-packages/nova/conductor/manager.py", line 1463,
> in schedule_and_build_instances\n    instance_uuids,
> return_alternates=True)\n  File
> "/usr/lib/python3.6/site-packages/nova/conductor/manager.py", line 870,
> in _schedule_instances\n    return_alternates=return_alternates)\n  File
> "/usr/lib/python3.6/site-packages/nova/scheduler/client/query.py", line
> 42, in select_destinations\n    instance_uuids, return_objects,
> return_alternates)\n  File
> "/usr/lib/python3.6/site-packages/nova/scheduler/rpcapi.py", line 160,
> in select_destinations\n    return cctxt.call(ctxt,
> \'select_destinations\', **msg_args)\n  File
> "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/client.py", line
> 181, in call\n    transport_options=self.transport_options)\n  File
> "/usr/lib/python3.6/site-packages/oslo_messaging/transport.py", line
> 129, in _send\n    transport_options=transport_options)\n  File
> "/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py",
> line 654, in send\n    transport_options=transport_options)\n  File
> "/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py",
> line 644, in _send\n    raise
> result\nnova.exception_Remote.NoValidHost_Remote: No valid host was
> found. There are not enough hosts available.\nTraceback (most recent
> call last):\n\n  File
> "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line
> 241, in inner\n    return func(*args, **kwargs)\n\n  File
> "/usr/lib/python3.6/site-packages/nova/scheduler/manager.py", line 215,
> in select_destinations\n    allocation_request_version,
> return_alternates)\n\n  File
> "/usr/lib/python3.6/site-packages/nova/scheduler/filter_scheduler.py",
> line 96, in select_destinations\n    allocation_request_version,
> return_alternates)\n\n  File
> "/usr/lib/python3.6/site-packages/nova/scheduler/filter_scheduler.py",
> line 265, in _schedule\n    claimed_instance_uuids)\n\n  File
> "/usr/lib/python3.6/site-packages/nova/scheduler/filter_scheduler.py",
> line 302, in _ensure_sufficient_hosts\n    raise
> exception.NoValidHost(reason=reason)\n\nnova.exception.NoValidHost: No
> valid host was found. There are not enough hosts available.\n\n'}
> ...
>
> We enabled debug logging for nova and found:
>
> ...
> 2020-10-13 13:50:49.193 33 DEBUG oslo_concurrency.lockutils
> [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
> f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
> default default] Lock "7b9846a9-f8be-4d60-92ad-0fb531e48e64" acquired by
> "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections"
> :: waited 0.000s inner
> /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:358
> 2020-10-13 13:50:49.196 33 DEBUG oslo_concurrency.lockutils
> [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
> f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
> default default] Lock "7b9846a9-f8be-4d60-92ad-0fb531e48e64" released by
> "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections"
> :: held 0.003s inner
> /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:370
> 2020-10-13 13:50:49.217 33 DEBUG oslo_db.sqlalchemy.engines
> [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
> f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
> default default] MySQL server mode set to
> STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
> _check_effective_sql_mode
> /usr/lib/python3.6/site-packages/oslo_db/sqlalchemy/engines.py:304
> 2020-10-13 13:50:49.245 33 DEBUG oslo_concurrency.lockutils
> [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
> f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
> default default] Lock "('os-cpt-103', 'os-cpt-103')" acquired by
> "nova.scheduler.host_manager.HostState.update.<locals>._locked_update"
> :: waited 0.000s inner
> /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:358
> 2020-10-13 13:50:49.246 33 DEBUG nova.scheduler.host_manager
> [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
> f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
> default default] Update host state from compute node:
> ComputeNode(cpu_allocation_ratio=4.0,cpu_info='{"arch": "x86_64",
> "model": "EPYC-IBPB", "vendor": "AMD", "topology": {"cells": 1,
> "sockets": 1, "cores": 24, "threads": 2}, "features": ["bmi1", "smep",
> "sha-ni", "fsgsbase", "adx", "cmov", "invtsc", "fpu", "mmx", "sep",
> "abm", "pni", "msr", "xsavec", "f16c", "fma", "nx", "pat", "sse4.1",
> "rdrand", "wbnoinvd", "vme", "lahf_lm", "cr8legacy", "xsave", "bmi2",
> "clzero", "mtrr", "arat", "amd-ssbd", "aes", "avx", "avx2", "cx8",
> "umip", "de", "ibpb", "misalignsse", "osvw", "perfctr_core", "pse36",
> "mce", "skinit", "syscall", "sse2", "apic", "fxsr_opt", "pse", "ht",
> "pdpe1gb", "rdtscp", "xsaves", "clflushopt", "pclmuldq", "sse4.2",
> "movbe", "smap", "ibs", "clwb", "xgetbv1", "sse4a", "cx16", "extapic",
> "wdt", "perfctr_nb", "tsc", "mca", "topoext", "pae", "fxsr", "lm",
> "cmp_legacy", "monitor", "3dnowprefetch", "ssse3", "pge", "popcnt",
> "rdseed", "mmxext", "tce", "clflush", "xsaveopt", "svm",
> "sse"]}',created_at=2020-09-08T09:22:00Z,current_workload=0,deleted=False,deleted_at=None,disk_allocation_ratio=1.0,disk_available_least=94,free_disk_gb=99,free_ram_mb=206453,host='os-cpt-103',host_ip=172.20.1.13,hypervisor_hostname='os-cpt-103',hypervisor_type='QEMU',hypervisor_version=4002000,id=9,local_gb=99,local_gb_used=0,mapped=0,memory_mb=257653,memory_mb_used=51200,metrics='[{"name":
> "cpu.user.percent", "timestamp": "2020-10-13T13:50:07.846401", "source":
> "libvirt.LibvirtDriver", "value": 0.0}, {"name": "cpu.kernel.percent",
> "timestamp": "2020-10-13T13:50:07.846401", "source":
> "libvirt.LibvirtDriver", "value": 0.01}, {"name": "cpu.iowait.percent",
> "timestamp": "2020-10-13T13:50:07.846401", "source":
> "libvirt.LibvirtDriver", "value": 0.0}, {"name": "cpu.kernel.time",
> "timestamp": "2020-10-13T13:50:07.846401", "source":
> "libvirt.LibvirtDriver", "value": 1373295770000000}, {"name":
> "cpu.percent", "timestamp": "2020-10-13T13:50:07.846401", "source":
> "libvirt.LibvirtDriver", "value": 0.01}, {"name": "cpu.frequency",
> "timestamp": "2020-10-13T13:50:07.846401", "source":
> "libvirt.LibvirtDriver", "value": 1796}, {"name": "cpu.user.time",
> "timestamp": "2020-10-13T13:50:07.846401", "source":
> "libvirt.LibvirtDriver", "value": 386827280000000}, {"name":
> "cpu.idle.time", "timestamp": "2020-10-13T13:50:07.846401", "source":
> "libvirt.LibvirtDriver", "value": 139696145240000000}, {"name":
> "cpu.idle.percent", "timestamp": "2020-10-13T13:50:07.846401", "source":
> "libvirt.LibvirtDriver", "value": 0.98}, {"name": "cpu.iowait.time",
> "timestamp": "2020-10-13T13:50:07.846401", "source":
> "libvirt.LibvirtDriver", "value":
> 3390650000000}]',numa_topology='{"nova_object.name": "NUMATopology",
> "nova_object.namespace": "nova", "nova_object.version": "1.2",
> "nova_object.data": {"cells": [{"nova_object.name": "NUMACell",
> "nova_object.namespace": "nova", "nova_object.version": "1.4",
> "nova_object.data": {"id": 0, "cpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9,
> 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27,
> 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45,
> 46, 47], "pcpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
> 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32,
> 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "memory":
> 257653, "cpu_usage": 0, "memory_usage": 0, "pinned_cpus": [],
> "siblings": [[12, 36], [10, 34], [2, 26], [1, 25], [9, 33], [0, 24], [8,
> 32], [11, 35], [13, 37], [19, 43], [21, 45], [18, 42], [38, 14], [46,
> 22], [4, 28], [20, 44], [15, 39], [17, 41], [23, 47], [16, 40], [6, 30],
> [7, 31], [3, 27], [5, 29]], "mempages": [{"nova_object.name":
> "NUMAPagesTopology", "nova_object.namespace": "nova",
> "nova_object.version": "1.1", "nova_object.data": {"size_kb": 4,
> "total": 65959381, "used": 0, "reserved": 0}, "nova_object.changes":
> ["used", "reserved", "size_kb", "total"]}, {"nova_object.name":
> "NUMAPagesTopology", "nova_object.namespace": "nova",
> "nova_object.version": "1.1", "nova_object.data": {"size_kb": 2048,
> "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["used",
> "reserved", "size_kb", "total"]}, {"nova_object.name":
> "NUMAPagesTopology", "nova_object.namespace": "nova",
> "nova_object.version": "1.1", "nova_object.data": {"size_kb": 1048576,
> "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["used",
> "reserved", "size_kb", "total"]}], "network_metadata":
> {"nova_object.name": "NetworkMetadata", "nova_object.namespace": "nova",
> "nova_object.version": "1.0", "nova_object.data": {"physnets": [],
> "tunneled": false}, "nova_object.changes": ["physnets", "tunneled"]}},
> "nova_object.changes": ["mempages", "cpu_usage", "memory",
> "memory_usage", "id", "pinned_cpus", "pcpuset", "network_metadata",
> "siblings", "cpuset"]}]}, "nova_object.changes":
> ["cells"]}',pci_device_pools=PciDevicePoolList,ram_allocation_ratio=1.2,running_vms=3,service_id=None,stats={failed_builds='0',io_workload='0',num_instances='3',num_os_type_None='3',num_proj_e06101e290f24563a836e16909146ea0='3',num_task_None='3',num_vm_active='3'},supported_hv_specs=[HVSpec,HVSpec,HVSpec,HVSpec],updated_at=2020-10-13T13:50:07Z,uuid=34993097-d180-49a0-ae3c-9111e6aa8968,vcpus=48,vcpus_used=7)
> _locked_update
> /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:172
> 2020-10-13 13:50:49.250 33 DEBUG nova.scheduler.host_manager
> [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
> f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
> default default] Update host state with aggregates:
> [Aggregate(created_at=2020-09-08T14:01:05Z,deleted=False,deleted_at=None,hosts=['os-cpt-101','os-cpt-102','os-cpt-103'],id=3,metadata={availability_zone='az-1'},name='az-1',updated_at=None,uuid=8f37e9f3-506d-40da-8997-38915f1dfe67)]
> _locked_update
> /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:175
> 2020-10-13 13:50:49.251 33 DEBUG nova.scheduler.host_manager
> [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
> f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
> default default] Update host state with service dict: {'id': 27, 'uuid':
> '62ea00b1-e61d-4f84-8b35-c1d3d7a333e7', 'host': 'os-cpt-103', 'binary':
> 'nova-compute', 'topic': 'compute', 'report_count': 294508, 'disabled':
> False, 'disabled_reason': None, 'last_seen_up': datetime.datetime(2020,
> 10, 13, 13, 50, 45, tzinfo=<iso8601.Utc>), 'forced_down': False,
> 'version': 51, 'created_at': datetime.datetime(2020, 9, 8, 9, 22,
> tzinfo=<iso8601.Utc>), 'updated_at': datetime.datetime(2020, 10, 13, 13,
> 50, 45, tzinfo=<iso8601.Utc>), 'deleted_at': None, 'deleted': False}
> _locked_update
> /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:178
> 2020-10-13 13:50:49.251 33 DEBUG nova.scheduler.host_manager
> [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
> f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
> default default] Update host state with instances:
> ['c88b77f7-2c59-497b-9633-139e62ba2bb1',
> '635fbd95-dd64-4e0a-9767-a0e0a6958610',
> 'a02369ab-9002-4bf8-b65c-03e93cbc7df5'] _locked_update
> /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:182
> 2020-10-13 13:50:49.252 33 DEBUG oslo_concurrency.lockutils
> [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
> f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
> default default] Lock "('os-cpt-103', 'os-cpt-103')" released by
> "nova.scheduler.host_manager.HostState.update.<locals>._locked_update"
> :: held 0.007s inner
> /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:370
> 2020-10-13 13:50:49.253 33 DEBUG oslo_concurrency.lockutils
> [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
> f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
> default default] Lock "('os-cpt-101', 'os-cpt-101')" acquired by
> "nova.scheduler.host_manager.HostState.update.<locals>._locked_update"
> :: waited 0.000s inner
> /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:358
> 2020-10-13 13:50:49.253 33 DEBUG nova.scheduler.host_manager
> [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
> f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
> default default] Update host state from compute node:
> ComputeNode(cpu_allocation_ratio=4.0,cpu_info='{"arch": "x86_64",
> "model": "EPYC-IBPB", "vendor": "AMD", "topology": {"cells": 1,
> "sockets": 1, "cores": 24, "threads": 2}, "features": ["3dnowprefetch",
> "fpu", "sse4.1", "fma", "arat", "avx2", "bmi1", "sse2", "extapic",
> "adx", "aes", "sha-ni", "tce", "osvw", "invtsc", "sse", "xsave", "de",
> "movbe", "pse", "pdpe1gb", "clflush", "mmx", "wdt", "cmov",
> "perfctr_core", "skinit", "umip", "bmi2", "cx8", "amd-ssbd",
> "cmp_legacy", "perfctr_nb", "rdrand", "ibpb", "monitor", "mtrr",
> "clflushopt", "smap", "msr", "sep", "f16c", "pat", "avx", "xsavec",
> "mca", "apic", "pni", "xsaves", "cr8legacy", "popcnt", "svm", "clzero",
> "pae", "lm", "pclmuldq", "pge", "rdseed", "xgetbv1", "sse4.2", "ht",
> "rdtscp", "fxsr", "lahf_lm", "vme", "sse4a", "tsc", "misalignsse",
> "abm", "fxsr_opt", "mce", "syscall", "ssse3", "cx16", "ibs", "smep",
> "fsgsbase", "topoext", "wbnoinvd", "xsaveopt", "mmxext", "nx", "pse36",
> "clwb"]}',created_at=2020-09-08T09:22:00Z,current_workload=0,deleted=False,deleted_at=None,disk_allocation_ratio=1.0,disk_available_least=64,free_disk_gb=79,free_ram_mb=207477,host='os-cpt-101',host_ip=172.20.1.11,hypervisor_hostname='os-cpt-101',hypervisor_type='QEMU',hypervisor_version=4002000,id=6,local_gb=99,local_gb_used=20,mapped=0,memory_mb=257653,memory_mb_used=50176,metrics='[{"name":
> "cpu.iowait.percent", "timestamp": "2020-10-13T13:50:21.645565",
> "source": "libvirt.LibvirtDriver", "value": 0.0}, {"name":
> "cpu.percent", "timestamp": "2020-10-13T13:50:21.645565", "source":
> "libvirt.LibvirtDriver", "value": 0.01}, {"name": "cpu.idle.percent",
> "timestamp": "2020-10-13T13:50:21.645565", "source":
> "libvirt.LibvirtDriver", "value": 0.98}, {"name": "cpu.frequency",
> "timestamp": "2020-10-13T13:50:21.645565", "source":
> "libvirt.LibvirtDriver", "value": 1796}, {"name": "cpu.idle.time",
> "timestamp": "2020-10-13T13:50:21.645565", "source":
> "libvirt.LibvirtDriver", "value": 139958595980000000}, {"name":
> "cpu.user.percent", "timestamp": "2020-10-13T13:50:21.645565", "source":
> "libvirt.LibvirtDriver", "value": 0.0}, {"name": "cpu.kernel.percent",
> "timestamp": "2020-10-13T13:50:21.645565", "source":
> "libvirt.LibvirtDriver", "value": 0.01}, {"name": "cpu.user.time",
> "timestamp": "2020-10-13T13:50:21.645565", "source":
> "libvirt.LibvirtDriver", "value": 388866090000000}, {"name":
> "cpu.kernel.time", "timestamp": "2020-10-13T13:50:21.645565", "source":
> "libvirt.LibvirtDriver", "value": 1118743070000000}, {"name":
> "cpu.iowait.time", "timestamp": "2020-10-13T13:50:21.645565", "source":
> "libvirt.LibvirtDriver", "value":
> 3637460000000}]',numa_topology='{"nova_object.name": "NUMATopology",
> "nova_object.namespace": "nova", "nova_object.version": "1.2",
> "nova_object.data": {"cells": [{"nova_object.name": "NUMACell",
> "nova_object.namespace": "nova", "nova_object.version": "1.4",
> "nova_object.data": {"id": 0, "cpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9,
> 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27,
> 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45,
> 46, 47], "pcpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
> 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32,
> 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "memory":
> 257653, "cpu_usage": 0, "memory_usage": 0, "pinned_cpus": [],
> "siblings": [[12, 36], [10, 34], [2, 26], [1, 25], [9, 33], [0, 24], [8,
> 32], [11, 35], [13, 37], [19, 43], [21, 45], [18, 42], [38, 14], [46,
> 22], [4, 28], [20, 44], [15, 39], [17, 41], [23, 47], [16, 40], [6, 30],
> [7, 31], [3, 27], [5, 29]], "mempages": [{"nova_object.name":
> "NUMAPagesTopology", "nova_object.namespace": "nova",
> "nova_object.version": "1.1", "nova_object.data": {"size_kb": 4,
> "total": 65959383, "used": 0, "reserved": 0}, "nova_object.changes":
> ["reserved", "used", "total", "size_kb"]}, {"nova_object.name":
> "NUMAPagesTopology", "nova_object.namespace": "nova",
> "nova_object.version": "1.1", "nova_object.data": {"size_kb": 2048,
> "total": 0, "used": 0, "reserved": 0}, "nova_object.changes":
> ["reserved", "used", "total", "size_kb"]}, {"nova_object.name":
> "NUMAPagesTopology", "nova_object.namespace": "nova",
> "nova_object.version": "1.1", "nova_object.data": {"size_kb": 1048576,
> "total": 0, "used": 0, "reserved": 0}, "nova_object.changes":
> ["reserved", "used", "total", "size_kb"]}], "network_metadata":
> {"nova_object.name": "NetworkMetadata", "nova_object.namespace": "nova",
> "nova_object.version": "1.0", "nova_object.data": {"physnets": [],
> "tunneled": false}, "nova_object.changes": ["physnets", "tunneled"]}},
> "nova_object.changes": ["id", "cpu_usage", "pcpuset", "memory",
> "cpuset", "siblings", "mempages", "network_metadata", "memory_usage",
> "pinned_cpus"]}]}, "nova_object.changes":
> ["cells"]}',pci_device_pools=PciDevicePoolList,ram_allocation_ratio=1.2,running_vms=4,service_id=None,stats={failed_builds='0',io_workload='0',num_instances='4',num_os_type_None='4',num_proj_05af2e78169748f69938d01b5238fc8b='1',num_proj_882ea7b2a43a41819ef796f797cdbe82='1',num_proj_d52e2dccbc7e46d8b518120fa7c8753a='1',num_proj_e06101e290f24563a836e16909146ea0='1',num_task_None='4',num_vm_active='4'},supported_hv_specs=[HVSpec,HVSpec,HVSpec,HVSpec],updated_at=2020-10-13T13:50:21Z,uuid=5ab97e91-6b03-48f3-a320-c8cbb032cd3a,vcpus=48,vcpus_used=7)
> _locked_update
> /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:172
> 2020-10-13 13:50:49.257 33 DEBUG nova.scheduler.host_manager
> [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
> f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
> default default] Update host state with aggregates:
> [Aggregate(created_at=2020-09-08T14:01:05Z,deleted=False,deleted_at=None,hosts=['os-cpt-101','os-cpt-102','os-cpt-103'],id=3,metadata={availability_zone='az-1'},name='az-1',updated_at=None,uuid=8f37e9f3-506d-40da-8997-38915f1dfe67)]
> _locked_update
> /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:175
> 2020-10-13 13:50:49.257 33 DEBUG nova.scheduler.host_manager
> [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
> f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
> default default] Update host state with service dict: {'id': 21, 'uuid':
> 'c178b502-9b8c-43f3-9053-e3d0e2405c89', 'host': 'os-cpt-101', 'binary':
> 'nova-compute', 'topic': 'compute', 'report_count': 294504, 'disabled':
> False, 'disabled_reason': None, 'last_seen_up': datetime.datetime(2020,
> 10, 13, 13, 50, 43, tzinfo=<iso8601.Utc>), 'forced_down': False,
> 'version': 51, 'created_at': datetime.datetime(2020, 9, 8, 9, 22,
> tzinfo=<iso8601.Utc>), 'updated_at': datetime.datetime(2020, 10, 13, 13,
> 50, 43, tzinfo=<iso8601.Utc>), 'deleted_at': None, 'deleted': False}
> _locked_update
> /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:178
> 2020-10-13 13:50:49.258 33 DEBUG nova.scheduler.host_manager
> [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
> f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
> default default] Update host state with instances:
> ['96f4c101-d75b-44b6-8af9-dd75068d207d',
> '121c5df5-2d8b-48e0-a270-9f6f79193dd6',
> 'e235063d-f7a8-4f38-8294-d7a289167348',
> '425b53c3-f6f5-43e9-beeb-25c9b6ead1f9'] _locked_update
> /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:182
> 2020-10-13 13:50:49.258 33 DEBUG oslo_concurrency.lockutils
> [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
> f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
> default default] Lock "('os-cpt-101', 'os-cpt-101')" released by
> "nova.scheduler.host_manager.HostState.update.<locals>._locked_update"
> :: held 0.005s inner
> /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:370
> 2020-10-13 13:50:49.259 33 DEBUG oslo_concurrency.lockutils
> [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
> f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
> default default] Lock "('os-cpt-102', 'os-cpt-102')" acquired by
> "nova.scheduler.host_manager.HostState.update.<locals>._locked_update"
> :: waited 0.000s inner
> /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:358
> 2020-10-13 13:50:49.260 33 DEBUG nova.scheduler.host_manager
> [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
> f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
> default default] Update host state from compute node:
> ComputeNode(cpu_allocation_ratio=4.0,cpu_info='{"arch": "x86_64",
> "model": "EPYC-IBPB", "vendor": "AMD", "topology": {"cells": 1,
> "sockets": 1, "cores": 24, "threads": 2}, "features": ["lahf_lm",
> "clflush", "osvw", "extapic", "vme", "wdt", "monitor", "msr", "adx",
> "pse36", "sse4a", "fma", "pat", "mce", "sse2", "nx", "f16c", "mca",
> "xsaveopt", "avx", "syscall", "rdrand", "clwb", "ssse3", "xsavec",
> "invtsc", "bmi2", "fpu", "movbe", "aes", "bmi1", "cr8legacy", "mmxext",
> "ibpb", "amd-ssbd", "skinit", "topoext", "umip", "cmp_legacy", "arat",
> "lm", "svm", "fxsr", "ibs", "pae", "misalignsse", "mtrr", "sep", "ht",
> "smap", "xgetbv1", "clzero", "pdpe1gb", "apic", "abm", "pge", "pni",
> "tsc", "xsaves", "wbnoinvd", "sse4.2", "cmov", "cx8", "pse", "rdtscp",
> "cx16", "sse", "sse4.1", "fxsr_opt", "popcnt", "sha-ni", "perfctr_core",
> "fsgsbase", "avx2", "mmx", "rdseed", "clflushopt", "pclmuldq",
> "perfctr_nb", "smep", "de", "3dnowprefetch", "xsave",
> "tce"]}',created_at=2020-09-08T09:21:58Z,current_workload=0,deleted=False,deleted_at=None,disk_allocation_ratio=1.0,disk_available_least=93,free_disk_gb=99,free_ram_mb=216693,host='os-cpt-102',host_ip=172.20.1.12,hypervisor_hostname='os-cpt-102',hypervisor_type='QEMU',hypervisor_version=4002000,id=3,local_gb=99,local_gb_used=0,mapped=0,memory_mb=257653,memory_mb_used=40960,metrics='[{"name":
> "cpu.user.time", "timestamp": "2020-10-13T13:50:03.450583", "source":
> "libvirt.LibvirtDriver", "value": 161246250000000}, {"name":
> "cpu.kernel.percent", "timestamp": "2020-10-13T13:50:03.450583",
> "source": "libvirt.LibvirtDriver", "value": 0.0}, {"name":
> "cpu.user.percent", "timestamp": "2020-10-13T13:50:03.450583", "source":
> "libvirt.LibvirtDriver", "value": 0.0}, {"name": "cpu.idle.percent",
> "timestamp": "2020-10-13T13:50:03.450583", "source":
> "libvirt.LibvirtDriver", "value": 0.99}, {"name": "cpu.percent",
> "timestamp": "2020-10-13T13:50:03.450583", "source":
> "libvirt.LibvirtDriver", "value": 0.0}, {"name": "cpu.idle.time",
> "timestamp": "2020-10-13T13:50:03.450583", "source":
> "libvirt.LibvirtDriver", "value": 140551238730000000}, {"name":
> "cpu.frequency", "timestamp": "2020-10-13T13:50:03.450583", "source":
> "libvirt.LibvirtDriver", "value": 3343}, {"name": "cpu.kernel.time",
> "timestamp": "2020-10-13T13:50:03.450583", "source":
> "libvirt.LibvirtDriver", "value": 366274340000000}, {"name":
> "cpu.iowait.time", "timestamp": "2020-10-13T13:50:03.450583", "source":
> "libvirt.LibvirtDriver", "value": 3829690000000}, {"name":
> "cpu.iowait.percent", "timestamp": "2020-10-13T13:50:03.450583",
> "source": "libvirt.LibvirtDriver", "value":
> 0.0}]',numa_topology='{"nova_object.name": "NUMATopology",
> "nova_object.namespace": "nova", "nova_object.version": "1.2",
> "nova_object.data": {"cells": [{"nova_object.name": "NUMACell",
> "nova_object.namespace": "nova", "nova_object.version": "1.4",
> "nova_object.data": {"id": 0, "cpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9,
> 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27,
> 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45,
> 46, 47], "pcpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
> 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32,
> 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "memory":
> 257653, "cpu_usage": 0, "memory_usage": 0, "pinned_cpus": [],
> "siblings": [[12, 36], [10, 34], [2, 26], [1, 25], [9, 33], [0, 24], [8,
> 32], [11, 35], [13, 37], [19, 43], [21, 45], [18, 42], [38, 14], [46,
> 22], [4, 28], [20, 44], [15, 39], [17, 41], [23, 47], [16, 40], [6, 30],
> [7, 31], [3, 27], [5, 29]], "mempages": [{"nova_object.name":
> "NUMAPagesTopology", "nova_object.namespace": "nova",
> "nova_object.version": "1.1", "nova_object.data": {"size_kb": 4,
> "total": 65959382, "used": 0, "reserved": 0}, "nova_object.changes":
> ["total", "used", "reserved", "size_kb"]}, {"nova_object.name":
> "NUMAPagesTopology", "nova_object.namespace": "nova",
> "nova_object.version": "1.1", "nova_object.data": {"size_kb": 2048,
> "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["total",
> "used", "reserved", "size_kb"]}, {"nova_object.name":
> "NUMAPagesTopology", "nova_object.namespace": "nova",
> "nova_object.version": "1.1", "nova_object.data": {"size_kb": 1048576,
> "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["total",
> "used", "reserved", "size_kb"]}], "network_metadata":
> {"nova_object.name": "NetworkMetadata", "nova_object.namespace": "nova",
> "nova_object.version": "1.0", "nova_object.data": {"physnets": [],
> "tunneled": false}, "nova_object.changes": ["physnets", "tunneled"]}},
> "nova_object.changes": ["cpu_usage", "pinned_cpus", "mempages",
> "memory", "siblings", "network_metadata", "cpuset", "memory_usage",
> "id", "pcpuset"]}]}, "nova_object.changes":
> ["cells"]}',pci_device_pools=PciDevicePoolList,ram_allocation_ratio=1.2,running_vms=0,service_id=None,stats={failed_builds='0'},supported_hv_specs=[HVSpec,HVSpec,HVSpec,HVSpec],updated_at=2020-10-13T13:50:03Z,uuid=81c2e888-56e4-4469-ab93-f204fa85a5c5,vcpus=48,vcpus_used=2)
> _locked_update
> /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:172
> 2020-10-13 13:50:49.263 33 DEBUG nova.scheduler.host_manager
> [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
> f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
> default default] Update host state with aggregates:
> [Aggregate(created_at=2020-09-08T14:01:05Z,deleted=False,deleted_at=None,hosts=['os-cpt-101','os-cpt-102','os-cpt-103'],id=3,metadata={availability_zone='az-1'},name='az-1',updated_at=None,uuid=8f37e9f3-506d-40da-8997-38915f1dfe67)]
> _locked_update
> /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:175
> 2020-10-13 13:50:49.264 33 DEBUG nova.scheduler.host_manager
> [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
> f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
> default default] Update host state with service dict: {'id': 18, 'uuid':
> '342a2d9b-4b32-45a2-ba16-b808343b3b8f', 'host': 'os-cpt-102', 'binary':
> 'nova-compute', 'topic': 'compute', 'report_count': 294494, 'disabled':
> False, 'disabled_reason': None, 'last_seen_up': datetime.datetime(2020,
> 10, 13, 13, 50, 49, tzinfo=<iso8601.Utc>), 'forced_down': False,
> 'version': 51, 'created_at': datetime.datetime(2020, 9, 8, 9, 21, 58,
> tzinfo=<iso8601.Utc>), 'updated_at': datetime.datetime(2020, 10, 13, 13,
> 50, 49, tzinfo=<iso8601.Utc>), 'deleted_at': None, 'deleted': False}
> _locked_update
> /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:178
> 2020-10-13 13:50:49.264 33 DEBUG nova.scheduler.host_manager
> [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
> f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
> default default] Update host state with instances: [] _locked_update
> /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:182
> 2020-10-13 13:50:49.265 33 DEBUG oslo_concurrency.lockutils
> [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
> f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
> default default] Lock "('os-cpt-102', 'os-cpt-102')" released by
> "nova.scheduler.host_manager.HostState.update.<locals>._locked_update"
> :: held 0.005s inner
> /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:370
> 2020-10-13 13:50:49.266 33 DEBUG nova.filters
> [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
> f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
> default default] Starting with 3 host(s) get_filtered_objects
> /usr/lib/python3.6/site-packages/nova/filters.py:70
> 2020-10-13 13:50:49.266 33 DEBUG
> nova.scheduler.filters.availability_zone_filter
> [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
> f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
> default default] Availability Zone 'az-2' requested. (os-cpt-101,
> os-cpt-101) ram: 207477MB disk: 65536MB io_ops: 0 instances: 4 has AZs:
> {'az-1'} host_passes
> /usr/lib/python3.6/site-packages/nova/scheduler/filters/availability_zone_filter.py:61
> 2020-10-13 13:50:49.267 33 DEBUG
> nova.scheduler.filters.availability_zone_filter
> [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
> f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
> default default] Availability Zone 'az-2' requested. (os-cpt-102,
> os-cpt-102) ram: 216693MB disk: 95232MB io_ops: 0 instances: 0 has AZs:
> {'az-1'} host_passes
> /usr/lib/python3.6/site-packages/nova/scheduler/filters/availability_zone_filter.py:61
> 2020-10-13 13:50:49.267 33 DEBUG
> nova.scheduler.filters.availability_zone_filter
> [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
> f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
> default default] Availability Zone 'az-2' requested. (os-cpt-103,
> os-cpt-103) ram: 206453MB disk: 96256MB io_ops: 0 instances: 3 has AZs:
> {'az-1'} host_passes
> /usr/lib/python3.6/site-packages/nova/scheduler/filters/availability_zone_filter.py:61
> 2020-10-13 13:50:49.268 33 INFO nova.filters
> [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
> f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
> default default] Filter AvailabilityZoneFilter returned 0 hosts
> 2020-10-13 13:50:49.269 33 DEBUG nova.filters
> [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
> f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
> default default] Filtering removed all hosts for the request with
> instance ID 'd18bf174-9ccb-4721-b7d8-36cd31834f62'. Filter results:
> [('AvailabilityZoneFilter', None)] get_filtered_objects
> /usr/lib/python3.6/site-packages/nova/filters.py:115
> 2020-10-13 13:50:49.269 33 INFO nova.filters
> [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
> f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
> default default] Filtering removed all hosts for the request with
> instance ID 'd18bf174-9ccb-4721-b7d8-36cd31834f62'. Filter results:
> ['AvailabilityZoneFilter: (start: 3, end: 0)']
> 2020-10-13 13:50:49.270 33 DEBUG nova.scheduler.filter_scheduler
> [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
> f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
> default default] Filtered [] _get_sorted_hosts
> /usr/lib/python3.6/site-packages/nova/scheduler/filter_scheduler.py:443
> 2020-10-13 13:50:49.270 33 DEBUG nova.scheduler.filter_scheduler
> [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b
> f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 -
> default default] There are 0 hosts available but 1 instances requested
> to build. _ensure_sufficient_hosts
> /usr/lib/python3.6/site-packages/nova/scheduler/filter_scheduler.py:300
> ...
>
> So nova seems to only consider compute nodes in az-1.
>
> The amphora image is available in both availability zones. Starting an
> instance from the image directly in az-2 works.
>
> Does anybody have any idea on how to debug this any further? Any hints,
> tips or ideas would be greatly appreciated!
>
> With kind regards,
> Björn Puttmann
>
>