Problem with octavia LBaaS and nova availability zones
Hi one and all! Currently, we have an ussuri openstack installation via kolla-ansible. The installation consists of three control nodes and two availability zones (az-1, az-2), each with three compute nodes: - os-cpt-10[1-3] > az-1 - os-cpt-20[1-3] > az-2 az-1 is the default compute availability zone. The compute nodes are hardwarewise equipped exactly the same. As storage system a ceph cluster is used. We wanted to also use Octavia LBaaS and got it up and running after some experimentation. We also wanted to be able to choose the availibility zones when starting a loadbalancer. We added the az info with: openstack --os-cloud service_octavia loadbalancer availabilityzoneprofile create --name az-1 --provider amphora --availability-zone-data '{"compute_zone": "az-1"}' openstack --os-cloud service_octavia loadbalancer availabilityzoneprofile create --name az-2 --provider amphora --availability-zone-data '{"compute_zone": "az-2"}' openstack --os-cloud service_octavia loadbalancer availabilityzone create --name az-1 --availabilityzoneprofile az-1 openstack --os-cloud service_octavia loadbalancer availabilityzone create --name az-2 --availabilityzoneprofile az-2 Creating a loadbalancer via cli: openstack --os-cloud $LB_PROJECT loadbalancer create --flavor $LB_FLAVOR --name $LB_NAME --vip-subnet-id $(openstack --os-cloud $LB_PROJECT subnet list --name $LB_SUBNET -f value -c ID) --availability-zone $LB_AZ works, as long as $LB_AZ == az-1. If we want to start the loadbalancer in az-2, this fails with an error in octavia-worker.log: ... 2020-09-30 07:21:01.482 34 ERROR oslo_messaging.rpc.server octavia.common.exceptions.ComputeBuildException: Failed to build compute instance due to: {'code': 500, 'created': '2020-09-30T07:20:55Z', 'message': 'No valid host was found. There are not enough hosts available.', 'details': 'Traceback (most recent call last):\n File "/usr/lib/python3.6/site-packages/nova/conductor/manager.py", line 1463, in schedule_and_build_instances\n instance_uuids, return_alternates=True)\n File "/usr/lib/python3.6/site-packages/nova/conductor/manager.py", line 870, in _schedule_instances\n return_alternates=return_alternates)\n File "/usr/lib/python3.6/site-packages/nova/scheduler/client/query.py", line 42, in select_destinations\n instance_uuids, return_objects, return_alternates)\n File "/usr/lib/python3.6/site-packages/nova/scheduler/rpcapi.py", line 160, in select_destinations\n return cctxt.call(ctxt, \'select_destinations\', **msg_args)\n File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/client.py", line 181, in call\n transport_options=self.transport_options)\n File "/usr/lib/python3.6/site-packages/oslo_messaging/transport.py", line 129, in _send\n transport_options=transport_options)\n File "/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 654, in send\n transport_options=transport_options)\n File "/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 644, in _send\n raise result\nnova.exception_Remote.NoValidHost_Remote: No valid host was found. There are not enough hosts available.\nTraceback (most recent call last):\n\n File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 241, in inner\n return func(*args, **kwargs)\n\n File "/usr/lib/python3.6/site-packages/nova/scheduler/manager.py", line 215, in select_destinations\n allocation_request_version, return_alternates)\n\n File "/usr/lib/python3.6/site-packages/nova/scheduler/filter_scheduler.py", line 96, in select_destinations\n allocation_request_version, return_alternates)\n\n File "/usr/lib/python3.6/site-packages/nova/scheduler/filter_scheduler.py", line 265, in _schedule\n claimed_instance_uuids)\n\n File "/usr/lib/python3.6/site-packages/nova/scheduler/filter_scheduler.py", line 302, in _ensure_sufficient_hosts\n raise exception.NoValidHost(reason=reason)\n\nnova.exception.NoValidHost: No valid host was found. There are not enough hosts available.\n\n'} ... We enabled debug logging for nova and found: ... 2020-10-13 13:50:49.193 33 DEBUG oslo_concurrency.lockutils [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Lock "7b9846a9-f8be-4d60-92ad-0fb531e48e64" acquired by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:358 2020-10-13 13:50:49.196 33 DEBUG oslo_concurrency.lockutils [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Lock "7b9846a9-f8be-4d60-92ad-0fb531e48e64" released by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: held 0.003s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:370 2020-10-13 13:50:49.217 33 DEBUG oslo_db.sqlalchemy.engines [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python3.6/site-packages/oslo_db/sqlalchemy/engines.py:304 2020-10-13 13:50:49.245 33 DEBUG oslo_concurrency.lockutils [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Lock "('os-cpt-103', 'os-cpt-103')" acquired by "nova.scheduler.host_manager.HostState.update.<locals>._locked_update" :: waited 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:358 2020-10-13 13:50:49.246 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state from compute node: ComputeNode(cpu_allocation_ratio=4.0,cpu_info='{"arch": "x86_64", "model": "EPYC-IBPB", "vendor": "AMD", "topology": {"cells": 1, "sockets": 1, "cores": 24, "threads": 2}, "features": ["bmi1", "smep", "sha-ni", "fsgsbase", "adx", "cmov", "invtsc", "fpu", "mmx", "sep", "abm", "pni", "msr", "xsavec", "f16c", "fma", "nx", "pat", "sse4.1", "rdrand", "wbnoinvd", "vme", "lahf_lm", "cr8legacy", "xsave", "bmi2", "clzero", "mtrr", "arat", "amd-ssbd", "aes", "avx", "avx2", "cx8", "umip", "de", "ibpb", "misalignsse", "osvw", "perfctr_core", "pse36", "mce", "skinit", "syscall", "sse2", "apic", "fxsr_opt", "pse", "ht", "pdpe1gb", "rdtscp", "xsaves", "clflushopt", "pclmuldq", "sse4.2", "movbe", "smap", "ibs", "clwb", "xgetbv1", "sse4a", "cx16", "extapic", "wdt", "perfctr_nb", "tsc", "mca", "topoext", "pae", "fxsr", "lm", "cmp_legacy", "monitor", "3dnowprefetch", "ssse3", "pge", "popcnt", "rdseed", "mmxext", "tce", "clflush", "xsaveopt", "svm", "sse"]}',created_at=2020-09-08T09:22:00Z,current_workload=0,deleted=False,deleted_at=None,disk_allocation_ratio=1.0,disk_available_least=94,free_disk_gb=99,free_ram_mb=206453,host='os-cpt-103',host_ip=172.20.1.13,hypervisor_hostname='os-cpt-103',hypervisor_type='QEMU',hypervisor_version=4002000,id=9,local_gb=99,local_gb_used=0,mapped=0,memory_mb=257653,memory_mb_used=51200,metrics='[{"name": "cpu.user.percent", "timestamp": "2020-10-13T13:50:07.846401", "source": "libvirt.LibvirtDriver", "value": 0.0}, {"name": "cpu.kernel.percent", "timestamp": "2020-10-13T13:50:07.846401", "source": "libvirt.LibvirtDriver", "value": 0.01}, {"name": "cpu.iowait.percent", "timestamp": "2020-10-13T13:50:07.846401", "source": "libvirt.LibvirtDriver", "value": 0.0}, {"name": "cpu.kernel.time", "timestamp": "2020-10-13T13:50:07.846401", "source": "libvirt.LibvirtDriver", "value": 1373295770000000}, {"name": "cpu.percent", "timestamp": "2020-10-13T13:50:07.846401", "source": "libvirt.LibvirtDriver", "value": 0.01}, {"name": "cpu.frequency", "timestamp": "2020-10-13T13:50:07.846401", "source": "libvirt.LibvirtDriver", "value": 1796}, {"name": "cpu.user.time", "timestamp": "2020-10-13T13:50:07.846401", "source": "libvirt.LibvirtDriver", "value": 386827280000000}, {"name": "cpu.idle.time", "timestamp": "2020-10-13T13:50:07.846401", "source": "libvirt.LibvirtDriver", "value": 139696145240000000}, {"name": "cpu.idle.percent", "timestamp": "2020-10-13T13:50:07.846401", "source": "libvirt.LibvirtDriver", "value": 0.98}, {"name": "cpu.iowait.time", "timestamp": "2020-10-13T13:50:07.846401", "source": "libvirt.LibvirtDriver", "value": 3390650000000}]',numa_topology='{"nova_object.name": "NUMATopology", "nova_object.namespace": "nova", "nova_object.version": "1.2", "nova_object.data": {"cells": [{"nova_object.name": "NUMACell", "nova_object.namespace": "nova", "nova_object.version": "1.4", "nova_object.data": {"id": 0, "cpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "pcpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "memory": 257653, "cpu_usage": 0, "memory_usage": 0, "pinned_cpus": [], "siblings": [[12, 36], [10, 34], [2, 26], [1, 25], [9, 33], [0, 24], [8, 32], [11, 35], [13, 37], [19, 43], [21, 45], [18, 42], [38, 14], [46, 22], [4, 28], [20, 44], [15, 39], [17, 41], [23, 47], [16, 40], [6, 30], [7, 31], [3, 27], [5, 29]], "mempages": [{"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 4, "total": 65959381, "used": 0, "reserved": 0}, "nova_object.changes": ["used", "reserved", "size_kb", "total"]}, {"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 2048, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["used", "reserved", "size_kb", "total"]}, {"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 1048576, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["used", "reserved", "size_kb", "total"]}], "network_metadata": {"nova_object.name": "NetworkMetadata", "nova_object.namespace": "nova", "nova_object.version": "1.0", "nova_object.data": {"physnets": [], "tunneled": false}, "nova_object.changes": ["physnets", "tunneled"]}}, "nova_object.changes": ["mempages", "cpu_usage", "memory", "memory_usage", "id", "pinned_cpus", "pcpuset", "network_metadata", "siblings", "cpuset"]}]}, "nova_object.changes": ["cells"]}',pci_device_pools=PciDevicePoolList,ram_allocation_ratio=1.2,running_vms=3,service_id=None,stats={failed_builds='0',io_workload='0',num_instances='3',num_os_type_None='3',num_proj_e06101e290f24563a836e16909146ea0='3',num_task_None='3',num_vm_active='3'},supported_hv_specs=[HVSpec,HVSpec,HVSpec,HVSpec],updated_at=2020-10-13T13:50:07Z,uuid=34993097-d180-49a0-ae3c-9111e6aa8968,vcpus=48,vcpus_used=7) _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:172 2020-10-13 13:50:49.250 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state with aggregates: [Aggregate(created_at=2020-09-08T14:01:05Z,deleted=False,deleted_at=None,hosts=['os-cpt-101','os-cpt-102','os-cpt-103'],id=3,metadata={availability_zone='az-1'},name='az-1',updated_at=None,uuid=8f37e9f3-506d-40da-8997-38915f1dfe67)] _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:175 2020-10-13 13:50:49.251 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state with service dict: {'id': 27, 'uuid': '62ea00b1-e61d-4f84-8b35-c1d3d7a333e7', 'host': 'os-cpt-103', 'binary': 'nova-compute', 'topic': 'compute', 'report_count': 294508, 'disabled': False, 'disabled_reason': None, 'last_seen_up': datetime.datetime(2020, 10, 13, 13, 50, 45, tzinfo=<iso8601.Utc>), 'forced_down': False, 'version': 51, 'created_at': datetime.datetime(2020, 9, 8, 9, 22, tzinfo=<iso8601.Utc>), 'updated_at': datetime.datetime(2020, 10, 13, 13, 50, 45, tzinfo=<iso8601.Utc>), 'deleted_at': None, 'deleted': False} _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:178 2020-10-13 13:50:49.251 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state with instances: ['c88b77f7-2c59-497b-9633-139e62ba2bb1', '635fbd95-dd64-4e0a-9767-a0e0a6958610', 'a02369ab-9002-4bf8-b65c-03e93cbc7df5'] _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:182 2020-10-13 13:50:49.252 33 DEBUG oslo_concurrency.lockutils [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Lock "('os-cpt-103', 'os-cpt-103')" released by "nova.scheduler.host_manager.HostState.update.<locals>._locked_update" :: held 0.007s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:370 2020-10-13 13:50:49.253 33 DEBUG oslo_concurrency.lockutils [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Lock "('os-cpt-101', 'os-cpt-101')" acquired by "nova.scheduler.host_manager.HostState.update.<locals>._locked_update" :: waited 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:358 2020-10-13 13:50:49.253 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state from compute node: ComputeNode(cpu_allocation_ratio=4.0,cpu_info='{"arch": "x86_64", "model": "EPYC-IBPB", "vendor": "AMD", "topology": {"cells": 1, "sockets": 1, "cores": 24, "threads": 2}, "features": ["3dnowprefetch", "fpu", "sse4.1", "fma", "arat", "avx2", "bmi1", "sse2", "extapic", "adx", "aes", "sha-ni", "tce", "osvw", "invtsc", "sse", "xsave", "de", "movbe", "pse", "pdpe1gb", "clflush", "mmx", "wdt", "cmov", "perfctr_core", "skinit", "umip", "bmi2", "cx8", "amd-ssbd", "cmp_legacy", "perfctr_nb", "rdrand", "ibpb", "monitor", "mtrr", "clflushopt", "smap", "msr", "sep", "f16c", "pat", "avx", "xsavec", "mca", "apic", "pni", "xsaves", "cr8legacy", "popcnt", "svm", "clzero", "pae", "lm", "pclmuldq", "pge", "rdseed", "xgetbv1", "sse4.2", "ht", "rdtscp", "fxsr", "lahf_lm", "vme", "sse4a", "tsc", "misalignsse", "abm", "fxsr_opt", "mce", "syscall", "ssse3", "cx16", "ibs", "smep", "fsgsbase", "topoext", "wbnoinvd", "xsaveopt", "mmxext", "nx", "pse36", "clwb"]}',created_at=2020-09-08T09:22:00Z,current_workload=0,deleted=False,deleted_at=None,disk_allocation_ratio=1.0,disk_available_least=64,free_disk_gb=79,free_ram_mb=207477,host='os-cpt-101',host_ip=172.20.1.11,hypervisor_hostname='os-cpt-101',hypervisor_type='QEMU',hypervisor_version=4002000,id=6,local_gb=99,local_gb_used=20,mapped=0,memory_mb=257653,memory_mb_used=50176,metrics='[{"name": "cpu.iowait.percent", "timestamp": "2020-10-13T13:50:21.645565", "source": "libvirt.LibvirtDriver", "value": 0.0}, {"name": "cpu.percent", "timestamp": "2020-10-13T13:50:21.645565", "source": "libvirt.LibvirtDriver", "value": 0.01}, {"name": "cpu.idle.percent", "timestamp": "2020-10-13T13:50:21.645565", "source": "libvirt.LibvirtDriver", "value": 0.98}, {"name": "cpu.frequency", "timestamp": "2020-10-13T13:50:21.645565", "source": "libvirt.LibvirtDriver", "value": 1796}, {"name": "cpu.idle.time", "timestamp": "2020-10-13T13:50:21.645565", "source": "libvirt.LibvirtDriver", "value": 139958595980000000}, {"name": "cpu.user.percent", "timestamp": "2020-10-13T13:50:21.645565", "source": "libvirt.LibvirtDriver", "value": 0.0}, {"name": "cpu.kernel.percent", "timestamp": "2020-10-13T13:50:21.645565", "source": "libvirt.LibvirtDriver", "value": 0.01}, {"name": "cpu.user.time", "timestamp": "2020-10-13T13:50:21.645565", "source": "libvirt.LibvirtDriver", "value": 388866090000000}, {"name": "cpu.kernel.time", "timestamp": "2020-10-13T13:50:21.645565", "source": "libvirt.LibvirtDriver", "value": 1118743070000000}, {"name": "cpu.iowait.time", "timestamp": "2020-10-13T13:50:21.645565", "source": "libvirt.LibvirtDriver", "value": 3637460000000}]',numa_topology='{"nova_object.name": "NUMATopology", "nova_object.namespace": "nova", "nova_object.version": "1.2", "nova_object.data": {"cells": [{"nova_object.name": "NUMACell", "nova_object.namespace": "nova", "nova_object.version": "1.4", "nova_object.data": {"id": 0, "cpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "pcpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "memory": 257653, "cpu_usage": 0, "memory_usage": 0, "pinned_cpus": [], "siblings": [[12, 36], [10, 34], [2, 26], [1, 25], [9, 33], [0, 24], [8, 32], [11, 35], [13, 37], [19, 43], [21, 45], [18, 42], [38, 14], [46, 22], [4, 28], [20, 44], [15, 39], [17, 41], [23, 47], [16, 40], [6, 30], [7, 31], [3, 27], [5, 29]], "mempages": [{"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 4, "total": 65959383, "used": 0, "reserved": 0}, "nova_object.changes": ["reserved", "used", "total", "size_kb"]}, {"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 2048, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["reserved", "used", "total", "size_kb"]}, {"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 1048576, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["reserved", "used", "total", "size_kb"]}], "network_metadata": {"nova_object.name": "NetworkMetadata", "nova_object.namespace": "nova", "nova_object.version": "1.0", "nova_object.data": {"physnets": [], "tunneled": false}, "nova_object.changes": ["physnets", "tunneled"]}}, "nova_object.changes": ["id", "cpu_usage", "pcpuset", "memory", "cpuset", "siblings", "mempages", "network_metadata", "memory_usage", "pinned_cpus"]}]}, "nova_object.changes": ["cells"]}',pci_device_pools=PciDevicePoolList,ram_allocation_ratio=1.2,running_vms=4,service_id=None,stats={failed_builds='0',io_workload='0',num_instances='4',num_os_type_None='4',num_proj_05af2e78169748f69938d01b5238fc8b='1',num_proj_882ea7b2a43a41819ef796f797cdbe82='1',num_proj_d52e2dccbc7e46d8b518120fa7c8753a='1',num_proj_e06101e290f24563a836e16909146ea0='1',num_task_None='4',num_vm_active='4'},supported_hv_specs=[HVSpec,HVSpec,HVSpec,HVSpec],updated_at=2020-10-13T13:50:21Z,uuid=5ab97e91-6b03-48f3-a320-c8cbb032cd3a,vcpus=48,vcpus_used=7) _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:172 2020-10-13 13:50:49.257 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state with aggregates: [Aggregate(created_at=2020-09-08T14:01:05Z,deleted=False,deleted_at=None,hosts=['os-cpt-101','os-cpt-102','os-cpt-103'],id=3,metadata={availability_zone='az-1'},name='az-1',updated_at=None,uuid=8f37e9f3-506d-40da-8997-38915f1dfe67)] _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:175 2020-10-13 13:50:49.257 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state with service dict: {'id': 21, 'uuid': 'c178b502-9b8c-43f3-9053-e3d0e2405c89', 'host': 'os-cpt-101', 'binary': 'nova-compute', 'topic': 'compute', 'report_count': 294504, 'disabled': False, 'disabled_reason': None, 'last_seen_up': datetime.datetime(2020, 10, 13, 13, 50, 43, tzinfo=<iso8601.Utc>), 'forced_down': False, 'version': 51, 'created_at': datetime.datetime(2020, 9, 8, 9, 22, tzinfo=<iso8601.Utc>), 'updated_at': datetime.datetime(2020, 10, 13, 13, 50, 43, tzinfo=<iso8601.Utc>), 'deleted_at': None, 'deleted': False} _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:178 2020-10-13 13:50:49.258 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state with instances: ['96f4c101-d75b-44b6-8af9-dd75068d207d', '121c5df5-2d8b-48e0-a270-9f6f79193dd6', 'e235063d-f7a8-4f38-8294-d7a289167348', '425b53c3-f6f5-43e9-beeb-25c9b6ead1f9'] _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:182 2020-10-13 13:50:49.258 33 DEBUG oslo_concurrency.lockutils [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Lock "('os-cpt-101', 'os-cpt-101')" released by "nova.scheduler.host_manager.HostState.update.<locals>._locked_update" :: held 0.005s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:370 2020-10-13 13:50:49.259 33 DEBUG oslo_concurrency.lockutils [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Lock "('os-cpt-102', 'os-cpt-102')" acquired by "nova.scheduler.host_manager.HostState.update.<locals>._locked_update" :: waited 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:358 2020-10-13 13:50:49.260 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state from compute node: ComputeNode(cpu_allocation_ratio=4.0,cpu_info='{"arch": "x86_64", "model": "EPYC-IBPB", "vendor": "AMD", "topology": {"cells": 1, "sockets": 1, "cores": 24, "threads": 2}, "features": ["lahf_lm", "clflush", "osvw", "extapic", "vme", "wdt", "monitor", "msr", "adx", "pse36", "sse4a", "fma", "pat", "mce", "sse2", "nx", "f16c", "mca", "xsaveopt", "avx", "syscall", "rdrand", "clwb", "ssse3", "xsavec", "invtsc", "bmi2", "fpu", "movbe", "aes", "bmi1", "cr8legacy", "mmxext", "ibpb", "amd-ssbd", "skinit", "topoext", "umip", "cmp_legacy", "arat", "lm", "svm", "fxsr", "ibs", "pae", "misalignsse", "mtrr", "sep", "ht", "smap", "xgetbv1", "clzero", "pdpe1gb", "apic", "abm", "pge", "pni", "tsc", "xsaves", "wbnoinvd", "sse4.2", "cmov", "cx8", "pse", "rdtscp", "cx16", "sse", "sse4.1", "fxsr_opt", "popcnt", "sha-ni", "perfctr_core", "fsgsbase", "avx2", "mmx", "rdseed", "clflushopt", "pclmuldq", "perfctr_nb", "smep", "de", "3dnowprefetch", "xsave", "tce"]}',created_at=2020-09-08T09:21:58Z,current_workload=0,deleted=False,deleted_at=None,disk_allocation_ratio=1.0,disk_available_least=93,free_disk_gb=99,free_ram_mb=216693,host='os-cpt-102',host_ip=172.20.1.12,hypervisor_hostname='os-cpt-102',hypervisor_type='QEMU',hypervisor_version=4002000,id=3,local_gb=99,local_gb_used=0,mapped=0,memory_mb=257653,memory_mb_used=40960,metrics='[{"name": "cpu.user.time", "timestamp": "2020-10-13T13:50:03.450583", "source": "libvirt.LibvirtDriver", "value": 161246250000000}, {"name": "cpu.kernel.percent", "timestamp": "2020-10-13T13:50:03.450583", "source": "libvirt.LibvirtDriver", "value": 0.0}, {"name": "cpu.user.percent", "timestamp": "2020-10-13T13:50:03.450583", "source": "libvirt.LibvirtDriver", "value": 0.0}, {"name": "cpu.idle.percent", "timestamp": "2020-10-13T13:50:03.450583", "source": "libvirt.LibvirtDriver", "value": 0.99}, {"name": "cpu.percent", "timestamp": "2020-10-13T13:50:03.450583", "source": "libvirt.LibvirtDriver", "value": 0.0}, {"name": "cpu.idle.time", "timestamp": "2020-10-13T13:50:03.450583", "source": "libvirt.LibvirtDriver", "value": 140551238730000000}, {"name": "cpu.frequency", "timestamp": "2020-10-13T13:50:03.450583", "source": "libvirt.LibvirtDriver", "value": 3343}, {"name": "cpu.kernel.time", "timestamp": "2020-10-13T13:50:03.450583", "source": "libvirt.LibvirtDriver", "value": 366274340000000}, {"name": "cpu.iowait.time", "timestamp": "2020-10-13T13:50:03.450583", "source": "libvirt.LibvirtDriver", "value": 3829690000000}, {"name": "cpu.iowait.percent", "timestamp": "2020-10-13T13:50:03.450583", "source": "libvirt.LibvirtDriver", "value": 0.0}]',numa_topology='{"nova_object.name": "NUMATopology", "nova_object.namespace": "nova", "nova_object.version": "1.2", "nova_object.data": {"cells": [{"nova_object.name": "NUMACell", "nova_object.namespace": "nova", "nova_object.version": "1.4", "nova_object.data": {"id": 0, "cpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "pcpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "memory": 257653, "cpu_usage": 0, "memory_usage": 0, "pinned_cpus": [], "siblings": [[12, 36], [10, 34], [2, 26], [1, 25], [9, 33], [0, 24], [8, 32], [11, 35], [13, 37], [19, 43], [21, 45], [18, 42], [38, 14], [46, 22], [4, 28], [20, 44], [15, 39], [17, 41], [23, 47], [16, 40], [6, 30], [7, 31], [3, 27], [5, 29]], "mempages": [{"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 4, "total": 65959382, "used": 0, "reserved": 0}, "nova_object.changes": ["total", "used", "reserved", "size_kb"]}, {"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 2048, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["total", "used", "reserved", "size_kb"]}, {"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 1048576, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["total", "used", "reserved", "size_kb"]}], "network_metadata": {"nova_object.name": "NetworkMetadata", "nova_object.namespace": "nova", "nova_object.version": "1.0", "nova_object.data": {"physnets": [], "tunneled": false}, "nova_object.changes": ["physnets", "tunneled"]}}, "nova_object.changes": ["cpu_usage", "pinned_cpus", "mempages", "memory", "siblings", "network_metadata", "cpuset", "memory_usage", "id", "pcpuset"]}]}, "nova_object.changes": ["cells"]}',pci_device_pools=PciDevicePoolList,ram_allocation_ratio=1.2,running_vms=0,service_id=None,stats={failed_builds='0'},supported_hv_specs=[HVSpec,HVSpec,HVSpec,HVSpec],updated_at=2020-10-13T13:50:03Z,uuid=81c2e888-56e4-4469-ab93-f204fa85a5c5,vcpus=48,vcpus_used=2) _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:172 2020-10-13 13:50:49.263 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state with aggregates: [Aggregate(created_at=2020-09-08T14:01:05Z,deleted=False,deleted_at=None,hosts=['os-cpt-101','os-cpt-102','os-cpt-103'],id=3,metadata={availability_zone='az-1'},name='az-1',updated_at=None,uuid=8f37e9f3-506d-40da-8997-38915f1dfe67)] _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:175 2020-10-13 13:50:49.264 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state with service dict: {'id': 18, 'uuid': '342a2d9b-4b32-45a2-ba16-b808343b3b8f', 'host': 'os-cpt-102', 'binary': 'nova-compute', 'topic': 'compute', 'report_count': 294494, 'disabled': False, 'disabled_reason': None, 'last_seen_up': datetime.datetime(2020, 10, 13, 13, 50, 49, tzinfo=<iso8601.Utc>), 'forced_down': False, 'version': 51, 'created_at': datetime.datetime(2020, 9, 8, 9, 21, 58, tzinfo=<iso8601.Utc>), 'updated_at': datetime.datetime(2020, 10, 13, 13, 50, 49, tzinfo=<iso8601.Utc>), 'deleted_at': None, 'deleted': False} _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:178 2020-10-13 13:50:49.264 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state with instances: [] _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:182 2020-10-13 13:50:49.265 33 DEBUG oslo_concurrency.lockutils [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Lock "('os-cpt-102', 'os-cpt-102')" released by "nova.scheduler.host_manager.HostState.update.<locals>._locked_update" :: held 0.005s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:370 2020-10-13 13:50:49.266 33 DEBUG nova.filters [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Starting with 3 host(s) get_filtered_objects /usr/lib/python3.6/site-packages/nova/filters.py:70 2020-10-13 13:50:49.266 33 DEBUG nova.scheduler.filters.availability_zone_filter [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Availability Zone 'az-2' requested. (os-cpt-101, os-cpt-101) ram: 207477MB disk: 65536MB io_ops: 0 instances: 4 has AZs: {'az-1'} host_passes /usr/lib/python3.6/site-packages/nova/scheduler/filters/availability_zone_filter.py:61 2020-10-13 13:50:49.267 33 DEBUG nova.scheduler.filters.availability_zone_filter [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Availability Zone 'az-2' requested. (os-cpt-102, os-cpt-102) ram: 216693MB disk: 95232MB io_ops: 0 instances: 0 has AZs: {'az-1'} host_passes /usr/lib/python3.6/site-packages/nova/scheduler/filters/availability_zone_filter.py:61 2020-10-13 13:50:49.267 33 DEBUG nova.scheduler.filters.availability_zone_filter [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Availability Zone 'az-2' requested. (os-cpt-103, os-cpt-103) ram: 206453MB disk: 96256MB io_ops: 0 instances: 3 has AZs: {'az-1'} host_passes /usr/lib/python3.6/site-packages/nova/scheduler/filters/availability_zone_filter.py:61 2020-10-13 13:50:49.268 33 INFO nova.filters [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Filter AvailabilityZoneFilter returned 0 hosts 2020-10-13 13:50:49.269 33 DEBUG nova.filters [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Filtering removed all hosts for the request with instance ID 'd18bf174-9ccb-4721-b7d8-36cd31834f62'. Filter results: [('AvailabilityZoneFilter', None)] get_filtered_objects /usr/lib/python3.6/site-packages/nova/filters.py:115 2020-10-13 13:50:49.269 33 INFO nova.filters [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Filtering removed all hosts for the request with instance ID 'd18bf174-9ccb-4721-b7d8-36cd31834f62'. Filter results: ['AvailabilityZoneFilter: (start: 3, end: 0)'] 2020-10-13 13:50:49.270 33 DEBUG nova.scheduler.filter_scheduler [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Filtered [] _get_sorted_hosts /usr/lib/python3.6/site-packages/nova/scheduler/filter_scheduler.py:443 2020-10-13 13:50:49.270 33 DEBUG nova.scheduler.filter_scheduler [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] There are 0 hosts available but 1 instances requested to build. _ensure_sufficient_hosts /usr/lib/python3.6/site-packages/nova/scheduler/filter_scheduler.py:300 ... So nova seems to only consider compute nodes in az-1. The amphora image is available in both availability zones. Starting an instance from the image directly in az-2 works. Does anybody have any idea on how to debug this any further? Any hints, tips or ideas would be greatly appreciated! With kind regards, Björn Puttmann
Hi Björn, Yeah, I don't see a reason in that nova log snippet, so I can 't point to the exact cause. It might be higher in the logs than the snippet included. That said, it might be that the AZ definition in the AZ profile does not include the appropriate lb-mgmt-net ID for az-2. Without that defined, Octavia will use the default lb-mgmt-net ID from the configuration file, which is likely only available in az-1. I would try defining the management_network and valid_vip_networks for az-2. Michael On Tue, Oct 13, 2020 at 8:48 AM <bjoernputtmann@netprojects.de> wrote:
Hi one and all!
Currently, we have an ussuri openstack installation via kolla-ansible. The installation consists of three control nodes and two availability zones (az-1, az-2), each with three compute nodes: - os-cpt-10[1-3] > az-1 - os-cpt-20[1-3] > az-2
az-1 is the default compute availability zone.
The compute nodes are hardwarewise equipped exactly the same. As storage system a ceph cluster is used.
We wanted to also use Octavia LBaaS and got it up and running after some experimentation. We also wanted to be able to choose the availibility zones when starting a loadbalancer. We added the az info with:
openstack --os-cloud service_octavia loadbalancer availabilityzoneprofile create --name az-1 --provider amphora --availability-zone-data '{"compute_zone": "az-1"}' openstack --os-cloud service_octavia loadbalancer availabilityzoneprofile create --name az-2 --provider amphora --availability-zone-data '{"compute_zone": "az-2"}' openstack --os-cloud service_octavia loadbalancer availabilityzone create --name az-1 --availabilityzoneprofile az-1 openstack --os-cloud service_octavia loadbalancer availabilityzone create --name az-2 --availabilityzoneprofile az-2
Creating a loadbalancer via cli:
openstack --os-cloud $LB_PROJECT loadbalancer create --flavor $LB_FLAVOR --name $LB_NAME --vip-subnet-id $(openstack --os-cloud $LB_PROJECT subnet list --name $LB_SUBNET -f value -c ID) --availability-zone $LB_AZ
works, as long as $LB_AZ == az-1.
If we want to start the loadbalancer in az-2, this fails with an error in octavia-worker.log: ... 2020-09-30 07:21:01.482 34 ERROR oslo_messaging.rpc.server octavia.common.exceptions.ComputeBuildException: Failed to build compute instance due to: {'code': 500, 'created': '2020-09-30T07:20:55Z', 'message': 'No valid host was found. There are not enough hosts available.', 'details': 'Traceback (most recent call last):\n File "/usr/lib/python3.6/site-packages/nova/conductor/manager.py", line 1463, in schedule_and_build_instances\n instance_uuids, return_alternates=True)\n File "/usr/lib/python3.6/site-packages/nova/conductor/manager.py", line 870, in _schedule_instances\n return_alternates=return_alternates)\n File "/usr/lib/python3.6/site-packages/nova/scheduler/client/query.py", line 42, in select_destinations\n instance_uuids, return_objects, return_alternates)\n File "/usr/lib/python3.6/site-packages/nova/scheduler/rpcapi.py", line 160, in select_destinations\n return cctxt.call(ctxt, \'select_destinations\', **msg_args)\n File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/client.py", line 181, in call\n transport_options=self.transport_options)\n File "/usr/lib/python3.6/site-packages/oslo_messaging/transport.py", line 129, in _send\n transport_options=transport_options)\n File "/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 654, in send\n transport_options=transport_options)\n File "/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 644, in _send\n raise result\nnova.exception_Remote.NoValidHost_Remote: No valid host was found. There are not enough hosts available.\nTraceback (most recent call last):\n\n File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 241, in inner\n return func(*args, **kwargs)\n\n File "/usr/lib/python3.6/site-packages/nova/scheduler/manager.py", line 215, in select_destinations\n allocation_request_version, return_alternates)\n\n File "/usr/lib/python3.6/site-packages/nova/scheduler/filter_scheduler.py", line 96, in select_destinations\n allocation_request_version, return_alternates)\n\n File "/usr/lib/python3.6/site-packages/nova/scheduler/filter_scheduler.py", line 265, in _schedule\n claimed_instance_uuids)\n\n File "/usr/lib/python3.6/site-packages/nova/scheduler/filter_scheduler.py", line 302, in _ensure_sufficient_hosts\n raise exception.NoValidHost(reason=reason)\n\nnova.exception.NoValidHost: No valid host was found. There are not enough hosts available.\n\n'} ...
We enabled debug logging for nova and found:
... 2020-10-13 13:50:49.193 33 DEBUG oslo_concurrency.lockutils [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Lock "7b9846a9-f8be-4d60-92ad-0fb531e48e64" acquired by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:358 2020-10-13 13:50:49.196 33 DEBUG oslo_concurrency.lockutils [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Lock "7b9846a9-f8be-4d60-92ad-0fb531e48e64" released by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: held 0.003s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:370 2020-10-13 13:50:49.217 33 DEBUG oslo_db.sqlalchemy.engines [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python3.6/site-packages/oslo_db/sqlalchemy/engines.py:304 2020-10-13 13:50:49.245 33 DEBUG oslo_concurrency.lockutils [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Lock "('os-cpt-103', 'os-cpt-103')" acquired by "nova.scheduler.host_manager.HostState.update.<locals>._locked_update" :: waited 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:358 2020-10-13 13:50:49.246 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state from compute node: ComputeNode(cpu_allocation_ratio=4.0,cpu_info='{"arch": "x86_64", "model": "EPYC-IBPB", "vendor": "AMD", "topology": {"cells": 1, "sockets": 1, "cores": 24, "threads": 2}, "features": ["bmi1", "smep", "sha-ni", "fsgsbase", "adx", "cmov", "invtsc", "fpu", "mmx", "sep", "abm", "pni", "msr", "xsavec", "f16c", "fma", "nx", "pat", "sse4.1", "rdrand", "wbnoinvd", "vme", "lahf_lm", "cr8legacy", "xsave", "bmi2", "clzero", "mtrr", "arat", "amd-ssbd", "aes", "avx", "avx2", "cx8", "umip", "de", "ibpb", "misalignsse", "osvw", "perfctr_core", "pse36", "mce", "skinit", "syscall", "sse2", "apic", "fxsr_opt", "pse", "ht", "pdpe1gb", "rdtscp", "xsaves", "clflushopt", "pclmuldq", "sse4.2", "movbe", "smap", "ibs", "clwb", "xgetbv1", "sse4a", "cx16", "extapic", "wdt", "perfctr_nb", "tsc", "mca", "topoext", "pae", "fxsr", "lm", "cmp_legacy", "monitor", "3dnowprefetch", "ssse3", "pge", "popcnt", "rdseed", "mmxext", "tce", "clflush", "xsaveopt", "svm", "sse"]}',created_at=2020-09-08T09:22:00Z,current_workload=0,deleted=False,deleted_at=None,disk_allocation_ratio=1.0,disk_available_least=94,free_disk_gb=99,free_ram_mb=206453,host='os-cpt-103',host_ip=172.20.1.13,hypervisor_hostname='os-cpt-103',hypervisor_type='QEMU',hypervisor_version=4002000,id=9,local_gb=99,local_gb_used=0,mapped=0,memory_mb=257653,memory_mb_used=51200,metrics='[{"name": "cpu.user.percent", "timestamp": "2020-10-13T13:50:07.846401", "source": "libvirt.LibvirtDriver", "value": 0.0}, {"name": "cpu.kernel.percent", "timestamp": "2020-10-13T13:50:07.846401", "source": "libvirt.LibvirtDriver", "value": 0.01}, {"name": "cpu.iowait.percent", "timestamp": "2020-10-13T13:50:07.846401", "source": "libvirt.LibvirtDriver", "value": 0.0}, {"name": "cpu.kernel.time", "timestamp": "2020-10-13T13:50:07.846401", "source": "libvirt.LibvirtDriver", "value": 1373295770000000}, {"name": "cpu.percent", "timestamp": "2020-10-13T13:50:07.846401", "source": "libvirt.LibvirtDriver", "value": 0.01}, {"name": "cpu.frequency", "timestamp": "2020-10-13T13:50:07.846401", "source": "libvirt.LibvirtDriver", "value": 1796}, {"name": "cpu.user.time", "timestamp": "2020-10-13T13:50:07.846401", "source": "libvirt.LibvirtDriver", "value": 386827280000000}, {"name": "cpu.idle.time", "timestamp": "2020-10-13T13:50:07.846401", "source": "libvirt.LibvirtDriver", "value": 139696145240000000}, {"name": "cpu.idle.percent", "timestamp": "2020-10-13T13:50:07.846401", "source": "libvirt.LibvirtDriver", "value": 0.98}, {"name": "cpu.iowait.time", "timestamp": "2020-10-13T13:50:07.846401", "source": "libvirt.LibvirtDriver", "value": 3390650000000}]',numa_topology='{"nova_object.name": "NUMATopology", "nova_object.namespace": "nova", "nova_object.version": "1.2", "nova_object.data": {"cells": [{"nova_object.name": "NUMACell", "nova_object.namespace": "nova", "nova_object.version": "1.4", "nova_object.data": {"id": 0, "cpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "pcpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "memory": 257653, "cpu_usage": 0, "memory_usage": 0, "pinned_cpus": [], "siblings": [[12, 36], [10, 34], [2, 26], [1, 25], [9, 33], [0, 24], [8, 32], [11, 35], [13, 37], [19, 43], [21, 45], [18, 42], [38, 14], [46, 22], [4, 28], [20, 44], [15, 39], [17, 41], [23, 47], [16, 40], [6, 30], [7, 31], [3, 27], [5, 29]], "mempages": [{"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 4, "total": 65959381, "used": 0, "reserved": 0}, "nova_object.changes": ["used", "reserved", "size_kb", "total"]}, {"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 2048, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["used", "reserved", "size_kb", "total"]}, {"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 1048576, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["used", "reserved", "size_kb", "total"]}], "network_metadata": {"nova_object.name": "NetworkMetadata", "nova_object.namespace": "nova", "nova_object.version": "1.0", "nova_object.data": {"physnets": [], "tunneled": false}, "nova_object.changes": ["physnets", "tunneled"]}}, "nova_object.changes": ["mempages", "cpu_usage", "memory", "memory_usage", "id", "pinned_cpus", "pcpuset", "network_metadata", "siblings", "cpuset"]}]}, "nova_object.changes": ["cells"]}',pci_device_pools=PciDevicePoolList,ram_allocation_ratio=1.2,running_vms=3,service_id=None,stats={failed_builds='0',io_workload='0',num_instances='3',num_os_type_None='3',num_proj_e06101e290f24563a836e16909146ea0='3',num_task_None='3',num_vm_active='3'},supported_hv_specs=[HVSpec,HVSpec,HVSpec,HVSpec],updated_at=2020-10-13T13:50:07Z,uuid=34993097-d180-49a0-ae3c-9111e6aa8968,vcpus=48,vcpus_used=7) _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:172 2020-10-13 13:50:49.250 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state with aggregates: [Aggregate(created_at=2020-09-08T14:01:05Z,deleted=False,deleted_at=None,hosts=['os-cpt-101','os-cpt-102','os-cpt-103'],id=3,metadata={availability_zone='az-1'},name='az-1',updated_at=None,uuid=8f37e9f3-506d-40da-8997-38915f1dfe67)] _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:175 2020-10-13 13:50:49.251 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state with service dict: {'id': 27, 'uuid': '62ea00b1-e61d-4f84-8b35-c1d3d7a333e7', 'host': 'os-cpt-103', 'binary': 'nova-compute', 'topic': 'compute', 'report_count': 294508, 'disabled': False, 'disabled_reason': None, 'last_seen_up': datetime.datetime(2020, 10, 13, 13, 50, 45, tzinfo=<iso8601.Utc>), 'forced_down': False, 'version': 51, 'created_at': datetime.datetime(2020, 9, 8, 9, 22, tzinfo=<iso8601.Utc>), 'updated_at': datetime.datetime(2020, 10, 13, 13, 50, 45, tzinfo=<iso8601.Utc>), 'deleted_at': None, 'deleted': False} _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:178 2020-10-13 13:50:49.251 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state with instances: ['c88b77f7-2c59-497b-9633-139e62ba2bb1', '635fbd95-dd64-4e0a-9767-a0e0a6958610', 'a02369ab-9002-4bf8-b65c-03e93cbc7df5'] _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:182 2020-10-13 13:50:49.252 33 DEBUG oslo_concurrency.lockutils [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Lock "('os-cpt-103', 'os-cpt-103')" released by "nova.scheduler.host_manager.HostState.update.<locals>._locked_update" :: held 0.007s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:370 2020-10-13 13:50:49.253 33 DEBUG oslo_concurrency.lockutils [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Lock "('os-cpt-101', 'os-cpt-101')" acquired by "nova.scheduler.host_manager.HostState.update.<locals>._locked_update" :: waited 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:358 2020-10-13 13:50:49.253 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state from compute node: ComputeNode(cpu_allocation_ratio=4.0,cpu_info='{"arch": "x86_64", "model": "EPYC-IBPB", "vendor": "AMD", "topology": {"cells": 1, "sockets": 1, "cores": 24, "threads": 2}, "features": ["3dnowprefetch", "fpu", "sse4.1", "fma", "arat", "avx2", "bmi1", "sse2", "extapic", "adx", "aes", "sha-ni", "tce", "osvw", "invtsc", "sse", "xsave", "de", "movbe", "pse", "pdpe1gb", "clflush", "mmx", "wdt", "cmov", "perfctr_core", "skinit", "umip", "bmi2", "cx8", "amd-ssbd", "cmp_legacy", "perfctr_nb", "rdrand", "ibpb", "monitor", "mtrr", "clflushopt", "smap", "msr", "sep", "f16c", "pat", "avx", "xsavec", "mca", "apic", "pni", "xsaves", "cr8legacy", "popcnt", "svm", "clzero", "pae", "lm", "pclmuldq", "pge", "rdseed", "xgetbv1", "sse4.2", "ht", "rdtscp", "fxsr", "lahf_lm", "vme", "sse4a", "tsc", "misalignsse", "abm", "fxsr_opt", "mce", "syscall", "ssse3", "cx16", "ibs", "smep", "fsgsbase", "topoext", "wbnoinvd", "xsaveopt", "mmxext", "nx", "pse36", "clwb"]}',created_at=2020-09-08T09:22:00Z,current_workload=0,deleted=False,deleted_at=None,disk_allocation_ratio=1.0,disk_available_least=64,free_disk_gb=79,free_ram_mb=207477,host='os-cpt-101',host_ip=172.20.1.11,hypervisor_hostname='os-cpt-101',hypervisor_type='QEMU',hypervisor_version=4002000,id=6,local_gb=99,local_gb_used=20,mapped=0,memory_mb=257653,memory_mb_used=50176,metrics='[{"name": "cpu.iowait.percent", "timestamp": "2020-10-13T13:50:21.645565", "source": "libvirt.LibvirtDriver", "value": 0.0}, {"name": "cpu.percent", "timestamp": "2020-10-13T13:50:21.645565", "source": "libvirt.LibvirtDriver", "value": 0.01}, {"name": "cpu.idle.percent", "timestamp": "2020-10-13T13:50:21.645565", "source": "libvirt.LibvirtDriver", "value": 0.98}, {"name": "cpu.frequency", "timestamp": "2020-10-13T13:50:21.645565", "source": "libvirt.LibvirtDriver", "value": 1796}, {"name": "cpu.idle.time", "timestamp": "2020-10-13T13:50:21.645565", "source": "libvirt.LibvirtDriver", "value": 139958595980000000}, {"name": "cpu.user.percent", "timestamp": "2020-10-13T13:50:21.645565", "source": "libvirt.LibvirtDriver", "value": 0.0}, {"name": "cpu.kernel.percent", "timestamp": "2020-10-13T13:50:21.645565", "source": "libvirt.LibvirtDriver", "value": 0.01}, {"name": "cpu.user.time", "timestamp": "2020-10-13T13:50:21.645565", "source": "libvirt.LibvirtDriver", "value": 388866090000000}, {"name": "cpu.kernel.time", "timestamp": "2020-10-13T13:50:21.645565", "source": "libvirt.LibvirtDriver", "value": 1118743070000000}, {"name": "cpu.iowait.time", "timestamp": "2020-10-13T13:50:21.645565", "source": "libvirt.LibvirtDriver", "value": 3637460000000}]',numa_topology='{"nova_object.name": "NUMATopology", "nova_object.namespace": "nova", "nova_object.version": "1.2", "nova_object.data": {"cells": [{"nova_object.name": "NUMACell", "nova_object.namespace": "nova", "nova_object.version": "1.4", "nova_object.data": {"id": 0, "cpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "pcpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "memory": 257653, "cpu_usage": 0, "memory_usage": 0, "pinned_cpus": [], "siblings": [[12, 36], [10, 34], [2, 26], [1, 25], [9, 33], [0, 24], [8, 32], [11, 35], [13, 37], [19, 43], [21, 45], [18, 42], [38, 14], [46, 22], [4, 28], [20, 44], [15, 39], [17, 41], [23, 47], [16, 40], [6, 30], [7, 31], [3, 27], [5, 29]], "mempages": [{"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 4, "total": 65959383, "used": 0, "reserved": 0}, "nova_object.changes": ["reserved", "used", "total", "size_kb"]}, {"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 2048, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["reserved", "used", "total", "size_kb"]}, {"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 1048576, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["reserved", "used", "total", "size_kb"]}], "network_metadata": {"nova_object.name": "NetworkMetadata", "nova_object.namespace": "nova", "nova_object.version": "1.0", "nova_object.data": {"physnets": [], "tunneled": false}, "nova_object.changes": ["physnets", "tunneled"]}}, "nova_object.changes": ["id", "cpu_usage", "pcpuset", "memory", "cpuset", "siblings", "mempages", "network_metadata", "memory_usage", "pinned_cpus"]}]}, "nova_object.changes": ["cells"]}',pci_device_pools=PciDevicePoolList,ram_allocation_ratio=1.2,running_vms=4,service_id=None,stats={failed_builds='0',io_workload='0',num_instances='4',num_os_type_None='4',num_proj_05af2e78169748f69938d01b5238fc8b='1',num_proj_882ea7b2a43a41819ef796f797cdbe82='1',num_proj_d52e2dccbc7e46d8b518120fa7c8753a='1',num_proj_e06101e290f24563a836e16909146ea0='1',num_task_None='4',num_vm_active='4'},supported_hv_specs=[HVSpec,HVSpec,HVSpec,HVSpec],updated_at=2020-10-13T13:50:21Z,uuid=5ab97e91-6b03-48f3-a320-c8cbb032cd3a,vcpus=48,vcpus_used=7) _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:172 2020-10-13 13:50:49.257 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state with aggregates: [Aggregate(created_at=2020-09-08T14:01:05Z,deleted=False,deleted_at=None,hosts=['os-cpt-101','os-cpt-102','os-cpt-103'],id=3,metadata={availability_zone='az-1'},name='az-1',updated_at=None,uuid=8f37e9f3-506d-40da-8997-38915f1dfe67)] _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:175 2020-10-13 13:50:49.257 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state with service dict: {'id': 21, 'uuid': 'c178b502-9b8c-43f3-9053-e3d0e2405c89', 'host': 'os-cpt-101', 'binary': 'nova-compute', 'topic': 'compute', 'report_count': 294504, 'disabled': False, 'disabled_reason': None, 'last_seen_up': datetime.datetime(2020, 10, 13, 13, 50, 43, tzinfo=<iso8601.Utc>), 'forced_down': False, 'version': 51, 'created_at': datetime.datetime(2020, 9, 8, 9, 22, tzinfo=<iso8601.Utc>), 'updated_at': datetime.datetime(2020, 10, 13, 13, 50, 43, tzinfo=<iso8601.Utc>), 'deleted_at': None, 'deleted': False} _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:178 2020-10-13 13:50:49.258 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state with instances: ['96f4c101-d75b-44b6-8af9-dd75068d207d', '121c5df5-2d8b-48e0-a270-9f6f79193dd6', 'e235063d-f7a8-4f38-8294-d7a289167348', '425b53c3-f6f5-43e9-beeb-25c9b6ead1f9'] _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:182 2020-10-13 13:50:49.258 33 DEBUG oslo_concurrency.lockutils [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Lock "('os-cpt-101', 'os-cpt-101')" released by "nova.scheduler.host_manager.HostState.update.<locals>._locked_update" :: held 0.005s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:370 2020-10-13 13:50:49.259 33 DEBUG oslo_concurrency.lockutils [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Lock "('os-cpt-102', 'os-cpt-102')" acquired by "nova.scheduler.host_manager.HostState.update.<locals>._locked_update" :: waited 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:358 2020-10-13 13:50:49.260 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state from compute node: ComputeNode(cpu_allocation_ratio=4.0,cpu_info='{"arch": "x86_64", "model": "EPYC-IBPB", "vendor": "AMD", "topology": {"cells": 1, "sockets": 1, "cores": 24, "threads": 2}, "features": ["lahf_lm", "clflush", "osvw", "extapic", "vme", "wdt", "monitor", "msr", "adx", "pse36", "sse4a", "fma", "pat", "mce", "sse2", "nx", "f16c", "mca", "xsaveopt", "avx", "syscall", "rdrand", "clwb", "ssse3", "xsavec", "invtsc", "bmi2", "fpu", "movbe", "aes", "bmi1", "cr8legacy", "mmxext", "ibpb", "amd-ssbd", "skinit", "topoext", "umip", "cmp_legacy", "arat", "lm", "svm", "fxsr", "ibs", "pae", "misalignsse", "mtrr", "sep", "ht", "smap", "xgetbv1", "clzero", "pdpe1gb", "apic", "abm", "pge", "pni", "tsc", "xsaves", "wbnoinvd", "sse4.2", "cmov", "cx8", "pse", "rdtscp", "cx16", "sse", "sse4.1", "fxsr_opt", "popcnt", "sha-ni", "perfctr_core", "fsgsbase", "avx2", "mmx", "rdseed", "clflushopt", "pclmuldq", "perfctr_nb", "smep", "de", "3dnowprefetch", "xsave", "tce"]}',created_at=2020-09-08T09:21:58Z,current_workload=0,deleted=False,deleted_at=None,disk_allocation_ratio=1.0,disk_available_least=93,free_disk_gb=99,free_ram_mb=216693,host='os-cpt-102',host_ip=172.20.1.12,hypervisor_hostname='os-cpt-102',hypervisor_type='QEMU',hypervisor_version=4002000,id=3,local_gb=99,local_gb_used=0,mapped=0,memory_mb=257653,memory_mb_used=40960,metrics='[{"name": "cpu.user.time", "timestamp": "2020-10-13T13:50:03.450583", "source": "libvirt.LibvirtDriver", "value": 161246250000000}, {"name": "cpu.kernel.percent", "timestamp": "2020-10-13T13:50:03.450583", "source": "libvirt.LibvirtDriver", "value": 0.0}, {"name": "cpu.user.percent", "timestamp": "2020-10-13T13:50:03.450583", "source": "libvirt.LibvirtDriver", "value": 0.0}, {"name": "cpu.idle.percent", "timestamp": "2020-10-13T13:50:03.450583", "source": "libvirt.LibvirtDriver", "value": 0.99}, {"name": "cpu.percent", "timestamp": "2020-10-13T13:50:03.450583", "source": "libvirt.LibvirtDriver", "value": 0.0}, {"name": "cpu.idle.time", "timestamp": "2020-10-13T13:50:03.450583", "source": "libvirt.LibvirtDriver", "value": 140551238730000000}, {"name": "cpu.frequency", "timestamp": "2020-10-13T13:50:03.450583", "source": "libvirt.LibvirtDriver", "value": 3343}, {"name": "cpu.kernel.time", "timestamp": "2020-10-13T13:50:03.450583", "source": "libvirt.LibvirtDriver", "value": 366274340000000}, {"name": "cpu.iowait.time", "timestamp": "2020-10-13T13:50:03.450583", "source": "libvirt.LibvirtDriver", "value": 3829690000000}, {"name": "cpu.iowait.percent", "timestamp": "2020-10-13T13:50:03.450583", "source": "libvirt.LibvirtDriver", "value": 0.0}]',numa_topology='{"nova_object.name": "NUMATopology", "nova_object.namespace": "nova", "nova_object.version": "1.2", "nova_object.data": {"cells": [{"nova_object.name": "NUMACell", "nova_object.namespace": "nova", "nova_object.version": "1.4", "nova_object.data": {"id": 0, "cpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "pcpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "memory": 257653, "cpu_usage": 0, "memory_usage": 0, "pinned_cpus": [], "siblings": [[12, 36], [10, 34], [2, 26], [1, 25], [9, 33], [0, 24], [8, 32], [11, 35], [13, 37], [19, 43], [21, 45], [18, 42], [38, 14], [46, 22], [4, 28], [20, 44], [15, 39], [17, 41], [23, 47], [16, 40], [6, 30], [7, 31], [3, 27], [5, 29]], "mempages": [{"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 4, "total": 65959382, "used": 0, "reserved": 0}, "nova_object.changes": ["total", "used", "reserved", "size_kb"]}, {"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 2048, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["total", "used", "reserved", "size_kb"]}, {"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 1048576, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["total", "used", "reserved", "size_kb"]}], "network_metadata": {"nova_object.name": "NetworkMetadata", "nova_object.namespace": "nova", "nova_object.version": "1.0", "nova_object.data": {"physnets": [], "tunneled": false}, "nova_object.changes": ["physnets", "tunneled"]}}, "nova_object.changes": ["cpu_usage", "pinned_cpus", "mempages", "memory", "siblings", "network_metadata", "cpuset", "memory_usage", "id", "pcpuset"]}]}, "nova_object.changes": ["cells"]}',pci_device_pools=PciDevicePoolList,ram_allocation_ratio=1.2,running_vms=0,service_id=None,stats={failed_builds='0'},supported_hv_specs=[HVSpec,HVSpec,HVSpec,HVSpec],updated_at=2020-10-13T13:50:03Z,uuid=81c2e888-56e4-4469-ab93-f204fa85a5c5,vcpus=48,vcpus_used=2) _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:172 2020-10-13 13:50:49.263 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state with aggregates: [Aggregate(created_at=2020-09-08T14:01:05Z,deleted=False,deleted_at=None,hosts=['os-cpt-101','os-cpt-102','os-cpt-103'],id=3,metadata={availability_zone='az-1'},name='az-1',updated_at=None,uuid=8f37e9f3-506d-40da-8997-38915f1dfe67)] _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:175 2020-10-13 13:50:49.264 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state with service dict: {'id': 18, 'uuid': '342a2d9b-4b32-45a2-ba16-b808343b3b8f', 'host': 'os-cpt-102', 'binary': 'nova-compute', 'topic': 'compute', 'report_count': 294494, 'disabled': False, 'disabled_reason': None, 'last_seen_up': datetime.datetime(2020, 10, 13, 13, 50, 49, tzinfo=<iso8601.Utc>), 'forced_down': False, 'version': 51, 'created_at': datetime.datetime(2020, 9, 8, 9, 21, 58, tzinfo=<iso8601.Utc>), 'updated_at': datetime.datetime(2020, 10, 13, 13, 50, 49, tzinfo=<iso8601.Utc>), 'deleted_at': None, 'deleted': False} _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:178 2020-10-13 13:50:49.264 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state with instances: [] _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:182 2020-10-13 13:50:49.265 33 DEBUG oslo_concurrency.lockutils [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Lock "('os-cpt-102', 'os-cpt-102')" released by "nova.scheduler.host_manager.HostState.update.<locals>._locked_update" :: held 0.005s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:370 2020-10-13 13:50:49.266 33 DEBUG nova.filters [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Starting with 3 host(s) get_filtered_objects /usr/lib/python3.6/site-packages/nova/filters.py:70 2020-10-13 13:50:49.266 33 DEBUG nova.scheduler.filters.availability_zone_filter [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Availability Zone 'az-2' requested. (os-cpt-101, os-cpt-101) ram: 207477MB disk: 65536MB io_ops: 0 instances: 4 has AZs: {'az-1'} host_passes /usr/lib/python3.6/site-packages/nova/scheduler/filters/availability_zone_filter.py:61 2020-10-13 13:50:49.267 33 DEBUG nova.scheduler.filters.availability_zone_filter [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Availability Zone 'az-2' requested. (os-cpt-102, os-cpt-102) ram: 216693MB disk: 95232MB io_ops: 0 instances: 0 has AZs: {'az-1'} host_passes /usr/lib/python3.6/site-packages/nova/scheduler/filters/availability_zone_filter.py:61 2020-10-13 13:50:49.267 33 DEBUG nova.scheduler.filters.availability_zone_filter [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Availability Zone 'az-2' requested. (os-cpt-103, os-cpt-103) ram: 206453MB disk: 96256MB io_ops: 0 instances: 3 has AZs: {'az-1'} host_passes /usr/lib/python3.6/site-packages/nova/scheduler/filters/availability_zone_filter.py:61 2020-10-13 13:50:49.268 33 INFO nova.filters [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Filter AvailabilityZoneFilter returned 0 hosts 2020-10-13 13:50:49.269 33 DEBUG nova.filters [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Filtering removed all hosts for the request with instance ID 'd18bf174-9ccb-4721-b7d8-36cd31834f62'. Filter results: [('AvailabilityZoneFilter', None)] get_filtered_objects /usr/lib/python3.6/site-packages/nova/filters.py:115 2020-10-13 13:50:49.269 33 INFO nova.filters [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Filtering removed all hosts for the request with instance ID 'd18bf174-9ccb-4721-b7d8-36cd31834f62'. Filter results: ['AvailabilityZoneFilter: (start: 3, end: 0)'] 2020-10-13 13:50:49.270 33 DEBUG nova.scheduler.filter_scheduler [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Filtered [] _get_sorted_hosts /usr/lib/python3.6/site-packages/nova/scheduler/filter_scheduler.py:443 2020-10-13 13:50:49.270 33 DEBUG nova.scheduler.filter_scheduler [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] There are 0 hosts available but 1 instances requested to build. _ensure_sufficient_hosts /usr/lib/python3.6/site-packages/nova/scheduler/filter_scheduler.py:300 ...
So nova seems to only consider compute nodes in az-1.
The amphora image is available in both availability zones. Starting an instance from the image directly in az-2 works.
Does anybody have any idea on how to debug this any further? Any hints, tips or ideas would be greatly appreciated!
With kind regards, Björn Puttmann
Hi Michael, thanks for taking the time reading this! The internal octavia network is a vlan on a physical network on the hosts. openstack --os-cloud admin subnet list --long gives: +--------------------------------------+----------------------------------------------------+--------+----------------------------------+-------+--------+--------------------------------------+--------------+-------------+--------------------+------+ | ID | Name | Status | Project | State | Shared | Subnets | Network Type | Router Type | Availability Zones | Tags | +--------------------------------------+----------------------------------------------------+--------+----------------------------------+-------+--------+--------------------------------------+--------------+-------------+--------------------+------+ ... | fe73f93f-9e76-4bd9-9ce4-e60882d313d9 | internal-octavia | ACTIVE | d52e2dccbc7e46d8b518120fa7c8753a | UP | False | fd10e49c-359a-45d0-b207-89ba3d438a68 | vlan | Internal | az-1, az-2 | | ... +--------------------------------------+----------------------------------------------------+--------+----------------------------------+-------+--------+--------------------------------------+--------------+-------------+--------------------+------+ so, Openstack seems to know the the network and it should work in both az. openstack --os-cloud admin subnet list --long gives: +--------------------------------------+---------------------------------------------------+--------------------------------------+--------------------+----------------------------------+-------+--------------+---------------------------------+-------------+------------+-----------------+---------------+------+ | ID | Name | Network | Subnet | Project | DHCP | Name Servers | Allocation Pools | Host Routes | IP Version | Gateway | Service Types | Tags | +--------------------------------------+---------------------------------------------------+--------------------------------------+--------------------+----------------------------------+-------+--------------+---------------------------------+-------------+------------+-----------------+---------------+------+ ... | fd10e49c-359a-45d0-b207-89ba3d438a68 | 172_20_48_0_20 | fe73f93f-9e76-4bd9-9ce4-e60882d313d9 | 172.20.48.0/20 | d52e2dccbc7e46d8b518120fa7c8753a | False | | 172.20.48.50-172.20.63.254 | | 4 | 172.20.48.1 | | | ... +--------------------------------------+---------------------------------------------------+--------------------------------------+--------------------+----------------------------------+-------+--------------+---------------------------------+-------------+------------+-----------------+---------------+------+ BTW, starting an instance manually in the service project with the correct network and the octavia image works. I recreated the availabilityzoneprofile via: openstack --os-cloud service_octavia loadbalancer availabilityzoneprofile create --name az-1 --provider amphora --availability-zone-data '{"compute_zone": "az-1", "management_network": "fe73f93f-9e76-4bd9-9ce4-e60882d313d9"}' openstack --os-cloud service_octavia loadbalancer availabilityzoneprofile create --name az-2 --provider amphora --availability-zone-data '{"compute_zone": "az-2", "management_network": "fe73f93f-9e76-4bd9-9ce4-e60882d313d9"}' but still no success. Am 2020-10-14 00:06, schrieb Michael Johnson:
Hi Björn,
Yeah, I don't see a reason in that nova log snippet, so I can 't point to the exact cause. It might be higher in the logs than the snippet included.
That said, it might be that the AZ definition in the AZ profile does not include the appropriate lb-mgmt-net ID for az-2. Without that defined, Octavia will use the default lb-mgmt-net ID from the configuration file, which is likely only available in az-1. I would try defining the management_network and valid_vip_networks for az-2.
Michael
Yeah, it's clearly a nova scheduling issue as we can see in the nova logs that Octavia passed in the correct AZ setting. Is there any way you can make the full chunk of the nova scheduler log available? Somewhere in there should be a filter log message that indicates why nova is unable to schedule this correctly. It might also be a good idea to ask this again on the discuss list and tag nova (Add [nova] in the subject) instead of Octavia. Michael On Wed, Oct 14, 2020 at 9:50 AM <bjoernputtmann@netprojects.de> wrote:
Hi Michael,
thanks for taking the time reading this!
The internal octavia network is a vlan on a physical network on the hosts. openstack --os-cloud admin subnet list --long gives:
+--------------------------------------+----------------------------------------------------+--------+----------------------------------+-------+--------+--------------------------------------+--------------+-------------+--------------------+------+ | ID | Name | Status | Project | State | Shared | Subnets | Network Type | Router Type | Availability Zones | Tags | +--------------------------------------+----------------------------------------------------+--------+----------------------------------+-------+--------+--------------------------------------+--------------+-------------+--------------------+------+ ... | fe73f93f-9e76-4bd9-9ce4-e60882d313d9 | internal-octavia | ACTIVE | d52e2dccbc7e46d8b518120fa7c8753a | UP | False | fd10e49c-359a-45d0-b207-89ba3d438a68 | vlan | Internal | az-1, az-2 | | ... +--------------------------------------+----------------------------------------------------+--------+----------------------------------+-------+--------+--------------------------------------+--------------+-------------+--------------------+------+
so, Openstack seems to know the the network and it should work in both az.
openstack --os-cloud admin subnet list --long gives: +--------------------------------------+---------------------------------------------------+--------------------------------------+--------------------+----------------------------------+-------+--------------+---------------------------------+-------------+------------+-----------------+---------------+------+ | ID | Name | Network | Subnet | Project | DHCP | Name Servers | Allocation Pools | Host Routes | IP Version | Gateway | Service Types | Tags | +--------------------------------------+---------------------------------------------------+--------------------------------------+--------------------+----------------------------------+-------+--------------+---------------------------------+-------------+------------+-----------------+---------------+------+ ... | fd10e49c-359a-45d0-b207-89ba3d438a68 | 172_20_48_0_20 | fe73f93f-9e76-4bd9-9ce4-e60882d313d9 | 172.20.48.0/20 | d52e2dccbc7e46d8b518120fa7c8753a | False | | 172.20.48.50-172.20.63.254 | | 4 | 172.20.48.1 | | | ... +--------------------------------------+---------------------------------------------------+--------------------------------------+--------------------+----------------------------------+-------+--------------+---------------------------------+-------------+------------+-----------------+---------------+------+
BTW, starting an instance manually in the service project with the correct network and the octavia image works.
I recreated the availabilityzoneprofile via:
openstack --os-cloud service_octavia loadbalancer availabilityzoneprofile create --name az-1 --provider amphora --availability-zone-data '{"compute_zone": "az-1", "management_network": "fe73f93f-9e76-4bd9-9ce4-e60882d313d9"}' openstack --os-cloud service_octavia loadbalancer availabilityzoneprofile create --name az-2 --provider amphora --availability-zone-data '{"compute_zone": "az-2", "management_network": "fe73f93f-9e76-4bd9-9ce4-e60882d313d9"}'
but still no success.
Am 2020-10-14 00:06, schrieb Michael Johnson:
Hi Björn,
Yeah, I don't see a reason in that nova log snippet, so I can 't point to the exact cause. It might be higher in the logs than the snippet included.
That said, it might be that the AZ definition in the AZ profile does not include the appropriate lb-mgmt-net ID for az-2. Without that defined, Octavia will use the default lb-mgmt-net ID from the configuration file, which is likely only available in az-1. I would try defining the management_network and valid_vip_networks for az-2.
Michael
participants (2)
-
bjoernputtmann＠netprojects.de
-
Michael Johnson