April 2020 - openstack-discuss - lists.openstack.org

RDO | OpenStack VMs | XFS Metadata Corruption
by Pradeep Antil 23 Apr '20

23 Apr '20

Hi Techies, I have below RDO setup, - RDO 13 - Base OS for Controllers & Compute is Ubuntu - Neutron with vxlan + VLAN (for provider N/W) - Cinder backend is CHEF - HugePages and CPU Pinning for VNF's VMs I am trying to deploy a stack which is suppose to create 18 VMs across 11 computes nodes internal disk, but every time 3 to 4 VMs out of 18 doesn't spawned properly. At console of these VMs i am getting below errors, Any idea and suggestion how to troubleshoot this? and resolve the issue. [ 100.681552] ffff8b37f8f86020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 100.681553] ffff8b37f8f86030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 100.681560] XFS (vda1): Metadata corruption detected at xfs_inode_buf_verify+0x79/0x100 [xfs], xfs_inode block 0x179b800 [ 100.681561] XFS (vda1): Unmount and run xfs_repair [ 100.681561] XFS (vda1): First 64 bytes of corrupted metadata buffer: [ 100.681562] ffff8b37f8f86000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 100.681562] ffff8b37f8f86010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 100.681563] ffff8b37f8f86020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 100.681564] ffff8b37f8f86030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 100.681596] XFS (vda1): metadata I/O error: block 0x179b800 ("xfs_trans_read_buf_map") error 117 numblks 32 [ 100.681599] XFS (vda1): xfs_imap_to_bp: xfs_trans_read_buf() returned error -117. [ 99.585766] [cloud-init[32m OK [0m[2530]: ] Cloud-init v. 18.2 running 'init-local' at Thu, 16 Apr 2020 10:44:21 +0000. Up 99.55 seconds.Started oVirt Guest Agent. [ 101.086566] XFS (vda1): Metadata corruption detected at xfs_inode_buf_verify+0x79/0x100 [xfs], xfs_inode block 0x179b800 [ 101.092093] XFS (vda1): Unmount and run xfs_repair [ 101.094660] XFS (vda1): First 64 bytes of corrupted metadata buffer: [ 101.097787] ffff8b37fef07000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 101.105959] ffff8b37fef07010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 101.110718] ffff8b37fef07020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 101.115412] ffff8b37fef07030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 101.120166] XFS (vda1): Metadata corruption detected at xfs_inode_buf_verify+0x79/0x100 [xfs], xfs_inode block 0x179b800 [ 101.125644] XFS (vda1): Unmount and run xfs_repair [ 101.128229] XFS (vda1): First 64 bytes of corrupted metadata buffer: [ 101.131370] ffff8b37fef07000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 101.138671] ffff8b37fef07010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 101.143427] ffff8b37fef07020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 101.148235] ffff8b37fef07030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 101.152999] XFS (vda1): Metadata corruption detected at xfs_inode_buf_verify+0x79/0x100 [xfs], xfs_inode block 0x179b800 [ 101.158479] XFS (vda1): Unmount and run xfs_repair [ 101.161068] XFS (vda1): First 64 bytes of corrupted metadata buffer: [ 101.169883] ffff8b37fef07000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 101.174751] ffff8b37fef07010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 101.179639] ffff8b37fef07020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 101.184285] ffff8b37fef07030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 101.189104] XFS (vda1): Metadata corruption detected at xfs_inode_buf_verify+0x79/0x100 [xfs], xfs_inode block 0x179b800 [ 101.194619] XFS (vda1): Unmount and run xfs_repair [ 101.197228] XFS (vda1): First 64 bytes of corrupted metadata buffer: [ 101.201109] ffff8b37fef07000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 101.205976] ffff8b37fef07010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 101.210709] ffff8b37fef07020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 101.215442] ffff8b37fef07030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 101.220196] XFS (vda1): Metadata corruption detected at xfs_inode_buf_verify+0x79/0x100 [xfs], xfs_inode block 0x179b800 [ 101.225708] XFS (vda1): Unmount and run xfs_repair [ 101.228296] XFS (vda1): First 64 bytes of corrupted metadata buffer: [ 101.232058] ffff8b37fef07000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 101.236803] ffff8b37fef07010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 101.241538] ffff8b37fef07020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 101.246252] ffff8b37fef07030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 101.250997] XFS (vda1): Metadata corruption detected at xfs_inode_buf_verify+0x79/0x100 [xfs], xfs_inode block 0x179b800 [ 101.256518] XFS (vda1): Unmount and run xfs_repair [ 101.259105] XFS (vda1): First 64 bytes of corrupted metadata buffer: [ 101.262912] ffff8b37fef07000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 101.267649] ffff8b37fef07010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 101.272360] ffff8b37fef07020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 101.277088] ffff8b37fef07030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 101.281831] XFS (vda1): Metadata corruption detected at xfs_inode_buf_verify+0x79/0x100 [xfs], xfs_inode block 0x179b800 [ 101.287322] XFS (vda1): Unmount and run xfs_repair [ 101.295401] XFS (vda1): First 64 bytes of corrupted metadata buffer: [ 101.298546] ffff8b37fef07000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 101.303283] ffff8b37fef07010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 101.308009] ffff8b37fef07020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 101.312747] ffff8b37fef07030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 101.317460] XFS (vda1): Metadata corruption detected at xfs_inode_buf_verify+0x79/0x100 [xfs], xfs_inode block 0x179b800 [ 101.322960] XFS (vda1): Unmount and run xfs_repair [ 101.326233] XFS (vda1): First 64 bytes of corrupted metadata buffer: [ 101.329383] ffff8b37fef07000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 101.334100] ffff8b37fef07010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 101.338822] ffff8b37fef07020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 101.343549] ffff8b37fef07030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 101.348297] XFS (vda1): Metadata corruption detected at xfs_inode_buf_verify+0x79/0x100 [xfs], xfs_inode block 0x179b800 [ 101.353793] XFS (vda1): Unmount and run xfs_repair [ 101.357102] XFS (vda1): First 64 bytes of corrupted metadata buffer: Below are the Nova Compute logs of the hypervisor where it is scheduled to spawned, 3T06:04:55Z,direct_url=<?>,disk_format='qcow2',id=c255bbbc-c8c3-462e-b827-1d35db08d283,min_disk=0,min_ram=0,name='vnf-scef-18.5',owner='36c70ae400e74fc2859f44815d0c9afb',properties=ImageMetaProps,protected=<?>,size=7143292928,status='active',tags=<?>,updated_at=2020-03-03T06:05:49Z,virtual_size=<?>,visibility=<?>) rescue=None block_device_info={'swap': None, 'root_device_name': u'/dev/vda', 'ephemerals': [], 'block_device_mapping': []} _get_guest_xml /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:5419 2020-04-16 16:12:28.310 219284 DEBUG oslo_concurrency.processutils [req-5a53263c-928c-4a0c-a03c-8b698339efca cbabd9368dc24fea84fd2e43935fddfa 975a7d3840a141b0a20a9dc60e3da6cd - default default] CMD "/openstack/venvs/nova-17.1.12/bin/python -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info /var/lib/nova/instances/dfa80e78-ee02-46e5-ba7a-0874fa37da56/disk --force-share" returned: 0 in 0.031s execute /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/oslo_concurrency/processutils.py:409 2020-04-16 16:12:28.310 219284 DEBUG oslo_concurrency.processutils [req-5a53263c-928c-4a0c-a03c-8b698339efca cbabd9368dc24fea84fd2e43935fddfa 975a7d3840a141b0a20a9dc60e3da6cd - default default] Running cmd (subprocess): qemu-img resize /var/lib/nova/instances/dfa80e78-ee02-46e5-ba7a-0874fa37da56/disk 64424509440 execute /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/oslo_concurrency/processutils.py:372 2020-04-16 16:12:28.322 219284 DEBUG oslo_concurrency.processutils [req-5a53263c-928c-4a0c-a03c-8b698339efca cbabd9368dc24fea84fd2e43935fddfa 975a7d3840a141b0a20a9dc60e3da6cd - default default] CMD "qemu-img resize /var/lib/nova/instances/dfa80e78-ee02-46e5-ba7a-0874fa37da56/disk 64424509440" returned: 0 in 0.012s execute /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/oslo_concurrency/processutils.py:409 2020-04-16 16:12:28.323 219284 DEBUG oslo_concurrency.lockutils [req-5a53263c-928c-4a0c-a03c-8b698339efca cbabd9368dc24fea84fd2e43935fddfa 975a7d3840a141b0a20a9dc60e3da6cd - default default] Lock "86692cd1e738b8df7cf1f951967c61e92222fc4c" released by "nova.virt.libvirt.imagebackend.copy_qcow2_image" :: held 0.092s inner /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:285 2020-04-16 16:12:28.323 219284 DEBUG oslo_concurrency.processutils [req-5a53263c-928c-4a0c-a03c-8b698339efca cbabd9368dc24fea84fd2e43935fddfa 975a7d3840a141b0a20a9dc60e3da6cd - default default] Running cmd (subprocess): /openstack/venvs/nova-17.1.12/bin/python -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info /var/lib/nova/instances/_base/86692cd1e738b8df7cf1f951967c61e92222fc4c --force-share execute /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/oslo_concurrency/processutils.py:372 2020-04-16 16:12:28.338 219284 DEBUG nova.virt.libvirt.driver [req-a7ee4c3e-ea3a-4237-ba75-4c85411c9889 cbabd9368dc24fea84fd2e43935fddfa 975a7d3840a141b0a20a9dc60e3da6cd - default default] CPU mode 'host-model' model '' was chosen, with extra flags: '' _get_guest_cpu_model_config /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:3909 2020-04-16 16:12:28.338 219284 DEBUG nova.virt.hardware [req-a7ee4c3e-ea3a-4237-ba75-4c85411c9889 cbabd9368dc24fea84fd2e43935fddfa 975a7d3840a141b0a20a9dc60e3da6cd - default default] Getting desirable topologies for flavor Flavor(created_at=2020-03-23T11:20:41Z,deleted=False,deleted_at=None,description=None,disabled=False,ephemeral_gb=0,extra_specs={hw:cpu_policy='dedicated',hw:mem_page_size='1048576'},flavorid='03e45d45-f4f4-4c24-8b70-678c3703402f',id=102,is_public=False,memory_mb=49152,name='dmdc-traffic-flavor',projects=<?>,root_gb=60,rxtx_factor=1.0,swap=0,updated_at=None,vcpu_weight=0,vcpus=16) and image_meta ImageMeta(checksum='69f8c18e59db9669d669d04824507a82',container_format='bare',created_at=2020-03-03T06:07:18Z,direct_url=<?>,disk_format='qcow2',id=d31e39bc-c2b7-42ad-968f-7e782dd72943,min_disk=0,min_ram=0,name='vnf-dmdc-18.5.0',owner='36c70ae400e74fc2859f44815d0c9afb',properties=ImageMetaProps,protected=<?>,size=5569380352,status='active',tags=<?>,updated_at=2020-03-03T06:08:03Z,virtual_size=<?>,visibility=<?>), allow threads: True _get_desirable_cpu_topologies /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/virt/hardware.py:551 2020-04-16 16:12:28.339 219284 DEBUG nova.virt.hardware [req-a7ee4c3e-ea3a-4237-ba75-4c85411c9889 cbabd9368dc24fea84fd2e43935fddfa 975a7d3840a141b0a20a9dc60e3da6cd - default default] Flavor limits 65536:65536:65536 _get_cpu_topology_constraints /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/virt/hardware.py:297 2020-04-16 16:12:28.339 219284 DEBUG nova.virt.hardware [req-a7ee4c3e-ea3a-4237-ba75-4c85411c9889 cbabd9368dc24fea84fd2e43935fddfa 975a7d3840a141b0a20a9dc60e3da6cd - default default] Image limits 65536:65536:65536 _get_cpu_topology_constraints /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/virt/hardware.py:308 2020-04-16 16:12:28.339 219284 DEBUG nova.virt.hardware [req-a7ee4c3e-ea3a-4237-ba75-4c85411c9889 cbabd9368dc24fea84fd2e43935fddfa 975a7d3840a141b0a20a9dc60e3da6cd - default default] Flavor pref -1:-1:-1 _get_cpu_topology_constraints /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/virt/hardware.py:331 2020-04-16 16:12:28.340 219284 DEBUG nova.virt.hardware [req-a7ee4c3e-ea3a-4237-ba75-4c85411c9889 cbabd9368dc24fea84fd2e43935fddfa 975a7d3840a141b0a20a9dc60e3da6cd - default default] Image pref -1:-1:-1 _get_cpu_topology_constraints /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/virt/hardware.py:350 2020-04-16 16:12:28.340 219284 DEBUG nova.virt.hardware [req-a7ee4c3e-ea3a-4237-ba75-4c85411c9889 cbabd9368dc24fea84fd2e43935fddfa 975a7d3840a141b0a20a9dc60e3da6cd - default default] Chosen -1:-1:-1 limits 65536:65536:65536 _get_cpu_topology_constraints /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/virt/hardware.py:379 2020-04-16 16:12:28.340 219284 DEBUG nova.virt.hardware [req-a7ee4c3e-ea3a-4237-ba75-4c85411c9889 cbabd9368dc24fea84fd2e43935fddfa 975a7d3840a141b0a20a9dc60e3da6cd - default default] Topology preferred VirtCPUTopology(cores=-1,sockets=-1,threads=-1), maximum VirtCPUTopology(cores=65536,sockets=65536,threads=65536) _get_desirable_cpu_topologies /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/virt/hardware.py:555 2020-04-16 16:12:28.340 219284 DEBUG nova.virt.hardware [req-a7ee4c3e-ea3a-4237-ba75-4c85411c9889 cbabd9368dc24fea84fd2e43935fddfa 975a7d3840a141b0a20a9dc60e3da6cd - default default] Build topologies for 16 vcpu(s) 16:16:16 _get_possible_cpu_topologies /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/virt/hardware.py:418 2020-04-16 16:12:28.341 219284 DEBUG nova.virt.hardware [req-a7ee4c3e-ea3a-4237-ba75-4c85411c9889 cbabd9368dc24fea84fd2e43935fddfa 975a7d3840a141b0a20a9dc60e3da6cd - default default] Got 15 possible topologies _get_possible_cpu_topologies /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/virt/hardware.py:445 2020-04-16 16:12:28.341 219284 DEBUG nova.virt.hardware [req-a7ee4c3e-ea3a-4237-ba75-4c85411c9889 cbabd9368dc24fea84fd2e43935fddfa 975a7d3840a141b0a20a9dc60e3da6cd - default default] Possible topologies [VirtCPUTopology(cores=1,sockets=16,threads=1), VirtCPUTopology(cores=2,sockets=8,threads=1), VirtCPUTopology(cores=4,sockets=4,threads=1), VirtCPUTopology(cores=8,sockets=2,threads=1), VirtCPUTopology(cores=16,sockets=1,threads=1), VirtCPUTopology(cores=1,sockets=8,threads=2), VirtCPUTopology(cores=2,sockets=4,threads=2), VirtCPUTopology(cores=4,sockets=2,threads=2), VirtCPUTopology(cores=8,sockets=1,threads=2), VirtCPUTopology(cores=1,sockets=4,threads=4), VirtCPUTopology(cores=2,sockets=2,threads=4), VirtCPUTopology(cores=4,sockets=1,threads=4), VirtCPUTopology(cores=1,sockets=2,threads=8), VirtCPUTopology(cores=2,sockets=1,threads=8), VirtCPUTopology(cores=1,sockets=1,threads=16)] _get_desirable_cpu_topologies /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/virt/hardware.py:560 2020-04-16 16:12:28.341 219284 DEBUG nova.virt.hardware [req-a7ee4c3e-ea3a-4237-ba75-4c85411c9889 cbabd9368dc24fea84fd2e43935fddfa 975a7d3840a141b0a20a9dc60e3da6cd - default default] Filtering topologies best for 2 threads _get_desirable_cpu_topologies /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/virt/hardware.py:578 2020-04-16 16:12:28.342 219284 DEBUG nova.virt.hardware [req-a7ee4c3e-ea3a-4237-ba75-4c85411c9889 cbabd9368dc24fea84fd2e43935fddfa 975a7d3840a141b0a20a9dc60e3da6cd - default default] Remaining possible topologies [VirtCPUTopology(cores=1,sockets=8,threads=2), VirtCPUTopology(cores=2,sockets=4,threads=2), VirtCPUTopology(cores=4,sockets=2,threads=2), VirtCPUTopology(cores=8,sockets=1,threads=2)] _get_desirable_cpu_topologies /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/virt/hardware.py:583 2020-04-16 16:12:28.342 219284 DEBUG nova.virt.hardware [req-a7ee4c3e-ea3a-4237-ba75-4c85411c9889 cbabd9368dc24fea84fd2e43935fddfa 975a7d3840a141b0a20a9dc60e3da6cd - default default] Sorted desired topologies [VirtCPUTopology(cores=1,sockets=8,threads=2), VirtCPUTopology(cores=2,sockets=4,threads=2), VirtCPUTopology(cores=4,sockets=2,threads=2), VirtCPUTopology(cores=8,sockets=1,threads=2)] _get_desirable_cpu_topologies /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/virt/hardware.py:586 2020-04-16 16:12:28.344 219284 DEBUG nova.virt.libvirt.driver [req-48d96e5d-f071-44e7-94d2-e9fcb2a13087 cbabd9368dc24fea84fd2e43935fddfa 975a7d3840a141b0a20a9dc60e3da6cd - default default] CPU mode 'host-model' model '' was chosen, with extra flags: '' _get_guest_cpu_model_config /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:3909 2020-04-16 16:12:28.345 219284 DEBUG nova.virt.hardware [req-48d96e5d-f071-44e7-94d2-e9fcb2a13087 cbabd9368dc24fea84fd2e43935fddfa 975a7d3840a141b0a20a9dc60e3da6cd - default default] Getting desirable topologies for flavor Flavor(created_at=2020-03-23T11:20:34Z,deleted=False,deleted_at=None,description=None,disabled=False,ephemeral_gb=0,extra_specs={hw:cpu_policy='dedicated',hw:mem_page_size='1048576'},flavorid='d60b66d4-c0e0-4292-9113-1df2d94d57a5',id=90,is_public=False,memory_mb=57344,name='scef-traffic-flavor',projects=<?>,root_gb=60,rxtx_factor=1.0,swap=0,updated_at=None,vcpu_weight=0,vcpus=16) and image_meta ImageMeta(checksum='3fe0e06194e0b5327ba38bb2367f760d',container_format='bare',created_at=2020-03-03T06:04:55Z,direct_url=<?>,disk_format='qcow2',id=c255bbbc-c8c3-462e-b827-1d35db08d283,min_disk=0,min_ram=0,name='vnf-scef-18.5',owner='36c70ae400e74fc2859f44815d0c9afb',properties=ImageMetaProps,protected=<?>,size=7143292928,status='active',tags=<?>,updated_at=2020-03-03T06:05:49Z,virtual_size=<?>,visibility=<?>), allow threads: True _get_desirable_cpu_topologies /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/virt/hardware.py:551 2020-04-16 16:12:28.345 219284 DEBUG nova.virt.hardware [req-48d96e5d-f071-44e7-94d2-e9fcb2a13087 cbabd9368dc24fea84fd2e43935fddfa 975a7d3840a141b0a20a9dc60e3da6cd - default default] Flavor limits 65536:65536:65536 _get_cpu_topology_constraints /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/virt/hardware.py:297 2020-04-16 16:12:28.345 219284 DEBUG nova.virt.hardware [req-48d96e5d-f071-44e7-94d2-e9fcb2a13087 cbabd9368dc24fea84fd2e43935fddfa 975a7d3840a141b0a20a9dc60e3da6cd - default default] Image limits 65536:65536:65536 _get_cpu_topology_constraints /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/virt/hardware.py:308 2020-04-16 16:12:28.346 219284 DEBUG nova.virt.hardware [req-48d96e5d-f071-44e7-94d2-e9fcb2a13087 cbabd9368dc24fea84fd2e43935fddfa 975a7d3840a141b0a20a9dc60e3da6cd - default default] Flavor pref -1:-1:-1 _get_cpu_topology_constraints /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/virt/hardware.py:331 2020-04-16 16:12:28.346 219284 DEBUG nova.virt.hardware [req-48d96e5d-f071-44e7-94d2-e9fcb2a13087 cbabd9368dc24fea84fd2e43935fddfa 975a7d3840a141b0a20a9dc60e3da6cd - default default] Image pref -1:-1:-1 _get_cpu_topology_constraints /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/virt/hardware.py:350 2020-04-16 16:12:28.346 219284 DEBUG nova.virt.hardware [req-48d96e5d-f071-44e7-94d2-e9fcb2a13087 cbabd9368dc24fea84fd2e43935fddfa 975a7d3840a141b0a20a9dc60e3da6cd - default default] Chosen -1:-1:-1 limits 65536:65536:65536 _get_cpu_topology_constraints /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/virt/hardware.py:379 packages/nova/network/base_api.py:48 2020-04-16 16:34:48.580 219284 DEBUG oslo_concurrency.lockutils [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] Releasing semaphore "refresh_cache-f33b2602-ac5f-491e-bdb8-7e7f9376bcad" lock /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:228 2020-04-16 16:34:48.580 219284 DEBUG nova.compute.manager [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] [instance: f33b2602-ac5f-491e-bdb8-7e7f9376bcad] Updated the network info_cache for instance _heal_instance_info_cache /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/compute/manager.py:6827 2020-04-16 16:34:50.580 219284 DEBUG oslo_service.periodic_task [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] Running periodic task ComputeManager._run_image_cache_manager_pass run_periodic_tasks /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 2020-04-16 16:34:50.581 219284 DEBUG oslo_concurrency.lockutils [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] Lock "storage-registry-lock" acquired by "nova.virt.storage_users.do_register_storage_use" :: waited 0.000s inner /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:273 2020-04-16 16:34:50.581 219284 DEBUG oslo_concurrency.lockutils [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] Lock "storage-registry-lock" released by "nova.virt.storage_users.do_register_storage_use" :: held 0.000s inner /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:285 2020-04-16 16:34:50.581 219284 DEBUG oslo_concurrency.lockutils [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] Lock "storage-registry-lock" acquired by "nova.virt.storage_users.do_get_storage_users" :: waited 0.000s inner /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:273 2020-04-16 16:34:50.582 219284 DEBUG oslo_concurrency.lockutils [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] Lock "storage-registry-lock" released by "nova.virt.storage_users.do_get_storage_users" :: held 0.000s inner /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:285 2020-04-16 16:34:50.628 219284 DEBUG nova.virt.libvirt.imagecache [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] Verify base images _age_and_verify_cached_images /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/virt/libvirt/imagecache.py:348 2020-04-16 16:34:50.628 219284 DEBUG nova.virt.libvirt.imagecache [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] Image id yields fingerprint da39a3ee5e6b4b0d3255bfef95601890afd80709 _age_and_verify_cached_images /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/virt/libvirt/imagecache.py:355 2020-04-16 16:34:50.628 219284 DEBUG nova.virt.libvirt.imagecache [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] Image id b8783f95-138b-4265-a09d-55ec9d9ad35d yields fingerprint b40b27e04896d063bc591b19642da8910da3eb1f _age_and_verify_cached_images /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/virt/libvirt/imagecache.py:355 2020-04-16 16:34:50.628 219284 INFO nova.virt.libvirt.imagecache [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] image b8783f95-138b-4265-a09d-55ec9d9ad35d at (/var/lib/nova/instances/_base/b40b27e04896d063bc591b19642da8910da3eb1f): checking 2020-04-16 16:34:50.628 219284 DEBUG nova.virt.libvirt.imagecache [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] image b8783f95-138b-4265-a09d-55ec9d9ad35d at (/var/lib/nova/instances/_base/b40b27e04896d063bc591b19642da8910da3eb1f): image is in use _mark_in_use /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/virt/libvirt/imagecache.py:329 2020-04-16 16:34:50.629 219284 DEBUG nova.virt.libvirt.imagecache [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] Image id c255bbbc-c8c3-462e-b827-1d35db08d283 yields fingerprint 86692cd1e738b8df7cf1f951967c61e92222fc4c _age_and_verify_cached_images /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/virt/libvirt/imagecache.py:355 2020-04-16 16:34:50.630 219284 INFO nova.virt.libvirt.imagecache [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] image c255bbbc-c8c3-462e-b827-1d35db08d283 at (/var/lib/nova/instances/_base/86692cd1e738b8df7cf1f951967c61e92222fc4c): checking 2020-04-16 16:34:50.630 219284 DEBUG nova.virt.libvirt.imagecache [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] image c255bbbc-c8c3-462e-b827-1d35db08d283 at (/var/lib/nova/instances/_base/86692cd1e738b8df7cf1f951967c61e92222fc4c): image is in use _mark_in_use /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/virt/libvirt/imagecache.py:329 2020-04-16 16:34:50.630 219284 DEBUG nova.virt.libvirt.imagecache [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] Image id d31e39bc-c2b7-42ad-968f-7e782dd72943 yields fingerprint 5c538ead16d8375e4890e8b9bb1aa080edc75f33 _age_and_verify_cached_images /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/virt/libvirt/imagecache.py:355 2020-04-16 16:34:50.630 219284 INFO nova.virt.libvirt.imagecache [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] image d31e39bc-c2b7-42ad-968f-7e782dd72943 at (/var/lib/nova/instances/_base/5c538ead16d8375e4890e8b9bb1aa080edc75f33): checking 2020-04-16 16:34:50.630 219284 DEBUG nova.virt.libvirt.imagecache [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] image d31e39bc-c2b7-42ad-968f-7e782dd72943 at (/var/lib/nova/instances/_base/5c538ead16d8375e4890e8b9bb1aa080edc75f33): image is in use _mark_in_use /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/virt/libvirt/imagecache.py:329 2020-04-16 16:34:50.631 219284 DEBUG nova.virt.libvirt.imagecache [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] Image id b3af2bf0-055b-48fb-aedc-4683468a3f74 yields fingerprint 7af98c4d49b766d82eec8169a5c87be4eb56e5eb _age_and_verify_cached_images /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/virt/libvirt/imagecache.py:355 2020-04-16 16:34:50.631 219284 INFO nova.virt.libvirt.imagecache [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] image b3af2bf0-055b-48fb-aedc-4683468a3f74 at (/var/lib/nova/instances/_base/7af98c4d49b766d82eec8169a5c87be4eb56e5eb): checking 2020-04-16 16:34:50.631 219284 DEBUG nova.virt.libvirt.imagecache [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] image b3af2bf0-055b-48fb-aedc-4683468a3f74 at (/var/lib/nova/instances/_base/7af98c4d49b766d82eec8169a5c87be4eb56e5eb): image is in use _mark_in_use /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/virt/libvirt/imagecache.py:329 2020-04-16 16:34:50.632 219284 DEBUG nova.virt.libvirt.imagecache [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] f33b2602-ac5f-491e-bdb8-7e7f9376bcad is a valid instance name _list_backing_images /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/virt/libvirt/imagecache.py:169 2020-04-16 16:34:50.632 219284 DEBUG nova.virt.libvirt.imagecache [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] f33b2602-ac5f-491e-bdb8-7e7f9376bcad has a disk file _list_backing_images /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/virt/libvirt/imagecache.py:172 2020-04-16 16:34:50.632 219284 DEBUG oslo_concurrency.processutils [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] Running cmd (subprocess): /openstack/venvs/nova-17.1.12/bin/python -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info /var/lib/nova/instances/f33b2602-ac5f-491e-bdb8-7e7f9376bcad/disk --force-share execute /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/oslo_concurrency/processutils.py:372 2020-04-16 16:34:50.663 219284 DEBUG oslo_concurrency.processutils [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] CMD "/openstack/venvs/nova-17.1.12/bin/python -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info /var/lib/nova/instances/f33b2602-ac5f-491e-bdb8-7e7f9376bcad/disk --force-share" returned: 0 in 0.031s execute /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/oslo_concurrency/processutils.py:409 2020-04-16 16:34:50.664 219284 DEBUG nova.virt.libvirt.imagecache [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] Instance f33b2602-ac5f-491e-bdb8-7e7f9376bcad is backed by b40b27e04896d063bc591b19642da8910da3eb1f _list_backing_images /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/virt/libvirt/imagecache.py:187 2020-04-16 16:34:50.665 219284 DEBUG nova.virt.libvirt.imagecache [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] f117eb96-06a9-4c91-9c5c-111228e24d66 is a valid instance name _list_backing_images /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/virt/libvirt/imagecache.py:169 2020-04-16 16:34:50.665 219284 DEBUG nova.virt.libvirt.imagecache [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] f117eb96-06a9-4c91-9c5c-111228e24d66 has a disk file _list_backing_images /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/virt/libvirt/imagecache.py:172 2020-04-16 16:34:50.665 219284 DEBUG oslo_concurrency.processutils [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] Running cmd (subprocess): /openstack/venvs/nova-17.1.12/bin/python -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info /var/lib/nova/instances/f117eb96-06a9-4c91-9c5c-111228e24d66/disk --force-share execute /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/oslo_concurrency/processutils.py:372 2020-04-16 16:34:50.694 219284 DEBUG oslo_concurrency.processutils [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] CMD "/openstack/venvs/nova-17.1.12/bin/python -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info /var/lib/nova/instances/f117eb96-06a9-4c91-9c5c-111228e24d66/disk --force-share" returned: 0 in 0.029s execute /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/oslo_concurrency/processutils.py:409 2020-04-16 16:34:50.695 219284 DEBUG nova.virt.libvirt.imagecache [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] Instance f117eb96-06a9-4c91-9c5c-111228e24d66 is backed by 7af98c4d49b766d82eec8169a5c87be4eb56e5eb _list_backing_images /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/virt/libvirt/imagecache.py:187 2020-04-16 16:34:50.695 219284 DEBUG nova.virt.libvirt.imagecache [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] 5ba39de3-f5f8-46a2-908d-c43b901e1696 is a valid instance name _list_backing_images /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/virt/libvirt/imagecache.py:169 2020-04-16 16:34:50.695 219284 DEBUG nova.virt.libvirt.imagecache [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] 5ba39de3-f5f8-46a2-908d-c43b901e1696 has a disk file _list_backing_images /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/virt/libvirt/imagecache.py:172 2020-04-16 16:34:50.695 219284 DEBUG oslo_concurrency.processutils [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] Running cmd (subprocess): /openstack/venvs/nova-17.1.12/bin/python -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info /var/lib/nova/instances/5ba39de3-f5f8-46a2-908d-c43b901e1696/disk --force-share execute /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/oslo_concurrency/processutils.py:372 2020-04-16 16:34:50.723 219284 DEBUG oslo_concurrency.processutils [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] CMD "/openstack/venvs/nova-17.1.12/bin/python -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info /var/lib/nova/instances/5ba39de3-f5f8-46a2-908d-c43b901e1696/disk --force-share" returned: 0 in 0.028s execute /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/oslo_concurrency/processutils.py:409 2020-04-16 16:34:50.724 219284 DEBUG nova.virt.libvirt.imagecache [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] Instance 5ba39de3-f5f8-46a2-908d-c43b901e1696 is backed by 86692cd1e738b8df7cf1f951967c61e92222fc4c _list_backing_images /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/virt/libvirt/imagecache.py:187 2020-04-16 16:34:50.724 219284 DEBUG nova.virt.libvirt.imagecache [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] d3d2837b-49c3-4822-b26b-4b3c03d344ae is a valid instance name _list_backing_images /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/virt/libvirt/imagecache.py:169 2020-04-16 16:34:50.724 219284 DEBUG nova.virt.libvirt.imagecache [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] d3d2837b-49c3-4822-b26b-4b3c03d344ae has a disk file _list_backing_images /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/virt/libvirt/imagecache.py:172 2020-04-16 16:34:50.725 219284 DEBUG oslo_concurrency.processutils [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] Running cmd (subprocess): /openstack/venvs/nova-17.1.12/bin/python -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info /var/lib/nova/instances/d3d2837b-49c3-4822-b26b-4b3c03d344ae/disk --force-share execute /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/oslo_concurrency/processutils.py:372 2020-04-16 16:34:50.752 219284 DEBUG oslo_concurrency.processutils [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] CMD "/openstack/venvs/nova-17.1.12/bin/python -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info /var/lib/nova/instances/d3d2837b-49c3-4822-b26b-4b3c03d344ae/disk --force-share" returned: 0 in 0.028s execute /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/oslo_concurrency/processutils.py:409 2020-04-16 16:34:50.753 219284 DEBUG nova.virt.libvirt.imagecache [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] Instance d3d2837b-49c3-4822-b26b-4b3c03d344ae is backed by 5c538ead16d8375e4890e8b9bb1aa080edc75f33 _list_backing_images /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/virt/libvirt/imagecache.py:187 2020-04-16 16:34:50.753 219284 DEBUG nova.virt.libvirt.imagecache [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] dfa80e78-ee02-46e5-ba7a-0874fa37da56 is a valid instance name _list_backing_images /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/virt/libvirt/imagecache.py:169 2020-04-16 16:34:50.754 219284 DEBUG nova.virt.libvirt.imagecache [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] dfa80e78-ee02-46e5-ba7a-0874fa37da56 has a disk file _list_backing_images /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/virt/libvirt/imagecache.py:172 2020-04-16 16:34:50.754 219284 DEBUG oslo_concurrency.processutils [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] Running cmd (subprocess): /openstack/venvs/nova-17.1.12/bin/python -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info /var/lib/nova/instances/dfa80e78-ee02-46e5-ba7a-0874fa37da56/disk --force-share execute /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/oslo_concurrency/processutils.py:372 2020-04-16 16:34:50.781 219284 DEBUG oslo_concurrency.processutils [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] CMD "/openstack/venvs/nova-17.1.12/bin/python -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info /var/lib/nova/instances/dfa80e78-ee02-46e5-ba7a-0874fa37da56/disk --force-share" returned: 0 in 0.027s execute /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/oslo_concurrency/processutils.py:409 2020-04-16 16:34:50.782 219284 DEBUG nova.virt.libvirt.imagecache [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] Instance dfa80e78-ee02-46e5-ba7a-0874fa37da56 is backed by 86692cd1e738b8df7cf1f951967c61e92222fc4c _list_backing_images /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/virt/libvirt/imagecache.py:187 2020-04-16 16:34:50.782 219284 INFO nova.virt.libvirt.imagecache [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] Active base files: /var/lib/nova/instances/_base/b40b27e04896d063bc591b19642da8910da3eb1f /var/lib/nova/instances/_base/86692cd1e738b8df7cf1f951967c61e92222fc4c /var/lib/nova/instances/_base/5c538ead16d8375e4890e8b9bb1aa080edc75f33 /var/lib/nova/instances/_base/7af98c4d49b766d82eec8169a5c87be4eb56e5eb 2020-04-16 16:34:50.783 219284 DEBUG nova.virt.libvirt.imagecache [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] Verification complete _age_and_verify_cached_images /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/virt/libvirt/imagecache.py:384 2020-04-16 16:34:50.783 219284 DEBUG nova.virt.libvirt.imagecache [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] Verify swap images _age_and_verify_swap_images /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/virt/libvirt/imagecache.py:333 2020-04-16 16:35:01.887 219284 DEBUG oslo_service.periodic_task [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] Running periodic task ComputeManager.update_available_resource run_periodic_tasks /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 2020-04-16 16:35:01.910 219284 DEBUG nova.compute.resource_tracker [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] Auditing locally available compute resources for KO1A3D02O131106CM07 (node: KO1A3D02O131106CM07.openstack.local) update_available_resource /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/compute/resource_tracker.py:689 2020-04-16 16:35:02.009 219284 DEBUG oslo_concurrency.processutils [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] Running cmd (subprocess): /openstack/venvs/nova-17.1.12/bin/python -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info /var/lib/nova/instances/f33b2602-ac5f-491e-bdb8-7e7f9376bcad/disk --force-share execute /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/oslo_concurrency/processutils.py:372 2020-04-16 16:35:02.040 219284 DEBUG oslo_concurrency.processutils [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] CMD "/openstack/venvs/nova-17.1.12/bin/python -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info /var/lib/nova/instances/f33b2602-ac5f-491e-bdb8-7e7f9376bcad/disk --force-share" returned: 0 in 0.031s execute /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/oslo_concurrency/processutils.py:409 2020-04-16 16:35:02.041 219284 DEBUG oslo_concurrency.processutils [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] Running cmd (subprocess): /openstack/venvs/nova-17.1.12/bin/python -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info /var/lib/nova/instances/f33b2602-ac5f-491e-bdb8-7e7f9376bcad/disk --force-share execute /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/oslo_concurrency/processutils.py:372 2020-04-16 16:35:02.070 219284 DEBUG oslo_concurrency.processutils [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] CMD "/openstack/venvs/nova-17.1.12/bin/python -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info /var/lib/nova/instances/f33b2602-ac5f-491e-bdb8-7e7f9376bcad/disk --force-share" returned: 0 in 0.029s execute /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/oslo_concurrency/processutils.py:409 2020-04-16 16:35:02.070 219284 DEBUG nova.virt.libvirt.driver [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] skipping disk for instance-00000636 as it does not have a path _get_instance_disk_info_from_config /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:7840 2020-04-16 16:35:02.071 219284 DEBUG nova.virt.libvirt.driver [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] skipping disk for instance-00000636 as it does not have a path _get_instance_disk_info_from_config /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:7840 2020-04-16 16:35:02.073 219284 DEBUG oslo_concurrency.processutils [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] Running cmd (subprocess): /openstack/venvs/nova-17.1.12/bin/python -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info /var/lib/nova/instances/d3d2837b-49c3-4822-b26b-4b3c03d344ae/disk --force-share execute /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/oslo_concurrency/processutils.py:372 2020-04-16 16:35:02.101 219284 DEBUG oslo_concurrency.processutils [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] CMD "/openstack/venvs/nova-17.1.12/bin/python -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info /var/lib/nova/instances/d3d2837b-49c3-4822-b26b-4b3c03d344ae/disk --force-share" returned: 0 in 0.028s execute /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/oslo_concurrency/processutils.py:409 2020-04-16 16:35:02.101 219284 DEBUG oslo_concurrency.processutils [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] Running cmd (subprocess): /openstack/venvs/nova-17.1.12/bin/python -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info /var/lib/nova/instances/d3d2837b-49c3-4822-b26b-4b3c03d344ae/disk --force-share execute /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/oslo_concurrency/processutils.py:372 2020-04-16 16:35:02.129 219284 DEBUG oslo_concurrency.processutils [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] CMD "/openstack/venvs/nova-17.1.12/bin/python -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info /var/lib/nova/instances/d3d2837b-49c3-4822-b26b-4b3c03d344ae/disk --force-share" returned: 0 in 0.028s execute /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/oslo_concurrency/processutils.py:409 2020-04-16 16:35:02.132 219284 DEBUG oslo_concurrency.processutils [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] Running cmd (subprocess): /openstack/venvs/nova-17.1.12/bin/python -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info /var/lib/nova/instances/dfa80e78-ee02-46e5-ba7a-0874fa37da56/disk --force-share execute /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/oslo_concurrency/processutils.py:372 2020-04-16 16:35:02.159 219284 DEBUG oslo_concurrency.processutils [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] CMD "/openstack/venvs/nova-17.1.12/bin/python -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info /var/lib/nova/instances/dfa80e78-ee02-46e5-ba7a-0874fa37da56/disk --force-share" returned: 0 in 0.028s execute /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/oslo_concurrency/processutils.py:409 2020-04-16 16:35:02.160 219284 DEBUG oslo_concurrency.processutils [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] Running cmd (subprocess): /openstack/venvs/nova-17.1.12/bin/python -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info /var/lib/nova/instances/dfa80e78-ee02-46e5-ba7a-0874fa37da56/disk --force-share execute /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/oslo_concurrency/processutils.py:372 2020-04-16 16:35:02.187 219284 DEBUG oslo_concurrency.processutils [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] CMD "/openstack/venvs/nova-17.1.12/bin/python -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info /var/lib/nova/instances/dfa80e78-ee02-46e5-ba7a-0874fa37da56/disk --force-share" returned: 0 in 0.027s execute /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/oslo_concurrency/processutils.py:409 2020-04-16 16:35:02.190 219284 DEBUG oslo_concurrency.processutils [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] Running cmd (subprocess): /openstack/venvs/nova-17.1.12/bin/python -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info /var/lib/nova/instances/f117eb96-06a9-4c91-9c5c-111228e24d66/disk --force-share execute /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/oslo_concurrency/processutils.py:372 2020-04-16 16:35:02.217 219284 DEBUG oslo_concurrency.processutils [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] CMD "/openstack/venvs/nova-17.1.12/bin/python -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info /var/lib/nova/instances/f117eb96-06a9-4c91-9c5c-111228e24d66/disk --force-share" returned: 0 in 0.027s execute /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/oslo_concurrency/processutils.py:409 2020-04-16 16:35:02.218 219284 DEBUG oslo_concurrency.processutils [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] Running cmd (subprocess): /openstack/venvs/nova-17.1.12/bin/python -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info /var/lib/nova/instances/f117eb96-06a9-4c91-9c5c-111228e24d66/disk --force-share execute /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/oslo_concurrency/processutils.py:372 2020-04-16 16:35:02.245 219284 DEBUG oslo_concurrency.processutils [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] CMD "/openstack/venvs/nova-17.1.12/bin/python -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info /var/lib/nova/instances/f117eb96-06a9-4c91-9c5c-111228e24d66/disk --force-share" returned: 0 in 0.027s execute /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/oslo_concurrency/processutils.py:409 2020-04-16 16:35:02.247 219284 DEBUG oslo_concurrency.processutils [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] Running cmd (subprocess): /openstack/venvs/nova-17.1.12/bin/python -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info /var/lib/nova/instances/5ba39de3-f5f8-46a2-908d-c43b901e1696/disk --force-share execute /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/oslo_concurrency/processutils.py:372 2020-04-16 16:35:02.274 219284 DEBUG oslo_concurrency.processutils [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] CMD "/openstack/venvs/nova-17.1.12/bin/python -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info /var/lib/nova/instances/5ba39de3-f5f8-46a2-908d-c43b901e1696/disk --force-share" returned: 0 in 0.027s execute /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/oslo_concurrency/processutils.py:409 2020-04-16 16:35:02.275 219284 DEBUG oslo_concurrency.processutils [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] Running cmd (subprocess): /openstack/venvs/nova-17.1.12/bin/python -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info /var/lib/nova/instances/5ba39de3-f5f8-46a2-908d-c43b901e1696/disk --force-share execute /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/oslo_concurrency/processutils.py:372 2020-04-16 16:35:02.302 219284 DEBUG oslo_concurrency.processutils [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] CMD "/openstack/venvs/nova-17.1.12/bin/python -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info /var/lib/nova/instances/5ba39de3-f5f8-46a2-908d-c43b901e1696/disk --force-share" returned: 0 in 0.027s execute /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/oslo_concurrency/processutils.py:409 2020-04-16 16:35:02.669 219284 DEBUG nova.compute.resource_tracker [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] Hypervisor/Node resource view: name=KO1A3D02O131106CM07.openstack.local free_ram=72406MB free_disk=523GB free_vcpus=10 pci_devices=[{"dev_id": "pci_0000_3a_0a_7", "product_id": "2047", /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/compute/resource_tracker.py:828 2020-04-16 16:35:02.670 219284 DEBUG oslo_concurrency.lockutils [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] Lock "compute_resources" acquired by "nova.compute.resource_tracker._update_available_resource" :: waited 0.000s inner /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:273 2020-04-16 16:35:02.729 219284 DEBUG nova.compute.resource_tracker [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] Compute driver doesn't require allocation refresh and we're on a compute host in a deployment that only has compute hosts with Nova versions >=16 (Pike). Skipping auto-correction of allocations. _update_usage_from_instances /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/compute/resource_tracker.py:1247 2020-04-16 16:35:02.784 219284 DEBUG nova.compute.resource_tracker [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] Instance 5ba39de3-f5f8-46a2-908d-c43b901e1696 actively managed on this compute host and has allocations in placement: {u'resources': {u'VCPU': 16, u'MEMORY_MB': 57344, u'DISK_GB': 60}}. _remove_deleted_instances_allocations /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/compute/resource_tracker.py:1269 2020-04-16 16:35:02.785 219284 DEBUG nova.compute.resource_tracker [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] Instance dfa80e78-ee02-46e5-ba7a-0874fa37da56 actively managed on this compute host and has allocations in placement: {u'resources': {u'VCPU': 12, u'MEMORY_MB': 24576, u'DISK_GB': 60}}. _remove_deleted_instances_allocations /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/compute/resource_tracker.py:1269 2020-04-16 16:35:02.785 219284 DEBUG nova.compute.resource_tracker [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] Instance f33b2602-ac5f-491e-bdb8-7e7f9376bcad actively managed on this compute host and has allocations in placement: {u'resources': {u'VCPU': 16, u'MEMORY_MB': 49152, u'DISK_GB': 40}}. _remove_deleted_instances_allocations /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/compute/resource_tracker.py:1269 2020-04-16 16:35:02.785 219284 DEBUG nova.compute.resource_tracker [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] Instance d3d2837b-49c3-4822-b26b-4b3c03d344ae actively managed on this compute host and has allocations in placement: {u'resources': {u'VCPU': 16, u'MEMORY_MB': 49152, u'DISK_GB': 60}}. _remove_deleted_instances_allocations /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/compute/resource_tracker.py:1269 2020-04-16 16:35:02.785 219284 DEBUG nova.compute.resource_tracker [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] Instance f117eb96-06a9-4c91-9c5c-111228e24d66 actively managed on this compute host and has allocations in placement: {u'resources': {u'VCPU': 2, u'MEMORY_MB': 4096, u'DISK_GB': 20}}. _remove_deleted_instances_allocations /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/compute/resource_tracker.py:1269 2020-04-16 16:35:02.785 219284 DEBUG nova.compute.resource_tracker [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] Total usable vcpus: 72, total allocated vcpus: 62 _report_final_resource_view /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/compute/resource_tracker.py:844 2020-04-16 16:35:02.786 219284 INFO nova.compute.resource_tracker [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] Final resource view: name=KO1A3D02O131106CM07.openstack.local phys_ram=385391MB used_ram=192512MB phys_disk=548GB used_disk=250GB total_vcpus=72 used_vcpus=62 pci_stats=[] 2020-04-16 16:35:02.814 219284 DEBUG nova.compute.resource_tracker [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] Compute_service record updated for KO1A3D02O131106CM07:KO1A3D02O131106CM07.openstack.local _update_available_resource /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/compute/resource_tracker.py:784 2020-04-16 16:35:02.814 219284 DEBUG oslo_concurrency.lockutils [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] Lock "compute_resources" released by "nova.compute.resource_tracker._update_available_resource" :: held 0.144s inner /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:285 2020-04-16 16:35:37.612 219284 DEBUG oslo_service.periodic_task [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] Running periodic task ComputeManager._reclaim_queued_deletes run_periodic_tasks /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 2020-04-16 16:35:37.613 219284 DEBUG nova.compute.manager [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] CONF.reclaim_instance_interval <= 0, skipping... _reclaim_queued_deletes /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/compute/manager.py:7438 2020-04-16 16:35:38.685 219284 DEBUG oslo_service.periodic_task [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] Running periodic task ComputeManager._poll_rebooting_instances run_periodic_tasks /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 2020-04-16 16:35:39.685 219284 DEBUG oslo_service.periodic_task [req-dd4a2032-bbbd-4c0b-87ac-11605ffbf6c2 - - - - -] Running periodic task ComputeManager._instance_usage_audit run_periodic_tasks /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 -- Best Regards Pradeep Kumar

4 8

[nova] [libvirt] New libvirt subteam
by Sylvain Bauza 23 Apr '20

23 Apr '20

Hi, As proposed and agreed during the last Nova meeting [1], a new libvirt subteam is about to be created. The overall idea of this subteam is to gather contributors from various companies working on the same Nova component (here, libvirt) and help them sharing their efforts. A common pattern is to have all the subteam members to actively engage themselves on reviewing changes from the other subteam members. See the Nova subteam principles [2] for more details. If you feel interested with this idea and you'd like to join this, I just created a new etherpad for tracking contributors and review requests : https://etherpad.opendev.org/p/nova-libvirt-subteam Feel free to put your name on the board and please also provide your timezone. Ideally, I'd love to have some kind of IRC discussion between all subteam members so let's see who's in and then when we can discuss. I'll just sit around in the nova channel if you have questions. Thanks, -Sylvain [1] http://eavesdrop.openstack.org/meetings/nova/2020/nova.2020-04-23-16.00.log… [2] https://wiki.openstack.org/wiki/Nova#Nova_subteams

1 0

[kolla] Kolla klub
by Mark Goddard 23 Apr '20

23 Apr '20

Hi, Thanks to everyone who attended today's Kolla Klub meeting. There is a recording [1] for anyone who missed it. I've updated the agenda [2] for the next meeting to include the case studies agreed today. So far I've got Gaël Therond and Дмитрий Грачев (Dmitry Grachev) signed up to present on 7th May, with a few others listed as interested in presenting during a future session. Please add your name if you are interested. No cloud too small! Please also add meeting topic ideas if you have them. [1] https://drive.google.com/open?id=1TLabWeOK65nTrZ278qN4fWLt0Az_zUFT [2] https://docs.google.com/document/d/1EwQs2GXF-EvJZamEx9vQAOSDB5tCjsDCJyHQN5_… Cheers, Mark

1 0

[ansible][tripleo][openstack-ansible-modules] Virtual PTG planning for Openstack Ansilbe modules
by Sagi Shnaidman 23 Apr '20

23 Apr '20

Hi, all As we discussed in the last meeting, I've created an etherpad for Virtual PTG session of Openstack Ansible modules project. Please feel free to add there topics for discussion and add yourself to a list if you plan to attend: https://etherpad.opendev.org/p/openstack-ansible-modules-ptg Thanks -- Best regards Sagi Shnaidman

1 0

[tempest] regex failing
by Eric Barrera 23 Apr '20

23 Apr '20

Hi, I started seeing an error yesterday when doing this command: tempest UPPER_CONSTRAINTS_FILE= https://releases.openstack.org/constraints/upper/master tox -eall -- baremetal --concurrency=1 The specified regex doesn't match with anythingError: InvocationError for command /opt/stack/new/tempest/.tox/tempest/bin/tempest run --regex baremetal --concurrency=1 Any ideas how could I resolve this? Thanks!

4 7

[qa][ptg] Virtual PTG planning
by Masayuki Igawa 23 Apr '20

23 Apr '20

Hi team, Short summary: 1. Please take a look at the QA PTG etherpad[2] and put your name and topics there 2. Please vote on the doodle poll[3] Details: As we discussed in the office hours, we're now collecting topics[3] for the PTG[1]. And, we need to fix our slots for the PTG. We have now 5 topics[2], and we'll have some more, possibly. So, we'll need 2 or 3 hours slots for 2 days or more if we have more topics. Anyway, please fill out the doodle poll[3] if you'd like to participate in the PTG. And, please add your name and topics to the etherpad so that we can discuss them during the following office hours. [1] http://lists.openstack.org/pipermail/openstack-discuss/2020-April/014126.ht… [2] https://etherpad.opendev.org/p/qa-victoria-ptg [3] https://doodle.com/poll/5awstyshtquuwcq3 -- Masayuki

1 0

[goals][Drop Python 2.7 Support][storlets] Request to hold on dropping python2.7 support in Storlets
by Takashi Kajinami 23 Apr '20

23 Apr '20

Hi, Sorry to be late to raise this up, but I'd like to request to postpone completion of "Dropping Python2.7 Support" work in Storlets during this cycle. Storlets is heavily dependent on swift and swift won't drop python 2.7 support during this cycle. This means that we still expect some users who want to run storlets with their swift running on python2 during Ussuri. Currently we've completed all python3 compatibility work, and py2 tests are already dropped. However we would like to postpone marking storlets as python3 only (and dropping all py2-3 compatibility codes) so that we don't prohibit users from using it in swift with python2. See the following patch to remove python3 only mark at the moment. (It was initially added, but we want to revert it) [1] https://review.opendev.org/#/c/720131/2 Thank you, Takashi -- ---------- Takashi Kajinami Senior Software Maintenance Engineer Customer Experience and Engagement Red Hat e-mail: tkajinam(a)redhat.com

2 8

[all][PTL] U Community Goal: Project PTL & Contrib Docs Update #7
by Kendall Nelson 23 Apr '20

23 Apr '20

Hello Everyone! It looks like a lot of good progress[1] has been made since my last update. Thank you for all the hard work people have been putting into their contributor docs! We still have a number of projects missing any progress though. If you have not started yet, PLEASE do so. If there is anything I can do to help, please let me know ASAP. Also, feel free to add me as a reviewer once you have patches pushed. All told, if you are doing the bare minimum of only filling out the template[2] it shouldn't take more than 5-10 minutes per repo. If you have questions about the goal itself, here is the link to that[3]. And as you push patches, please be sure to update the StoryBoard story[4]. Thanks! -Kendall (diablo_rojo) [1] Progress: https://review.opendev.org/#/q/topic:project-ptl-and-contrib-docs+(status:o… [2] Cookiecutter Template: https://opendev.org/openstack/cookiecutter/src/branch/master/%7b%7bcookiecu… [3] Accepted Goal: https://governance.openstack.org/tc/goals/selected/ussuri/project-ptl-and-c… [4] Task Tracking: https://storyboard.openstack.org/#!/story/2007236

2 2

RH-OSP 13 | Queens | Troubleshooting transient 403 in RBAC networks.
by Laurent Dumont 22 Apr '20

22 Apr '20

Hi! I'm currently troubleshooting some transient Keystone/Neutron 403 issues when creating ports in RBAC-ed networks. - As part of our CI process,we are running a few sanity tests in virtual Openstack setups with 3 controllers and 2 computes. - One of those test validates the following scenarios (everything is done through the Openstack python sdk) : - Create projectA / UserA / ProjectB / UserB - Create Network in project A - RBAC Network in projectA to ProjectB - With a token issued to a user in projectB, try to create the following ports in project. - regular port - update regular port with a fixed IP - update regular port with an allowed_address_pair - create port with a fixed IP - create port with an allowed address pair. - The behavior that I'm seeing the most is that the port creation will fail with the following error (the logs are from the sdk with logging/debug enabled. - It fails maybe 1/3 - and I'm unable to see a common pattern. - This is testing against the same Openstack env and the failures/success can happen within a few seconds of each others. Sun Apr 19 01:54:06 2020 -- From ProjectB - create a port with a fixed > address in the network shared from ProjectA > REQ: curl -g -i -X GET > https://OPENSTACK_URL_HERE:13696/v2.0/subnets/77c3fad2-bf78-4b4e-bcfb-8114f… > -H "User-Agent: openstacksdk/0.37.0 keystoneauth1/3.18.0 > python-requests/2.22.0 CPython/3.7.5" -H "X-Auth-Token: > {SHA256}d80fe300742786033376f0591f98d3b4c342103cda1f802087809963694a7935" > RESP: [200] Content-Length: 619 Content-Type: application/json Date: Sun, > 19 Apr 2020 01:54:06 GMT X-Openstack-Request-Id: > req-fcee8345-7daa-499e-a985-92d59a31d0dd > RESP BODY: > {"subnet":{"service_types":[],"description":"","enable_dhcp":true,"tags":[],"network_id":"4ba08203-4328-418b-89e6-b82d474275d5","tenant_id":"472631dbfc5f4dfda9293fb93b5e493f","created_at":"2020-04-19T01:54:03Z","dns_nameservers":[],"updated_at":"2020-04-19T01:54:03Z","gateway_ip":"1.1.1.1","ipv6_ra_mode":null,"allocation_pools":[{"start":"1.1.1.2","end":"1.1.1.254"}],"host_routes":[],"revision_number":0,"ip_version":4,"ipv6_address_mode":null,"cidr":" > 1.1.1.0/24 > ","project_id":"472631dbfc5f4dfda9293fb93b5e493f","id":"77c3fad2-bf78-4b4e-bcfb-8114f95d7884","subnetpool_id":null,"name":"neutronpolicy_shared_network"}} > GET call to network for > https://OPENSTACK_URL_HERE:13696/v2.0/subnets/77c3fad2-bf78-4b4e-bcfb-8114f… > used request id req-fcee8345-7daa-499e-a985-92d59a31d0dd > REQ: curl -g -i -X POST https://OPENSTACK_URL_HERE:13696/v2.0/ports -H > "Content-Type: application/json" -H "User-Agent: openstacksdk/0.37.0 > keystoneauth1/3.18.0 python-requests/2.22.0 CPython/3.7.5" -H > "X-Auth-Token: > {SHA256}d80fe300742786033376f0591f98d3b4c342103cda1f802087809963694a7935" > -d '{"port": {"name": "neutronpolicy_portb-fixed-ip", "network_id": > "4ba08203-4328-418b-89e6-b82d474275d5", "fixed_ips": [{"subnet_id": > "77c3fad2-bf78-4b4e-bcfb-8114f95d7884", "ip_address": "1.1.1.100"}]}}' > RESP: [403] Content-Length: 151 Content-Type: application/json Date: Sun, > 19 Apr 2020 01:54:07 GMT X-Openstack-Request-Id: > req-00277dbc-faca-479f-8736-563595d2729b > RESP BODY: {"NeutronError": {"message": "(rule:create_port and > rule:create_port:fixed_ips) is disallowed by policy", "type": > "PolicyNotAuthorized", "detail": ""}} > POST call to network for https://OPENSTACK_URL_HERE:13696/v2.0/ports used > request id req-00277dbc-faca-479f-8736-563595d2729b When it works, it looks the same, just without the 403 during the port creation. Sun Apr 19 02:29:54 2020 -- From ProjectB - create a port with a fixed > address in the network shared from ProjectA > REQ: curl -g -i -X GET > https://OPENSTACK_URL_HERE:13696/v2.0/subnets/6d2231b6-3ce2-4121-a9a6-72848… > -H "User-Agent: openstacksdk/0.37.0 keystoneauth1/3.18.0 > python-requests/2.22.0 CPython/3.7.5" -H "X-Auth-Token: > {SHA256}d39995434f6a0f106cae7408b07eee93846230f2bb5c922a8e16cfc7c8b26599" > RESP: [200] Content-Length: 619 Content-Type: application/json Date: Sun, > 19 Apr 2020 02:29:54 GMT X-Openstack-Request-Id: > req-bd41b51f-b6e9-454c-8c9a-ee444140459e > RESP BODY: > {"subnet":{"service_types":[],"description":"","enable_dhcp":true,"tags":[],"network_id":"e0d7e7a1-2863-45ec-bd7b-1767a1d02822","tenant_id":"4530b70143ec4cd0ad929a4ca571bdd8","created_at":"2020-04-19T02:29:51Z","dns_nameservers":[],"updated_at":"2020-04-19T02:29:51Z","gateway_ip":"1.1.1.1","ipv6_ra_mode":null,"allocation_pools":[{"start":"1.1.1.2","end":"1.1.1.254"}],"host_routes":[],"revision_number":0,"ip_version":4,"ipv6_address_mode":null,"cidr":" > 1.1.1.0/24 > ","project_id":"4530b70143ec4cd0ad929a4ca571bdd8","id":"6d2231b6-3ce2-4121-a9a6-7284823c48ab","subnetpool_id":null,"name":"neutronpolicy_shared_network"}} > GET call to network for > https:/OPENSTACK_URL_HERE:13696/v2.0/subnets/6d2231b6-3ce2-4121-a9a6-7284823c48ab > used request id req-bd41b51f-b6e9-454c-8c9a-ee444140459e > REQ: curl -g -i -X POST https://OPENSTACK_URL_HERE:13696/v2.0/ports -H > "Content-Type: application/json" -H "User-Agent: openstacksdk/0.37.0 > keystoneauth1/3.18.0 python-requests/2.22.0 CPython/3.7.5" -H > "X-Auth-Token: > {SHA256}d39995434f6a0f106cae7408b07eee93846230f2bb5c922a8e16cfc7c8b26599" > -d '{"port": {"name": "neutronpolicy_portb-fixed-ip", "network_id": > "e0d7e7a1-2863-45ec-bd7b-1767a1d02822", "fixed_ips": [{"subnet_id": > "6d2231b6-3ce2-4121-a9a6-7284823c48ab", "ip_address": "1.1.1.100"}]}}' > RESP: [201] Content-Length: 769 Content-Type: application/json Date: Sun, > 19 Apr 2020 02:29:56 GMT X-Openstack-Request-Id: > req-898dadfb-366d-4a78-8e89-c9d5e9a512f6 > RESP BODY: > {"port":{"allowed_address_pairs":[],"extra_dhcp_opts":[],"updated_at":"2020-04-19T02:29:55Z","device_owner":"","revision_number":6,"port_security_enabled":true,"fixed_ips":[{"subnet_id":"6d2231b6-3ce2-4121-a9a6-7284823c48ab","ip_address":"1.1.1.100"}],"id":"9999784b-c38d-4cf2-b951-e8255d3483f4","security_groups":["4c022e69-b575-435b-bfb7-9cc81f50d49e"],"mac_address":"fa:16:3e:3e:22:fb","device_id":"","status":"DOWN","description":"","tags":[],"name":"neutronpolicy_portb-fixed-ip","project_id":"253ee9b6b5964185ab0818580129c011","qos_policy_id":null,"admin_state_up":true,"network_id":"e0d7e7a1-2863-45ec-bd7b-1767a1d02822","tenant_id":"253ee9b6b5964185ab0818580129c011","created_at":"2020-04-19T02:29:55Z","binding:vnic_type":"normal","ip_allocation":"immediate"}} > POST call to network for https://OPENSTACK_URL_HERE:13696/v2.0/ports used > request id req-898dadfb-366d-4a78-8e89-c9d5e9a512f6 > Sun Apr 19 02:29:56 2020 -- PORT-OK This is what I see in the Neutron server logs (with debug enabled) DEBUG neutron.api.v2.base [req-00277dbc-faca-479f-8736-563595d2729b > c905a7d0011e4bb4ae388cfd699e5989 4bae655884b44e8e9c8dbb5983fb3cd8 - default > default] Request body: {u'port': {u'network_id': > u'4ba08203-4328-418b-89e6-b82d474275d5', u'fixed_ips': [{u'subnet_id': > u'77c3fad2-bf78-4b4e-bcfb-8114f95d7884', u'ip_address': u'1.1.1.100'}], > u'name': u'neutronpolicy_portb-fixed-ip'}} prepare_request_body > /usr/lib/python2.7/site-packages/neutron/api/v2/base.py:690 > DEBUG neutron.db.quota.driver [req-00277dbc-faca-479f-8736-563595d2729b > c905a7d0011e4bb4ae388cfd699e5989 4bae655884b44e8e9c8dbb5983fb3cd8 - default > default] Resources subnetpool,trunk have unlimited quota limit. It is not > required to calculate headroom make_reservation > /usr/lib/python2.7/site-packages/neutron/db/quota/driver.py:223 > DEBUG neutron.quota.resource [req-00277dbc-faca-479f-8736-563595d2729b > c905a7d0011e4bb4ae388cfd699e5989 4bae655884b44e8e9c8dbb5983fb3cd8 - default > default] Usage tracker for resource:port and > tenant:4bae655884b44e8e9c8dbb5983fb3cd8 is out of sync, need to count used > quota count_used > /usr/lib/python2.7/site-packages/neutron/quota/resource.py:272 > DEBUG neutron.quota.resource [req-00277dbc-faca-479f-8736-563595d2729b > c905a7d0011e4bb4ae388cfd699e5989 4bae655884b44e8e9c8dbb5983fb3cd8 - default > default] Quota usage for port was recalculated. Used quota:0. count_used > /usr/lib/python2.7/site-packages/neutron/quota/resource.py:292 > DEBUG neutron.db.quota.driver [req-00277dbc-faca-479f-8736-563595d2729b > c905a7d0011e4bb4ae388cfd699e5989 4bae655884b44e8e9c8dbb5983fb3cd8 - default > default] Attempting to reserve 1 items for resource port. Total usage: 0; > quota limit: 500; headroom:500 make_reservation > /usr/lib/python2.7/site-packages/neutron/db/quota/driver.py:255 > DEBUG neutron.pecan_wsgi.hooks.quota_enforcement > [req-00277dbc-faca-479f-8736-563595d2729b c905a7d0011e4bb4ae388cfd699e5989 > 4bae655884b44e8e9c8dbb5983fb3cd8 - default default] Made reservation on > behalf of 4bae655884b44e8e9c8dbb5983fb3cd8 for: {'port': 1} before > /usr/lib/python2.7/site-packages/neutron/pecan_wsgi/hooks/quota_enforcement.py:55 > DEBUG neutron.policy [req-00277dbc-faca-479f-8736-563595d2729b > c905a7d0011e4bb4ae388cfd699e5989 4bae655884b44e8e9c8dbb5983fb3cd8 - default > default] Unable to find requested field: shared in target: > {'binding:host_id': <neutron_lib.constants.Sentinel object at > 0x7f2a3b214d50>, u'name': u'neutronpolicy_portb-fixed-ip', > 'allowed_address_pairs': <neutron_lib.constants.Sentinel object at > 0x7f2a3b214d50>, 'admin_state_up': True, u'network_id': > u'4ba08203-4328-418b-89e6-b82d474275d5', 'tenant_id': > u'4bae655884b44e8e9c8dbb5983fb3cd8', 'extra_dhcp_opts': None, > 'mac_address': <neutron_lib.constants.Sentinel object at 0x7f2a3b214d50>, > 'binding:vnic_type': 'normal', 'device_owner': '', 'qos_policy_id': None, > 'device_id': '', 'binding:profile': <neutron_lib.constants.Sentinel object > at 0x7f2a3b214d50>, 'port_security_enabled': > <neutron_lib.constants.Sentinel object at 0x7f2a3b214d50>, 'project_id': > u'4bae655884b44e8e9c8dbb5983fb3cd8', u'fixed_ips': [{u'subnet_id': > u'77c3fad2-bf78-4b4e-bcfb-8114f95d7884', u'ip_address': u'1.1.1.100'}], > u'network:tenant_id': u'472631dbfc5f4dfda9293fb93b5e493f', > 'security_groups': <neutron_lib.constants.Sentinel object at > 0x7f2a3b214d50>, 'description': ''} __call__ > /usr/lib/python2.7/site-packages/neutron/policy.py:314 > DEBUG neutron.policy [req-00277dbc-faca-479f-8736-563595d2729b > c905a7d0011e4bb4ae388cfd699e5989 4bae655884b44e8e9c8dbb5983fb3cd8 - default > default] Enforcing rules: ['create_port', 'create_port:fixed_ips'] > log_rule_list /usr/lib/python2.7/site-packages/neutron/policy.py:334 > DEBUG neutron.policy [req-00277dbc-faca-479f-8736-563595d2729b > c905a7d0011e4bb4ae388cfd699e5989 4bae655884b44e8e9c8dbb5983fb3cd8 - default > default] Failed policy check for 'create_port' enforce > /usr/lib/python2.7/site-packages/neutron/policy.py:405 > INFO neutron.pecan_wsgi.hooks.translation > [req-00277dbc-faca-479f-8736-563595d2729b c905a7d0011e4bb4ae388cfd699e5989 > 4bae655884b44e8e9c8dbb5983fb3cd8 - default default] POST failed (client > error): Access was denied to this resource. > DEBUG neutron.pecan_wsgi.hooks.notifier > [req-00277dbc-faca-479f-8736-563595d2729b c905a7d0011e4bb4ae388cfd699e5989 > 4bae655884b44e8e9c8dbb5983fb3cd8 - default default] No notification will be > sent due to unsuccessful status code: 403 after > /usr/lib/python2.7/site-packages/neutron/pecan_wsgi/hooks/notifier.py:79 > INFO neutron.wsgi [req-00277dbc-faca-479f-8736-563595d2729b > c905a7d0011e4bb4ae388cfd699e5989 4bae655884b44e8e9c8dbb5983fb3cd8 - default > default] 192.168.20.29 "POST /v2.0/ports HTTP/1.1" status: 403 len: 333 > time: 0.3625510 > I haven't found a whole lot online - just a couple of possibly related bugs https://bugs.launchpad.net/neutron/+bug/1808112 https://bugzilla.redhat.com/show_bug.cgi?id=1683385 Does anyone have any good tips/debugging methods? Thanks!

2 3

[release] Release countdown for week R-3, April 20 - 24
by Sean McGinnis 22 Apr '20

22 Apr '20

Development Focus ----------------- The Release Candidate (RC) deadline is next Thursday, April 23. Work should be focused on fixing any release-critical bugs. General Information ------------------- All deliverables released under a cycle-with-rc model should have a first release candidate by the end of the week, from which a stable/ussuri branch will be cut. This branch will track the ussuri release. Once stable/ussuri has been created, master will be ready to switch to victoria development. While master will no longer be feature-frozen, please prioritize any work necessary for completing ussuri plans. Release-critical bugfixes will need to be merged in the master branch first, then backported to the stable/ussuri branch before a new release candidate can be proposed. Actions ------- Early in the week, the release team will be proposing RC1 patches for all cycle-with-rc projects, using the latest commit from master. If your team is ready to go for cutting RC1, please let us know by leaving a +1 on these patches. If there are still a few more patches needed before RC1, you can -1 the patch and update it later in the week with the new commit hash you would like to use. Remember, stable/ussuri branches will be created with this, so you will want to make sure you have what you need included to avoid needing to backport changes from master (which will technically then be victoria) to this stable branch for any additional RCs before the final release. The release team will also be proposing releases for any deliverable following a cycle-with-intermediary model that has not produced any ussuri release so far. Upcoming Deadlines & Dates -------------------------- RC1 deadline: April 23 (R-3 week) Final RC deadline: May 7 (R-1 week) Final Ussuri release: May 13 Virtual Victoria PTG: June 1-5

1 1

[octavia] Virtual PTG etherpad created
by Michael Johnson 22 Apr '20

22 Apr '20

The Octavia team has started their virtual Victoria PTG etherpad[1]. Please let us know if you will be joining our sessions and if you have any topics you would like to cover. [1] https://etherpad.opendev.org/p/octavia-virtual-V-ptg Michael

1 0

[all] [PTL][TC] PTG Signup Reminder & PTG Registration
by Kendall Nelson 22 Apr '20

22 Apr '20

Hello! ==Virtual PTG Signup== Wanted to take a second to remind teams about the process for signing up for virtual PTG slots. 0. Look at the slots available and talk to your contributors and team members to find out the best times to signup for. 1. Sign up in the ethercalc [1] for the time slots your team needs. You can sign up for up to 4 hours/day and 16 hours/ the week (restrictions in place until *April 28th at 7:00 UTC*). If you are a SIG or WG, try to keep your conversations at the start of the week wherever possible. Vertical teams, try to focus your conversations towards the latter part of the week (Wednesday-Friday). *AND* 2. Fill out the survey [2] to give us the other team information we need to help facilitate the event. We NEED you to do both steps! I can see a number of teams that have signed up for time, but haven't filled out the survey so please do so ASAP. There are still a lot of slots left! Original thread[3]. ==Registration== After a lot of discussion, we have decided to have registration. It is free but its still extremely important that you register as this is primarily so we have a way to contact attendees with information about the event as we figure the rest of the details out. We will have the registration live next week. -the Kendalls (diablo_rojo & wendallkaters) [1] Ethercalc:https://ethercalc.openstack.org/126u8ek25noy [2] Survey: https://openstackfoundation.formstack.com/forms/june2020_virtual_ptg_survey [3] PTG Layout & Signup Thread: http://lists.openstack.org/pipermail/openstack-discuss/2020-April/014126.ht…

1 0

Virtual OpenDev Update
by Ashlee Ferguson 22 Apr '20

22 Apr '20

Hi everyone, As we continue to gather feedback and discuss what a virtual OpenDev will look like, we wanted to provide you with an update around the event. While we are still attempting to solve some of the riddles of virtual event world, we did come up with some decisions that give us something to build on, which are based on input and experience from members of the Programming Committee. Decisions: - OpenDev will happen after the Virtual PTG (exact format and timing TBD) - Will not be simultaneous/parallel Tracks so everyone has the opportunity to participate in more than one - 3 Tracks, each on different date: - Hardware Automation - Large Scale Ops - Containers in Production We also discussed the original intent and goals of OpenDev to make sure we’re still heading in a direction that’s beneficial for the community, especially since it will no longer directly precede the upcoming PTG. Similar to past OpenDev events, the goal is to identify the questions we don’t have the answers for, and to use that time to dive in and determine the work ahead of us, working together, sharing use cases, and learning from other people asking those same questions. High-level decisions, goals, and some areas we’re looking for feedback are in the etherpad below. As you continue to experience this new amazing world of virtual events, please add any feedback or ideas to the etherpad below. We’re looking forward to creating virtual OpenDev with you all! https://etherpad.openstack.org/p/Virtual_OpenDev_Planning <https://etherpad.openstack.org/p/Virtual_OpenDev_Planning> Ashlee Ashlee Ferguson Community & Events Coordinator OpenStack Foundation

1 2

[requirements][vitrage][ffe] ffe request for update constraint for XStatic-Graphlib to new release 2.1.7.0
by Eyal B 22 Apr '20

22 Apr '20

Hi, I'd like to request FFE for: https://review.opendev.org/#/c/721818/ This is needed to fix a dashboard bug https://storyboard.openstack.org/#!/task/30604 Thanks Eyal

2 1

[sahara] sahara-core has been adjusted
by Jeremy Freudberg 22 Apr '20

22 Apr '20

The sahara-core group now contains those who have performed at least one review during the calendar year 2019.

1 0

[nova] [cinder] [ceph] Questioning deployment choices
by Radosław Piliszek 22 Apr '20

22 Apr '20

Hi Fellow OpenStackers! I have a question regarding nova-cinder-ceph deployment configuration. Most notably ephemeral storage. So - if I configure Nova to use Ceph as ephemeral storage and Cinder's only backend is Ceph as well then how does ephemeral storage differ from persistent storage now except for interface from Cinder rather than Nova (and all that comes with that) and a different pool? Was Nova's Ceph integration added just to be able to avoid Cinder in certain scenarios? Other Cinder backends did not get support from Nova. Looking forward to your insights. -yoctozepto

3 5

[kolla][tripleo][nova] nova scheduler startup errors in Train on CentOS 8
by Mark Goddard 22 Apr '20

22 Apr '20

Hi, Our (kolla-ansible) CentOS 8 binary (RDO) deploy jobs have been failing on stable/train since added. Because the source jobs were passing, I was kind of hoping it was a packaging issue that would right itself, but no such luck yet. The majority of the job succeeds, but we grep our logs for CRITICAL messages, and nova-scheduler has a few when it starts up that cause us to fail: 2020-04-21 08:59:30.125 24 CRITICAL nova [req-86f6dad1-25f2-4c77-9d7d-ac7422801b4e - - - - -] Unhandled error: RuntimeError: dictionary changed size during iteration 2020-04-21 08:59:30.125 24 ERROR nova Traceback (most recent call last): 2020-04-21 08:59:30.125 24 ERROR nova File "/usr/bin/nova-scheduler", line 10, in <module> 2020-04-21 08:59:30.125 24 ERROR nova sys.exit(main()) 2020-04-21 08:59:30.125 24 ERROR nova File "/usr/lib/python3.6/site-packages/nova/cmd/scheduler.py", line 53, in main 2020-04-21 08:59:30.125 24 ERROR nova service.serve(server, workers=workers) 2020-04-21 08:59:30.125 24 ERROR nova File "/usr/lib/python3.6/site-packages/nova/service.py", line 454, in serve 2020-04-21 08:59:30.125 24 ERROR nova restart_method='mutate') 2020-04-21 08:59:30.125 24 ERROR nova File "/usr/lib/python3.6/site-packages/oslo_service/service.py", line 842, in launch 2020-04-21 08:59:30.125 24 ERROR nova launcher.launch_service(service, workers=workers) 2020-04-21 08:59:30.125 24 ERROR nova File "/usr/lib/python3.6/site-packages/oslo_service/service.py", line 606, in launch_service 2020-04-21 08:59:30.125 24 ERROR nova self._start_child(wrap) 2020-04-21 08:59:30.125 24 ERROR nova File "/usr/lib/python3.6/site-packages/oslo_service/service.py", line 575, in _start_child 2020-04-21 08:59:30.125 24 ERROR nova self.launcher.restart() 2020-04-21 08:59:30.125 24 ERROR nova File "/usr/lib/python3.6/site-packages/oslo_service/service.py", line 311, in restart 2020-04-21 08:59:30.125 24 ERROR nova self.conf.mutate_config_files() 2020-04-21 08:59:30.125 24 ERROR nova File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 3013, in mutate_config_files 2020-04-21 08:59:30.125 24 ERROR nova self._warn_immutability() 2020-04-21 08:59:30.125 24 ERROR nova File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 3040, in _warn_immutability 2020-04-21 08:59:30.125 24 ERROR nova for info, group in self._all_opt_infos(): 2020-04-21 08:59:30.125 24 ERROR nova File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 2502, in _all_opt_infos 2020-04-21 08:59:30.125 24 ERROR nova for info in self._opts.values(): 2020-04-21 08:59:30.125 24 ERROR nova RuntimeError: dictionary changed size during iteration https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/… The service does appear to start up successfully. It doesn't happen on master. Has anyone seen this before? Cheers, Mark

3 4

[cinder] nominate Luigi Toscano for cinder-tempest-plugin-core
by Brian Rosmaita 22 Apr '20

22 Apr '20

Luigi Toscano (tosky on IRC) has been very helpful this cycle addressing testing questions, doing reviews, and adding code related to testing to the various Cinder project repositories. He's been an active participant at the weekly Cinder meetings and has been good about getting visibility on testing-related patches. I'd like to put him in a position to be able to assign priority to reviews for the cinder-tempest-plugin and approve patches to that repository; hence I am nominating him to be a member of the cinder-tempest-plugin-core team. In the absence of objections, I'll make the change before the next Cinder team meeting (Wednesday, 22 April at 1400 UTC in #openstack-meeting-alt). cheers, brian

8 8

Re: [nova][cinder][ops] question/confirmation of legacy vol attachment migration
by Brett Milford 22 Apr '20

22 Apr '20

On Tue, Dec 10, 2019 at 11:15 PM Gorka Eguileor <geguileo(a)redhat.com> wrote: > > On 06/12, Brett Milford wrote: > > On Fri, Nov 22, 2019 at 3:54 AM Matt Riedemann <mriedemos(a)gmail.com> wrote: > > > > > > On 10/17/2019 5:24 AM, Gorka Eguileor wrote: > > > > I stand by my initial recommendation, being able to update the existing > > > > attachment to add the connection information from Nova. > > > > > > OK, thanks for the input and thoughtfulness on this. I've abandoned my > > > change since I'm not going to be pushing this boulder anymore but left > > > notes in the change in case someone else wants to pick it up some day. > > > > > > Note to nova cores: this means we'll have legacy volume attachment > > > compat code around forever. > > > > > > -- > > > > > > Thanks, > > > > > > Matt > > > > > > > Hi Groka, Cinder & Nova devs, > > > > Following up this thread from the context of > > https://review.opendev.org/#/c/579004/ > > > > To summarise the discussion: > > - initialize_connection shouldn't be called more than once, as it > > may mess up some drivers. > > - To (safely) refresh connection_info a new api for cinder is required. > > Hi, > > It may be a new API or a modification of one of the existing ones. > > > > - a patch to nova, such as #579004 could make a call to this new api > > to refresh connection info on events such as reboot. > > > > Is there any other context to this issue I've missed or an alternate > > path to solving this? > > > > Does the creation of a new api for cinder have any implications for > > being able to backport this patch for nova? > > > > I don't think backporting it in Nova would do us much good, since the > Cinder code would not be backportable (it's a new feature, not a bug > fix). > > > > What would be the process for me to kick off this work? > > > > You should propose a Cinder spec [1] for your work and you may provide a > WIP patch to Cinder at the same time, though depending on the review you > may have to completely change that code later. > > Cheers, > Gorka. > > [1]: https://opendev.org/openstack/cinder-specs > > > Thanks for your help, > > Brett (bcm) > > -- > > ❯ brett > > > Cheers Matt, Groka, I thought this might be the case. -- ❯ brett

2 2

[largescale-sig] Next meeting: Apr 22, 8utc
by Thierry Carrez 22 Apr '20

22 Apr '20

Hi everyone, The Large Scale SIG will have a meeting this week on Wednesday, April 22 at 8 UTC[1] in the #openstack-meeting-3 channel on IRC: https://www.timeanddate.com/worldclock/fixedtime.html?iso=20200422T08 Feel free to add topics to our agenda at: https://etherpad.openstack.org/p/large-scale-sig-meeting A reminder of the TODOs we had from last meeting, in case you have time to make progress on them: - masahito to prepare oslo.metric POC code release - oneswig to contribute a scaling story on bare metal cluster scaling - belmoreira to submit a scaling story to https://etherpad.openstack.org/p/scaling-stories - ttx to copy scaling stories on the wiki for long-term storage - amorin propose patch against Nova doc Talk to you all on Wednesday, -- Thierry Carrez

1 1

[requirements][release][monasca] FFE request for blocking confluent-kafka 1.4.0
by Witek Bedyk 22 Apr '20

22 Apr '20

Hi, I'd like to request FFE for: https://review.opendev.org/721685 The last release of confluent-kafka includes a buggy source package [1] which block our CI. Thanks Witek [1] https://github.com/confluentinc/confluent-kafka-python/issues/830

3 2

[keystone][ptg] Virtual PTG planning
by Colleen Murphy 21 Apr '20

21 Apr '20

Hi team, As discussed in today's meeting, we're collecting input on how to structure the keystone team's virtual PTG sessions. See Kendall's email[1] for the full details of how the gathering will work. I suggest we have two 4-hour sessions during a time that works for the main keystone contributors, and perhaps one or two shorter office-hour sessions at alternate times for ad-hoc discussions or cross-project collaboration. As we refine the agenda, we may decide that we want more or less time, and we can probably adjust accordingly down the line. I've created a poll[2] with two-hour blocks to represent the available session times we could schedule. Please fill out the poll if you're a member of the team or if you have a topic to discuss with the team. Please also add your name and topics to the brainstorming etherpad[3]. Colleen [1] http://lists.openstack.org/pipermail/openstack-discuss/2020-April/014126.ht… [2] https://doodle.com/poll/8agdsvxy2ve9rr4x [3] https://etherpad.opendev.org/p/yvr-ptg-keystone

1 0

[neutron] Neutron Bug Deputy Report Apr. 13-21
by Ryan Tidwell 21 Apr '20

21 Apr '20

Hello Neutron Team, This is my bug deputy report for the previous week. It was an active week of bug reports, here are the highlights: RFE: -------- Neutron ports dns_assignment does not match the designate DNS records for Neutron port https://bugs.launchpad.net/bugs/1873091 - This could use some follow-up from someone who is acquainted with the neutron-designate integration FWaaS: ------------ The firewall group's status is active when firewall group has no any policy https://bugs.launchpad.net/bugs/1872439 - Could simply be a misunderstanding by the user, some clarification by someone with more FWaaS expertise would be helpful. The firewall group's function is failed in the dvr scene. https://bugs.launchpad.net/bugs/1872407 - On the surface this looks like a DVR-related error. Need more info from the user to be able to debug further. DVR: -------- Internal IP leak to physical interface from qrouter in DVR mode https://bugs.launchpad.net/bugs/1873761 - RST packets seem to not get NAT'ed properly, needs some investigation to understand why. Can not use vrrp in a dvr openstack environment https://bugs.launchpad.net/bugs/1873375 - The reporter references https://review.opendev.org/#/c/716302/, I have posted a follow-up question for clarity. Floating IP was not removed from the router when the port forwarding deletion was completed https://bugs.launchpad.net/bugs/1873708 - This had some initial triage, an effort has been made to reproduce the issue without success CI Issues: --------------- [stable/stein] Neutron-tempest-plugin jobs are failing due to wrong python version https://bugs.launchpad.net/bugs/1873776 - https://review.opendev.org/721277 OVN Bugs Filed: ------------------------- [OVN] External ports creation is not account for VNIC_DIRECT_PHYSICAL https://bugs.launchpad.net/bugs/1874065 - This appears to have an owner Misc. Bugs ----------------- VM can't be accessed, openflow rules are removed and neutron-ovs-agent is always reporting errors after rabbitmq is in split-brain state https://bugs.launchpad.net/bugs/1873265 - Things get into a weird state when rabbitmq gets into a split-brain scenario. Lots of discussion happening in the bug report. vpnaas pluto.pid file path wrong https://bugs.launchpad.net/bugs/1872940 Failing to terminate Windows processes https://bugs.launchpad.net/bugs/1872663 - Fix proposed here: https://review.opendev.org/719765 -Ryan

1 0

[oslo] Virtual PTG etherpad
by Ben Nemec 21 Apr '20

21 Apr '20

As discussed in the meeting this week, I've created an etherpad[0] where people can propose topics for a virtual PTG. If you have anything you'd like to discuss please add it there so we can decide whether to meet. Thanks. -Ben 0: https://etherpad.opendev.org/p/oslo-victoria-topics

1 0

[puppet] broken CI for stable/pike and stable/ocata
by Takashi Kajinami 21 Apr '20

21 Apr '20

Hi All, While I'm updating job configurations in each branch to avoid unexpected conflicts between them[1], I noticed that currently CI for stable/pike and stable/ocata is broken. Pike: https://review.opendev.org/#/c/718725/ Ocata: https://review.opendev.org/#/c/718726/ While I guess we need to update the job script to fix the issue, May I know whether anybody still has an interest in maintaining these branches? If nobody is interested in keeping these branches, I'd propose to make these old branches eol, so that we can reduce the number of open branches. I'd appreciate any feedback about this topic. [1] https://bugs.launchpad.net/puppet-openstack-integration/+bug/1871866 Thank you, Takashi -- ---------- Takashi Kajinami Senior Software Maintenance Engineer Customer Experience and Engagement Red Hat e-mail: tkajinam(a)redhat.com

3 4