From 1713783 at bugs.launchpad.net Fri Sep 1 13:47:20 2017 From: 1713783 at bugs.launchpad.net (OpenStack Infra) Date: Fri, 01 Sep 2017 13:47:20 -0000 Subject: [Openstack-security] [Bug 1713783] Related fix proposed to nova (master) References: <150402703560.19393.11649471063519714290.malonedeb@chaenomeles.canonical.com> Message-ID: <150427364047.19611.7265390203788304676.malone@chaenomeles.canonical.com> Related fix proposed to branch: master Review: https://review.openstack.org/500057 -- You received this bug notification because you are a member of OpenStack Security, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1713783 Title: After failed evacuation the recovered source compute tries to delete the instance Status in OpenStack Compute (nova): Triaged Status in OpenStack Compute (nova) newton series: Triaged Status in OpenStack Compute (nova) ocata series: Triaged Status in OpenStack Compute (nova) pike series: Triaged Status in OpenStack Security Advisory: Won't Fix Bug description: Description =========== In case of a failed evacuation attempt the status of the migration is 'accepted' instead of 'failed' so when source compute is recovered the compute manager tries to delete the instance from the source host. However a secondary fault prevents deleting the allocation in placement so the actual deletion of the instance fails as well. Steps to reproduce ================== The following functional test reproduces the bug: https://review.openstack.org/#/c/498482/ What it does: initiate evacuation when no valid host is available and evacuation fails, but nova manager still tries to delete the instance. Logs:     2017-08-29 19:11:15,751 ERROR [oslo_messaging.rpc.server] Exception during message handling     NoValidHost: No valid host was found. There are not enough hosts available.     2017-08-29 19:11:16,103 INFO [nova.tests.functional.test_servers] Running periodic for compute1 (host1)     2017-08-29 19:11:16,115 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,120 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,131 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/allocations" status: 200 len: 152 microversion: 1.0     2017-08-29 19:11:16,138 INFO [nova.compute.resource_tracker] Final resource view: name=host1 phys_ram=8192MB used_ram=1024MB phys_disk=1028GB used_disk=1GB total_vcpus=10 used_vcpus=1 pci_stats=[]     2017-08-29 19:11:16,146 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,151 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,152 INFO [nova.tests.functional.test_servers] Running periodic for compute2 (host2)     2017-08-29 19:11:16,163 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,168 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,176 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/allocations" status: 200 len: 54 microversion: 1.0     2017-08-29 19:11:16,184 INFO [nova.compute.resource_tracker] Final resource view: name=host2 phys_ram=8192MB used_ram=512MB phys_disk=1028GB used_disk=0GB total_vcpus=10 used_vcpus=0 pci_stats=[]     2017-08-29 19:11:16,192 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,197 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,198 INFO [nova.tests.functional.test_servers] Finished with periodics     2017-08-29 19:11:16,255 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/6f70656e737461636b20342065766572/servers/5058200c-478e-4449-88c1-906fdd572662" status: 200 len: 1875 microversion: 2.53 time: 0.056198     2017-08-29 19:11:16,262 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/6f70656e737461636b20342065766572/os-migrations" status: 200 len: 373 microversion: 2.53 time: 0.004618     2017-08-29 19:11:16,280 INFO [nova.api.openstack.requestlog] 127.0.0.1 "PUT /v2.1/6f70656e737461636b20342065766572/os-services/c269bc74-4720-4de4-a6e5-889080b892a0" status: 200 len: 245 microversion: 2.53 time: 0.016442     2017-08-29 19:11:16,281 INFO [nova.service] Starting compute node (version 16.0.0)     2017-08-29 19:11:16,296 INFO [nova.compute.manager] Deleting instance as it has been evacuated from this host To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1713783/+subscriptions From 1713783 at bugs.launchpad.net Fri Sep 1 14:45:56 2017 From: 1713783 at bugs.launchpad.net (OpenStack Infra) Date: Fri, 01 Sep 2017 14:45:56 -0000 Subject: [Openstack-security] [Bug 1713783] Re: After failed evacuation the recovered source compute tries to delete the instance References: <150402703560.19393.11649471063519714290.malonedeb@chaenomeles.canonical.com> Message-ID: <150427715847.8620.3877574380648768769.launchpad@soybean.canonical.com> ** Changed in: nova Status: Triaged => In Progress ** Changed in: nova Assignee: (unassigned) => Illes Elod (elod-illes) -- You received this bug notification because you are a member of OpenStack Security, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1713783 Title: After failed evacuation the recovered source compute tries to delete the instance Status in OpenStack Compute (nova): In Progress Status in OpenStack Compute (nova) newton series: Triaged Status in OpenStack Compute (nova) ocata series: Triaged Status in OpenStack Compute (nova) pike series: Triaged Status in OpenStack Security Advisory: Won't Fix Bug description: Description =========== In case of a failed evacuation attempt the status of the migration is 'accepted' instead of 'failed' so when source compute is recovered the compute manager tries to delete the instance from the source host. However a secondary fault prevents deleting the allocation in placement so the actual deletion of the instance fails as well. Steps to reproduce ================== The following functional test reproduces the bug: https://review.openstack.org/#/c/498482/ What it does: initiate evacuation when no valid host is available and evacuation fails, but nova manager still tries to delete the instance. Logs:     2017-08-29 19:11:15,751 ERROR [oslo_messaging.rpc.server] Exception during message handling     NoValidHost: No valid host was found. There are not enough hosts available.     2017-08-29 19:11:16,103 INFO [nova.tests.functional.test_servers] Running periodic for compute1 (host1)     2017-08-29 19:11:16,115 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,120 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,131 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/allocations" status: 200 len: 152 microversion: 1.0     2017-08-29 19:11:16,138 INFO [nova.compute.resource_tracker] Final resource view: name=host1 phys_ram=8192MB used_ram=1024MB phys_disk=1028GB used_disk=1GB total_vcpus=10 used_vcpus=1 pci_stats=[]     2017-08-29 19:11:16,146 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,151 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,152 INFO [nova.tests.functional.test_servers] Running periodic for compute2 (host2)     2017-08-29 19:11:16,163 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,168 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,176 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/allocations" status: 200 len: 54 microversion: 1.0     2017-08-29 19:11:16,184 INFO [nova.compute.resource_tracker] Final resource view: name=host2 phys_ram=8192MB used_ram=512MB phys_disk=1028GB used_disk=0GB total_vcpus=10 used_vcpus=0 pci_stats=[]     2017-08-29 19:11:16,192 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,197 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,198 INFO [nova.tests.functional.test_servers] Finished with periodics     2017-08-29 19:11:16,255 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/6f70656e737461636b20342065766572/servers/5058200c-478e-4449-88c1-906fdd572662" status: 200 len: 1875 microversion: 2.53 time: 0.056198     2017-08-29 19:11:16,262 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/6f70656e737461636b20342065766572/os-migrations" status: 200 len: 373 microversion: 2.53 time: 0.004618     2017-08-29 19:11:16,280 INFO [nova.api.openstack.requestlog] 127.0.0.1 "PUT /v2.1/6f70656e737461636b20342065766572/os-services/c269bc74-4720-4de4-a6e5-889080b892a0" status: 200 len: 245 microversion: 2.53 time: 0.016442     2017-08-29 19:11:16,281 INFO [nova.service] Starting compute node (version 16.0.0)     2017-08-29 19:11:16,296 INFO [nova.compute.manager] Deleting instance as it has been evacuated from this host To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1713783/+subscriptions From 1713783 at bugs.launchpad.net Fri Sep 1 16:34:46 2017 From: 1713783 at bugs.launchpad.net (OpenStack Infra) Date: Fri, 01 Sep 2017 16:34:46 -0000 Subject: [Openstack-security] [Bug 1713783] Re: After failed evacuation the recovered source compute tries to delete the instance References: <150402703560.19393.11649471063519714290.malonedeb@chaenomeles.canonical.com> Message-ID: <150428368901.19681.14608581571214084189.launchpad@chaenomeles.canonical.com> ** Changed in: nova Assignee: Illes Elod (elod-illes) => Matt Riedemann (mriedem) -- You received this bug notification because you are a member of OpenStack Security, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1713783 Title: After failed evacuation the recovered source compute tries to delete the instance Status in OpenStack Compute (nova): In Progress Status in OpenStack Compute (nova) newton series: Triaged Status in OpenStack Compute (nova) ocata series: Triaged Status in OpenStack Compute (nova) pike series: Triaged Status in OpenStack Security Advisory: Won't Fix Bug description: Description =========== In case of a failed evacuation attempt the status of the migration is 'accepted' instead of 'failed' so when source compute is recovered the compute manager tries to delete the instance from the source host. However a secondary fault prevents deleting the allocation in placement so the actual deletion of the instance fails as well. Steps to reproduce ================== The following functional test reproduces the bug: https://review.openstack.org/#/c/498482/ What it does: initiate evacuation when no valid host is available and evacuation fails, but nova manager still tries to delete the instance. Logs:     2017-08-29 19:11:15,751 ERROR [oslo_messaging.rpc.server] Exception during message handling     NoValidHost: No valid host was found. There are not enough hosts available.     2017-08-29 19:11:16,103 INFO [nova.tests.functional.test_servers] Running periodic for compute1 (host1)     2017-08-29 19:11:16,115 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,120 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,131 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/allocations" status: 200 len: 152 microversion: 1.0     2017-08-29 19:11:16,138 INFO [nova.compute.resource_tracker] Final resource view: name=host1 phys_ram=8192MB used_ram=1024MB phys_disk=1028GB used_disk=1GB total_vcpus=10 used_vcpus=1 pci_stats=[]     2017-08-29 19:11:16,146 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,151 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,152 INFO [nova.tests.functional.test_servers] Running periodic for compute2 (host2)     2017-08-29 19:11:16,163 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,168 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,176 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/allocations" status: 200 len: 54 microversion: 1.0     2017-08-29 19:11:16,184 INFO [nova.compute.resource_tracker] Final resource view: name=host2 phys_ram=8192MB used_ram=512MB phys_disk=1028GB used_disk=0GB total_vcpus=10 used_vcpus=0 pci_stats=[]     2017-08-29 19:11:16,192 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,197 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,198 INFO [nova.tests.functional.test_servers] Finished with periodics     2017-08-29 19:11:16,255 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/6f70656e737461636b20342065766572/servers/5058200c-478e-4449-88c1-906fdd572662" status: 200 len: 1875 microversion: 2.53 time: 0.056198     2017-08-29 19:11:16,262 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/6f70656e737461636b20342065766572/os-migrations" status: 200 len: 373 microversion: 2.53 time: 0.004618     2017-08-29 19:11:16,280 INFO [nova.api.openstack.requestlog] 127.0.0.1 "PUT /v2.1/6f70656e737461636b20342065766572/os-services/c269bc74-4720-4de4-a6e5-889080b892a0" status: 200 len: 245 microversion: 2.53 time: 0.016442     2017-08-29 19:11:16,281 INFO [nova.service] Starting compute node (version 16.0.0)     2017-08-29 19:11:16,296 INFO [nova.compute.manager] Deleting instance as it has been evacuated from this host To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1713783/+subscriptions From 1713783 at bugs.launchpad.net Fri Sep 1 16:45:09 2017 From: 1713783 at bugs.launchpad.net (OpenStack Infra) Date: Fri, 01 Sep 2017 16:45:09 -0000 Subject: [Openstack-security] [Bug 1713783] Related fix merged to nova (master) References: <150402703560.19393.11649471063519714290.malonedeb@chaenomeles.canonical.com> Message-ID: <150428430967.19681.2616090160884300031.malone@chaenomeles.canonical.com> Reviewed: https://review.openstack.org/500057 Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5687c170ea003aafa2720fb00af9d84836bb41df Submitter: Jenkins Branch: master commit 5687c170ea003aafa2720fb00af9d84836bb41df Author: Előd Illés Date: Fri Sep 1 15:36:20 2017 +0200 Functional test for regression bug #1713783 Add functional test for evacuation, when no valid host available. Migration should end up in 'error' state. Change-Id: I1adc20f2a5261e6906a18b9aee5cd2c8ecf0cf4d Related-bug: #1713783 -- You received this bug notification because you are a member of OpenStack Security, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1713783 Title: After failed evacuation the recovered source compute tries to delete the instance Status in OpenStack Compute (nova): In Progress Status in OpenStack Compute (nova) newton series: Triaged Status in OpenStack Compute (nova) ocata series: Triaged Status in OpenStack Compute (nova) pike series: Triaged Status in OpenStack Security Advisory: Won't Fix Bug description: Description =========== In case of a failed evacuation attempt the status of the migration is 'accepted' instead of 'failed' so when source compute is recovered the compute manager tries to delete the instance from the source host. However a secondary fault prevents deleting the allocation in placement so the actual deletion of the instance fails as well. Steps to reproduce ================== The following functional test reproduces the bug: https://review.openstack.org/#/c/498482/ What it does: initiate evacuation when no valid host is available and evacuation fails, but nova manager still tries to delete the instance. Logs:     2017-08-29 19:11:15,751 ERROR [oslo_messaging.rpc.server] Exception during message handling     NoValidHost: No valid host was found. There are not enough hosts available.     2017-08-29 19:11:16,103 INFO [nova.tests.functional.test_servers] Running periodic for compute1 (host1)     2017-08-29 19:11:16,115 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,120 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,131 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/allocations" status: 200 len: 152 microversion: 1.0     2017-08-29 19:11:16,138 INFO [nova.compute.resource_tracker] Final resource view: name=host1 phys_ram=8192MB used_ram=1024MB phys_disk=1028GB used_disk=1GB total_vcpus=10 used_vcpus=1 pci_stats=[]     2017-08-29 19:11:16,146 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,151 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,152 INFO [nova.tests.functional.test_servers] Running periodic for compute2 (host2)     2017-08-29 19:11:16,163 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,168 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,176 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/allocations" status: 200 len: 54 microversion: 1.0     2017-08-29 19:11:16,184 INFO [nova.compute.resource_tracker] Final resource view: name=host2 phys_ram=8192MB used_ram=512MB phys_disk=1028GB used_disk=0GB total_vcpus=10 used_vcpus=0 pci_stats=[]     2017-08-29 19:11:16,192 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,197 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,198 INFO [nova.tests.functional.test_servers] Finished with periodics     2017-08-29 19:11:16,255 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/6f70656e737461636b20342065766572/servers/5058200c-478e-4449-88c1-906fdd572662" status: 200 len: 1875 microversion: 2.53 time: 0.056198     2017-08-29 19:11:16,262 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/6f70656e737461636b20342065766572/os-migrations" status: 200 len: 373 microversion: 2.53 time: 0.004618     2017-08-29 19:11:16,280 INFO [nova.api.openstack.requestlog] 127.0.0.1 "PUT /v2.1/6f70656e737461636b20342065766572/os-services/c269bc74-4720-4de4-a6e5-889080b892a0" status: 200 len: 245 microversion: 2.53 time: 0.016442     2017-08-29 19:11:16,281 INFO [nova.service] Starting compute node (version 16.0.0)     2017-08-29 19:11:16,296 INFO [nova.compute.manager] Deleting instance as it has been evacuated from this host To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1713783/+subscriptions From 1713783 at bugs.launchpad.net Fri Sep 1 17:58:46 2017 From: 1713783 at bugs.launchpad.net (OpenStack Infra) Date: Fri, 01 Sep 2017 17:58:46 -0000 Subject: [Openstack-security] [Bug 1713783] Related fix proposed to nova (master) References: <150402703560.19393.11649471063519714290.malonedeb@chaenomeles.canonical.com> Message-ID: <150428872610.8358.16486329117165788060.malone@soybean.canonical.com> Related fix proposed to branch: master Review: https://review.openstack.org/500144 -- You received this bug notification because you are a member of OpenStack Security, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1713783 Title: After failed evacuation the recovered source compute tries to delete the instance Status in OpenStack Compute (nova): In Progress Status in OpenStack Compute (nova) newton series: Triaged Status in OpenStack Compute (nova) ocata series: Triaged Status in OpenStack Compute (nova) pike series: Triaged Status in OpenStack Security Advisory: Won't Fix Bug description: Description =========== In case of a failed evacuation attempt the status of the migration is 'accepted' instead of 'failed' so when source compute is recovered the compute manager tries to delete the instance from the source host. However a secondary fault prevents deleting the allocation in placement so the actual deletion of the instance fails as well. Steps to reproduce ================== The following functional test reproduces the bug: https://review.openstack.org/#/c/498482/ What it does: initiate evacuation when no valid host is available and evacuation fails, but nova manager still tries to delete the instance. Logs:     2017-08-29 19:11:15,751 ERROR [oslo_messaging.rpc.server] Exception during message handling     NoValidHost: No valid host was found. There are not enough hosts available.     2017-08-29 19:11:16,103 INFO [nova.tests.functional.test_servers] Running periodic for compute1 (host1)     2017-08-29 19:11:16,115 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,120 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,131 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/allocations" status: 200 len: 152 microversion: 1.0     2017-08-29 19:11:16,138 INFO [nova.compute.resource_tracker] Final resource view: name=host1 phys_ram=8192MB used_ram=1024MB phys_disk=1028GB used_disk=1GB total_vcpus=10 used_vcpus=1 pci_stats=[]     2017-08-29 19:11:16,146 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,151 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,152 INFO [nova.tests.functional.test_servers] Running periodic for compute2 (host2)     2017-08-29 19:11:16,163 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,168 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,176 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/allocations" status: 200 len: 54 microversion: 1.0     2017-08-29 19:11:16,184 INFO [nova.compute.resource_tracker] Final resource view: name=host2 phys_ram=8192MB used_ram=512MB phys_disk=1028GB used_disk=0GB total_vcpus=10 used_vcpus=0 pci_stats=[]     2017-08-29 19:11:16,192 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,197 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,198 INFO [nova.tests.functional.test_servers] Finished with periodics     2017-08-29 19:11:16,255 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/6f70656e737461636b20342065766572/servers/5058200c-478e-4449-88c1-906fdd572662" status: 200 len: 1875 microversion: 2.53 time: 0.056198     2017-08-29 19:11:16,262 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/6f70656e737461636b20342065766572/os-migrations" status: 200 len: 373 microversion: 2.53 time: 0.004618     2017-08-29 19:11:16,280 INFO [nova.api.openstack.requestlog] 127.0.0.1 "PUT /v2.1/6f70656e737461636b20342065766572/os-services/c269bc74-4720-4de4-a6e5-889080b892a0" status: 200 len: 245 microversion: 2.53 time: 0.016442     2017-08-29 19:11:16,281 INFO [nova.service] Starting compute node (version 16.0.0)     2017-08-29 19:11:16,296 INFO [nova.compute.manager] Deleting instance as it has been evacuated from this host To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1713783/+subscriptions From 1702571 at bugs.launchpad.net Wed Sep 13 17:00:57 2017 From: 1702571 at bugs.launchpad.net (OpenStack Infra) Date: Wed, 13 Sep 2017 17:00:57 -0000 Subject: [Openstack-security] [Bug 1702571] Fix included in openstack/ansible-hardening 16.0.0.0rc2 References: <149928884089.15922.6070230979479956825.malonedeb@wampee.canonical.com> Message-ID: <150532205709.13396.18124938826544082839.malone@gac.canonical.com> This issue was fixed in the openstack/ansible-hardening 16.0.0.0rc2 release candidate. -- You received this bug notification because you are a member of OpenStack Security, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1702571 Title: ansible-hardening docs are vague about using as standalone Status in openstack-ansible: Fix Released Bug description: The ansible-hardening docs are a bit vague about how to handle standalone deployments. Questions to answer: 1) How/where to provide variables? 2) What about roles in non standard locations? /home/myuser/ansible-hardening? 3) How do you install via ansible-galaxy from git? 4) What do you need to install first before running the role? 5) What would an inventory file look like? To manage notifications about this bug go to: https://bugs.launchpad.net/openstack-ansible/+bug/1702571/+subscriptions From 1702123 at bugs.launchpad.net Wed Sep 13 17:01:20 2017 From: 1702123 at bugs.launchpad.net (OpenStack Infra) Date: Wed, 13 Sep 2017 17:01:20 -0000 Subject: [Openstack-security] [Bug 1702123] Fix included in openstack/openstack-ansible 16.0.0.0rc2 References: <149909478116.6327.1678093967892023529.malonedeb@gac.canonical.com> Message-ID: <150532208098.16999.7465558363343070597.malone@soybean.canonical.com> This issue was fixed in the openstack/openstack-ansible 16.0.0.0rc2 release candidate. -- You received this bug notification because you are a member of OpenStack Security, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1702123 Title: SELinux error: keepalived reading haproxy pid file Status in openstack-ansible: Fix Released Bug description: When keepalived tries to read the haproxy PID file, SELinux denies the access. This should be added into the haproxy role. To manage notifications about this bug go to: https://bugs.launchpad.net/openstack-ansible/+bug/1702123/+subscriptions From 1693343 at bugs.launchpad.net Wed Sep 13 17:01:04 2017 From: 1693343 at bugs.launchpad.net (OpenStack Infra) Date: Wed, 13 Sep 2017 17:01:04 -0000 Subject: [Openstack-security] [Bug 1693343] Fix included in openstack/ansible-hardening 16.0.0.0rc2 References: <149565524123.24782.9127842748680337701.malonedeb@soybean.canonical.com> Message-ID: <150532206503.17056.11146221462597977925.malone@soybean.canonical.com> This issue was fixed in the openstack/ansible-hardening 16.0.0.0rc2 release candidate. -- You received this bug notification because you are a member of OpenStack Security, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1693343 Title: V-71945 fails on CentOS 7 Status in openstack-ansible: Fix Released Bug description: 2017-05-24 17:35:21.290912 | TASK [openstack-ansible-security : V-71945 - If three unsuccessful logon attempts within 15 minutes occur the associated account must be locked.] *** 2017-05-24 17:35:21.312253 | Wednesday 24 May 2017 17:35:21 +0000 (0:00:00.320) 0:00:19.083 ********* 2017-05-24 17:35:21.468598 | fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg": "Path pam_password_file does not exist !", "rc": 257} To manage notifications about this bug go to: https://bugs.launchpad.net/openstack-ansible/+bug/1693343/+subscriptions From 1686110 at bugs.launchpad.net Wed Sep 13 17:01:08 2017 From: 1686110 at bugs.launchpad.net (OpenStack Infra) Date: Wed, 13 Sep 2017 17:01:08 -0000 Subject: [Openstack-security] [Bug 1686110] Fix included in openstack/ansible-hardening 16.0.0.0rc2 References: <20170425143229.16670.26982.malonedeb@wampee.canonical.com> Message-ID: <150532206866.5765.13225500084032998122.malone@wampee.canonical.com> This issue was fixed in the openstack/ansible-hardening 16.0.0.0rc2 release candidate. -- You received this bug notification because you are a member of OpenStack Security, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1686110 Title: AIDE configuration is set AFTER the initial run Status in openstack-ansible: Fix Released Bug description: The "Configure AIDE to verify additional properties" task runs *after* the tasks which do the AIDE initialization. This isn't a problem on CentOS since the default properties meet the STIG requirements, but it does affect Ubuntu. The result is that Ubuntu users may see a huge AIDE update upon their second AIDE run. To manage notifications about this bug go to: https://bugs.launchpad.net/openstack-ansible/+bug/1686110/+subscriptions From 1704398 at bugs.launchpad.net Sat Sep 16 04:17:44 2017 From: 1704398 at bugs.launchpad.net (Launchpad Bug Tracker) Date: Sat, 16 Sep 2017 04:17:44 -0000 Subject: [Openstack-security] [Bug 1704398] Re: Inheriting trunk subport segmentation details leaks information References: <150004087773.6647.4215962220125184884.malonedeb@gac.canonical.com> Message-ID: <150553546584.12243.8927827418585582164.malone@loganberry.canonical.com> [Expired for neutron because there has been no activity for 60 days.] ** Changed in: neutron Status: Incomplete => Expired -- You received this bug notification because you are a member of OpenStack Security, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1704398 Title: Inheriting trunk subport segmentation details leaks information Status in neutron: Expired Status in OpenStack Security Advisory: Won't Fix Bug description: The following information is discoverable for non-admin users of neutron despite the policy: * the segmentation type and id of all vlan based networks * the segmentation type of all non-vlan based networks Reproduction: configuration) [[local|localrc]] enable_plugin neutron ... enable_service q-trunk versions) devstack b79531a9f96736225a8991052a0be5767c217377 neutron 3a894d0b9e8e71bf7ee7e42350685065df5a8b3c 1) for vlan networks source openrc admin demo openstack network create net0 --provider-network-type vlan --provider-physical-network test --provider-segment 100 openstack network create net1 --provider-network-type vlan --provider-physical-network test --provider-segment 101 source openrc demo demo openstack port create port0 --network net0 openstack port create port1 --network net1 openstack network trunk create trunk0 --parent-port port0 openstack network trunk set trunk0 --subport port=port1,segmentation-type=inherit # demo user has no right to see segmentation type and id of vlan provider net openstack network show net1 # but both the type and id are available in the subport list openstack network subport list --trunk trunk0 2) for other network types source openrc admin demo openstack network create net0 openstack network create net1 source openrc demo demo openstack port create port0 --network net0 openstack port create port1 --network net1 openstack network trunk create trunk0 --parent-port port0 # the type of net1 comes back in the error message (in my environment 'vxlan') openstack network trunk set trunk0 --subport port=port1,segmentation-type=inherit While according to the default policy this information should be admin-only. See around here: https://github.com/openstack/neutron/blob/93390579da5d044a4e725faafd2b1f1d2d072a2a/etc/policy.json#L45 This behavior was introduced in response to this rfe: https://bugs.launchpad.net/neutron/+bug/1648129 Actually sharing this information is the point of the rfe. IIRC the concern about the information leak was even raised during a review, but it seems it went unaddressed. (Yep, I found the comment again: https://review.openstack.org/436756 comment at 03-07 21:12) I'm not sure if the segmentation details should be considered sensitive information or not. But the current behavior (admin-only here, not admin-only there) is clearly inconsistent. We could probably solve this by just synchronizing the default policies of provider network get operation and subport segmentation detail inheritance. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1704398/+subscriptions From lhinds at redhat.com Sun Sep 17 11:28:26 2017 From: lhinds at redhat.com (Luke Hinds) Date: Sun, 17 Sep 2017 11:28:26 -0000 Subject: [Openstack-security] [Bug 1668503] Re: sha512_crypt is insufficient, use pbkdf2_sha512 for password hashing References: <20170228052347.7890.80677.malonedeb@chaenomeles.canonical.com> Message-ID: <150564770980.16396.16721751922859956263.launchpad@soybean.canonical.com> ** Changed in: ossn Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of OpenStack Security, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1668503 Title: sha512_crypt is insufficient, use pbkdf2_sha512 for password hashing Status in OpenStack Identity (keystone): Fix Released Status in OpenStack Identity (keystone) mitaka series: Won't Fix Status in OpenStack Identity (keystone) newton series: Won't Fix Status in OpenStack Identity (keystone) ocata series: Won't Fix Status in OpenStack Identity (keystone) pike series: Fix Released Status in OpenStack Security Advisory: Won't Fix Status in OpenStack Security Notes: Fix Released Bug description: Keystone uses sha512_crypt for password hashing. This is insufficient and provides limited protection (even with 10,000 rounds) against brute-forcing of the password hashes (especially with FPGAs and/or GPU processing). The correct mechanism is to use bcrypt, scrypt, or pbkdf2_sha512 instead of sha512_crypt. This bug is marked as public security as bug #1543048 has already highlighted this issue. To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1668503/+subscriptions From 1713783 at bugs.launchpad.net Mon Sep 18 13:37:45 2017 From: 1713783 at bugs.launchpad.net (OpenStack Infra) Date: Mon, 18 Sep 2017 13:37:45 -0000 Subject: [Openstack-security] [Bug 1713783] Related fix proposed to nova (stable/pike) References: <150402703560.19393.11649471063519714290.malonedeb@chaenomeles.canonical.com> Message-ID: <150574186550.5156.1471247657780184300.malone@wampee.canonical.com> Related fix proposed to branch: stable/pike Review: https://review.openstack.org/504943 -- You received this bug notification because you are a member of OpenStack Security, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1713783 Title: After failed evacuation the recovered source compute tries to delete the instance Status in OpenStack Compute (nova): In Progress Status in OpenStack Compute (nova) newton series: Triaged Status in OpenStack Compute (nova) ocata series: Triaged Status in OpenStack Compute (nova) pike series: Triaged Status in OpenStack Security Advisory: Won't Fix Bug description: Description =========== In case of a failed evacuation attempt the status of the migration is 'accepted' instead of 'failed' so when source compute is recovered the compute manager tries to delete the instance from the source host. However a secondary fault prevents deleting the allocation in placement so the actual deletion of the instance fails as well. Steps to reproduce ================== The following functional test reproduces the bug: https://review.openstack.org/#/c/498482/ What it does: initiate evacuation when no valid host is available and evacuation fails, but nova manager still tries to delete the instance. Logs:     2017-08-29 19:11:15,751 ERROR [oslo_messaging.rpc.server] Exception during message handling     NoValidHost: No valid host was found. There are not enough hosts available.     2017-08-29 19:11:16,103 INFO [nova.tests.functional.test_servers] Running periodic for compute1 (host1)     2017-08-29 19:11:16,115 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,120 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,131 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/allocations" status: 200 len: 152 microversion: 1.0     2017-08-29 19:11:16,138 INFO [nova.compute.resource_tracker] Final resource view: name=host1 phys_ram=8192MB used_ram=1024MB phys_disk=1028GB used_disk=1GB total_vcpus=10 used_vcpus=1 pci_stats=[]     2017-08-29 19:11:16,146 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,151 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,152 INFO [nova.tests.functional.test_servers] Running periodic for compute2 (host2)     2017-08-29 19:11:16,163 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,168 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,176 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/allocations" status: 200 len: 54 microversion: 1.0     2017-08-29 19:11:16,184 INFO [nova.compute.resource_tracker] Final resource view: name=host2 phys_ram=8192MB used_ram=512MB phys_disk=1028GB used_disk=0GB total_vcpus=10 used_vcpus=0 pci_stats=[]     2017-08-29 19:11:16,192 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,197 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,198 INFO [nova.tests.functional.test_servers] Finished with periodics     2017-08-29 19:11:16,255 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/6f70656e737461636b20342065766572/servers/5058200c-478e-4449-88c1-906fdd572662" status: 200 len: 1875 microversion: 2.53 time: 0.056198     2017-08-29 19:11:16,262 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/6f70656e737461636b20342065766572/os-migrations" status: 200 len: 373 microversion: 2.53 time: 0.004618     2017-08-29 19:11:16,280 INFO [nova.api.openstack.requestlog] 127.0.0.1 "PUT /v2.1/6f70656e737461636b20342065766572/os-services/c269bc74-4720-4de4-a6e5-889080b892a0" status: 200 len: 245 microversion: 2.53 time: 0.016442     2017-08-29 19:11:16,281 INFO [nova.service] Starting compute node (version 16.0.0)     2017-08-29 19:11:16,296 INFO [nova.compute.manager] Deleting instance as it has been evacuated from this host To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1713783/+subscriptions From 1713783 at bugs.launchpad.net Mon Sep 18 14:50:56 2017 From: 1713783 at bugs.launchpad.net (OpenStack Infra) Date: Mon, 18 Sep 2017 14:50:56 -0000 Subject: [Openstack-security] [Bug 1713783] Re: After failed evacuation the recovered source compute tries to delete the instance References: <150402703560.19393.11649471063519714290.malonedeb@chaenomeles.canonical.com> Message-ID: <150574625619.5308.2964953994590472884.malone@wampee.canonical.com> Reviewed: https://review.openstack.org/499237 Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=a8ebf5f1aac080854704e27146e8c98b053c6224 Submitter: Jenkins Branch: master commit a8ebf5f1aac080854704e27146e8c98b053c6224 Author: Előd Illés Date: Wed Aug 30 16:54:36 2017 +0200 Set error state after failed evacuation When evacuation fails with NoValidHost, the migration status remains 'accepted' instead of 'error'. This causes problem in case the compute service starts up again and looks for evacuations with status 'accepted', as it then removes the local instances for those evacuations even though the instance was never actually evacuated to another host. Change-Id: I06d78c744fa75ae5f34c5cfa76bc3c9460767b84 Closes-Bug: #1713783 ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of OpenStack Security, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1713783 Title: After failed evacuation the recovered source compute tries to delete the instance Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) newton series: Triaged Status in OpenStack Compute (nova) ocata series: Triaged Status in OpenStack Compute (nova) pike series: Triaged Status in OpenStack Security Advisory: Won't Fix Bug description: Description =========== In case of a failed evacuation attempt the status of the migration is 'accepted' instead of 'failed' so when source compute is recovered the compute manager tries to delete the instance from the source host. However a secondary fault prevents deleting the allocation in placement so the actual deletion of the instance fails as well. Steps to reproduce ================== The following functional test reproduces the bug: https://review.openstack.org/#/c/498482/ What it does: initiate evacuation when no valid host is available and evacuation fails, but nova manager still tries to delete the instance. Logs:     2017-08-29 19:11:15,751 ERROR [oslo_messaging.rpc.server] Exception during message handling     NoValidHost: No valid host was found. There are not enough hosts available.     2017-08-29 19:11:16,103 INFO [nova.tests.functional.test_servers] Running periodic for compute1 (host1)     2017-08-29 19:11:16,115 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,120 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,131 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/allocations" status: 200 len: 152 microversion: 1.0     2017-08-29 19:11:16,138 INFO [nova.compute.resource_tracker] Final resource view: name=host1 phys_ram=8192MB used_ram=1024MB phys_disk=1028GB used_disk=1GB total_vcpus=10 used_vcpus=1 pci_stats=[]     2017-08-29 19:11:16,146 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,151 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,152 INFO [nova.tests.functional.test_servers] Running periodic for compute2 (host2)     2017-08-29 19:11:16,163 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,168 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,176 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/allocations" status: 200 len: 54 microversion: 1.0     2017-08-29 19:11:16,184 INFO [nova.compute.resource_tracker] Final resource view: name=host2 phys_ram=8192MB used_ram=512MB phys_disk=1028GB used_disk=0GB total_vcpus=10 used_vcpus=0 pci_stats=[]     2017-08-29 19:11:16,192 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,197 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,198 INFO [nova.tests.functional.test_servers] Finished with periodics     2017-08-29 19:11:16,255 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/6f70656e737461636b20342065766572/servers/5058200c-478e-4449-88c1-906fdd572662" status: 200 len: 1875 microversion: 2.53 time: 0.056198     2017-08-29 19:11:16,262 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/6f70656e737461636b20342065766572/os-migrations" status: 200 len: 373 microversion: 2.53 time: 0.004618     2017-08-29 19:11:16,280 INFO [nova.api.openstack.requestlog] 127.0.0.1 "PUT /v2.1/6f70656e737461636b20342065766572/os-services/c269bc74-4720-4de4-a6e5-889080b892a0" status: 200 len: 245 microversion: 2.53 time: 0.016442     2017-08-29 19:11:16,281 INFO [nova.service] Starting compute node (version 16.0.0)     2017-08-29 19:11:16,296 INFO [nova.compute.manager] Deleting instance as it has been evacuated from this host To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1713783/+subscriptions From 1668410 at bugs.launchpad.net Mon Sep 18 15:32:12 2017 From: 1668410 at bugs.launchpad.net (Launchpad Bug Tracker) Date: Mon, 18 Sep 2017 15:32:12 -0000 Subject: [Openstack-security] [Bug 1668410] Re: [SRU] Infinite loop trying to delete deleted HA router References: <20170227213540.6394.32961.malonedeb@wampee.canonical.com> Message-ID: <150574873730.31869.6129042550405371366.malone@ackee.canonical.com> This bug was fixed in the package neutron - 2:8.4.0-0ubuntu5 --------------- neutron (2:8.4.0-0ubuntu5) xenial; urgency=medium * d/p/l3-ha-don-t-send-routers-without-_ha_interface.patch: Backport fix for l3 ha: don't send routers without '_ha_interface' (LP: #1668410) -- Hua Zhang Thu, 24 Aug 2017 12:19:23 +0800 ** Changed in: neutron (Ubuntu Xenial) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of OpenStack Security, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1668410 Title: [SRU] Infinite loop trying to delete deleted HA router Status in Ubuntu Cloud Archive: Invalid Status in Ubuntu Cloud Archive mitaka series: Triaged Status in neutron: In Progress Status in OpenStack Security Advisory: Won't Fix Status in neutron package in Ubuntu: Invalid Status in neutron source package in Xenial: Fix Released Bug description: [Descriptoin] When deleting a router the logfile is filled up. See full log - http://paste.ubuntu.com/25429257/ I can see the error 'Error while deleting router c0dab368-5ac8-4996-88c9-f5d345a774a6' occured 3343386 times from _safe_router_removed() [1]: $ grep -r 'Error while deleting router c0dab368-5ac8-4996-88c9-f5d345a774a6' |wc -l 3343386 This _safe_router_removed() is invoked by L488 [2], if _safe_router_removed() goes wrong it will return False, then self._resync_router(update) [3] will make the code _safe_router_removed be run again and again. So we saw so many errors 'Error while deleting router XXXXX'. [1] https://github.com/openstack/neutron/blob/mitaka-eol/neutron/agent/l3/agent.py#L361 [2] https://github.com/openstack/neutron/blob/mitaka-eol/neutron/agent/l3/agent.py#L488 [3] https://github.com/openstack/neutron/blob/mitaka-eol/neutron/agent/l3/agent.py#L457 [Test Case] That's because race condition between neutron server and L3 agent, after neutron server deletes HA interfaces the L3 agent may sync a HA router without HA interface info (just need to trigger L708[1] after deleting HA interfaces and before deleting HA router). If we delete HA router at this time, this problem will happen. So test case we design is as below: 1, First update fixed package, and restart neutron-server by 'sudo service neutron-server restart' 2, Create ha_router neutron router-create harouter --ha=True 3, Delete ports associated with ha_router before deleting ha_router neutron router-port-list harouter |grep 'HA port' |awk '{print $2}' |xargs -l neutron port-delete neutron router-port-list harouter 4, Update ha_router to trigger l3-agent to update ha_router info without ha_port into self.router_info neutron router-update harouter --description=test 5, Delete ha_router this time neutron router-delete harouter [1] https://github.com/openstack/neutron/blob/mitaka- eol/neutron/db/l3_hamode_db.py#L708 [Regression Potential] The fixed patch [1] for neutron-server will no longer return ha_router which is missing ha_ports, so L488 will no longer have chance to call _safe_router_removed() for a ha_router, so the problem has been fundamentally fixed by this patch and no regression potential. Besides, this fixed patch has been in mitaka-eol branch now, and neutron-server mitaka package is based on neutron-8.4.0, so we need to backport it to xenial and mitaka. $ git tag --contains 8c77ee6b20dd38cc0246e854711cb91cffe3a069 mitaka-eol [1] https://review.openstack.org/#/c/440799/2/neutron/db/l3_hamode_db.py [2] https://github.com/openstack/neutron/blob/mitaka-eol/neutron/agent/l3/agent.py#L488 To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1668410/+subscriptions From 1668410 at bugs.launchpad.net Mon Sep 18 17:47:02 2017 From: 1668410 at bugs.launchpad.net (Ryan Beisner) Date: Mon, 18 Sep 2017 17:47:02 -0000 Subject: [Openstack-security] [Bug 1668410] Please test proposed package References: <20170227213540.6394.32961.malonedeb@wampee.canonical.com> Message-ID: <150575682214.13585.5307183385591824731.malone@gac.canonical.com> Hello Ann, or anyone else affected, Accepted neutron into mitaka-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository. Please help us by testing this new package. To enable the -proposed repository: sudo add-apt-repository cloud-archive:mitaka-proposed sudo apt-get update Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-mitaka-needed to verification-mitaka-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-mitaka-failed. In either case, details of your testing will help us make a better decision. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance! ** Changed in: cloud-archive/mitaka Status: Triaged => Fix Committed ** Tags added: verification-mitaka-needed -- You received this bug notification because you are a member of OpenStack Security, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1668410 Title: [SRU] Infinite loop trying to delete deleted HA router Status in Ubuntu Cloud Archive: Invalid Status in Ubuntu Cloud Archive mitaka series: Fix Committed Status in neutron: In Progress Status in OpenStack Security Advisory: Won't Fix Status in neutron package in Ubuntu: Invalid Status in neutron source package in Xenial: Fix Released Bug description: [Descriptoin] When deleting a router the logfile is filled up. See full log - http://paste.ubuntu.com/25429257/ I can see the error 'Error while deleting router c0dab368-5ac8-4996-88c9-f5d345a774a6' occured 3343386 times from _safe_router_removed() [1]: $ grep -r 'Error while deleting router c0dab368-5ac8-4996-88c9-f5d345a774a6' |wc -l 3343386 This _safe_router_removed() is invoked by L488 [2], if _safe_router_removed() goes wrong it will return False, then self._resync_router(update) [3] will make the code _safe_router_removed be run again and again. So we saw so many errors 'Error while deleting router XXXXX'. [1] https://github.com/openstack/neutron/blob/mitaka-eol/neutron/agent/l3/agent.py#L361 [2] https://github.com/openstack/neutron/blob/mitaka-eol/neutron/agent/l3/agent.py#L488 [3] https://github.com/openstack/neutron/blob/mitaka-eol/neutron/agent/l3/agent.py#L457 [Test Case] That's because race condition between neutron server and L3 agent, after neutron server deletes HA interfaces the L3 agent may sync a HA router without HA interface info (just need to trigger L708[1] after deleting HA interfaces and before deleting HA router). If we delete HA router at this time, this problem will happen. So test case we design is as below: 1, First update fixed package, and restart neutron-server by 'sudo service neutron-server restart' 2, Create ha_router neutron router-create harouter --ha=True 3, Delete ports associated with ha_router before deleting ha_router neutron router-port-list harouter |grep 'HA port' |awk '{print $2}' |xargs -l neutron port-delete neutron router-port-list harouter 4, Update ha_router to trigger l3-agent to update ha_router info without ha_port into self.router_info neutron router-update harouter --description=test 5, Delete ha_router this time neutron router-delete harouter [1] https://github.com/openstack/neutron/blob/mitaka- eol/neutron/db/l3_hamode_db.py#L708 [Regression Potential] The fixed patch [1] for neutron-server will no longer return ha_router which is missing ha_ports, so L488 will no longer have chance to call _safe_router_removed() for a ha_router, so the problem has been fundamentally fixed by this patch and no regression potential. Besides, this fixed patch has been in mitaka-eol branch now, and neutron-server mitaka package is based on neutron-8.4.0, so we need to backport it to xenial and mitaka. $ git tag --contains 8c77ee6b20dd38cc0246e854711cb91cffe3a069 mitaka-eol [1] https://review.openstack.org/#/c/440799/2/neutron/db/l3_hamode_db.py [2] https://github.com/openstack/neutron/blob/mitaka-eol/neutron/agent/l3/agent.py#L488 To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1668410/+subscriptions From fungi at yuggoth.org Mon Sep 18 19:28:18 2017 From: fungi at yuggoth.org (Jeremy Stanley) Date: Mon, 18 Sep 2017 19:28:18 -0000 Subject: [Openstack-security] [Bug 1668503] Re: sha512_crypt is insufficient, use pbkdf2_sha512 for password hashing References: <20170228052347.7890.80677.malonedeb@chaenomeles.canonical.com> Message-ID: <150576289817.13324.4965661686286897032.malone@gac.canonical.com> It's worth noting that on review of the original source, keystone.common.password_hashing.hash_password() was not using a bare SHA-512 hash like hashlib.sha512() but instead already performed key derivation via passlib.hash.sha512_crypt.hash(), a wrapper for the vaguely-named SHA-512 method for glibc's crypt() function. Per the ML thread starting at http://www.openwall.com/lists/oss- security/2017/09/17/2 the change itself is at least for the better, but the risk from the previous implementation is not nearly as severe as this bug and the subsequent OSSN would seem to suggest. Apologies if my poorly-researched comment in bug 1543048 led to a fire drill, I'll propose an errata revision to OSSN-0081 this week. -- You received this bug notification because you are a member of OpenStack Security, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1668503 Title: sha512_crypt is insufficient, use pbkdf2_sha512 for password hashing Status in OpenStack Identity (keystone): Fix Released Status in OpenStack Identity (keystone) mitaka series: Won't Fix Status in OpenStack Identity (keystone) newton series: Won't Fix Status in OpenStack Identity (keystone) ocata series: Won't Fix Status in OpenStack Identity (keystone) pike series: Fix Released Status in OpenStack Security Advisory: Won't Fix Status in OpenStack Security Notes: Fix Released Bug description: Keystone uses sha512_crypt for password hashing. This is insufficient and provides limited protection (even with 10,000 rounds) against brute-forcing of the password hashes (especially with FPGAs and/or GPU processing). The correct mechanism is to use bcrypt, scrypt, or pbkdf2_sha512 instead of sha512_crypt. This bug is marked as public security as bug #1543048 has already highlighted this issue. To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1668503/+subscriptions From morgan.fainberg at gmail.com Mon Sep 18 21:43:35 2017 From: morgan.fainberg at gmail.com (Morgan Fainberg) Date: Mon, 18 Sep 2017 21:43:35 -0000 Subject: [Openstack-security] [Bug 1668503] Re: sha512_crypt is insufficient, use pbkdf2_sha512 for password hashing References: <20170228052347.7890.80677.malonedeb@chaenomeles.canonical.com> Message-ID: <150577101550.23063.13267815780984618957.malone@wampee.canonical.com> I also apologize if I didn't make it clear it was pbkdf still with sha512, just that sha512_crypt (even pbkdf) is not sufficient for applications (e.g. exposing your shadow file on linux *is* still a compromise). The implication is that there is significantly more surface area for Keystone to deal with than a system shadow file, therefore we should be using bcrypt, scrypt, or at *least* pbkdf2 instead of sha512_crypt. tl;dr I may have also missed on communicating sha512_crypt is pbkdf rather than bare sha512 hashing. -- You received this bug notification because you are a member of OpenStack Security, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1668503 Title: sha512_crypt is insufficient, use pbkdf2_sha512 for password hashing Status in OpenStack Identity (keystone): Fix Released Status in OpenStack Identity (keystone) mitaka series: Won't Fix Status in OpenStack Identity (keystone) newton series: Won't Fix Status in OpenStack Identity (keystone) ocata series: Won't Fix Status in OpenStack Identity (keystone) pike series: Fix Released Status in OpenStack Security Advisory: Won't Fix Status in OpenStack Security Notes: Fix Released Bug description: Keystone uses sha512_crypt for password hashing. This is insufficient and provides limited protection (even with 10,000 rounds) against brute-forcing of the password hashes (especially with FPGAs and/or GPU processing). The correct mechanism is to use bcrypt, scrypt, or pbkdf2_sha512 instead of sha512_crypt. This bug is marked as public security as bug #1543048 has already highlighted this issue. To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1668503/+subscriptions From 1713783 at bugs.launchpad.net Tue Sep 19 00:14:18 2017 From: 1713783 at bugs.launchpad.net (OpenStack Infra) Date: Tue, 19 Sep 2017 00:14:18 -0000 Subject: [Openstack-security] [Bug 1713783] Re: After failed evacuation the recovered source compute tries to delete the instance References: <150402703560.19393.11649471063519714290.malonedeb@chaenomeles.canonical.com> Message-ID: <150578005831.13703.12221603358544780758.malone@gac.canonical.com> Fix proposed to branch: stable/pike Review: https://review.openstack.org/504979 ** Changed in: nova/pike Status: Triaged => In Progress ** Changed in: nova/pike Assignee: (unassigned) => Matt Riedemann (mriedem) -- You received this bug notification because you are a member of OpenStack Security, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1713783 Title: After failed evacuation the recovered source compute tries to delete the instance Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) newton series: Triaged Status in OpenStack Compute (nova) ocata series: Triaged Status in OpenStack Compute (nova) pike series: In Progress Status in OpenStack Security Advisory: Won't Fix Bug description: Description =========== In case of a failed evacuation attempt the status of the migration is 'accepted' instead of 'failed' so when source compute is recovered the compute manager tries to delete the instance from the source host. However a secondary fault prevents deleting the allocation in placement so the actual deletion of the instance fails as well. Steps to reproduce ================== The following functional test reproduces the bug: https://review.openstack.org/#/c/498482/ What it does: initiate evacuation when no valid host is available and evacuation fails, but nova manager still tries to delete the instance. Logs:     2017-08-29 19:11:15,751 ERROR [oslo_messaging.rpc.server] Exception during message handling     NoValidHost: No valid host was found. There are not enough hosts available.     2017-08-29 19:11:16,103 INFO [nova.tests.functional.test_servers] Running periodic for compute1 (host1)     2017-08-29 19:11:16,115 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,120 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,131 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/allocations" status: 200 len: 152 microversion: 1.0     2017-08-29 19:11:16,138 INFO [nova.compute.resource_tracker] Final resource view: name=host1 phys_ram=8192MB used_ram=1024MB phys_disk=1028GB used_disk=1GB total_vcpus=10 used_vcpus=1 pci_stats=[]     2017-08-29 19:11:16,146 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,151 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,152 INFO [nova.tests.functional.test_servers] Running periodic for compute2 (host2)     2017-08-29 19:11:16,163 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,168 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,176 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/allocations" status: 200 len: 54 microversion: 1.0     2017-08-29 19:11:16,184 INFO [nova.compute.resource_tracker] Final resource view: name=host2 phys_ram=8192MB used_ram=512MB phys_disk=1028GB used_disk=0GB total_vcpus=10 used_vcpus=0 pci_stats=[]     2017-08-29 19:11:16,192 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,197 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,198 INFO [nova.tests.functional.test_servers] Finished with periodics     2017-08-29 19:11:16,255 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/6f70656e737461636b20342065766572/servers/5058200c-478e-4449-88c1-906fdd572662" status: 200 len: 1875 microversion: 2.53 time: 0.056198     2017-08-29 19:11:16,262 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/6f70656e737461636b20342065766572/os-migrations" status: 200 len: 373 microversion: 2.53 time: 0.004618     2017-08-29 19:11:16,280 INFO [nova.api.openstack.requestlog] 127.0.0.1 "PUT /v2.1/6f70656e737461636b20342065766572/os-services/c269bc74-4720-4de4-a6e5-889080b892a0" status: 200 len: 245 microversion: 2.53 time: 0.016442     2017-08-29 19:11:16,281 INFO [nova.service] Starting compute node (version 16.0.0)     2017-08-29 19:11:16,296 INFO [nova.compute.manager] Deleting instance as it has been evacuated from this host To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1713783/+subscriptions From 1713783 at bugs.launchpad.net Tue Sep 19 02:58:54 2017 From: 1713783 at bugs.launchpad.net (OpenStack Infra) Date: Tue, 19 Sep 2017 02:58:54 -0000 Subject: [Openstack-security] [Bug 1713783] Re: After failed evacuation the recovered source compute tries to delete the instance References: <150402703560.19393.11649471063519714290.malonedeb@chaenomeles.canonical.com> Message-ID: <150578993430.24618.4707045128007385432.malone@chaenomeles.canonical.com> Reviewed: https://review.openstack.org/504943 Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=e87d9f51abea0b1c97628ef02435157da4b334ec Submitter: Jenkins Branch: stable/pike commit e87d9f51abea0b1c97628ef02435157da4b334ec Author: Előd Illés Date: Fri Sep 1 15:36:20 2017 +0200 Functional test for regression bug #1713783 Add functional test for evacuation, when no valid host available. Migration should end up in 'error' state. Change-Id: I1adc20f2a5261e6906a18b9aee5cd2c8ecf0cf4d Related-bug: #1713783 (cherry picked from commit 5687c170ea003aafa2720fb00af9d84836bb41df) ** Tags added: in-stable-pike -- You received this bug notification because you are a member of OpenStack Security, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1713783 Title: After failed evacuation the recovered source compute tries to delete the instance Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) newton series: Triaged Status in OpenStack Compute (nova) ocata series: Triaged Status in OpenStack Compute (nova) pike series: In Progress Status in OpenStack Security Advisory: Won't Fix Bug description: Description =========== In case of a failed evacuation attempt the status of the migration is 'accepted' instead of 'failed' so when source compute is recovered the compute manager tries to delete the instance from the source host. However a secondary fault prevents deleting the allocation in placement so the actual deletion of the instance fails as well. Steps to reproduce ================== The following functional test reproduces the bug: https://review.openstack.org/#/c/498482/ What it does: initiate evacuation when no valid host is available and evacuation fails, but nova manager still tries to delete the instance. Logs:     2017-08-29 19:11:15,751 ERROR [oslo_messaging.rpc.server] Exception during message handling     NoValidHost: No valid host was found. There are not enough hosts available.     2017-08-29 19:11:16,103 INFO [nova.tests.functional.test_servers] Running periodic for compute1 (host1)     2017-08-29 19:11:16,115 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,120 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,131 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/allocations" status: 200 len: 152 microversion: 1.0     2017-08-29 19:11:16,138 INFO [nova.compute.resource_tracker] Final resource view: name=host1 phys_ram=8192MB used_ram=1024MB phys_disk=1028GB used_disk=1GB total_vcpus=10 used_vcpus=1 pci_stats=[]     2017-08-29 19:11:16,146 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,151 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,152 INFO [nova.tests.functional.test_servers] Running periodic for compute2 (host2)     2017-08-29 19:11:16,163 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,168 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,176 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/allocations" status: 200 len: 54 microversion: 1.0     2017-08-29 19:11:16,184 INFO [nova.compute.resource_tracker] Final resource view: name=host2 phys_ram=8192MB used_ram=512MB phys_disk=1028GB used_disk=0GB total_vcpus=10 used_vcpus=0 pci_stats=[]     2017-08-29 19:11:16,192 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,197 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,198 INFO [nova.tests.functional.test_servers] Finished with periodics     2017-08-29 19:11:16,255 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/6f70656e737461636b20342065766572/servers/5058200c-478e-4449-88c1-906fdd572662" status: 200 len: 1875 microversion: 2.53 time: 0.056198     2017-08-29 19:11:16,262 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/6f70656e737461636b20342065766572/os-migrations" status: 200 len: 373 microversion: 2.53 time: 0.004618     2017-08-29 19:11:16,280 INFO [nova.api.openstack.requestlog] 127.0.0.1 "PUT /v2.1/6f70656e737461636b20342065766572/os-services/c269bc74-4720-4de4-a6e5-889080b892a0" status: 200 len: 245 microversion: 2.53 time: 0.016442     2017-08-29 19:11:16,281 INFO [nova.service] Starting compute node (version 16.0.0)     2017-08-29 19:11:16,296 INFO [nova.compute.manager] Deleting instance as it has been evacuated from this host To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1713783/+subscriptions From 1713783 at bugs.launchpad.net Tue Sep 19 10:40:23 2017 From: 1713783 at bugs.launchpad.net (OpenStack Infra) Date: Tue, 19 Sep 2017 10:40:23 -0000 Subject: [Openstack-security] [Bug 1713783] Related fix proposed to nova (stable/ocata) References: <150402703560.19393.11649471063519714290.malonedeb@chaenomeles.canonical.com> Message-ID: <150581762314.16751.9346258194437533579.malone@soybean.canonical.com> Related fix proposed to branch: stable/ocata Review: https://review.openstack.org/505160 -- You received this bug notification because you are a member of OpenStack Security, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1713783 Title: After failed evacuation the recovered source compute tries to delete the instance Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) newton series: Triaged Status in OpenStack Compute (nova) ocata series: Triaged Status in OpenStack Compute (nova) pike series: In Progress Status in OpenStack Security Advisory: Won't Fix Bug description: Description =========== In case of a failed evacuation attempt the status of the migration is 'accepted' instead of 'failed' so when source compute is recovered the compute manager tries to delete the instance from the source host. However a secondary fault prevents deleting the allocation in placement so the actual deletion of the instance fails as well. Steps to reproduce ================== The following functional test reproduces the bug: https://review.openstack.org/#/c/498482/ What it does: initiate evacuation when no valid host is available and evacuation fails, but nova manager still tries to delete the instance. Logs:     2017-08-29 19:11:15,751 ERROR [oslo_messaging.rpc.server] Exception during message handling     NoValidHost: No valid host was found. There are not enough hosts available.     2017-08-29 19:11:16,103 INFO [nova.tests.functional.test_servers] Running periodic for compute1 (host1)     2017-08-29 19:11:16,115 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,120 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,131 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/allocations" status: 200 len: 152 microversion: 1.0     2017-08-29 19:11:16,138 INFO [nova.compute.resource_tracker] Final resource view: name=host1 phys_ram=8192MB used_ram=1024MB phys_disk=1028GB used_disk=1GB total_vcpus=10 used_vcpus=1 pci_stats=[]     2017-08-29 19:11:16,146 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,151 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,152 INFO [nova.tests.functional.test_servers] Running periodic for compute2 (host2)     2017-08-29 19:11:16,163 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,168 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,176 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/allocations" status: 200 len: 54 microversion: 1.0     2017-08-29 19:11:16,184 INFO [nova.compute.resource_tracker] Final resource view: name=host2 phys_ram=8192MB used_ram=512MB phys_disk=1028GB used_disk=0GB total_vcpus=10 used_vcpus=0 pci_stats=[]     2017-08-29 19:11:16,192 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,197 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,198 INFO [nova.tests.functional.test_servers] Finished with periodics     2017-08-29 19:11:16,255 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/6f70656e737461636b20342065766572/servers/5058200c-478e-4449-88c1-906fdd572662" status: 200 len: 1875 microversion: 2.53 time: 0.056198     2017-08-29 19:11:16,262 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/6f70656e737461636b20342065766572/os-migrations" status: 200 len: 373 microversion: 2.53 time: 0.004618     2017-08-29 19:11:16,280 INFO [nova.api.openstack.requestlog] 127.0.0.1 "PUT /v2.1/6f70656e737461636b20342065766572/os-services/c269bc74-4720-4de4-a6e5-889080b892a0" status: 200 len: 245 microversion: 2.53 time: 0.016442     2017-08-29 19:11:16,281 INFO [nova.service] Starting compute node (version 16.0.0)     2017-08-29 19:11:16,296 INFO [nova.compute.manager] Deleting instance as it has been evacuated from this host To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1713783/+subscriptions From 1668410 at bugs.launchpad.net Tue Sep 19 14:00:00 2017 From: 1668410 at bugs.launchpad.net (Ryan Beisner) Date: Tue, 19 Sep 2017 14:00:00 -0000 Subject: [Openstack-security] [Bug 1668410] Update Released References: <20170227213540.6394.32961.malonedeb@wampee.canonical.com> Message-ID: <150582960043.12107.6425603759508770523.malone@gac.canonical.com> The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions. -- You received this bug notification because you are a member of OpenStack Security, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1668410 Title: [SRU] Infinite loop trying to delete deleted HA router Status in Ubuntu Cloud Archive: Invalid Status in Ubuntu Cloud Archive mitaka series: Fix Released Status in neutron: In Progress Status in OpenStack Security Advisory: Won't Fix Status in neutron package in Ubuntu: Invalid Status in neutron source package in Xenial: Fix Released Bug description: [Descriptoin] When deleting a router the logfile is filled up. See full log - http://paste.ubuntu.com/25429257/ I can see the error 'Error while deleting router c0dab368-5ac8-4996-88c9-f5d345a774a6' occured 3343386 times from _safe_router_removed() [1]: $ grep -r 'Error while deleting router c0dab368-5ac8-4996-88c9-f5d345a774a6' |wc -l 3343386 This _safe_router_removed() is invoked by L488 [2], if _safe_router_removed() goes wrong it will return False, then self._resync_router(update) [3] will make the code _safe_router_removed be run again and again. So we saw so many errors 'Error while deleting router XXXXX'. [1] https://github.com/openstack/neutron/blob/mitaka-eol/neutron/agent/l3/agent.py#L361 [2] https://github.com/openstack/neutron/blob/mitaka-eol/neutron/agent/l3/agent.py#L488 [3] https://github.com/openstack/neutron/blob/mitaka-eol/neutron/agent/l3/agent.py#L457 [Test Case] That's because race condition between neutron server and L3 agent, after neutron server deletes HA interfaces the L3 agent may sync a HA router without HA interface info (just need to trigger L708[1] after deleting HA interfaces and before deleting HA router). If we delete HA router at this time, this problem will happen. So test case we design is as below: 1, First update fixed package, and restart neutron-server by 'sudo service neutron-server restart' 2, Create ha_router neutron router-create harouter --ha=True 3, Delete ports associated with ha_router before deleting ha_router neutron router-port-list harouter |grep 'HA port' |awk '{print $2}' |xargs -l neutron port-delete neutron router-port-list harouter 4, Update ha_router to trigger l3-agent to update ha_router info without ha_port into self.router_info neutron router-update harouter --description=test 5, Delete ha_router this time neutron router-delete harouter [1] https://github.com/openstack/neutron/blob/mitaka- eol/neutron/db/l3_hamode_db.py#L708 [Regression Potential] The fixed patch [1] for neutron-server will no longer return ha_router which is missing ha_ports, so L488 will no longer have chance to call _safe_router_removed() for a ha_router, so the problem has been fundamentally fixed by this patch and no regression potential. Besides, this fixed patch has been in mitaka-eol branch now, and neutron-server mitaka package is based on neutron-8.4.0, so we need to backport it to xenial and mitaka. $ git tag --contains 8c77ee6b20dd38cc0246e854711cb91cffe3a069 mitaka-eol [1] https://review.openstack.org/#/c/440799/2/neutron/db/l3_hamode_db.py [2] https://github.com/openstack/neutron/blob/mitaka-eol/neutron/agent/l3/agent.py#L488 To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1668410/+subscriptions From 1668410 at bugs.launchpad.net Tue Sep 19 14:00:02 2017 From: 1668410 at bugs.launchpad.net (Ryan Beisner) Date: Tue, 19 Sep 2017 14:00:02 -0000 Subject: [Openstack-security] [Bug 1668410] Re: [SRU] Infinite loop trying to delete deleted HA router References: <20170227213540.6394.32961.malonedeb@wampee.canonical.com> Message-ID: <150582960267.13396.10030908930875818269.malone@gac.canonical.com> This bug was fixed in the package neutron - 2:8.4.0-0ubuntu5~cloud0 --------------- neutron (2:8.4.0-0ubuntu5~cloud0) trusty-mitaka; urgency=medium . * New update for the Ubuntu Cloud Archive. . neutron (2:8.4.0-0ubuntu5) xenial; urgency=medium . * d/p/l3-ha-don-t-send-routers-without-_ha_interface.patch: Backport fix for l3 ha: don't send routers without '_ha_interface' (LP: #1668410) ** Changed in: cloud-archive/mitaka Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of OpenStack Security, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1668410 Title: [SRU] Infinite loop trying to delete deleted HA router Status in Ubuntu Cloud Archive: Invalid Status in Ubuntu Cloud Archive mitaka series: Fix Released Status in neutron: In Progress Status in OpenStack Security Advisory: Won't Fix Status in neutron package in Ubuntu: Invalid Status in neutron source package in Xenial: Fix Released Bug description: [Descriptoin] When deleting a router the logfile is filled up. See full log - http://paste.ubuntu.com/25429257/ I can see the error 'Error while deleting router c0dab368-5ac8-4996-88c9-f5d345a774a6' occured 3343386 times from _safe_router_removed() [1]: $ grep -r 'Error while deleting router c0dab368-5ac8-4996-88c9-f5d345a774a6' |wc -l 3343386 This _safe_router_removed() is invoked by L488 [2], if _safe_router_removed() goes wrong it will return False, then self._resync_router(update) [3] will make the code _safe_router_removed be run again and again. So we saw so many errors 'Error while deleting router XXXXX'. [1] https://github.com/openstack/neutron/blob/mitaka-eol/neutron/agent/l3/agent.py#L361 [2] https://github.com/openstack/neutron/blob/mitaka-eol/neutron/agent/l3/agent.py#L488 [3] https://github.com/openstack/neutron/blob/mitaka-eol/neutron/agent/l3/agent.py#L457 [Test Case] That's because race condition between neutron server and L3 agent, after neutron server deletes HA interfaces the L3 agent may sync a HA router without HA interface info (just need to trigger L708[1] after deleting HA interfaces and before deleting HA router). If we delete HA router at this time, this problem will happen. So test case we design is as below: 1, First update fixed package, and restart neutron-server by 'sudo service neutron-server restart' 2, Create ha_router neutron router-create harouter --ha=True 3, Delete ports associated with ha_router before deleting ha_router neutron router-port-list harouter |grep 'HA port' |awk '{print $2}' |xargs -l neutron port-delete neutron router-port-list harouter 4, Update ha_router to trigger l3-agent to update ha_router info without ha_port into self.router_info neutron router-update harouter --description=test 5, Delete ha_router this time neutron router-delete harouter [1] https://github.com/openstack/neutron/blob/mitaka- eol/neutron/db/l3_hamode_db.py#L708 [Regression Potential] The fixed patch [1] for neutron-server will no longer return ha_router which is missing ha_ports, so L488 will no longer have chance to call _safe_router_removed() for a ha_router, so the problem has been fundamentally fixed by this patch and no regression potential. Besides, this fixed patch has been in mitaka-eol branch now, and neutron-server mitaka package is based on neutron-8.4.0, so we need to backport it to xenial and mitaka. $ git tag --contains 8c77ee6b20dd38cc0246e854711cb91cffe3a069 mitaka-eol [1] https://review.openstack.org/#/c/440799/2/neutron/db/l3_hamode_db.py [2] https://github.com/openstack/neutron/blob/mitaka-eol/neutron/agent/l3/agent.py#L488 To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1668410/+subscriptions From 1713783 at bugs.launchpad.net Tue Sep 19 16:03:17 2017 From: 1713783 at bugs.launchpad.net (OpenStack Infra) Date: Tue, 19 Sep 2017 16:03:17 -0000 Subject: [Openstack-security] [Bug 1713783] Re: After failed evacuation the recovered source compute tries to delete the instance References: <150402703560.19393.11649471063519714290.malonedeb@chaenomeles.canonical.com> Message-ID: <150583699750.29603.7105323353993188930.malone@chaenomeles.canonical.com> Reviewed: https://review.openstack.org/504979 Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=a3f286f43d866cd343d26d9bafadecab1c225e4b Submitter: Jenkins Branch: stable/pike commit a3f286f43d866cd343d26d9bafadecab1c225e4b Author: Előd Illés Date: Wed Aug 30 16:54:36 2017 +0200 Set error state after failed evacuation When evacuation fails with NoValidHost, the migration status remains 'accepted' instead of 'error'. This causes problem in case the compute service starts up again and looks for evacuations with status 'accepted', as it then removes the local instances for those evacuations even though the instance was never actually evacuated to another host. Change-Id: I06d78c744fa75ae5f34c5cfa76bc3c9460767b84 Closes-Bug: #1713783 (cherry picked from commit a8ebf5f1aac080854704e27146e8c98b053c6224) ** Changed in: nova/pike Status: In Progress => Fix Committed -- You received this bug notification because you are a member of OpenStack Security, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1713783 Title: After failed evacuation the recovered source compute tries to delete the instance Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) newton series: Triaged Status in OpenStack Compute (nova) ocata series: Triaged Status in OpenStack Compute (nova) pike series: Fix Committed Status in OpenStack Security Advisory: Won't Fix Bug description: Description =========== In case of a failed evacuation attempt the status of the migration is 'accepted' instead of 'failed' so when source compute is recovered the compute manager tries to delete the instance from the source host. However a secondary fault prevents deleting the allocation in placement so the actual deletion of the instance fails as well. Steps to reproduce ================== The following functional test reproduces the bug: https://review.openstack.org/#/c/498482/ What it does: initiate evacuation when no valid host is available and evacuation fails, but nova manager still tries to delete the instance. Logs:     2017-08-29 19:11:15,751 ERROR [oslo_messaging.rpc.server] Exception during message handling     NoValidHost: No valid host was found. There are not enough hosts available.     2017-08-29 19:11:16,103 INFO [nova.tests.functional.test_servers] Running periodic for compute1 (host1)     2017-08-29 19:11:16,115 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,120 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,131 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/allocations" status: 200 len: 152 microversion: 1.0     2017-08-29 19:11:16,138 INFO [nova.compute.resource_tracker] Final resource view: name=host1 phys_ram=8192MB used_ram=1024MB phys_disk=1028GB used_disk=1GB total_vcpus=10 used_vcpus=1 pci_stats=[]     2017-08-29 19:11:16,146 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,151 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,152 INFO [nova.tests.functional.test_servers] Running periodic for compute2 (host2)     2017-08-29 19:11:16,163 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,168 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,176 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/allocations" status: 200 len: 54 microversion: 1.0     2017-08-29 19:11:16,184 INFO [nova.compute.resource_tracker] Final resource view: name=host2 phys_ram=8192MB used_ram=512MB phys_disk=1028GB used_disk=0GB total_vcpus=10 used_vcpus=0 pci_stats=[]     2017-08-29 19:11:16,192 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,197 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,198 INFO [nova.tests.functional.test_servers] Finished with periodics     2017-08-29 19:11:16,255 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/6f70656e737461636b20342065766572/servers/5058200c-478e-4449-88c1-906fdd572662" status: 200 len: 1875 microversion: 2.53 time: 0.056198     2017-08-29 19:11:16,262 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/6f70656e737461636b20342065766572/os-migrations" status: 200 len: 373 microversion: 2.53 time: 0.004618     2017-08-29 19:11:16,280 INFO [nova.api.openstack.requestlog] 127.0.0.1 "PUT /v2.1/6f70656e737461636b20342065766572/os-services/c269bc74-4720-4de4-a6e5-889080b892a0" status: 200 len: 245 microversion: 2.53 time: 0.016442     2017-08-29 19:11:16,281 INFO [nova.service] Starting compute node (version 16.0.0)     2017-08-29 19:11:16,296 INFO [nova.compute.manager] Deleting instance as it has been evacuated from this host To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1713783/+subscriptions From 1713783 at bugs.launchpad.net Wed Sep 20 14:55:28 2017 From: 1713783 at bugs.launchpad.net (OpenStack Infra) Date: Wed, 20 Sep 2017 14:55:28 -0000 Subject: [Openstack-security] [Bug 1713783] Related fix merged to nova (master) References: <150402703560.19393.11649471063519714290.malonedeb@chaenomeles.canonical.com> Message-ID: <150591932874.29965.12853805007817460308.malone@wampee.canonical.com> Reviewed: https://review.openstack.org/500144 Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=aec81e7d98c1ca082abd157dab5cc83b5bedfd89 Submitter: Jenkins Branch: master commit aec81e7d98c1ca082abd157dab5cc83b5bedfd89 Author: Matt Riedemann Date: Fri Sep 1 13:56:38 2017 -0400 Update docs for _destroy_evacuated_instances The docstring on this method no longer reflects reality, and hasn't since we started filtering on migration records starting back in Liberty. This change updates the docstring for what the method actually does now, and also includes a comment about why we include 'accepted' but not yet complete migrations when looking for things to remove from this source node. Change-Id: Ia0c67aeffe41e133de9dee5080be9c5ec1739aa6 Related-Bug: #1713783 -- You received this bug notification because you are a member of OpenStack Security, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1713783 Title: After failed evacuation the recovered source compute tries to delete the instance Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) newton series: Triaged Status in OpenStack Compute (nova) ocata series: Triaged Status in OpenStack Compute (nova) pike series: Fix Committed Status in OpenStack Security Advisory: Won't Fix Bug description: Description =========== In case of a failed evacuation attempt the status of the migration is 'accepted' instead of 'failed' so when source compute is recovered the compute manager tries to delete the instance from the source host. However a secondary fault prevents deleting the allocation in placement so the actual deletion of the instance fails as well. Steps to reproduce ================== The following functional test reproduces the bug: https://review.openstack.org/#/c/498482/ What it does: initiate evacuation when no valid host is available and evacuation fails, but nova manager still tries to delete the instance. Logs:     2017-08-29 19:11:15,751 ERROR [oslo_messaging.rpc.server] Exception during message handling     NoValidHost: No valid host was found. There are not enough hosts available.     2017-08-29 19:11:16,103 INFO [nova.tests.functional.test_servers] Running periodic for compute1 (host1)     2017-08-29 19:11:16,115 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,120 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,131 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/allocations" status: 200 len: 152 microversion: 1.0     2017-08-29 19:11:16,138 INFO [nova.compute.resource_tracker] Final resource view: name=host1 phys_ram=8192MB used_ram=1024MB phys_disk=1028GB used_disk=1GB total_vcpus=10 used_vcpus=1 pci_stats=[]     2017-08-29 19:11:16,146 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,151 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,152 INFO [nova.tests.functional.test_servers] Running periodic for compute2 (host2)     2017-08-29 19:11:16,163 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,168 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,176 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/allocations" status: 200 len: 54 microversion: 1.0     2017-08-29 19:11:16,184 INFO [nova.compute.resource_tracker] Final resource view: name=host2 phys_ram=8192MB used_ram=512MB phys_disk=1028GB used_disk=0GB total_vcpus=10 used_vcpus=0 pci_stats=[]     2017-08-29 19:11:16,192 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,197 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,198 INFO [nova.tests.functional.test_servers] Finished with periodics     2017-08-29 19:11:16,255 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/6f70656e737461636b20342065766572/servers/5058200c-478e-4449-88c1-906fdd572662" status: 200 len: 1875 microversion: 2.53 time: 0.056198     2017-08-29 19:11:16,262 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/6f70656e737461636b20342065766572/os-migrations" status: 200 len: 373 microversion: 2.53 time: 0.004618     2017-08-29 19:11:16,280 INFO [nova.api.openstack.requestlog] 127.0.0.1 "PUT /v2.1/6f70656e737461636b20342065766572/os-services/c269bc74-4720-4de4-a6e5-889080b892a0" status: 200 len: 245 microversion: 2.53 time: 0.016442     2017-08-29 19:11:16,281 INFO [nova.service] Starting compute node (version 16.0.0)     2017-08-29 19:11:16,296 INFO [nova.compute.manager] Deleting instance as it has been evacuated from this host To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1713783/+subscriptions From gerrit2 at review.openstack.org Mon Sep 25 15:34:07 2017 From: gerrit2 at review.openstack.org (gerrit2 at review.openstack.org) Date: Mon, 25 Sep 2017 15:34:07 +0000 Subject: [Openstack-security] [openstack/cursive] SecurityImpact review request change openstack%2Fcursive~master~I8d7f43fb4c0573ac3681147eac213b369bbbcb3b Message-ID: Hi, I'd like you to take a look at this patch for potential SecurityImpact. https://review.openstack.org/357202 Log: commit 9f8eb5eea90837f121323383e569d5def11682d7 Author: Peter Hamilton Date: Thu Aug 18 08:50:38 2016 -0400 Add certificate validation This change adds support for certificate validation, including certificate inspection utilities. Validating a certificate requires the certificate UUID of the certificate to validate, a set of UUIDs corresponding to the set of trusted certificates needed to validate the certificate, and a user context for authentication to the key manager. A new certificate verification context is included that is used to store the set of trusted certificates once they are loaded from the key manager. This context is used to validate the signing certificate, verifying that the certificate belongs to a valid certificate chain rooted in the set of trusted certificates. All new certificate utility code is added in a new module named certificate_utils. For more information on this work, see the spec: https://review.openstack.org/#/c/357151/ SecurityImpact DocImpact Change-Id: I8d7f43fb4c0573ac3681147eac213b369bbbcb3b Implements: blueprint nova-validate-certificates From 1713783 at bugs.launchpad.net Mon Sep 25 14:27:43 2017 From: 1713783 at bugs.launchpad.net (OpenStack Infra) Date: Mon, 25 Sep 2017 14:27:43 -0000 Subject: [Openstack-security] [Bug 1713783] Fix included in openstack/nova 16.0.1 References: <150402703560.19393.11649471063519714290.malonedeb@chaenomeles.canonical.com> Message-ID: <150634966386.30835.6639228949462120902.malone@wampee.canonical.com> This issue was fixed in the openstack/nova 16.0.1 release. -- You received this bug notification because you are a member of OpenStack Security, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1713783 Title: After failed evacuation the recovered source compute tries to delete the instance Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) newton series: Triaged Status in OpenStack Compute (nova) ocata series: Triaged Status in OpenStack Compute (nova) pike series: Fix Committed Status in OpenStack Security Advisory: Won't Fix Bug description: Description =========== In case of a failed evacuation attempt the status of the migration is 'accepted' instead of 'failed' so when source compute is recovered the compute manager tries to delete the instance from the source host. However a secondary fault prevents deleting the allocation in placement so the actual deletion of the instance fails as well. Steps to reproduce ================== The following functional test reproduces the bug: https://review.openstack.org/#/c/498482/ What it does: initiate evacuation when no valid host is available and evacuation fails, but nova manager still tries to delete the instance. Logs:     2017-08-29 19:11:15,751 ERROR [oslo_messaging.rpc.server] Exception during message handling     NoValidHost: No valid host was found. There are not enough hosts available.     2017-08-29 19:11:16,103 INFO [nova.tests.functional.test_servers] Running periodic for compute1 (host1)     2017-08-29 19:11:16,115 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,120 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,131 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/allocations" status: 200 len: 152 microversion: 1.0     2017-08-29 19:11:16,138 INFO [nova.compute.resource_tracker] Final resource view: name=host1 phys_ram=8192MB used_ram=1024MB phys_disk=1028GB used_disk=1GB total_vcpus=10 used_vcpus=1 pci_stats=[]     2017-08-29 19:11:16,146 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,151 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,152 INFO [nova.tests.functional.test_servers] Running periodic for compute2 (host2)     2017-08-29 19:11:16,163 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,168 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,176 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/allocations" status: 200 len: 54 microversion: 1.0     2017-08-29 19:11:16,184 INFO [nova.compute.resource_tracker] Final resource view: name=host2 phys_ram=8192MB used_ram=512MB phys_disk=1028GB used_disk=0GB total_vcpus=10 used_vcpus=0 pci_stats=[]     2017-08-29 19:11:16,192 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,197 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,198 INFO [nova.tests.functional.test_servers] Finished with periodics     2017-08-29 19:11:16,255 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/6f70656e737461636b20342065766572/servers/5058200c-478e-4449-88c1-906fdd572662" status: 200 len: 1875 microversion: 2.53 time: 0.056198     2017-08-29 19:11:16,262 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/6f70656e737461636b20342065766572/os-migrations" status: 200 len: 373 microversion: 2.53 time: 0.004618     2017-08-29 19:11:16,280 INFO [nova.api.openstack.requestlog] 127.0.0.1 "PUT /v2.1/6f70656e737461636b20342065766572/os-services/c269bc74-4720-4de4-a6e5-889080b892a0" status: 200 len: 245 microversion: 2.53 time: 0.016442     2017-08-29 19:11:16,281 INFO [nova.service] Starting compute node (version 16.0.0)     2017-08-29 19:11:16,296 INFO [nova.compute.manager] Deleting instance as it has been evacuated from this host To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1713783/+subscriptions From tdecacqu at redhat.com Mon Sep 25 18:17:04 2017 From: tdecacqu at redhat.com (Tristan Cacqueray) Date: Mon, 25 Sep 2017 18:17:04 -0000 Subject: [Openstack-security] [Bug 1606495] Re: copy_from in api v1 allows network port scan References: <20160726093308.5782.75451.malonedeb@chaenomeles.canonical.com> Message-ID: <150636342624.30499.17895101252010898515.launchpad@wampee.canonical.com> ** Description changed: - This issue is being treated as a potential security risk under embargo. - Please do not make any public mention of embargoed (private) security - vulnerabilities before their coordinated publication by the OpenStack - Vulnerability Management Team in the form of an official OpenStack - Security Advisory. This includes discussion of the bug or associated - fixes in public forums such as mailing lists, code review systems and - bug trackers. Please also avoid private disclosure to other individuals - not already approved for access to this information, and provide this - same reminder to those who are made aware of the issue prior to - publication. All discussion should remain confined to this private bug - report, and any proposed fixes should be added to the bug as - attachments. - - copy_from allows to create Images with an url like http://localhost:22 The remote content gets copied unverified in the defined glance store. E.g. after downloading the image with copy_from url http://localhost:22, you see the OpenSSH banner. This is a security issue, as it allows users to do network "scans" for open ports and it copies remote (potentially malicious) content unverified to your configured glance store. glance api v1 is still the default in horizon. -- You received this bug notification because you are a member of OpenStack Security, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1606495 Title: copy_from in api v1 allows network port scan Status in Glance: New Status in OpenStack Security Advisory: Opinion Status in OpenStack Security Notes: Fix Released Bug description: copy_from allows to create Images with an url like http://localhost:22 The remote content gets copied unverified in the defined glance store. E.g. after downloading the image with copy_from url http://localhost:22, you see the OpenSSH banner. This is a security issue, as it allows users to do network "scans" for open ports and it copies remote (potentially malicious) content unverified to your configured glance store. glance api v1 is still the default in horizon. To manage notifications about this bug go to: https://bugs.launchpad.net/glance/+bug/1606495/+subscriptions From 1605914 at bugs.launchpad.net Thu Sep 28 02:56:46 2017 From: 1605914 at bugs.launchpad.net (Amrith Kumar) Date: Thu, 28 Sep 2017 02:56:46 -0000 Subject: [Openstack-security] [Bug 1605914] Re: Hard coded security group References: <20160723182211.8674.89923.malonedeb@soybean.canonical.com> Message-ID: <150656740836.30499.9869850315071800166.launchpad@wampee.canonical.com> ** Changed in: trove Assignee: Amrith Kumar (amrith) => (unassigned) -- You received this bug notification because you are a member of OpenStack Security, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1605914 Title: Hard coded security group Status in OpenStack DBaaS (Trove): Confirmed Bug description: "trove create" also creates a security group, with hard coded values. Some of which is not what I want/need - especially the 'allow everything' rule! Please allow to specify existing SG(s) (preferably plural) on either the command line (preferably), or in the datastore/datastore_version configuration. Or possibly in the datastore section in trove*.conf. To manage notifications about this bug go to: https://bugs.launchpad.net/trove/+bug/1605914/+subscriptions From 1434545 at bugs.launchpad.net Thu Sep 28 02:56:27 2017 From: 1434545 at bugs.launchpad.net (Amrith Kumar) Date: Thu, 28 Sep 2017 02:56:27 -0000 Subject: [Openstack-security] [Bug 1434545] Re: Several command injection vulnerabilities in guestagent/pkg References: <20150320131103.23017.37206.malonedeb@chaenomeles.canonical.com> Message-ID: <150656738997.30110.12766708281554446415.launchpad@wampee.canonical.com> ** Changed in: trove Assignee: Amrith Kumar (amrith) => (unassigned) -- You received this bug notification because you are a member of OpenStack Security, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1434545 Title: Several command injection vulnerabilities in guestagent/pkg Status in OpenStack Security Advisory: Won't Fix Status in OpenStack DBaaS (Trove): Triaged Bug description: At several places in the file guestagent/pkg.py, there are shell injection vulnerabilities: https://github.com/openstack/trove/blob/master/trove/guestagent/pkg.py#L209 In this line, the cmd_list is being built parameterized, but then it is just combined into one big string and called directly on a shell through the command getstatusoutput, which does a popen. If package name is set maliciously, the command will execute arbitrary code with the privilege of the trove process. The same is true on this line, https://github.com/openstack/trove/blob/master/trove/guestagent/pkg.py#L258 , where a package named something like "abc; rm -rf /etc" will cause all files in /etc which Trove has permissions for, to be deleted. Again, on this line: https://github.com/openstack/trove/blob/master/trove/guestagent/pkg.py#L371 , a malicious package name will cause arbitrary code injection with the privileges of the Trove process. I'm not nearly familiar enough with the Trove code and uses to know all the ways that package names for this code can be set, but these commands should be parameterized. Finally, os.popen is a deprecated function. The subprocess module should be used instead. To manage notifications about this bug go to: https://bugs.launchpad.net/ossa/+bug/1434545/+subscriptions From 1343657 at bugs.launchpad.net Thu Sep 28 02:56:23 2017 From: 1343657 at bugs.launchpad.net (Amrith Kumar) Date: Thu, 28 Sep 2017 02:56:23 -0000 Subject: [Openstack-security] [Bug 1343657] Re: Shell Injection in backup strategies References: <20140717230743.8188.12182.malonedeb@soybean.canonical.com> Message-ID: <150656738751.22699.8383350905811366301.launchpad@chaenomeles.canonical.com> ** Changed in: trove Assignee: Amrith Kumar (amrith) => (unassigned) -- You received this bug notification because you are a member of OpenStack Security, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1343657 Title: Shell Injection in backup strategies Status in OpenStack Security Advisory: Won't Fix Status in OpenStack Security Notes: Fix Released Status in OpenStack DBaaS (Trove): Triaged Bug description: Trove uses subprocess.Popen with shell=True in trove/trove/guestagent/strategies/backup/base.py line 61: def run(self): self.process = subprocess.Popen(self.command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, preexec_fn=os.setsid) self.pid = self.process.pid This could be used, maliciously or not, to inject arbitrary commands into a command line string. An example of this could be triggered is in trove/trove/guestagent/strategies/backup/mysql_imply.py line 37. It is creating a MySQL string with single quote. If the password, either maliciously or just happens to contain another single quote, it will escape from the command and arbitrary data will be executed instead. For more information on subprocess, shell=True and command injection see: https://docs.python.org/2/library/subprocess.html#frequently- used-arguments To manage notifications about this bug go to: https://bugs.launchpad.net/ossa/+bug/1343657/+subscriptions