From 1777460 at bugs.launchpad.net Tue Oct 2 16:06:33 2018 From: 1777460 at bugs.launchpad.net (OpenStack Infra) Date: Tue, 02 Oct 2018 16:06:33 -0000 Subject: [Openstack-security] [Bug 1777460] Re: Whitelist two more SSBD-related CPU flags for AMD ('amd-ssbd', 'amd-no-ssb') References: <152933134329.1989.7672538528184873175.malonedeb@chaenomeles.canonical.com> Message-ID: <153849639399.30636.15812051707162276321.malone@chaenomeles.canonical.com> Fix proposed to branch: stable/ocata Review: https://review.openstack.org/607296 ** Changed in: nova/ocata Status: Confirmed => In Progress ** Changed in: nova/ocata Assignee: (unassigned) => Elod Illes (elod-illes) -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1777460 Title: Whitelist two more SSBD-related CPU flags for AMD ('amd-ssbd', 'amd- no-ssb') Status in OpenStack Compute (nova): Won't Fix Status in OpenStack Compute (nova) ocata series: In Progress Status in OpenStack Compute (nova) pike series: Fix Committed Status in OpenStack Compute (nova) queens series: Fix Committed Bug description: In addition to the existing 'virt-ssbd', future AMD CPUs will have another (architectural) way to deal with SSBD (Speculative Store Bypass Disable), via the CPU flag: 'amd-ssbd'. Furthermore, future AMD CPUs also will expose a mechanism to tell the guest that the "Speculative Store Bypass Disable" (SSBD) is not needed and that the CPU is all good. This is via the CPU flag: 'amd-no-ssb' In summary, two new flags are[1][2]:     amd-ssbd     amd-no-ssb It is recommended to add the above two flags to the whitelist of Nova's `cpu_model_extra_flags` config attribute -- for stable branches (Queens, Pike and Ocata). For Rocky and above release, no such white-listing is required, since we allow free-form CPU flags[3].     * * * Additional notes (from the QEMU mailing list thread[4]) related to performance and live migration:   - tl;dr: On an AMD Compute node, a guest should be presented with     'amd-ssbd', if available, in preference to 'virt-ssbd'.     Details: Tom Lendacky from AMD writes[4] -- "The idea behind     'virt-ssbd' was to provide an architectural method for a guest to do     SSBD when 'amd-ssbd' isn't present. The 'amd-ssbd' feature will use     SPEC_CTRL which is intended to not be intercepted and will be fast.     The use of 'virt-ssbd' will always be intercepted and therefore will     not be as fast. So a guest should be presented with 'amd-ssbd', if     available, in preference to 'virt-ssbd'."   - It is safe to use 'amd-ssbd' (it is an architectural method for guest to do SSBD) in a guest which can be live migrated between different generations/families of AMD CPU. [1] libvirt patch:     https://www.redhat.com/archives/libvir-list/2018-June/msg01111.html [2] QEMU patch:     https://lists.nongnu.org/archive/html/qemu-devel/2018-06/msg00222.html [3] http://git.openstack.org/cgit/openstack/nova/commit/?id=cc27a20 --     libvirt: Lift the restriction of choices for `cpu_model_extra_flags` [4] https://lists.nongnu.org/archive/html/qemu-devel/2018-06/msg02301.html To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1777460/+subscriptions From 1777460 at bugs.launchpad.net Wed Oct 3 15:15:13 2018 From: 1777460 at bugs.launchpad.net (OpenStack Infra) Date: Wed, 03 Oct 2018 15:15:13 -0000 Subject: [Openstack-security] [Bug 1777460] Re: Whitelist two more SSBD-related CPU flags for AMD ('amd-ssbd', 'amd-no-ssb') References: <152933134329.1989.7672538528184873175.malonedeb@chaenomeles.canonical.com> Message-ID: <153857971375.7797.8487321625316392411.malone@soybean.canonical.com> Reviewed: https://review.openstack.org/607296 Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c85f5e22e1cb8afd756341517bd7284ffc8e505b Submitter: Zuul Branch: stable/ocata commit c85f5e22e1cb8afd756341517bd7284ffc8e505b Author: Dan Smith Date: Mon Jun 18 14:13:29 2018 -0700 [Stable Only] Add amd-ssbd and amd-no-ssb CPU flags Update the whitelist for the latest new CPU flags for mitigation of recent security issues. Change-Id: I8686a4755777c8c720c40d4111cc469676d2a5fd Closes-Bug: #1777460 (cherry picked from commit f8aca778f704983bc7ebb0a75d42914fee2dac06) (cherry picked from commit 682ee60803c0e6a468e701282a18cee1c118c9df) ** Changed in: nova/ocata Status: In Progress => Fix Committed -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1777460 Title: Whitelist two more SSBD-related CPU flags for AMD ('amd-ssbd', 'amd- no-ssb') Status in OpenStack Compute (nova): Won't Fix Status in OpenStack Compute (nova) ocata series: Fix Committed Status in OpenStack Compute (nova) pike series: Fix Committed Status in OpenStack Compute (nova) queens series: Fix Committed Bug description: In addition to the existing 'virt-ssbd', future AMD CPUs will have another (architectural) way to deal with SSBD (Speculative Store Bypass Disable), via the CPU flag: 'amd-ssbd'. Furthermore, future AMD CPUs also will expose a mechanism to tell the guest that the "Speculative Store Bypass Disable" (SSBD) is not needed and that the CPU is all good. This is via the CPU flag: 'amd-no-ssb' In summary, two new flags are[1][2]:     amd-ssbd     amd-no-ssb It is recommended to add the above two flags to the whitelist of Nova's `cpu_model_extra_flags` config attribute -- for stable branches (Queens, Pike and Ocata). For Rocky and above release, no such white-listing is required, since we allow free-form CPU flags[3].     * * * Additional notes (from the QEMU mailing list thread[4]) related to performance and live migration:   - tl;dr: On an AMD Compute node, a guest should be presented with     'amd-ssbd', if available, in preference to 'virt-ssbd'.     Details: Tom Lendacky from AMD writes[4] -- "The idea behind     'virt-ssbd' was to provide an architectural method for a guest to do     SSBD when 'amd-ssbd' isn't present. The 'amd-ssbd' feature will use     SPEC_CTRL which is intended to not be intercepted and will be fast.     The use of 'virt-ssbd' will always be intercepted and therefore will     not be as fast. So a guest should be presented with 'amd-ssbd', if     available, in preference to 'virt-ssbd'."   - It is safe to use 'amd-ssbd' (it is an architectural method for guest to do SSBD) in a guest which can be live migrated between different generations/families of AMD CPU. [1] libvirt patch:     https://www.redhat.com/archives/libvir-list/2018-June/msg01111.html [2] QEMU patch:     https://lists.nongnu.org/archive/html/qemu-devel/2018-06/msg00222.html [3] http://git.openstack.org/cgit/openstack/nova/commit/?id=cc27a20 --     libvirt: Lift the restriction of choices for `cpu_model_extra_flags` [4] https://lists.nongnu.org/archive/html/qemu-devel/2018-06/msg02301.html To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1777460/+subscriptions From fungi at yuggoth.org Wed Oct 3 18:15:33 2018 From: fungi at yuggoth.org (Jeremy Stanley) Date: Wed, 03 Oct 2018 18:15:33 -0000 Subject: [Openstack-security] [Bug 1795800] Re: Username enumeration via response timing difference References: <153854884921.22301.15072922945681177517.malonedeb@gac.canonical.com> Message-ID: <153859053352.21995.12749235302719125565.malone@gac.canonical.com> Thanks, I've gone ahead and triaged this as a hardening opportunity for now. ** Information type changed from Private Security to Public ** Changed in: ossa Status: Incomplete => Won't Fix ** Tags added: security -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1795800 Title: Username enumeration via response timing difference Status in OpenStack Identity (keystone): New Status in OpenStack Security Advisory: Won't Fix Bug description: This issue is being treated as a potential security risk under embargo. Please do not make any public mention of embargoed (private) security vulnerabilities before their coordinated publication by the OpenStack Vulnerability Management Team in the form of an official OpenStack Security Advisory. This includes discussion of the bug or associated fixes in public forums such as mailing lists, code review systems and bug trackers. Please also avoid private disclosure to other individuals not already approved for access to this information, and provide this same reminder to those who are made aware of the issue prior to publication. All discussion should remain confined to this private bug report, and any proposed fixes should be added to the bug as attachments. The response times for POST /v3/auth/tokens are significantly higher for valid usernames compared to those of invalid ones, making it possible to enumerate users on the system. Examples: # For invalid username + Request POST /v3/auth/tokens HTTP/1.1 Host: hostname:5000 Connection: close Content-Length: 141 Content-Type: application/json {"auth":{"identity":{"methods": ["password"],"password":{"user":{"name": "nonexisting","domain":{"name": "Default"},"password": "devstacker"}}}}} + Response Time: <150ms # For valid username ('admin' in this case) + Request POST /v3/auth/tokens HTTP/1.1 Host: hostname:5000 Connection: close Content-Length: 139 Content-Type: application/json {"auth":{"identity":{"methods": ["password"],"password":{"user":{"name": "admin","domain":{"name": "Default"},"password": "devstacker"}}}}} + Response time: >600ms # Tested version v3.8 [UPDATE 3 Oct 2018 5:01 AEST] Looks like it's also possible to enumerate for valid "domain" too. There're 2 ways that I can see: * With valid username: use the above user enum bug to guess the valid username, then brute the "domain" parameter. Response times are significantly higher for valid compared to invalid domains. * Without valid username: get a baseline response time using invalid username AND invalid domain name. Bruteforce the "domain" param until the response time hits an average high. For me invalid domain falls in the 90-100ms range whereas valid ones show 100+ms. This one looks a bit more obscure i.e. timing difference is not as distinguishable, but should still be recognisable with a good sample size. To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1795800/+subscriptions From 1792047 at bugs.launchpad.net Mon Oct 8 19:13:04 2018 From: 1792047 at bugs.launchpad.net (OpenStack Infra) Date: Mon, 08 Oct 2018 19:13:04 -0000 Subject: [Openstack-security] [Bug 1792047] Re: keystone rbacenforcer not populating policy dict with view args References: <153670878287.17124.3225642471309073829.malonedeb@chaenomeles.canonical.com> Message-ID: <153902598438.21959.4290146749720489405.malone@gac.canonical.com> Reviewed: https://review.openstack.org/601882 Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=0c71cdd23bd2a7e4f7ec1a5ecec91f3ed7457d00 Submitter: Zuul Branch: stable/rocky commit 0c71cdd23bd2a7e4f7ec1a5ecec91f3ed7457d00 Author: morgan fainberg Date: Tue Sep 11 16:03:54 2018 -0700 Ensure view args is in policy dict The policy_dict (in enforcement) was not populating the view args in a similar manner to the old style @protected decorator. This change ensures that we mirror the old behavior (required for proper use of v3cloud policy). Conflicts: keystone/tests/unit/common/test_rbac_enforcer.py Change-Id: Ida9009a95a874be9cc60c3152d4e3225726562eb Partial-Bug: #1776504 Closes-Bug: #1792047 (cherry picked from commit 4975b79e8174587f7639347939cf679460d4896b) ** Changed in: keystone/rocky Status: In Progress => Fix Committed -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1792047 Title: keystone rbacenforcer not populating policy dict with view args Status in OpenStack Identity (keystone): Fix Released Status in OpenStack Identity (keystone) rocky series: Fix Committed Status in OpenStack Identity (keystone) stein series: Fix Released Bug description: The old @protected decorator pushed the view arguments into the policy_dict for enforcement purposes[0]. This was missed in the new RBACEnforcer. [0] https://github.com/openstack/keystone/blob/294ca38554bb229f66a772e7dba35a5b08a36b20/keystone/common/authorization.py#L152 To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1792047/+subscriptions From 1750074 at bugs.launchpad.net Mon Oct 8 20:33:24 2018 From: 1750074 at bugs.launchpad.net (OpenStack Infra) Date: Mon, 08 Oct 2018 20:33:24 -0000 Subject: [Openstack-security] [Bug 1750074] Fix included in openstack/manila 5.0.2 References: <151882033265.13602.7690410348638039764.malonedeb@gac.canonical.com> Message-ID: <153903080467.8207.5725727196550637155.malone@soybean.canonical.com> This issue was fixed in the openstack/manila 5.0.2 release. -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1750074 Title: Cinder logs rabbitmq password on connection log Status in Cinder: Fix Released Status in Manila: Fix Released Status in OpenStack Security Advisory: Won't Fix Bug description: Cinder may log rabbitmq password on connection when DEBUG is on. Example on cinder-scheduler.log file after enabling DEBUG: (Password has been replaced with XXX) 2018-02-05 19:21:52.721 35 DEBUG cinder.service [req-a2dbe0dd- 14c9-4123-a69a-3623e5f0a4d7 - - - - -] transport_url : rabbit://guest:XXX at 10.10.10.1:5672,guest:XXX at 10.10.10.2:5672,guest:XXX at 10.10.10.3:5672 wait /usr/lib/python2.7/site-packages/cinder/service.py:611 In a production environment, this is pretty bad. To manage notifications about this bug go to: https://bugs.launchpad.net/cinder/+bug/1750074/+subscriptions From 1777460 at bugs.launchpad.net Mon Oct 8 21:49:03 2018 From: 1777460 at bugs.launchpad.net (OpenStack Infra) Date: Mon, 08 Oct 2018 21:49:03 -0000 Subject: [Openstack-security] [Bug 1777460] Fix included in openstack/nova 15.1.5 References: <152933134329.1989.7672538528184873175.malonedeb@chaenomeles.canonical.com> Message-ID: <153903534366.8207.1828260039816429493.malone@soybean.canonical.com> This issue was fixed in the openstack/nova 15.1.5 release. -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1777460 Title: Whitelist two more SSBD-related CPU flags for AMD ('amd-ssbd', 'amd- no-ssb') Status in OpenStack Compute (nova): Won't Fix Status in OpenStack Compute (nova) ocata series: Fix Committed Status in OpenStack Compute (nova) pike series: Fix Committed Status in OpenStack Compute (nova) queens series: Fix Committed Bug description: In addition to the existing 'virt-ssbd', future AMD CPUs will have another (architectural) way to deal with SSBD (Speculative Store Bypass Disable), via the CPU flag: 'amd-ssbd'. Furthermore, future AMD CPUs also will expose a mechanism to tell the guest that the "Speculative Store Bypass Disable" (SSBD) is not needed and that the CPU is all good. This is via the CPU flag: 'amd-no-ssb' In summary, two new flags are[1][2]:     amd-ssbd     amd-no-ssb It is recommended to add the above two flags to the whitelist of Nova's `cpu_model_extra_flags` config attribute -- for stable branches (Queens, Pike and Ocata). For Rocky and above release, no such white-listing is required, since we allow free-form CPU flags[3].     * * * Additional notes (from the QEMU mailing list thread[4]) related to performance and live migration:   - tl;dr: On an AMD Compute node, a guest should be presented with     'amd-ssbd', if available, in preference to 'virt-ssbd'.     Details: Tom Lendacky from AMD writes[4] -- "The idea behind     'virt-ssbd' was to provide an architectural method for a guest to do     SSBD when 'amd-ssbd' isn't present. The 'amd-ssbd' feature will use     SPEC_CTRL which is intended to not be intercepted and will be fast.     The use of 'virt-ssbd' will always be intercepted and therefore will     not be as fast. So a guest should be presented with 'amd-ssbd', if     available, in preference to 'virt-ssbd'."   - It is safe to use 'amd-ssbd' (it is an architectural method for guest to do SSBD) in a guest which can be live migrated between different generations/families of AMD CPU. [1] libvirt patch:     https://www.redhat.com/archives/libvir-list/2018-June/msg01111.html [2] QEMU patch:     https://lists.nongnu.org/archive/html/qemu-devel/2018-06/msg00222.html [3] http://git.openstack.org/cgit/openstack/nova/commit/?id=cc27a20 --     libvirt: Lift the restriction of choices for `cpu_model_extra_flags` [4] https://lists.nongnu.org/archive/html/qemu-devel/2018-06/msg02301.html To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1777460/+subscriptions From 1734320 at bugs.launchpad.net Wed Oct 10 20:29:34 2018 From: 1734320 at bugs.launchpad.net (OpenStack Infra) Date: Wed, 10 Oct 2018 20:29:34 -0000 Subject: [Openstack-security] [Bug 1734320] Change abandoned on nova (master) References: <151152217834.14483.1577991310209811902.malonedeb@soybean.canonical.com> Message-ID: <153920337437.9259.8728771331520128087.malone@chaenomeles.canonical.com> Change abandoned by sean mooney (work at seanmooney.info) on branch: master Review: https://review.openstack.org/602432 Reason: this should not be needed -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1734320 Title: Eavesdropping private traffic Status in neutron: Triaged Status in OpenStack Compute (nova): In Progress Status in os-vif: In Progress Status in OpenStack Security Advisory: Won't Fix Bug description: Eavesdropping private traffic ============================= Abstract -------- We've discovered a security issue that allows end users within their own private network to receive from, and send traffic to, other private networks on the same compute node. Description ----------- During live-migration there is a small time window where the ports of instances are untagged. Instances have a port trunked to the integration bridge and receive 802.1Q tagged private traffic from other tenants. If the port is administratively down during live migration, the port will remain in trunk mode indefinitely. Traffic is possible between ports is that are administratively down, even between tenants self-service networks. Conditions ---------- The following conditions are necessary. * Openvswitch Self-service networks * An Openstack administrator or an automated process needs to schedule a Live migration We tested this on newton. Issues ------ This outcome is the result of multiple independent issues. We will list the most important first, and follow with bugs that create a fragile situation. Issue #1 Initially creating a trunk port When the port is initially created, it is in trunk mode. This creates a fail-open situation. See: https://github.com/openstack/os-vif/blob/newton-eol/vif_plug_ovs/linux_net.py#L52 Recommendation: create ports in the port_dead state, don't leave it dangling in trunk-mode. Add a drop-flow initially. Issue #2 Order of creation. The instance is actually migrated before the (networking) configuration is completed. Recommendation: wait with finishing the live migration until the underlying configuration has been applied completely. Issue #3 Not closing the port when it is down. Neutron calls the port_dead function to ensure the port is down. It sets the tag to 4095 and adds a "drop" flow if (and only if) there is already another tag on the port. The port_dead function will keep untagged ports untagged. https://github.com/openstack/neutron/blob/stable/newton/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L995 Recommendation: Make port_dead also shut the port if no tag is found. Log a warning if this happens. Issue #4 Putting the port administratively down actually puts the port on a compute node shared vlan Instances from different projects on different private networks can talk to each other if they put their ports down. The code does install an openflow "drop" rule but it has a lower priority (2) than the allow rules. Recommendation: Increase the port_dead openflow drop rule priority to MAX Timeline --------  2017-09-14 Discovery eavesdropping issue  2017-09-15 Verify workaround.  2017-10-04 Discovery port-down-traffic issue  2017-11-24 Vendor Disclosure to Openstack Steps to reproduce ------------------ 1. Attach an instance to two networks: admin$ openstack server create --nic net-id= --nic net-id = --image --flavor instance_temp 2. Attach a FIP to the instance to be able to log in to this instance 3. Verify: admin$ openstack server show -c name -c addresses fe28a2ee-098f-4425 -9d3c-8e2cd383547d +-----------+-----------------------------------------------------------------------------+ | Field | Value | +-----------+-----------------------------------------------------------------------------+ | addresses | network1=192.168.99.8, ; network2=192.168.80.14 | | name | instance_temp | +-----------+-----------------------------------------------------------------------------+ 4. Ssh to the instance using network1 and run a tcpdump on the other port network2 [root at instance_temp]$ tcpdump -eeenni eth1 5. Get port-id of network2 admin$ nova interface-list fe28a2ee-098f-4425-9d3c-8e2cd383547d +------------+--------------------------------------+--------------------------------------+---------------+-------------------+ | Port State | Port ID | Net ID | IP addresses | MAC Addr | +------------+--------------------------------------+--------------------------------------+---------------+-------------------+ | ACTIVE | a848520b-0814-4030-bb48-49e4b5cf8160 | d69028f7-9558-4f14-8ce6-29cb8f1c19cd | 192.168.80.14 | fa:16:3e:2d:8b:7b | | ACTIVE | fad148ca-cf7a-4839-aac3-a2cd8d1d2260 | d22c22ae-0a42-4e3b-8144-f28534c3439a | 192.168.99.8 | fa:16:3e:60:2c:fa | +------------+--------------------------------------+--------------------------------------+---------------+-------------------+ 6. Force port down on network 2 admin$ neutron port-update a848520b-0814-4030-bb48-49e4b5cf8160 --admin-state-up False 7. Port gets tagged with vlan 4095, the dead vlan tag, which is normal: compute1# grep a848520b-0814-4030-bb48-49e4b5cf8160 /var/log/neutron/neutron-openvswitch-agent.log | tail -1 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-e008feb3-8a35-4c97-adac-b48ff88165b2 - - - - -] VIF port: a848520b-0814-4030-bb48-49e4b5cf8160 admin state up disabled, putting on the dead VLAN 8. Verify the port is tagged with vlan 4095 compute1# ovs-vsctl show | grep -A3 qvoa848520b-08       Port "qvoa848520b-08"           tag: 4095           Interface "qvoa848520b-08" 9. Now live-migrate the instance: admin# nova live-migration fe28a2ee-098f-4425-9d3c-8e2cd383547d 10. Verify the tag is gone on compute2, and take a deep breath compute2# ovs-vsctl show | grep -A3 qvoa848520b-08       Port "qvoa848520b-08"           Interface "qvoa848520b-08"       Port... compute2# echo "Wut!" 11. Now traffic of all other self-service networks present on compute2 can be sniffed from instance_temp [root at instance_temp] tcpdump -eenni eth1 13:14:31.748266 fa:16:3e:6a:17:38 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.103.12.160 tell 10.103.12.152, length 28 13:14:31.804573 fa:16:3e:e8:a2:d2 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.70, length 28 13:14:31.810482 fa:16:3e:95:ca:3a > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.154, length 28 13:14:31.977820 fa:16:3e:6f:f4:9b > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.150, length 28 13:14:31.979590 fa:16:3e:0f:3d:cc > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 9, p 0, ethertype ARP, Request who-has 10.103.9.163 tell 10.103.9.1, length 28 13:14:32.048082 fa:16:3e:65:64:38 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.101, length 28 13:14:32.127400 fa:16:3e:30:cb:b5 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.103.12.160 tell 10.103.12.165, length 28 13:14:32.141982 fa:16:3e:96:cd:b0 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.100, length 28 13:14:32.205327 fa:16:3e:a2:0b:76 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.153, length 28 13:14:32.444142 fa:16:3e:1f:db:ed > 01:00:5e:00:00:12, ethertype 802.1Q (0x8100), length 58: vlan 72, p 0, ethertype IPv4, 192.168.99.212 > 224.0.0.18: VRRPv2, Advertisement, vrid 50, prio 103, authtype none, intvl 1s, length 20 13:14:32.449497 fa:16:3e:1c:24:c0 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.20, length 28 13:14:32.476015 fa:16:3e:f2:3b:97 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.22, length 28 13:14:32.575034 fa:16:3e:44:fe:35 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.103.12.160 tell 10.103.12.163, length 28 13:14:32.676185 fa:16:3e:1e:92:d7 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.103.12.160 tell 10.103.12.150, length 28 13:14:32.711755 fa:16:3e:99:6c:c8 > 01:00:5e:00:00:12, ethertype 802.1Q (0x8100), length 62: vlan 10, p 0, ethertype IPv4, 10.103.12.154 > 224.0.0.18: VRRPv2, Advertisement, vrid 2, prio 49, authtype simple, intvl 1s, length 24 13:14:32.711773 fa:16:3e:f5:23:d5 > 01:00:5e:00:00:12, ethertype 802.1Q (0x8100), length 58: vlan 12, p 0, ethertype IPv4, 10.103.15.154 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 49, authtype simple, intvl 1s, length 20 Workaround ---------- We temporary fixed this issue by forcing the dead vlan tag on port creation on compute nodes: /usr/lib/python2.7/site-packages/vif_plug_ovs/linux_net.py: def _create_ovs_vif_cmd(bridge, dev, iface_id, mac,                         instance_id, interface_type=None,                         vhost_server_path=None): + # ODCN: initialize port as dead + # ODCN: TODO: set drop flow     cmd = ['--', '--if-exists', 'del-port', dev, '--',             'add-port', bridge, dev, + 'tag=4095',             '--', 'set', 'Interface', dev,             'external-ids:iface-id=%s' % iface_id,             'external-ids:iface-status=active',             'external-ids:attached-mac=%s' % mac,             'external-ids:vm-uuid=%s' % instance_id]     if interface_type:         cmd += ['type=%s' % interface_type]     if vhost_server_path:         cmd += ['options:vhost-server-path=%s' % vhost_server_path]     return cmd https://github.com/openstack/neutron/blob/stable/newton/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L995     def port_dead(self, port, log_errors=True):         '''Once a port has no binding, put it on the "dead vlan".         :param port: an ovs_lib.VifPort object.         '''         # Don't kill a port if it's already dead         cur_tag = self.int_br.db_get_val("Port", port.port_name, "tag",                                          log_errors=log_errors) + # ODCN GM 20170915 + if not cur_tag: + LOG.error('port_dead(): port %s has no tag', port.port_name) + # ODCN AJS 20170915 + if not cur_tag or cur_tag != constants.DEAD_VLAN_TAG: - if cur_tag and cur_tag != constants.DEAD_VLAN_TAG:            LOG.info('port_dead(): put port %s on dead vlan', port.port_name)            self.int_br.set_db_attribute("Port", port.port_name, "tag",                                          constants.DEAD_VLAN_TAG,                                          log_errors=log_errors)             self.int_br.drop_port(in_port=port.ofport) plugins/ml2/drivers/openvswitch/agent/openflow/ovs_ofctl/ovs_bridge.py     def drop_port(self, in_port): + # ODCN AJS 20171004: - self.install_drop(priority=2, in_port=in_port) + self.install_drop(priority=65535, in_port=in_port) Regards, ODC Noord. Gerhard Muntingh Albert Siersema Paul Peereboom To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1734320/+subscriptions From 1734320 at bugs.launchpad.net Thu Oct 11 23:09:38 2018 From: 1734320 at bugs.launchpad.net (OpenStack Infra) Date: Thu, 11 Oct 2018 23:09:38 -0000 Subject: [Openstack-security] [Bug 1734320] Fix proposed to os-vif (stable/rocky) References: <151152217834.14483.1577991310209811902.malonedeb@soybean.canonical.com> Message-ID: <153929937894.12962.9946786463206512646.malone@wampee.canonical.com> Fix proposed to branch: stable/rocky Review: https://review.openstack.org/609850 -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1734320 Title: Eavesdropping private traffic Status in neutron: Triaged Status in OpenStack Compute (nova): In Progress Status in os-vif: In Progress Status in OpenStack Security Advisory: Won't Fix Bug description: Eavesdropping private traffic ============================= Abstract -------- We've discovered a security issue that allows end users within their own private network to receive from, and send traffic to, other private networks on the same compute node. Description ----------- During live-migration there is a small time window where the ports of instances are untagged. Instances have a port trunked to the integration bridge and receive 802.1Q tagged private traffic from other tenants. If the port is administratively down during live migration, the port will remain in trunk mode indefinitely. Traffic is possible between ports is that are administratively down, even between tenants self-service networks. Conditions ---------- The following conditions are necessary. * Openvswitch Self-service networks * An Openstack administrator or an automated process needs to schedule a Live migration We tested this on newton. Issues ------ This outcome is the result of multiple independent issues. We will list the most important first, and follow with bugs that create a fragile situation. Issue #1 Initially creating a trunk port When the port is initially created, it is in trunk mode. This creates a fail-open situation. See: https://github.com/openstack/os-vif/blob/newton-eol/vif_plug_ovs/linux_net.py#L52 Recommendation: create ports in the port_dead state, don't leave it dangling in trunk-mode. Add a drop-flow initially. Issue #2 Order of creation. The instance is actually migrated before the (networking) configuration is completed. Recommendation: wait with finishing the live migration until the underlying configuration has been applied completely. Issue #3 Not closing the port when it is down. Neutron calls the port_dead function to ensure the port is down. It sets the tag to 4095 and adds a "drop" flow if (and only if) there is already another tag on the port. The port_dead function will keep untagged ports untagged. https://github.com/openstack/neutron/blob/stable/newton/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L995 Recommendation: Make port_dead also shut the port if no tag is found. Log a warning if this happens. Issue #4 Putting the port administratively down actually puts the port on a compute node shared vlan Instances from different projects on different private networks can talk to each other if they put their ports down. The code does install an openflow "drop" rule but it has a lower priority (2) than the allow rules. Recommendation: Increase the port_dead openflow drop rule priority to MAX Timeline --------  2017-09-14 Discovery eavesdropping issue  2017-09-15 Verify workaround.  2017-10-04 Discovery port-down-traffic issue  2017-11-24 Vendor Disclosure to Openstack Steps to reproduce ------------------ 1. Attach an instance to two networks: admin$ openstack server create --nic net-id= --nic net-id = --image --flavor instance_temp 2. Attach a FIP to the instance to be able to log in to this instance 3. Verify: admin$ openstack server show -c name -c addresses fe28a2ee-098f-4425 -9d3c-8e2cd383547d +-----------+-----------------------------------------------------------------------------+ | Field | Value | +-----------+-----------------------------------------------------------------------------+ | addresses | network1=192.168.99.8, ; network2=192.168.80.14 | | name | instance_temp | +-----------+-----------------------------------------------------------------------------+ 4. Ssh to the instance using network1 and run a tcpdump on the other port network2 [root at instance_temp]$ tcpdump -eeenni eth1 5. Get port-id of network2 admin$ nova interface-list fe28a2ee-098f-4425-9d3c-8e2cd383547d +------------+--------------------------------------+--------------------------------------+---------------+-------------------+ | Port State | Port ID | Net ID | IP addresses | MAC Addr | +------------+--------------------------------------+--------------------------------------+---------------+-------------------+ | ACTIVE | a848520b-0814-4030-bb48-49e4b5cf8160 | d69028f7-9558-4f14-8ce6-29cb8f1c19cd | 192.168.80.14 | fa:16:3e:2d:8b:7b | | ACTIVE | fad148ca-cf7a-4839-aac3-a2cd8d1d2260 | d22c22ae-0a42-4e3b-8144-f28534c3439a | 192.168.99.8 | fa:16:3e:60:2c:fa | +------------+--------------------------------------+--------------------------------------+---------------+-------------------+ 6. Force port down on network 2 admin$ neutron port-update a848520b-0814-4030-bb48-49e4b5cf8160 --admin-state-up False 7. Port gets tagged with vlan 4095, the dead vlan tag, which is normal: compute1# grep a848520b-0814-4030-bb48-49e4b5cf8160 /var/log/neutron/neutron-openvswitch-agent.log | tail -1 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-e008feb3-8a35-4c97-adac-b48ff88165b2 - - - - -] VIF port: a848520b-0814-4030-bb48-49e4b5cf8160 admin state up disabled, putting on the dead VLAN 8. Verify the port is tagged with vlan 4095 compute1# ovs-vsctl show | grep -A3 qvoa848520b-08       Port "qvoa848520b-08"           tag: 4095           Interface "qvoa848520b-08" 9. Now live-migrate the instance: admin# nova live-migration fe28a2ee-098f-4425-9d3c-8e2cd383547d 10. Verify the tag is gone on compute2, and take a deep breath compute2# ovs-vsctl show | grep -A3 qvoa848520b-08       Port "qvoa848520b-08"           Interface "qvoa848520b-08"       Port... compute2# echo "Wut!" 11. Now traffic of all other self-service networks present on compute2 can be sniffed from instance_temp [root at instance_temp] tcpdump -eenni eth1 13:14:31.748266 fa:16:3e:6a:17:38 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.103.12.160 tell 10.103.12.152, length 28 13:14:31.804573 fa:16:3e:e8:a2:d2 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.70, length 28 13:14:31.810482 fa:16:3e:95:ca:3a > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.154, length 28 13:14:31.977820 fa:16:3e:6f:f4:9b > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.150, length 28 13:14:31.979590 fa:16:3e:0f:3d:cc > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 9, p 0, ethertype ARP, Request who-has 10.103.9.163 tell 10.103.9.1, length 28 13:14:32.048082 fa:16:3e:65:64:38 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.101, length 28 13:14:32.127400 fa:16:3e:30:cb:b5 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.103.12.160 tell 10.103.12.165, length 28 13:14:32.141982 fa:16:3e:96:cd:b0 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.100, length 28 13:14:32.205327 fa:16:3e:a2:0b:76 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.153, length 28 13:14:32.444142 fa:16:3e:1f:db:ed > 01:00:5e:00:00:12, ethertype 802.1Q (0x8100), length 58: vlan 72, p 0, ethertype IPv4, 192.168.99.212 > 224.0.0.18: VRRPv2, Advertisement, vrid 50, prio 103, authtype none, intvl 1s, length 20 13:14:32.449497 fa:16:3e:1c:24:c0 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.20, length 28 13:14:32.476015 fa:16:3e:f2:3b:97 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.22, length 28 13:14:32.575034 fa:16:3e:44:fe:35 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.103.12.160 tell 10.103.12.163, length 28 13:14:32.676185 fa:16:3e:1e:92:d7 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.103.12.160 tell 10.103.12.150, length 28 13:14:32.711755 fa:16:3e:99:6c:c8 > 01:00:5e:00:00:12, ethertype 802.1Q (0x8100), length 62: vlan 10, p 0, ethertype IPv4, 10.103.12.154 > 224.0.0.18: VRRPv2, Advertisement, vrid 2, prio 49, authtype simple, intvl 1s, length 24 13:14:32.711773 fa:16:3e:f5:23:d5 > 01:00:5e:00:00:12, ethertype 802.1Q (0x8100), length 58: vlan 12, p 0, ethertype IPv4, 10.103.15.154 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 49, authtype simple, intvl 1s, length 20 Workaround ---------- We temporary fixed this issue by forcing the dead vlan tag on port creation on compute nodes: /usr/lib/python2.7/site-packages/vif_plug_ovs/linux_net.py: def _create_ovs_vif_cmd(bridge, dev, iface_id, mac,                         instance_id, interface_type=None,                         vhost_server_path=None): + # ODCN: initialize port as dead + # ODCN: TODO: set drop flow     cmd = ['--', '--if-exists', 'del-port', dev, '--',             'add-port', bridge, dev, + 'tag=4095',             '--', 'set', 'Interface', dev,             'external-ids:iface-id=%s' % iface_id,             'external-ids:iface-status=active',             'external-ids:attached-mac=%s' % mac,             'external-ids:vm-uuid=%s' % instance_id]     if interface_type:         cmd += ['type=%s' % interface_type]     if vhost_server_path:         cmd += ['options:vhost-server-path=%s' % vhost_server_path]     return cmd https://github.com/openstack/neutron/blob/stable/newton/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L995     def port_dead(self, port, log_errors=True):         '''Once a port has no binding, put it on the "dead vlan".         :param port: an ovs_lib.VifPort object.         '''         # Don't kill a port if it's already dead         cur_tag = self.int_br.db_get_val("Port", port.port_name, "tag",                                          log_errors=log_errors) + # ODCN GM 20170915 + if not cur_tag: + LOG.error('port_dead(): port %s has no tag', port.port_name) + # ODCN AJS 20170915 + if not cur_tag or cur_tag != constants.DEAD_VLAN_TAG: - if cur_tag and cur_tag != constants.DEAD_VLAN_TAG:            LOG.info('port_dead(): put port %s on dead vlan', port.port_name)            self.int_br.set_db_attribute("Port", port.port_name, "tag",                                          constants.DEAD_VLAN_TAG,                                          log_errors=log_errors)             self.int_br.drop_port(in_port=port.ofport) plugins/ml2/drivers/openvswitch/agent/openflow/ovs_ofctl/ovs_bridge.py     def drop_port(self, in_port): + # ODCN AJS 20171004: - self.install_drop(priority=2, in_port=in_port) + self.install_drop(priority=65535, in_port=in_port) Regards, ODC Noord. Gerhard Muntingh Albert Siersema Paul Peereboom To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1734320/+subscriptions From 1734320 at bugs.launchpad.net Thu Oct 11 23:12:18 2018 From: 1734320 at bugs.launchpad.net (OpenStack Infra) Date: Thu, 11 Oct 2018 23:12:18 -0000 Subject: [Openstack-security] [Bug 1734320] Fix proposed to os-vif (stable/queens) References: <151152217834.14483.1577991310209811902.malonedeb@soybean.canonical.com> Message-ID: <153929953898.12541.3412054702152394348.malone@wampee.canonical.com> Fix proposed to branch: stable/queens Review: https://review.openstack.org/609851 -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1734320 Title: Eavesdropping private traffic Status in neutron: Triaged Status in OpenStack Compute (nova): In Progress Status in os-vif: In Progress Status in OpenStack Security Advisory: Won't Fix Bug description: Eavesdropping private traffic ============================= Abstract -------- We've discovered a security issue that allows end users within their own private network to receive from, and send traffic to, other private networks on the same compute node. Description ----------- During live-migration there is a small time window where the ports of instances are untagged. Instances have a port trunked to the integration bridge and receive 802.1Q tagged private traffic from other tenants. If the port is administratively down during live migration, the port will remain in trunk mode indefinitely. Traffic is possible between ports is that are administratively down, even between tenants self-service networks. Conditions ---------- The following conditions are necessary. * Openvswitch Self-service networks * An Openstack administrator or an automated process needs to schedule a Live migration We tested this on newton. Issues ------ This outcome is the result of multiple independent issues. We will list the most important first, and follow with bugs that create a fragile situation. Issue #1 Initially creating a trunk port When the port is initially created, it is in trunk mode. This creates a fail-open situation. See: https://github.com/openstack/os-vif/blob/newton-eol/vif_plug_ovs/linux_net.py#L52 Recommendation: create ports in the port_dead state, don't leave it dangling in trunk-mode. Add a drop-flow initially. Issue #2 Order of creation. The instance is actually migrated before the (networking) configuration is completed. Recommendation: wait with finishing the live migration until the underlying configuration has been applied completely. Issue #3 Not closing the port when it is down. Neutron calls the port_dead function to ensure the port is down. It sets the tag to 4095 and adds a "drop" flow if (and only if) there is already another tag on the port. The port_dead function will keep untagged ports untagged. https://github.com/openstack/neutron/blob/stable/newton/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L995 Recommendation: Make port_dead also shut the port if no tag is found. Log a warning if this happens. Issue #4 Putting the port administratively down actually puts the port on a compute node shared vlan Instances from different projects on different private networks can talk to each other if they put their ports down. The code does install an openflow "drop" rule but it has a lower priority (2) than the allow rules. Recommendation: Increase the port_dead openflow drop rule priority to MAX Timeline --------  2017-09-14 Discovery eavesdropping issue  2017-09-15 Verify workaround.  2017-10-04 Discovery port-down-traffic issue  2017-11-24 Vendor Disclosure to Openstack Steps to reproduce ------------------ 1. Attach an instance to two networks: admin$ openstack server create --nic net-id= --nic net-id = --image --flavor instance_temp 2. Attach a FIP to the instance to be able to log in to this instance 3. Verify: admin$ openstack server show -c name -c addresses fe28a2ee-098f-4425 -9d3c-8e2cd383547d +-----------+-----------------------------------------------------------------------------+ | Field | Value | +-----------+-----------------------------------------------------------------------------+ | addresses | network1=192.168.99.8, ; network2=192.168.80.14 | | name | instance_temp | +-----------+-----------------------------------------------------------------------------+ 4. Ssh to the instance using network1 and run a tcpdump on the other port network2 [root at instance_temp]$ tcpdump -eeenni eth1 5. Get port-id of network2 admin$ nova interface-list fe28a2ee-098f-4425-9d3c-8e2cd383547d +------------+--------------------------------------+--------------------------------------+---------------+-------------------+ | Port State | Port ID | Net ID | IP addresses | MAC Addr | +------------+--------------------------------------+--------------------------------------+---------------+-------------------+ | ACTIVE | a848520b-0814-4030-bb48-49e4b5cf8160 | d69028f7-9558-4f14-8ce6-29cb8f1c19cd | 192.168.80.14 | fa:16:3e:2d:8b:7b | | ACTIVE | fad148ca-cf7a-4839-aac3-a2cd8d1d2260 | d22c22ae-0a42-4e3b-8144-f28534c3439a | 192.168.99.8 | fa:16:3e:60:2c:fa | +------------+--------------------------------------+--------------------------------------+---------------+-------------------+ 6. Force port down on network 2 admin$ neutron port-update a848520b-0814-4030-bb48-49e4b5cf8160 --admin-state-up False 7. Port gets tagged with vlan 4095, the dead vlan tag, which is normal: compute1# grep a848520b-0814-4030-bb48-49e4b5cf8160 /var/log/neutron/neutron-openvswitch-agent.log | tail -1 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-e008feb3-8a35-4c97-adac-b48ff88165b2 - - - - -] VIF port: a848520b-0814-4030-bb48-49e4b5cf8160 admin state up disabled, putting on the dead VLAN 8. Verify the port is tagged with vlan 4095 compute1# ovs-vsctl show | grep -A3 qvoa848520b-08       Port "qvoa848520b-08"           tag: 4095           Interface "qvoa848520b-08" 9. Now live-migrate the instance: admin# nova live-migration fe28a2ee-098f-4425-9d3c-8e2cd383547d 10. Verify the tag is gone on compute2, and take a deep breath compute2# ovs-vsctl show | grep -A3 qvoa848520b-08       Port "qvoa848520b-08"           Interface "qvoa848520b-08"       Port... compute2# echo "Wut!" 11. Now traffic of all other self-service networks present on compute2 can be sniffed from instance_temp [root at instance_temp] tcpdump -eenni eth1 13:14:31.748266 fa:16:3e:6a:17:38 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.103.12.160 tell 10.103.12.152, length 28 13:14:31.804573 fa:16:3e:e8:a2:d2 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.70, length 28 13:14:31.810482 fa:16:3e:95:ca:3a > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.154, length 28 13:14:31.977820 fa:16:3e:6f:f4:9b > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.150, length 28 13:14:31.979590 fa:16:3e:0f:3d:cc > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 9, p 0, ethertype ARP, Request who-has 10.103.9.163 tell 10.103.9.1, length 28 13:14:32.048082 fa:16:3e:65:64:38 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.101, length 28 13:14:32.127400 fa:16:3e:30:cb:b5 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.103.12.160 tell 10.103.12.165, length 28 13:14:32.141982 fa:16:3e:96:cd:b0 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.100, length 28 13:14:32.205327 fa:16:3e:a2:0b:76 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.153, length 28 13:14:32.444142 fa:16:3e:1f:db:ed > 01:00:5e:00:00:12, ethertype 802.1Q (0x8100), length 58: vlan 72, p 0, ethertype IPv4, 192.168.99.212 > 224.0.0.18: VRRPv2, Advertisement, vrid 50, prio 103, authtype none, intvl 1s, length 20 13:14:32.449497 fa:16:3e:1c:24:c0 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.20, length 28 13:14:32.476015 fa:16:3e:f2:3b:97 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.22, length 28 13:14:32.575034 fa:16:3e:44:fe:35 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.103.12.160 tell 10.103.12.163, length 28 13:14:32.676185 fa:16:3e:1e:92:d7 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.103.12.160 tell 10.103.12.150, length 28 13:14:32.711755 fa:16:3e:99:6c:c8 > 01:00:5e:00:00:12, ethertype 802.1Q (0x8100), length 62: vlan 10, p 0, ethertype IPv4, 10.103.12.154 > 224.0.0.18: VRRPv2, Advertisement, vrid 2, prio 49, authtype simple, intvl 1s, length 24 13:14:32.711773 fa:16:3e:f5:23:d5 > 01:00:5e:00:00:12, ethertype 802.1Q (0x8100), length 58: vlan 12, p 0, ethertype IPv4, 10.103.15.154 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 49, authtype simple, intvl 1s, length 20 Workaround ---------- We temporary fixed this issue by forcing the dead vlan tag on port creation on compute nodes: /usr/lib/python2.7/site-packages/vif_plug_ovs/linux_net.py: def _create_ovs_vif_cmd(bridge, dev, iface_id, mac,                         instance_id, interface_type=None,                         vhost_server_path=None): + # ODCN: initialize port as dead + # ODCN: TODO: set drop flow     cmd = ['--', '--if-exists', 'del-port', dev, '--',             'add-port', bridge, dev, + 'tag=4095',             '--', 'set', 'Interface', dev,             'external-ids:iface-id=%s' % iface_id,             'external-ids:iface-status=active',             'external-ids:attached-mac=%s' % mac,             'external-ids:vm-uuid=%s' % instance_id]     if interface_type:         cmd += ['type=%s' % interface_type]     if vhost_server_path:         cmd += ['options:vhost-server-path=%s' % vhost_server_path]     return cmd https://github.com/openstack/neutron/blob/stable/newton/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L995     def port_dead(self, port, log_errors=True):         '''Once a port has no binding, put it on the "dead vlan".         :param port: an ovs_lib.VifPort object.         '''         # Don't kill a port if it's already dead         cur_tag = self.int_br.db_get_val("Port", port.port_name, "tag",                                          log_errors=log_errors) + # ODCN GM 20170915 + if not cur_tag: + LOG.error('port_dead(): port %s has no tag', port.port_name) + # ODCN AJS 20170915 + if not cur_tag or cur_tag != constants.DEAD_VLAN_TAG: - if cur_tag and cur_tag != constants.DEAD_VLAN_TAG:            LOG.info('port_dead(): put port %s on dead vlan', port.port_name)            self.int_br.set_db_attribute("Port", port.port_name, "tag",                                          constants.DEAD_VLAN_TAG,                                          log_errors=log_errors)             self.int_br.drop_port(in_port=port.ofport) plugins/ml2/drivers/openvswitch/agent/openflow/ovs_ofctl/ovs_bridge.py     def drop_port(self, in_port): + # ODCN AJS 20171004: - self.install_drop(priority=2, in_port=in_port) + self.install_drop(priority=65535, in_port=in_port) Regards, ODC Noord. Gerhard Muntingh Albert Siersema Paul Peereboom To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1734320/+subscriptions From 1734320 at bugs.launchpad.net Mon Oct 22 23:22:43 2018 From: 1734320 at bugs.launchpad.net (OpenStack Infra) Date: Mon, 22 Oct 2018 23:22:43 -0000 Subject: [Openstack-security] [Bug 1734320] Fix proposed to os-vif (master) References: <151152217834.14483.1577991310209811902.malonedeb@soybean.canonical.com> Message-ID: <154025056335.16201.8128664177506562496.malone@chaenomeles.canonical.com> Fix proposed to branch: master Review: https://review.openstack.org/612534 -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1734320 Title: Eavesdropping private traffic Status in neutron: Triaged Status in OpenStack Compute (nova): In Progress Status in os-vif: In Progress Status in OpenStack Security Advisory: Won't Fix Bug description: Eavesdropping private traffic ============================= Abstract -------- We've discovered a security issue that allows end users within their own private network to receive from, and send traffic to, other private networks on the same compute node. Description ----------- During live-migration there is a small time window where the ports of instances are untagged. Instances have a port trunked to the integration bridge and receive 802.1Q tagged private traffic from other tenants. If the port is administratively down during live migration, the port will remain in trunk mode indefinitely. Traffic is possible between ports is that are administratively down, even between tenants self-service networks. Conditions ---------- The following conditions are necessary. * Openvswitch Self-service networks * An Openstack administrator or an automated process needs to schedule a Live migration We tested this on newton. Issues ------ This outcome is the result of multiple independent issues. We will list the most important first, and follow with bugs that create a fragile situation. Issue #1 Initially creating a trunk port When the port is initially created, it is in trunk mode. This creates a fail-open situation. See: https://github.com/openstack/os-vif/blob/newton-eol/vif_plug_ovs/linux_net.py#L52 Recommendation: create ports in the port_dead state, don't leave it dangling in trunk-mode. Add a drop-flow initially. Issue #2 Order of creation. The instance is actually migrated before the (networking) configuration is completed. Recommendation: wait with finishing the live migration until the underlying configuration has been applied completely. Issue #3 Not closing the port when it is down. Neutron calls the port_dead function to ensure the port is down. It sets the tag to 4095 and adds a "drop" flow if (and only if) there is already another tag on the port. The port_dead function will keep untagged ports untagged. https://github.com/openstack/neutron/blob/stable/newton/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L995 Recommendation: Make port_dead also shut the port if no tag is found. Log a warning if this happens. Issue #4 Putting the port administratively down actually puts the port on a compute node shared vlan Instances from different projects on different private networks can talk to each other if they put their ports down. The code does install an openflow "drop" rule but it has a lower priority (2) than the allow rules. Recommendation: Increase the port_dead openflow drop rule priority to MAX Timeline --------  2017-09-14 Discovery eavesdropping issue  2017-09-15 Verify workaround.  2017-10-04 Discovery port-down-traffic issue  2017-11-24 Vendor Disclosure to Openstack Steps to reproduce ------------------ 1. Attach an instance to two networks: admin$ openstack server create --nic net-id= --nic net-id = --image --flavor instance_temp 2. Attach a FIP to the instance to be able to log in to this instance 3. Verify: admin$ openstack server show -c name -c addresses fe28a2ee-098f-4425 -9d3c-8e2cd383547d +-----------+-----------------------------------------------------------------------------+ | Field | Value | +-----------+-----------------------------------------------------------------------------+ | addresses | network1=192.168.99.8, ; network2=192.168.80.14 | | name | instance_temp | +-----------+-----------------------------------------------------------------------------+ 4. Ssh to the instance using network1 and run a tcpdump on the other port network2 [root at instance_temp]$ tcpdump -eeenni eth1 5. Get port-id of network2 admin$ nova interface-list fe28a2ee-098f-4425-9d3c-8e2cd383547d +------------+--------------------------------------+--------------------------------------+---------------+-------------------+ | Port State | Port ID | Net ID | IP addresses | MAC Addr | +------------+--------------------------------------+--------------------------------------+---------------+-------------------+ | ACTIVE | a848520b-0814-4030-bb48-49e4b5cf8160 | d69028f7-9558-4f14-8ce6-29cb8f1c19cd | 192.168.80.14 | fa:16:3e:2d:8b:7b | | ACTIVE | fad148ca-cf7a-4839-aac3-a2cd8d1d2260 | d22c22ae-0a42-4e3b-8144-f28534c3439a | 192.168.99.8 | fa:16:3e:60:2c:fa | +------------+--------------------------------------+--------------------------------------+---------------+-------------------+ 6. Force port down on network 2 admin$ neutron port-update a848520b-0814-4030-bb48-49e4b5cf8160 --admin-state-up False 7. Port gets tagged with vlan 4095, the dead vlan tag, which is normal: compute1# grep a848520b-0814-4030-bb48-49e4b5cf8160 /var/log/neutron/neutron-openvswitch-agent.log | tail -1 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-e008feb3-8a35-4c97-adac-b48ff88165b2 - - - - -] VIF port: a848520b-0814-4030-bb48-49e4b5cf8160 admin state up disabled, putting on the dead VLAN 8. Verify the port is tagged with vlan 4095 compute1# ovs-vsctl show | grep -A3 qvoa848520b-08       Port "qvoa848520b-08"           tag: 4095           Interface "qvoa848520b-08" 9. Now live-migrate the instance: admin# nova live-migration fe28a2ee-098f-4425-9d3c-8e2cd383547d 10. Verify the tag is gone on compute2, and take a deep breath compute2# ovs-vsctl show | grep -A3 qvoa848520b-08       Port "qvoa848520b-08"           Interface "qvoa848520b-08"       Port... compute2# echo "Wut!" 11. Now traffic of all other self-service networks present on compute2 can be sniffed from instance_temp [root at instance_temp] tcpdump -eenni eth1 13:14:31.748266 fa:16:3e:6a:17:38 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.103.12.160 tell 10.103.12.152, length 28 13:14:31.804573 fa:16:3e:e8:a2:d2 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.70, length 28 13:14:31.810482 fa:16:3e:95:ca:3a > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.154, length 28 13:14:31.977820 fa:16:3e:6f:f4:9b > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.150, length 28 13:14:31.979590 fa:16:3e:0f:3d:cc > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 9, p 0, ethertype ARP, Request who-has 10.103.9.163 tell 10.103.9.1, length 28 13:14:32.048082 fa:16:3e:65:64:38 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.101, length 28 13:14:32.127400 fa:16:3e:30:cb:b5 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.103.12.160 tell 10.103.12.165, length 28 13:14:32.141982 fa:16:3e:96:cd:b0 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.100, length 28 13:14:32.205327 fa:16:3e:a2:0b:76 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.153, length 28 13:14:32.444142 fa:16:3e:1f:db:ed > 01:00:5e:00:00:12, ethertype 802.1Q (0x8100), length 58: vlan 72, p 0, ethertype IPv4, 192.168.99.212 > 224.0.0.18: VRRPv2, Advertisement, vrid 50, prio 103, authtype none, intvl 1s, length 20 13:14:32.449497 fa:16:3e:1c:24:c0 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.20, length 28 13:14:32.476015 fa:16:3e:f2:3b:97 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.22, length 28 13:14:32.575034 fa:16:3e:44:fe:35 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.103.12.160 tell 10.103.12.163, length 28 13:14:32.676185 fa:16:3e:1e:92:d7 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.103.12.160 tell 10.103.12.150, length 28 13:14:32.711755 fa:16:3e:99:6c:c8 > 01:00:5e:00:00:12, ethertype 802.1Q (0x8100), length 62: vlan 10, p 0, ethertype IPv4, 10.103.12.154 > 224.0.0.18: VRRPv2, Advertisement, vrid 2, prio 49, authtype simple, intvl 1s, length 24 13:14:32.711773 fa:16:3e:f5:23:d5 > 01:00:5e:00:00:12, ethertype 802.1Q (0x8100), length 58: vlan 12, p 0, ethertype IPv4, 10.103.15.154 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 49, authtype simple, intvl 1s, length 20 Workaround ---------- We temporary fixed this issue by forcing the dead vlan tag on port creation on compute nodes: /usr/lib/python2.7/site-packages/vif_plug_ovs/linux_net.py: def _create_ovs_vif_cmd(bridge, dev, iface_id, mac,                         instance_id, interface_type=None,                         vhost_server_path=None): + # ODCN: initialize port as dead + # ODCN: TODO: set drop flow     cmd = ['--', '--if-exists', 'del-port', dev, '--',             'add-port', bridge, dev, + 'tag=4095',             '--', 'set', 'Interface', dev,             'external-ids:iface-id=%s' % iface_id,             'external-ids:iface-status=active',             'external-ids:attached-mac=%s' % mac,             'external-ids:vm-uuid=%s' % instance_id]     if interface_type:         cmd += ['type=%s' % interface_type]     if vhost_server_path:         cmd += ['options:vhost-server-path=%s' % vhost_server_path]     return cmd https://github.com/openstack/neutron/blob/stable/newton/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L995     def port_dead(self, port, log_errors=True):         '''Once a port has no binding, put it on the "dead vlan".         :param port: an ovs_lib.VifPort object.         '''         # Don't kill a port if it's already dead         cur_tag = self.int_br.db_get_val("Port", port.port_name, "tag",                                          log_errors=log_errors) + # ODCN GM 20170915 + if not cur_tag: + LOG.error('port_dead(): port %s has no tag', port.port_name) + # ODCN AJS 20170915 + if not cur_tag or cur_tag != constants.DEAD_VLAN_TAG: - if cur_tag and cur_tag != constants.DEAD_VLAN_TAG:            LOG.info('port_dead(): put port %s on dead vlan', port.port_name)            self.int_br.set_db_attribute("Port", port.port_name, "tag",                                          constants.DEAD_VLAN_TAG,                                          log_errors=log_errors)             self.int_br.drop_port(in_port=port.ofport) plugins/ml2/drivers/openvswitch/agent/openflow/ovs_ofctl/ovs_bridge.py     def drop_port(self, in_port): + # ODCN AJS 20171004: - self.install_drop(priority=2, in_port=in_port) + self.install_drop(priority=65535, in_port=in_port) Regards, ODC Noord. Gerhard Muntingh Albert Siersema Paul Peereboom To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1734320/+subscriptions From morgan.fainberg at gmail.com Wed Oct 24 17:33:36 2018 From: morgan.fainberg at gmail.com (Morgan Fainberg) Date: Wed, 24 Oct 2018 17:33:36 -0000 Subject: [Openstack-security] [Bug 1680911] Re: Revoking an unscoped token does not revoke all tokens scoped from the unscoped token References: <20170407170932.30940.87220.malonedeb@wampee.canonical.com> Message-ID: <154040241642.20671.3783089913017322040.malone@gac.canonical.com> Marking this as wont fix. This really is not something we can address in a meaningful way at this time. It expands through a huge set of issues across all of openstack and is not in line with the direction of the project now. ** Changed in: keystone Status: Triaged => Won't Fix -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1680911 Title: Revoking an unscoped token does not revoke all tokens scoped from the unscoped token Status in OpenStack Identity (keystone): Won't Fix Status in OpenStack Security Advisory: Won't Fix Bug description: If you create an unscoped token (A) and you then use token A to create a project-scoped token (B) you now have token (A) [audit_id] = audit_id_a token (A) [audit_chain_id] = [audit_id_a] token (B) [audit_id] = audit_id_b token (B) [audit_chain_id] = [audit_id_a, audit_id_b] If you Revoke(token A) then token B should also be invalid. However, this is not the case currently as there are two reasons for this. There is a bug that doesn't correctly catch this in revoke_models because it accidently changes a list to a list in a tuple: https://github.com/openstack/keystone/blob/master/keystone/models/revoke_model.py#L200-L201 This needs to have the comma removed from not in (token_values['audit_chain_id'],) to not in (token_values['audit_chain_id']) The second and main reason is because this functionality is never exposed to the user and in the code it is never run here: https://github.com/openstack/keystone/blob/master/keystone/token/provider.py#L255-L277 because revoke_chain=False in the parameter is never set to True in a call anywhere in the code. To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1680911/+subscriptions From morgan.fainberg at gmail.com Wed Oct 24 17:34:09 2018 From: morgan.fainberg at gmail.com (Morgan Fainberg) Date: Wed, 24 Oct 2018 17:34:09 -0000 Subject: [Openstack-security] [Bug 1680911] Re: Revoking an unscoped token does not revoke all tokens scoped from the unscoped token References: <20170407170932.30940.87220.malonedeb@wampee.canonical.com> Message-ID: <154040244916.29308.14168482002244501992.malone@soybean.canonical.com> Especially since we have eliminated most cases of revoke-by-id. Revocation is easier to handle in the cases of revoke-by-user-at-time, etc. -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1680911 Title: Revoking an unscoped token does not revoke all tokens scoped from the unscoped token Status in OpenStack Identity (keystone): Won't Fix Status in OpenStack Security Advisory: Won't Fix Bug description: If you create an unscoped token (A) and you then use token A to create a project-scoped token (B) you now have token (A) [audit_id] = audit_id_a token (A) [audit_chain_id] = [audit_id_a] token (B) [audit_id] = audit_id_b token (B) [audit_chain_id] = [audit_id_a, audit_id_b] If you Revoke(token A) then token B should also be invalid. However, this is not the case currently as there are two reasons for this. There is a bug that doesn't correctly catch this in revoke_models because it accidently changes a list to a list in a tuple: https://github.com/openstack/keystone/blob/master/keystone/models/revoke_model.py#L200-L201 This needs to have the comma removed from not in (token_values['audit_chain_id'],) to not in (token_values['audit_chain_id']) The second and main reason is because this functionality is never exposed to the user and in the code it is never run here: https://github.com/openstack/keystone/blob/master/keystone/token/provider.py#L255-L277 because revoke_chain=False in the parameter is never set to True in a call anywhere in the code. To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1680911/+subscriptions From morgan.fainberg at gmail.com Wed Oct 24 17:55:07 2018 From: morgan.fainberg at gmail.com (Morgan Fainberg) Date: Wed, 24 Oct 2018 17:55:07 -0000 Subject: [Openstack-security] [Bug 1703369] Re: get_identity_providers policy should be singular References: <149969257444.24611.1539875531360774756.malonedeb@soybean.canonical.com> Message-ID: <154040370984.20557.4509852100604195771.launchpad@gac.canonical.com> ** Changed in: keystone/ocata Status: Fix Committed => Fix Released ** Changed in: keystone/newton Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1703369 Title: get_identity_providers policy should be singular Status in OpenStack Dashboard (Horizon): Fix Released Status in OpenStack Identity (keystone): Fix Released Status in OpenStack Identity (keystone) newton series: Fix Released Status in OpenStack Identity (keystone) ocata series: Fix Released Status in OpenStack Security Advisory: Won't Fix Status in OpenStack Security Notes: Fix Released Bug description: identity:get_identity_providers should be identity:get_identity_provider (singular) since a GET is targeted on a single provider and the code is setup to check for identity:get_identity_provider (singular). See https://github.com/openstack/keystone/blob/c7e29560b7bf7a44e44722eea0645bf18ad56af3/keystone/federation/controllers.py#L112 found in master (pike) The ocata default policy.json also has this problem. Unless someone manually overrode policy to specify identity:get_identity_provider (singular), the result would be that the default rule was actually used for that check instead of identity:get_identity_providers. We could go back and fix the default policy.json for past releases, but the default actually has the same value as identity:get_identity_providers, and if nobody has complained it's probably safer to just leave it. It is, after all, just defaults there and anyone can override by specifying the correct value. But we must fix in pike to go along with the shift of policy into code. Policy defaults in code definitely need to match up with what the code actually checks. There should no longer be any reliance on the default rule. To manage notifications about this bug go to: https://bugs.launchpad.net/horizon/+bug/1703369/+subscriptions From morgan.fainberg at gmail.com Wed Oct 24 18:00:36 2018 From: morgan.fainberg at gmail.com (Morgan Fainberg) Date: Wed, 24 Oct 2018 18:00:36 -0000 Subject: [Openstack-security] [Bug 1795800] Re: Username enumeration via response timing difference References: <153854884921.22301.15072922945681177517.malonedeb@gac.canonical.com> Message-ID: <154040403619.28754.48135937534076058.malone@soybean.canonical.com> I don't know how we'll address this. Realistically, I think this is going to have to be marked as invalid/wont fix/opinion. I'm going to mark it as wont fix, we can circle back on it if there is more discussion to be had. ** Changed in: keystone Status: New => Won't Fix -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1795800 Title: Username enumeration via response timing difference Status in OpenStack Identity (keystone): Won't Fix Status in OpenStack Security Advisory: Won't Fix Bug description: This issue is being treated as a potential security risk under embargo. Please do not make any public mention of embargoed (private) security vulnerabilities before their coordinated publication by the OpenStack Vulnerability Management Team in the form of an official OpenStack Security Advisory. This includes discussion of the bug or associated fixes in public forums such as mailing lists, code review systems and bug trackers. Please also avoid private disclosure to other individuals not already approved for access to this information, and provide this same reminder to those who are made aware of the issue prior to publication. All discussion should remain confined to this private bug report, and any proposed fixes should be added to the bug as attachments. The response times for POST /v3/auth/tokens are significantly higher for valid usernames compared to those of invalid ones, making it possible to enumerate users on the system. Examples: # For invalid username + Request POST /v3/auth/tokens HTTP/1.1 Host: hostname:5000 Connection: close Content-Length: 141 Content-Type: application/json {"auth":{"identity":{"methods": ["password"],"password":{"user":{"name": "nonexisting","domain":{"name": "Default"},"password": "devstacker"}}}}} + Response Time: <150ms # For valid username ('admin' in this case) + Request POST /v3/auth/tokens HTTP/1.1 Host: hostname:5000 Connection: close Content-Length: 139 Content-Type: application/json {"auth":{"identity":{"methods": ["password"],"password":{"user":{"name": "admin","domain":{"name": "Default"},"password": "devstacker"}}}}} + Response time: >600ms # Tested version v3.8 [UPDATE 3 Oct 2018 5:01 AEST] Looks like it's also possible to enumerate for valid "domain" too. There're 2 ways that I can see: * With valid username: use the above user enum bug to guess the valid username, then brute the "domain" parameter. Response times are significantly higher for valid compared to invalid domains. * Without valid username: get a baseline response time using invalid username AND invalid domain name. Bruteforce the "domain" param until the response time hits an average high. For me invalid domain falls in the 90-100ms range whereas valid ones show 100+ms. This one looks a bit more obscure i.e. timing difference is not as distinguishable, but should still be recognisable with a good sample size. To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1795800/+subscriptions From morgan.fainberg at gmail.com Wed Oct 24 18:04:24 2018 From: morgan.fainberg at gmail.com (Morgan Fainberg) Date: Wed, 24 Oct 2018 18:04:24 -0000 Subject: [Openstack-security] [Bug 1791973] Re: User cannot list their own trusts References: <153668011462.24906.14477100206448263755.malonedeb@gac.canonical.com> Message-ID: <154040426540.19878.9279793135249368941.launchpad@gac.canonical.com> ** Changed in: keystone Status: New => Triaged ** Changed in: keystone Importance: Undecided => Medium -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1791973 Title: User cannot list their own trusts Status in OpenStack Identity (keystone): Triaged Status in OpenStack Security Advisory: Won't Fix Bug description: Heat and Admin users both commonly create trusts for other users. But any application is capable of doing this, as it requires only a scoped token to create a trust, which users pass around regularly. If I am concerned that some other application (or unauthorized user) has created a trust with me as the trustor, I need to be able to confirm this. If I cannot perform "trust list" and see the set of trusts that have me as a trustor, I am not able to clear out spurious ones. Thus, I would not be aware of any trusts set up in my name. To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1791973/+subscriptions From morgan.fainberg at gmail.com Wed Oct 24 18:25:39 2018 From: morgan.fainberg at gmail.com (Morgan Fainberg) Date: Wed, 24 Oct 2018 18:25:39 -0000 Subject: [Openstack-security] [Bug 1071815] Re: auth_token middleware does not check if an endpoint is in the service catalog References: <20121026164130.13962.23200.malonedeb@gac.canonical.com> Message-ID: <154040553949.24125.16040364830115768864.malone@wampee.canonical.com> This is not part of the scope of keystonemiddleware. We do not deny based up on the endpoint/catalog. ** Changed in: keystonemiddleware Status: Triaged => Won't Fix -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1071815 Title: auth_token middleware does not check if an endpoint is in the service catalog Status in keystonemiddleware: Won't Fix Bug description: We include the catalog in the token, but it is not checked. Thus, a token that is intended for a subset of the endpoints can be used on additional endpoints. This prevents a user from creating a token specific to an endpoint. The comparable mechanism is service tickets in Kerberos. If a rogue service gets a ticket in Kerberos, it cannot reuse that ticket elsewhere. WIth the current token scheme, all tokens on a compromised server are at risk of being abused throughout an openstack deployment. To manage notifications about this bug go to: https://bugs.launchpad.net/keystonemiddleware/+bug/1071815/+subscriptions From morgan.fainberg at gmail.com Wed Oct 24 18:30:50 2018 From: morgan.fainberg at gmail.com (Morgan Fainberg) Date: Wed, 24 Oct 2018 18:30:50 -0000 Subject: [Openstack-security] [Bug 1435530] Re: keystonemiddleware without TRL checking and default cache config can allow access after token revocation References: <20150323201925.7521.48815.malonedeb@chaenomeles.canonical.com> Message-ID: <154040585038.13629.2359299790928175495.malone@chaenomeles.canonical.com> This is relatively by design. See previous comments. ** Changed in: keystonemiddleware Status: Triaged => Won't Fix -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1435530 Title: keystonemiddleware without TRL checking and default cache config can allow access after token revocation Status in keystonemiddleware: Won't Fix Status in OpenStack Security Advisory: Won't Fix Status in OpenStack Security Notes: Fix Released Bug description: *** This can probably be made public right away, but I am erring on the side of caution and letting the VMT make this call *** Yukihiro KAWADA reported an issue with Keystone that indirectly led/confirmed an issue with Keystonemiddleware when the Token Revocation List (TRL) is not utilized and caching is enabled (default). If the TRL is not used (common with UUID tokens, as PKI signing is not setup), a token that is used on an endpoint is valid for 300s (default, may be more/less based on config) even if the token is revoked within keystone (this include disabling of the user). Worse, the default is is to cache the tokens in-process memory, which means that the token may appear to be revoked/invalid in some cases and then become re-valid on subsequent requests unless a shared cache is used. This appears to be insane default(s) that lead to a window of exposure that is not clearly communicated either with defaults or when caching times are adjusted. To manage notifications about this bug go to: https://bugs.launchpad.net/keystonemiddleware/+bug/1435530/+subscriptions From morgan.fainberg at gmail.com Wed Oct 24 19:06:01 2018 From: morgan.fainberg at gmail.com (Morgan Fainberg) Date: Wed, 24 Oct 2018 19:06:01 -0000 Subject: [Openstack-security] [Bug 1534284] Re: keystoneauth auth plugins should not use etree XML parsing References: <20160114194930.29365.34205.malonedeb@chaenomeles.canonical.com> Message-ID: <154040796191.19833.6691218079164963300.malone@gac.canonical.com> Unassigning due to inactivity ** Changed in: keystoneauth Status: In Progress => Triaged ** Changed in: keystoneauth Assignee: Pavlo Shchelokovskyy (pshchelo) => (unassigned) -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1534284 Title: keystoneauth auth plugins should not use etree XML parsing Status in keystoneauth: Triaged Status in OpenStack Security Advisory: Won't Fix Status in python-keystoneclient: Won't Fix Bug description: XML parsing is surprisingly difficult and fraught with danger, for example entity expansion makes it easy to cause a lot of memory to be used and therefore crash your system. keystoneclient is using etree parsing which has these potential issues, although in the case of keystoneclient it's the response from the IdP which I think is generally trusted. This is in python- keystoneclient/keystoneclient/contrib/auth/v3/saml2.py There's a defusedxml parser that has protections against these attacks and should therefore be used instead if possible - https://pypi.python.org/pypi/defusedxml - the docs for this page also include some examples of other possible attacks. This was caught by bandit 0.17.0. I'm going to start this out as private security so we can think about it some more before it goes public, even though it's probably not something that needs an issue since I think the source is generally trusted. If you can't trust your IdP then who can you trust? To manage notifications about this bug go to: https://bugs.launchpad.net/keystoneauth/+bug/1534284/+subscriptions From 1713783 at bugs.launchpad.net Thu Oct 25 19:46:10 2018 From: 1713783 at bugs.launchpad.net (OpenStack Infra) Date: Thu, 25 Oct 2018 19:46:10 -0000 Subject: [Openstack-security] [Bug 1713783] Change abandoned on nova (master) References: <150402703560.19393.11649471063519714290.malonedeb@chaenomeles.canonical.com> Message-ID: <154049677053.24662.3053555415227331691.malone@wampee.canonical.com> Change abandoned by Matt Riedemann (mriedem.os at gmail.com) on branch: master Review: https://review.openstack.org/561707 -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1713783 Title: After failed evacuation the recovered source compute tries to delete the instance Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) ocata series: Fix Committed Status in OpenStack Compute (nova) pike series: Fix Committed Status in OpenStack Security Advisory: Won't Fix Bug description: Description =========== In case of a failed evacuation attempt the status of the migration is 'accepted' instead of 'failed' so when source compute is recovered the compute manager tries to delete the instance from the source host. However a secondary fault prevents deleting the allocation in placement so the actual deletion of the instance fails as well. Steps to reproduce ================== The following functional test reproduces the bug: https://review.openstack.org/#/c/498482/ What it does: initiate evacuation when no valid host is available and evacuation fails, but nova manager still tries to delete the instance. Logs:     2017-08-29 19:11:15,751 ERROR [oslo_messaging.rpc.server] Exception during message handling     NoValidHost: No valid host was found. There are not enough hosts available.     2017-08-29 19:11:16,103 INFO [nova.tests.functional.test_servers] Running periodic for compute1 (host1)     2017-08-29 19:11:16,115 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,120 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,131 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/allocations" status: 200 len: 152 microversion: 1.0     2017-08-29 19:11:16,138 INFO [nova.compute.resource_tracker] Final resource view: name=host1 phys_ram=8192MB used_ram=1024MB phys_disk=1028GB used_disk=1GB total_vcpus=10 used_vcpus=1 pci_stats=[]     2017-08-29 19:11:16,146 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,151 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,152 INFO [nova.tests.functional.test_servers] Running periodic for compute2 (host2)     2017-08-29 19:11:16,163 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,168 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,176 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/allocations" status: 200 len: 54 microversion: 1.0     2017-08-29 19:11:16,184 INFO [nova.compute.resource_tracker] Final resource view: name=host2 phys_ram=8192MB used_ram=512MB phys_disk=1028GB used_disk=0GB total_vcpus=10 used_vcpus=0 pci_stats=[]     2017-08-29 19:11:16,192 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,197 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,198 INFO [nova.tests.functional.test_servers] Finished with periodics     2017-08-29 19:11:16,255 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/6f70656e737461636b20342065766572/servers/5058200c-478e-4449-88c1-906fdd572662" status: 200 len: 1875 microversion: 2.53 time: 0.056198     2017-08-29 19:11:16,262 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/6f70656e737461636b20342065766572/os-migrations" status: 200 len: 373 microversion: 2.53 time: 0.004618     2017-08-29 19:11:16,280 INFO [nova.api.openstack.requestlog] 127.0.0.1 "PUT /v2.1/6f70656e737461636b20342065766572/os-services/c269bc74-4720-4de4-a6e5-889080b892a0" status: 200 len: 245 microversion: 2.53 time: 0.016442     2017-08-29 19:11:16,281 INFO [nova.service] Starting compute node (version 16.0.0)     2017-08-29 19:11:16,296 INFO [nova.compute.manager] Deleting instance as it has been evacuated from this host To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1713783/+subscriptions From morgan.fainberg at gmail.com Fri Oct 26 16:36:50 2018 From: morgan.fainberg at gmail.com (Morgan Fainberg) Date: Fri, 26 Oct 2018 16:36:50 -0000 Subject: [Openstack-security] [Bug 1791973] Re: User cannot list their own trusts References: <153668011462.24906.14477100206448263755.malonedeb@gac.canonical.com> Message-ID: <154057181167.24199.8015763917919623543.launchpad@wampee.canonical.com> ** Tags added: policy -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1791973 Title: User cannot list their own trusts Status in OpenStack Identity (keystone): Triaged Status in OpenStack Security Advisory: Won't Fix Bug description: Heat and Admin users both commonly create trusts for other users. But any application is capable of doing this, as it requires only a scoped token to create a trust, which users pass around regularly. If I am concerned that some other application (or unauthorized user) has created a trust with me as the trustor, I need to be able to confirm this. If I cannot perform "trust list" and see the set of trusts that have me as a trustor, I am not able to clear out spurious ones. Thus, I would not be aware of any trusts set up in my name. To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1791973/+subscriptions From 1791973 at bugs.launchpad.net Fri Oct 26 16:53:46 2018 From: 1791973 at bugs.launchpad.net (Adam Young) Date: Fri, 26 Oct 2018 16:53:46 -0000 Subject: [Openstack-security] [Bug 1791973] Re: User cannot list their own trusts References: <153668011462.24906.14477100206448263755.malonedeb@gac.canonical.com> Message-ID: <154057282677.20190.11924988844678030672.malone@gac.canonical.com> Looking at the policy check: # FIXME(lbragstad): Trusts have the ability to optionally include a # project, but until trusts deal with system scope it's not really # useful. For now, this should be a project only operation. scope_types=['project'], It seems to me that this should be an unscoped only operation for users, where they are listing their own trusts, not scoped to a project. -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1791973 Title: User cannot list their own trusts Status in OpenStack Identity (keystone): Triaged Status in OpenStack Security Advisory: Won't Fix Bug description: Heat and Admin users both commonly create trusts for other users. But any application is capable of doing this, as it requires only a scoped token to create a trust, which users pass around regularly. If I am concerned that some other application (or unauthorized user) has created a trust with me as the trustor, I need to be able to confirm this. If I cannot perform "trust list" and see the set of trusts that have me as a trustor, I am not able to clear out spurious ones. Thus, I would not be aware of any trusts set up in my name. To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1791973/+subscriptions