From 1501808 at bugs.launchpad.net Tue Sep 4 05:52:15 2018 From: 1501808 at bugs.launchpad.net (huanhongda) Date: Tue, 04 Sep 2018 05:52:15 -0000 Subject: [Openstack-security] [Bug 1501808] Re: Enabling soft-deletes opens a DOS on compute hosts References: <20151001152652.22834.78736.malonedeb@gac.canonical.com> Message-ID: <153604033511.31926.11990472065723355709.malone@soybean.canonical.com> Is anyone working on this? Can we change the code not to release quota until the instance is reclaimed? -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1501808 Title: Enabling soft-deletes opens a DOS on compute hosts Status in OpenStack Compute (nova): In Progress Status in OpenStack Security Advisory: Won't Fix Bug description: If the user sets reclaim_instance_interval to anything other than 0, then when a user requests an instance delete, it will instead be soft deleted. Soft delete explicitly releases the user's quota, but does not release the instance's resources until period task _reclaim_queued_deletes runs with a period of reclaim_instance_interval seconds. A malicious authenticated user can repeatedly create and delete instances without limit, which will consume resources on the host without consuming their quota. If done quickly enough, this will exhaust host resources. I'm not entirely sure what to suggest in remediation, as this seems to be a deliberate design. The most obvious fix would be to not release quota until the instance is reaped, but that would be a significant change in behaviour. This is very similar to https://bugs.launchpad.net/bugs/cve/2015-3280 , except that we do it deliberately. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1501808/+subscriptions From 1757300 at bugs.launchpad.net Tue Sep 4 19:54:43 2018 From: 1757300 at bugs.launchpad.net (OpenStack Infra) Date: Tue, 04 Sep 2018 19:54:43 -0000 Subject: [Openstack-security] [Bug 1757300] Fix included in openstack/heat 8.0.7 References: <152159262637.18509.15861333391173783708.malonedeb@chaenomeles.canonical.com> Message-ID: <153609088314.16471.9386489074312886437.malone@chaenomeles.canonical.com> This issue was fixed in the openstack/heat 8.0.7 release. -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1757300 Title: RandomString may have less entropy than expected Status in OpenStack Heat: Fix Released Status in OpenStack Security Advisory: Won't Fix Bug description: When generating a random string, once we have selected from the various required pools, we continue by selecting a pool at random and then selecting a character from that pool at random. This does not take into account the differing sizes of the available pools, nor the fact that the same character could appear in multiple pools. This results in a non-uniform probability distribution of characters. For example, in the following resource: type: OS::Heat::RandomString properties: length: 66 character_classes: - class: lettersdigits character_sequences: - sequence: "*$" one might reasonably expect to find an average of 3 '*' or '$' characters in the output, but in fact there would be an average of 33. Since users mostly make use of this feature to generate default passwords for services they are deploying, this would result in the generated passwords having slightly less entropy than expected. Pathological cases where the entropy is massively reduced (like the one above - where it is only 229.5 bits vs. the expected 391 bits) are possible, although it's probably unlikely that users would encounter them by accident. To manage notifications about this bug go to: https://bugs.launchpad.net/heat/+bug/1757300/+subscriptions From 1757300 at bugs.launchpad.net Tue Sep 4 19:55:26 2018 From: 1757300 at bugs.launchpad.net (OpenStack Infra) Date: Tue, 04 Sep 2018 19:55:26 -0000 Subject: [Openstack-security] [Bug 1757300] Fix included in openstack/heat 9.0.5 References: <152159262637.18509.15861333391173783708.malonedeb@chaenomeles.canonical.com> Message-ID: <153609092695.28911.6636737554945369429.malone@wampee.canonical.com> This issue was fixed in the openstack/heat 9.0.5 release. -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1757300 Title: RandomString may have less entropy than expected Status in OpenStack Heat: Fix Released Status in OpenStack Security Advisory: Won't Fix Bug description: When generating a random string, once we have selected from the various required pools, we continue by selecting a pool at random and then selecting a character from that pool at random. This does not take into account the differing sizes of the available pools, nor the fact that the same character could appear in multiple pools. This results in a non-uniform probability distribution of characters. For example, in the following resource: type: OS::Heat::RandomString properties: length: 66 character_classes: - class: lettersdigits character_sequences: - sequence: "*$" one might reasonably expect to find an average of 3 '*' or '$' characters in the output, but in fact there would be an average of 33. Since users mostly make use of this feature to generate default passwords for services they are deploying, this would result in the generated passwords having slightly less entropy than expected. Pathological cases where the entropy is massively reduced (like the one above - where it is only 229.5 bits vs. the expected 391 bits) are possible, although it's probably unlikely that users would encounter them by accident. To manage notifications about this bug go to: https://bugs.launchpad.net/heat/+bug/1757300/+subscriptions From 1625833 at bugs.launchpad.net Thu Sep 6 08:11:46 2018 From: 1625833 at bugs.launchpad.net (OpenStack Infra) Date: Thu, 06 Sep 2018 08:11:46 -0000 Subject: [Openstack-security] [Bug 1625833] Change abandoned on horizon (master) References: <20160920214728.9448.43144.malonedeb@chaenomeles.canonical.com> Message-ID: <153622150676.31580.14355600339401363994.malone@soybean.canonical.com> Change abandoned by Ivan Kolodyazhny (e0ne at e0ne.info) on branch: master Review: https://review.openstack.org/373540 Reason: This review is > 4 months without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results. -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1625833 Title: Prevent open redirects as a result of workflow action Status in OpenStack Dashboard (Horizon): Opinion Status in OpenStack Security Advisory: Won't Fix Bug description: For example: /admin/flavors/create/?next=http://www.foobar.com/ If a user is tricked into clicking that link, the flavor create workflow will be shown, but the redirect on form post will unexpectedly take the user to another site. Prevent this by checking that the next_url in WorkflowView.post is same origin. To manage notifications about this bug go to: https://bugs.launchpad.net/horizon/+bug/1625833/+subscriptions From 1734320 at bugs.launchpad.net Mon Sep 10 13:28:54 2018 From: 1734320 at bugs.launchpad.net (OpenStack Infra) Date: Mon, 10 Sep 2018 13:28:54 -0000 Subject: [Openstack-security] [Bug 1734320] Change abandoned on os-vif (master) References: <151152217834.14483.1577991310209811902.malonedeb@soybean.canonical.com> Message-ID: <153658613475.16673.12943249891292342541.malone@chaenomeles.canonical.com> Change abandoned by Slawek Kaplonski (skaplons at redhat.com) on branch: master Review: https://review.openstack.org/594118 Reason: That isn't good approach for sure. We will have to find something better -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1734320 Title: Eavesdropping private traffic Status in neutron: Triaged Status in OpenStack Compute (nova): Confirmed Status in os-vif: In Progress Status in OpenStack Security Advisory: Won't Fix Bug description: Eavesdropping private traffic ============================= Abstract -------- We've discovered a security issue that allows end users within their own private network to receive from, and send traffic to, other private networks on the same compute node. Description ----------- During live-migration there is a small time window where the ports of instances are untagged. Instances have a port trunked to the integration bridge and receive 802.1Q tagged private traffic from other tenants. If the port is administratively down during live migration, the port will remain in trunk mode indefinitely. Traffic is possible between ports is that are administratively down, even between tenants self-service networks. Conditions ---------- The following conditions are necessary. * Openvswitch Self-service networks * An Openstack administrator or an automated process needs to schedule a Live migration We tested this on newton. Issues ------ This outcome is the result of multiple independent issues. We will list the most important first, and follow with bugs that create a fragile situation. Issue #1 Initially creating a trunk port When the port is initially created, it is in trunk mode. This creates a fail-open situation. See: https://github.com/openstack/os-vif/blob/newton-eol/vif_plug_ovs/linux_net.py#L52 Recommendation: create ports in the port_dead state, don't leave it dangling in trunk-mode. Add a drop-flow initially. Issue #2 Order of creation. The instance is actually migrated before the (networking) configuration is completed. Recommendation: wait with finishing the live migration until the underlying configuration has been applied completely. Issue #3 Not closing the port when it is down. Neutron calls the port_dead function to ensure the port is down. It sets the tag to 4095 and adds a "drop" flow if (and only if) there is already another tag on the port. The port_dead function will keep untagged ports untagged. https://github.com/openstack/neutron/blob/stable/newton/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L995 Recommendation: Make port_dead also shut the port if no tag is found. Log a warning if this happens. Issue #4 Putting the port administratively down actually puts the port on a compute node shared vlan Instances from different projects on different private networks can talk to each other if they put their ports down. The code does install an openflow "drop" rule but it has a lower priority (2) than the allow rules. Recommendation: Increase the port_dead openflow drop rule priority to MAX Timeline --------  2017-09-14 Discovery eavesdropping issue  2017-09-15 Verify workaround.  2017-10-04 Discovery port-down-traffic issue  2017-11-24 Vendor Disclosure to Openstack Steps to reproduce ------------------ 1. Attach an instance to two networks: admin$ openstack server create --nic net-id= --nic net-id = --image --flavor instance_temp 2. Attach a FIP to the instance to be able to log in to this instance 3. Verify: admin$ openstack server show -c name -c addresses fe28a2ee-098f-4425 -9d3c-8e2cd383547d +-----------+-----------------------------------------------------------------------------+ | Field | Value | +-----------+-----------------------------------------------------------------------------+ | addresses | network1=192.168.99.8, ; network2=192.168.80.14 | | name | instance_temp | +-----------+-----------------------------------------------------------------------------+ 4. Ssh to the instance using network1 and run a tcpdump on the other port network2 [root at instance_temp]$ tcpdump -eeenni eth1 5. Get port-id of network2 admin$ nova interface-list fe28a2ee-098f-4425-9d3c-8e2cd383547d +------------+--------------------------------------+--------------------------------------+---------------+-------------------+ | Port State | Port ID | Net ID | IP addresses | MAC Addr | +------------+--------------------------------------+--------------------------------------+---------------+-------------------+ | ACTIVE | a848520b-0814-4030-bb48-49e4b5cf8160 | d69028f7-9558-4f14-8ce6-29cb8f1c19cd | 192.168.80.14 | fa:16:3e:2d:8b:7b | | ACTIVE | fad148ca-cf7a-4839-aac3-a2cd8d1d2260 | d22c22ae-0a42-4e3b-8144-f28534c3439a | 192.168.99.8 | fa:16:3e:60:2c:fa | +------------+--------------------------------------+--------------------------------------+---------------+-------------------+ 6. Force port down on network 2 admin$ neutron port-update a848520b-0814-4030-bb48-49e4b5cf8160 --admin-state-up False 7. Port gets tagged with vlan 4095, the dead vlan tag, which is normal: compute1# grep a848520b-0814-4030-bb48-49e4b5cf8160 /var/log/neutron/neutron-openvswitch-agent.log | tail -1 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-e008feb3-8a35-4c97-adac-b48ff88165b2 - - - - -] VIF port: a848520b-0814-4030-bb48-49e4b5cf8160 admin state up disabled, putting on the dead VLAN 8. Verify the port is tagged with vlan 4095 compute1# ovs-vsctl show | grep -A3 qvoa848520b-08       Port "qvoa848520b-08"           tag: 4095           Interface "qvoa848520b-08" 9. Now live-migrate the instance: admin# nova live-migration fe28a2ee-098f-4425-9d3c-8e2cd383547d 10. Verify the tag is gone on compute2, and take a deep breath compute2# ovs-vsctl show | grep -A3 qvoa848520b-08       Port "qvoa848520b-08"           Interface "qvoa848520b-08"       Port... compute2# echo "Wut!" 11. Now traffic of all other self-service networks present on compute2 can be sniffed from instance_temp [root at instance_temp] tcpdump -eenni eth1 13:14:31.748266 fa:16:3e:6a:17:38 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.103.12.160 tell 10.103.12.152, length 28 13:14:31.804573 fa:16:3e:e8:a2:d2 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.70, length 28 13:14:31.810482 fa:16:3e:95:ca:3a > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.154, length 28 13:14:31.977820 fa:16:3e:6f:f4:9b > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.150, length 28 13:14:31.979590 fa:16:3e:0f:3d:cc > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 9, p 0, ethertype ARP, Request who-has 10.103.9.163 tell 10.103.9.1, length 28 13:14:32.048082 fa:16:3e:65:64:38 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.101, length 28 13:14:32.127400 fa:16:3e:30:cb:b5 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.103.12.160 tell 10.103.12.165, length 28 13:14:32.141982 fa:16:3e:96:cd:b0 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.100, length 28 13:14:32.205327 fa:16:3e:a2:0b:76 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.153, length 28 13:14:32.444142 fa:16:3e:1f:db:ed > 01:00:5e:00:00:12, ethertype 802.1Q (0x8100), length 58: vlan 72, p 0, ethertype IPv4, 192.168.99.212 > 224.0.0.18: VRRPv2, Advertisement, vrid 50, prio 103, authtype none, intvl 1s, length 20 13:14:32.449497 fa:16:3e:1c:24:c0 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.20, length 28 13:14:32.476015 fa:16:3e:f2:3b:97 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.22, length 28 13:14:32.575034 fa:16:3e:44:fe:35 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.103.12.160 tell 10.103.12.163, length 28 13:14:32.676185 fa:16:3e:1e:92:d7 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.103.12.160 tell 10.103.12.150, length 28 13:14:32.711755 fa:16:3e:99:6c:c8 > 01:00:5e:00:00:12, ethertype 802.1Q (0x8100), length 62: vlan 10, p 0, ethertype IPv4, 10.103.12.154 > 224.0.0.18: VRRPv2, Advertisement, vrid 2, prio 49, authtype simple, intvl 1s, length 24 13:14:32.711773 fa:16:3e:f5:23:d5 > 01:00:5e:00:00:12, ethertype 802.1Q (0x8100), length 58: vlan 12, p 0, ethertype IPv4, 10.103.15.154 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 49, authtype simple, intvl 1s, length 20 Workaround ---------- We temporary fixed this issue by forcing the dead vlan tag on port creation on compute nodes: /usr/lib/python2.7/site-packages/vif_plug_ovs/linux_net.py: def _create_ovs_vif_cmd(bridge, dev, iface_id, mac,                         instance_id, interface_type=None,                         vhost_server_path=None): + # ODCN: initialize port as dead + # ODCN: TODO: set drop flow     cmd = ['--', '--if-exists', 'del-port', dev, '--',             'add-port', bridge, dev, + 'tag=4095',             '--', 'set', 'Interface', dev,             'external-ids:iface-id=%s' % iface_id,             'external-ids:iface-status=active',             'external-ids:attached-mac=%s' % mac,             'external-ids:vm-uuid=%s' % instance_id]     if interface_type:         cmd += ['type=%s' % interface_type]     if vhost_server_path:         cmd += ['options:vhost-server-path=%s' % vhost_server_path]     return cmd https://github.com/openstack/neutron/blob/stable/newton/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L995     def port_dead(self, port, log_errors=True):         '''Once a port has no binding, put it on the "dead vlan".         :param port: an ovs_lib.VifPort object.         '''         # Don't kill a port if it's already dead         cur_tag = self.int_br.db_get_val("Port", port.port_name, "tag",                                          log_errors=log_errors) + # ODCN GM 20170915 + if not cur_tag: + LOG.error('port_dead(): port %s has no tag', port.port_name) + # ODCN AJS 20170915 + if not cur_tag or cur_tag != constants.DEAD_VLAN_TAG: - if cur_tag and cur_tag != constants.DEAD_VLAN_TAG:            LOG.info('port_dead(): put port %s on dead vlan', port.port_name)            self.int_br.set_db_attribute("Port", port.port_name, "tag",                                          constants.DEAD_VLAN_TAG,                                          log_errors=log_errors)             self.int_br.drop_port(in_port=port.ofport) plugins/ml2/drivers/openvswitch/agent/openflow/ovs_ofctl/ovs_bridge.py     def drop_port(self, in_port): + # ODCN AJS 20171004: - self.install_drop(priority=2, in_port=in_port) + self.install_drop(priority=65535, in_port=in_port) Regards, ODC Noord. Gerhard Muntingh Albert Siersema Paul Peereboom To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1734320/+subscriptions From 1734320 at bugs.launchpad.net Tue Sep 11 14:20:09 2018 From: 1734320 at bugs.launchpad.net (Slawek Kaplonski) Date: Tue, 11 Sep 2018 14:20:09 -0000 Subject: [Openstack-security] [Bug 1734320] Re: Eavesdropping private traffic References: <151152217834.14483.1577991310209811902.malonedeb@soybean.canonical.com> Message-ID: <153667560924.24829.15700092607143413952.malone@gac.canonical.com> Yesterday I talked with Brian about this issue and we though about possible solution only in Neutron. Maybe change of openflows in br-int can fix this issue. We can try to use something like: 1. default action in br-int set to DROP 2. For each lvm id (local vlan id used for each network on host) set OF rule with ACTION=normal to proceed such packets in same way as it is now for all rules. What do You think about such solution? -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1734320 Title: Eavesdropping private traffic Status in neutron: Triaged Status in OpenStack Compute (nova): Confirmed Status in os-vif: In Progress Status in OpenStack Security Advisory: Won't Fix Bug description: Eavesdropping private traffic ============================= Abstract -------- We've discovered a security issue that allows end users within their own private network to receive from, and send traffic to, other private networks on the same compute node. Description ----------- During live-migration there is a small time window where the ports of instances are untagged. Instances have a port trunked to the integration bridge and receive 802.1Q tagged private traffic from other tenants. If the port is administratively down during live migration, the port will remain in trunk mode indefinitely. Traffic is possible between ports is that are administratively down, even between tenants self-service networks. Conditions ---------- The following conditions are necessary. * Openvswitch Self-service networks * An Openstack administrator or an automated process needs to schedule a Live migration We tested this on newton. Issues ------ This outcome is the result of multiple independent issues. We will list the most important first, and follow with bugs that create a fragile situation. Issue #1 Initially creating a trunk port When the port is initially created, it is in trunk mode. This creates a fail-open situation. See: https://github.com/openstack/os-vif/blob/newton-eol/vif_plug_ovs/linux_net.py#L52 Recommendation: create ports in the port_dead state, don't leave it dangling in trunk-mode. Add a drop-flow initially. Issue #2 Order of creation. The instance is actually migrated before the (networking) configuration is completed. Recommendation: wait with finishing the live migration until the underlying configuration has been applied completely. Issue #3 Not closing the port when it is down. Neutron calls the port_dead function to ensure the port is down. It sets the tag to 4095 and adds a "drop" flow if (and only if) there is already another tag on the port. The port_dead function will keep untagged ports untagged. https://github.com/openstack/neutron/blob/stable/newton/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L995 Recommendation: Make port_dead also shut the port if no tag is found. Log a warning if this happens. Issue #4 Putting the port administratively down actually puts the port on a compute node shared vlan Instances from different projects on different private networks can talk to each other if they put their ports down. The code does install an openflow "drop" rule but it has a lower priority (2) than the allow rules. Recommendation: Increase the port_dead openflow drop rule priority to MAX Timeline --------  2017-09-14 Discovery eavesdropping issue  2017-09-15 Verify workaround.  2017-10-04 Discovery port-down-traffic issue  2017-11-24 Vendor Disclosure to Openstack Steps to reproduce ------------------ 1. Attach an instance to two networks: admin$ openstack server create --nic net-id= --nic net-id = --image --flavor instance_temp 2. Attach a FIP to the instance to be able to log in to this instance 3. Verify: admin$ openstack server show -c name -c addresses fe28a2ee-098f-4425 -9d3c-8e2cd383547d +-----------+-----------------------------------------------------------------------------+ | Field | Value | +-----------+-----------------------------------------------------------------------------+ | addresses | network1=192.168.99.8, ; network2=192.168.80.14 | | name | instance_temp | +-----------+-----------------------------------------------------------------------------+ 4. Ssh to the instance using network1 and run a tcpdump on the other port network2 [root at instance_temp]$ tcpdump -eeenni eth1 5. Get port-id of network2 admin$ nova interface-list fe28a2ee-098f-4425-9d3c-8e2cd383547d +------------+--------------------------------------+--------------------------------------+---------------+-------------------+ | Port State | Port ID | Net ID | IP addresses | MAC Addr | +------------+--------------------------------------+--------------------------------------+---------------+-------------------+ | ACTIVE | a848520b-0814-4030-bb48-49e4b5cf8160 | d69028f7-9558-4f14-8ce6-29cb8f1c19cd | 192.168.80.14 | fa:16:3e:2d:8b:7b | | ACTIVE | fad148ca-cf7a-4839-aac3-a2cd8d1d2260 | d22c22ae-0a42-4e3b-8144-f28534c3439a | 192.168.99.8 | fa:16:3e:60:2c:fa | +------------+--------------------------------------+--------------------------------------+---------------+-------------------+ 6. Force port down on network 2 admin$ neutron port-update a848520b-0814-4030-bb48-49e4b5cf8160 --admin-state-up False 7. Port gets tagged with vlan 4095, the dead vlan tag, which is normal: compute1# grep a848520b-0814-4030-bb48-49e4b5cf8160 /var/log/neutron/neutron-openvswitch-agent.log | tail -1 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-e008feb3-8a35-4c97-adac-b48ff88165b2 - - - - -] VIF port: a848520b-0814-4030-bb48-49e4b5cf8160 admin state up disabled, putting on the dead VLAN 8. Verify the port is tagged with vlan 4095 compute1# ovs-vsctl show | grep -A3 qvoa848520b-08       Port "qvoa848520b-08"           tag: 4095           Interface "qvoa848520b-08" 9. Now live-migrate the instance: admin# nova live-migration fe28a2ee-098f-4425-9d3c-8e2cd383547d 10. Verify the tag is gone on compute2, and take a deep breath compute2# ovs-vsctl show | grep -A3 qvoa848520b-08       Port "qvoa848520b-08"           Interface "qvoa848520b-08"       Port... compute2# echo "Wut!" 11. Now traffic of all other self-service networks present on compute2 can be sniffed from instance_temp [root at instance_temp] tcpdump -eenni eth1 13:14:31.748266 fa:16:3e:6a:17:38 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.103.12.160 tell 10.103.12.152, length 28 13:14:31.804573 fa:16:3e:e8:a2:d2 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.70, length 28 13:14:31.810482 fa:16:3e:95:ca:3a > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.154, length 28 13:14:31.977820 fa:16:3e:6f:f4:9b > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.150, length 28 13:14:31.979590 fa:16:3e:0f:3d:cc > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 9, p 0, ethertype ARP, Request who-has 10.103.9.163 tell 10.103.9.1, length 28 13:14:32.048082 fa:16:3e:65:64:38 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.101, length 28 13:14:32.127400 fa:16:3e:30:cb:b5 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.103.12.160 tell 10.103.12.165, length 28 13:14:32.141982 fa:16:3e:96:cd:b0 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.100, length 28 13:14:32.205327 fa:16:3e:a2:0b:76 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.153, length 28 13:14:32.444142 fa:16:3e:1f:db:ed > 01:00:5e:00:00:12, ethertype 802.1Q (0x8100), length 58: vlan 72, p 0, ethertype IPv4, 192.168.99.212 > 224.0.0.18: VRRPv2, Advertisement, vrid 50, prio 103, authtype none, intvl 1s, length 20 13:14:32.449497 fa:16:3e:1c:24:c0 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.20, length 28 13:14:32.476015 fa:16:3e:f2:3b:97 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.22, length 28 13:14:32.575034 fa:16:3e:44:fe:35 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.103.12.160 tell 10.103.12.163, length 28 13:14:32.676185 fa:16:3e:1e:92:d7 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.103.12.160 tell 10.103.12.150, length 28 13:14:32.711755 fa:16:3e:99:6c:c8 > 01:00:5e:00:00:12, ethertype 802.1Q (0x8100), length 62: vlan 10, p 0, ethertype IPv4, 10.103.12.154 > 224.0.0.18: VRRPv2, Advertisement, vrid 2, prio 49, authtype simple, intvl 1s, length 24 13:14:32.711773 fa:16:3e:f5:23:d5 > 01:00:5e:00:00:12, ethertype 802.1Q (0x8100), length 58: vlan 12, p 0, ethertype IPv4, 10.103.15.154 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 49, authtype simple, intvl 1s, length 20 Workaround ---------- We temporary fixed this issue by forcing the dead vlan tag on port creation on compute nodes: /usr/lib/python2.7/site-packages/vif_plug_ovs/linux_net.py: def _create_ovs_vif_cmd(bridge, dev, iface_id, mac,                         instance_id, interface_type=None,                         vhost_server_path=None): + # ODCN: initialize port as dead + # ODCN: TODO: set drop flow     cmd = ['--', '--if-exists', 'del-port', dev, '--',             'add-port', bridge, dev, + 'tag=4095',             '--', 'set', 'Interface', dev,             'external-ids:iface-id=%s' % iface_id,             'external-ids:iface-status=active',             'external-ids:attached-mac=%s' % mac,             'external-ids:vm-uuid=%s' % instance_id]     if interface_type:         cmd += ['type=%s' % interface_type]     if vhost_server_path:         cmd += ['options:vhost-server-path=%s' % vhost_server_path]     return cmd https://github.com/openstack/neutron/blob/stable/newton/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L995     def port_dead(self, port, log_errors=True):         '''Once a port has no binding, put it on the "dead vlan".         :param port: an ovs_lib.VifPort object.         '''         # Don't kill a port if it's already dead         cur_tag = self.int_br.db_get_val("Port", port.port_name, "tag",                                          log_errors=log_errors) + # ODCN GM 20170915 + if not cur_tag: + LOG.error('port_dead(): port %s has no tag', port.port_name) + # ODCN AJS 20170915 + if not cur_tag or cur_tag != constants.DEAD_VLAN_TAG: - if cur_tag and cur_tag != constants.DEAD_VLAN_TAG:            LOG.info('port_dead(): put port %s on dead vlan', port.port_name)            self.int_br.set_db_attribute("Port", port.port_name, "tag",                                          constants.DEAD_VLAN_TAG,                                          log_errors=log_errors)             self.int_br.drop_port(in_port=port.ofport) plugins/ml2/drivers/openvswitch/agent/openflow/ovs_ofctl/ovs_bridge.py     def drop_port(self, in_port): + # ODCN AJS 20171004: - self.install_drop(priority=2, in_port=in_port) + self.install_drop(priority=65535, in_port=in_port) Regards, ODC Noord. Gerhard Muntingh Albert Siersema Paul Peereboom To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1734320/+subscriptions From morgan.fainberg at gmail.com Wed Sep 12 14:59:09 2018 From: morgan.fainberg at gmail.com (Morgan Fainberg) Date: Wed, 12 Sep 2018 14:59:09 -0000 Subject: [Openstack-security] [Bug 1792047] Re: keystone rbacenforcer not populating policy dict with view args References: <153670878287.17124.3225642471309073829.malonedeb@chaenomeles.canonical.com> Message-ID: <153676434928.17161.10979535434354201870.malone@chaenomeles.canonical.com> he concern is the opposite of exploitable. It can lock keystone's api too closed. It is security in that sense, it should be a tag I guess. Hide quoted ** Information type changed from Public Security to Public ** Tags added: policy security -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1792047 Title: keystone rbacenforcer not populating policy dict with view args Status in OpenStack Identity (keystone): In Progress Status in OpenStack Identity (keystone) rocky series: In Progress Status in OpenStack Identity (keystone) stein series: In Progress Bug description: The old @protected decorator pushed the view arguments into the policy_dict for enforcement purposes[0]. This was missed in the new RBACEnforcer. [0] https://github.com/openstack/keystone/blob/294ca38554bb229f66a772e7dba35a5b08a36b20/keystone/common/authorization.py#L152 To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1792047/+subscriptions From morgan.fainberg at gmail.com Wed Sep 12 14:57:58 2018 From: morgan.fainberg at gmail.com (Morgan Fainberg) Date: Wed, 12 Sep 2018 14:57:58 -0000 Subject: [Openstack-security] [Bug 1792047] Re: keystone rbacenforcer not populating policy dict with view args References: <153670878287.17124.3225642471309073829.malonedeb@chaenomeles.canonical.com> <153676280154.16997.5454166496161449120.malone@chaenomeles.canonical.com> Message-ID: The concern is the opposite of exploitable. It can lock keystone's api too closed. It is security in that sense, it should be a tag I guess. On Wed, Sep 12, 2018, 08:41 Jeremy Stanley wrote: > Is this considered exploitable (class A vulnerability report)? Or should > it be using the security bugtag to indicate a hardening opportunity > instead of the Public Security bug type? > > -- > You received this bug notification because you are subscribed to the bug > report. > Matching subscriptions: Private security bugs > https://bugs.launchpad.net/bugs/1792047 > > Title: > keystone rbacenforcer not populating policy dict with view args > > Status in OpenStack Identity (keystone): > In Progress > Status in OpenStack Identity (keystone) rocky series: > In Progress > Status in OpenStack Identity (keystone) stein series: > In Progress > > Bug description: > The old @protected decorator pushed the view arguments into the > policy_dict for enforcement purposes[0]. This was missed in the new > RBACEnforcer. > > [0] > > https://github.com/openstack/keystone/blob/294ca38554bb229f66a772e7dba35a5b08a36b20/keystone/common/authorization.py#L152 > > To manage notifications about this bug go to: > https://bugs.launchpad.net/keystone/+bug/1792047/+subscriptions > -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1792047 Title: keystone rbacenforcer not populating policy dict with view args Status in OpenStack Identity (keystone): In Progress Status in OpenStack Identity (keystone) rocky series: In Progress Status in OpenStack Identity (keystone) stein series: In Progress Bug description: The old @protected decorator pushed the view arguments into the policy_dict for enforcement purposes[0]. This was missed in the new RBACEnforcer. [0] https://github.com/openstack/keystone/blob/294ca38554bb229f66a772e7dba35a5b08a36b20/keystone/common/authorization.py#L152 To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1792047/+subscriptions From 1734320 at bugs.launchpad.net Thu Sep 13 16:01:56 2018 From: 1734320 at bugs.launchpad.net (OpenStack Infra) Date: Thu, 13 Sep 2018 16:01:56 -0000 Subject: [Openstack-security] [Bug 1734320] Re: Eavesdropping private traffic References: <151152217834.14483.1577991310209811902.malonedeb@soybean.canonical.com> Message-ID: <153685451663.28670.473010720584589417.malone@wampee.canonical.com> Fix proposed to branch: master Review: https://review.openstack.org/602384 ** Changed in: os-vif Assignee: Slawek Kaplonski (slaweq) => sean mooney (sean-k-mooney) -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1734320 Title: Eavesdropping private traffic Status in neutron: Triaged Status in OpenStack Compute (nova): Confirmed Status in os-vif: In Progress Status in OpenStack Security Advisory: Won't Fix Bug description: Eavesdropping private traffic ============================= Abstract -------- We've discovered a security issue that allows end users within their own private network to receive from, and send traffic to, other private networks on the same compute node. Description ----------- During live-migration there is a small time window where the ports of instances are untagged. Instances have a port trunked to the integration bridge and receive 802.1Q tagged private traffic from other tenants. If the port is administratively down during live migration, the port will remain in trunk mode indefinitely. Traffic is possible between ports is that are administratively down, even between tenants self-service networks. Conditions ---------- The following conditions are necessary. * Openvswitch Self-service networks * An Openstack administrator or an automated process needs to schedule a Live migration We tested this on newton. Issues ------ This outcome is the result of multiple independent issues. We will list the most important first, and follow with bugs that create a fragile situation. Issue #1 Initially creating a trunk port When the port is initially created, it is in trunk mode. This creates a fail-open situation. See: https://github.com/openstack/os-vif/blob/newton-eol/vif_plug_ovs/linux_net.py#L52 Recommendation: create ports in the port_dead state, don't leave it dangling in trunk-mode. Add a drop-flow initially. Issue #2 Order of creation. The instance is actually migrated before the (networking) configuration is completed. Recommendation: wait with finishing the live migration until the underlying configuration has been applied completely. Issue #3 Not closing the port when it is down. Neutron calls the port_dead function to ensure the port is down. It sets the tag to 4095 and adds a "drop" flow if (and only if) there is already another tag on the port. The port_dead function will keep untagged ports untagged. https://github.com/openstack/neutron/blob/stable/newton/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L995 Recommendation: Make port_dead also shut the port if no tag is found. Log a warning if this happens. Issue #4 Putting the port administratively down actually puts the port on a compute node shared vlan Instances from different projects on different private networks can talk to each other if they put their ports down. The code does install an openflow "drop" rule but it has a lower priority (2) than the allow rules. Recommendation: Increase the port_dead openflow drop rule priority to MAX Timeline --------  2017-09-14 Discovery eavesdropping issue  2017-09-15 Verify workaround.  2017-10-04 Discovery port-down-traffic issue  2017-11-24 Vendor Disclosure to Openstack Steps to reproduce ------------------ 1. Attach an instance to two networks: admin$ openstack server create --nic net-id= --nic net-id = --image --flavor instance_temp 2. Attach a FIP to the instance to be able to log in to this instance 3. Verify: admin$ openstack server show -c name -c addresses fe28a2ee-098f-4425 -9d3c-8e2cd383547d +-----------+-----------------------------------------------------------------------------+ | Field | Value | +-----------+-----------------------------------------------------------------------------+ | addresses | network1=192.168.99.8, ; network2=192.168.80.14 | | name | instance_temp | +-----------+-----------------------------------------------------------------------------+ 4. Ssh to the instance using network1 and run a tcpdump on the other port network2 [root at instance_temp]$ tcpdump -eeenni eth1 5. Get port-id of network2 admin$ nova interface-list fe28a2ee-098f-4425-9d3c-8e2cd383547d +------------+--------------------------------------+--------------------------------------+---------------+-------------------+ | Port State | Port ID | Net ID | IP addresses | MAC Addr | +------------+--------------------------------------+--------------------------------------+---------------+-------------------+ | ACTIVE | a848520b-0814-4030-bb48-49e4b5cf8160 | d69028f7-9558-4f14-8ce6-29cb8f1c19cd | 192.168.80.14 | fa:16:3e:2d:8b:7b | | ACTIVE | fad148ca-cf7a-4839-aac3-a2cd8d1d2260 | d22c22ae-0a42-4e3b-8144-f28534c3439a | 192.168.99.8 | fa:16:3e:60:2c:fa | +------------+--------------------------------------+--------------------------------------+---------------+-------------------+ 6. Force port down on network 2 admin$ neutron port-update a848520b-0814-4030-bb48-49e4b5cf8160 --admin-state-up False 7. Port gets tagged with vlan 4095, the dead vlan tag, which is normal: compute1# grep a848520b-0814-4030-bb48-49e4b5cf8160 /var/log/neutron/neutron-openvswitch-agent.log | tail -1 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-e008feb3-8a35-4c97-adac-b48ff88165b2 - - - - -] VIF port: a848520b-0814-4030-bb48-49e4b5cf8160 admin state up disabled, putting on the dead VLAN 8. Verify the port is tagged with vlan 4095 compute1# ovs-vsctl show | grep -A3 qvoa848520b-08       Port "qvoa848520b-08"           tag: 4095           Interface "qvoa848520b-08" 9. Now live-migrate the instance: admin# nova live-migration fe28a2ee-098f-4425-9d3c-8e2cd383547d 10. Verify the tag is gone on compute2, and take a deep breath compute2# ovs-vsctl show | grep -A3 qvoa848520b-08       Port "qvoa848520b-08"           Interface "qvoa848520b-08"       Port... compute2# echo "Wut!" 11. Now traffic of all other self-service networks present on compute2 can be sniffed from instance_temp [root at instance_temp] tcpdump -eenni eth1 13:14:31.748266 fa:16:3e:6a:17:38 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.103.12.160 tell 10.103.12.152, length 28 13:14:31.804573 fa:16:3e:e8:a2:d2 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.70, length 28 13:14:31.810482 fa:16:3e:95:ca:3a > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.154, length 28 13:14:31.977820 fa:16:3e:6f:f4:9b > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.150, length 28 13:14:31.979590 fa:16:3e:0f:3d:cc > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 9, p 0, ethertype ARP, Request who-has 10.103.9.163 tell 10.103.9.1, length 28 13:14:32.048082 fa:16:3e:65:64:38 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.101, length 28 13:14:32.127400 fa:16:3e:30:cb:b5 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.103.12.160 tell 10.103.12.165, length 28 13:14:32.141982 fa:16:3e:96:cd:b0 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.100, length 28 13:14:32.205327 fa:16:3e:a2:0b:76 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.153, length 28 13:14:32.444142 fa:16:3e:1f:db:ed > 01:00:5e:00:00:12, ethertype 802.1Q (0x8100), length 58: vlan 72, p 0, ethertype IPv4, 192.168.99.212 > 224.0.0.18: VRRPv2, Advertisement, vrid 50, prio 103, authtype none, intvl 1s, length 20 13:14:32.449497 fa:16:3e:1c:24:c0 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.20, length 28 13:14:32.476015 fa:16:3e:f2:3b:97 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.22, length 28 13:14:32.575034 fa:16:3e:44:fe:35 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.103.12.160 tell 10.103.12.163, length 28 13:14:32.676185 fa:16:3e:1e:92:d7 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.103.12.160 tell 10.103.12.150, length 28 13:14:32.711755 fa:16:3e:99:6c:c8 > 01:00:5e:00:00:12, ethertype 802.1Q (0x8100), length 62: vlan 10, p 0, ethertype IPv4, 10.103.12.154 > 224.0.0.18: VRRPv2, Advertisement, vrid 2, prio 49, authtype simple, intvl 1s, length 24 13:14:32.711773 fa:16:3e:f5:23:d5 > 01:00:5e:00:00:12, ethertype 802.1Q (0x8100), length 58: vlan 12, p 0, ethertype IPv4, 10.103.15.154 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 49, authtype simple, intvl 1s, length 20 Workaround ---------- We temporary fixed this issue by forcing the dead vlan tag on port creation on compute nodes: /usr/lib/python2.7/site-packages/vif_plug_ovs/linux_net.py: def _create_ovs_vif_cmd(bridge, dev, iface_id, mac,                         instance_id, interface_type=None,                         vhost_server_path=None): + # ODCN: initialize port as dead + # ODCN: TODO: set drop flow     cmd = ['--', '--if-exists', 'del-port', dev, '--',             'add-port', bridge, dev, + 'tag=4095',             '--', 'set', 'Interface', dev,             'external-ids:iface-id=%s' % iface_id,             'external-ids:iface-status=active',             'external-ids:attached-mac=%s' % mac,             'external-ids:vm-uuid=%s' % instance_id]     if interface_type:         cmd += ['type=%s' % interface_type]     if vhost_server_path:         cmd += ['options:vhost-server-path=%s' % vhost_server_path]     return cmd https://github.com/openstack/neutron/blob/stable/newton/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L995     def port_dead(self, port, log_errors=True):         '''Once a port has no binding, put it on the "dead vlan".         :param port: an ovs_lib.VifPort object.         '''         # Don't kill a port if it's already dead         cur_tag = self.int_br.db_get_val("Port", port.port_name, "tag",                                          log_errors=log_errors) + # ODCN GM 20170915 + if not cur_tag: + LOG.error('port_dead(): port %s has no tag', port.port_name) + # ODCN AJS 20170915 + if not cur_tag or cur_tag != constants.DEAD_VLAN_TAG: - if cur_tag and cur_tag != constants.DEAD_VLAN_TAG:            LOG.info('port_dead(): put port %s on dead vlan', port.port_name)            self.int_br.set_db_attribute("Port", port.port_name, "tag",                                          constants.DEAD_VLAN_TAG,                                          log_errors=log_errors)             self.int_br.drop_port(in_port=port.ofport) plugins/ml2/drivers/openvswitch/agent/openflow/ovs_ofctl/ovs_bridge.py     def drop_port(self, in_port): + # ODCN AJS 20171004: - self.install_drop(priority=2, in_port=in_port) + self.install_drop(priority=65535, in_port=in_port) Regards, ODC Noord. Gerhard Muntingh Albert Siersema Paul Peereboom To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1734320/+subscriptions From 1734320 at bugs.launchpad.net Thu Sep 13 20:04:01 2018 From: 1734320 at bugs.launchpad.net (OpenStack Infra) Date: Thu, 13 Sep 2018 20:04:01 -0000 Subject: [Openstack-security] [Bug 1734320] Re: Eavesdropping private traffic References: <151152217834.14483.1577991310209811902.malonedeb@soybean.canonical.com> Message-ID: <153686904161.29221.2880945713114437907.malone@wampee.canonical.com> Fix proposed to branch: master Review: https://review.openstack.org/602432 ** Changed in: nova Status: Confirmed => In Progress ** Changed in: nova Assignee: (unassigned) => sean mooney (sean-k-mooney) -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1734320 Title: Eavesdropping private traffic Status in neutron: Triaged Status in OpenStack Compute (nova): In Progress Status in os-vif: In Progress Status in OpenStack Security Advisory: Won't Fix Bug description: Eavesdropping private traffic ============================= Abstract -------- We've discovered a security issue that allows end users within their own private network to receive from, and send traffic to, other private networks on the same compute node. Description ----------- During live-migration there is a small time window where the ports of instances are untagged. Instances have a port trunked to the integration bridge and receive 802.1Q tagged private traffic from other tenants. If the port is administratively down during live migration, the port will remain in trunk mode indefinitely. Traffic is possible between ports is that are administratively down, even between tenants self-service networks. Conditions ---------- The following conditions are necessary. * Openvswitch Self-service networks * An Openstack administrator or an automated process needs to schedule a Live migration We tested this on newton. Issues ------ This outcome is the result of multiple independent issues. We will list the most important first, and follow with bugs that create a fragile situation. Issue #1 Initially creating a trunk port When the port is initially created, it is in trunk mode. This creates a fail-open situation. See: https://github.com/openstack/os-vif/blob/newton-eol/vif_plug_ovs/linux_net.py#L52 Recommendation: create ports in the port_dead state, don't leave it dangling in trunk-mode. Add a drop-flow initially. Issue #2 Order of creation. The instance is actually migrated before the (networking) configuration is completed. Recommendation: wait with finishing the live migration until the underlying configuration has been applied completely. Issue #3 Not closing the port when it is down. Neutron calls the port_dead function to ensure the port is down. It sets the tag to 4095 and adds a "drop" flow if (and only if) there is already another tag on the port. The port_dead function will keep untagged ports untagged. https://github.com/openstack/neutron/blob/stable/newton/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L995 Recommendation: Make port_dead also shut the port if no tag is found. Log a warning if this happens. Issue #4 Putting the port administratively down actually puts the port on a compute node shared vlan Instances from different projects on different private networks can talk to each other if they put their ports down. The code does install an openflow "drop" rule but it has a lower priority (2) than the allow rules. Recommendation: Increase the port_dead openflow drop rule priority to MAX Timeline --------  2017-09-14 Discovery eavesdropping issue  2017-09-15 Verify workaround.  2017-10-04 Discovery port-down-traffic issue  2017-11-24 Vendor Disclosure to Openstack Steps to reproduce ------------------ 1. Attach an instance to two networks: admin$ openstack server create --nic net-id= --nic net-id = --image --flavor instance_temp 2. Attach a FIP to the instance to be able to log in to this instance 3. Verify: admin$ openstack server show -c name -c addresses fe28a2ee-098f-4425 -9d3c-8e2cd383547d +-----------+-----------------------------------------------------------------------------+ | Field | Value | +-----------+-----------------------------------------------------------------------------+ | addresses | network1=192.168.99.8, ; network2=192.168.80.14 | | name | instance_temp | +-----------+-----------------------------------------------------------------------------+ 4. Ssh to the instance using network1 and run a tcpdump on the other port network2 [root at instance_temp]$ tcpdump -eeenni eth1 5. Get port-id of network2 admin$ nova interface-list fe28a2ee-098f-4425-9d3c-8e2cd383547d +------------+--------------------------------------+--------------------------------------+---------------+-------------------+ | Port State | Port ID | Net ID | IP addresses | MAC Addr | +------------+--------------------------------------+--------------------------------------+---------------+-------------------+ | ACTIVE | a848520b-0814-4030-bb48-49e4b5cf8160 | d69028f7-9558-4f14-8ce6-29cb8f1c19cd | 192.168.80.14 | fa:16:3e:2d:8b:7b | | ACTIVE | fad148ca-cf7a-4839-aac3-a2cd8d1d2260 | d22c22ae-0a42-4e3b-8144-f28534c3439a | 192.168.99.8 | fa:16:3e:60:2c:fa | +------------+--------------------------------------+--------------------------------------+---------------+-------------------+ 6. Force port down on network 2 admin$ neutron port-update a848520b-0814-4030-bb48-49e4b5cf8160 --admin-state-up False 7. Port gets tagged with vlan 4095, the dead vlan tag, which is normal: compute1# grep a848520b-0814-4030-bb48-49e4b5cf8160 /var/log/neutron/neutron-openvswitch-agent.log | tail -1 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-e008feb3-8a35-4c97-adac-b48ff88165b2 - - - - -] VIF port: a848520b-0814-4030-bb48-49e4b5cf8160 admin state up disabled, putting on the dead VLAN 8. Verify the port is tagged with vlan 4095 compute1# ovs-vsctl show | grep -A3 qvoa848520b-08       Port "qvoa848520b-08"           tag: 4095           Interface "qvoa848520b-08" 9. Now live-migrate the instance: admin# nova live-migration fe28a2ee-098f-4425-9d3c-8e2cd383547d 10. Verify the tag is gone on compute2, and take a deep breath compute2# ovs-vsctl show | grep -A3 qvoa848520b-08       Port "qvoa848520b-08"           Interface "qvoa848520b-08"       Port... compute2# echo "Wut!" 11. Now traffic of all other self-service networks present on compute2 can be sniffed from instance_temp [root at instance_temp] tcpdump -eenni eth1 13:14:31.748266 fa:16:3e:6a:17:38 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.103.12.160 tell 10.103.12.152, length 28 13:14:31.804573 fa:16:3e:e8:a2:d2 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.70, length 28 13:14:31.810482 fa:16:3e:95:ca:3a > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.154, length 28 13:14:31.977820 fa:16:3e:6f:f4:9b > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.150, length 28 13:14:31.979590 fa:16:3e:0f:3d:cc > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 9, p 0, ethertype ARP, Request who-has 10.103.9.163 tell 10.103.9.1, length 28 13:14:32.048082 fa:16:3e:65:64:38 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.101, length 28 13:14:32.127400 fa:16:3e:30:cb:b5 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.103.12.160 tell 10.103.12.165, length 28 13:14:32.141982 fa:16:3e:96:cd:b0 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.100, length 28 13:14:32.205327 fa:16:3e:a2:0b:76 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.153, length 28 13:14:32.444142 fa:16:3e:1f:db:ed > 01:00:5e:00:00:12, ethertype 802.1Q (0x8100), length 58: vlan 72, p 0, ethertype IPv4, 192.168.99.212 > 224.0.0.18: VRRPv2, Advertisement, vrid 50, prio 103, authtype none, intvl 1s, length 20 13:14:32.449497 fa:16:3e:1c:24:c0 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.20, length 28 13:14:32.476015 fa:16:3e:f2:3b:97 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.22, length 28 13:14:32.575034 fa:16:3e:44:fe:35 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.103.12.160 tell 10.103.12.163, length 28 13:14:32.676185 fa:16:3e:1e:92:d7 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.103.12.160 tell 10.103.12.150, length 28 13:14:32.711755 fa:16:3e:99:6c:c8 > 01:00:5e:00:00:12, ethertype 802.1Q (0x8100), length 62: vlan 10, p 0, ethertype IPv4, 10.103.12.154 > 224.0.0.18: VRRPv2, Advertisement, vrid 2, prio 49, authtype simple, intvl 1s, length 24 13:14:32.711773 fa:16:3e:f5:23:d5 > 01:00:5e:00:00:12, ethertype 802.1Q (0x8100), length 58: vlan 12, p 0, ethertype IPv4, 10.103.15.154 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 49, authtype simple, intvl 1s, length 20 Workaround ---------- We temporary fixed this issue by forcing the dead vlan tag on port creation on compute nodes: /usr/lib/python2.7/site-packages/vif_plug_ovs/linux_net.py: def _create_ovs_vif_cmd(bridge, dev, iface_id, mac,                         instance_id, interface_type=None,                         vhost_server_path=None): + # ODCN: initialize port as dead + # ODCN: TODO: set drop flow     cmd = ['--', '--if-exists', 'del-port', dev, '--',             'add-port', bridge, dev, + 'tag=4095',             '--', 'set', 'Interface', dev,             'external-ids:iface-id=%s' % iface_id,             'external-ids:iface-status=active',             'external-ids:attached-mac=%s' % mac,             'external-ids:vm-uuid=%s' % instance_id]     if interface_type:         cmd += ['type=%s' % interface_type]     if vhost_server_path:         cmd += ['options:vhost-server-path=%s' % vhost_server_path]     return cmd https://github.com/openstack/neutron/blob/stable/newton/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L995     def port_dead(self, port, log_errors=True):         '''Once a port has no binding, put it on the "dead vlan".         :param port: an ovs_lib.VifPort object.         '''         # Don't kill a port if it's already dead         cur_tag = self.int_br.db_get_val("Port", port.port_name, "tag",                                          log_errors=log_errors) + # ODCN GM 20170915 + if not cur_tag: + LOG.error('port_dead(): port %s has no tag', port.port_name) + # ODCN AJS 20170915 + if not cur_tag or cur_tag != constants.DEAD_VLAN_TAG: - if cur_tag and cur_tag != constants.DEAD_VLAN_TAG:            LOG.info('port_dead(): put port %s on dead vlan', port.port_name)            self.int_br.set_db_attribute("Port", port.port_name, "tag",                                          constants.DEAD_VLAN_TAG,                                          log_errors=log_errors)             self.int_br.drop_port(in_port=port.ofport) plugins/ml2/drivers/openvswitch/agent/openflow/ovs_ofctl/ovs_bridge.py     def drop_port(self, in_port): + # ODCN AJS 20171004: - self.install_drop(priority=2, in_port=in_port) + self.install_drop(priority=65535, in_port=in_port) Regards, ODC Noord. Gerhard Muntingh Albert Siersema Paul Peereboom To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1734320/+subscriptions From 1792047 at bugs.launchpad.net Sat Sep 15 00:15:52 2018 From: 1792047 at bugs.launchpad.net (OpenStack Infra) Date: Sat, 15 Sep 2018 00:15:52 -0000 Subject: [Openstack-security] [Bug 1792047] Re: keystone rbacenforcer not populating policy dict with view args References: <153670878287.17124.3225642471309073829.malonedeb@chaenomeles.canonical.com> Message-ID: <153697055209.12067.3616590399783916378.malone@gac.canonical.com> Reviewed: https://review.openstack.org/601875 Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=4975b79e8174587f7639347939cf679460d4896b Submitter: Zuul Branch: master commit 4975b79e8174587f7639347939cf679460d4896b Author: morgan fainberg Date: Tue Sep 11 16:03:54 2018 -0700 Ensure view args is in policy dict The policy_dict (in enforcement) was not populating the view args in a similar manner to the old style @protected decorator. This change ensures that we mirror the old behavior (required for proper use of v3cloud policy). Change-Id: Ida9009a95a874be9cc60c3152d4e3225726562eb Partial-Bug: #1776504 Closes-Bug: #1792047 ** Changed in: keystone Status: In Progress => Fix Released -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1792047 Title: keystone rbacenforcer not populating policy dict with view args Status in OpenStack Identity (keystone): Fix Released Status in OpenStack Identity (keystone) rocky series: In Progress Status in OpenStack Identity (keystone) stein series: Fix Released Bug description: The old @protected decorator pushed the view arguments into the policy_dict for enforcement purposes[0]. This was missed in the new RBACEnforcer. [0] https://github.com/openstack/keystone/blob/294ca38554bb229f66a772e7dba35a5b08a36b20/keystone/common/authorization.py#L152 To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1792047/+subscriptions From fungi at yuggoth.org Thu Sep 20 13:16:51 2018 From: fungi at yuggoth.org (Jeremy Stanley) Date: Thu, 20 Sep 2018 13:16:51 -0000 Subject: [Openstack-security] [Bug 1793029] Re: adding 0.0.0.0/0 address pair to a port bypasses all other vm security groups References: <153722407420.8513.15403635939639716317.malonedeb@chaenomeles.canonical.com> Message-ID: <153744941106.2634.10575931267286040060.malone@wampee.canonical.com> Thanks everyone for the prompt analysis. I've triaged this as a class B2 port per the OpenStack VMT taxonomy: https://security.openstack.org/vmt- process.html#incident-report-taxonomy ** Information type changed from Private Security to Public ** Changed in: ossa Status: Incomplete => Won't Fix ** Also affects: ossn Importance: Undecided Status: New ** Description changed: - This issue is being treated as a potential security risk under embargo. - Please do not make any public mention of embargoed (private) security - vulnerabilities before their coordinated publication by the OpenStack - Vulnerability Management Team in the form of an official OpenStack - Security Advisory. This includes discussion of the bug or associated - fixes in public forums such as mailing lists, code review systems and - bug trackers. Please also avoid private disclosure to other individuals - not already approved for access to this information, and provide this - same reminder to those who are made aware of the issue prior to - publication. All discussion should remain confined to this private bug - report, and any proposed fixes should be added to the bug as - attachments. - On an openstack-ansible / newton setup with linuxbridge, a customer ran: neutron port-update $port-uuid --allowed-address-pairs type=dict list=true ip_address=0.0.0.0/0 to bypass the ip source restriction (pfsense router and had to route packets). The impact of running the above, was an allow all rule was added to all ports in the network, bypassing all security groups. The iptables rule:  905K 55M RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 match-set NIPv44046d62c-59c8-4fd0-a547- src used on all ports, now triggers as: 0.0.0.0/1 128.0.0.0/1 was added to the ipset NIPv44046d62c-59c8-4fd0-a547 (verified by looking at the ipset on the nova hosts). Removing the two lines from the ipset restored all security groups. Expected result was to remove ip filtering on the single port. This sounds similar to: https://bugs.launchpad.net/neutron/+bug/1461054 but is marked fixed long ago. I've marked this as a security bug as a change to a single port can bypass other ports security groups. ** Tags added: security -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1793029 Title: adding 0.0.0.0/0 address pair to a port bypasses all other vm security groups Status in neutron: New Status in OpenStack Security Advisory: Won't Fix Status in OpenStack Security Notes: New Bug description: On an openstack-ansible / newton setup with linuxbridge, a customer ran: neutron port-update $port-uuid --allowed-address-pairs type=dict list=true ip_address=0.0.0.0/0 to bypass the ip source restriction (pfsense router and had to route packets). The impact of running the above, was an allow all rule was added to all ports in the network, bypassing all security groups. The iptables rule:  905K 55M RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 match-set NIPv44046d62c-59c8-4fd0-a547- src used on all ports, now triggers as: 0.0.0.0/1 128.0.0.0/1 was added to the ipset NIPv44046d62c-59c8-4fd0-a547 (verified by looking at the ipset on the nova hosts). Removing the two lines from the ipset restored all security groups. Expected result was to remove ip filtering on the single port. This sounds similar to: https://bugs.launchpad.net/neutron/+bug/1461054 but is marked fixed long ago. I've marked this as a security bug as a change to a single port can bypass other ports security groups. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1793029/+subscriptions From fungi at yuggoth.org Thu Sep 20 16:35:28 2018 From: fungi at yuggoth.org (Jeremy Stanley) Date: Thu, 20 Sep 2018 16:35:28 -0000 Subject: [Openstack-security] [Bug 1791973] Re: User cannot list their own trusts References: <153668011462.24906.14477100206448263755.malonedeb@gac.canonical.com> Message-ID: <153746132856.30636.15153372917616306276.malone@chaenomeles.canonical.com> Based on consensus above, I've switched the bug to public and triaged it as a class D report, tagging it as a potential hardening opportunity or security-related improvement. ** Information type changed from Private Security to Public ** Description changed: - This issue is being treated as a potential security risk under embargo. - Please do not make any public mention of embargoed (private) security - vulnerabilities before their coordinated publication by the OpenStack - Vulnerability Management Team in the form of an official OpenStack - Security Advisory. This includes discussion of the bug or associated - fixes in public forums such as mailing lists, code review systems and - bug trackers. Please also avoid private disclosure to other individuals - not already approved for access to this information, and provide this - same reminder to those who are made aware of the issue prior to - publication. All discussion should remain confined to this private bug - report, and any proposed fixes should be added to the bug as - attachments. - Heat and Admin users both commonly create trusts for other users. But any application is capable of doing this, as it requires only a scoped token to create a trust, which users pass around regularly. If I am concerned that some other application (or unauthorized user) has created a trust with me as the trustor, I need to be able to confirm this. If I cannot perform "trust list" and see the set of trusts that have me as a trustor, I am not able to clear out spurious ones. Thus, I would not be aware of any trusts set up in my name. ** Changed in: ossa Status: New => Won't Fix ** Tags added: security -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1791973 Title: User cannot list their own trusts Status in OpenStack Identity (keystone): New Status in OpenStack Security Advisory: Won't Fix Bug description: Heat and Admin users both commonly create trusts for other users. But any application is capable of doing this, as it requires only a scoped token to create a trust, which users pass around regularly. If I am concerned that some other application (or unauthorized user) has created a trust with me as the trustor, I need to be able to confirm this. If I cannot perform "trust list" and see the set of trusts that have me as a trustor, I am not able to clear out spurious ones. Thus, I would not be aware of any trusts set up in my name. To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1791973/+subscriptions From 1761054 at bugs.launchpad.net Mon Sep 24 14:47:11 2018 From: 1761054 at bugs.launchpad.net (OpenStack Infra) Date: Mon, 24 Sep 2018 14:47:11 -0000 Subject: [Openstack-security] [Bug 1761054] Fix included in openstack/nova 16.1.5 References: <152281951978.29145.13868060795008433631.malonedeb@chaenomeles.canonical.com> Message-ID: <153780043173.30787.8876680059750034597.malone@chaenomeles.canonical.com> This issue was fixed in the openstack/nova 16.1.5 release. -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1761054 Title: nova log expose password when swapvolume Status in OpenStack Compute (nova): Fix Released Status in OpenStack Security Advisory: Won't Fix Bug description: http://logs.openstack.org/50/557150/6/check/tempest- full/1f9c9f2/controller/logs/screen-n-cpu.txt.gz#_Mar_30_08_37_13_371323 u'auth_password': u'8KigD3KKykJkJixs', u'auth_username': u'6m4wAHCZVqFfTQaF4eZu', To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1761054/+subscriptions From 1739646 at bugs.launchpad.net Mon Sep 24 14:47:03 2018 From: 1739646 at bugs.launchpad.net (OpenStack Infra) Date: Mon, 24 Sep 2018 14:47:03 -0000 Subject: [Openstack-security] [Bug 1739646] Fix included in openstack/nova 16.1.5 References: <151387701431.11131.16472149525534236557.malonedeb@gac.canonical.com> Message-ID: <153780042368.30215.14562454652245139337.malone@soybean.canonical.com> This issue was fixed in the openstack/nova 16.1.5 release. -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1739646 Title: Instance type with disk set to 0 can cause DoS Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) ocata series: Fix Committed Status in OpenStack Compute (nova) pike series: Fix Committed Status in OpenStack Compute (nova) queens series: Fix Committed Status in OpenStack Security Advisory: Won't Fix Status in OpenStack Security Notes: New Bug description: In OpenStack at the moment, there is the ability to create instance types with disk size 0. The API documentation states the following: "The size of the root disk that will be created in GiB. If 0 the root disk will be set to exactly the size of the image used to deploy the instance. However, in this case filter scheduler cannot select the compute host based on the virtual image size. Therefore, 0 should only be used for volume booted instances or for testing purposes." In a cloud environment where a deployer wants to offer boot-from- volume instances, those instance types will be there. However, this means that a user can upload an image of 4TB and boot small instances where each one will have 4TB of storage, potentially exhausting the disks local storage (or Ceph cluster if using Ceph for ephemeral storage). I'm not sure if this is a security issue or it should be published as an advisory, but I believe there should be an option to disable the feature of booting an instance with the exact size of the image used so deployers have the ability/choice to provide boot-from-volume instance types. I can confirm this in our environment that if a customer creates an instance with 200GB of ephemeral disk space, they can take an image of it, then create an instance with that image on an instance type that has no ephemeral disk space and get 200GB of disk. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1739646/+subscriptions From 1739646 at bugs.launchpad.net Mon Sep 24 14:47:39 2018 From: 1739646 at bugs.launchpad.net (OpenStack Infra) Date: Mon, 24 Sep 2018 14:47:39 -0000 Subject: [Openstack-security] [Bug 1739646] Fix included in openstack/nova 17.0.6 References: <151387701431.11131.16472149525534236557.malonedeb@gac.canonical.com> Message-ID: <153780045930.1958.16474317304445457825.malone@wampee.canonical.com> This issue was fixed in the openstack/nova 17.0.6 release. -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1739646 Title: Instance type with disk set to 0 can cause DoS Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) ocata series: Fix Committed Status in OpenStack Compute (nova) pike series: Fix Committed Status in OpenStack Compute (nova) queens series: Fix Committed Status in OpenStack Security Advisory: Won't Fix Status in OpenStack Security Notes: New Bug description: In OpenStack at the moment, there is the ability to create instance types with disk size 0. The API documentation states the following: "The size of the root disk that will be created in GiB. If 0 the root disk will be set to exactly the size of the image used to deploy the instance. However, in this case filter scheduler cannot select the compute host based on the virtual image size. Therefore, 0 should only be used for volume booted instances or for testing purposes." In a cloud environment where a deployer wants to offer boot-from- volume instances, those instance types will be there. However, this means that a user can upload an image of 4TB and boot small instances where each one will have 4TB of storage, potentially exhausting the disks local storage (or Ceph cluster if using Ceph for ephemeral storage). I'm not sure if this is a security issue or it should be published as an advisory, but I believe there should be an option to disable the feature of booting an instance with the exact size of the image used so deployers have the ability/choice to provide boot-from-volume instance types. I can confirm this in our environment that if a customer creates an instance with 200GB of ephemeral disk space, they can take an image of it, then create an instance with that image on an instance type that has no ephemeral disk space and get 200GB of disk. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1739646/+subscriptions From 1777460 at bugs.launchpad.net Mon Sep 24 14:47:38 2018 From: 1777460 at bugs.launchpad.net (OpenStack Infra) Date: Mon, 24 Sep 2018 14:47:38 -0000 Subject: [Openstack-security] [Bug 1777460] Fix included in openstack/nova 17.0.6 References: <152933134329.1989.7672538528184873175.malonedeb@chaenomeles.canonical.com> Message-ID: <153780045820.30903.14114757145858108635.malone@chaenomeles.canonical.com> This issue was fixed in the openstack/nova 17.0.6 release. -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1777460 Title: Whitelist two more SSBD-related CPU flags for AMD ('amd-ssbd', 'amd- no-ssb') Status in OpenStack Compute (nova): Won't Fix Status in OpenStack Compute (nova) ocata series: Confirmed Status in OpenStack Compute (nova) pike series: Fix Committed Status in OpenStack Compute (nova) queens series: Fix Committed Bug description: In addition to the existing 'virt-ssbd', future AMD CPUs will have another (architectural) way to deal with SSBD (Speculative Store Bypass Disable), via the CPU flag: 'amd-ssbd'. Furthermore, future AMD CPUs also will expose a mechanism to tell the guest that the "Speculative Store Bypass Disable" (SSBD) is not needed and that the CPU is all good. This is via the CPU flag: 'amd-no-ssb' In summary, two new flags are[1][2]:     amd-ssbd     amd-no-ssb It is recommended to add the above two flags to the whitelist of Nova's `cpu_model_extra_flags` config attribute -- for stable branches (Queens, Pike and Ocata). For Rocky and above release, no such white-listing is required, since we allow free-form CPU flags[3].     * * * Additional notes (from the QEMU mailing list thread[4]) related to performance and live migration:   - tl;dr: On an AMD Compute node, a guest should be presented with     'amd-ssbd', if available, in preference to 'virt-ssbd'.     Details: Tom Lendacky from AMD writes[4] -- "The idea behind     'virt-ssbd' was to provide an architectural method for a guest to do     SSBD when 'amd-ssbd' isn't present. The 'amd-ssbd' feature will use     SPEC_CTRL which is intended to not be intercepted and will be fast.     The use of 'virt-ssbd' will always be intercepted and therefore will     not be as fast. So a guest should be presented with 'amd-ssbd', if     available, in preference to 'virt-ssbd'."   - It is safe to use 'amd-ssbd' (it is an architectural method for guest to do SSBD) in a guest which can be live migrated between different generations/families of AMD CPU. [1] libvirt patch:     https://www.redhat.com/archives/libvir-list/2018-June/msg01111.html [2] QEMU patch:     https://lists.nongnu.org/archive/html/qemu-devel/2018-06/msg00222.html [3] http://git.openstack.org/cgit/openstack/nova/commit/?id=cc27a20 --     libvirt: Lift the restriction of choices for `cpu_model_extra_flags` [4] https://lists.nongnu.org/archive/html/qemu-devel/2018-06/msg02301.html To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1777460/+subscriptions From 1777460 at bugs.launchpad.net Mon Sep 24 14:47:01 2018 From: 1777460 at bugs.launchpad.net (OpenStack Infra) Date: Mon, 24 Sep 2018 14:47:01 -0000 Subject: [Openstack-security] [Bug 1777460] Fix included in openstack/nova 16.1.5 References: <152933134329.1989.7672538528184873175.malonedeb@chaenomeles.canonical.com> Message-ID: <153780042150.24297.13693011622880469289.malone@gac.canonical.com> This issue was fixed in the openstack/nova 16.1.5 release. -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1777460 Title: Whitelist two more SSBD-related CPU flags for AMD ('amd-ssbd', 'amd- no-ssb') Status in OpenStack Compute (nova): Won't Fix Status in OpenStack Compute (nova) ocata series: Confirmed Status in OpenStack Compute (nova) pike series: Fix Committed Status in OpenStack Compute (nova) queens series: Fix Committed Bug description: In addition to the existing 'virt-ssbd', future AMD CPUs will have another (architectural) way to deal with SSBD (Speculative Store Bypass Disable), via the CPU flag: 'amd-ssbd'. Furthermore, future AMD CPUs also will expose a mechanism to tell the guest that the "Speculative Store Bypass Disable" (SSBD) is not needed and that the CPU is all good. This is via the CPU flag: 'amd-no-ssb' In summary, two new flags are[1][2]:     amd-ssbd     amd-no-ssb It is recommended to add the above two flags to the whitelist of Nova's `cpu_model_extra_flags` config attribute -- for stable branches (Queens, Pike and Ocata). For Rocky and above release, no such white-listing is required, since we allow free-form CPU flags[3].     * * * Additional notes (from the QEMU mailing list thread[4]) related to performance and live migration:   - tl;dr: On an AMD Compute node, a guest should be presented with     'amd-ssbd', if available, in preference to 'virt-ssbd'.     Details: Tom Lendacky from AMD writes[4] -- "The idea behind     'virt-ssbd' was to provide an architectural method for a guest to do     SSBD when 'amd-ssbd' isn't present. The 'amd-ssbd' feature will use     SPEC_CTRL which is intended to not be intercepted and will be fast.     The use of 'virt-ssbd' will always be intercepted and therefore will     not be as fast. So a guest should be presented with 'amd-ssbd', if     available, in preference to 'virt-ssbd'."   - It is safe to use 'amd-ssbd' (it is an architectural method for guest to do SSBD) in a guest which can be live migrated between different generations/families of AMD CPU. [1] libvirt patch:     https://www.redhat.com/archives/libvir-list/2018-June/msg01111.html [2] QEMU patch:     https://lists.nongnu.org/archive/html/qemu-devel/2018-06/msg00222.html [3] http://git.openstack.org/cgit/openstack/nova/commit/?id=cc27a20 --     libvirt: Lift the restriction of choices for `cpu_model_extra_flags` [4] https://lists.nongnu.org/archive/html/qemu-devel/2018-06/msg02301.html To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1777460/+subscriptions From 1739646 at bugs.launchpad.net Wed Sep 26 10:43:53 2018 From: 1739646 at bugs.launchpad.net (OpenStack Infra) Date: Wed, 26 Sep 2018 10:43:53 -0000 Subject: [Openstack-security] [Bug 1739646] Fix included in openstack/nova 15.1.4 References: <151387701431.11131.16472149525534236557.malonedeb@gac.canonical.com> Message-ID: <153795863354.1761.12132682749145856620.malone@wampee.canonical.com> This issue was fixed in the openstack/nova 15.1.4 release. -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1739646 Title: Instance type with disk set to 0 can cause DoS Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) ocata series: Fix Committed Status in OpenStack Compute (nova) pike series: Fix Committed Status in OpenStack Compute (nova) queens series: Fix Committed Status in OpenStack Security Advisory: Won't Fix Status in OpenStack Security Notes: New Bug description: In OpenStack at the moment, there is the ability to create instance types with disk size 0. The API documentation states the following: "The size of the root disk that will be created in GiB. If 0 the root disk will be set to exactly the size of the image used to deploy the instance. However, in this case filter scheduler cannot select the compute host based on the virtual image size. Therefore, 0 should only be used for volume booted instances or for testing purposes." In a cloud environment where a deployer wants to offer boot-from- volume instances, those instance types will be there. However, this means that a user can upload an image of 4TB and boot small instances where each one will have 4TB of storage, potentially exhausting the disks local storage (or Ceph cluster if using Ceph for ephemeral storage). I'm not sure if this is a security issue or it should be published as an advisory, but I believe there should be an option to disable the feature of booting an instance with the exact size of the image used so deployers have the ability/choice to provide boot-from-volume instance types. I can confirm this in our environment that if a customer creates an instance with 200GB of ephemeral disk space, they can take an image of it, then create an instance with that image on an instance type that has no ephemeral disk space and get 200GB of disk. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1739646/+subscriptions