From 1819957 at bugs.launchpad.net Thu May 2 21:31:44 2019 From: 1819957 at bugs.launchpad.net (OpenStack Infra) Date: Thu, 02 May 2019 21:31:44 -0000 Subject: [Openstack-security] [Bug 1819957] Re: Caching with stale data when a server disconnects due to network partition and reconnects References: <155250258380.27992.5839797432076968036.malonedeb@wampee.canonical.com> Message-ID: <155683270475.32057.14596778356754369426.malone@wampee.canonical.com> Reviewed: https://review.opendev.org/656419 Committed: https://git.openstack.org/cgit/openstack/oslo.cache/commit/?id=aeb95779c7bea4117ccb69b17eab31d452397964 Submitter: Zuul Branch: stable/stein commit aeb95779c7bea4117ccb69b17eab31d452397964 Author: Morgan Fainberg Date: Fri Mar 22 12:35:16 2019 -0700 Pass `flush_on_reconnect` to memcache pooled backend If a memcache server disappears and then reconnects when multiple memcache servers are used (specific to the python-memcached based backends) it is possible that the server will contain stale data. The default is now to supply the ``flush_on_reconnect`` optional argument to the backend. This means that when the service connects to a memcache server, it will flush all cached data in the server. The pooled backend is more likely to run into issues with this as it does not explicitly use a thread.local for the client. The non-pooled backend was not touched, it is not the recommended production use-case. See the help from python-memcached: @param flush_on_reconnect: optional flag which prevents a scenario that can cause stale data to be read: If there's more than one memcached server and the connection to one is interrupted, keys that mapped to that server will get reassigned to another. If the first server comes back, those keys will map to it again. If it still has its data, get()s can read stale data that was overwritten on another server. This flag is off by default for backwards compatibility. Change-Id: I3e335261f749ad065e8abe972f4ac476d334e6b3 closes-bug: #1819957 (cherry picked from commit 1192f185a5fd2fa6177655f157146488a3de81d1) Signed-off-by: Matthew Thode ** Tags added: in-stable-stein -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1819957 Title: Caching with stale data when a server disconnects due to network partition and reconnects Status in OpenStack Identity (keystone): Invalid Status in keystonemiddleware: Triaged Status in oslo.cache: Fix Released Status in OpenStack Security Advisory: Won't Fix Bug description: The flush_on_reconnect optional flag is not used. This can cause stale data to be utilized from a cache server that disconnected due to a network partition. This has security concerns as follows: 1* Password changes/user changes may be reverted for the cache TTL 1a* User may get locked out if PCI-DSS is on and the password change happens during the network partition. 2* Grant changes may be reverted for the cache TTL 3* Resources (all types) may become "undeleted" for the cache TTL 4* Tokens (KSM) may become valid again during the cache TTL As noted in the python-memcached library: @param flush_on_reconnect: optional flag which prevents a scenario that can cause stale data to be read: If there's more than one memcached server and the connection to one is interrupted, keys that mapped to that server will get reassigned to another. If the first server comes back, those keys will map to it again. If it still has its data, get()s can read stale data that was overwritten on another server. This flag is off by default for backwards compatibility. The solution is to explicitly pass flush_on_reconnect as an optional argument. A concern with this model is that the memcached servers may be utilized by other tooling and may lose cache state (in the case the oslo.cache connection is the only thing affected by the network partitioning). This similarly needs to be addressed in pymemcache when it is utilized in lieu of python-memcached. To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1819957/+subscriptions From 1819957 at bugs.launchpad.net Thu May 2 23:35:30 2019 From: 1819957 at bugs.launchpad.net (OpenStack Infra) Date: Thu, 02 May 2019 23:35:30 -0000 Subject: [Openstack-security] [Bug 1819957] Re: Caching with stale data when a server disconnects due to network partition and reconnects References: <155250258380.27992.5839797432076968036.malonedeb@wampee.canonical.com> Message-ID: <155684013018.31990.6614869059112264258.malone@chaenomeles.canonical.com> Reviewed: https://review.opendev.org/656420 Committed: https://git.openstack.org/cgit/openstack/oslo.cache/commit/?id=1d6d08f4911edb50debab4639e6a2825260b9e09 Submitter: Zuul Branch: stable/rocky commit 1d6d08f4911edb50debab4639e6a2825260b9e09 Author: Morgan Fainberg Date: Fri Mar 22 12:35:16 2019 -0700 Pass `flush_on_reconnect` to memcache pooled backend If a memcache server disappears and then reconnects when multiple memcache servers are used (specific to the python-memcached based backends) it is possible that the server will contain stale data. The default is now to supply the ``flush_on_reconnect`` optional argument to the backend. This means that when the service connects to a memcache server, it will flush all cached data in the server. The pooled backend is more likely to run into issues with this as it does not explicitly use a thread.local for the client. The non-pooled backend was not touched, it is not the recommended production use-case. See the help from python-memcached: @param flush_on_reconnect: optional flag which prevents a scenario that can cause stale data to be read: If there's more than one memcached server and the connection to one is interrupted, keys that mapped to that server will get reassigned to another. If the first server comes back, those keys will map to it again. If it still has its data, get()s can read stale data that was overwritten on another server. This flag is off by default for backwards compatibility. Change-Id: I3e335261f749ad065e8abe972f4ac476d334e6b3 closes-bug: #1819957 (cherry picked from commit 1192f185a5fd2fa6177655f157146488a3de81d1) Signed-off-by: Matthew Thode ** Tags added: in-stable-rocky -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1819957 Title: Caching with stale data when a server disconnects due to network partition and reconnects Status in OpenStack Identity (keystone): Invalid Status in keystonemiddleware: Triaged Status in oslo.cache: Fix Released Status in OpenStack Security Advisory: Won't Fix Bug description: The flush_on_reconnect optional flag is not used. This can cause stale data to be utilized from a cache server that disconnected due to a network partition. This has security concerns as follows: 1* Password changes/user changes may be reverted for the cache TTL 1a* User may get locked out if PCI-DSS is on and the password change happens during the network partition. 2* Grant changes may be reverted for the cache TTL 3* Resources (all types) may become "undeleted" for the cache TTL 4* Tokens (KSM) may become valid again during the cache TTL As noted in the python-memcached library: @param flush_on_reconnect: optional flag which prevents a scenario that can cause stale data to be read: If there's more than one memcached server and the connection to one is interrupted, keys that mapped to that server will get reassigned to another. If the first server comes back, those keys will map to it again. If it still has its data, get()s can read stale data that was overwritten on another server. This flag is off by default for backwards compatibility. The solution is to explicitly pass flush_on_reconnect as an optional argument. A concern with this model is that the memcached servers may be utilized by other tooling and may lose cache state (in the case the oslo.cache connection is the only thing affected by the network partitioning). This similarly needs to be addressed in pymemcache when it is utilized in lieu of python-memcached. To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1819957/+subscriptions From 1819957 at bugs.launchpad.net Thu May 16 06:21:51 2019 From: 1819957 at bugs.launchpad.net (OpenStack Infra) Date: Thu, 16 May 2019 06:21:51 -0000 Subject: [Openstack-security] [Bug 1819957] Fix included in openstack/oslo.cache 1.33.3 References: <155250258380.27992.5839797432076968036.malonedeb@wampee.canonical.com> Message-ID: <155798771124.14985.9277625995867494092.malone@gac.canonical.com> This issue was fixed in the openstack/oslo.cache 1.33.3 release. -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1819957 Title: Caching with stale data when a server disconnects due to network partition and reconnects Status in OpenStack Identity (keystone): Invalid Status in keystonemiddleware: Triaged Status in oslo.cache: Fix Released Status in OpenStack Security Advisory: Won't Fix Bug description: The flush_on_reconnect optional flag is not used. This can cause stale data to be utilized from a cache server that disconnected due to a network partition. This has security concerns as follows: 1* Password changes/user changes may be reverted for the cache TTL 1a* User may get locked out if PCI-DSS is on and the password change happens during the network partition. 2* Grant changes may be reverted for the cache TTL 3* Resources (all types) may become "undeleted" for the cache TTL 4* Tokens (KSM) may become valid again during the cache TTL As noted in the python-memcached library: @param flush_on_reconnect: optional flag which prevents a scenario that can cause stale data to be read: If there's more than one memcached server and the connection to one is interrupted, keys that mapped to that server will get reassigned to another. If the first server comes back, those keys will map to it again. If it still has its data, get()s can read stale data that was overwritten on another server. This flag is off by default for backwards compatibility. The solution is to explicitly pass flush_on_reconnect as an optional argument. A concern with this model is that the memcached servers may be utilized by other tooling and may lose cache state (in the case the oslo.cache connection is the only thing affected by the network partitioning). This similarly needs to be addressed in pymemcache when it is utilized in lieu of python-memcached. To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1819957/+subscriptions From 1819957 at bugs.launchpad.net Thu May 16 06:34:02 2019 From: 1819957 at bugs.launchpad.net (OpenStack Infra) Date: Thu, 16 May 2019 06:34:02 -0000 Subject: [Openstack-security] [Bug 1819957] Fix included in openstack/oslo.cache 1.30.4 References: <155250258380.27992.5839797432076968036.malonedeb@wampee.canonical.com> Message-ID: <155798844262.15065.8701527830614195837.malone@gac.canonical.com> This issue was fixed in the openstack/oslo.cache 1.30.4 release. -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1819957 Title: Caching with stale data when a server disconnects due to network partition and reconnects Status in OpenStack Identity (keystone): Invalid Status in keystonemiddleware: Triaged Status in oslo.cache: Fix Released Status in OpenStack Security Advisory: Won't Fix Bug description: The flush_on_reconnect optional flag is not used. This can cause stale data to be utilized from a cache server that disconnected due to a network partition. This has security concerns as follows: 1* Password changes/user changes may be reverted for the cache TTL 1a* User may get locked out if PCI-DSS is on and the password change happens during the network partition. 2* Grant changes may be reverted for the cache TTL 3* Resources (all types) may become "undeleted" for the cache TTL 4* Tokens (KSM) may become valid again during the cache TTL As noted in the python-memcached library: @param flush_on_reconnect: optional flag which prevents a scenario that can cause stale data to be read: If there's more than one memcached server and the connection to one is interrupted, keys that mapped to that server will get reassigned to another. If the first server comes back, those keys will map to it again. If it still has its data, get()s can read stale data that was overwritten on another server. This flag is off by default for backwards compatibility. The solution is to explicitly pass flush_on_reconnect as an optional argument. A concern with this model is that the memcached servers may be utilized by other tooling and may lose cache state (in the case the oslo.cache connection is the only thing affected by the network partitioning). This similarly needs to be addressed in pymemcache when it is utilized in lieu of python-memcached. To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1819957/+subscriptions From fungi at yuggoth.org Mon May 20 15:39:23 2019 From: fungi at yuggoth.org (Jeremy Stanley) Date: Mon, 20 May 2019 15:39:23 -0000 Subject: [Openstack-security] [Bug 1824248] Re: Security Group filtering hides rules from user References: <155493822208.21486.11348171578680982627.malonedeb@soybean.canonical.com> Message-ID: <155836676326.21696.15679801678571703098.malone@chaenomeles.canonical.com> Seems there's support for switching this to public and no real disagreement with my suggestion a month ago to treat this as a security hardening opportunity rather than a vulnerability, so I'll triage it publicly as such now and we can continue to debate/fix without any further embargo. Treating as a class D report per https://security.openstack.org/vmt- process.html#incident-report-taxonomy . ** Description changed: - This issue is being treated as a potential security risk under embargo. - Please do not make any public mention of embargoed (private) security - vulnerabilities before their coordinated publication by the OpenStack - Vulnerability Management Team in the form of an official OpenStack - Security Advisory. This includes discussion of the bug or associated - fixes in public forums such as mailing lists, code review systems and - bug trackers. Please also avoid private disclosure to other individuals - not already approved for access to this information, and provide this - same reminder to those who are made aware of the issue prior to - publication. All discussion should remain confined to this private bug - report, and any proposed fixes should be added to the bug as - attachments. - Manage Rules part of the GUI hides the rules currently visible in the Launch Instance modal window. It allows a malicious admin to add backdoor access rules that might be later added to VMs without the knowledge of owner of those VMs. When sending GET request as below, it responds only with the rules that are created by user and this happens when using Manage Rules part of the GUI: On the other hand when using GET request as below, it responds with all SG and it includes all rules, and there is no filtering and this is used in Launch Instance modal window: Here is example of rules display in Manage Rules part of GUI: > /opt/stack/horizon/openstack_dashboard/dashboards/project/security_groups/views.py(50)_get_data() -> return api.neutron.security_group_get(self.request, sg_id) (Pdb) l  45 @memoized.memoized_method  46 def _get_data(self):  47 sg_id = filters.get_int_or_uuid(self.kwargs['security_group_id'])  48 try:  49 from remote_pdb import RemotePdb; RemotePdb('127.0.0.1', 444).set_trace()  50 -> return api.neutron.security_group_get(self.request, sg_id)  51 except Exception:  52 redirect = reverse('horizon:project:security_groups:index')  53 exceptions.handle(self.request,  54 _('Unable to retrieve security group.'),  55 redirect=redirect) (Pdb) p api.neutron.security_group_get(self.request, sg_id) , , , ]}> (Pdb) (Pdb) p self.request As you might have noticed there are no ports access 44 and 22 (SSH) And from the Launch Instance Modal Window, as well as CLI we can see that there are two more rules that are invisible for user, port 44 and 22 (SSH) as displayed below: > /opt/stack/horizon/openstack_dashboard/api/rest/network.py(47)get() -> return {'items': [sg.to_dict() for sg in security_groups]} (Pdb) l  42 """  43  44 security_groups = api.neutron.security_group_list(request)  45 from remote_pdb import RemotePdb; RemotePdb('127.0.0.1', 444).set_trace()  46  47 -> return {'items': [sg.to_dict() for sg in security_groups]}  48  49  50 @urls.register  51 class FloatingIP(generic.View):  52 """API for a single floating IP address.""" (Pdb) p security_groups [, , , , , ]}>] (Pdb) (Pdb) p request Thank you, Robin ** Information type changed from Private Security to Public ** Changed in: ossa Status: Incomplete => Won't Fix ** Tags added: security -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1824248 Title: Security Group filtering hides rules from user Status in neutron: New Status in OpenStack Security Advisory: Won't Fix Bug description: Manage Rules part of the GUI hides the rules currently visible in the Launch Instance modal window. It allows a malicious admin to add backdoor access rules that might be later added to VMs without the knowledge of owner of those VMs. When sending GET request as below, it responds only with the rules that are created by user and this happens when using Manage Rules part of the GUI: On the other hand when using GET request as below, it responds with all SG and it includes all rules, and there is no filtering and this is used in Launch Instance modal window: Here is example of rules display in Manage Rules part of GUI: > /opt/stack/horizon/openstack_dashboard/dashboards/project/security_groups/views.py(50)_get_data() -> return api.neutron.security_group_get(self.request, sg_id) (Pdb) l  45 @memoized.memoized_method  46 def _get_data(self):  47 sg_id = filters.get_int_or_uuid(self.kwargs['security_group_id'])  48 try:  49 from remote_pdb import RemotePdb; RemotePdb('127.0.0.1', 444).set_trace()  50 -> return api.neutron.security_group_get(self.request, sg_id)  51 except Exception:  52 redirect = reverse('horizon:project:security_groups:index')  53 exceptions.handle(self.request,  54 _('Unable to retrieve security group.'),  55 redirect=redirect) (Pdb) p api.neutron.security_group_get(self.request, sg_id) , , , ]}> (Pdb) (Pdb) p self.request As you might have noticed there are no ports access 44 and 22 (SSH) And from the Launch Instance Modal Window, as well as CLI we can see that there are two more rules that are invisible for user, port 44 and 22 (SSH) as displayed below: > /opt/stack/horizon/openstack_dashboard/api/rest/network.py(47)get() -> return {'items': [sg.to_dict() for sg in security_groups]} (Pdb) l  42 """  43  44 security_groups = api.neutron.security_group_list(request)  45 from remote_pdb import RemotePdb; RemotePdb('127.0.0.1', 444).set_trace()  46  47 -> return {'items': [sg.to_dict() for sg in security_groups]}  48  49  50 @urls.register  51 class FloatingIP(generic.View):  52 """API for a single floating IP address.""" (Pdb) p security_groups [, , , , , ]}>] (Pdb) (Pdb) p request Thank you, Robin To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1824248/+subscriptions From 1824248 at bugs.launchpad.net Mon May 20 16:51:36 2019 From: 1824248 at bugs.launchpad.net (OpenStack Infra) Date: Mon, 20 May 2019 16:51:36 -0000 Subject: [Openstack-security] [Bug 1824248] Re: Security Group filtering hides rules from user References: <155493822208.21486.11348171578680982627.malonedeb@soybean.canonical.com> Message-ID: <155837109628.15551.17995500638307660054.malone@wampee.canonical.com> Fix proposed to branch: master Review: https://review.opendev.org/660174 ** Changed in: neutron Status: New => In Progress ** Changed in: neutron Assignee: (unassigned) => Slawek Kaplonski (slaweq) -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1824248 Title: Security Group filtering hides rules from user Status in neutron: In Progress Status in OpenStack Security Advisory: Won't Fix Bug description: Manage Rules part of the GUI hides the rules currently visible in the Launch Instance modal window. It allows a malicious admin to add backdoor access rules that might be later added to VMs without the knowledge of owner of those VMs. When sending GET request as below, it responds only with the rules that are created by user and this happens when using Manage Rules part of the GUI: On the other hand when using GET request as below, it responds with all SG and it includes all rules, and there is no filtering and this is used in Launch Instance modal window: Here is example of rules display in Manage Rules part of GUI: > /opt/stack/horizon/openstack_dashboard/dashboards/project/security_groups/views.py(50)_get_data() -> return api.neutron.security_group_get(self.request, sg_id) (Pdb) l  45 @memoized.memoized_method  46 def _get_data(self):  47 sg_id = filters.get_int_or_uuid(self.kwargs['security_group_id'])  48 try:  49 from remote_pdb import RemotePdb; RemotePdb('127.0.0.1', 444).set_trace()  50 -> return api.neutron.security_group_get(self.request, sg_id)  51 except Exception:  52 redirect = reverse('horizon:project:security_groups:index')  53 exceptions.handle(self.request,  54 _('Unable to retrieve security group.'),  55 redirect=redirect) (Pdb) p api.neutron.security_group_get(self.request, sg_id) , , , ]}> (Pdb) (Pdb) p self.request As you might have noticed there are no ports access 44 and 22 (SSH) And from the Launch Instance Modal Window, as well as CLI we can see that there are two more rules that are invisible for user, port 44 and 22 (SSH) as displayed below: > /opt/stack/horizon/openstack_dashboard/api/rest/network.py(47)get() -> return {'items': [sg.to_dict() for sg in security_groups]} (Pdb) l  42 """  43  44 security_groups = api.neutron.security_group_list(request)  45 from remote_pdb import RemotePdb; RemotePdb('127.0.0.1', 444).set_trace()  46  47 -> return {'items': [sg.to_dict() for sg in security_groups]}  48  49  50 @urls.register  51 class FloatingIP(generic.View):  52 """API for a single floating IP address.""" (Pdb) p security_groups [, , , , , ]}>] (Pdb) (Pdb) p request Thank you, Robin To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1824248/+subscriptions From 1824248 at bugs.launchpad.net Mon May 20 16:53:18 2019 From: 1824248 at bugs.launchpad.net (OpenStack Infra) Date: Mon, 20 May 2019 16:53:18 -0000 Subject: [Openstack-security] [Bug 1824248] Related fix proposed to neutron-tempest-plugin (master) References: <155493822208.21486.11348171578680982627.malonedeb@soybean.canonical.com> Message-ID: <155837119877.14865.7504568158792360026.malone@wampee.canonical.com> Related fix proposed to branch: master Review: https://review.opendev.org/660175 -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1824248 Title: Security Group filtering hides rules from user Status in neutron: In Progress Status in OpenStack Security Advisory: Won't Fix Bug description: Manage Rules part of the GUI hides the rules currently visible in the Launch Instance modal window. It allows a malicious admin to add backdoor access rules that might be later added to VMs without the knowledge of owner of those VMs. When sending GET request as below, it responds only with the rules that are created by user and this happens when using Manage Rules part of the GUI: On the other hand when using GET request as below, it responds with all SG and it includes all rules, and there is no filtering and this is used in Launch Instance modal window: Here is example of rules display in Manage Rules part of GUI: > /opt/stack/horizon/openstack_dashboard/dashboards/project/security_groups/views.py(50)_get_data() -> return api.neutron.security_group_get(self.request, sg_id) (Pdb) l  45 @memoized.memoized_method  46 def _get_data(self):  47 sg_id = filters.get_int_or_uuid(self.kwargs['security_group_id'])  48 try:  49 from remote_pdb import RemotePdb; RemotePdb('127.0.0.1', 444).set_trace()  50 -> return api.neutron.security_group_get(self.request, sg_id)  51 except Exception:  52 redirect = reverse('horizon:project:security_groups:index')  53 exceptions.handle(self.request,  54 _('Unable to retrieve security group.'),  55 redirect=redirect) (Pdb) p api.neutron.security_group_get(self.request, sg_id) , , , ]}> (Pdb) (Pdb) p self.request As you might have noticed there are no ports access 44 and 22 (SSH) And from the Launch Instance Modal Window, as well as CLI we can see that there are two more rules that are invisible for user, port 44 and 22 (SSH) as displayed below: > /opt/stack/horizon/openstack_dashboard/api/rest/network.py(47)get() -> return {'items': [sg.to_dict() for sg in security_groups]} (Pdb) l  42 """  43  44 security_groups = api.neutron.security_group_list(request)  45 from remote_pdb import RemotePdb; RemotePdb('127.0.0.1', 444).set_trace()  46  47 -> return {'items': [sg.to_dict() for sg in security_groups]}  48  49  50 @urls.register  51 class FloatingIP(generic.View):  52 """API for a single floating IP address.""" (Pdb) p security_groups [, , , , , ]}>] (Pdb) (Pdb) p request Thank you, Robin To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1824248/+subscriptions From 1824248 at bugs.launchpad.net Fri May 24 12:59:19 2019 From: 1824248 at bugs.launchpad.net (OpenStack Infra) Date: Fri, 24 May 2019 12:59:19 -0000 Subject: [Openstack-security] [Bug 1824248] Fix proposed to neutron (stable/stein) References: <155493822208.21486.11348171578680982627.malonedeb@soybean.canonical.com> Message-ID: <155870275909.25498.1785467297099955963.malone@gac.canonical.com> Fix proposed to branch: stable/stein Review: https://review.opendev.org/661281 -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1824248 Title: Security Group filtering hides rules from user Status in neutron: In Progress Status in OpenStack Security Advisory: Won't Fix Bug description: Manage Rules part of the GUI hides the rules currently visible in the Launch Instance modal window. It allows a malicious admin to add backdoor access rules that might be later added to VMs without the knowledge of owner of those VMs. When sending GET request as below, it responds only with the rules that are created by user and this happens when using Manage Rules part of the GUI: On the other hand when using GET request as below, it responds with all SG and it includes all rules, and there is no filtering and this is used in Launch Instance modal window: Here is example of rules display in Manage Rules part of GUI: > /opt/stack/horizon/openstack_dashboard/dashboards/project/security_groups/views.py(50)_get_data() -> return api.neutron.security_group_get(self.request, sg_id) (Pdb) l  45 @memoized.memoized_method  46 def _get_data(self):  47 sg_id = filters.get_int_or_uuid(self.kwargs['security_group_id'])  48 try:  49 from remote_pdb import RemotePdb; RemotePdb('127.0.0.1', 444).set_trace()  50 -> return api.neutron.security_group_get(self.request, sg_id)  51 except Exception:  52 redirect = reverse('horizon:project:security_groups:index')  53 exceptions.handle(self.request,  54 _('Unable to retrieve security group.'),  55 redirect=redirect) (Pdb) p api.neutron.security_group_get(self.request, sg_id) , , , ]}> (Pdb) (Pdb) p self.request As you might have noticed there are no ports access 44 and 22 (SSH) And from the Launch Instance Modal Window, as well as CLI we can see that there are two more rules that are invisible for user, port 44 and 22 (SSH) as displayed below: > /opt/stack/horizon/openstack_dashboard/api/rest/network.py(47)get() -> return {'items': [sg.to_dict() for sg in security_groups]} (Pdb) l  42 """  43  44 security_groups = api.neutron.security_group_list(request)  45 from remote_pdb import RemotePdb; RemotePdb('127.0.0.1', 444).set_trace()  46  47 -> return {'items': [sg.to_dict() for sg in security_groups]}  48  49  50 @urls.register  51 class FloatingIP(generic.View):  52 """API for a single floating IP address.""" (Pdb) p security_groups [, , , , , ]}>] (Pdb) (Pdb) p request Thank you, Robin To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1824248/+subscriptions From 1824248 at bugs.launchpad.net Fri May 24 13:14:59 2019 From: 1824248 at bugs.launchpad.net (OpenStack Infra) Date: Fri, 24 May 2019 13:14:59 -0000 Subject: [Openstack-security] [Bug 1824248] Fix proposed to neutron (stable/rocky) References: <155493822208.21486.11348171578680982627.malonedeb@soybean.canonical.com> Message-ID: <155870369917.12772.10497517077337848471.malone@soybean.canonical.com> Fix proposed to branch: stable/rocky Review: https://review.opendev.org/661283 -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1824248 Title: Security Group filtering hides rules from user Status in neutron: In Progress Status in OpenStack Security Advisory: Won't Fix Bug description: Manage Rules part of the GUI hides the rules currently visible in the Launch Instance modal window. It allows a malicious admin to add backdoor access rules that might be later added to VMs without the knowledge of owner of those VMs. When sending GET request as below, it responds only with the rules that are created by user and this happens when using Manage Rules part of the GUI: On the other hand when using GET request as below, it responds with all SG and it includes all rules, and there is no filtering and this is used in Launch Instance modal window: Here is example of rules display in Manage Rules part of GUI: > /opt/stack/horizon/openstack_dashboard/dashboards/project/security_groups/views.py(50)_get_data() -> return api.neutron.security_group_get(self.request, sg_id) (Pdb) l  45 @memoized.memoized_method  46 def _get_data(self):  47 sg_id = filters.get_int_or_uuid(self.kwargs['security_group_id'])  48 try:  49 from remote_pdb import RemotePdb; RemotePdb('127.0.0.1', 444).set_trace()  50 -> return api.neutron.security_group_get(self.request, sg_id)  51 except Exception:  52 redirect = reverse('horizon:project:security_groups:index')  53 exceptions.handle(self.request,  54 _('Unable to retrieve security group.'),  55 redirect=redirect) (Pdb) p api.neutron.security_group_get(self.request, sg_id) , , , ]}> (Pdb) (Pdb) p self.request As you might have noticed there are no ports access 44 and 22 (SSH) And from the Launch Instance Modal Window, as well as CLI we can see that there are two more rules that are invisible for user, port 44 and 22 (SSH) as displayed below: > /opt/stack/horizon/openstack_dashboard/api/rest/network.py(47)get() -> return {'items': [sg.to_dict() for sg in security_groups]} (Pdb) l  42 """  43  44 security_groups = api.neutron.security_group_list(request)  45 from remote_pdb import RemotePdb; RemotePdb('127.0.0.1', 444).set_trace()  46  47 -> return {'items': [sg.to_dict() for sg in security_groups]}  48  49  50 @urls.register  51 class FloatingIP(generic.View):  52 """API for a single floating IP address.""" (Pdb) p security_groups [, , , , , ]}>] (Pdb) (Pdb) p request Thank you, Robin To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1824248/+subscriptions From 1824248 at bugs.launchpad.net Fri May 24 13:17:07 2019 From: 1824248 at bugs.launchpad.net (OpenStack Infra) Date: Fri, 24 May 2019 13:17:07 -0000 Subject: [Openstack-security] [Bug 1824248] Fix proposed to neutron (stable/queens) References: <155493822208.21486.11348171578680982627.malonedeb@soybean.canonical.com> Message-ID: <155870382738.12691.15987266540875677168.malone@soybean.canonical.com> Fix proposed to branch: stable/queens Review: https://review.opendev.org/661284 -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1824248 Title: Security Group filtering hides rules from user Status in neutron: In Progress Status in OpenStack Security Advisory: Won't Fix Bug description: Manage Rules part of the GUI hides the rules currently visible in the Launch Instance modal window. It allows a malicious admin to add backdoor access rules that might be later added to VMs without the knowledge of owner of those VMs. When sending GET request as below, it responds only with the rules that are created by user and this happens when using Manage Rules part of the GUI: On the other hand when using GET request as below, it responds with all SG and it includes all rules, and there is no filtering and this is used in Launch Instance modal window: Here is example of rules display in Manage Rules part of GUI: > /opt/stack/horizon/openstack_dashboard/dashboards/project/security_groups/views.py(50)_get_data() -> return api.neutron.security_group_get(self.request, sg_id) (Pdb) l  45 @memoized.memoized_method  46 def _get_data(self):  47 sg_id = filters.get_int_or_uuid(self.kwargs['security_group_id'])  48 try:  49 from remote_pdb import RemotePdb; RemotePdb('127.0.0.1', 444).set_trace()  50 -> return api.neutron.security_group_get(self.request, sg_id)  51 except Exception:  52 redirect = reverse('horizon:project:security_groups:index')  53 exceptions.handle(self.request,  54 _('Unable to retrieve security group.'),  55 redirect=redirect) (Pdb) p api.neutron.security_group_get(self.request, sg_id) , , , ]}> (Pdb) (Pdb) p self.request As you might have noticed there are no ports access 44 and 22 (SSH) And from the Launch Instance Modal Window, as well as CLI we can see that there are two more rules that are invisible for user, port 44 and 22 (SSH) as displayed below: > /opt/stack/horizon/openstack_dashboard/api/rest/network.py(47)get() -> return {'items': [sg.to_dict() for sg in security_groups]} (Pdb) l  42 """  43  44 security_groups = api.neutron.security_group_list(request)  45 from remote_pdb import RemotePdb; RemotePdb('127.0.0.1', 444).set_trace()  46  47 -> return {'items': [sg.to_dict() for sg in security_groups]}  48  49  50 @urls.register  51 class FloatingIP(generic.View):  52 """API for a single floating IP address.""" (Pdb) p security_groups [, , , , , ]}>] (Pdb) (Pdb) p request Thank you, Robin To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1824248/+subscriptions From 1824248 at bugs.launchpad.net Sat May 25 01:03:16 2019 From: 1824248 at bugs.launchpad.net (OpenStack Infra) Date: Sat, 25 May 2019 01:03:16 -0000 Subject: [Openstack-security] [Bug 1824248] Re: Security Group filtering hides rules from user References: <155493822208.21486.11348171578680982627.malonedeb@soybean.canonical.com> Message-ID: <155874619669.12407.14756639734514867354.malone@soybean.canonical.com> Reviewed: https://review.opendev.org/660174 Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=1920a37a94b7a9589dcf83f6ff0765068560dbf8 Submitter: Zuul Branch: master commit 1920a37a94b7a9589dcf83f6ff0765068560dbf8 Author: Slawek Kaplonski Date: Mon May 20 18:47:18 2019 +0200 Show all SG rules belong to SG in group's details If security group contains rule(s) which were created by different user (admin), owner of this security group should see such rules even if those rules don't belong to him. This patch changes to use admin_context to get security group rules in get_security_group() method to achieve that. Test to cover such case is added in neutron-tempest-plugin repo. Change-Id: I890c81bb6eabc5caa620ed4fcc4dc88ebfa6e1b0 Closes-Bug: #1824248 ** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1824248 Title: Security Group filtering hides rules from user Status in neutron: Fix Released Status in OpenStack Security Advisory: Won't Fix Bug description: Manage Rules part of the GUI hides the rules currently visible in the Launch Instance modal window. It allows a malicious admin to add backdoor access rules that might be later added to VMs without the knowledge of owner of those VMs. When sending GET request as below, it responds only with the rules that are created by user and this happens when using Manage Rules part of the GUI: On the other hand when using GET request as below, it responds with all SG and it includes all rules, and there is no filtering and this is used in Launch Instance modal window: Here is example of rules display in Manage Rules part of GUI: > /opt/stack/horizon/openstack_dashboard/dashboards/project/security_groups/views.py(50)_get_data() -> return api.neutron.security_group_get(self.request, sg_id) (Pdb) l  45 @memoized.memoized_method  46 def _get_data(self):  47 sg_id = filters.get_int_or_uuid(self.kwargs['security_group_id'])  48 try:  49 from remote_pdb import RemotePdb; RemotePdb('127.0.0.1', 444).set_trace()  50 -> return api.neutron.security_group_get(self.request, sg_id)  51 except Exception:  52 redirect = reverse('horizon:project:security_groups:index')  53 exceptions.handle(self.request,  54 _('Unable to retrieve security group.'),  55 redirect=redirect) (Pdb) p api.neutron.security_group_get(self.request, sg_id) , , , ]}> (Pdb) (Pdb) p self.request As you might have noticed there are no ports access 44 and 22 (SSH) And from the Launch Instance Modal Window, as well as CLI we can see that there are two more rules that are invisible for user, port 44 and 22 (SSH) as displayed below: > /opt/stack/horizon/openstack_dashboard/api/rest/network.py(47)get() -> return {'items': [sg.to_dict() for sg in security_groups]} (Pdb) l  42 """  43  44 security_groups = api.neutron.security_group_list(request)  45 from remote_pdb import RemotePdb; RemotePdb('127.0.0.1', 444).set_trace()  46  47 -> return {'items': [sg.to_dict() for sg in security_groups]}  48  49  50 @urls.register  51 class FloatingIP(generic.View):  52 """API for a single floating IP address.""" (Pdb) p security_groups [, , , , , ]}>] (Pdb) (Pdb) p request Thank you, Robin To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1824248/+subscriptions From 1793029 at bugs.launchpad.net Mon May 27 13:30:30 2019 From: 1793029 at bugs.launchpad.net (OpenStack Infra) Date: Mon, 27 May 2019 13:30:30 -0000 Subject: [Openstack-security] [Bug 1793029] Re: adding 0.0.0.0/0 address pair to a port bypasses all other vm security groups References: <153722407420.8513.15403635939639716317.malonedeb@chaenomeles.canonical.com> Message-ID: <155896383055.12481.12827837676009251191.malone@soybean.canonical.com> Fix proposed to branch: master Review: https://review.opendev.org/661594 ** Changed in: neutron Status: Triaged => In Progress ** Changed in: neutron Assignee: (unassigned) => Slawek Kaplonski (slaweq) -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1793029 Title: adding 0.0.0.0/0 address pair to a port bypasses all other vm security groups Status in neutron: In Progress Status in OpenStack Security Advisory: Won't Fix Status in OpenStack Security Notes: New Bug description: On an openstack-ansible / newton setup with linuxbridge, a customer ran: neutron port-update $port-uuid --allowed-address-pairs type=dict list=true ip_address=0.0.0.0/0 to bypass the ip source restriction (pfsense router and had to route packets). The impact of running the above, was an allow all rule was added to all ports in the network, bypassing all security groups. The iptables rule:  905K 55M RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 match-set NIPv44046d62c-59c8-4fd0-a547- src used on all ports, now triggers as: 0.0.0.0/1 128.0.0.0/1 was added to the ipset NIPv44046d62c-59c8-4fd0-a547 (verified by looking at the ipset on the nova hosts). Removing the two lines from the ipset restored all security groups. Expected result was to remove ip filtering on the single port. This sounds similar to: https://bugs.launchpad.net/neutron/+bug/1461054 but is marked fixed long ago. I've marked this as a security bug as a change to a single port can bypass other ports security groups. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1793029/+subscriptions From 1824248 at bugs.launchpad.net Tue May 28 00:57:55 2019 From: 1824248 at bugs.launchpad.net (OpenStack Infra) Date: Tue, 28 May 2019 00:57:55 -0000 Subject: [Openstack-security] [Bug 1824248] Re: Security Group filtering hides rules from user References: <155493822208.21486.11348171578680982627.malonedeb@soybean.canonical.com> Message-ID: <155900507596.16310.3649446777961651278.malone@chaenomeles.canonical.com> Reviewed: https://review.opendev.org/661281 Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=acd081c298052e678eb2ff74434d1529544088d8 Submitter: Zuul Branch: stable/stein commit acd081c298052e678eb2ff74434d1529544088d8 Author: Slawek Kaplonski Date: Mon May 20 18:47:18 2019 +0200 Show all SG rules belong to SG in group's details If security group contains rule(s) which were created by different user (admin), owner of this security group should see such rules even if those rules don't belong to him. This patch changes to use admin_context to get security group rules in get_security_group() method to achieve that. Test to cover such case is added in neutron-tempest-plugin repo. Change-Id: I890c81bb6eabc5caa620ed4fcc4dc88ebfa6e1b0 Closes-Bug: #1824248 (cherry picked from commit 1920a37a94b7a9589dcf83f6ff0765068560dbf8) ** Tags added: in-stable-stein -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1824248 Title: Security Group filtering hides rules from user Status in neutron: Fix Released Status in OpenStack Security Advisory: Won't Fix Bug description: Manage Rules part of the GUI hides the rules currently visible in the Launch Instance modal window. It allows a malicious admin to add backdoor access rules that might be later added to VMs without the knowledge of owner of those VMs. When sending GET request as below, it responds only with the rules that are created by user and this happens when using Manage Rules part of the GUI: On the other hand when using GET request as below, it responds with all SG and it includes all rules, and there is no filtering and this is used in Launch Instance modal window: Here is example of rules display in Manage Rules part of GUI: > /opt/stack/horizon/openstack_dashboard/dashboards/project/security_groups/views.py(50)_get_data() -> return api.neutron.security_group_get(self.request, sg_id) (Pdb) l  45 @memoized.memoized_method  46 def _get_data(self):  47 sg_id = filters.get_int_or_uuid(self.kwargs['security_group_id'])  48 try:  49 from remote_pdb import RemotePdb; RemotePdb('127.0.0.1', 444).set_trace()  50 -> return api.neutron.security_group_get(self.request, sg_id)  51 except Exception:  52 redirect = reverse('horizon:project:security_groups:index')  53 exceptions.handle(self.request,  54 _('Unable to retrieve security group.'),  55 redirect=redirect) (Pdb) p api.neutron.security_group_get(self.request, sg_id) , , , ]}> (Pdb) (Pdb) p self.request As you might have noticed there are no ports access 44 and 22 (SSH) And from the Launch Instance Modal Window, as well as CLI we can see that there are two more rules that are invisible for user, port 44 and 22 (SSH) as displayed below: > /opt/stack/horizon/openstack_dashboard/api/rest/network.py(47)get() -> return {'items': [sg.to_dict() for sg in security_groups]} (Pdb) l  42 """  43  44 security_groups = api.neutron.security_group_list(request)  45 from remote_pdb import RemotePdb; RemotePdb('127.0.0.1', 444).set_trace()  46  47 -> return {'items': [sg.to_dict() for sg in security_groups]}  48  49  50 @urls.register  51 class FloatingIP(generic.View):  52 """API for a single floating IP address.""" (Pdb) p security_groups [, , , , , ]}>] (Pdb) (Pdb) p request Thank you, Robin To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1824248/+subscriptions From 1824248 at bugs.launchpad.net Tue May 28 17:42:03 2019 From: 1824248 at bugs.launchpad.net (OpenStack Infra) Date: Tue, 28 May 2019 17:42:03 -0000 Subject: [Openstack-security] [Bug 1824248] Re: Security Group filtering hides rules from user References: <155493822208.21486.11348171578680982627.malonedeb@soybean.canonical.com> Message-ID: <155906532382.16496.14136222850505232431.malone@chaenomeles.canonical.com> Reviewed: https://review.opendev.org/661283 Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=252d80058cacc797fbfb970ad42031efe301b0e2 Submitter: Zuul Branch: stable/rocky commit 252d80058cacc797fbfb970ad42031efe301b0e2 Author: Slawek Kaplonski Date: Mon May 20 18:47:18 2019 +0200 Show all SG rules belong to SG in group's details If security group contains rule(s) which were created by different user (admin), owner of this security group should see such rules even if those rules don't belong to him. This patch changes to use admin_context to get security group rules in get_security_group() method to achieve that. Test to cover such case is added in neutron-tempest-plugin repo. Conflicts: neutron/db/securitygroups_db.py Change-Id: I890c81bb6eabc5caa620ed4fcc4dc88ebfa6e1b0 Closes-Bug: #1824248 (cherry picked from commit 1920a37a94b7a9589dcf83f6ff0765068560dbf8) ** Tags added: in-stable-rocky ** Tags added: in-stable-queens -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1824248 Title: Security Group filtering hides rules from user Status in neutron: Fix Released Status in OpenStack Security Advisory: Won't Fix Bug description: Manage Rules part of the GUI hides the rules currently visible in the Launch Instance modal window. It allows a malicious admin to add backdoor access rules that might be later added to VMs without the knowledge of owner of those VMs. When sending GET request as below, it responds only with the rules that are created by user and this happens when using Manage Rules part of the GUI: On the other hand when using GET request as below, it responds with all SG and it includes all rules, and there is no filtering and this is used in Launch Instance modal window: Here is example of rules display in Manage Rules part of GUI: > /opt/stack/horizon/openstack_dashboard/dashboards/project/security_groups/views.py(50)_get_data() -> return api.neutron.security_group_get(self.request, sg_id) (Pdb) l  45 @memoized.memoized_method  46 def _get_data(self):  47 sg_id = filters.get_int_or_uuid(self.kwargs['security_group_id'])  48 try:  49 from remote_pdb import RemotePdb; RemotePdb('127.0.0.1', 444).set_trace()  50 -> return api.neutron.security_group_get(self.request, sg_id)  51 except Exception:  52 redirect = reverse('horizon:project:security_groups:index')  53 exceptions.handle(self.request,  54 _('Unable to retrieve security group.'),  55 redirect=redirect) (Pdb) p api.neutron.security_group_get(self.request, sg_id) , , , ]}> (Pdb) (Pdb) p self.request As you might have noticed there are no ports access 44 and 22 (SSH) And from the Launch Instance Modal Window, as well as CLI we can see that there are two more rules that are invisible for user, port 44 and 22 (SSH) as displayed below: > /opt/stack/horizon/openstack_dashboard/api/rest/network.py(47)get() -> return {'items': [sg.to_dict() for sg in security_groups]} (Pdb) l  42 """  43  44 security_groups = api.neutron.security_group_list(request)  45 from remote_pdb import RemotePdb; RemotePdb('127.0.0.1', 444).set_trace()  46  47 -> return {'items': [sg.to_dict() for sg in security_groups]}  48  49  50 @urls.register  51 class FloatingIP(generic.View):  52 """API for a single floating IP address.""" (Pdb) p security_groups [, , , , , ]}>] (Pdb) (Pdb) p request Thank you, Robin To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1824248/+subscriptions From 1824248 at bugs.launchpad.net Tue May 28 17:43:10 2019 From: 1824248 at bugs.launchpad.net (OpenStack Infra) Date: Tue, 28 May 2019 17:43:10 -0000 Subject: [Openstack-security] [Bug 1824248] Fix merged to neutron (stable/queens) References: <155493822208.21486.11348171578680982627.malonedeb@soybean.canonical.com> Message-ID: <155906539054.7772.9066743692512491303.malone@wampee.canonical.com> Reviewed: https://review.opendev.org/661284 Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=e02b66a858b552783d442b52e08f296c849951db Submitter: Zuul Branch: stable/queens commit e02b66a858b552783d442b52e08f296c849951db Author: Slawek Kaplonski Date: Mon May 20 18:47:18 2019 +0200 Show all SG rules belong to SG in group's details If security group contains rule(s) which were created by different user (admin), owner of this security group should see such rules even if those rules don't belong to him. This patch changes to use admin_context to get security group rules in get_security_group() method to achieve that. Test to cover such case is added in neutron-tempest-plugin repo. Conflicts: neutron/db/securitygroups_db.py Change-Id: I890c81bb6eabc5caa620ed4fcc4dc88ebfa6e1b0 Closes-Bug: #1824248 (cherry picked from commit 1920a37a94b7a9589dcf83f6ff0765068560dbf8) (cherry picked from commit 252d80058cacc797fbfb970ad42031efe301b0e2) -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1824248 Title: Security Group filtering hides rules from user Status in neutron: Fix Released Status in OpenStack Security Advisory: Won't Fix Bug description: Manage Rules part of the GUI hides the rules currently visible in the Launch Instance modal window. It allows a malicious admin to add backdoor access rules that might be later added to VMs without the knowledge of owner of those VMs. When sending GET request as below, it responds only with the rules that are created by user and this happens when using Manage Rules part of the GUI: On the other hand when using GET request as below, it responds with all SG and it includes all rules, and there is no filtering and this is used in Launch Instance modal window: Here is example of rules display in Manage Rules part of GUI: > /opt/stack/horizon/openstack_dashboard/dashboards/project/security_groups/views.py(50)_get_data() -> return api.neutron.security_group_get(self.request, sg_id) (Pdb) l  45 @memoized.memoized_method  46 def _get_data(self):  47 sg_id = filters.get_int_or_uuid(self.kwargs['security_group_id'])  48 try:  49 from remote_pdb import RemotePdb; RemotePdb('127.0.0.1', 444).set_trace()  50 -> return api.neutron.security_group_get(self.request, sg_id)  51 except Exception:  52 redirect = reverse('horizon:project:security_groups:index')  53 exceptions.handle(self.request,  54 _('Unable to retrieve security group.'),  55 redirect=redirect) (Pdb) p api.neutron.security_group_get(self.request, sg_id) , , , ]}> (Pdb) (Pdb) p self.request As you might have noticed there are no ports access 44 and 22 (SSH) And from the Launch Instance Modal Window, as well as CLI we can see that there are two more rules that are invisible for user, port 44 and 22 (SSH) as displayed below: > /opt/stack/horizon/openstack_dashboard/api/rest/network.py(47)get() -> return {'items': [sg.to_dict() for sg in security_groups]} (Pdb) l  42 """  43  44 security_groups = api.neutron.security_group_list(request)  45 from remote_pdb import RemotePdb; RemotePdb('127.0.0.1', 444).set_trace()  46  47 -> return {'items': [sg.to_dict() for sg in security_groups]}  48  49  50 @urls.register  51 class FloatingIP(generic.View):  52 """API for a single floating IP address.""" (Pdb) p security_groups [, , , , , ]}>] (Pdb) (Pdb) p request Thank you, Robin To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1824248/+subscriptions From 1824248 at bugs.launchpad.net Tue May 28 17:43:18 2019 From: 1824248 at bugs.launchpad.net (OpenStack Infra) Date: Tue, 28 May 2019 17:43:18 -0000 Subject: [Openstack-security] [Bug 1824248] Related fix merged to neutron-tempest-plugin (master) References: <155493822208.21486.11348171578680982627.malonedeb@soybean.canonical.com> Message-ID: <155906539889.16885.2477621637037977000.malone@chaenomeles.canonical.com> Reviewed: https://review.opendev.org/660175 Committed: https://git.openstack.org/cgit/openstack/neutron-tempest-plugin/commit/?id=87c3f941a3dd168d64226a4e238545ed69e05383 Submitter: Zuul Branch: master commit 87c3f941a3dd168d64226a4e238545ed69e05383 Author: Slawek Kaplonski Date: Mon May 20 18:50:53 2019 +0200 Add API test case to check if SG displays all rules This patch adds new API test which checks if owner of security group can see rules which belongs to his security group even if rule was created and belongs to other user (admin). Patch for master branch: Depends-On: https://review.opendev.org/660174 Backport to stable/Stein: Depends-On: https://review.opendev.org/661281 Backport to stable/Rocky: Depends-On: https://review.opendev.org/661283 Backport to stable/Queens: Depends-On: https://review.opendev.org/661284 Change-Id: I728cd8252d27e27e91bd95e4734d9db470dee35a Related-Bug: #1824248 -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1824248 Title: Security Group filtering hides rules from user Status in neutron: Fix Released Status in OpenStack Security Advisory: Won't Fix Bug description: Manage Rules part of the GUI hides the rules currently visible in the Launch Instance modal window. It allows a malicious admin to add backdoor access rules that might be later added to VMs without the knowledge of owner of those VMs. When sending GET request as below, it responds only with the rules that are created by user and this happens when using Manage Rules part of the GUI: On the other hand when using GET request as below, it responds with all SG and it includes all rules, and there is no filtering and this is used in Launch Instance modal window: Here is example of rules display in Manage Rules part of GUI: > /opt/stack/horizon/openstack_dashboard/dashboards/project/security_groups/views.py(50)_get_data() -> return api.neutron.security_group_get(self.request, sg_id) (Pdb) l  45 @memoized.memoized_method  46 def _get_data(self):  47 sg_id = filters.get_int_or_uuid(self.kwargs['security_group_id'])  48 try:  49 from remote_pdb import RemotePdb; RemotePdb('127.0.0.1', 444).set_trace()  50 -> return api.neutron.security_group_get(self.request, sg_id)  51 except Exception:  52 redirect = reverse('horizon:project:security_groups:index')  53 exceptions.handle(self.request,  54 _('Unable to retrieve security group.'),  55 redirect=redirect) (Pdb) p api.neutron.security_group_get(self.request, sg_id) , , , ]}> (Pdb) (Pdb) p self.request As you might have noticed there are no ports access 44 and 22 (SSH) And from the Launch Instance Modal Window, as well as CLI we can see that there are two more rules that are invisible for user, port 44 and 22 (SSH) as displayed below: > /opt/stack/horizon/openstack_dashboard/api/rest/network.py(47)get() -> return {'items': [sg.to_dict() for sg in security_groups]} (Pdb) l  42 """  43  44 security_groups = api.neutron.security_group_list(request)  45 from remote_pdb import RemotePdb; RemotePdb('127.0.0.1', 444).set_trace()  46  47 -> return {'items': [sg.to_dict() for sg in security_groups]}  48  49  50 @urls.register  51 class FloatingIP(generic.View):  52 """API for a single floating IP address.""" (Pdb) p security_groups [, , , , , ]}>] (Pdb) (Pdb) p request Thank you, Robin To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1824248/+subscriptions From 1793029 at bugs.launchpad.net Wed May 29 15:46:22 2019 From: 1793029 at bugs.launchpad.net (OpenStack Infra) Date: Wed, 29 May 2019 15:46:22 -0000 Subject: [Openstack-security] [Bug 1793029] Re: adding 0.0.0.0/0 address pair to a port bypasses all other vm security groups References: <153722407420.8513.15403635939639716317.malonedeb@chaenomeles.canonical.com> Message-ID: <155914478514.17038.12745077322051753413.launchpad@chaenomeles.canonical.com> ** Changed in: neutron Assignee: Slawek Kaplonski (slaweq) => Brian Haley (brian-haley) -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1793029 Title: adding 0.0.0.0/0 address pair to a port bypasses all other vm security groups Status in neutron: In Progress Status in OpenStack Security Advisory: Won't Fix Status in OpenStack Security Notes: New Bug description: On an openstack-ansible / newton setup with linuxbridge, a customer ran: neutron port-update $port-uuid --allowed-address-pairs type=dict list=true ip_address=0.0.0.0/0 to bypass the ip source restriction (pfsense router and had to route packets). The impact of running the above, was an allow all rule was added to all ports in the network, bypassing all security groups. The iptables rule:  905K 55M RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 match-set NIPv44046d62c-59c8-4fd0-a547- src used on all ports, now triggers as: 0.0.0.0/1 128.0.0.0/1 was added to the ipset NIPv44046d62c-59c8-4fd0-a547 (verified by looking at the ipset on the nova hosts). Removing the two lines from the ipset restored all security groups. Expected result was to remove ip filtering on the single port. This sounds similar to: https://bugs.launchpad.net/neutron/+bug/1461054 but is marked fixed long ago. I've marked this as a security bug as a change to a single port can bypass other ports security groups. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1793029/+subscriptions From 1793029 at bugs.launchpad.net Wed May 29 16:32:37 2019 From: 1793029 at bugs.launchpad.net (OpenStack Infra) Date: Wed, 29 May 2019 16:32:37 -0000 Subject: [Openstack-security] [Bug 1793029] Re: adding 0.0.0.0/0 address pair to a port bypasses all other vm security groups References: <153722407420.8513.15403635939639716317.malonedeb@chaenomeles.canonical.com> Message-ID: <155914755713.12854.3393927712003453780.malone@soybean.canonical.com> Reviewed: https://review.opendev.org/661594 Committed: https://git.openstack.org/cgit/openstack/neutron-lib/commit/?id=e02c4b214b6762823c5e1ae5719f08f5f51910e8 Submitter: Zuul Branch: master commit e02c4b214b6762823c5e1ae5719f08f5f51910e8 Author: Slawek Kaplonski Date: Mon May 27 02:45:15 2019 +0200 [api-ref] Add short warning about ANY IP address in allowed address pair Change-Id: Ie3ed5ea81e0df50a699174fef95eb32337ed5682 Closes-Bug: #1793029 ** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1793029 Title: adding 0.0.0.0/0 address pair to a port bypasses all other vm security groups Status in neutron: Fix Released Status in OpenStack Security Advisory: Won't Fix Status in OpenStack Security Notes: New Bug description: On an openstack-ansible / newton setup with linuxbridge, a customer ran: neutron port-update $port-uuid --allowed-address-pairs type=dict list=true ip_address=0.0.0.0/0 to bypass the ip source restriction (pfsense router and had to route packets). The impact of running the above, was an allow all rule was added to all ports in the network, bypassing all security groups. The iptables rule:  905K 55M RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 match-set NIPv44046d62c-59c8-4fd0-a547- src used on all ports, now triggers as: 0.0.0.0/1 128.0.0.0/1 was added to the ipset NIPv44046d62c-59c8-4fd0-a547 (verified by looking at the ipset on the nova hosts). Removing the two lines from the ipset restored all security groups. Expected result was to remove ip filtering on the single port. This sounds similar to: https://bugs.launchpad.net/neutron/+bug/1461054 but is marked fixed long ago. I've marked this as a security bug as a change to a single port can bypass other ports security groups. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1793029/+subscriptions From 1734320 at bugs.launchpad.net Thu May 30 17:12:58 2019 From: 1734320 at bugs.launchpad.net (OpenStack Infra) Date: Thu, 30 May 2019 17:12:58 -0000 Subject: [Openstack-security] [Bug 1734320] Change abandoned on os-vif (stable/queens) References: <151152217834.14483.1577991310209811902.malonedeb@soybean.canonical.com> Message-ID: <155923637893.12443.17837889341434354722.malone@soybean.canonical.com> Change abandoned by sean mooney (smooney at redhat.com) on branch: stable/queens Review: https://review.opendev.org/609851 -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1734320 Title: Eavesdropping private traffic Status in neutron: Fix Committed Status in OpenStack Compute (nova): In Progress Status in os-vif: Fix Released Status in OpenStack Security Advisory: Won't Fix Bug description: Eavesdropping private traffic ============================= Abstract -------- We've discovered a security issue that allows end users within their own private network to receive from, and send traffic to, other private networks on the same compute node. Description ----------- During live-migration there is a small time window where the ports of instances are untagged. Instances have a port trunked to the integration bridge and receive 802.1Q tagged private traffic from other tenants. If the port is administratively down during live migration, the port will remain in trunk mode indefinitely. Traffic is possible between ports is that are administratively down, even between tenants self-service networks. Conditions ---------- The following conditions are necessary. * Openvswitch Self-service networks * An Openstack administrator or an automated process needs to schedule a Live migration We tested this on newton. Issues ------ This outcome is the result of multiple independent issues. We will list the most important first, and follow with bugs that create a fragile situation. Issue #1 Initially creating a trunk port When the port is initially created, it is in trunk mode. This creates a fail-open situation. See: https://github.com/openstack/os-vif/blob/newton-eol/vif_plug_ovs/linux_net.py#L52 Recommendation: create ports in the port_dead state, don't leave it dangling in trunk-mode. Add a drop-flow initially. Issue #2 Order of creation. The instance is actually migrated before the (networking) configuration is completed. Recommendation: wait with finishing the live migration until the underlying configuration has been applied completely. Issue #3 Not closing the port when it is down. Neutron calls the port_dead function to ensure the port is down. It sets the tag to 4095 and adds a "drop" flow if (and only if) there is already another tag on the port. The port_dead function will keep untagged ports untagged. https://github.com/openstack/neutron/blob/stable/newton/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L995 Recommendation: Make port_dead also shut the port if no tag is found. Log a warning if this happens. Issue #4 Putting the port administratively down actually puts the port on a compute node shared vlan Instances from different projects on different private networks can talk to each other if they put their ports down. The code does install an openflow "drop" rule but it has a lower priority (2) than the allow rules. Recommendation: Increase the port_dead openflow drop rule priority to MAX Timeline --------  2017-09-14 Discovery eavesdropping issue  2017-09-15 Verify workaround.  2017-10-04 Discovery port-down-traffic issue  2017-11-24 Vendor Disclosure to Openstack Steps to reproduce ------------------ 1. Attach an instance to two networks: admin$ openstack server create --nic net-id= --nic net-id = --image --flavor instance_temp 2. Attach a FIP to the instance to be able to log in to this instance 3. Verify: admin$ openstack server show -c name -c addresses fe28a2ee-098f-4425 -9d3c-8e2cd383547d +-----------+-----------------------------------------------------------------------------+ | Field | Value | +-----------+-----------------------------------------------------------------------------+ | addresses | network1=192.168.99.8, ; network2=192.168.80.14 | | name | instance_temp | +-----------+-----------------------------------------------------------------------------+ 4. Ssh to the instance using network1 and run a tcpdump on the other port network2 [root at instance_temp]$ tcpdump -eeenni eth1 5. Get port-id of network2 admin$ nova interface-list fe28a2ee-098f-4425-9d3c-8e2cd383547d +------------+--------------------------------------+--------------------------------------+---------------+-------------------+ | Port State | Port ID | Net ID | IP addresses | MAC Addr | +------------+--------------------------------------+--------------------------------------+---------------+-------------------+ | ACTIVE | a848520b-0814-4030-bb48-49e4b5cf8160 | d69028f7-9558-4f14-8ce6-29cb8f1c19cd | 192.168.80.14 | fa:16:3e:2d:8b:7b | | ACTIVE | fad148ca-cf7a-4839-aac3-a2cd8d1d2260 | d22c22ae-0a42-4e3b-8144-f28534c3439a | 192.168.99.8 | fa:16:3e:60:2c:fa | +------------+--------------------------------------+--------------------------------------+---------------+-------------------+ 6. Force port down on network 2 admin$ neutron port-update a848520b-0814-4030-bb48-49e4b5cf8160 --admin-state-up False 7. Port gets tagged with vlan 4095, the dead vlan tag, which is normal: compute1# grep a848520b-0814-4030-bb48-49e4b5cf8160 /var/log/neutron/neutron-openvswitch-agent.log | tail -1 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-e008feb3-8a35-4c97-adac-b48ff88165b2 - - - - -] VIF port: a848520b-0814-4030-bb48-49e4b5cf8160 admin state up disabled, putting on the dead VLAN 8. Verify the port is tagged with vlan 4095 compute1# ovs-vsctl show | grep -A3 qvoa848520b-08       Port "qvoa848520b-08"           tag: 4095           Interface "qvoa848520b-08" 9. Now live-migrate the instance: admin# nova live-migration fe28a2ee-098f-4425-9d3c-8e2cd383547d 10. Verify the tag is gone on compute2, and take a deep breath compute2# ovs-vsctl show | grep -A3 qvoa848520b-08       Port "qvoa848520b-08"           Interface "qvoa848520b-08"       Port... compute2# echo "Wut!" 11. Now traffic of all other self-service networks present on compute2 can be sniffed from instance_temp [root at instance_temp] tcpdump -eenni eth1 13:14:31.748266 fa:16:3e:6a:17:38 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.103.12.160 tell 10.103.12.152, length 28 13:14:31.804573 fa:16:3e:e8:a2:d2 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.70, length 28 13:14:31.810482 fa:16:3e:95:ca:3a > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.154, length 28 13:14:31.977820 fa:16:3e:6f:f4:9b > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.150, length 28 13:14:31.979590 fa:16:3e:0f:3d:cc > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 9, p 0, ethertype ARP, Request who-has 10.103.9.163 tell 10.103.9.1, length 28 13:14:32.048082 fa:16:3e:65:64:38 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.101, length 28 13:14:32.127400 fa:16:3e:30:cb:b5 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.103.12.160 tell 10.103.12.165, length 28 13:14:32.141982 fa:16:3e:96:cd:b0 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.100, length 28 13:14:32.205327 fa:16:3e:a2:0b:76 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.153, length 28 13:14:32.444142 fa:16:3e:1f:db:ed > 01:00:5e:00:00:12, ethertype 802.1Q (0x8100), length 58: vlan 72, p 0, ethertype IPv4, 192.168.99.212 > 224.0.0.18: VRRPv2, Advertisement, vrid 50, prio 103, authtype none, intvl 1s, length 20 13:14:32.449497 fa:16:3e:1c:24:c0 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.20, length 28 13:14:32.476015 fa:16:3e:f2:3b:97 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.22, length 28 13:14:32.575034 fa:16:3e:44:fe:35 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.103.12.160 tell 10.103.12.163, length 28 13:14:32.676185 fa:16:3e:1e:92:d7 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.103.12.160 tell 10.103.12.150, length 28 13:14:32.711755 fa:16:3e:99:6c:c8 > 01:00:5e:00:00:12, ethertype 802.1Q (0x8100), length 62: vlan 10, p 0, ethertype IPv4, 10.103.12.154 > 224.0.0.18: VRRPv2, Advertisement, vrid 2, prio 49, authtype simple, intvl 1s, length 24 13:14:32.711773 fa:16:3e:f5:23:d5 > 01:00:5e:00:00:12, ethertype 802.1Q (0x8100), length 58: vlan 12, p 0, ethertype IPv4, 10.103.15.154 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 49, authtype simple, intvl 1s, length 20 Workaround ---------- We temporary fixed this issue by forcing the dead vlan tag on port creation on compute nodes: /usr/lib/python2.7/site-packages/vif_plug_ovs/linux_net.py: def _create_ovs_vif_cmd(bridge, dev, iface_id, mac,                         instance_id, interface_type=None,                         vhost_server_path=None): + # ODCN: initialize port as dead + # ODCN: TODO: set drop flow     cmd = ['--', '--if-exists', 'del-port', dev, '--',             'add-port', bridge, dev, + 'tag=4095',             '--', 'set', 'Interface', dev,             'external-ids:iface-id=%s' % iface_id,             'external-ids:iface-status=active',             'external-ids:attached-mac=%s' % mac,             'external-ids:vm-uuid=%s' % instance_id]     if interface_type:         cmd += ['type=%s' % interface_type]     if vhost_server_path:         cmd += ['options:vhost-server-path=%s' % vhost_server_path]     return cmd https://github.com/openstack/neutron/blob/stable/newton/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L995     def port_dead(self, port, log_errors=True):         '''Once a port has no binding, put it on the "dead vlan".         :param port: an ovs_lib.VifPort object.         '''         # Don't kill a port if it's already dead         cur_tag = self.int_br.db_get_val("Port", port.port_name, "tag",                                          log_errors=log_errors) + # ODCN GM 20170915 + if not cur_tag: + LOG.error('port_dead(): port %s has no tag', port.port_name) + # ODCN AJS 20170915 + if not cur_tag or cur_tag != constants.DEAD_VLAN_TAG: - if cur_tag and cur_tag != constants.DEAD_VLAN_TAG:            LOG.info('port_dead(): put port %s on dead vlan', port.port_name)            self.int_br.set_db_attribute("Port", port.port_name, "tag",                                          constants.DEAD_VLAN_TAG,                                          log_errors=log_errors)             self.int_br.drop_port(in_port=port.ofport) plugins/ml2/drivers/openvswitch/agent/openflow/ovs_ofctl/ovs_bridge.py     def drop_port(self, in_port): + # ODCN AJS 20171004: - self.install_drop(priority=2, in_port=in_port) + self.install_drop(priority=65535, in_port=in_port) Regards, ODC Noord. Gerhard Muntingh Albert Siersema Paul Peereboom To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1734320/+subscriptions From 1734320 at bugs.launchpad.net Thu May 30 17:13:12 2019 From: 1734320 at bugs.launchpad.net (OpenStack Infra) Date: Thu, 30 May 2019 17:13:12 -0000 Subject: [Openstack-security] [Bug 1734320] Change abandoned on os-vif (stable/rocky) References: <151152217834.14483.1577991310209811902.malonedeb@soybean.canonical.com> Message-ID: <155923639265.25369.3311170319868874133.malone@gac.canonical.com> Change abandoned by sean mooney (smooney at redhat.com) on branch: stable/rocky Review: https://review.opendev.org/609850 -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1734320 Title: Eavesdropping private traffic Status in neutron: Fix Committed Status in OpenStack Compute (nova): In Progress Status in os-vif: Fix Released Status in OpenStack Security Advisory: Won't Fix Bug description: Eavesdropping private traffic ============================= Abstract -------- We've discovered a security issue that allows end users within their own private network to receive from, and send traffic to, other private networks on the same compute node. Description ----------- During live-migration there is a small time window where the ports of instances are untagged. Instances have a port trunked to the integration bridge and receive 802.1Q tagged private traffic from other tenants. If the port is administratively down during live migration, the port will remain in trunk mode indefinitely. Traffic is possible between ports is that are administratively down, even between tenants self-service networks. Conditions ---------- The following conditions are necessary. * Openvswitch Self-service networks * An Openstack administrator or an automated process needs to schedule a Live migration We tested this on newton. Issues ------ This outcome is the result of multiple independent issues. We will list the most important first, and follow with bugs that create a fragile situation. Issue #1 Initially creating a trunk port When the port is initially created, it is in trunk mode. This creates a fail-open situation. See: https://github.com/openstack/os-vif/blob/newton-eol/vif_plug_ovs/linux_net.py#L52 Recommendation: create ports in the port_dead state, don't leave it dangling in trunk-mode. Add a drop-flow initially. Issue #2 Order of creation. The instance is actually migrated before the (networking) configuration is completed. Recommendation: wait with finishing the live migration until the underlying configuration has been applied completely. Issue #3 Not closing the port when it is down. Neutron calls the port_dead function to ensure the port is down. It sets the tag to 4095 and adds a "drop" flow if (and only if) there is already another tag on the port. The port_dead function will keep untagged ports untagged. https://github.com/openstack/neutron/blob/stable/newton/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L995 Recommendation: Make port_dead also shut the port if no tag is found. Log a warning if this happens. Issue #4 Putting the port administratively down actually puts the port on a compute node shared vlan Instances from different projects on different private networks can talk to each other if they put their ports down. The code does install an openflow "drop" rule but it has a lower priority (2) than the allow rules. Recommendation: Increase the port_dead openflow drop rule priority to MAX Timeline --------  2017-09-14 Discovery eavesdropping issue  2017-09-15 Verify workaround.  2017-10-04 Discovery port-down-traffic issue  2017-11-24 Vendor Disclosure to Openstack Steps to reproduce ------------------ 1. Attach an instance to two networks: admin$ openstack server create --nic net-id= --nic net-id = --image --flavor instance_temp 2. Attach a FIP to the instance to be able to log in to this instance 3. Verify: admin$ openstack server show -c name -c addresses fe28a2ee-098f-4425 -9d3c-8e2cd383547d +-----------+-----------------------------------------------------------------------------+ | Field | Value | +-----------+-----------------------------------------------------------------------------+ | addresses | network1=192.168.99.8, ; network2=192.168.80.14 | | name | instance_temp | +-----------+-----------------------------------------------------------------------------+ 4. Ssh to the instance using network1 and run a tcpdump on the other port network2 [root at instance_temp]$ tcpdump -eeenni eth1 5. Get port-id of network2 admin$ nova interface-list fe28a2ee-098f-4425-9d3c-8e2cd383547d +------------+--------------------------------------+--------------------------------------+---------------+-------------------+ | Port State | Port ID | Net ID | IP addresses | MAC Addr | +------------+--------------------------------------+--------------------------------------+---------------+-------------------+ | ACTIVE | a848520b-0814-4030-bb48-49e4b5cf8160 | d69028f7-9558-4f14-8ce6-29cb8f1c19cd | 192.168.80.14 | fa:16:3e:2d:8b:7b | | ACTIVE | fad148ca-cf7a-4839-aac3-a2cd8d1d2260 | d22c22ae-0a42-4e3b-8144-f28534c3439a | 192.168.99.8 | fa:16:3e:60:2c:fa | +------------+--------------------------------------+--------------------------------------+---------------+-------------------+ 6. Force port down on network 2 admin$ neutron port-update a848520b-0814-4030-bb48-49e4b5cf8160 --admin-state-up False 7. Port gets tagged with vlan 4095, the dead vlan tag, which is normal: compute1# grep a848520b-0814-4030-bb48-49e4b5cf8160 /var/log/neutron/neutron-openvswitch-agent.log | tail -1 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-e008feb3-8a35-4c97-adac-b48ff88165b2 - - - - -] VIF port: a848520b-0814-4030-bb48-49e4b5cf8160 admin state up disabled, putting on the dead VLAN 8. Verify the port is tagged with vlan 4095 compute1# ovs-vsctl show | grep -A3 qvoa848520b-08       Port "qvoa848520b-08"           tag: 4095           Interface "qvoa848520b-08" 9. Now live-migrate the instance: admin# nova live-migration fe28a2ee-098f-4425-9d3c-8e2cd383547d 10. Verify the tag is gone on compute2, and take a deep breath compute2# ovs-vsctl show | grep -A3 qvoa848520b-08       Port "qvoa848520b-08"           Interface "qvoa848520b-08"       Port... compute2# echo "Wut!" 11. Now traffic of all other self-service networks present on compute2 can be sniffed from instance_temp [root at instance_temp] tcpdump -eenni eth1 13:14:31.748266 fa:16:3e:6a:17:38 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.103.12.160 tell 10.103.12.152, length 28 13:14:31.804573 fa:16:3e:e8:a2:d2 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.70, length 28 13:14:31.810482 fa:16:3e:95:ca:3a > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.154, length 28 13:14:31.977820 fa:16:3e:6f:f4:9b > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.150, length 28 13:14:31.979590 fa:16:3e:0f:3d:cc > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 9, p 0, ethertype ARP, Request who-has 10.103.9.163 tell 10.103.9.1, length 28 13:14:32.048082 fa:16:3e:65:64:38 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.101, length 28 13:14:32.127400 fa:16:3e:30:cb:b5 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.103.12.160 tell 10.103.12.165, length 28 13:14:32.141982 fa:16:3e:96:cd:b0 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.100, length 28 13:14:32.205327 fa:16:3e:a2:0b:76 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.153, length 28 13:14:32.444142 fa:16:3e:1f:db:ed > 01:00:5e:00:00:12, ethertype 802.1Q (0x8100), length 58: vlan 72, p 0, ethertype IPv4, 192.168.99.212 > 224.0.0.18: VRRPv2, Advertisement, vrid 50, prio 103, authtype none, intvl 1s, length 20 13:14:32.449497 fa:16:3e:1c:24:c0 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.20, length 28 13:14:32.476015 fa:16:3e:f2:3b:97 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.22, length 28 13:14:32.575034 fa:16:3e:44:fe:35 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.103.12.160 tell 10.103.12.163, length 28 13:14:32.676185 fa:16:3e:1e:92:d7 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.103.12.160 tell 10.103.12.150, length 28 13:14:32.711755 fa:16:3e:99:6c:c8 > 01:00:5e:00:00:12, ethertype 802.1Q (0x8100), length 62: vlan 10, p 0, ethertype IPv4, 10.103.12.154 > 224.0.0.18: VRRPv2, Advertisement, vrid 2, prio 49, authtype simple, intvl 1s, length 24 13:14:32.711773 fa:16:3e:f5:23:d5 > 01:00:5e:00:00:12, ethertype 802.1Q (0x8100), length 58: vlan 12, p 0, ethertype IPv4, 10.103.15.154 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 49, authtype simple, intvl 1s, length 20 Workaround ---------- We temporary fixed this issue by forcing the dead vlan tag on port creation on compute nodes: /usr/lib/python2.7/site-packages/vif_plug_ovs/linux_net.py: def _create_ovs_vif_cmd(bridge, dev, iface_id, mac,                         instance_id, interface_type=None,                         vhost_server_path=None): + # ODCN: initialize port as dead + # ODCN: TODO: set drop flow     cmd = ['--', '--if-exists', 'del-port', dev, '--',             'add-port', bridge, dev, + 'tag=4095',             '--', 'set', 'Interface', dev,             'external-ids:iface-id=%s' % iface_id,             'external-ids:iface-status=active',             'external-ids:attached-mac=%s' % mac,             'external-ids:vm-uuid=%s' % instance_id]     if interface_type:         cmd += ['type=%s' % interface_type]     if vhost_server_path:         cmd += ['options:vhost-server-path=%s' % vhost_server_path]     return cmd https://github.com/openstack/neutron/blob/stable/newton/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L995     def port_dead(self, port, log_errors=True):         '''Once a port has no binding, put it on the "dead vlan".         :param port: an ovs_lib.VifPort object.         '''         # Don't kill a port if it's already dead         cur_tag = self.int_br.db_get_val("Port", port.port_name, "tag",                                          log_errors=log_errors) + # ODCN GM 20170915 + if not cur_tag: + LOG.error('port_dead(): port %s has no tag', port.port_name) + # ODCN AJS 20170915 + if not cur_tag or cur_tag != constants.DEAD_VLAN_TAG: - if cur_tag and cur_tag != constants.DEAD_VLAN_TAG:            LOG.info('port_dead(): put port %s on dead vlan', port.port_name)            self.int_br.set_db_attribute("Port", port.port_name, "tag",                                          constants.DEAD_VLAN_TAG,                                          log_errors=log_errors)             self.int_br.drop_port(in_port=port.ofport) plugins/ml2/drivers/openvswitch/agent/openflow/ovs_ofctl/ovs_bridge.py     def drop_port(self, in_port): + # ODCN AJS 20171004: - self.install_drop(priority=2, in_port=in_port) + self.install_drop(priority=65535, in_port=in_port) Regards, ODC Noord. Gerhard Muntingh Albert Siersema Paul Peereboom To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1734320/+subscriptions From 1734320 at bugs.launchpad.net Thu May 30 17:13:24 2019 From: 1734320 at bugs.launchpad.net (OpenStack Infra) Date: Thu, 30 May 2019 17:13:24 -0000 Subject: [Openstack-security] [Bug 1734320] Change abandoned on os-vif (stable/rocky) References: <151152217834.14483.1577991310209811902.malonedeb@soybean.canonical.com> Message-ID: <155923640501.25369.4261239957809876756.malone@gac.canonical.com> Change abandoned by sean mooney (smooney at redhat.com) on branch: stable/rocky Review: https://review.opendev.org/616285 -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1734320 Title: Eavesdropping private traffic Status in neutron: Fix Committed Status in OpenStack Compute (nova): In Progress Status in os-vif: Fix Released Status in OpenStack Security Advisory: Won't Fix Bug description: Eavesdropping private traffic ============================= Abstract -------- We've discovered a security issue that allows end users within their own private network to receive from, and send traffic to, other private networks on the same compute node. Description ----------- During live-migration there is a small time window where the ports of instances are untagged. Instances have a port trunked to the integration bridge and receive 802.1Q tagged private traffic from other tenants. If the port is administratively down during live migration, the port will remain in trunk mode indefinitely. Traffic is possible between ports is that are administratively down, even between tenants self-service networks. Conditions ---------- The following conditions are necessary. * Openvswitch Self-service networks * An Openstack administrator or an automated process needs to schedule a Live migration We tested this on newton. Issues ------ This outcome is the result of multiple independent issues. We will list the most important first, and follow with bugs that create a fragile situation. Issue #1 Initially creating a trunk port When the port is initially created, it is in trunk mode. This creates a fail-open situation. See: https://github.com/openstack/os-vif/blob/newton-eol/vif_plug_ovs/linux_net.py#L52 Recommendation: create ports in the port_dead state, don't leave it dangling in trunk-mode. Add a drop-flow initially. Issue #2 Order of creation. The instance is actually migrated before the (networking) configuration is completed. Recommendation: wait with finishing the live migration until the underlying configuration has been applied completely. Issue #3 Not closing the port when it is down. Neutron calls the port_dead function to ensure the port is down. It sets the tag to 4095 and adds a "drop" flow if (and only if) there is already another tag on the port. The port_dead function will keep untagged ports untagged. https://github.com/openstack/neutron/blob/stable/newton/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L995 Recommendation: Make port_dead also shut the port if no tag is found. Log a warning if this happens. Issue #4 Putting the port administratively down actually puts the port on a compute node shared vlan Instances from different projects on different private networks can talk to each other if they put their ports down. The code does install an openflow "drop" rule but it has a lower priority (2) than the allow rules. Recommendation: Increase the port_dead openflow drop rule priority to MAX Timeline --------  2017-09-14 Discovery eavesdropping issue  2017-09-15 Verify workaround.  2017-10-04 Discovery port-down-traffic issue  2017-11-24 Vendor Disclosure to Openstack Steps to reproduce ------------------ 1. Attach an instance to two networks: admin$ openstack server create --nic net-id= --nic net-id = --image --flavor instance_temp 2. Attach a FIP to the instance to be able to log in to this instance 3. Verify: admin$ openstack server show -c name -c addresses fe28a2ee-098f-4425 -9d3c-8e2cd383547d +-----------+-----------------------------------------------------------------------------+ | Field | Value | +-----------+-----------------------------------------------------------------------------+ | addresses | network1=192.168.99.8, ; network2=192.168.80.14 | | name | instance_temp | +-----------+-----------------------------------------------------------------------------+ 4. Ssh to the instance using network1 and run a tcpdump on the other port network2 [root at instance_temp]$ tcpdump -eeenni eth1 5. Get port-id of network2 admin$ nova interface-list fe28a2ee-098f-4425-9d3c-8e2cd383547d +------------+--------------------------------------+--------------------------------------+---------------+-------------------+ | Port State | Port ID | Net ID | IP addresses | MAC Addr | +------------+--------------------------------------+--------------------------------------+---------------+-------------------+ | ACTIVE | a848520b-0814-4030-bb48-49e4b5cf8160 | d69028f7-9558-4f14-8ce6-29cb8f1c19cd | 192.168.80.14 | fa:16:3e:2d:8b:7b | | ACTIVE | fad148ca-cf7a-4839-aac3-a2cd8d1d2260 | d22c22ae-0a42-4e3b-8144-f28534c3439a | 192.168.99.8 | fa:16:3e:60:2c:fa | +------------+--------------------------------------+--------------------------------------+---------------+-------------------+ 6. Force port down on network 2 admin$ neutron port-update a848520b-0814-4030-bb48-49e4b5cf8160 --admin-state-up False 7. Port gets tagged with vlan 4095, the dead vlan tag, which is normal: compute1# grep a848520b-0814-4030-bb48-49e4b5cf8160 /var/log/neutron/neutron-openvswitch-agent.log | tail -1 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-e008feb3-8a35-4c97-adac-b48ff88165b2 - - - - -] VIF port: a848520b-0814-4030-bb48-49e4b5cf8160 admin state up disabled, putting on the dead VLAN 8. Verify the port is tagged with vlan 4095 compute1# ovs-vsctl show | grep -A3 qvoa848520b-08       Port "qvoa848520b-08"           tag: 4095           Interface "qvoa848520b-08" 9. Now live-migrate the instance: admin# nova live-migration fe28a2ee-098f-4425-9d3c-8e2cd383547d 10. Verify the tag is gone on compute2, and take a deep breath compute2# ovs-vsctl show | grep -A3 qvoa848520b-08       Port "qvoa848520b-08"           Interface "qvoa848520b-08"       Port... compute2# echo "Wut!" 11. Now traffic of all other self-service networks present on compute2 can be sniffed from instance_temp [root at instance_temp] tcpdump -eenni eth1 13:14:31.748266 fa:16:3e:6a:17:38 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.103.12.160 tell 10.103.12.152, length 28 13:14:31.804573 fa:16:3e:e8:a2:d2 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.70, length 28 13:14:31.810482 fa:16:3e:95:ca:3a > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.154, length 28 13:14:31.977820 fa:16:3e:6f:f4:9b > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.150, length 28 13:14:31.979590 fa:16:3e:0f:3d:cc > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 9, p 0, ethertype ARP, Request who-has 10.103.9.163 tell 10.103.9.1, length 28 13:14:32.048082 fa:16:3e:65:64:38 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.101, length 28 13:14:32.127400 fa:16:3e:30:cb:b5 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.103.12.160 tell 10.103.12.165, length 28 13:14:32.141982 fa:16:3e:96:cd:b0 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.100, length 28 13:14:32.205327 fa:16:3e:a2:0b:76 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.153, length 28 13:14:32.444142 fa:16:3e:1f:db:ed > 01:00:5e:00:00:12, ethertype 802.1Q (0x8100), length 58: vlan 72, p 0, ethertype IPv4, 192.168.99.212 > 224.0.0.18: VRRPv2, Advertisement, vrid 50, prio 103, authtype none, intvl 1s, length 20 13:14:32.449497 fa:16:3e:1c:24:c0 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.20, length 28 13:14:32.476015 fa:16:3e:f2:3b:97 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.22, length 28 13:14:32.575034 fa:16:3e:44:fe:35 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.103.12.160 tell 10.103.12.163, length 28 13:14:32.676185 fa:16:3e:1e:92:d7 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.103.12.160 tell 10.103.12.150, length 28 13:14:32.711755 fa:16:3e:99:6c:c8 > 01:00:5e:00:00:12, ethertype 802.1Q (0x8100), length 62: vlan 10, p 0, ethertype IPv4, 10.103.12.154 > 224.0.0.18: VRRPv2, Advertisement, vrid 2, prio 49, authtype simple, intvl 1s, length 24 13:14:32.711773 fa:16:3e:f5:23:d5 > 01:00:5e:00:00:12, ethertype 802.1Q (0x8100), length 58: vlan 12, p 0, ethertype IPv4, 10.103.15.154 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 49, authtype simple, intvl 1s, length 20 Workaround ---------- We temporary fixed this issue by forcing the dead vlan tag on port creation on compute nodes: /usr/lib/python2.7/site-packages/vif_plug_ovs/linux_net.py: def _create_ovs_vif_cmd(bridge, dev, iface_id, mac,                         instance_id, interface_type=None,                         vhost_server_path=None): + # ODCN: initialize port as dead + # ODCN: TODO: set drop flow     cmd = ['--', '--if-exists', 'del-port', dev, '--',             'add-port', bridge, dev, + 'tag=4095',             '--', 'set', 'Interface', dev,             'external-ids:iface-id=%s' % iface_id,             'external-ids:iface-status=active',             'external-ids:attached-mac=%s' % mac,             'external-ids:vm-uuid=%s' % instance_id]     if interface_type:         cmd += ['type=%s' % interface_type]     if vhost_server_path:         cmd += ['options:vhost-server-path=%s' % vhost_server_path]     return cmd https://github.com/openstack/neutron/blob/stable/newton/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L995     def port_dead(self, port, log_errors=True):         '''Once a port has no binding, put it on the "dead vlan".         :param port: an ovs_lib.VifPort object.         '''         # Don't kill a port if it's already dead         cur_tag = self.int_br.db_get_val("Port", port.port_name, "tag",                                          log_errors=log_errors) + # ODCN GM 20170915 + if not cur_tag: + LOG.error('port_dead(): port %s has no tag', port.port_name) + # ODCN AJS 20170915 + if not cur_tag or cur_tag != constants.DEAD_VLAN_TAG: - if cur_tag and cur_tag != constants.DEAD_VLAN_TAG:            LOG.info('port_dead(): put port %s on dead vlan', port.port_name)            self.int_br.set_db_attribute("Port", port.port_name, "tag",                                          constants.DEAD_VLAN_TAG,                                          log_errors=log_errors)             self.int_br.drop_port(in_port=port.ofport) plugins/ml2/drivers/openvswitch/agent/openflow/ovs_ofctl/ovs_bridge.py     def drop_port(self, in_port): + # ODCN AJS 20171004: - self.install_drop(priority=2, in_port=in_port) + self.install_drop(priority=65535, in_port=in_port) Regards, ODC Noord. Gerhard Muntingh Albert Siersema Paul Peereboom To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1734320/+subscriptions From 1734320 at bugs.launchpad.net Thu May 30 17:19:55 2019 From: 1734320 at bugs.launchpad.net (OpenStack Infra) Date: Thu, 30 May 2019 17:19:55 -0000 Subject: [Openstack-security] [Bug 1734320] Change abandoned on neutron (master) References: <151152217834.14483.1577991310209811902.malonedeb@soybean.canonical.com> Message-ID: <155923679590.12528.7738284613466238999.malone@soybean.canonical.com> Change abandoned by sean mooney (smooney at redhat.com) on branch: master Review: https://review.opendev.org/635083 Reason: we have changed the direction somewhat so this is not needed in it current form. -- You received this bug notification because you are a member of OpenStack Security SIG, which is subscribed to OpenStack. https://bugs.launchpad.net/bugs/1734320 Title: Eavesdropping private traffic Status in neutron: Fix Committed Status in OpenStack Compute (nova): In Progress Status in os-vif: Fix Released Status in OpenStack Security Advisory: Won't Fix Bug description: Eavesdropping private traffic ============================= Abstract -------- We've discovered a security issue that allows end users within their own private network to receive from, and send traffic to, other private networks on the same compute node. Description ----------- During live-migration there is a small time window where the ports of instances are untagged. Instances have a port trunked to the integration bridge and receive 802.1Q tagged private traffic from other tenants. If the port is administratively down during live migration, the port will remain in trunk mode indefinitely. Traffic is possible between ports is that are administratively down, even between tenants self-service networks. Conditions ---------- The following conditions are necessary. * Openvswitch Self-service networks * An Openstack administrator or an automated process needs to schedule a Live migration We tested this on newton. Issues ------ This outcome is the result of multiple independent issues. We will list the most important first, and follow with bugs that create a fragile situation. Issue #1 Initially creating a trunk port When the port is initially created, it is in trunk mode. This creates a fail-open situation. See: https://github.com/openstack/os-vif/blob/newton-eol/vif_plug_ovs/linux_net.py#L52 Recommendation: create ports in the port_dead state, don't leave it dangling in trunk-mode. Add a drop-flow initially. Issue #2 Order of creation. The instance is actually migrated before the (networking) configuration is completed. Recommendation: wait with finishing the live migration until the underlying configuration has been applied completely. Issue #3 Not closing the port when it is down. Neutron calls the port_dead function to ensure the port is down. It sets the tag to 4095 and adds a "drop" flow if (and only if) there is already another tag on the port. The port_dead function will keep untagged ports untagged. https://github.com/openstack/neutron/blob/stable/newton/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L995 Recommendation: Make port_dead also shut the port if no tag is found. Log a warning if this happens. Issue #4 Putting the port administratively down actually puts the port on a compute node shared vlan Instances from different projects on different private networks can talk to each other if they put their ports down. The code does install an openflow "drop" rule but it has a lower priority (2) than the allow rules. Recommendation: Increase the port_dead openflow drop rule priority to MAX Timeline --------  2017-09-14 Discovery eavesdropping issue  2017-09-15 Verify workaround.  2017-10-04 Discovery port-down-traffic issue  2017-11-24 Vendor Disclosure to Openstack Steps to reproduce ------------------ 1. Attach an instance to two networks: admin$ openstack server create --nic net-id= --nic net-id = --image --flavor instance_temp 2. Attach a FIP to the instance to be able to log in to this instance 3. Verify: admin$ openstack server show -c name -c addresses fe28a2ee-098f-4425 -9d3c-8e2cd383547d +-----------+-----------------------------------------------------------------------------+ | Field | Value | +-----------+-----------------------------------------------------------------------------+ | addresses | network1=192.168.99.8, ; network2=192.168.80.14 | | name | instance_temp | +-----------+-----------------------------------------------------------------------------+ 4. Ssh to the instance using network1 and run a tcpdump on the other port network2 [root at instance_temp]$ tcpdump -eeenni eth1 5. Get port-id of network2 admin$ nova interface-list fe28a2ee-098f-4425-9d3c-8e2cd383547d +------------+--------------------------------------+--------------------------------------+---------------+-------------------+ | Port State | Port ID | Net ID | IP addresses | MAC Addr | +------------+--------------------------------------+--------------------------------------+---------------+-------------------+ | ACTIVE | a848520b-0814-4030-bb48-49e4b5cf8160 | d69028f7-9558-4f14-8ce6-29cb8f1c19cd | 192.168.80.14 | fa:16:3e:2d:8b:7b | | ACTIVE | fad148ca-cf7a-4839-aac3-a2cd8d1d2260 | d22c22ae-0a42-4e3b-8144-f28534c3439a | 192.168.99.8 | fa:16:3e:60:2c:fa | +------------+--------------------------------------+--------------------------------------+---------------+-------------------+ 6. Force port down on network 2 admin$ neutron port-update a848520b-0814-4030-bb48-49e4b5cf8160 --admin-state-up False 7. Port gets tagged with vlan 4095, the dead vlan tag, which is normal: compute1# grep a848520b-0814-4030-bb48-49e4b5cf8160 /var/log/neutron/neutron-openvswitch-agent.log | tail -1 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-e008feb3-8a35-4c97-adac-b48ff88165b2 - - - - -] VIF port: a848520b-0814-4030-bb48-49e4b5cf8160 admin state up disabled, putting on the dead VLAN 8. Verify the port is tagged with vlan 4095 compute1# ovs-vsctl show | grep -A3 qvoa848520b-08       Port "qvoa848520b-08"           tag: 4095           Interface "qvoa848520b-08" 9. Now live-migrate the instance: admin# nova live-migration fe28a2ee-098f-4425-9d3c-8e2cd383547d 10. Verify the tag is gone on compute2, and take a deep breath compute2# ovs-vsctl show | grep -A3 qvoa848520b-08       Port "qvoa848520b-08"           Interface "qvoa848520b-08"       Port... compute2# echo "Wut!" 11. Now traffic of all other self-service networks present on compute2 can be sniffed from instance_temp [root at instance_temp] tcpdump -eenni eth1 13:14:31.748266 fa:16:3e:6a:17:38 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.103.12.160 tell 10.103.12.152, length 28 13:14:31.804573 fa:16:3e:e8:a2:d2 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.70, length 28 13:14:31.810482 fa:16:3e:95:ca:3a > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.154, length 28 13:14:31.977820 fa:16:3e:6f:f4:9b > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.150, length 28 13:14:31.979590 fa:16:3e:0f:3d:cc > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 9, p 0, ethertype ARP, Request who-has 10.103.9.163 tell 10.103.9.1, length 28 13:14:32.048082 fa:16:3e:65:64:38 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.101, length 28 13:14:32.127400 fa:16:3e:30:cb:b5 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.103.12.160 tell 10.103.12.165, length 28 13:14:32.141982 fa:16:3e:96:cd:b0 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.100, length 28 13:14:32.205327 fa:16:3e:a2:0b:76 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.153, length 28 13:14:32.444142 fa:16:3e:1f:db:ed > 01:00:5e:00:00:12, ethertype 802.1Q (0x8100), length 58: vlan 72, p 0, ethertype IPv4, 192.168.99.212 > 224.0.0.18: VRRPv2, Advertisement, vrid 50, prio 103, authtype none, intvl 1s, length 20 13:14:32.449497 fa:16:3e:1c:24:c0 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.20, length 28 13:14:32.476015 fa:16:3e:f2:3b:97 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 33, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.22, length 28 13:14:32.575034 fa:16:3e:44:fe:35 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.103.12.160 tell 10.103.12.163, length 28 13:14:32.676185 fa:16:3e:1e:92:d7 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.103.12.160 tell 10.103.12.150, length 28 13:14:32.711755 fa:16:3e:99:6c:c8 > 01:00:5e:00:00:12, ethertype 802.1Q (0x8100), length 62: vlan 10, p 0, ethertype IPv4, 10.103.12.154 > 224.0.0.18: VRRPv2, Advertisement, vrid 2, prio 49, authtype simple, intvl 1s, length 24 13:14:32.711773 fa:16:3e:f5:23:d5 > 01:00:5e:00:00:12, ethertype 802.1Q (0x8100), length 58: vlan 12, p 0, ethertype IPv4, 10.103.15.154 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 49, authtype simple, intvl 1s, length 20 Workaround ---------- We temporary fixed this issue by forcing the dead vlan tag on port creation on compute nodes: /usr/lib/python2.7/site-packages/vif_plug_ovs/linux_net.py: def _create_ovs_vif_cmd(bridge, dev, iface_id, mac,                         instance_id, interface_type=None,                         vhost_server_path=None): + # ODCN: initialize port as dead + # ODCN: TODO: set drop flow     cmd = ['--', '--if-exists', 'del-port', dev, '--',             'add-port', bridge, dev, + 'tag=4095',             '--', 'set', 'Interface', dev,             'external-ids:iface-id=%s' % iface_id,             'external-ids:iface-status=active',             'external-ids:attached-mac=%s' % mac,             'external-ids:vm-uuid=%s' % instance_id]     if interface_type:         cmd += ['type=%s' % interface_type]     if vhost_server_path:         cmd += ['options:vhost-server-path=%s' % vhost_server_path]     return cmd https://github.com/openstack/neutron/blob/stable/newton/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L995     def port_dead(self, port, log_errors=True):         '''Once a port has no binding, put it on the "dead vlan".         :param port: an ovs_lib.VifPort object.         '''         # Don't kill a port if it's already dead         cur_tag = self.int_br.db_get_val("Port", port.port_name, "tag",                                          log_errors=log_errors) + # ODCN GM 20170915 + if not cur_tag: + LOG.error('port_dead(): port %s has no tag', port.port_name) + # ODCN AJS 20170915 + if not cur_tag or cur_tag != constants.DEAD_VLAN_TAG: - if cur_tag and cur_tag != constants.DEAD_VLAN_TAG:            LOG.info('port_dead(): put port %s on dead vlan', port.port_name)            self.int_br.set_db_attribute("Port", port.port_name, "tag",                                          constants.DEAD_VLAN_TAG,                                          log_errors=log_errors)             self.int_br.drop_port(in_port=port.ofport) plugins/ml2/drivers/openvswitch/agent/openflow/ovs_ofctl/ovs_bridge.py     def drop_port(self, in_port): + # ODCN AJS 20171004: - self.install_drop(priority=2, in_port=in_port) + self.install_drop(priority=65535, in_port=in_port) Regards, ODC Noord. Gerhard Muntingh Albert Siersema Paul Peereboom To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1734320/+subscriptions