[openstack-dev] [nova] How to properly detect and fence a compromised host (and why I dislike TrustedFilter)
shane.wang at intel.com
Wed Jun 24 04:25:50 UTC 2015
AFAIK, TrustedFilter is using a sort of cache to cache the trusted state, which is designed to solve the performance issue mentioned here.
My thoughts for deprecating it are:
#1. We already have customers here in China who are using that filter. How are they going to do upgrade in the future?
#2. Dependency should not be a reason to deprecate a module in OpenStack, Nova is not a stand-alone module, and it depends on various technologies and libraries.
Intel is setting up the third party CI for TCP/OAT in Liberty, which is to address the concerns mentioned in the thread. And also, OAT is an open source project which is being maintained as the long-term strategy.
For the situation that a host gets compromised, OAT checks trusted or untrusted from the start point of boot/reboot, it is hard for OAT to detect whether a host gets compromised when it is running, I don't know how to detect that without the filter?
Back to Michael's question, the process of the verification is done by software automatically when a host boots or reboots, will that be an overhead for the admin to have a separate job?
From: Michael Still [mailto:mikal at stillhq.com]
Sent: Wednesday, June 24, 2015 7:49 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [nova] How to properly detect and fence a compromised host (and why I dislike TrustedFilter)
I agree. I feel like this is another example of functionality which is trivially implemented outside nova, and where it works much better if we don't do it. Couldn't an admin just have a cron job which verifies hosts, and then adds them to a compromised-hosts host aggregate if they're owned? I assume without testing it that you can migrate instances _out_ of a host aggregate you can't boot in?
On Tue, Jun 23, 2015 at 8:41 PM, Sylvain Bauza <sbauza at redhat.com> wrote:
> Hi team,
> Some discussion occurred over IRC about a bug which was publicly open
> related to TrustedFilter  I want to take the opportunity for
> raising my concerns about that specific filter, why I dislike it and
> how I think we could improve the situation - and clarify everyone's
> The current situation is that way : Nova only checks if one host is
> compromised only when the scheduler is called, ie. only when
> booting/migrating/evacuating/unshelving an instance (well, not exactly
> all the evacuate/live-migrate cases, but let's not discuss about that
> now). When the request goes in the scheduler, all the hosts are
> checked against all the enabled filters and the TrustedFilter is
> making an external HTTP(S) call to the Attestation API service (not
> handled by Nova) for *each host* to see if the host is valid (not compromised) or not.
> To be clear, that's the only in-tree scheduler filter which explicitly
> does an external call to a separate service that Nova is not managing.
> I can see at least 3 reasons for thinking about why it's bad :
> #1 : that's a terrible bottleneck for performance, because we're
> IO-blocking N times given N hosts (we're even not multiplexing the
> HTTP requests)
> #2 : all the filters are checking an internal Nova state for the host
> (called HostState) but that the TrustedFilter, which means that
> conceptually we defer the decision to a 3rd-party engine
> #3 : that Attestation API services becomes a de facto dependency for
> Nova (since it's an in-tree filter) while it's not listed as a
> dependency and thus not gated.
> All of these reasons could be acceptable if that would cover the
> exposed usecase given in  (ie. I want to make sure that if my host
> gets compromised, my instances will not be running on that host) but
> that just doesn't work, due to the situation I mentioned above.
> So, given that, here are my thoughts :
> a/ if a host gets compromised, we can just disable its service to
> prevent its election as a valid destination host. There is no need for
> a specialised filter.
> b/ if a host is compromised, we can assume that the instances have to
> resurrect elsewhere, ie. we can call a nova evacuate c/ checking if an
> host is compromised or not is not a Nova responsibility since it's
> already perfectly done by 
> In other words, I'm considering that "security" usecase as something
> analog as the HA usecase  where we need a 3rd-party tool
> responsible for periodically checking the state of the hosts, and if
> compromised then call the Nova API for fencing the host and evacuating the compromised instances.
> Given that, I'm proposing to deprecate TrustedFilter and explictly
> mention to drop it from in-tree in a later cycle
> Thoughts ?
>  https://bugs.launchpad.net/nova/+bug/1456228
>  https://github.com/OpenAttestation/OpenAttestation
> ____ OpenStack Development Mailing List (not for usage questions)
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
More information about the OpenStack-dev