[openstack-dev] [glance] [nova] Verification of glance images before boot
Flavio Percoco
flavio at redhat.com
Fri Sep 11 15:10:19 UTC 2015
On 11/09/15 10:49 -0400, Nikhil Komawar wrote:
>
>
>On 9/11/15 10:40 AM, Flavio Percoco wrote:
>> On 11/09/15 07:58 -0400, Nikhil Komawar wrote:
>>> You are right in the sense that's the ideal scenario.
>>>
>>> (Impl-wise) However, even today we do not guarantee that behavior. If
>>> someone
>>> were to propose a new driver or a change driver capability or any
>>> thing of such
>>> order, images in status killed won't be guaranteed to have removed
>>> the garbage
>>> data. The driver may not choose to be resilient enough or would not
>>> take the
>>> responsibility of data removal synchronously on failures.
>>
>> I think it's glance's responsibility to make sure the driver deletes
>> the image data. If the API is not strong enough to guarantee this,
>> then we should change that.
>>
>
>Sounds like a good direction but I still think we cannot commit to it.
>It really depends on the driver and introduction of new or upgrades to
>the existing drivers (if not their capabilities) is a not a good idea.
>This will result in backward incompatibility in many cases and there can
>be enough corner cases when we think about v1, v2 and tasks to address
>them at Glance API level.
mmh, I guess my reply was confusing again. I meant the glance_store
API and not Glance's API. What I'm trying to say is that just
depending on the scrubber (which not everyone deploys) won't be enough
to fix this issue entirely.
>
>I personally prefer the image-data delete to be asynchronous and fault
>tolerant (on DELETE of image record). However, even that cannot match
>the performance of a scrubber like service.
Yes, I also prefer the data to be deleted asynchrounously when it
comes to not blocking glance-api threads. However, that doesn't make
this step completely fault-tolerant.
What I'm saying is that depending on a asynchronous deletion only when
there's also the chance to possibility of having a synchrounous one is
not good.
>So, in certain cases we need to advice running the scrubber if the data
>has the tendency to be left behind.
This is where we difer in thoughts. I understand where you're coming
from with this but I think we shouldn't have those cases. The fact
that there's data being left behind is the real issue. We shouldn't
have those cases.
Having async deletions is fine and I agree they are useful.
Cheers,
Flavio
>
>>> Taking that fact in account, I have thought of Brianna's patch to be
>>> okay.
>>
>> Oh sure, I'm not trying to say it was a wrong choice. Sorry if it
>> sounded like that. I was replying to the thought of extending scrubber
>> (unless there's a patch that does this that I might have missed).
>>
>> Cheers,
>> Flavio
>>
>>>
>>> On 9/11/15 4:42 AM, Flavio Percoco wrote:
>>>
>>> On 10/09/15 15:36 -0400, Nikhil Komawar wrote:
>>>
>>> The solution to this problem is to improve the scrubber to
>>> clean up the
>>> garbage data left behind in the backend store during such failed
>>> uploads.
>>>
>>> Currently, scrubber cleans up images in pending_delete and
>>> extending
>>> that to images in killed status would avoid such a situation.
>>>
>>>
>>> While the above would certainly help, I think it's not the right
>>> solution. Images in status "killed" should not have data to begin
>>> with.
>>>
>>> I'd rather find a way to clean that data as soon as the image is
>>> moved to a "killed" state instead of extending the scrubber.
>>>
>>> Cheers,
>>> Flavio
>>>
>>>
>>> On 9/10/15 3:28 PM, Poulos, Brianna L. wrote:
>>>
>>> Malini,
>>>
>>> Thank you for bringing up the ³killed² state as it
>>> relates to
>>> quota. We
>>> opted to move the image to a killed state since that is
>>> what occurs
>>> when
>>> an upload fails, and the signature verification failure
>>> would occur
>>> during
>>> an upload. But we should keep in mind the potential to
>>> take up
>>> space and
>>> yet not take up quota when signature verification fails.
>>>
>>> Regarding the MD5 hash, there is currently a glance spec
>>> [1] to
>>> allow the
>>> hash method used for the checksum to be
>>> configurable‹currently it
>>> is
>>> hardcoded in glance. After making it configurable, the
>>> default
>>> would
>>> transition from MD5 to something more secure (like SHA-256).
>>>
>>> [1] https://review.openstack.org/#/c/191542/
>>>
>>> Thanks,
>>> ~Brianna
>>>
>>>
>>>
>>>
>>> On 9/10/15, 5:10 , "Bhandaru, Malini K"
>>> <malini.k.bhandaru at intel.com>
>>> wrote:
>>>
>>>
>>> Brianna, I can imagine a denial of service attack by
>>> uploading
>>> images
>>> whose signature is invalid if we allow them to reside
>>> in Glance
>>> In a "killed" state. This would be less of an issue
>>> "killed"
>>> images still
>>> consume storage quota until actually deleted.
>>> Also given MD-5 less secure, why not have the default
>>> hash be
>>> SHA-1 or 2?
>>> Regards
>>> Malini
>>>
>>> -----Original Message-----
>>> From: Poulos, Brianna L.
>>> [mailto:Brianna.Poulos at jhuapl.edu]
>>> Sent: Wednesday, September 09, 2015 9:54 AM
>>> To: OpenStack Development Mailing List (not for usage
>>> questions)
>>> Cc: stuart.mclaren at hp.com
>>> Subject: Re: [openstack-dev] [glance] [nova]
>>> Verification of
>>> glance
>>> images before boot
>>>
>>> Stuart is right about what will currently happen in
>>> Nova when
>>> an image is
>>> downloaded, which protects against unintentional
>>> modifications
>>> to the
>>> image data.
>>>
>>> What is currently being worked on is adding the
>>> ability to
>>> verify a
>>> signature of the checksum. The flow of this is as
>>> follows:
>>> 1. The user creates a signature of the "checksum hash"
>>> (currently MD5) of
>>> the image data offline.
>>> 2. The user uploads a public key certificate, which
>>> can be used
>>> to verify
>>> the signature to a key manager (currently Barbican).
>>> 3. The user creates an image in glance, with signature
>>> metadata
>>> properties.
>>> 4. The user uploads the image data to glance.
>>> 5. If the signature metadata properties exist, glance
>>> verifies
>>> the
>>> signature of the "checksum hash", including retrieving
>>> the
>>> certificate
>>>
>>> >from the key manager.
>>>
>>> 6. If the signature verification fails, glance moves
>>> the image
>>> to a
>>> killed state, and returns an error message to the user.
>>> 7. If the signature verification succeeds, a log message
>>> indicates that
>>> it succeeded, and the image upload finishes successfully.
>>>
>>> 8. Nova requests the image from glance, along with the
>>> image
>>> properties,
>>> in order to boot it.
>>> 9. Nova uses the signature metadata properties to
>>> verify the
>>> signature
>>> (if a configuration option is set).
>>> 10. If the signature verification fails, nova does not
>>> boot the
>>> image,
>>> but errors out.
>>> 11. If the signature verification succeeds, nova boots
>>> the
>>> image, and a
>>> log message notes that the verification succeeded.
>>>
>>> Regarding what is currently in Liberty, the blueprint
>>> mentioned
>>> [1] has
>>> merged, and code [2] has also been merged in glance,
>>> which
>>> handles steps
>>> 1-7 of the flow above.
>>>
>>> For steps 7-11, there is currently a nova blueprint
>>> [3], along
>>> with code
>>> [4], which are proposed for Mitaka.
>>>
>>> Note that we are in the process of adding official
>>> documentation, with
>>> examples of creating the signature as well as the
>>> properties
>>> that need to
>>> be added for the image before upload. In the
>>> meantime, there's
>>> an
>>> etherpad that describes how to test the signature
>>> verification
>>> functionality in Glance [5].
>>>
>>> Also note that this is the initial approach, and there
>>> are some
>>> limitations. For example, ideally the signature would
>>> be based
>>> on a
>>> cryptographically secure (i.e. not MD5) hash of the
>>> image.
>>> There is a
>>> spec in glance to allow this hash to be configurable [6].
>>>
>>> [1]
>>> https://blueprints.launchpad.net/glance/+spec/
>>> image-signing-and-verificati
>>> o
>>> n-support
>>> [2]
>>> https://github.com/openstack/glance/commit/
>>> 484ef1b40b738c87adb203bba6107dd
>>> b
>>> 4b04ff6e
>>> [3] https://review.openstack.org/#/c/188874/
>>> [4] https://review.openstack.org/#/c/189843/
>>> [5]
>>> https://etherpad.openstack.org/p/
>>> liberty-glance-image-signing-instructions
>>> [6] https://review.openstack.org/#/c/191542/
>>>
>>>
>>> Thanks,
>>> ~Brianna
>>>
>>>
>>>
>>>
>>> On 9/9/15, 12:16 , "Nikhil Komawar"
>>> <nik.komawar at gmail.com>
>>> wrote:
>>>
>>>
>>> That's correct.
>>>
>>> The size and the checksum are to be verified
>>> outside of
>>> Glance, in this
>>> case Nova. However, you may want to note that it's
>>> not
>>> necessary that
>>> all Nova virt drivers would use py-glanceclient so
>>> you
>>> would want to
>>> check the download specific code in the virt
>>> driver your
>>> Nova
>>> deployment is using.
>>>
>>> Having said that, essentially the flow seems
>>> appropriate.
>>> Error must be
>>> raise on mismatch.
>>>
>>> The signing BP was to help prevent the compromised
>>> Glance
>>> from changing
>>> the checksum and image blob at the same time. Using a
>>> digital
>>> signature, you can prevent download of compromised
>>> data.
>>> However, the
>>> feature has just been implemented in Glance;
>>> Glance users
>>> may take time
>>> to adopt.
>>>
>>>
>>>
>>> On 9/9/15 11:15 AM, stuart.mclaren at hp.com wrote:
>>>
>>> The glance client (running 'inside' the Nova
>>> server)
>>> will
>>> re-calculate the checksum as it downloads the
>>> image and
>>> then compare
>>> it against the expected value. If they don't
>>> match an
>>> error will be
>>> raised.
>>>
>>>
>>> How can I know that the image that a new
>>> instance
>>> is spawned from -
>>> is actually the image that was originally
>>> registered in glance - and
>>> has not been maliciously tampered with in
>>> some way?
>>>
>>> Is there some kind of verification that is
>>> performed against the
>>> md5sum of the registered image in glance
>>> before a
>>> new instance is
>>> spawned?
>>>
>>> Is that done by Nova?
>>> Glance?
>>> Both? Neither?
>>>
>>> The reason I ask is some 'paranoid'
>>> security (that
>>> is their job I
>>> suppose) people have raised these questions.
>>>
>>> I know there is a glance BP already merged
>>> for L
>>> [1] - but I would
>>> like to understand the actual flow in a
>>> bit more
>>> detail.
>>>
>>> Thanks.
>>>
>>> [1]
>>>
>>>
>>> https://blueprints.launchpad.net/glance/+spec/
>>> image-signing-and-verif
>>> ica
>>> tion-support
>>>
>>>
>>> --
>>> Best Regards,
>>> Maish Saidel-Keesing
>>>
>>>
>>>
>>> ------------------------------
>>>
>>>
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>>
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo
>>> /openstack-dev
>>>
>>>
>>> End of OpenStack-dev Digest, Vol 41, Issue 22
>>> *********************************************
>>>
>>>
>>>
>>>
>>> ______________________________________________________________________
>>>
>>> ___
>>> _
>>>
>>> OpenStack Development Mailing List (not for usage
>>> questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?
>>> subject:unsubscribe
>>>
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/
>>> openstack-dev
>>>
>>> --
>>>
>>> Thanks,
>>> Nikhil
>>>
>>>
>>>
>>> _______________________________________________________________________
>>>
>>> ___ OpenStack Development Mailing List (not for usage
>>> questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?
>>> subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/
>>> openstack-dev
>>>
>>>
>>>
>>> __________________________________________________________________________
>>>
>>> OpenStack Development Mailing List (not for usage
>>> questions)
>>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?
>>> subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/
>>> openstack-dev
>>>
>>>
>>> __________________________________________________________________________
>>>
>>> OpenStack Development Mailing List (not for usage
>>> questions)
>>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?
>>> subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/
>>> openstack-dev
>>>
>>>
>>>
>>> __________________________________________________________________________
>>>
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?
>>> subject:unsubscribe
>>>
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>> --
>>>
>>> Thanks,
>>> Nikhil
>>>
>>>
>>>
>>> __________________________________________________________________________
>>>
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?
>>> subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>>
>>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>> --
>>>
>>> Thanks,
>>> Nikhil
>>>
>>
>
>--
>
>Thanks,
>Nikhil
>
--
@flaper87
Flavio Percoco
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150911/7b6bffa3/attachment.pgp>
More information about the OpenStack-dev
mailing list