[openstack-dev] [glance] [nova] Verification of glance images before boot
Flavio Percoco
flavio at redhat.com
Fri Sep 11 14:40:23 UTC 2015
On 11/09/15 07:58 -0400, Nikhil Komawar wrote:
>You are right in the sense that's the ideal scenario.
>
>(Impl-wise) However, even today we do not guarantee that behavior. If someone
>were to propose a new driver or a change driver capability or any thing of such
>order, images in status killed won't be guaranteed to have removed the garbage
>data. The driver may not choose to be resilient enough or would not take the
>responsibility of data removal synchronously on failures.
I think it's glance's responsibility to make sure the driver deletes
the image data. If the API is not strong enough to guarantee this,
then we should change that.
>Taking that fact in account, I have thought of Brianna's patch to be okay.
Oh sure, I'm not trying to say it was a wrong choice. Sorry if it
sounded like that. I was replying to the thought of extending scrubber
(unless there's a patch that does this that I might have missed).
Cheers,
Flavio
>
>On 9/11/15 4:42 AM, Flavio Percoco wrote:
>
> On 10/09/15 15:36 -0400, Nikhil Komawar wrote:
>
> The solution to this problem is to improve the scrubber to clean up the
> garbage data left behind in the backend store during such failed
> uploads.
>
> Currently, scrubber cleans up images in pending_delete and extending
> that to images in killed status would avoid such a situation.
>
>
> While the above would certainly help, I think it's not the right
> solution. Images in status "killed" should not have data to begin
> with.
>
> I'd rather find a way to clean that data as soon as the image is
> moved to a "killed" state instead of extending the scrubber.
>
> Cheers,
> Flavio
>
>
> On 9/10/15 3:28 PM, Poulos, Brianna L. wrote:
>
> Malini,
>
> Thank you for bringing up the ³killed² state as it relates to
> quota. We
> opted to move the image to a killed state since that is what occurs
> when
> an upload fails, and the signature verification failure would occur
> during
> an upload. But we should keep in mind the potential to take up
> space and
> yet not take up quota when signature verification fails.
>
> Regarding the MD5 hash, there is currently a glance spec [1] to
> allow the
> hash method used for the checksum to be configurable‹currently it
> is
> hardcoded in glance. After making it configurable, the default
> would
> transition from MD5 to something more secure (like SHA-256).
>
> [1] https://review.openstack.org/#/c/191542/
>
> Thanks,
> ~Brianna
>
>
>
>
> On 9/10/15, 5:10 , "Bhandaru, Malini K"
> <malini.k.bhandaru at intel.com>
> wrote:
>
>
> Brianna, I can imagine a denial of service attack by uploading
> images
> whose signature is invalid if we allow them to reside in Glance
> In a "killed" state. This would be less of an issue "killed"
> images still
> consume storage quota until actually deleted.
> Also given MD-5 less secure, why not have the default hash be
> SHA-1 or 2?
> Regards
> Malini
>
> -----Original Message-----
> From: Poulos, Brianna L. [mailto:Brianna.Poulos at jhuapl.edu]
> Sent: Wednesday, September 09, 2015 9:54 AM
> To: OpenStack Development Mailing List (not for usage
> questions)
> Cc: stuart.mclaren at hp.com
> Subject: Re: [openstack-dev] [glance] [nova] Verification of
> glance
> images before boot
>
> Stuart is right about what will currently happen in Nova when
> an image is
> downloaded, which protects against unintentional modifications
> to the
> image data.
>
> What is currently being worked on is adding the ability to
> verify a
> signature of the checksum. The flow of this is as follows:
> 1. The user creates a signature of the "checksum hash"
> (currently MD5) of
> the image data offline.
> 2. The user uploads a public key certificate, which can be used
> to verify
> the signature to a key manager (currently Barbican).
> 3. The user creates an image in glance, with signature metadata
> properties.
> 4. The user uploads the image data to glance.
> 5. If the signature metadata properties exist, glance verifies
> the
> signature of the "checksum hash", including retrieving the
> certificate
>
> >from the key manager.
>
> 6. If the signature verification fails, glance moves the image
> to a
> killed state, and returns an error message to the user.
> 7. If the signature verification succeeds, a log message
> indicates that
> it succeeded, and the image upload finishes successfully.
>
> 8. Nova requests the image from glance, along with the image
> properties,
> in order to boot it.
> 9. Nova uses the signature metadata properties to verify the
> signature
> (if a configuration option is set).
> 10. If the signature verification fails, nova does not boot the
> image,
> but errors out.
> 11. If the signature verification succeeds, nova boots the
> image, and a
> log message notes that the verification succeeded.
>
> Regarding what is currently in Liberty, the blueprint mentioned
> [1] has
> merged, and code [2] has also been merged in glance, which
> handles steps
> 1-7 of the flow above.
>
> For steps 7-11, there is currently a nova blueprint [3], along
> with code
> [4], which are proposed for Mitaka.
>
> Note that we are in the process of adding official
> documentation, with
> examples of creating the signature as well as the properties
> that need to
> be added for the image before upload. In the meantime, there's
> an
> etherpad that describes how to test the signature verification
> functionality in Glance [5].
>
> Also note that this is the initial approach, and there are some
> limitations. For example, ideally the signature would be based
> on a
> cryptographically secure (i.e. not MD5) hash of the image.
> There is a
> spec in glance to allow this hash to be configurable [6].
>
> [1]
> https://blueprints.launchpad.net/glance/+spec/
> image-signing-and-verificati
> o
> n-support
> [2]
> https://github.com/openstack/glance/commit/
> 484ef1b40b738c87adb203bba6107dd
> b
> 4b04ff6e
> [3] https://review.openstack.org/#/c/188874/
> [4] https://review.openstack.org/#/c/189843/
> [5]
> https://etherpad.openstack.org/p/
> liberty-glance-image-signing-instructions
> [6] https://review.openstack.org/#/c/191542/
>
>
> Thanks,
> ~Brianna
>
>
>
>
> On 9/9/15, 12:16 , "Nikhil Komawar" <nik.komawar at gmail.com>
> wrote:
>
>
> That's correct.
>
> The size and the checksum are to be verified outside of
> Glance, in this
> case Nova. However, you may want to note that it's not
> necessary that
> all Nova virt drivers would use py-glanceclient so you
> would want to
> check the download specific code in the virt driver your
> Nova
> deployment is using.
>
> Having said that, essentially the flow seems appropriate.
> Error must be
> raise on mismatch.
>
> The signing BP was to help prevent the compromised Glance
> from changing
> the checksum and image blob at the same time. Using a
> digital
> signature, you can prevent download of compromised data.
> However, the
> feature has just been implemented in Glance; Glance users
> may take time
> to adopt.
>
>
>
> On 9/9/15 11:15 AM, stuart.mclaren at hp.com wrote:
>
> The glance client (running 'inside' the Nova server)
> will
> re-calculate the checksum as it downloads the image and
> then compare
> it against the expected value. If they don't match an
> error will be
> raised.
>
>
> How can I know that the image that a new instance
> is spawned from -
> is actually the image that was originally
> registered in glance - and
> has not been maliciously tampered with in some way?
>
> Is there some kind of verification that is
> performed against the
> md5sum of the registered image in glance before a
> new instance is
> spawned?
>
> Is that done by Nova?
> Glance?
> Both? Neither?
>
> The reason I ask is some 'paranoid' security (that
> is their job I
> suppose) people have raised these questions.
>
> I know there is a glance BP already merged for L
> [1] - but I would
> like to understand the actual flow in a bit more
> detail.
>
> Thanks.
>
> [1]
>
> https://blueprints.launchpad.net/glance/+spec/
> image-signing-and-verif
> ica
> tion-support
>
>
> --
> Best Regards,
> Maish Saidel-Keesing
>
>
>
> ------------------------------
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo
> /openstack-dev
>
>
> End of OpenStack-dev Digest, Vol 41, Issue 22
> *********************************************
>
>
>
> ______________________________________________________________________
>
> ___
> _
>
> OpenStack Development Mailing List (not for usage
> questions)
> Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?
> subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack-dev
>
> --
>
> Thanks,
> Nikhil
>
>
> _______________________________________________________________________
>
> ___ OpenStack Development Mailing List (not for usage
> questions)
> Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?
> subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack-dev
>
>
> __________________________________________________________________________
>
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?
> subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack-dev
>
> __________________________________________________________________________
>
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?
> subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack-dev
>
>
> __________________________________________________________________________
>
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?
> subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> --
>
> Thanks,
> Nikhil
>
>
> __________________________________________________________________________
>
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?
> subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>--
>
>Thanks,
>Nikhil
>
--
@flaper87
Flavio Percoco
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150911/e4a28d87/attachment.pgp>
More information about the OpenStack-dev
mailing list