[openstack-dev] [glance] [nova] Verification of glance images before boot

Flavio Percoco flavio at redhat.com
Fri Sep 11 14:40:23 UTC 2015


On 11/09/15 07:58 -0400, Nikhil Komawar wrote:
>You are right in the sense that's the ideal scenario.
>
>(Impl-wise) However, even today we do not guarantee that behavior. If someone
>were to propose a new driver or a change driver capability or any thing of such
>order, images in status killed won't be guaranteed to have removed the garbage
>data. The driver may not choose to be resilient enough or would not take the
>responsibility of data removal synchronously on failures.

I think it's glance's responsibility to make sure the driver deletes
the image data. If the API is not strong enough to guarantee this,
then we should change that.

>Taking that fact in account, I have thought of Brianna's patch to be okay.

Oh sure, I'm not trying to say it was a wrong choice. Sorry if it
sounded like that. I was replying to the thought of extending scrubber
(unless there's a patch that does this that I might have missed).

Cheers,
Flavio

>
>On 9/11/15 4:42 AM, Flavio Percoco wrote:
>
>    On 10/09/15 15:36 -0400, Nikhil Komawar wrote:
>
>        The solution to this problem is to improve the scrubber to clean up the
>        garbage data left behind in the backend store during such failed
>        uploads.
>
>        Currently, scrubber cleans up images in pending_delete and extending
>        that to images in killed status would avoid such a situation.
>
>
>    While the above would certainly help, I think it's not the right
>    solution. Images in status "killed" should not have data to begin
>    with.
>
>    I'd rather find a way to clean that data as soon as the image is
>    moved to a "killed" state instead of extending the scrubber.
>
>    Cheers,
>    Flavio
>
>
>        On 9/10/15 3:28 PM, Poulos, Brianna L. wrote:
>
>            Malini,
>
>            Thank you for bringing up the ³killed² state as it relates to
>            quota.  We
>            opted to move the image to a killed state since that is what occurs
>            when
>            an upload fails, and the signature verification failure would occur
>            during
>            an upload.  But we should keep in mind the potential to take up
>            space and
>            yet not take up quota when signature verification fails.
>
>            Regarding the MD5 hash, there is currently a glance spec [1] to
>            allow the
>            hash method used for the checksum to be configurable‹currently it
>            is
>            hardcoded in glance.  After making it configurable, the default
>            would
>            transition from MD5 to something more secure (like SHA-256).
>
>            [1] https://review.openstack.org/#/c/191542/
>
>            Thanks,
>            ~Brianna
>
>
>
>
>            On 9/10/15, 5:10 , "Bhandaru, Malini K"
>            <malini.k.bhandaru at intel.com>
>            wrote:
>
>
>                Brianna, I can imagine a denial of service attack by uploading
>                images
>                whose signature is invalid if we allow them to reside in Glance
>                In a "killed" state. This would be less of an issue "killed"
>                images still
>                consume storage quota until actually deleted.
>                Also given MD-5 less secure, why not have the default hash be
>                SHA-1 or 2?
>                Regards
>                Malini
>
>                -----Original Message-----
>                From: Poulos, Brianna L. [mailto:Brianna.Poulos at jhuapl.edu]
>                Sent: Wednesday, September 09, 2015 9:54 AM
>                To: OpenStack Development Mailing List (not for usage
>                questions)
>                Cc: stuart.mclaren at hp.com
>                Subject: Re: [openstack-dev] [glance] [nova] Verification of
>                glance
>                images before boot
>
>                Stuart is right about what will currently happen in Nova when
>                an image is
>                downloaded, which protects against unintentional modifications
>                to the
>                image data.
>
>                What is currently being worked on is adding the ability to
>                verify a
>                signature of the checksum.  The flow of this is as follows:
>                1. The user creates a signature of the "checksum hash"
>                (currently MD5) of
>                the image data offline.
>                2. The user uploads a public key certificate, which can be used
>                to verify
>                the signature to a key manager (currently Barbican).
>                3. The user creates an image in glance, with signature metadata
>                properties.
>                4. The user uploads the image data to glance.
>                5. If the signature metadata properties exist, glance verifies
>                the
>                signature of the "checksum hash", including retrieving the
>                certificate
>
>            >from the key manager.
>
>                6. If the signature verification fails, glance moves the image
>                to a
>                killed state, and returns an error message to the user.
>                7. If the signature verification succeeds, a log message
>                indicates that
>                it succeeded, and the image upload finishes successfully.
>
>                8. Nova requests the image from glance, along with the image
>                properties,
>                in order to boot it.
>                9. Nova uses the signature metadata properties to verify the
>                signature
>                (if a configuration option is set).
>                10. If the signature verification fails, nova does not boot the
>                image,
>                but errors out.
>                11. If the signature verification succeeds, nova boots the
>                image, and a
>                log message notes that the verification succeeded.
>
>                Regarding what is currently in Liberty, the blueprint mentioned
>                [1] has
>                merged, and code [2] has also been merged in glance, which
>                handles steps
>                1-7 of the flow above.
>
>                For steps 7-11, there is currently a nova blueprint [3], along
>                with code
>                [4], which are proposed for Mitaka.
>
>                Note that we are in the process of adding official
>                documentation, with
>                examples of creating the signature as well as the properties
>                that need to
>                be added for the image before upload.  In the meantime, there's
>                an
>                etherpad that describes how to test the signature verification
>                functionality in Glance [5].
>
>                Also note that this is the initial approach, and there are some
>                limitations.  For example, ideally the signature would be based
>                on a
>                cryptographically secure (i.e. not MD5) hash of the image. 
>                There is a
>                spec in glance to allow this hash to be configurable [6].
>
>                [1]
>                https://blueprints.launchpad.net/glance/+spec/
>                image-signing-and-verificati
>                o
>                n-support
>                [2]
>                https://github.com/openstack/glance/commit/
>                484ef1b40b738c87adb203bba6107dd
>                b
>                4b04ff6e
>                [3] https://review.openstack.org/#/c/188874/
>                [4] https://review.openstack.org/#/c/189843/
>                [5]
>                https://etherpad.openstack.org/p/
>                liberty-glance-image-signing-instructions
>                [6] https://review.openstack.org/#/c/191542/
>
>
>                Thanks,
>                ~Brianna
>
>
>
>
>                On 9/9/15, 12:16 , "Nikhil Komawar" <nik.komawar at gmail.com>
>                wrote:
>
>
>                    That's correct.
>
>                    The size and the checksum are to be verified outside of
>                    Glance, in this
>                    case Nova. However, you may want to note that it's not
>                    necessary that
>                    all Nova virt drivers would use py-glanceclient so you
>                    would want to
>                    check the download specific code in the virt driver your
>                    Nova
>                    deployment is using.
>
>                    Having said that, essentially the flow seems appropriate.
>                    Error must be
>                    raise on mismatch.
>
>                    The signing BP was to help prevent the compromised Glance
>                    from changing
>                    the checksum and image blob at the same time. Using a
>                    digital
>                    signature, you can prevent download of compromised data.
>                    However, the
>                    feature has just been implemented in Glance; Glance users
>                    may take time
>                    to adopt.
>
>
>
>                    On 9/9/15 11:15 AM, stuart.mclaren at hp.com wrote:
>
>                        The glance client (running 'inside' the Nova server)
>                        will
>                        re-calculate the checksum as it downloads the image and
>                        then compare
>                        it against the expected value. If they don't match an
>                        error will be
>                        raised.
>
>
>                            How can I know that the image that a new instance
>                            is spawned from -
>                            is actually the image that was originally
>                            registered in glance - and
>                            has not been maliciously tampered with in some way?
>
>                            Is there some kind of verification that is
>                            performed against the
>                            md5sum of the registered image in glance before a
>                            new instance is
>                            spawned?
>
>                            Is that done by Nova?
>                            Glance?
>                            Both? Neither?
>
>                            The reason I ask is some 'paranoid' security (that
>                            is their job I
>                            suppose) people have raised these questions.
>
>                            I know there is a glance BP already merged for L
>                            [1] - but I would
>                            like to understand the actual flow in a bit more
>                            detail.
>
>                            Thanks.
>
>                            [1]
>
>                            https://blueprints.launchpad.net/glance/+spec/
>                            image-signing-and-verif
>                            ica
>                            tion-support
>
>
>                            --
>                            Best Regards,
>                            Maish Saidel-Keesing
>
>
>
>                            ------------------------------
>
>                            _______________________________________________
>                            OpenStack-dev mailing list
>                            OpenStack-dev at lists.openstack.org
>                            http://lists.openstack.org/cgi-bin/mailman/listinfo
>                            /openstack-dev
>
>
>                            End of OpenStack-dev Digest, Vol 41, Issue 22
>                            *********************************************
>
>
>
>                        ______________________________________________________________________
>
>                        ___
>                        _
>
>                        OpenStack Development Mailing List (not for usage
>                        questions)
>                        Unsubscribe:
>                        OpenStack-dev-request at lists.openstack.org?
>                        subject:unsubscribe
>                        http://lists.openstack.org/cgi-bin/mailman/listinfo/
>                        openstack-dev
>
>                    --
>
>                    Thanks,
>                    Nikhil
>
>
>                    _______________________________________________________________________
>
>                    ___ OpenStack Development Mailing List (not for usage
>                    questions)
>                    Unsubscribe:
>                    OpenStack-dev-request at lists.openstack.org?
>                    subject:unsubscribe
>                    http://lists.openstack.org/cgi-bin/mailman/listinfo/
>                    openstack-dev
>
>
>                __________________________________________________________________________
>
>                OpenStack Development Mailing List (not for usage questions)
>                Unsubscribe: OpenStack-dev-request at lists.openstack.org?
>                subject:unsubscribe
>                http://lists.openstack.org/cgi-bin/mailman/listinfo/
>                openstack-dev
>
>                __________________________________________________________________________
>
>                OpenStack Development Mailing List (not for usage questions)
>                Unsubscribe: OpenStack-dev-request at lists.openstack.org?
>                subject:unsubscribe
>                http://lists.openstack.org/cgi-bin/mailman/listinfo/
>                openstack-dev
>
>
>            __________________________________________________________________________
>
>            OpenStack Development Mailing List (not for usage questions)
>            Unsubscribe: OpenStack-dev-request at lists.openstack.org?
>            subject:unsubscribe
>            http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>        -- 
>
>        Thanks,
>        Nikhil
>
>
>        __________________________________________________________________________
>
>        OpenStack Development Mailing List (not for usage questions)
>        Unsubscribe: OpenStack-dev-request at lists.openstack.org?
>        subject:unsubscribe
>        http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>   
>    __________________________________________________________________________
>    OpenStack Development Mailing List (not for usage questions)
>    Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>    http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>--
>
>Thanks,
>Nikhil
>

-- 
@flaper87
Flavio Percoco
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150911/e4a28d87/attachment.pgp>


More information about the OpenStack-dev mailing list