[openstack-dev] [glance] [nova] Glance bug with Kilo upgrade & Nova

Flavio Percoco flavio at redhat.com
Tue Jun 9 12:33:17 UTC 2015


On 09/06/15 07:38 +0000, Bhandaru, Malini K wrote:
>A DB migrate script that puts some token default string only for glance properties that are NULL. Should not change anything else.
>Hope it works Flavio.
>Regards
>Malini

Proposed fix, please read the commit message and comments:

https://review.openstack.org/#/c/189661/

>
>-----Original Message-----
>From: Flavio Percoco [mailto:flavio at redhat.com]
>Sent: Tuesday, June 09, 2015 12:33 AM
>To: Bhandaru, Malini K
>Cc: OpenStack Development Mailing List (not for usage questions); openstack-operators at lists.openstack.org
>Subject: Re: [openstack-dev] [glance] [nova] Glance bug with Kilo upgrade & Nova
>
>On 09/06/15 09:22 +0200, Flavio Percoco wrote:
>>On 09/06/15 07:09 +0000, Bhandaru, Malini K wrote:
>>>Flavio, would a DB script that writes an empty string or NOP or something instead of NULL In the column do the trick?
>>>Then the problem degenerates to a new DB upgrade script.
>>
>>I now remembered about a change[0] - that I wrote myself - that
>>required a bump on the API version - which we did - that allows None
>>values to be returned in the API. This is probably what is causing this
>>behavior.
>>
>>A DB migration would certainly work, I'd love to avoid it but I guess
>>that's the best solution in this case.
>
>Actually, we can't just migrate the database as there might be custom properties that were explicitly set to None.
>
>I'll keep you posted
>
>>
>>Cheers,
>>Flavio
>>
>>[0] https://review.openstack.org/#/c/138184/
>>
>>>
>>>Regards
>>>Malini
>>>
>>>-----Original Message-----
>>>From: Flavio Percoco [mailto:flavio at redhat.com]
>>>Sent: Monday, June 08, 2015 11:47 PM
>>>To: OpenStack Development Mailing List (not for usage questions)
>>>Cc: openstack-operators at lists.openstack.org
>>>Subject: Re: [openstack-dev] [glance] [nova] Glance bug with Kilo
>>>upgrade & Nova
>>>
>>>On 08/06/15 11:46 -0400, Clayton O'Neill wrote:
>>>>We tested testing Kilo upgrades in our hardware dev environments last
>>>>week and the second time through ran into this bug which right now is
>>>>probably a show-stopper for us.
>>>>
>>>>https://bugs.launchpad.net/glance/+bug/1419823
>>>>
>>>>The issue here is that the v1 Glance API allows you to create images
>>>>with properties that are 'NULL' in the Glance database.  For example:
>>>>
>>>>    glance image-create --name cirros_test --disk-format qcow2
>>>>--container-format bare --file cirros-0.3.4-x86_64-disk.img
>>>>--is-public True --is-protected True --progress --property
>>>>description=
>>>>
>>>>It's apparently also fairly easy to end up with a NULL description
>>>>when editing images properties via Horizon.
>>>>
>>>>The issue is that the v2 Glance API returns these NULL properties to
>>>>the client, which then validates them against the schema returned by the v2 API.
>>>>This schema returns specifies that the description property *must* be a string.
>>>>
>>>>In the Kilo release, Nova has been changed to use the v2 API, so
>>>>suddenly this matters.  The net effect is that end users can pretty
>>>>easily create properties with NULL values, and then won't be able to boot instances using those images.
>>>>What makes this worse is that it's completely opaque to end users,
>>>>since this just reports that no node was available to schedule the instance.
>>>>
>>>>However, Nova *only* uses the v2 api to list images if the
>>>>glance.allowed_direct_url_schemes config key is set in the config file.
>>>>However, this config item defaults to an empty array, meaning that by
>>>>default it's *always* set.  There doesn't appear to be a way to unset
>>>>a value with oslo-config that has a default value, blocking off that
>>>>route to work around the issue.  Disabling the v2 Glance API we don't
>>>>think will work, since Nova appears to assume the v2 API is available.
>>>
>>>AFAIK, Nova supports V2 image-lists since before Juno when the allowed_direct_url_schemes config option is set. Are you referring to another change? Has the default been changed in Nova? I'm asking because I was working on this migration to V2 and we decided to postpone it to L.
>>>
>>>>
>>>>Another work around we've looked at is to change the DB schema for
>>>>image properties (yuck) to not allow NULL values.  This results in
>>>>Glance returning a
>>>>500 error since glance-api is attempting to insert an invalid value.
>>>>This is better than instances failing in an opaque fashion, but still pretty horrible.
>>>>
>>>>Has anyone else run into this issue yet?  Are there other work
>>>>arounds that we've not thought of other than "Don't create images with NULL properties?"
>>>>User education is definitely an option, but given the failure mode,
>>>>it's not a great solution for us.
>>>
>>>I believe this needs to be fixed in the client rather than the API and/or the schemas. I'll take a look at this right away.
>>>
>>>Another workaround could be updateting `schema-image.json` to add the schemas that are missing and let the client download the final schema from the V2 API.
>>>
>>>Keep an eye on the bug, patches coming your way (assuming what I have in mind will work).
>>>Flavio
>>>
>>>--
>>>@flaper87
>>>Flavio Percoco
>>
>>--
>>@flaper87
>>Flavio Percoco
>
>
>
>>__________________________________________________________________________
>>OpenStack Development Mailing List (not for usage questions)
>>Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>-- 
>@flaper87
>Flavio Percoco

-- 
@flaper87
Flavio Percoco
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150609/fb6922b9/attachment.pgp>


More information about the OpenStack-dev mailing list