Open Stack

Flavio's fix was merged to master and back ported to the stable/kilo
master.  We're hoping to run it through our upgrade testing tomorrow.

On Tue, Jun 9, 2015 at 8:33 AM, Flavio Percoco <flavio at redhat.com> wrote:

> On 09/06/15 07:38 +0000, Bhandaru, Malini K wrote:
>
>> A DB migrate script that puts some token default string only for glance
>> properties that are NULL. Should not change anything else.
>> Hope it works Flavio.
>> Regards
>> Malini
>>
>
> Proposed fix, please read the commit message and comments:
>
> https://review.openstack.org/#/c/189661/
>
>
>
>> -----Original Message-----
>> From: Flavio Percoco [mailto:flavio at redhat.com]
>> Sent: Tuesday, June 09, 2015 12:33 AM
>> To: Bhandaru, Malini K
>> Cc: OpenStack Development Mailing List (not for usage questions);
>> openstack-operators at lists.openstack.org
>> Subject: Re: [openstack-dev] [glance] [nova] Glance bug with Kilo upgrade
>> & Nova
>>
>> On 09/06/15 09:22 +0200, Flavio Percoco wrote:
>>
>>> On 09/06/15 07:09 +0000, Bhandaru, Malini K wrote:
>>>
>>>> Flavio, would a DB script that writes an empty string or NOP or
>>>> something instead of NULL In the column do the trick?
>>>> Then the problem degenerates to a new DB upgrade script.
>>>>
>>>
>>> I now remembered about a change[0] - that I wrote myself - that
>>> required a bump on the API version - which we did - that allows None
>>> values to be returned in the API. This is probably what is causing this
>>> behavior.
>>>
>>> A DB migration would certainly work, I'd love to avoid it but I guess
>>> that's the best solution in this case.
>>>
>>
>> Actually, we can't just migrate the database as there might be custom
>> properties that were explicitly set to None.
>>
>> I'll keep you posted
>>
>>
>>> Cheers,
>>> Flavio
>>>
>>> [0] https://review.openstack.org/#/c/138184/
>>>
>>>
>>>> Regards
>>>> Malini
>>>>
>>>> -----Original Message-----
>>>> From: Flavio Percoco [mailto:flavio at redhat.com]
>>>> Sent: Monday, June 08, 2015 11:47 PM
>>>> To: OpenStack Development Mailing List (not for usage questions)
>>>> Cc: openstack-operators at lists.openstack.org
>>>> Subject: Re: [openstack-dev] [glance] [nova] Glance bug with Kilo
>>>> upgrade & Nova
>>>>
>>>> On 08/06/15 11:46 -0400, Clayton O'Neill wrote:
>>>>
>>>>> We tested testing Kilo upgrades in our hardware dev environments last
>>>>> week and the second time through ran into this bug which right now is
>>>>> probably a show-stopper for us.
>>>>>
>>>>> https://bugs.launchpad.net/glance/+bug/1419823
>>>>>
>>>>> The issue here is that the v1 Glance API allows you to create images
>>>>> with properties that are 'NULL' in the Glance database.  For example:
>>>>>
>>>>>     glance image-create --name cirros_test --disk-format qcow2
>>>>> --container-format bare --file cirros-0.3.4-x86_64-disk.img
>>>>> --is-public True --is-protected True --progress --property
>>>>> description=
>>>>>
>>>>> It's apparently also fairly easy to end up with a NULL description
>>>>> when editing images properties via Horizon.
>>>>>
>>>>> The issue is that the v2 Glance API returns these NULL properties to
>>>>> the client, which then validates them against the schema returned by
>>>>> the v2 API.
>>>>> This schema returns specifies that the description property *must* be
>>>>> a string.
>>>>>
>>>>> In the Kilo release, Nova has been changed to use the v2 API, so
>>>>> suddenly this matters.  The net effect is that end users can pretty
>>>>> easily create properties with NULL values, and then won't be able to
>>>>> boot instances using those images.
>>>>> What makes this worse is that it's completely opaque to end users,
>>>>> since this just reports that no node was available to schedule the
>>>>> instance.
>>>>>
>>>>> However, Nova *only* uses the v2 api to list images if the
>>>>> glance.allowed_direct_url_schemes config key is set in the config file.
>>>>> However, this config item defaults to an empty array, meaning that by
>>>>> default it's *always* set.  There doesn't appear to be a way to unset
>>>>> a value with oslo-config that has a default value, blocking off that
>>>>> route to work around the issue.  Disabling the v2 Glance API we don't
>>>>> think will work, since Nova appears to assume the v2 API is available.
>>>>>
>>>>
>>>> AFAIK, Nova supports V2 image-lists since before Juno when the
>>>> allowed_direct_url_schemes config option is set. Are you referring to
>>>> another change? Has the default been changed in Nova? I'm asking because I
>>>> was working on this migration to V2 and we decided to postpone it to L.
>>>>
>>>>
>>>>> Another work around we've looked at is to change the DB schema for
>>>>> image properties (yuck) to not allow NULL values.  This results in
>>>>> Glance returning a
>>>>> 500 error since glance-api is attempting to insert an invalid value.
>>>>> This is better than instances failing in an opaque fashion, but still
>>>>> pretty horrible.
>>>>>
>>>>> Has anyone else run into this issue yet?  Are there other work
>>>>> arounds that we've not thought of other than "Don't create images with
>>>>> NULL properties?"
>>>>> User education is definitely an option, but given the failure mode,
>>>>> it's not a great solution for us.
>>>>>
>>>>
>>>> I believe this needs to be fixed in the client rather than the API
>>>> and/or the schemas. I'll take a look at this right away.
>>>>
>>>> Another workaround could be updateting `schema-image.json` to add the
>>>> schemas that are missing and let the client download the final schema from
>>>> the V2 API.
>>>>
>>>> Keep an eye on the bug, patches coming your way (assuming what I have
>>>> in mind will work).
>>>> Flavio
>>>>
>>>> --
>>>> @flaper87
>>>> Flavio Percoco
>>>>
>>>
>>> --
>>> @flaper87
>>> Flavio Percoco
>>>
>>
>>
>>
>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>>
>> --
>> @flaper87
>> Flavio Percoco
>>
>
> --
> @flaper87
> Flavio Percoco
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20150609/3857177a/attachment.html>