<div dir="ltr">Flavio's fix was merged to master and back ported to the stable/kilo master. We're hoping to run it through our upgrade testing tomorrow.</div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jun 9, 2015 at 8:33 AM, Flavio Percoco <span dir="ltr"><<a href="mailto:flavio@redhat.com" target="_blank">flavio@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On 09/06/15 07:38 +0000, Bhandaru, Malini K wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
A DB migrate script that puts some token default string only for glance properties that are NULL. Should not change anything else.<br>
Hope it works Flavio.<br>
Regards<br>
Malini<br>
</blockquote>
<br></span>
Proposed fix, please read the commit message and comments:<br>
<br>
<a href="https://review.openstack.org/#/c/189661/" target="_blank">https://review.openstack.org/#/c/189661/</a><div class="HOEnZb"><div class="h5"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
-----Original Message-----<br>
From: Flavio Percoco [mailto:<a href="mailto:flavio@redhat.com" target="_blank">flavio@redhat.com</a>]<br>
Sent: Tuesday, June 09, 2015 12:33 AM<br>
To: Bhandaru, Malini K<br>
Cc: OpenStack Development Mailing List (not for usage questions); <a href="mailto:openstack-operators@lists.openstack.org" target="_blank">openstack-operators@lists.openstack.org</a><br>
Subject: Re: [openstack-dev] [glance] [nova] Glance bug with Kilo upgrade & Nova<br>
<br>
On 09/06/15 09:22 +0200, Flavio Percoco wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On 09/06/15 07:09 +0000, Bhandaru, Malini K wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Flavio, would a DB script that writes an empty string or NOP or something instead of NULL In the column do the trick?<br>
Then the problem degenerates to a new DB upgrade script.<br>
</blockquote>
<br>
I now remembered about a change[0] - that I wrote myself - that<br>
required a bump on the API version - which we did - that allows None<br>
values to be returned in the API. This is probably what is causing this<br>
behavior.<br>
<br>
A DB migration would certainly work, I'd love to avoid it but I guess<br>
that's the best solution in this case.<br>
</blockquote>
<br>
Actually, we can't just migrate the database as there might be custom properties that were explicitly set to None.<br>
<br>
I'll keep you posted<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Cheers,<br>
Flavio<br>
<br>
[0] <a href="https://review.openstack.org/#/c/138184/" target="_blank">https://review.openstack.org/#/c/138184/</a><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Regards<br>
Malini<br>
<br>
-----Original Message-----<br>
From: Flavio Percoco [mailto:<a href="mailto:flavio@redhat.com" target="_blank">flavio@redhat.com</a>]<br>
Sent: Monday, June 08, 2015 11:47 PM<br>
To: OpenStack Development Mailing List (not for usage questions)<br>
Cc: <a href="mailto:openstack-operators@lists.openstack.org" target="_blank">openstack-operators@lists.openstack.org</a><br>
Subject: Re: [openstack-dev] [glance] [nova] Glance bug with Kilo<br>
upgrade & Nova<br>
<br>
On 08/06/15 11:46 -0400, Clayton O'Neill wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
We tested testing Kilo upgrades in our hardware dev environments last<br>
week and the second time through ran into this bug which right now is<br>
probably a show-stopper for us.<br>
<br>
<a href="https://bugs.launchpad.net/glance/+bug/1419823" target="_blank">https://bugs.launchpad.net/glance/+bug/1419823</a><br>
<br>
The issue here is that the v1 Glance API allows you to create images<br>
with properties that are 'NULL' in the Glance database. For example:<br>
<br>
glance image-create --name cirros_test --disk-format qcow2<br>
--container-format bare --file cirros-0.3.4-x86_64-disk.img<br>
--is-public True --is-protected True --progress --property<br>
description=<br>
<br>
It's apparently also fairly easy to end up with a NULL description<br>
when editing images properties via Horizon.<br>
<br>
The issue is that the v2 Glance API returns these NULL properties to<br>
the client, which then validates them against the schema returned by the v2 API.<br>
This schema returns specifies that the description property *must* be a string.<br>
<br>
In the Kilo release, Nova has been changed to use the v2 API, so<br>
suddenly this matters. The net effect is that end users can pretty<br>
easily create properties with NULL values, and then won't be able to boot instances using those images.<br>
What makes this worse is that it's completely opaque to end users,<br>
since this just reports that no node was available to schedule the instance.<br>
<br>
However, Nova *only* uses the v2 api to list images if the<br>
glance.allowed_direct_url_schemes config key is set in the config file.<br>
However, this config item defaults to an empty array, meaning that by<br>
default it's *always* set. There doesn't appear to be a way to unset<br>
a value with oslo-config that has a default value, blocking off that<br>
route to work around the issue. Disabling the v2 Glance API we don't<br>
think will work, since Nova appears to assume the v2 API is available.<br>
</blockquote>
<br>
AFAIK, Nova supports V2 image-lists since before Juno when the allowed_direct_url_schemes config option is set. Are you referring to another change? Has the default been changed in Nova? I'm asking because I was working on this migration to V2 and we decided to postpone it to L.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Another work around we've looked at is to change the DB schema for<br>
image properties (yuck) to not allow NULL values. This results in<br>
Glance returning a<br>
500 error since glance-api is attempting to insert an invalid value.<br>
This is better than instances failing in an opaque fashion, but still pretty horrible.<br>
<br>
Has anyone else run into this issue yet? Are there other work<br>
arounds that we've not thought of other than "Don't create images with NULL properties?"<br>
User education is definitely an option, but given the failure mode,<br>
it's not a great solution for us.<br>
</blockquote>
<br>
I believe this needs to be fixed in the client rather than the API and/or the schemas. I'll take a look at this right away.<br>
<br>
Another workaround could be updateting `schema-image.json` to add the schemas that are missing and let the client download the final schema from the V2 API.<br>
<br>
Keep an eye on the bug, patches coming your way (assuming what I have in mind will work).<br>
Flavio<br>
<br>
--<br>
@flaper87<br>
Flavio Percoco<br>
</blockquote>
<br>
--<br>
@flaper87<br>
Flavio Percoco<br>
</blockquote>
<br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote>
<br>
<br>
-- <br>
@flaper87<br>
Flavio Percoco<br>
</blockquote>
<br></div></div><span class="HOEnZb"><font color="#888888">
-- <br>
@flaper87<br>
Flavio Percoco<br>
</font></span><br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div>