[Openstack-operators] nova-compute won't restart (on some nodes) after Grizzly upgrade

Jonathan Proulx jon at jonproulx.com
Thu Aug 8 14:26:31 UTC 2013


Thank you  Devendran.  That did it

-Jon


On Wed, Aug 7, 2013 at 9:45 PM, Devendran Jagadisan <
devendran.jagadisan at gmail.com> wrote:

> Hi Jonathan,
>
> You could try to run virsh  list --all on the node and check if there are
> deleted instances defined.
>
> Had the same error message during grizzly update and running virsh undefine
> the deleted instances solved the issue.
>
> Devendran
>
>
>
> On Thu, Aug 8, 2013 at 2:36 AM, Jonathan Proulx <jon at jonproulx.com> wrote:
>
>> Hi All,
>>
>> I'm almost through my grizzly upgrade.  I'd upgraded everything except
>> nova-compute before upgrading that (ubuntu 12.04 cloud archieve pkgs).
>>
>> On most nodes the nova-compute service upgraded and restarted properly,
>> but on some it imediately exits with:
>>
>> CRITICAL nova [-] 'instance_type_memory_mb'
>>
>> It would seem like this is https://code.launchpad.net/bugs/1161022 but
>> the fix for that was released in March and I've verified is in the packaged
>> version I'm using.
>>
>> The referenced bug involves the DB migration only updating non-deleted
>> instances in the instance-system-metatata table and the patch skips the
>> lookups that are broken (and irrelevant) for deleted instances.
>>
>> Tracing the DB calls from the host shows it is trying to do lookups for
>> instances that were deleted last October, which is a bit surprising as it's
>> run thousands of instances since & it's not looking those up.
>>
>> It is note worthy that that is around the time I upgraded from Essex ->
>> Folsom so it's possible their state  is weirder than most having run
>> through that update.
>>
>> There were directories for the instances in question in
>> /var/lib/nova/instances, so I thought "Aha!" and moved them, but on restart
>> I still get the same failure and same DB query for the old instances. Where
>> is nova getting the idea it should look these up & how can I stop it?
>>
>> Suggestions for other solutions welcome as well.
>>
>> Thanks,
>> -Jon
>>
>>
>>
>> _______________________________________________
>> OpenStack-operators mailing list
>> OpenStack-operators at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20130808/a68474b9/attachment.html>


More information about the OpenStack-operators mailing list