[Openstack-operators] [openstack-dev] [nova] nova-manage db archive_deleted_rows broken
mriedem at linux.vnet.ibm.com
Wed Nov 18 17:51:45 UTC 2015
On 11/18/2015 7:32 AM, Andrew Laski wrote:
> On 11/17/15 at 07:43pm, Jay Pipes wrote:
>> On 11/17/2015 05:43 PM, Matt Riedemann wrote:
>>> I found some time to work on a reverse sort of nova's tables for the db
>>> archive command, that looks like . It works fine in the unit tests,
>>> but fails because the deleted instances are referenced by
>>> instance_actions that aren't deleted. I see any DB APIs for deleting
>>> instance actions.
>>> Were we just planning on instance_actions living forever in the
>> Not as far as I understand.
> They were never intended to live forever. However there is a use case
> for holding on to the deleted action so that someone could query when or
> by whom their instance was deleted. But the current API does not
> provide a good way to query for that so this may be something better
> left to the growing list of things that Tasks could address.
>>> Should we soft delete instance_actions when we delete the referenced
>>> Or should we (hard) delete instance_actions when we archive (move to
>>> shadow tables) soft deleted instances?
>>> This is going to be a blocker to getting nova-manage db
>>> archive_deleted_rows working.
>>>  https://review.openstack.org/#/c/246635/
>> OpenStack Development Mailing List (not for usage questions)
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
OK, so we talked a bit about this in IRC today. I'll try to summarize
here. There are really two options it sounds like, either soft delete
instance_actions when we soft delete instances, or just hard delete the
instance_actions when archiving/purging soft deleted instances (to avoid
the referential constraint failure when hard deleting the instances).
We have to note that the os-instance-actions API only works on instances
that are not deleted . So once you delete an instance (soft deleted
in this case, i.e. instances.deleted != 0 in the DB), then the
os-instance-actions API is useless. We could change that with a
microversion to read deleted instances...but I'm not sure if anyone
wants that (operators might?).
Even if we start soft deleting instance_actions, the archive/purge code
will still have to account for existing resources in the database and
handle them, i.e. when archiving an instance, find it's related
instance_actions and if they are not soft deleted, soft delete and
archive them first before archiving the instance. We could probably also
provide a DB migration to do this, but that would be a lot of queries
and updates potentially during an offline upgrade. We could also provide
a nova-manage command to do it explicitly.
So hard-deleting instance_actions when archiving/purging instances is
easier from a development perspective, it's much more straight forward
(fewer bugs, yay!). The downside to hard delete is they are gone, they
wouldn't be in shadow tables or anything in the nova database.
The proposed remedy for wanting to store instance_actions after
hard-delete is to basically have a monitoring service setup that is
storing the instance delete notificiations, i.e. StackTach, gnocchi,
probably others (monasca?). This is kind of a cop out for the nova dev
team since it puts the burden on the operators to have this extra
service running, but I suspect most already have something like this
I'll cross post this to the operators ML and have it on this weeks' nova
meeting agenda to see if we can find some agreement on a path forward here.
More information about the OpenStack-operators