[openstack-dev] [TROVE] Point in time recovery design

Kevin Conway kevinjacobconway at gmail.com
Tue Mar 4 18:20:05 UTC 2014 

Have we discussed what level of granularity will be supported by point in
time recovery?

For example, there is already an initial implementation of incremental
backups/restores for the MySQL/InnoDB family of data stores. Is that not
already a form of point in time recover?

Is the goal to provide a developer with an API call for /recover {"at":
"2014-03-04 00:00:00.00Z"}? Supporting a true point in time recovery like
that is going to be very difficult. In a former life I worked with a group
of DBAs that managed a large Oracle deployment and dealt with point in time
recovery quite a bit. In fact, the ability to recover to a particular
transaction ID from the bin logs rather than just a timestamp was critical
to support any kind of fault tolerance with our replication setup (just a
heads up that these things are likely going to be problems when we combine
backup/restore with clustering/replication).

Incremental backups and point in time recovery are great features that I
would love to see a part of Trove. What I'm worried about is that we're
pushing forward on these features without enough datastore experts weighing
in on what realistic expectations are for these features across data stores
and how these features might impact replication/clustering. Don't get me
wrong, Xtrabackup is a neat tool, but I don't want to see us build an entire
API spec around it.

As it is, Trove only supports backups for MySQL. We don't even support
incremental backups for MySQL; only the subset of MySQL stores running
InnoDB. I'd like to see more backup operations implemented by the
requested/merged data stores before we start talking about point in time
recovery.

From:  Daniel Morris <daniel.morris at RACKSPACE.COM>
Reply-To:  "OpenStack Development Mailing List (not for usage questions)"
<openstack-dev at lists.openstack.org>
Date:  Tuesday, March 4, 2014 10:36 AM
To:  "OpenStack Development Mailing List (not for usage questions)"
<openstack-dev at lists.openstack.org>
Subject:  Re: [openstack-dev] [TROVE] Point in time recovery design

Nice write-up Denis.  Generally, I think this should merge with the work
going on for scheduled tasks and scheduled backups.

https://wiki.openstack.org/wiki/Trove/scheduled-tasks

Point in time recovery was one of the original goals of the scheduled /
automated backup work, but had not been fully worked out.  Currently this
work is sitting idle - https://review.openstack.org/#/c/73702/

I believe that at the time this was originally discussed, the idea was that
this would be handled on a new instance creation (not an active instance),
and would be accomplished via a new instance creation as follows:

POST /instances
 
{
         "instance": {
         "flavorRef": "https://service//v1.0/1234/flavors/1",
         "name": "myinstance",
         "volume": {
             "size": 2
         }
         "restorePoint": {
             "point_in_time" : "2012-03-28T22:00Z",
             "instanceRef":
"https://service/v1.0/1234/instances/2450c73f-7805-4afe-a42c-4094ab42666b"
         }
    }
}

Regardless of the API design (up for debate), we need this capability
integrated and just need to work out the best way to do it.

Daniel

From: Denis Makogon <dmakogon at mirantis.com>
Reply-To: "OpenStack Development Mailing List (not for usage questions)"
<openstack-dev at lists.openstack.org>
Date: Tuesday, March 4, 2014 5:15 AM
To: OpenStack Development Mailing List <openstack-dev at lists.openstack.org>
Subject: [openstack-dev] [TROVE] Point in time recovery design

> Trove. Point-in-Time recovery.
> 
> 
> 
> 
> 1. Introduction 
> <https://docs.google.com/a/mirantis.com/document/d/12qHHYCQ3BTOKCEcbfp-75NPJc1
> 5xPD01WEQe9OmyOxc/edit#heading=h.tg53tr6tfa3> .
> 2. 
> 3. What is a point in time recovery?
> <https://docs.google.com/a/mirantis.com/document/d/12qHHYCQ3BTOKCEcbfp-75NPJc1
> 5xPD01WEQe9OmyOxc/edit#heading=h.rnvnnwld05c2>
> 4. 
> 5. What does it take to do a point in time recovery?
> <https://docs.google.com/a/mirantis.com/document/d/12qHHYCQ3BTOKCEcbfp-75NPJc1
> 5xPD01WEQe9OmyOxc/edit#heading=h.cp013g5it8qq>
> 6. 
> 7. What to consider once you know your database is corrupted?
> <https://docs.google.com/a/mirantis.com/document/d/12qHHYCQ3BTOKCEcbfp-75NPJc1
> 5xPD01WEQe9OmyOxc/edit#heading=h.oe6kdjz502c7>
> 8. 
> 9. Trove and Point-in-time recovery.
> <https://docs.google.com/a/mirantis.com/document/d/12qHHYCQ3BTOKCEcbfp-75NPJc1
> 5xPD01WEQe9OmyOxc/edit#heading=h.1m5g1t97bfdx>
> 10. 
> 11. Trove core ReST API and Point-in-Time Recovery/Restore flow.
> <https://docs.google.com/a/mirantis.com/document/d/12qHHYCQ3BTOKCEcbfp-75NPJc1
> 5xPD01WEQe9OmyOxc/edit#heading=h.qjqnx4eo6du8>
>> 1. ReST routes. 
>> <https://docs.google.com/a/mirantis.com/document/d/12qHHYCQ3BTOKCEcbfp-75NPJc
>> 15xPD01WEQe9OmyOxc/edit#heading=h.yedhftl8z7td>
>> 2. 
>> 3. Request body.
>> <https://docs.google.com/a/mirantis.com/document/d/12qHHYCQ3BTOKCEcbfp-75NPJc
>> 15xPD01WEQe9OmyOxc/edit#heading=h.7grnwb9z2u3g>
>> 4. 
>> 5. Response object.
>> <https://docs.google.com/a/mirantis.com/document/d/12qHHYCQ3BTOKCEcbfp-75NPJc
>> 15xPD01WEQe9OmyOxc/edit#heading=h.3n48u9fe0w43>
> 12. Trove taskmanager RPC API and Point-in-Time Recovery/Restore flow.
> <https://docs.google.com/a/mirantis.com/document/d/12qHHYCQ3BTOKCEcbfp-75NPJc1
> 5xPD01WEQe9OmyOxc/edit#heading=h.1pr74mrhlana>
>> 1. RPC message. 
>> <https://docs.google.com/a/mirantis.com/document/d/12qHHYCQ3BTOKCEcbfp-75NPJc
>> 15xPD01WEQe9OmyOxc/edit#heading=h.hk9iralbbn60>
>> 2. 
>> 3. RPC message type.
>> <https://docs.google.com/a/mirantis.com/document/d/12qHHYCQ3BTOKCEcbfp-75NPJc
>> 15xPD01WEQe9OmyOxc/edit#heading=h.kgt3wgj6q4j7>
> 13. Trove guestagent RPC API and Point-in-Time Recovery/Restore flow.
> <https://docs.google.com/a/mirantis.com/document/d/12qHHYCQ3BTOKCEcbfp-75NPJc1
> 5xPD01WEQe9OmyOxc/edit#heading=h.gme8lf1lvok2>
>> 1. RPC message. 
>> <https://docs.google.com/a/mirantis.com/document/d/12qHHYCQ3BTOKCEcbfp-75NPJc
>> 15xPD01WEQe9OmyOxc/edit#heading=h.wxienab8nvi7>
>> 2. 
>> 3. RPC message type.
>> <https://docs.google.com/a/mirantis.com/document/d/12qHHYCQ3BTOKCEcbfp-75NPJc
>> 15xPD01WEQe9OmyOxc/edit#heading=h.stpp0ym7nd04>
>> 4. 
>> 5. Method implementation.
>> <https://docs.google.com/a/mirantis.com/document/d/12qHHYCQ3BTOKCEcbfp-75NPJc
>> 15xPD01WEQe9OmyOxc/edit#heading=h.y4xmxn4pju8f>
> 14. Proposed implementation for Trove and for Python-troveclient.
> <https://docs.google.com/a/mirantis.com/document/d/12qHHYCQ3BTOKCEcbfp-75NPJc1
> 5xPD01WEQe9OmyOxc/edit#heading=h.5bpxcfj9gujs>
> 15. 
> 16. Useful links.
> <https://docs.google.com/a/mirantis.com/document/d/12qHHYCQ3BTOKCEcbfp-75NPJc1
> 5xPD01WEQe9OmyOxc/edit#heading=h.p0h05fqgo9oj>
>  
> <https://docs.google.com/a/mirantis.com/document/d/12qHHYCQ3BTOKCEcbfp-75NPJc1
> 5xPD01WEQe9OmyOxc/edit#heading=h.p0h05fqgo9oj>
> I 
> <https://docs.google.com/a/mirantis.com/document/d/12qHHYCQ3BTOKCEcbfp-75NPJc1
> 5xPD01WEQe9OmyOxc/edit#heading=h.p0h05fqgo9oj> ntroduction
> 
> 
> Every once in a while, an event might happen that corrupts a database. We have
> all made a stupid mistake at least once that had trashed a database. When this
> happens what do you do? If you do not have a database backup, then you had
> better own up to the problem you caused and tell your boss that you screwed
> up. If you do have at least a complete database backup then you most likely
> will be able to recover the corrupted database, up to the point that you
> corrupted the data. This article will discuss how to use a point in time
> restore to recover your databases.
> If you google ³Point in time recovery² you also could find ³Point in time
> restore². So, let decide how to call it. Historically, database has a feature
> called Point in time recovery.
> 
> What is a point-in-time recovery?
> So what is a point in time recovery? A point in time recovery is restoring a
> database to a specified date and time. When you have completed a point in time
> recovery, your database will be in the state it was at the specific date and
> time you identified when restoring your database. A point in time recovery is
> a method to recover your database to any point in time since the last database
> backup.
> 
> What does it take to do a point-in-time recovery?
> In order to perform a point in time recovery you will need to have an entire
> series of backups (complete, differential, and transaction log backups) up to
> and/or beyond the point in time in which you want to recover. If you are
> missing any backups, or have truncated the transaction log without first
> performing a transaction log backup, then you will not be able to perform a
> point in time recovery. At a minimum, you will need a complete backup and all
> the transaction log backups taken following the complete backup. Optionally if
> you are taking differential backups, then you will need the complete backup,
> the last differential backup prior to the corruption, then all the transaction
> log backups taken following the differential backup.
> 
> 
> Trove and Point-in-time recovery
> 
> 
> OpenStack DBaaS Trove is able to perform instance restoration (whole new
> instance, from scratch) from previously stored backup in remote storage
> (OpenStack Swift, Amazon AWS S3, etc). From administration/regular user
> perspective Trove should be able to perform point in time recovery. Basically
> it¹s almost the same as restoring new instance, but the difference between
> restore (in terms of Trove) and recovery is huge.
> Restore gives an ability to spin-up new instance from backup (as mentioned
> earlier), but the Recovery gives an ability to restore already running
> instance from backup. For the beginning Trove would be able to recover/restore
> running instance from full backup.
> Trove core ReST API and Point-in-Time Recovery/Restore flow
> 
> ReST routes
> 
> HTTP methodRoutes
> 
> 
> POST{tenant_id}/instances/{instance_id}/recover
> or
> {tenant_id}/instances/{instance_id}/restore
> Request body
> 
> 
> ³recovery²: {
>     ³instance²: UUID,
>     ³backup²: UUID,
> }
> Response object
> 
> 
> ³recovery²: {
>     ³id²: ³instance_id²,
>     ³name²: ³instance_name²,
>     ³status²: ³BUILDING²,
>     ³datastore²: ³mysql²,
>     ³recovered_from_backup²: ³backup_id²,
>     ³point_in_time²: ³2011-01-22T13:25:27-06:00²,
> }
> 
> Trove taskmanager RPC API and Point-in-Time Recovery/Restore flow
> 
> RPC message
> RPC methodMethod parameters
> do_instance_recoveryinstance_idbackup_id
> RPC message type
>     CAST with poll until instance reach ACTIVE status.
> Trove guestagent RPC API and Point-in-Time Recovery/Restore flow
> 
> RPC message
> RPC methodMethod parameters
> 
> 
> do_recovery       backup_info: {                     'id': backup_id,
> Œlocation': location,                     ¹type': backup_type,
> 'checksum': checksum,       }
> RPC message type
>     CAST
> 
> Method implementation
> 
> 
> Re-used restore functionality (restore from full backup).
> 
> Proposed implementation for Trove and for Python-troveclient
> 
> 1. Trove: [1]
> 2. 
> 3. Python-troveclient: [2]
> Useful links
>      [1] https://review.openstack.org/#/c/77222/
> <https://review.openstack.org/#/c/77222/>
>      [2] https://review.openstack.org/#/c/77223/
> <https://review.openstack.org/#/c/77223/>
> 
> 
> Best Regards,
> Denis Makogon
> dmakogon at mirantis.com
> www.mirantis.com <http://www.mirantis.com>
> 
> _______________________________________________ OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140304/8df84dc3/attachment-0001.html>

More information about the OpenStack-dev mailing list