<div dir="ltr"><div class="gmail_default" style="font-family:courier new,monospace"><br></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Aug 30, 2013 at 8:02 AM, Murali Balcha <span dir="ltr"><<a href="mailto:Murali.Balcha@triliodata.com" target="_blank">Murali.Balcha@triliodata.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="auto">
<div>Hi John,</div>
<div>Thanks for your comments. I am planning to attend summit we can have a wider discussion there.<br>
<br>
Thanks,
<div>Murali Balcha</div>
<div><br>
</div>
</div><div><div class="h5">
<div><br>
On Aug 30, 2013, at 12:05 AM, "John Griffith" <<a href="mailto:john.griffith@solidfire.com" target="_blank">john.griffith@solidfire.com</a>> wrote:<br>
<br>
</div>
<blockquote type="cite">
<div>
<div dir="ltr">
<div style="font-family:'courier new',monospace"><br>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Thu, Aug 29, 2013 at 6:36 PM, Murali Balcha <span dir="ltr">
<<a href="mailto:Murali.Balcha@triliodata.com" target="_blank">Murali.Balcha@triliodata.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div><br>
>>> My question is, would it make sense to add to the current mechanisms in<br>
>>> Nova and Cinder than add the complexity of a new project?<br>
><br>
> I think the answer is yes :)<br>
<br>
<br>
</div>
I meant there is a clear need for Raksha project. :)<br>
<br>
Thanks,<br>
Murali Balcha<br>
<div>
<div><br>
On Aug 29, 2013, at 7:45 PM, "Murali Balcha" <<a href="mailto:Murali.Balcha@triliodata.com" target="_blank">Murali.Balcha@triliodata.com</a>> wrote:<br>
<br>
><br>
> ________________________________________<br>
>>> From: Ronen Kat <<a href="mailto:RONENKAT@il.ibm.com" target="_blank">RONENKAT@il.ibm.com</a>><br>
>>> Sen: Thursday, August 29, 2013 2:55 PM<br>
>>> To: <a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a>;
<a href="mailto:openstack-dev@lists.launchpad.net" target="_blank">openstack-dev@lists.launchpad.net</a><br>
>>> Subject: Re: [openstack-dev] Proposal for Raksha, a Data Protection As a Service project<br>
><br>
>>> Hi Murali,<br>
><br>
>>> I think the idea to provide enhanced data protection in OpenStack is a<br>
>>> great idea, and I have been thinking about backup in OpenStack for a while<br>
>>> now.<br>
>>> I just not sure a new project is the only way to do.<br>
><br>
>>> (as disclosure, I contributed code to enable IBM TSM as a Cinder backup<br>
>>> driver)<br>
><br>
> Hi Kat,<br>
> Consider the following use cases that Raksha will addresses. I will discuss from simple to complex use case and then address your specific questions with inline comments.<br>
> 1. VM1 that is created on the local file system with a cinder volume attached<br>
> 2. VM2 that is booted off from a cinder volume and has couple of cinder volumes attached<br>
> 3. VM1 and VM2 all booted from cinder volumes and has couple of volumes attached. They also share a private network for internal communication.<br>
> 4.<br>
> In all these cases Raksha will take a consistent snap of VMs, walk thru each VM resources and backup the resources to swift end point.<br>
> In case 1, that means backup VM image and Cinder volume image to swift<br>
> In case 2 is an extension of case 1.<br>
> In case 3, Raksha not only backup VM1 and VM2 and its associated resources, it also backup the network configuration<br>
><br>
> Now lets consider the restore case. The restore operation walks thru the backup resources and calls into respective openstack services to restore those objects. In case1, it first calls Nova API to restore the VM, it calls into Cinder to restore the volume
and attach the volume to the newly restored VM instance. In case of 3, it also calls into Neutron API to restore the networking. Hence my argument is that not one OpenStack project has a global view of VM and all its resources to implement an effective backup
and restore services.<br>
><br>
><br>
>>> I wonder what is the added-value of a project approach versus enhancements<br>
>>> to the current Nova and Cinder implementations of backup. Let me elaborate.<br>
><br>
>>> Nova has a "nova backup" feature that performs a backup of a VM to Glance,<br>
>>> the backup is managed by tenants in the same way that you propose.<br>
>>> While today it provides only point-in-time full backup, it seems reasonable<br>
>>> that it can be extended support incremental and consistent backup as well -<br>
>>> as the actual work is done either by the Storage or Hypervisor in any case.<br>
><br>
> Though Nova has API to upload a snapshot of the VM to glance, it does not snapshot any volumes associated with the VM. When a snapshot is uploaded to glance, Nova creates an image by collapsing the qemu image with delta file and uploads the larger file to
glance. If we were to perform periodic backups of VMs, this is a very inefficient way to do backup. Also having to manage two end points, one for Nova and Cinder is inefficient. These are the gaps I called out in Raksha wiki page.<br>
><br>
><br>
>>> Cinder has a cinder backup command that performs a volume backup to Swift,<br>
>>> Ceph or TSM. The Ceph implementation also support incremental backup (Ceph<br>
>>> to Ceph).<br>
>>> I envision that Cinder could be expanded to support incremental backup (for<br>
>>> persistent storage) by adding drivers/plug-ins that will leverage<br>
>>> incremental backup features of either the storage or Hypervisors.<br>
>>> Independently, in Havana the ability to do consistent volume snapshots was<br>
>>> added to GlusterFS. I assume that this consistency support could be<br>
>>> generalized to support other volume drivers, and be utilized as part of a<br>
>>> backup code.<br>
><br>
> I think we are talking specific implementations here. Yes, I am aware of Ceph blueprint to support incremental backup, but Cinder backup APIs are volume specific. That means if a VM has multiple volumes mapped as in the case 2 I discussed, tenant need to
call backup api three times. Also if you look at the swift layout of the cinder, it is very difficult to tie the swift images back to a particular VM. Imagine a tenant were to restore a VM and all its resources from a backup copy that was performed a week
ago. The restore operation is not straight forward.<br>
> It is my understanding that consistency should be maintained at the VM, not at individual volume. It is very difficult to assume how the application data inside VM is laid out.<br>
><br>
>>> Looking at the key features in Raksha, it seems that the main features<br>
>>> (2,3,4,7) could be addressed by improving the current mechanisms in Nova<br>
>>> and Cinder. I didn't included 1 as a feature as it is more a statement of<br>
>>> intent (or goal) than a feature.<br>
>>> Features 5 (dedup) and 6 (scheduler) are indeed new in your proposal.<br>
><br>
>>> Looking at the source de-duplication feature, and taking Swift as an<br>
>>> example, it seems reasonable that if Swift will implement de-duplication,<br>
>>> then doing backup to Swift will give us de-duplication for free.<br>
>>> In fact it would make sense to do the de-duplication at the Swift level<br>
>>> instead of just the backup layer to gain more duplication opportunities.<br>
><br>
> I agree, however Swift is not the only object store that need to support dedupe. Ceph is another popular object store too. GlusterFS supports Swift end point and there are other commercially available object stores too. So you argument becomes very product
specific. However source level dedupes is different than dedupe at rest. Source level dedupe reduces the backup windows and also reduces the amount of data that need to be pumped to backup end point like swift.<br>
><br>
>>> Following the above, and assuming it all come true (at times I am known to<br>
>>> be an optimistic), then we are left with backup job scheduling, and I<br>
>>> wonder if that is enough for a new project.<br>
><br>
> I hope I convinced that Raksha has more to offer than a simple cron job. Please take a look at the backup apis, its database schema and the usecases it addresses in its wiki page.<br>
><br>
><br>
> Bottom line is irrespective how OpenStack is deployed; here is how Raksha workflow looks like<br>
> * Create-backupjob VM1, VM2<br>
> --> Returns backup job id, id1<br>
> * Run-backupjob id1<br>
> --> Returns runid rid1<br>
> * Run backup job id1<br>
> --> Returns run id rid2<br>
><br>
> * Restore rid1<br>
> --> Restores PiT of VM1 and VM2 and its associated volumes<br>
><br>
><br>
>>> My question is, would it make sense to add to the current mechanisms in<br>
>>> Nova and Cinder than add the complexity of a new project?<br>
><br>
> I think the answer is yes :)<br>
><br>
> Regards,<br>
> Murali Balcha<br>
>>> __________________________________________<br>
>>> Ronen I. Kat<br>
>>> Storage Research<br>
> IBM Research - Haifa<br>
> Phone: <a href="tel:%2B972.3.7689493" value="+97237689493" target="_blank">+972.3.7689493</a><br>
> Email: <a href="mailto:ronenkat@il.ibm.com" target="_blank">ronenkat@il.ibm.com</a><br>
><br>
> From: Murali Balcha <<a href="mailto:Murali.Balcha@triliodata.com" target="_blank">Murali.Balcha@triliodata.com</a>><br>
> To: "<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a>"<br>
> <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a>>,<br>
> "<a href="mailto:openstack@list.openstack.org" target="_blank">openstack@list.openstack.org</a>" <<a href="mailto:openstack@list.openstack.org" target="_blank">openstack@list.openstack.org</a>>,<br>
> Date: 29/08/2013 01:18 AM<br>
> Subject: [openstack-dev] Proposal for Raksha, a Data Protection As a<br>
> Service project<br>
><br>
><br>
><br>
> Hello Stackers,<br>
> We would like to introduce a new project Raksha, a Data Protection As a<br>
> Service (DPaaS) for OpenStack Cloud.<br>
> Raksha’s primary goal is to provide a comprehensive Data Protection for<br>
> OpenStack by leveraging Nova, Swift, Glance and Cinder. Raksha has<br>
> following key features:<br>
> 1. Provide an enterprise grade data protection for OpenStack<br>
> based clouds<br>
> 2. Tenant administered backups and restores<br>
> 3. Application consistent backups<br>
> 4. Point In Time(PiT) full and incremental backups and restores<br>
> 5. Dedupe at source for efficient backups<br>
> 6. A job scheduler for periodic backups<br>
> 7. Noninvasive backup solution that does not require service<br>
> interruption during backup window<br>
><br>
> You will find the rationale behind the need for Raksha in OpenStack in its<br>
> Wiki. The wiki also has the preliminary design and the API description.<br>
> Some of the Raksha functionality may overlap with Nova and Cinder projects<br>
> and as a community lets work together to coordinate the features among<br>
> these projects. We would like to seek out early feedback so we can address<br>
> as many issues as we can in the first code drop. We are hoping to enlist<br>
> the OpenStack community help in making Raksha a part of OpenStack.<br>
> Raksha’s project resources:<br>
> Wiki: <a href="https://wiki.openstack.org/wiki/Raksha" target="_blank">https://wiki.openstack.org/wiki/Raksha</a><br>
> Launchpad: <a href="https://launchpad.net/raksha" target="_blank">https://launchpad.net/raksha</a><br>
> Github: <a href="https://github.com/DPaaS-Raksha/Raksha" target="_blank">https://github.com/DPaaS-Raksha/Raksha</a> (We will upload a prototype<br>
> code in few days)<br>
> If you want to talk to us, send an email to<br>
> <a href="mailto:openstack-dev@lists.launchpad.net" target="_blank">openstack-dev@lists.launchpad.net</a> with "[raksha]" in the subject or use<br>
> #openstack-raksha irc channel.<br>
><br>
> Best Regards,<br>
> Murali Balcha_______________________________________________<br>
> OpenStack-dev mailing list<br>
> <a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
> _______________________________________________<br>
> OpenStack-dev mailing list<br>
> <a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
> _______________________________________________<br>
> OpenStack-dev mailing list<br>
> <a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
<div class="gmail_extra">
<div style="font-family:'courier new',monospace">Hi <span style="font-family:arial,sans-serif;font-size:13px">Murali,</span></div>
<div style="font-family:'courier new',monospace"><br>
</div>
<div style="font-family:'courier new',monospace">This sounds pretty neat, but in my opinion it seems that we have most of the items in your list covered with the Cinder backup service. As far as backing up instances, I'm personally not
sure about backing up ephemeral objects? We already have the ability to create and upload an image which is "kind of a backup". Also if you want a persistent instance wouldn't it be better to have it reside on persistent storage and back that up?</div>
<div style="font-family:'courier new',monospace"><br>
</div>
<div style="font-family:'courier new',monospace">Anyway, my personal thought is that it might be more efficient to see how things develop with the backup service in Cinder.</div>
<div style="font-family:'courier new',monospace"><br>
</div>
<div style="font-family:'courier new',monospace">As far as the deduplication idea, I really think that's much better done on the target rather than trying to process it on the source. processing this in the backup service is pretty expensive
and there are a lot of trouble spots, not the least of which is a pretty big hit on performance. </div>
<div style="font-family:'courier new',monospace"><br>
</div>
<div style="font-family:'courier new',monospace">Also as was pointed out, there are quite a few efficiencies and optimizations that can be realized by leaving the work closer to the backend storage itself. There are a number of cases
already pointed out where there are some good optimizations, in addition there are also a number of back-ends in Cinder already that have plans for further enhancements/optimizations as well.</div>
<div style="font-family:'courier new',monospace"><br>
</div>
<div style="font-family:'courier new',monospace">Anyway, that's just my opinion. I'd be really interested in talking with you more (maybe at the summit) regarding some of the work you're doing and some of the ideas that you have. It
would be interesting to see what we could do to improve the backup service already in Cinder.</div>
<div style="font-family:'courier new',monospace"><br>
</div>
<div style="font-family:'courier new',monospace">Thanks,</div>
<div style="font-family:'courier new',monospace">John</div>
</div>
</div>
</div>
</blockquote>
<blockquote type="cite">
<div><span>_______________________________________________</span><br>
<span>OpenStack-dev mailing list</span><br>
<span><a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a></span><br>
<span><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a></span><br>
</div>
</blockquote>
</div></div></div>
<br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><div class="gmail_default" style="font-family:'courier new',monospace">Great, I think this will be a good summit topic for sure.</div><br></div></div>