Open Stack

Hi Preston,
Thank you. You saw Fabrizio in Vancouver, I'm Fausto, but it's allright, : P

The challenge is interesting. If we want to build a dedicated backup API
service (which is always what we wanted to do), probably we need to:


   - Place out of Nova and Cinder the backup features, as it wouldn't make
   much sense to me to have a Backup service and also have backups managed
   independently by Nova and Cinder.


That said, I'm not a big fan of the following:

   - Interacting with the hypervisors and the volumes directly without
   passing through the Nova and Cinder API.
   - Adding any additional workload on the compute nodes or block storage
   nodes.
   - Computing incremental, compression, encryption is expensive. Have many
   simultaneous process doing that may lead  to bad behaviours on core
   services.


Some open questions:

   - Are we going to implement in Nova and Cinder, the features we need
   from the hypervisor for the backups i.e.:
      - http://wiki.qemu.org/Features/Snapshots2
      - http://wiki.qemu.org/Features/Livebackup


   - Are we going to talk directly with the hypervisors without passing
   through the services API?
   - Are we going to provide a backup service using the feature from Nova
   and Cinder API currently implemented, and improve it over time as long as
   more related advanced features are available?
   - No other service touch directly volumes and hypervisors for reasons.
   We would need to make a diamond case to get an exception for backups.
   - There's any requirement from vendors and any of them will allocate
   resources to make a backup service, as a Team?


My (flexible) thoughts are:

   - The feature is needed and is brilliant.
   - We should probably implement the newest feature provided by the
   hypervisor in Nova and export them from the Nova API.
   - Create a plugin that is integrated with Freezer to leverage that new
   features.
   - Same apply for Cinder.
   - The VMs and Volumes backup feature is already available by Nova,
   Cinder and Freezer. It needs to be improved for sure a lot, but do we need
   to create a new project for a feature that needs to be improved, rather
   than work with the existing Teams?
   - No one wants to block others, Sam proposed solution is indeed
   remarkable, but this is OpenStack, we work in Teams, why we cannot do that
   and be less fragmented.


Thanks,
Fausto


On Mon, Feb 1, 2016 at 9:23 PM, Preston L. Bannister <preston at bannister.us>
wrote:

> Hi Fausto,
>
> To be clear, I am not in any way critical of Freezer and the folk putting
> in work. (Please, I want to be *entirely* clear on this point! Also, saw
> your presentation in Vancouver.)
>
> That said, Freezer is a bit of a Swiss-Army-Knife set of combined backup
> functions. Sometimes it is better to focus on a single aspect (or few). The
> new features landing in QEMU present an opportunity. A project focused
> solely on that opportunity, to work through initial set of issues, makes a
> lot of sense to me. (Something like how KVM forked QEMU for a time, to
> build faster x86 emulation.)
>
> I do not see these as competing projects, and more as cooperative. The
> Ekko work should be able to plug into Freezer, cleanly.
>
> Aspects of the problem, as I see:
>
>    1. Extracting efficient instance backups from OpenStack (via new QEMU
>    function).
>    2. Storing backups in efficient form (general implementation, and
>    vendor-specific supersets).
>    3. Offering an OpenStack backup-service API, with core and
>    vendor-specific extensions.
>
>
> Vendors (like my employer, EMC) might be somewhat opinionated about (2),
> and for reason. :)
>
> The huge missing piece is (1), and a focused project seems to make a lot
> of sense.
>
> As to (3), that looks like a good topic for further discussion. :)
>
>
> My $.02.
>
>
>
>
> On Sat, Jan 30, 2016 at 5:36 PM, Fausto Marzi <fausto.marzi at gmail.com>
> wrote:
>
>> Hi Preston,
>> No need to apologize. They are aspect of the same problem.
>> However, VMs backup is one of the many aspects that we are approaching
>> here, such as:
>>
>> - VM backups
>> - Volumes backups
>> - Specific applications consistent data backup (i.e. MySQL, Mongo, file
>> system, etc)
>> - Provide capabilities to restore data even if keystone and swift are not
>> available
>> - Upload data during backup to multiple media storage in parallel
>> - Web Interface
>> - Provide capability to synchronize backups for sharded data on multiple
>> nodes
>> - Encryption
>> - File based incremental
>> - Block based incremental
>> - Tenant related data backup and restore
>> - Multi platform OS support (i.e. Linux, BSD, OSX, Windows, iOS, Android,
>> etc)
>> - Everything is upstreamed.
>>
>> This looks like a list of features... and actually it is.
>>
>> Block based and some multi platform OS aside, all the mentioned features
>> are provided to the date. Most of them are available since Kilo.
>>
>> I agree with the multi API, room for vendors and to provide different
>> approaches, but please let me say something (*not referring specifically
>> to you or Sam or anyone*)
>>
>> All the time people say you have to do this and that, but the facts are
>> that at the end of the day, always the same 6 engineers (not even full
>> time) are working on it since 2 years, investing professional and personal
>> time on it.
>>
>> We try to be open, to accept everybody (even the most arrogant), to
>> implement features for whoever needs it, but the facts are that the only
>> Companies that invested on it are HP, a bit Ericsson and Orange (apologize
>> if I forgot anyone). We never said no to anyone about anything, never
>> focused only to a single Company influence, never blocked a thing... and
>> never will.
>>
>> Wouldn't be better to join efforts if companies need a backup solution
>> and have their own requirements implemented by a common public Team, rather
>> then start creating many tools to solve the same set of problems? How can
>> ever competition benefit this? How can ever fragmenting projects help to
>> provide a better solution?
>>
>> I'm sorry, but unless the TC or many people from the Community, tell us
>> to do something different (in that case we'll do it straight away), we'll
>> keep doing what we are doing, focusing on delivering what we think is the
>> most advanced solution, according the resources and time we have.
>>
>> We need to understand that here the most important thing is to work in
>> Team, to provide great tools to the Community, rather then thinking to be
>> PTL or maintain independence just for the sake of it or focusing only on
>> what's the best for a single Company. If this vision is not shared, then,
>> unfortunately, good luck competing, while if the vision is shared... let's
>> do together unprecedented things.
>>
>> Many thanks,
>> Fausto
>>
>>
>> On Sun, Jan 31, 2016 at 1:01 AM, Preston L. Bannister <
>> preston at bannister.us> wrote:
>>
>>> Seems to me there are three threads here.
>>>
>>> The Freezer folk were given a task, and did the best possible to support
>>> backup given what OpenStack allowed. To date, OpenStack is simply not very
>>> good at supporting backup as a service. (Apologies to the Freezer folk if I
>>> misinterpreted.)
>>>
>>> The patches (finally) landing in QEMU in support of incremental backup
>>> could be the basis for efficient backup services in OpenStack. This is all
>>> fairly high risk, in the short term. The bits that landed in QEMU 2.4 may
>>> not be sufficient (there are more QEMU patches trying to land). When put
>>> into production, we may find faults. For use in OpenStack, we may need
>>> changes in libvirt, and/or in Nova. (Or *maybe* not if usage for backup
>>> proves orthogonal.)  The only way to work out the prior is to start. The
>>> timeline could be months or years.
>>>
>>> There is a need for a common API for backup as a service in the cloud.
>>> Something more than imitating AWS. Allow some room for vendors with
>>> differing approach.
>>>
>>> I see the above as not competing, but aspects of the same problem.
>>>
>>>
>>> ​
>>>
>>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160201/7527a365/attachment.html>