[openstack][backup] Experience for instance backup
Hello guys. I am looking for instance backup solution. I am using Cinder backup with nfs backup but it looks not too fast. I am using a 10Gbps network. I would like to know experience for best practice for instance backup solutions on Openstack. Thank you. Nguyen Huu Khoi
Hi, could you be more specific what is "not too fast" for you? I don't really have too much information from my side, but I can describe how we do it. We use Ceph as back end for all services (nova, glance, cinder), and the most important machines are backed up by our backup server directly via rbd commands: - We create a snapshot of the running instances and export the snapshot to an external drive, this is a full backup. In addition to that many of our machines have their home or working directories mounted from CephFS which we also backup as a tar ball to an external drive once a week (also full backup). And then we have also a "real" backup solution in place (bacula) where we store incremental as well as full backups from individually configured resources for different intervals. All these different approaches have different runtimes, of course. Just as an example, one 40 GB VM (rbd image) which has areound 24 GB in-use takes around 6 minutes for the full backup. Although we also have the cinder-backup service up and running nobody is using it because the important volumes are attached to instances which we backup by our rbd solution, so there's no real need for that. Regards, Eugen Zitat von Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com>:
Hello guys. I am looking for instance backup solution. I am using Cinder backup with nfs backup but it looks not too fast. I am using a 10Gbps network. I would like to know experience for best practice for instance backup solutions on Openstack. Thank you. Nguyen Huu Khoi
Hey there, On 06/03/2023 22:34, Nguyễn Hữu Khôi wrote:
I am looking for instance backup solution. I am using Cinder backup with nfs backup but it looks not too fast. I am using a 10Gbps network. I would like to know experience for best practice for instance backup solutions on Openstack.
On 13/03/2023 12:46, Eugen Block wrote:
We use Ceph as back end for all services (nova, glance, cinder), and the most important machines are backed up by our backup server directly via rbd commands:
There is RBD and "the other" drivers. While RBD uses the native export / import feature of Ceph, all other drivers (file, NFS, object storages like S3) are based on the abstract chunked driver (https://opendev.org/openstack/cinder/src/branch/master/cinder/backup/chunked...). This driver reads the volume / image and treats it as chunks before making use of a concrete driver (e.g. NFS or S3) to send those chunks off somewhere to be stored. Restore works just the opposite way. The performance of the chunked driver based back-ends is not (yet) comparable to what RBD can achieve due to various reasons. But again, while "RBD" uses Ceph's mechanisms internally all other "targets" for backup storage work differently. We ourselves were looking into using and S3-compatible storage and thus I started a dicsussion about the state of those other drivers at https://lists.openstack.org/pipermail/openstack-discuss/2022-September/03026... This then led to a discussion at the Cinder PTG https://etherpad.opendev.org/p/antelope-ptg-cinder#L119 with many observations. There also are changes in the works, like restore into sparse volumes (https://review.opendev.org/c/openstack/cinder/+/852654) when going via the chunked driver. But also features like "encryption" (https://review.opendev.org/c/openstack/cinder-specs/+/862601) are being discussed. Regards Christian
Hello, Indeed and interesting topic for the PTG, we are using the Swift backup driver and also had issues with backups timing out due to Keystone tokens of which we have done some work to mitigate. Best regards Tobias
On 13 Mar 2023, at 17:11, Christian Rohmann <christian.rohmann@inovex.de> wrote:
Hey there,
On 06/03/2023 22:34, Nguyễn Hữu Khôi wrote:
I am looking for instance backup solution. I am using Cinder backup with nfs backup but it looks not too fast. I am using a 10Gbps network. I would like to know experience for best practice for instance backup solutions on Openstack.
On 13/03/2023 12:46, Eugen Block wrote:
We use Ceph as back end for all services (nova, glance, cinder), and the most important machines are backed up by our backup server directly via rbd commands:
There is RBD and "the other" drivers. While RBD uses the native export / import feature of Ceph, all other drivers (file, NFS, object storages like S3) are based on the abstract chunked driver (https://opendev.org/openstack/cinder/src/branch/master/cinder/backup/chunked...). This driver reads the volume / image and treats it as chunks before making use of a concrete driver (e.g. NFS or S3) to send those chunks off somewhere to be stored. Restore works just the opposite way. The performance of the chunked driver based back-ends is not (yet) comparable to what RBD can achieve due to various reasons.
But again, while "RBD" uses Ceph's mechanisms internally all other "targets" for backup storage work differently. We ourselves were looking into using and S3-compatible storage and thus I started a dicsussion about the state of those other drivers at https://lists.openstack.org/pipermail/openstack-discuss/2022-September/03026...
This then led to a discussion at the Cinder PTG https://etherpad.opendev.org/p/antelope-ptg-cinder#L119 with many observations.
There also are changes in the works, like restore into sparse volumes (https://review.opendev.org/c/openstack/cinder/+/852654) when going via the chunked driver. But also features like "encryption" (https://review.opendev.org/c/openstack/cinder-specs/+/862601) are being discussed.
Regards
Christian
Thank Christian, I will try to follow it. *Hello * Eugen, I use SAN to back our openstack services and I planned to use NFS for Cinder backup. Because of that, We separate tenants for different departments. So they can back up by themself. Thank you for your sharing. Nguyen Huu Khoi On Mon, Mar 13, 2023 at 11:11 PM Christian Rohmann < christian.rohmann@inovex.de> wrote:
Hey there,
On 06/03/2023 22:34, Nguyễn Hữu Khôi wrote:
I am looking for instance backup solution. I am using Cinder backup with nfs backup but it looks not too fast. I am using a 10Gbps network. I would like to know experience for best practice for instance backup solutions on Openstack.
On 13/03/2023 12:46, Eugen Block wrote:
We use Ceph as back end for all services (nova, glance, cinder), and the most important machines are backed up by our backup server directly via rbd commands:
There is RBD and "the other" drivers. While RBD uses the native export / import feature of Ceph, all other drivers (file, NFS, object storages like S3) are based on the abstract chunked driver ( https://opendev.org/openstack/cinder/src/branch/master/cinder/backup/chunked... ). This driver reads the volume / image and treats it as chunks before making use of a concrete driver (e.g. NFS or S3) to send those chunks off somewhere to be stored. Restore works just the opposite way. The performance of the chunked driver based back-ends is not (yet) comparable to what RBD can achieve due to various reasons.
But again, while "RBD" uses Ceph's mechanisms internally all other "targets" for backup storage work differently. We ourselves were looking into using and S3-compatible storage and thus I started a dicsussion about the state of those other drivers at
https://lists.openstack.org/pipermail/openstack-discuss/2022-September/03026...
This then led to a discussion at the Cinder PTG https://etherpad.opendev.org/p/antelope-ptg-cinder#L119 with many observations.
There also are changes in the works, like restore into sparse volumes (https://review.opendev.org/c/openstack/cinder/+/852654) when going via the chunked driver. But also features like "encryption" (https://review.opendev.org/c/openstack/cinder-specs/+/862601) are being discussed.
Regards
Christian
participants (4)
-
Christian Rohmann
-
Eugen Block
-
Nguyễn Hữu Khôi
-
Tobias Urdin