[neutron][ovn] ovn database backup and restore
Folks, I have OVN running on 3 node controllers with a RAFT clustering. What is the best way to take backup of all the databases of nb_db and sb_db and how to restore them during a full disaster situation? Is the following the correct way to take backup? ovsdb-client backup unix:///run/ovn/ovnnb_db.sock OVN_Northbound > nb-snap-$(date +%s).txt ovsdb-client backup unix:///run/ovn/ovnsb_db.sock OVN_Southbound > sb-snap-$(date +%s).txt During restore on what node we should restore and does restore work on fresh cluster or existing cluster? How do other folks handle disaster scenarios where all 3 nodes are down or required migration?
On Wed, Jan 1, 2025 at 10:17 PM Satish Patel <satish.txt@gmail.com> wrote:
Folks,
I have OVN running on 3 node controllers with a RAFT clustering. What is the best way to take backup of all the databases of nb_db and sb_db and how to restore them during a full disaster situation?
Is the following the correct way to take backup?
ovsdb-client backup unix:///run/ovn/ovnnb_db.sock OVN_Northbound > nb-snap-$(date +%s).txt
ovsdb-client backup unix:///run/ovn/ovnsb_db.sock OVN_Southbound > sb-snap-$(date +%s).txt
That is what we do when migrating OVN data from RHOSP 17 to RHOSO 18 (Red Hat distro): https://openstack-k8s-operators.github.io/data-plane-adoption/user/index.htm... We also update to new schema but you probably don't need that for simple backup.
During restore on what node we should restore and does restore work on fresh cluster or existing cluster?
I believe you will have to rebuild your RAFT cluster. Start with one node, restore data there, then join cluster for more nodes. You may find some useful instructions in upstream docs for ovs raft implementation: https://docs.openvswitch.org/en/latest/ref/ovsdb.7/#manual-cluster-recovery
How do other folks handle disaster scenarios where all 3 nodes are down or required migration?
Hi Ihar, Thank you so much for your reply, There is one more utility called "neutron-ovn-db-sync-util" which sync data from neutron DC to ovn DB. What is the difference between both methods? When should I use backup/restore vs neutron-ovn-db-sync-util utility? On Thu, Jan 2, 2025 at 9:45 AM Ihar Hrachyshka <ihrachys@redhat.com> wrote:
On Wed, Jan 1, 2025 at 10:17 PM Satish Patel <satish.txt@gmail.com> wrote:
Folks,
I have OVN running on 3 node controllers with a RAFT clustering. What is the best way to take backup of all the databases of nb_db and sb_db and how to restore them during a full disaster situation?
Is the following the correct way to take backup?
ovsdb-client backup unix:///run/ovn/ovnnb_db.sock OVN_Northbound > nb-snap-$(date +%s).txt
ovsdb-client backup unix:///run/ovn/ovnsb_db.sock OVN_Southbound > sb-snap-$(date +%s).txt
That is what we do when migrating OVN data from RHOSP 17 to RHOSO 18 (Red Hat distro): https://openstack-k8s-operators.github.io/data-plane-adoption/user/index.htm... We also update to new schema but you probably don't need that for simple backup.
During restore on what node we should restore and does restore work on fresh cluster or existing cluster?
I believe you will have to rebuild your RAFT cluster. Start with one node, restore data there, then join cluster for more nodes. You may find some useful instructions in upstream docs for ovs raft implementation: https://docs.openvswitch.org/en/latest/ref/ovsdb.7/#manual-cluster-recovery
How do other folks handle disaster scenarios where all 3 nodes are down or required migration?
The neutron tool will enforce translation from neutron db to ovn db. It's useful when you do not have backup files for OVN. It mostly works, except for some resources that are stored in OVN db but are not managed by neutron (e.g. Octavia creates some Load Balancers that cannot be restored by neutron tool; I believe Octavia team was working on a similar tool but I don't know if it was ever completed). The neutron tool may take longer time to restore because it runs through complete transformation of each neutron object into the corresponding OVN NB entities. tl;dr if you have OVN backups, restore using the official OVS raft method; otherwise, - or if you are not sure how to reset a RAFT cluster from scratch, - run the neutron tool. On Thu, Jan 2, 2025 at 9:25 PM Satish Patel <satish.txt@gmail.com> wrote:
Hi Ihar,
Thank you so much for your reply, There is one more utility called "neutron-ovn-db-sync-util" which sync data from neutron DC to ovn DB. What is the difference between both methods? When should I use backup/restore vs neutron-ovn-db-sync-util utility?
On Thu, Jan 2, 2025 at 9:45 AM Ihar Hrachyshka <ihrachys@redhat.com> wrote:
On Wed, Jan 1, 2025 at 10:17 PM Satish Patel <satish.txt@gmail.com> wrote:
Folks,
I have OVN running on 3 node controllers with a RAFT clustering. What is the best way to take backup of all the databases of nb_db and sb_db and how to restore them during a full disaster situation?
Is the following the correct way to take backup?
ovsdb-client backup unix:///run/ovn/ovnnb_db.sock OVN_Northbound > nb-snap-$(date +%s).txt
ovsdb-client backup unix:///run/ovn/ovnsb_db.sock OVN_Southbound > sb-snap-$(date +%s).txt
That is what we do when migrating OVN data from RHOSP 17 to RHOSO 18 (Red Hat distro): https://openstack-k8s-operators.github.io/data-plane-adoption/user/index.htm... We also update to new schema but you probably don't need that for simple backup.
During restore on what node we should restore and does restore work on fresh cluster or existing cluster?
I believe you will have to rebuild your RAFT cluster. Start with one node, restore data there, then join cluster for more nodes. You may find some useful instructions in upstream docs for ovs raft implementation: https://docs.openvswitch.org/en/latest/ref/ovsdb.7/#manual-cluster-recovery
How do other folks handle disaster scenarios where all 3 nodes are down or required migration?
For Octavia, we contributed this months ago but it's not being reviewed after many attempts :( https://review.opendev.org/c/openstack/ovn-octavia-provider/+/929039/21 I'm seeing this a bit all over OpenStack where key fixes get ignored and it's really unfortunate when we invest time and effort into a key fix and upstream it but no reviewing gets done. Get Outlook for iOS<https://aka.ms/o0ukef> ________________________________ From: Ihar Hrachyshka <ihrachys@redhat.com> Sent: Thursday, January 2, 2025 10:24:00 PM To: Satish Patel <satish.txt@gmail.com> Cc: OpenStack Discuss <openstack-discuss@lists.openstack.org> Subject: Re: [neutron][ovn] ovn database backup and restore The neutron tool will enforce translation from neutron db to ovn db. It's useful when you do not have backup files for OVN. It mostly works, except for some resources that are stored in OVN db but are not managed by neutron (e.g. Octavia creates some Load Balancers that cannot be restored by neutron tool; I believe Octavia team was working on a similar tool but I don't know if it was ever completed). The neutron tool may take longer time to restore because it runs through complete transformation of each neutron object into the corresponding OVN NB entities. tl;dr if you have OVN backups, restore using the official OVS raft method; otherwise, - or if you are not sure how to reset a RAFT cluster from scratch, - run the neutron tool. On Thu, Jan 2, 2025 at 9:25 PM Satish Patel <satish.txt@gmail.com<mailto:satish.txt@gmail.com>> wrote: Hi Ihar, Thank you so much for your reply, There is one more utility called "neutron-ovn-db-sync-util" which sync data from neutron DC to ovn DB. What is the difference between both methods? When should I use backup/restore vs neutron-ovn-db-sync-util utility? On Thu, Jan 2, 2025 at 9:45 AM Ihar Hrachyshka <ihrachys@redhat.com<mailto:ihrachys@redhat.com>> wrote: On Wed, Jan 1, 2025 at 10:17 PM Satish Patel <satish.txt@gmail.com<mailto:satish.txt@gmail.com>> wrote: Folks, I have OVN running on 3 node controllers with a RAFT clustering. What is the best way to take backup of all the databases of nb_db and sb_db and how to restore them during a full disaster situation? Is the following the correct way to take backup? ovsdb-client backup unix:///run/ovn/ovnnb_db.sock OVN_Northbound > nb-snap-$(date +%s).txt ovsdb-client backup unix:///run/ovn/ovnsb_db.sock OVN_Southbound > sb-snap-$(date +%s).txt That is what we do when migrating OVN data from RHOSP 17 to RHOSO 18 (Red Hat distro): https://openstack-k8s-operators.github.io/data-plane-adoption/user/index.htm... We also update to new schema but you probably don't need that for simple backup. During restore on what node we should restore and does restore work on fresh cluster or existing cluster? I believe you will have to rebuild your RAFT cluster. Start with one node, restore data there, then join cluster for more nodes. You may find some useful instructions in upstream docs for ovs raft implementation: https://docs.openvswitch.org/en/latest/ref/ovsdb.7/#manual-cluster-recovery How do other folks handle disaster scenarios where all 3 nodes are down or required migration?
Thank you Mohammed for the link I will test myself. But agreed that important patches shouldn't be left out. I Hope there is a flag somewhere showing the criticality of the patch. On Thu, Jan 2, 2025 at 11:40 PM Mohammed Naser <mnaser@vexxhost.com> wrote:
For Octavia, we contributed this months ago but it's not being reviewed after many attempts :(
https://review.opendev.org/c/openstack/ovn-octavia-provider/+/929039/21
I'm seeing this a bit all over OpenStack where key fixes get ignored and it's really unfortunate when we invest time and effort into a key fix and upstream it but no reviewing gets done.
Get Outlook for iOS <https://aka.ms/o0ukef>
------------------------------
*From:* Ihar Hrachyshka <ihrachys@redhat.com> *Sent:* Thursday, January 2, 2025 10:24:00 PM *To:* Satish Patel <satish.txt@gmail.com> *Cc:* OpenStack Discuss <openstack-discuss@lists.openstack.org> *Subject:* Re: [neutron][ovn] ovn database backup and restore
The neutron tool will enforce translation from neutron db to ovn db. It's useful when you do not have backup files for OVN. It mostly works, except for some resources that are stored in OVN db but are not managed by neutron (e.g. Octavia creates some Load Balancers that cannot be restored by neutron tool; I believe Octavia team was working on a similar tool but I don't know if it was ever completed). The neutron tool may take longer time to restore because it runs through complete transformation of each neutron object into the corresponding OVN NB entities.
tl;dr if you have OVN backups, restore using the official OVS raft method; otherwise, - or if you are not sure how to reset a RAFT cluster from scratch, - run the neutron tool.
On Thu, Jan 2, 2025 at 9:25 PM Satish Patel <satish.txt@gmail.com> wrote:
Hi Ihar,
Thank you so much for your reply, There is one more utility called "neutron-ovn-db-sync-util" which sync data from neutron DC to ovn DB. What is the difference between both methods? When should I use backup/restore vs neutron-ovn-db-sync-util utility?
On Thu, Jan 2, 2025 at 9:45 AM Ihar Hrachyshka <ihrachys@redhat.com> wrote:
On Wed, Jan 1, 2025 at 10:17 PM Satish Patel <satish.txt@gmail.com> wrote:
Folks,
I have OVN running on 3 node controllers with a RAFT clustering. What is the best way to take backup of all the databases of nb_db and sb_db and how to restore them during a full disaster situation?
Is the following the correct way to take backup?
ovsdb-client backup unix:///run/ovn/ovnnb_db.sock OVN_Northbound > nb-snap-$(date +%s).txt
ovsdb-client backup unix:///run/ovn/ovnsb_db.sock OVN_Southbound > sb-snap-$(date +%s).txt
That is what we do when migrating OVN data from RHOSP 17 to RHOSO 18 (Red Hat distro): https://openstack-k8s-operators.github.io/data-plane-adoption/user/index.htm... We also update to new schema but you probably don't need that for simple backup.
During restore on what node we should restore and does restore work on fresh cluster or existing cluster?
I believe you will have to rebuild your RAFT cluster. Start with one node, restore data there, then join cluster for more nodes. You may find some useful instructions in upstream docs for ovs raft implementation: https://docs.openvswitch.org/en/latest/ref/ovsdb.7/#manual-cluster-recovery
How do other folks handle disaster scenarios where all 3 nodes are down or required migration?
This is very informative. Thank you Ihar On Thu, Jan 2, 2025 at 10:24 PM Ihar Hrachyshka <ihrachys@redhat.com> wrote:
The neutron tool will enforce translation from neutron db to ovn db. It's useful when you do not have backup files for OVN. It mostly works, except for some resources that are stored in OVN db but are not managed by neutron (e.g. Octavia creates some Load Balancers that cannot be restored by neutron tool; I believe Octavia team was working on a similar tool but I don't know if it was ever completed). The neutron tool may take longer time to restore because it runs through complete transformation of each neutron object into the corresponding OVN NB entities.
tl;dr if you have OVN backups, restore using the official OVS raft method; otherwise, - or if you are not sure how to reset a RAFT cluster from scratch, - run the neutron tool.
On Thu, Jan 2, 2025 at 9:25 PM Satish Patel <satish.txt@gmail.com> wrote:
Hi Ihar,
Thank you so much for your reply, There is one more utility called "neutron-ovn-db-sync-util" which sync data from neutron DC to ovn DB. What is the difference between both methods? When should I use backup/restore vs neutron-ovn-db-sync-util utility?
On Thu, Jan 2, 2025 at 9:45 AM Ihar Hrachyshka <ihrachys@redhat.com> wrote:
On Wed, Jan 1, 2025 at 10:17 PM Satish Patel <satish.txt@gmail.com> wrote:
Folks,
I have OVN running on 3 node controllers with a RAFT clustering. What is the best way to take backup of all the databases of nb_db and sb_db and how to restore them during a full disaster situation?
Is the following the correct way to take backup?
ovsdb-client backup unix:///run/ovn/ovnnb_db.sock OVN_Northbound > nb-snap-$(date +%s).txt
ovsdb-client backup unix:///run/ovn/ovnsb_db.sock OVN_Southbound > sb-snap-$(date +%s).txt
That is what we do when migrating OVN data from RHOSP 17 to RHOSO 18 (Red Hat distro): https://openstack-k8s-operators.github.io/data-plane-adoption/user/index.htm... We also update to new schema but you probably don't need that for simple backup.
During restore on what node we should restore and does restore work on fresh cluster or existing cluster?
I believe you will have to rebuild your RAFT cluster. Start with one node, restore data there, then join cluster for more nodes. You may find some useful instructions in upstream docs for ovs raft implementation: https://docs.openvswitch.org/en/latest/ref/ovsdb.7/#manual-cluster-recovery
How do other folks handle disaster scenarios where all 3 nodes are down or required migration?
participants (3)
-
Ihar Hrachyshka
-
Mohammed Naser
-
Satish Patel