<div dir="auto">Hello, I suggest hystax for openstack failover and failback between two openstack sites.<div dir="auto">It works with openstack upstream As well.</div><div dir="auto">Ignazio </div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Il Gio 16 Gen 2020, 23:46 Burak Hoban <<a href="mailto:Burak.Hoban@iag.com.au">Burak.Hoban@iag.com.au</a>> ha scritto:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hey Tony,<br>
<br>
Keep in mind that if you're looking to run OpenStack, but you're not feeling comfortable with the community support then there's always the option to go with a vendor backed version. These are usually a good option for those a little more risk adverse, or who don't have the time/or skills to maintain upstream releases - however going down that path usually means you can do less with OpenStack (depending on the vendor), but you have a large pool of resources to help troubleshoot and answer questions. We do both approaches internally for different clusters, so both approaches have their pro and cons.<br>
<br>
You touched on a few points in your original email...<br>
<br>
> If you had two OpenStack clusters, one in "site 1" and another in "site 2", then you could look at below for backup/restore of instances cross-cluster:<br>
- Freezer -> <a href="https://wiki.openstack.org/wiki/Freezer" rel="noreferrer noreferrer" target="_blank">https://wiki.openstack.org/wiki/Freezer</a><br>
- Trillio (basically just a series of nova snapshots under the cover) -> <a href="https://www.trilio.io/" rel="noreferrer noreferrer" target="_blank">https://www.trilio.io/</a><br>
<br>
You could then over the top roll out a file level based backup tool on each instance, this would pretty much offer you replication functionality without having to do block-level tinkering.<br>
<br>
> Failover of OpenStack controller/computes<br>
If you have two sites, you can always go for 3x Controller deployment spanning cross site. Depending on latency obviously, however all you really need is a good enough link for RabbitMQ/Galera to talk reliably etc.<br>
<br>
Failing that, I'd recommend backing up your Controller with ReaR. From there you can also schedule frequent automated jobs to do a OpenStack DB backups. Recovering should be a case of ReaR restore, load latest OpenStack DB and start everything up... You'll probably want to ensure your VLANs are spanned cross-site so you can reuse same IP addresses.<br>
<br>
<a href="https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/post_deployment/backup_and_restore/05_rear.html" rel="noreferrer noreferrer" target="_blank">https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/post_deployment/backup_and_restore/05_rear.html</a><br>
<a href="https://superuser.openstack.org/articles/tutorial-rear-openstack-deployment/" rel="noreferrer noreferrer" target="_blank">https://superuser.openstack.org/articles/tutorial-rear-openstack-deployment/</a> <br>
<br>
<br>
In reality, the best solution would be to have two isolated clusters, and your workloads spanned across both sites. Obviously that isn't always possible (from personal experience), but pushing people down the Kubernetes path and then for the rest automation/backup utilities may cater for your needs.<br>
<br>
Having said that, Albert's link does look promising -> <a href="https://docs.openstack.org/cinder/pike/contributor/replication.html" rel="noreferrer noreferrer" target="_blank">https://docs.openstack.org/cinder/pike/contributor/replication.html</a> <br>
<br>
<br>
<br>
Date: Thu, 16 Jan 2020 19:49:08 +0000<br>
From: Albert Braden <<a href="mailto:Albert.Braden@synopsys.com" target="_blank" rel="noreferrer">Albert.Braden@synopsys.com</a>><br>
To: Tony Pearce <<a href="mailto:tony.pearce@cinglevue.com" target="_blank" rel="noreferrer">tony.pearce@cinglevue.com</a>>,<br>
"<a href="mailto:openstack-discuss@lists.openstack.org" target="_blank" rel="noreferrer">openstack-discuss@lists.openstack.org</a>"<br>
<<a href="mailto:openstack-discuss@lists.openstack.org" target="_blank" rel="noreferrer">openstack-discuss@lists.openstack.org</a>><br>
Subject: RE: DR options with openstack<br>
Message-ID:<br>
<<a href="mailto:BN8PR12MB3636451FC8E2BC6A50216425D9360@BN8PR12MB3636.namprd12.prod.outlook.com" target="_blank" rel="noreferrer">BN8PR12MB3636451FC8E2BC6A50216425D9360@BN8PR12MB3636.namprd12.prod.outlook.com</a>><br>
<br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
Hi Tony,<br>
<br>
It looks like Cheesecake didn’t survive but apparently some components of it did; details in <a href="https://docs.openstack.org/cinder/pike/contributor/replication.html" rel="noreferrer noreferrer" target="_blank">https://docs.openstack.org/cinder/pike/contributor/replication.html</a><br>
<br>
I’m not using Cinder now; we used it at eBay with Ceph and Netapp backends. Netapp makes it easy but is expensive; Ceph is free but you have to figure out how to make it work. You’re right about forking; we did it and then upgrading turned from an incredibly difficult ordeal to an impossible one. It’s better to stay with the “official” code so that upgrading remains an option.<br>
<br>
I’m just an operator; hopefully someone more expert will reply with more useful info.<br>
<br>
It’s true that our community lacks participation. It’s very difficult for a new operator to start using openstack and get help with the issues that they encounter. So far this mailing list has been the best resource for me. IRC and Ask Openstack are mostly unattended. I try to help out in #openstack when I can, but I don’t know a lot so I mostly end up telling people to ask on the list. On IRC sometimes I find help by asking in other openstack-* channels. Sometimes people complain that I’m asking in a developer channel, but sometimes I get help. Persistence is the key. If I keep asking long enough in enough places, eventually someone will answer. If all else fails, I open a bug.<br>
<br>
Good luck and welcome to the Openstack community!<br>
<br>
From: Tony Pearce <<a href="mailto:tony.pearce@cinglevue.com" target="_blank" rel="noreferrer">tony.pearce@cinglevue.com</a>><br>
Sent: Wednesday, January 15, 2020 11:37 PM<br>
To: <a href="mailto:openstack-discuss@lists.openstack.org" target="_blank" rel="noreferrer">openstack-discuss@lists.openstack.org</a><br>
Subject: DR options with openstack<br>
<br>
Hi all<br>
<br>
My questions are;<br>
<br>
<br>
1. How are people using iSCSI Cinder storage with Openstack to-date? For example a Nimble Storage array backend. I mean to say, are people using backend integration drivers for other hardware (like netapp)? Or are they using backend iscsi for example?<br>
2. How are people managing DR with Openstack in terms of backend storage replication to another array in another location and continuing to use Openstack?<br>
<br>
The environment which I am currently using;<br>
1 x Nimble Storage array (iSCSI) with nimble.py Cinder driver<br>
1 x virtualised Controller node<br>
2 x physical compute nodes<br>
This is Openstack Pike.<br>
<br>
In addition, I have a 2nd Nimble Storage array in another location.<br>
<br>
To explain the questions I’d like to put forward my thoughts for question 2 first:<br>
For point 2 above, I have been searching for a way to utilise replicated volumes on the 2nd array from Openstack with existing instances. For example, if site 1 goes down how would I bring up openstack in the 2nd location and boot up the instances where their volumes are stored on the 2nd array. I found a proposal for something called “cheesecake” ref: <a href="https://specs.openstack.org/openstack/cinder-specs/specs/rocky/cheesecake-promote-backend.html" rel="noreferrer noreferrer" target="_blank">https://specs.openstack.org/openstack/cinder-specs/specs/rocky/cheesecake-promote-backend.html</a><<a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__specs.openstack.org_openstack_cinder-2Dspecs_specs_rocky_cheesecake-2Dpromote-2Dbackend.html&d=DwMFaQ&c=DPL6_X_6JkXFx7AXWqB0tg&r=XrJBXYlVPpvOXkMqGPz6KucRW_ils95ZMrEmlTflPm8&m=e2jC0sFEUAs6byl7JOv5IAZTKPkABl-Eh6rQwQ55tWk&s=oVEr3DpxprOpbuxZ_4WSfSqAVCaZUlPCFT6g6DsqQHQ&e=" rel="noreferrer noreferrer" target="_blank">https://urldefense.proofpoint.com/v2/url?u=https-3A__specs.openstack.org_openstack_cinder-2Dspecs_specs_rocky_cheesecake-2Dpromote-2Dbackend.html&d=DwMFaQ&c=DPL6_X_6JkXFx7AXWqB0tg&r=XrJBXYlVPpvOXkMqGPz6KucRW_ils95ZMrEmlTflPm8&m=e2jC0sFEUAs6byl7JOv5IAZTKPkABl-Eh6rQwQ55tWk&s=oVEr3DpxprOpbuxZ_4WSfSqAVCaZUlPCFT6g6DsqQHQ&e=</a>><br>
But I could not find if it had been approved or implemented. So I return to square 1. I have some thoughts about failing over the controller VM and compute node but I don’t think there’s any need to go into here because of the above blocker and for brevity anyway.<br>
<br>
The nimble.py driver which I am using came with Openstack Pike and it appears Nimble / HPE are not maintaining it any longer. I saw a commit to remove nimble.py in Openstack Train release. The driver uses the REST API to perform actions on the array. Such as creating a volume, downloading the image, mounting the volume to the instance, snapshots, clones etc. This is great for me because to date I have around 10TB of openstack storage data allocated and the Nimble array shows the amount of data being consumed is <900GB. This is due to the compression and zero-byte snapshots and clones.<br>
<br>
So coming back to question 2 – is it possible? Can you drop me some keywords that I can search for such as an Openstack component like Cheesecake? I think basically what I am looking for is a supported way of telling Openstack that the instance volumes are now located at the new / second array. This means a new cinder backend. Example, new iqn, IP address, volume serial number. I think I could probably hack the cinder db but I really want to avoid that.<br>
<br>
So failing the above, it brings me to the question 1 I asked before. How are people using Cinder volumes? May be I am going about this the wrong way and need to take a few steps backwards to go forwards? I need storage to be able to deploy instances onto. Snapshots and clones are desired. At the moment these operations take less time than the horizon dashboard takes to load because of the waiting API responses.<br>
<br>
When searching for information about the above as an end-user / consumer I get a bit concerned. Is it right that Openstack usage is dropping? There’s no web forum to post questions. The chatroom on freenode is filled with ~300 ghosts. Ask Openstack questions go without response. Earlier this week (before I found this mail list) I had to use facebook to report that the Openstack.org website had been hacked. Basically it seems that if you’re a developer that can write code then you’re in but that’s it. I have never been a coder and so I am somewhat stuck.<br>
<br>
Thanks in advance<br>
<br>
Sent from Mail<<a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__go.microsoft.com_fwlink_-3FLinkId-3D550986&d=DwMFaQ&c=DPL6_X_6JkXFx7AXWqB0tg&r=XrJBXYlVPpvOXkMqGPz6KucRW_ils95ZMrEmlTflPm8&m=e2jC0sFEUAs6byl7JOv5IAZTKPkABl-Eh6rQwQ55tWk&s=Qo1wKkAeo1uTCH83dVO-IVt4MWhQRk7rg3xKmlzPGhI&e=" rel="noreferrer noreferrer" target="_blank">https://urldefense.proofpoint.com/v2/url?u=https-3A__go.microsoft.com_fwlink_-3FLinkId-3D550986&d=DwMFaQ&c=DPL6_X_6JkXFx7AXWqB0tg&r=XrJBXYlVPpvOXkMqGPz6KucRW_ils95ZMrEmlTflPm8&m=e2jC0sFEUAs6byl7JOv5IAZTKPkABl-Eh6rQwQ55tWk&s=Qo1wKkAeo1uTCH83dVO-IVt4MWhQRk7rg3xKmlzPGhI&e=</a>> for Windows 10<br>
<br>
<br>
_____________________________________________________________________<br>
<br>
The information transmitted in this message and its attachments (if any) is intended <br>
only for the person or entity to which it is addressed.<br>
The message may contain confidential and/or privileged material. Any review, <br>
retransmission, dissemination or other use of, or taking of any action in reliance <br>
upon this information, by persons or entities other than the intended recipient is <br>
prohibited.<br>
<br>
If you have received this in error, please contact the sender and delete this e-mail <br>
and associated material from any computer.<br>
<br>
The intended recipient of this e-mail may only use, reproduce, disclose or distribute <br>
the information contained in this e-mail and any attached files, with the permission <br>
of the sender.<br>
<br>
This message has been scanned for viruses.<br>
_____________________________________________________________________<br>
</blockquote></div>