On Sun, Nov 1, 2020 at 4:32 AM Oliver Weinmann oliver.weinmann@me.com wrote:

Hi Jerome,

thanks for your reply. Ok, but can rook also be used outside of kubernetes? Can I provision e.g. volumes for openstack cinder?

Rook provides storage for k8s and Ceph is one of its storage providers. I don't think a Ceph cluster as orchestrated by rook was designed for access outside of k8s. I've seen people expose parts of it outside of k8s [0] but that type of thing doesn't seem to be a standard rook pattern. I wonder how many people are doing that.

If you want Ceph to provide pools to both k8s and OpenStack, then one approach I see is the following:

1. Install Ceph on its own independent system so it's external to both OpenStack and k8s and use cephadm to do that deployment [1] 2. Install rook for k8s access to Ceph but direct rook to the external Ceph cluster from step 1 [2] 3. Install OpenStack but direct it to the external Ceph cluster from step 1, e.g. TripleO can do this [3]

I'm suggesting a tool to install Ceph, cephadm [1], which isn't in the current list of tools in the subject. One benefit of this tool is that you'd have the new Orchestrator APIs that Jerome mentioned but you wouldn't have to use k8s if you don't want to. The downside is that you don't have k8s operators directly managing Ceph. An upside of using the 3-step approach above is that one Ceph cluster can be used as the same backend to both OpenStack and k8s services. I haven't done this myself but I don't see why this wouldn't work. You'd just manage different pools/cephx keys for different services.

If you're not planning to use k8s, the above (without step 2) would also work well for a large deployment. If you're interested in a smaller footprint with just OpenStack and Ceph, perhaps where you collocate OSD and Compute containers (aka "hyperconverged"), then TripleO will deploy that too and it uses ceph-ansible to deploy Ceph behind the scenes [4]. TripleO can also deploy CephFS for NFS access from OpenStack tenants via ganesha. This can work even if the Ceph cluster itself is external [6] though access to the NFS service is meant for OpenStack tenants, not e.g. VMWare tenants. This method of deployment will likely evolve over the next two TripleO cycles to use cephadm in place of ceph-ansible [7].

John

[0] https://www.adaltas.com/en/2020/04/16/expose-ceph-from-rook-kubernetes/ [1] https://docs.ceph.com/en/latest/cephadm [2] https://github.com/rook/rook/blob/master/design/ceph/ceph-external-cluster.m... [3] https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/features... [4] https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/features... [5] https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/features... [6] https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/features... [7] https://review.opendev.org/#/c/723108/

Am 01.11.2020 um 09:49 schrieb Jerome Pansanel:

...

Hi Oliver,

Due to recent changes in CEPH Nautilus and Octopus (new Orchestrator APIs), we decided to switch to Rook. It is still in an experimental stage on our platform, but it seems promising.

Cheers,

Jerome

Le 01/11/2020 à 08:55, Oliver Weinmann a écrit :

...

Hi,

I'm still in the process of preparing a OpenStack POC. I'm 100% sure that I want to use CEPH and so I purchased the book Mastering CEPH 2nd edition. First of all, It's a really good book. It basically explains the various methods how ceph can be deployed and also the components that CEPH is build of. So I played around a lot with ceph-ansible and rook in my virtualbox environment. I also started to play with tripleo ceph deployment, although I haven't had the time yet to sucessfully deploy a openstack cluster with CEPH. Now I'm wondering, which of these 3 methods should I use?

rook

ceph-ansible

tripleo

I also want to use CEPH for NFS and CIFS (Samba) as we have plenty of VMs running under vSphere that currently consume storage from a ZFS storage via CIFS and NFS. I don't know if rook can be used for this. I have the feeling that it is purely meant to be used for kubernetes? And If I would like to have CIFS and NFS maybe tripleo is not capable of enabling these features in CEPH? So I would be left with ceph-ansible?

Any suggestions are very welcome.

Best Regards,

Oliver

Hi Tom,

Thanks a lot for your input. Highly appreciated. John really convinced me to deploy a Standalone cluster with cephadm and so I started to deploy It. I stumbled upon a few obstacles, but mostly because commands didn’t behave as expected or myself missing some important steps like adding 3 dashes in a yml file. Cephadm looks really promising. I hope that by tomorrow I will have a simple 3 node cluster. I have to look deeper into it as of how to configure separate networks for the cluster and the Frontend but it shouldn’t be too hard. Once the cluster is functioning I will re-deploy my tripleo POC pointing to the standalone Ceph cluster. Currently I have a zfs nfs Storage configured that I would like to keep. I have to figure out how to configure multiple backends in tripleo.

Cheers, Oliver

Von meinem iPhone gesendet

Am 03.11.2020 um 16:20 schrieb Tom Barron tpb@dyncloud.net:

On 01/11/20 08:55 +0100, Oliver Weinmann wrote:

...

Hi,

I'm still in the process of preparing a OpenStack POC. I'm 100% sure that I want to use CEPH and so I purchased the book Mastering CEPH 2nd edition. First of all, It's a really good book. It basically explains the various methods how ceph can be deployed and also the components that CEPH is build of. So I played around a lot with ceph-ansible and rook in my virtualbox environment. I also started to play with tripleo ceph deployment, although I haven't had the time yet to sucessfully deploy a openstack cluster with CEPH. Now I'm wondering, which of these 3 methods should I use?

rook

ceph-ansible

tripleo

I also want to use CEPH for NFS and CIFS (Samba) as we have plenty of VMs running under vSphere that currently consume storage from a ZFS storage via CIFS and NFS. I don't know if rook can be used for this. I have the feeling that it is purely meant to be used for kubernetes? And If I would like to have CIFS and NFS maybe tripleo is not capable of enabling these features in CEPH? So I would be left with ceph-ansible?

Any suggestions are very welcome.

Best Regards,

Oliver

If your core use case is OpenStack (OpenStack POC with CEPH) then of the three tools mentioned only tripleo will deploy OpenStack. It can deploy a Ceph cluster for use by OpenStack as well, or you can deploy Ceph externally and it will "point" to it from OpenStack. For an OpenStack POC (and maybe for production too) I'd just have TripleO deploy the Ceph cluster as well. TripleO will use a Ceph deployment tool -- today, ceph-ansible; in the future, cephadm -- to do the actual Ceph cluster deployment but it figures out how to run the Ceph deployment for you.

I don't think any of these tools will install a Samba/CIFs gateway to CephFS. It's reportedly on the road map for the new cephadm tool. You may be able to manually retrofit it to your Ceph cluster by consulting upstream guidance such as [1].

NFS shares provisioned in OpenStack (Manila file service) *could* be shared out-of-cloud to VSphere VMs if networking is set up to make them accessible but the shares would remain OpenStack managed. To use the same Ceph cluster for OpenStack and vSphere but have separate storage pools behind them, and independent management, you'd need to deploy the Ceph cluster independently of OpenStack and vSphere. TripleO could "point" to this cluster as mentioned previously.

I agree with your assessment that rook is intended to provide PVs for Kubernetes consumers. You didn't mention Kubernetes as a use case, but as John Fulton has mentioned it is possible to use rook on Kubernetes in a mode where it "points" to an external ceph cluster rather than provisioning its own as well. Or if you run k8s clusters on OpenStack, they can just consume the OpenStack storage, which will be Ceph storage when that is used to back OpenStack Cinder and Manila.

-- Tom Barron

[1] https://www.youtube.com/watch?v=5v8L7FhIyOw&list=PLrBUGiINAakNCnQUosh63L...

Hi,

just for the sake of completeness. I deployed a 3 node ceph cluster using cephadm and hooked it up to openstack using tripleo. I have to say it was really easy and worked like a charm. The performance is not great but this is something I have to look into next.

Thanks again everyone and have a nice day.

Am 03.11.2020 um 21:05 schrieb Oliver Weinmann:

Hi Tom,

Thanks a lot for your input. Highly appreciated. John really convinced me to deploy a Standalone cluster with cephadm and so I started to deploy It. I stumbled upon a few obstacles, but mostly because commands didn’t behave as expected or myself missing some important steps like adding 3 dashes in a yml file. Cephadm looks really promising. I hope that by tomorrow I will have a simple 3 node cluster. I have to look deeper into it as of how to configure separate networks for the cluster and the Frontend but it shouldn’t be too hard. Once the cluster is functioning I will re-deploy my tripleo POC pointing to the standalone Ceph cluster. Currently I have a zfs nfs Storage configured that I would like to keep. I have to figure out how to configure multiple backends in tripleo.

Cheers, Oliver

Von meinem iPhone gesendet

...

Am 03.11.2020 um 16:20 schrieb Tom Barron tpb@dyncloud.net:

On 01/11/20 08:55 +0100, Oliver Weinmann wrote:

...

Hi,

I'm still in the process of preparing a OpenStack POC. I'm 100% sure that I want to use CEPH and so I purchased the book Mastering CEPH 2nd edition. First of all, It's a really good book. It basically explains the various methods how ceph can be deployed and also the components that CEPH is build of. So I played around a lot with ceph-ansible and rook in my virtualbox environment. I also started to play with tripleo ceph deployment, although I haven't had the time yet to sucessfully deploy a openstack cluster with CEPH. Now I'm wondering, which of these 3 methods should I use?

rook

ceph-ansible

tripleo

I also want to use CEPH for NFS and CIFS (Samba) as we have plenty of VMs running under vSphere that currently consume storage from a ZFS storage via CIFS and NFS. I don't know if rook can be used for this. I have the feeling that it is purely meant to be used for kubernetes? And If I would like to have CIFS and NFS maybe tripleo is not capable of enabling these features in CEPH? So I would be left with ceph-ansible?

Any suggestions are very welcome.

Best Regards,

Oliver

If your core use case is OpenStack (OpenStack POC with CEPH) then of the three tools mentioned only tripleo will deploy OpenStack. It can deploy a Ceph cluster for use by OpenStack as well, or you can deploy Ceph externally and it will "point" to it from OpenStack. For an OpenStack POC (and maybe for production too) I'd just have TripleO deploy the Ceph cluster as well. TripleO will use a Ceph deployment tool -- today, ceph-ansible; in the future, cephadm -- to do the actual Ceph cluster deployment but it figures out how to run the Ceph deployment for you.

I don't think any of these tools will install a Samba/CIFs gateway to CephFS. It's reportedly on the road map for the new cephadm tool. You may be able to manually retrofit it to your Ceph cluster by consulting upstream guidance such as [1].

NFS shares provisioned in OpenStack (Manila file service) *could* be shared out-of-cloud to VSphere VMs if networking is set up to make them accessible but the shares would remain OpenStack managed. To use the same Ceph cluster for OpenStack and vSphere but have separate storage pools behind them, and independent management, you'd need to deploy the Ceph cluster independently of OpenStack and vSphere. TripleO could "point" to this cluster as mentioned previously.

I agree with your assessment that rook is intended to provide PVs for Kubernetes consumers. You didn't mention Kubernetes as a use case, but as John Fulton has mentioned it is possible to use rook on Kubernetes in a mode where it "points" to an external ceph cluster rather than provisioning its own as well. Or if you run k8s clusters on OpenStack, they can just consume the OpenStack storage, which will be Ceph storage when that is used to back OpenStack Cinder and Manila.

-- Tom Barron

[1] https://www.youtube.com/watch?v=5v8L7FhIyOw&list=PLrBUGiINAakNCnQUosh63L...