Thank you, I'd read it, but as there also are other methods such as ceph-ansible and puppet-ceph I am trying to get a feedback from other operators about their experiences, as in this case I'm particularly interested in integration of CEPH with OpenStack Best regards -- Francesco Di Nucci System Administrator Compute & Networking Service, INFN Naples Email:francesco.dinucci@na.infn.it On 19/05/25 16:29, Maksim Malchuk wrote:

Hi Francesco,

The CEPH community recommends using CEPHADM as the primary tool for deploying CEPH: https://docs.ceph.com/en/latest/install/#recommended-methods

On Mon, May 19, 2025 at 4:19 PM Francesco Di Nucci <francesco.dinucci@na.infn.it> wrote:

Hi all,

we're planning (finally) to setup a CEPH cluster to be used as OpenStack backend.

The cloud is actually set up with Foreman+Puppet, to setup CEPH what would you advice?

cephadm as it's the preferred method in the CEPH docs or Puppet with the puppet-ceph module, as it's part of openstack?

Thanks in advance

-- Francesco Di Nucci System Administrator Compute & Networking Service, INFN Naples

Email: francesco.dinucci@na.infn.it

-- Regards, Maksim Malchuk

Hi Francesco, RedHat/IBM decided to deprecate ceph-ansible in CEPH 5 (Pacific) in favor of CEPHADM: https://github.com/ceph/ceph-ansible/commit/a9d1ec844d24fcc3ddea7c030eff4cd6... On Tue, May 20, 2025 at 10:04 AM Joel McLean <joel.mclean@micron21.com> wrote:

G’day Francesco,

Over the last year we stood up an Openstack using Kolla-ansible, with storage provided by a few Ceph clusters, which we provisioned using cephadm.

We’ve also used ProxmoxVE and PVE-ceph, and well as PVE + cephadm; the modern cephadm process is just too easy, and is fairly well documented.

Integration of external ceph with kolla-ansible is also very well documented, although is pretty agnostic to how Ceph itself is deployed.

Kind Regards,

Joel McLean

*From:* Francesco Di Nucci <francesco.dinucci@na.infn.it> *Sent:* Tuesday, 20 May 2025 4:55 PM *To:* Maksim Malchuk <maksim.malchuk@gmail.com> *Cc:* openstack-discuss <openstack-discuss@lists.openstack.org> *Subject:* Re: CEPH deployment method: cephadm vs puppet

Thank you,

I'd read it, but as there also are other methods such as ceph-ansible and puppet-ceph I am trying to get a feedback from other operators about their experiences, as in this case I'm particularly interested in integration of CEPH with OpenStack

Best regards

--

Francesco Di Nucci

System Administrator

Compute & Networking Service, INFN Naples

Email: francesco.dinucci@na.infn.it

On 19/05/25 16:29, Maksim Malchuk wrote:

Hi Francesco,

The CEPH community recommends using CEPHADM as the primary tool for deploying CEPH: https://docs.ceph.com/en/latest/install/#recommended-methods

On Mon, May 19, 2025 at 4:19 PM Francesco Di Nucci < francesco.dinucci@na.infn.it> wrote:

Hi all,

we're planning (finally) to setup a CEPH cluster to be used as OpenStack backend.

The cloud is actually set up with Foreman+Puppet, to setup CEPH what would you advice?

cephadm as it's the preferred method in the CEPH docs or Puppet with the puppet-ceph module, as it's part of openstack?

Thanks in advance

-- Francesco Di Nucci System Administrator Compute & Networking Service, INFN Naples

Email: francesco.dinucci@na.infn.it

--

Regards,

Maksim Malchuk

-- Regards, Maksim Malchuk

Hi, I would say this is opinion based and depends on your experience and infrastructure. Even if you decided to use cephadm as a Ceph deployment tool, you still need to have some installation and configuration management in place, at least if you have more than a few hosts. Because with the Ceph orchestrator (cephadm) you can only add hosts that have been configured to your needs, like ssh keys, podman/docker, chrony etc. If you use puppet for that, it might be the right choice for you. We are using a combination of cobbler and Salt (Uyuni project) to perform automatic OS installation via PXE boot and configuration via Salt. Once the systems are ready to join the Ceph cluster, we just add them via orchestrator (ceph orch host add ...) and then the rest is managed by cephadm. So in our case, Ceph is decoupled from OpenStack management, although the OpenStack hosts are also installed and configured via Salt. I'd say choose the method you're most familiar with. Regards, Eugen Zitat von Francesco Di Nucci <francesco.dinucci@na.infn.it>:

Thank you,

I'd read it, but as there also are other methods such as ceph-ansible and puppet-ceph I am trying to get a feedback from other operators about their experiences, as in this case I'm particularly interested in integration of CEPH with OpenStack

Best regards

-- Francesco Di Nucci System Administrator Compute & Networking Service, INFN Naples

Email:francesco.dinucci@na.infn.it

On 19/05/25 16:29, Maksim Malchuk wrote:

...

Hi Francesco,

The CEPH community recommends using CEPHADM as the primary tool for deploying CEPH: https://docs.ceph.com/en/latest/install/#recommended-methods

On Mon, May 19, 2025 at 4:19 PM Francesco Di Nucci <francesco.dinucci@na.infn.it> wrote:

Hi all,

we're planning (finally) to setup a CEPH cluster to be used as OpenStack backend.

The cloud is actually set up with Foreman+Puppet, to setup CEPH what would you advice?

cephadm as it's the preferred method in the CEPH docs or Puppet with the puppet-ceph module, as it's part of openstack?

Thanks in advance

-- Francesco Di Nucci System Administrator Compute & Networking Service, INFN Naples

Email: francesco.dinucci@na.infn.it

-- Regards, Maksim Malchuk

Hi Francesco, We used to manage our Ceph cluster with a Puppet -like tool and decided to switch to cephadm to do the cluster manager in Octopus, using our system management tool only to provision the OS and Podman and configure the SSH key required by cephadm. It has been a huge improving terms of management efficiency, in particular the time required by any Ceph upgrade operation, including OS upgrades/réinstallation where cephadm redeploys everything required automatically. We will never move back to previous way of managing our cluster! Michel Sent from my mobile Le 20 mai 2025 09:33:07 Francesco Di Nucci <francesco.dinucci@na.infn.it> a écrit :

Thank you,

this might be a solution too (using other tools to setup the OS and then switch to cephadm)

It's not only about familiarity, I was also thinking about feasibility in the long run, with upgrades, new nodes etc, so nice to know that CEPH and OpenStack managements can be decoupled

Best regards

-- Francesco Di Nucci System Administrator Compute & Networking Service, INFN Naples

Email: francesco.dinucci@na.infn.it

On 20/05/25 09:12, Eugen Block wrote:

...

Hi,

I would say this is opinion based and depends on your experience and infrastructure. Even if you decided to use cephadm as a Ceph deployment tool, you still need to have some installation and configuration management in place, at least if you have more than a few hosts. Because with the Ceph orchestrator (cephadm) you can only add hosts that have been configured to your needs, like ssh keys, podman/docker, chrony etc. If you use puppet for that, it might be the right choice for you. We are using a combination of cobbler and Salt (Uyuni project) to perform automatic OS installation via PXE boot and configuration via Salt. Once the systems are ready to join the Ceph cluster, we just add them via orchestrator (ceph orch host add ...) and then the rest is managed by cephadm. So in our case, Ceph is decoupled from OpenStack management, although the OpenStack hosts are also installed and configured via Salt.

I'd say choose the method you're most familiar with.

Regards, Eugen

Zitat von Francesco Di Nucci <francesco.dinucci@na.infn.it>:

...

Thank you,

I'd read it, but as there also are other methods such as ceph-ansible and puppet-ceph I am trying to get a feedback from other operators about their experiences, as in this case I'm particularly interested in integration of CEPH with OpenStack

Best regards

-- Francesco Di Nucci System Administrator Compute & Networking Service, INFN Naples

Email:francesco.dinucci@na.infn.it

On 19/05/25 16:29, Maksim Malchuk wrote:

...

Hi Francesco,

The CEPH community recommends using CEPHADM as the primary tool for deploying CEPH: https://docs.ceph.com/en/latest/install/#recommended-methods

On Mon, May 19, 2025 at 4:19 PM Francesco Di Nucci <francesco.dinucci@na.infn.it> wrote:

Hi all,

we're planning (finally) to setup a CEPH cluster to be used as OpenStack backend.

The cloud is actually set up with Foreman+Puppet, to setup CEPH what would you advice?

cephadm as it's the preferred method in the CEPH docs or Puppet with the puppet-ceph module, as it's part of openstack?

Thanks in advance

-- Francesco Di Nucci System Administrator Compute & Networking Service, INFN Naples

Email: francesco.dinucci@na.infn.it

-- Regards, Maksim Malchuk

Francesco, Cephadm implements "rolling upgrade". It will stop the services one by one so that the cluster always remains healthy, with no impact on the services (except for rgw or MDS that may be single instance but it is just the time of a restart). And it will pause if at any point Moving forward may put the cluster at risk. Michel Sent from my mobile Le 20 mai 2025 14:04:07 Francesco Di Nucci <francesco.dinucci@na.infn.it> a écrit :

Thanks a lot, this discussion is being very helpful! -- Francesco Di Nucci System Administrator Compute & Networking Service, INFN Naples Email: francesco.dinucci@na.infn.it On 20/05/25 10:57, Michel Jouvin wrote:

...

Hi Francesco,

We used to manage our Ceph cluster with a Puppet -like tool and decided to switch to cephadm to do the cluster manager in Octopus, using our system management tool only to provision the OS and Podman and configure the SSH key required by cephadm. It has been a huge improving terms of management efficiency, in particular the time required by any Ceph upgrade operation, including OS upgrades/réinstallation where cephadm redeploys everything required automatically.

We will never move back to previous way of managing our cluster!

Michel Sent from my mobile

Le 20 mai 2025 09:33:07 Francesco Di Nucci <francesco.dinucci@na.infn.it> a écrit :

...

Thank you,

this might be a solution too (using other tools to setup the OS and then switch to cephadm)

It's not only about familiarity, I was also thinking about feasibility in the long run, with upgrades, new nodes etc, so nice to know that CEPH and OpenStack managements can be decoupled

Best regards

-- Francesco Di Nucci System Administrator Compute & Networking Service, INFN Naples

Email: francesco.dinucci@na.infn.it

On 20/05/25 09:12, Eugen Block wrote:

...

Hi,

I would say this is opinion based and depends on your experience and infrastructure. Even if you decided to use cephadm as a Ceph deployment tool, you still need to have some installation and configuration management in place, at least if you have more than a few hosts. Because with the Ceph orchestrator (cephadm) you can only add hosts that have been configured to your needs, like ssh keys, podman/docker, chrony etc. If you use puppet for that, it might be the right choice for you. We are using a combination of cobbler and Salt (Uyuni project) to perform automatic OS installation via PXE boot and configuration via Salt. Once the systems are ready to join the Ceph cluster, we just add them via orchestrator (ceph orch host add ...) and then the rest is managed by cephadm. So in our case, Ceph is decoupled from OpenStack management, although the OpenStack hosts are also installed and configured via Salt.

I'd say choose the method you're most familiar with.

Regards, Eugen

Zitat von Francesco Di Nucci <francesco.dinucci@na.infn.it>:

...

Thank you,

I'd read it, but as there also are other methods such as ceph-ansible and puppet-ceph I am trying to get a feedback from other operators about their experiences, as in this case I'm particularly interested in integration of CEPH with OpenStack

Best regards

-- Francesco Di Nucci System Administrator Compute & Networking Service, INFN Naples

Email:francesco.dinucci@na.infn.it

On 19/05/25 16:29, Maksim Malchuk wrote:

...

Hi Francesco,

The CEPH community recommends using CEPHADM as the primary tool for deploying CEPH: https://docs.ceph.com/en/latest/install/#recommended-methods

On Mon, May 19, 2025 at 4:19 PM Francesco Di Nucci <francesco.dinucci@na.infn.it> wrote:

Hi all,

we're planning (finally) to setup a CEPH cluster to be used as OpenStack backend.

The cloud is actually set up with Foreman+Puppet, to setup CEPH what would you advice?

cephadm as it's the preferred method in the CEPH docs or Puppet with the puppet-ceph module, as it's part of openstack?

Thanks in advance

-- Francesco Di Nucci System Administrator Compute & Networking Service, INFN Naples

Email: francesco.dinucci@na.infn.it

-- Regards, Maksim Malchuk

Hey! Deploying and operating a ceph cluster is a complex endeavour. And much of what can be done (in a holistic way from an operator perspective) depends on the tooling. 1. Cephadm: cephadm IS the go-to upstream tooling for ceph orchestration and has both pros and cons. It deploys Ceph in a containerised fashion (docker/podman) and manages the containers on operator's behalf. It is definitely worth checking out. It is also worth mentioning that cephadm is a tool and is abstracted from the ceph cluster itself (i.e. containers constituting the cluster) which allows you to choose the ceph container image at deploy time (or via a systematic upgrade laterally). The choice basically being between upstream images built IIRC using CentOS (!verify) or downstream images like Ceph Rock built using ceph packages from Ubuntu LTS. 2. MicroCeph: MicroCeph is a machine orchestrator for Ceph which spawns isolated Ceph services on the host (and not containers). Again like any operator it has its pros and cons. The idea behind MicroCeph is to make Ceph orchestration/operation easy and over the past cycles it has grown more featureful. It uses trust tokens generated by an existing cluster member for scaling the cluster horizontally and the cluster itself is isolated from the underlying host (no SSHfoo as well). Since it is the same Ceph underneath, native integrations (with openstack services like keystone, cinder, glance, or K8s via ceph CSI) work as is. It is worth checking out if it suits your use cases. 3. Rook Rook is an interesting way of operating a ceph cluster more suited for k8s native clouds. Interesting read however, for you it is less relevant. I would love to hear more from your ceph adventures, please keep us posted! Utkarsh On Tue, May 20, 2025 at 1:02 PM Francesco Di Nucci < francesco.dinucci@na.infn.it> wrote:

Thank you,

this might be a solution too (using other tools to setup the OS and then switch to cephadm)

It's not only about familiarity, I was also thinking about feasibility in the long run, with upgrades, new nodes etc, so nice to know that CEPH and OpenStack managements can be decoupled

Best regards

-- Francesco Di Nucci System Administrator Compute & Networking Service, INFN Naples

Email: francesco.dinucci@na.infn.it

On 20/05/25 09:12, Eugen Block wrote:

...

Hi,

I would say this is opinion based and depends on your experience and infrastructure. Even if you decided to use cephadm as a Ceph deployment tool, you still need to have some installation and configuration management in place, at least if you have more than a few hosts. Because with the Ceph orchestrator (cephadm) you can only add hosts that have been configured to your needs, like ssh keys, podman/docker, chrony etc. If you use puppet for that, it might be the right choice for you. We are using a combination of cobbler and Salt (Uyuni project) to perform automatic OS installation via PXE boot and configuration via Salt. Once the systems are ready to join the Ceph cluster, we just add them via orchestrator (ceph orch host add ...) and then the rest is managed by cephadm. So in our case, Ceph is decoupled from OpenStack management, although the OpenStack hosts are also installed and configured via Salt.

I'd say choose the method you're most familiar with.

Regards, Eugen

Zitat von Francesco Di Nucci <francesco.dinucci@na.infn.it>:

...

Thank you,

I'd read it, but as there also are other methods such as ceph-ansible and puppet-ceph I am trying to get a feedback from other operators about their experiences, as in this case I'm particularly interested in integration of CEPH with OpenStack

Best regards

-- Francesco Di Nucci System Administrator Compute & Networking Service, INFN Naples

Email:francesco.dinucci@na.infn.it

On 19/05/25 16:29, Maksim Malchuk wrote:

...

Hi Francesco,

The CEPH community recommends using CEPHADM as the primary tool for deploying CEPH: https://docs.ceph.com/en/latest/install/#recommended-methods

On Mon, May 19, 2025 at 4:19 PM Francesco Di Nucci <francesco.dinucci@na.infn.it> wrote:

Hi all,

we're planning (finally) to setup a CEPH cluster to be used as OpenStack backend.

The cloud is actually set up with Foreman+Puppet, to setup CEPH what would you advice?

cephadm as it's the preferred method in the CEPH docs or Puppet with the puppet-ceph module, as it's part of openstack?

Thanks in advance

-- Francesco Di Nucci System Administrator Compute & Networking Service, INFN Naples

Email: francesco.dinucci@na.infn.it

-- Regards, Maksim Malchuk

Thank you very much! cephadm for sure seems to have a lot of advantages, especially regarding the upgrades - by the way, does it support "online" upgrades or is a cluster downtime needed? Best regards -- Francesco Di Nucci System Administrator Compute & Networking Service, INFN Naples Email:francesco.dinucci@na.infn.it On 20/05/25 12:58, Michel Jouvin wrote:

Hi,

Management tools are always subjet to personal preferences but honnestly, I cannot really think of cons for cephadm. It is really a great tool, progressing a lot versions after versions and I'd rate a major advantage the fact it is a management layer on top (or aside as you prefer) the cluster: it means you can even repair a somewhat broken cluster with cephadm (as long as you have a mgr running as it is a mgr module).

If you have a cluster that you manage with something else, you can discuss whether you want to move to cephadm and when... but it you are starting a new cluster, I think there is no real discussion! Just, as mentioned by others, you still need a server provisionning tool to deploy the OS, Podman or Docker and the cephadm SSH keys on the server that are/will be part of the cluster.

Michel

Le 20/05/2025 à 11:25, Utkarsh Bhatt a écrit :

...

Hey! Deploying and operating a ceph cluster is a complex endeavour. And much of what can be done (in a holistic way from an operator perspective) depends on the tooling.

1. Cephadm:

cephadm IS the go-to upstream tooling for ceph orchestration and has both pros and cons. It deploys Ceph in a containerised fashion (docker/podman) and manages the containers on operator's behalf. It is definitely worth checking out. It is also worth mentioning that cephadm is a tool and is abstracted from the ceph cluster itself (i.e. containers constituting the cluster) which allows you to choose the ceph container image at deploy time (or via a systematic upgrade laterally). The choice basically being between upstream images built IIRC using CentOS (!verify) or downstream images like Ceph Rock built using ceph packages from Ubuntu LTS.

2. MicroCeph:

MicroCeph is a machine orchestrator for Ceph which spawns isolated Ceph services on the host (and not containers). Again like any operator it has its pros and cons. The idea behind MicroCeph is to make Ceph orchestration/operation easy and over the past cycles it has grown more featureful. It uses trust tokens generated by an existing cluster member for scaling the cluster horizontally and the cluster itself is isolated from the underlying host (no SSHfoo as well). Since it is the same Ceph underneath, native integrations (with openstack services like keystone, cinder, glance, or K8s via ceph CSI) work as is. It is worth checking out if it suits your use cases.

3. Rook

Rook is an interesting way of operating a ceph cluster more suited for k8s native clouds. Interesting read however, for you it is less relevant.

I would love to hear more from your ceph adventures, please keep us posted!

Utkarsh

On Tue, May 20, 2025 at 1:02 PM Francesco Di Nucci <francesco.dinucci@na.infn.it> wrote:

Thank you,

this might be a solution too (using other tools to setup the OS and then switch to cephadm)

It's not only about familiarity, I was also thinking about feasibility in the long run, with upgrades, new nodes etc, so nice to know that CEPH and OpenStack managements can be decoupled

Best regards

-- Francesco Di Nucci System Administrator Compute & Networking Service, INFN Naples

Email: francesco.dinucci@na.infn.it

On 20/05/25 09:12, Eugen Block wrote: > Hi, > > I would say this is opinion based and depends on your experience and > infrastructure. > Even if you decided to use cephadm as a Ceph deployment tool, you > still need to have some installation and configuration management in > place, at least if you have more than a few hosts. Because with the > Ceph orchestrator (cephadm) you can only add hosts that have been > configured to your needs, like ssh keys, podman/docker, chrony etc. If > you use puppet for that, it might be the right choice for you. > We are using a combination of cobbler and Salt (Uyuni project) to > perform automatic OS installation via PXE boot and configuration via > Salt. Once the systems are ready to join the Ceph cluster, we just add > them via orchestrator (ceph orch host add ...) and then the rest is > managed by cephadm. So in our case, Ceph is decoupled from OpenStack > management, although the OpenStack hosts are also installed and > configured via Salt. > > I'd say choose the method you're most familiar with. > > Regards, > Eugen > > Zitat von Francesco Di Nucci <francesco.dinucci@na.infn.it>: > >> Thank you, >> >> I'd read it, but as there also are other methods such as ceph-ansible >> and puppet-ceph I am trying to get a feedback from other operators >> about their experiences, as in this case I'm particularly interested >> in integration of CEPH with OpenStack >> >> Best regards >> >> -- >> Francesco Di Nucci >> System Administrator >> Compute & Networking Service, INFN Naples >> >> Email:francesco.dinucci@na.infn.it <mailto:Email%3Afrancesco.dinucci@na.infn.it> >> >> On 19/05/25 16:29, Maksim Malchuk wrote: >>> Hi Francesco, >>> >>> The CEPH community recommends using CEPHADM as the primary tool for >>> deploying CEPH: >>> https://docs.ceph.com/en/latest/install/#recommended-methods >>> >>> >>> On Mon, May 19, 2025 at 4:19 PM Francesco Di Nucci >>> <francesco.dinucci@na.infn.it> wrote: >>> >>>    Hi all, >>> >>>    we're planning (finally) to setup a CEPH cluster to be used as >>>    OpenStack >>>    backend. >>> >>>    The cloud is actually set up with Foreman+Puppet, to setup CEPH what >>>    would you advice? >>> >>>    cephadm as it's the preferred method in the CEPH docs or Puppet >>>    with the >>>    puppet-ceph module, as it's part of openstack? >>> >>>    Thanks in advance >>> >>>    --     Francesco Di Nucci >>>    System Administrator >>>    Compute & Networking Service, INFN Naples >>> >>>    Email: francesco.dinucci@na.infn.it >>> >>> >>> >>> >>> -- >>> Regards, >>> Maksim Malchuk >>> > > > >