On 7/22/22 08:49, Mahendra Paipuri wrote:

Hello all,

We are going to deploy Openstack cloud with researchers as primary target users. We will have around 100 GPUs with 12-15 servers with Infiniband interconnect. We still do not know the exact spec of servers nor GPUs but mostly we will have A100s and Intel Xeon processors. What sort of RAM and Storage requirements we need for a cluster of this size? Of course, this depends a lot on use cases and this cloud will be primarily used for HPC and AI. For the storage, we are mainly interested in the block storage requirements for provisioning VMs. We will most probably have a shared parallel file system as scratch and project spaces.

Is there any rule of thumb to get to the RAM and storage requirement numbers based on compute infrastructure we will have? If we can estimate a sort of "lower bound" that would be really helpful for us. If anyone have clusters of this size at your organizations and if you can share the RAM ans storage details of your clouds, that would be very useful for us too.

Thanks a lot and have a great day!!

Regards

Mahendra

Hi, Usually, on compute nodes, we set the Nova reserved RAM to 16 or 32 GB of RAM (depending on the amount of instance), and the rest of is for your VMs. So it really depends on your workload. If you're about to setup 15 servers with 100 A100s, that's 6 GPU board per server, so probably you will want to run one VM per GPU. In such case, just estimate how much RAM you want to assign to each VM, and multiply by the number GPU (6 in your case?). Let's say you want 32 GB of RAM per VM, then you'll be good with 256 GB of RAM per server. Now, for storage, the recommendation is 8 GB per OSD (and usually, you'll set 2 OSD per drive). Of course, more RAM is ok. You may want to read this: https://docs.ceph.com/en/quincy/start/hardware-recommendations/ Does this help? Cheers, Thomas Goirand (zigo)