Mike, We have an Ironic+Nova deployment with ~14k VMs and ~10k bare-metal nodes and use Glance with a Ceph backend for our images (no Swift). We have structured the setup into conductor groups with ~500 nodes each (mostly to limit the resource tracker time), and we distribute new server deliveries across several conductor groups (each managed by a small VM running a conductor): ~50 additional servers max per group. It is rare that we deploy more than a few hundred nodes simultaneously (and when we do it is typically with the same image), and this setup has served us well so far. Hope this helps! Cheers, Arne -- Dr. Arne Wiebalck Head of Compute & Devices CERN IT ________________________________________ From: Julia Kreger <juliaashleykreger@gmail.com> Sent: Friday, 5 January 2024 19:13 To: Michael Sherman Cc: openstack-discuss@lists.openstack.org Subject: Re: [ironic][glance] tradeoffs between ceph and swift glance backends for ironic+kvm? Greetings! Disclaimer: It is super rare for me to go look at glance back ends. From an Ironic point of view, the recommendation is geared so you can use TempURL access and that URL can just be provided through to the deployment agent so it can directly download the image contents from the object storage backend. Be that Swift, or ultimately Ceph operating with the Swift compatibility enabled. If that is not desirable, Ironic can cache the image locally on the conductor and still allow the nodes being deployed to stream the image out from the conductor instead. It really depends on the size of images you're dealing with and how much you're doing in terms of rapid deployments. Consult the [agent] section option image_download_source in ironic.conf[0]. That being said, if the glance image can be downloaded by a more recent glance v2 client, then in all likelihood it is just more a terminology/confusion issue. Hope that helps! -Julia [0]: https://docs.openstack.org/ironic/latest/configuration/sample-config.html On Fri, Jan 5, 2024 at 9:12 AM Michael Sherman <shermanm@uchicago.edu<mailto:shermanm@uchicago.edu>> wrote: Hey! We’re looking into deploying a cloud with a mix of VM and baremetal hosts (nova-compute vs ironic), and discovered some tension between the recommendations for glance for each case. At scale, ironic seems to recommend usage of the swift backend to remove the conductor as a bottleneck when streaming disk images, but for KVM usage, the RBD backend allows for faster launches by cloning the source image. Any thoughts on these or any other tradeoffs, or some recommended configurations? Thanks! -Mike -- Michael Sherman Infrastructure Lead – Chameleon Computer Science, University of Chicago MCS, Argonne National Lab