November 2023 - openstack-discuss

[kolla-ansible][16.1.0] Error container timeout
by Satish Patel 02 Nov '23

02 Nov '23

Folks, I am deploying the 2023.1 release of openstack using kolla-ansible (tag 16.1.0) . I noticed a very strange error during deployment. This is very close to this bug report: https://bugs.launchpad.net/kolla-ansible/+bug/2038914 RUNNING HANDLER [common : Restart fluentd container] ***************************************************************************************************************************** fatal: [os1-bos-ctrl02]: FAILED! => {"Command": "dumb-init --single-child -- kolla_start", "Created": 1698865177, "HostConfig": {"NetworkMode": "host"}, "Id": "ae17d4d276fd278d15c5df03fdee70eafbff66fecbc78ae0bbae64274daa910b", "Image": "registry.example.com/kolla/fluentd:2023.1-ubuntu-jammy", "ImageID": "sha256:41752d28100fd85074d112942f843ce144df8bb023d7bdf664df0ef7be9a50ea", "Labels": {"build-date": "20231029", "kolla_version": "16.1.1", "maintainer": "Kolla Project (https://launchpad.net/kolla)", "name": "fluentd", "org.opencontainers.image.ref.name": "ubuntu", "org.opencontainers.image.version": "22.04"}, "Mounts": [{"Destination": "/etc/localtime", "Mode": "ro", "Propagation": "rprivate", "RW": false, "Source": "/etc/localtime", "Type": "bind"}, {"Destination": "/etc/timezone", "Mode": "ro", "Propagation": "rprivate", "RW": false, "Source": "/etc/timezone", "Type": "bind"}, {"Destination": "/var/lib/fluentd/data", "Driver": "local", "Mode": "rw", "Name": "fluentd_data", "Propagation": "", "RW": true, "Source": "/var/lib/docker/volumes/fluentd_data/_data", "Type": "volume"}, {"Destination": "/var/lib/kolla/config_files", "Mode": "ro", "Propagation": "rprivate", "RW": false, "Source": "/etc/kolla/fluentd", "Type": "bind"}, {"Destination": "/var/log/kolla", "Driver": "local", "Mode": "rw", "Name": "kolla_logs", "Propagation": "", "RW": true, "Source": "/var/lib/docker/volumes/kolla_logs/_data", "Type": "volume"}], "Names": ["/fluentd"], "NetworkSettings": {"Networks": {"host": {"Aliases": null, "DriverOpts": null, "EndpointID": "", "Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "IPAMConfig": null, "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "Links": null, "MacAddress": "", "NetworkID": ""}}}, "Ports": [], "State": "created", "Status": "Created", "changed": true, "msg": "Container timed out"} fatal: [os1-bos-ctrl01]: FAILED! => {"Command": "dumb-init --single-child -- kolla_start", "Created": 1698865177, "HostConfig": {"NetworkMode": "host"}, "Id": "1ca559230483b490830dc2e38791fc8c567702d3280713a6f0c7bc45f2c8a038", "Image": "registry.example.com/kolla/fluentd:2023.1-ubuntu-jammy", "ImageID": "sha256:41752d28100fd85074d112942f843ce144df8bb023d7bdf664df0ef7be9a50ea", "Labels": {"build-date": "20231029", "kolla_version": "16.1.1", "maintainer": "Kolla Project (https://launchpad.net/kolla)", "name": "fluentd", "org.opencontainers.image.ref.name": "ubuntu", "org.opencontainers.image.version": "22.04"}, "Mounts": [{"Destination": "/etc/timezone", "Mode": "ro", "Propagation": "rprivate", "RW": false, "Source": "/etc/timezone", "Type": "bind"}, {"Destination": "/var/lib/fluentd/data", "Driver": "local", "Mode": "rw", "Name": "fluentd_data", "Propagation": "", "RW": true, "Source": "/var/lib/docker/volumes/fluentd_data/_data", "Type": "volume"}, {"Destination": "/var/lib/kolla/config_files", "Mode": "ro", "Propagation": "rprivate", "RW": false, "Source": "/etc/kolla/fluentd", "Type": "bind"}, {"Destination": "/var/log/kolla", "Driver": "local", "Mode": "rw", "Name": "kolla_logs", "Propagation": "", "RW": true, "Source": "/var/lib/docker/volumes/kolla_logs/_data", "Type": "volume"}, {"Destination": "/etc/localtime", "Mode": "ro", "Propagation": "rprivate", "RW": false, "Source": "/etc/localtime", "Type": "bind"}], "Names": ["/fluentd"], "NetworkSettings": {"Networks": {"host": {"Aliases": null, "DriverOpts": null, "EndpointID": "", "Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "IPAMConfig": null, "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "Links": null, "MacAddress": "", "NetworkID": ""}}}, "Ports": [], "State": "created", "Status": "Created", "changed": true, "msg": "Container timed out"} fatal: [os1-bos-ctrl03]: FAILED! => {"Command": "dumb-init --single-child -- kolla_start", "Created": 1698865177, "HostConfig": {"NetworkMode": "host"}, "Id": "8f3599681f56695b0e809acb5645d79deb3fbb476dc7bda5d75f838f67ed085d", "Image": "registry.example.com/kolla/fluentd:2023.1-ubuntu-jammy", "ImageID": "sha256:41752d28100fd85074d112942f843ce144df8bb023d7bdf664df0ef7be9a50ea", "Labels": {"build-date": "20231029", "kolla_version": "16.1.1", "maintainer": "Kolla Project (https://launchpad.net/kolla)", "name": "fluentd", "org.opencontainers.image.ref.name": "ubuntu", "org.opencontainers.image.version": "22.04"}, "Mounts": [{"Destination": "/etc/localtime", "Mode": "ro", "Propagation": "rprivate", "RW": false, "Source": "/etc/localtime", "Type": "bind"}, {"Destination": "/etc/timezone", "Mode": "ro", "Propagation": "rprivate", "RW": false, "Source": "/etc/timezone", "Type": "bind"}, {"Destination": "/var/lib/fluentd/data", "Driver": "local", "Mode": "rw", "Name": "fluentd_data", "Propagation": "", "RW": true, "Source": "/var/lib/docker/volumes/fluentd_data/_data", "Type": "volume"}, {"Destination": "/var/lib/kolla/config_files", "Mode": "ro", "Propagation": "rprivate", "RW": false, "Source": "/etc/kolla/fluentd", "Type": "bind"}, {"Destination": "/var/log/kolla", "Driver": "local", "Mode": "rw", "Name": "kolla_logs", "Propagation": "", "RW": true, "Source": "/var/lib/docker/volumes/kolla_logs/_data", "Type": "volume"}], "Names": ["/fluentd"], "NetworkSettings": {"Networks": {"host": {"Aliases": null, "DriverOpts": null, "EndpointID": "", "Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "IPAMConfig": null, "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "Links": null, "MacAddress": "", "NetworkID": ""}}}, "Ports": [], "State": "created", "Status": "Created", "changed": true, "msg": "Container timed out"}

1 0

[ironic] Announcement: Future of Metalsmith
by Jay Faulkner 01 Nov '23

01 Nov '23

Hi all, Just announcing that at the vPTG, the Ironic core team came to a consensus about the future of the metalsmith project. It was originally created to improve the user experience around provisioning Ironic nodes when using Ironic's API directly. At this, it was mostly successful and has completed this task. It's our intention to work on the Ironic API and python-ironicclient to provide a similarly simple deployment user experience, even for users without nova, integrating lessons learned from the metalsmith project. To this end; we are letting the overall community know we no longer expect to add additional functionality to metalsmith. We will ensure it remains maintained and well-tested until it's functionality has been duplicated, but all new work in this area will be directed at the primary Ironic API and Ironic client projects. To this end, and as discussed at the PTG, I've proposed a change to the Metalsmith readme, notifying users of this state. https://review.opendev.org/c/openstack/metalsmith/+/899761 This change, or this mailing list post, is a great place to comment if you have any comments on this decision. Thanks, Jay Faulkner Ironic PTL

3 3

Cinder Bug Report 2023-11-01
by Jon Bernard 01 Nov '23

01 Nov '23

Hello Argonauts, Cinder Bug Meeting Etherpad <https://etherpad.opendev.org/p/cinder-bug-squad-meeting> Undecided - hitachi and OEM : output REST API token <https://bugs.launchpad.net/cinder/+bug/2040966> - Status: New - hitachi AIX as host OS type does not work <https://bugs.launchpad.net/cinder/+bug/2041784> - Status: In Progress - The 'signature_verified' parameter should not be added to image when upload volume to image <https://bugs.launchpad.net/cinder/+bug/2041793> - Status: In Progress - Missing encryption param handling in Quobyte driver <https://bugs.launchpad.net/cinder/+bug/2042102> - Status: New Thanks, -- Jon

1 0

[manila] 2024.1 Caracal PTG summary
by Carlos Silva 01 Nov '23

01 Nov '23

Hello. Here is a summary of all topics we have discussed during last week's PTG and the AIs that were a result of the discussion. Thank you to all contributors that participated. If you missed one or more sessions, we got you covered. The Manila PTG, as well as the operator hour were recorded and the videos are available in the OpenStack Manila YouTube Channel [0]. Deferred deletion ------------------------- We discussed a bug that exists and the need for a deferred deletion approach. We agreed that the deferred deletion would be ideal in case the driver needs to do multiple things in the deletion and we don't want the operators to wait. #Agreed: The service should release user quotas as soon as the driver says it is processing the deletion; and handle errors internally AI: kpdev and NetApp/SAP will look into the deferred deletion implementation and provide a spec All things CephFS -------------------------- We discussed deprecating support for "standalone" NFS-Ganesha with the CephFS/NFS driver. We've supported using a clustered Ceph NFS service since the Zed release. We've also added upgrade helpers to migrate from a standalone NFS-Ganesha service to the Ceph NFS service; this code was backported to stable/2023.1 (Antelope) Testing and stabilization: We'll continue testing cephfs/nfs with standalone NFS-Ganesha until we drop support for ganesha's DBUS mechanism from the CephFS/NFS driver. IBM is currently testing the scale, performance and stability of clustered-nfs-ganesha and both, nfs-ganesha and cephadm are being updated accordingly Mulitnode devstack ceph in CI: - devstack testing of cephfs/nfs has been stable, we're trying to run this job on stable/2023.1 - AI: ashrodri will enable ingress service with cephadm Ceph NFS upgrades The code patches for upgrading to a cephadm deployed NFS cluster were merged and backported to stable/2023.1 (Antelope). When upgrading, access rules are reapplied and export locations will indicate the preferred export path for mounting shares. Deprecating use of DBUS API with NFS-Ganesha As the clustered nfs solution is hardened, and an upgrade mechanism is invented, we want to stop supporting the use of "standalone-nfs/ganesha" server with CephFS. AI: add a deprecation warning in the Caracal cycle and communicate it in the mailing list Multiple CephFS filesystems We discussed pros/cons of using a single driver representing multiple file systems as Manila storage pools - there are not a lot of benefits to this, especially considering the ease in deployment/configuration of the manila share services currently For multi-fs environments, the user experience to mount shares with Native CephFS is poor since the export paths no longer expose the filesystem name Operators suggested that we encode the filesystem name in export location metadata so automation tools (e.g.: manila CSI plugin, nova/virtiofs) can consume it programmatically AI: Elias Wimmer will report a bug against the CephFS driver AI: Ceph driver documentation will be updated regarding multi-fs and the behavior of the driver Ceph Mon address is never re-validated [3] Today, the only way to re-export a share is by restarting the manila-share manager service. We discussed an API based trigger to the ensure shares logic that will recalculate the export locations of the shares. We agreed that this implementation should add a micro-state to the share and that the API should deny other management operations on the shares when "ensuring". AI: carloss will propose a specification to implement this change Human readable export locations ----------------------------------------------- Export locations are constructed for machines, and they do not use human friendly names. NetApp proposed user customizable export locations a few releases ago [4] We discussed the proposal, and find it is still relevant - we brainstormed using a share type extra specification for customizing a prefix to the export location names We agreed that updating the export location of a share is bound to be disruptive and will not be part of the implementation. We may consider this enhancement if there is a very strong use case AI: NetApp folks will propose updates to the lite-spec and work on the implementation Share backups --------------------- A generic NFS implementation of share backups was introduced in the previous (Bobcat) cycle We discussed a share-manager driven advantaged share backup driver implementations, as well as the data service backup driver interface Cern found that the backup driver interface called back into the data service, and the driver interface needs changes AI: Luis from Cern will update the Backup driver to decouple the data manager interface on-demand in preparation for their implementation of the restic backup driver Tech debt -------------- During the tech debt session, gouthamr walked us through some multi-release tech debt items, such as the progress on scenario tests and the implementation of mechanisms to prevent race conditions. It is a good opportunity for someone in the community to step up and do some of the work, which can be a good technical challenge. AI: Discuss the specification in weekly IRC meetings and find new owner/s Barbican integration ----------------------------- Backend data encryption has been supported for several releases in Manila, but users don't have the ability to configure their encryption keys SAP is pursuing an integration with Barbican for Manila and has worked on a specification. #Agreed: Manila should have a Barbican/Castellan API layer to interact with the keystore, and share backends shouldn't have code to interact directly with Barbican. AI: kpdev will update the proposed spec [5] also documenting the interactions with Castellan Manila/Cinder cross-project session --------------------------------------------------- We organized a cross-project session with the Cinder team to discuss a couple of things: - An enhancement to the manila volume extend API calls in the generic driver: - as the Cinder APIs evolved and Manila would benefit from it. Rajat can work on the enhancement in case the Manila contributors don't have the bandwidth. - The way DHSS=True works in Manila - we brainstormed hypothetical support in Cinder for service instances/storage virtual machines and tenant driven volume networking - We also talked about the progress both the projects achieved with respect to Secure RBAC, OpenStackClient and OpenStackSDK. Expectations while submitting features -------------------------------------------------------- We are working on documenting the process one should go through while submitting features [6]. This also covers what are the deadlines and what reviewers expect in terms of feature submission. AI: carloss will update the new documentation with a checklist that will illustrate the definition of done for a feature Howcasts -------------- This is something we have been ideating on for a while. The idea is to produce short videos that will help newcomers to the project. We have an idea of what to work on [7] and help from the community would be appreciated. AI: we'll start the work on the howcasts and try to split the load within the community members SQLAlchemy 2.0 ----------------------- stephenfin has been working on the SQLAlchemy 2.0 changes for Manila and we have made good progress so far. One of the problems to test the changes is that we need to wait a whole entire CI run that can take up to 3 hours Agreed: - We'll work with stephenfin to reduce the amount of wait time to get CI results by reducing the tests using the dummy drivers - We'll review the changes and help figure out the issues with migration of share replicas and share instances queries Tech Debt Roundup - 2; multi-release efforts that have attention --------------------------------------------------------------------------- ---------------- Metadata spec We currently have metadata implemented for three resources (shares, share snapshots and share network subnets). We have open patches for access rules and export locations. AI: Update the export locations and access rules metadata changes and work on them during the Caracal cycle. OSC Support: We have parity between OSC and the manila shell client and we added a deprecation warning in the previous cycle and the removal will happen prior to the next SLURP release (E). This means that Manila's shell client will not be shipped as part of the D release. - More negative functional tests would be welcome, as we have implemented only the happy-paths during a hackathon. - AI: consider a OSC test hackathon during the Caracal release hackathon - AI: all documentation will be updated to utilize OSC in examples SDK We have completed a lot of this with the help of university students/outreachy interns. Most of the work that was pressing and features that were needed to integrate with other services were covered. AI: Continue working towards full parity between openstacksdk and manilaclient SDK Agreed: There are no plans to deprecate manilaclient SDK Share Transfers: We have share transfers between projects for DHSS=False, but we are still lacking support for DHSS=True. This is a cool technical challenge to work in Manila, and having someone willing to continue this work would be appreciated. Follow up on Resource locks ------------------------------------------ The two featureful changes for resource locks were merged in the Bobcat cycle (share resource locks and locks for access rules deletion and visibility restrictions) There are two changes yet to be merged, being the OpenStack SDK patch and the manila-tempest-plugin tests for the access rules locks. We brainstormed implementations of other resource locks, such as a visibility lock on sensitive fields of a security service, or deletion locks for snapshots, backups and replicas AI: We'll implement new resource locks opportunistically during the Caracal release Operator Hour --------------------- We have reflected over the results of the latest OpenStack user survey [9] and talked about the progress of the most requested features and also requested feedback from operators. We discussed the use cases for the disabling manila services feature, and chatted about one possible enhancement to this feature which would be to add backends as disabled and enable them later. AI: we should enhance our documentation to talk about start and stopping services, as well as the differences between stopping and disabling them. [0] https://www.youtube.com/@openstackmanila [1] https://github.com/openstack-k8s-operators/manila-operator/ - a replacement for TripleO [2] https://prow.ci.openshift.org/job-history/gs/origin-ci-test/pr-logs/directo… (Native CephFS) [3] https://bugs.launchpad.net/manila/+bug/1996793 [4] https://opendev.org/openstack/manila-specs/src/commit/baf6209da8e5b47228024… [5] https://review.opendev.org/c/openstack/manila-specs/+/898999 [6] https://review.opendev.org/c/openstack/manila/+/898855 [7] https://etherpad.opendev.org/p/manila-howcasts [8] https://etherpad.opendev.org/p/caracal-ptg-manila-operator-hour [9] https://drive.google.com/file/d/1ZM722TOX3ouP_7Cet2JdURFttTuI5ZLe/view?usp=… [10] https://etherpad.opendev.org/p/caracal-ptg-manila If you have questions or would like to follow-up on any of the topics, don't hesitate to reach out. Thank you! carloss

1 0

[manila] Nov 2nd upstream meeting is cancelled
by Carlos Silva 01 Nov '23

01 Nov '23

Hello! As some of our regular contributors will be on PTO tomorrow and we are following up on all-things-ptg, we are cancelling this week's weekly meeting. The next meeting will take place next week (Nov 9th). Thanks and regards, carloss

1 0

Volumes not deleting
by Russell Stather 01 Nov '23

01 Nov '23

Hi We have a juju openstack installation and we are finding that when we delete servers the associated volume doesnt get deleted. The volume stays as showing attached to a non existent server instance. We are having to marked them as not attached in the database and then the volume delete works. Any ideas what could be wrong here? Thanks Russell

2 1

Openstack volume deletion
by Vijay Thilak S 01 Nov '23

01 Nov '23

Hi, OpenStack Kolla Ansible has been installed on Ubuntu 20 and integrated with Ceph storage. The volume associated with the instance is not being deleted by Cinder when I terminate the instance with this option (Delete Volume on Instance Delete), which is causing me problems. I need to log in to the server and run the cinder commands to manually delete the volume that is not associated with instances. I would appreciate it if you could provide a solution. Thanks, Vijay

4 10

[kolla] Cancelling weekly meeting
by Michał Nasiadka 01 Nov '23

01 Nov '23

Hello Koalas, Weekly meeting today is cancelled due to public holiday in some of the countries including Poland. Best regards, Michal

1 0

[Octavia] weekly meeting 11/1 canceled
by Gregory Thiemonge 01 Nov '23

01 Nov '23

Hi Folks Many contributors are off today (public holiday), so we are canceling this week's meeting, Thanks

1 0

RE: openstack-discuss Digest, Vol 61, Issue 1 Openstack- Instance creation
by Asma Naz Shariq 01 Nov '23

01 Nov '23

Hi Openstack Team, When I tried to create instance in Openstack dashboard, I encountered below error: Error: Failed to perform requested operation on instance "Test-Instance", the instance has an error status: Please try again later [Error: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance f8cb7bb1-64db-4242-a26a-65f3568ff4e9.]. Can anyone guide me what is the root cause for this error? Regards, -----Original Message----- From: openstack-discuss-request(a)lists.openstack.org <openstack-discuss-request(a)lists.openstack.org> Sent: Wednesday, November 1, 2023 7:26 AM To: openstack-discuss(a)lists.openstack.org Subject: openstack-discuss Digest, Vol 61, Issue 1 Send openstack-discuss mailing list submissions to openstack-discuss(a)lists.openstack.org To subscribe or unsubscribe via email, send a message with subject or body 'help' to openstack-discuss-request(a)lists.openstack.org You can reach the person managing the list at openstack-discuss-owner(a)lists.openstack.org When replying, please edit your Subject line so it is more specific than "Re: Contents of openstack-discuss digest..." Today's Topics: 1. Re: [ops] Seeking Advice on Service Deployment and Layout for an 8-Node Cluster (Forrest Fuqua) ---------------------------------------------------------------------- Message: 1 Date: Wed, 1 Nov 2023 02:26:02 +0000 From: Forrest Fuqua <fffics(a)rit.edu> Subject: Re: [ops] Seeking Advice on Service Deployment and Layout for an 8-Node Cluster To: Karl Kloppenborg <kkloppenborg(a)resetdata.com.au>, "openstack-discuss(a)lists.openstack.org" <openstack-discuss(a)lists.openstack.org> Message-ID: <CO6PR16MB420923931283EAF219118489D0A7A(a)CO6PR16MB4209.namp rd16.prod.outlook.com> Content-Type: multipart/alternative; boundary="_000_CO6PR16MB420923 931283EAF219118489D0A7ACO6PR16MB4209namp_" I am currently doing COW images in Ceph right now, that seems to be the same thing as volume backed images, I have about 40 images I use ( I even published the set here:http://mirrors.rit.edu/oszoo/ ) Ill take a look at Openstack Helm, I've never deployed Kerb before. ________________________________ From: Karl Kloppenborg <kkloppenborg(a)resetdata.com.au> Sent: Tuesday, October 31, 2023 8:03 PM To: Forrest Fuqua <fffics(a)rit.edu>; openstack-discuss(a)lists.openstack.org <openstack-discuss(a)lists.openstack.org> Subject: Re: [ops] Seeking Advice on Service Deployment and Layout for an 8-Node Cluster Hi Forrest, One of the first recommendations I would suggest is to utilise volume-backed-images. https://docs.openstack.org/cinder/latest/admin/volume-backed-image.html If you have say a set of linux and windows images, you can use this to instead of RBD imaging, ceph will snapshot from the base image the RBD and this can provision in seconds. I would also suggest (and this is completely biased because I’m on the core team) looking at OpenstackHelm – we run this at scale with many thousands of VM’s. Thanks, Karl. From: Forrest Fuqua <fffics(a)rit.edu> Date: Wednesday, 1 November 2023 at 2:15 am To: Karl Kloppenborg <kkloppenborg(a)resetdata.com.au>, openstack-discuss(a)lists.openstack.org <openstack-discuss(a)lists.openstack.org> Subject: Re: [ops] Seeking Advice on Service Deployment and Layout for an 8-Node Cluster In response to Karl's request, Here is last year's major load I ran: http://mirrors.rit.edu/cptc/2023/virtual-machines/1-README.txt<http://mirrors.rit.edu/cptc/2023/virtual-machines/1-README.txt> The TL;DR its a mix of Windows and Linux VMs at about 4GB of Ram, and 4vCPU at about 50 machines per team, and having up to 50 teams at a time, with all the extras support VMs at various workloads, we do about 2500 VMs just for the yearly compitations All of these VMs will be under high load since they will be under a consistent cyberattack by students. The biggest issues I've had is Ceph starts to slow down on RBD creation of a COW image and Nova will timeout the creation, If it gets too back, entries like volumes, and VMs can get stuck in weird error states of error_deleting or some other weird state (I've had Volumes stuck attached to VMs that don't exist!) At peak, Netdata was showing about 500kIOP/s across the whole cluster before it really started to go north and went into a death spiral, A single windows VM was a 4 hour boot time. Ceph started showing slow writes with up to 40s commit latency When I was running a multi-node management, the MariaDB and RabbitMQ would get super mad and start dropping messages and database issues, having a single dedicated compute node has worked well so far. but I am thinking of moving away from Kolla and going to Ansible during my summer rebuild. ________________________________ From: Karl Kloppenborg <kkloppenborg(a)resetdata.com.au> Sent: Monday, October 30, 2023 6:44 PM To: Forrest Fuqua <fffics(a)rit.edu>; openstack-discuss(a)lists.openstack.org <openstack-discuss(a)lists.openstack.org> Subject: Re: [ops] Seeking Advice on Service Deployment and Layout for an 8-Node Cluster Hi Forrest, Based on the configuration specified, I will assume that 1.8TB is assignable to VM resources, less the management node (though honestly why not just share over the cluster, doesn’t seem like it’s a production workload?) However, (1.8TB*7)*1000 = 12600GB usable ram, with no oversubscription. (roughly 180-200GB each system for host and ceph services) Therefore 12600/3100 = ~4GB ram per VM at peak load, no oversubscription. Based on the CPU arch: 128*2 threads = 256 vCPU cores (which in this model I would count as vCPU cores, there’s a LOT of contention about what defines a vCPU, there’s many camps and religious arguments on this)… However, we want some space for the host so as a general thought, I will put 250 vCPU available in this calculation. 250*7 = 1750 vCPU cores avail. There are 3100-1750 = 1,350 deficit vCPU cores. As such to make up a 1vCPU core to VM at peak workload an oversubscription would be 1.8 1750*1.8 = 3150 vCPU available. Honestly, depending on these workloads and your requirements you can probably bust this subscription ratio up more and again, I would instead share the management workloads over the cluster more and induct the first node as another compute resource. I would like to know more about the workloads, what is the flavour of each VM, Is the workload disk intensive, cpu or memory, or io? In terms of running the workload, I don’t see an obvious issue with your nova config, the allocation ratio is high but that’s not going to probably be an issue here based on the numbers I mentioned. Ensuring host reservation memory is good, which I can see you have around 128GB in reservation and ram_allocation_ratio as 1.0 will ensure no OOM scenarios. Overall, I think your settings are sane and my only comment would be to add that management node into the compute mix if you can but otherwise without knowing more on the workload, seems fine. Have you done any scale testing? Spun up a workload of 3100 VMs? ( easiest way to do this would likely be to heat template…) then run an image with a default start workload of something you want to test with. --Karl. From: Forrest Fuqua <fffics(a)rit.edu> Date: Tuesday, 31 October 2023 at 5:40 am To: openstack-discuss(a)lists.openstack.org <openstack-discuss(a)lists.openstack.org> Subject: [ops] Seeking Advice on Service Deployment and Layout for an 8-Node Cluster Hi there, I'm currently exploring some strategies and seeking advice on the deployment and layout of services on an 8-node cluster. Each node is equipped with dual AMD EYPC 7702s, two 100G Links, and 2TB of RAM. Additionally, I've set up two 13TB NVME drives in a Ceph configuration for each node. At present, the setup involves one management node, while the others are dedicated to running Nova, Neutron, Glance, and Cinder. The workload, while fairly basic, involves handling substantial numbers. It's primarily used for one-off research projects and a few classes, serving around 60 end users. The most demanding scenario occurs during cybersecurity competitions when we need to deploy 3100 VMs at once. When this is spread out over a few days, it's manageable. However, there's a risk of overloading the system, leading to DiskIO issues and Database inconsistencies. Currently, I'm utilizing Kolla for deployment, and I've shared my configurations on this GitHub repository: Link to Configs<https://github.com/RIT-GCI-CyberRange/openstack-configs>. I'd greatly appreciate any insights or advice on optimizing this setup to handle the peak workload more efficiently and prevent potential performance issues. Thanks in advance for your help!

1 0