- openstack-discuss - lists.openstack.org

[kolla-ansible][16.1.0] Could not load namespace_haproxy_for_cell
by Satish Patel 02 Nov '23

02 Nov '23

Folks, I am deploying new cloud and planning to use 2023.1 with kolla-ansible tag 16.1.0 but I hit this error and after google I found this reference bug: https://bugs.launchpad.net/kolla-ansible/+bug/2039573 Very curious about the workaround here. How does it pass CI-CD if it's really a bug?

3 3

[nova][compute] HugePages question
by Satish Patel 02 Nov '23

02 Nov '23

Folks, What are the advantages and disadvantages to enable HugePages on compute nodes? I know that HugePages improve performance of VMs because of TLB lookup etc. But if it does improve in most cases then why people why default not turning on HugePage for deployment? Just trying to find out what could go wrong if I turn on by default HugePage for all deployment?

2 2

[kolla-ansible][16.1.0] Error container timeout
by Satish Patel 02 Nov '23

02 Nov '23

Folks, I am deploying the 2023.1 release of openstack using kolla-ansible (tag 16.1.0) . I noticed a very strange error during deployment. This is very close to this bug report: https://bugs.launchpad.net/kolla-ansible/+bug/2038914 RUNNING HANDLER [common : Restart fluentd container] ***************************************************************************************************************************** fatal: [os1-bos-ctrl02]: FAILED! => {"Command": "dumb-init --single-child -- kolla_start", "Created": 1698865177, "HostConfig": {"NetworkMode": "host"}, "Id": "ae17d4d276fd278d15c5df03fdee70eafbff66fecbc78ae0bbae64274daa910b", "Image": "registry.example.com/kolla/fluentd:2023.1-ubuntu-jammy", "ImageID": "sha256:41752d28100fd85074d112942f843ce144df8bb023d7bdf664df0ef7be9a50ea", "Labels": {"build-date": "20231029", "kolla_version": "16.1.1", "maintainer": "Kolla Project (https://launchpad.net/kolla)", "name": "fluentd", "org.opencontainers.image.ref.name": "ubuntu", "org.opencontainers.image.version": "22.04"}, "Mounts": [{"Destination": "/etc/localtime", "Mode": "ro", "Propagation": "rprivate", "RW": false, "Source": "/etc/localtime", "Type": "bind"}, {"Destination": "/etc/timezone", "Mode": "ro", "Propagation": "rprivate", "RW": false, "Source": "/etc/timezone", "Type": "bind"}, {"Destination": "/var/lib/fluentd/data", "Driver": "local", "Mode": "rw", "Name": "fluentd_data", "Propagation": "", "RW": true, "Source": "/var/lib/docker/volumes/fluentd_data/_data", "Type": "volume"}, {"Destination": "/var/lib/kolla/config_files", "Mode": "ro", "Propagation": "rprivate", "RW": false, "Source": "/etc/kolla/fluentd", "Type": "bind"}, {"Destination": "/var/log/kolla", "Driver": "local", "Mode": "rw", "Name": "kolla_logs", "Propagation": "", "RW": true, "Source": "/var/lib/docker/volumes/kolla_logs/_data", "Type": "volume"}], "Names": ["/fluentd"], "NetworkSettings": {"Networks": {"host": {"Aliases": null, "DriverOpts": null, "EndpointID": "", "Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "IPAMConfig": null, "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "Links": null, "MacAddress": "", "NetworkID": ""}}}, "Ports": [], "State": "created", "Status": "Created", "changed": true, "msg": "Container timed out"} fatal: [os1-bos-ctrl01]: FAILED! => {"Command": "dumb-init --single-child -- kolla_start", "Created": 1698865177, "HostConfig": {"NetworkMode": "host"}, "Id": "1ca559230483b490830dc2e38791fc8c567702d3280713a6f0c7bc45f2c8a038", "Image": "registry.example.com/kolla/fluentd:2023.1-ubuntu-jammy", "ImageID": "sha256:41752d28100fd85074d112942f843ce144df8bb023d7bdf664df0ef7be9a50ea", "Labels": {"build-date": "20231029", "kolla_version": "16.1.1", "maintainer": "Kolla Project (https://launchpad.net/kolla)", "name": "fluentd", "org.opencontainers.image.ref.name": "ubuntu", "org.opencontainers.image.version": "22.04"}, "Mounts": [{"Destination": "/etc/timezone", "Mode": "ro", "Propagation": "rprivate", "RW": false, "Source": "/etc/timezone", "Type": "bind"}, {"Destination": "/var/lib/fluentd/data", "Driver": "local", "Mode": "rw", "Name": "fluentd_data", "Propagation": "", "RW": true, "Source": "/var/lib/docker/volumes/fluentd_data/_data", "Type": "volume"}, {"Destination": "/var/lib/kolla/config_files", "Mode": "ro", "Propagation": "rprivate", "RW": false, "Source": "/etc/kolla/fluentd", "Type": "bind"}, {"Destination": "/var/log/kolla", "Driver": "local", "Mode": "rw", "Name": "kolla_logs", "Propagation": "", "RW": true, "Source": "/var/lib/docker/volumes/kolla_logs/_data", "Type": "volume"}, {"Destination": "/etc/localtime", "Mode": "ro", "Propagation": "rprivate", "RW": false, "Source": "/etc/localtime", "Type": "bind"}], "Names": ["/fluentd"], "NetworkSettings": {"Networks": {"host": {"Aliases": null, "DriverOpts": null, "EndpointID": "", "Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "IPAMConfig": null, "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "Links": null, "MacAddress": "", "NetworkID": ""}}}, "Ports": [], "State": "created", "Status": "Created", "changed": true, "msg": "Container timed out"} fatal: [os1-bos-ctrl03]: FAILED! => {"Command": "dumb-init --single-child -- kolla_start", "Created": 1698865177, "HostConfig": {"NetworkMode": "host"}, "Id": "8f3599681f56695b0e809acb5645d79deb3fbb476dc7bda5d75f838f67ed085d", "Image": "registry.example.com/kolla/fluentd:2023.1-ubuntu-jammy", "ImageID": "sha256:41752d28100fd85074d112942f843ce144df8bb023d7bdf664df0ef7be9a50ea", "Labels": {"build-date": "20231029", "kolla_version": "16.1.1", "maintainer": "Kolla Project (https://launchpad.net/kolla)", "name": "fluentd", "org.opencontainers.image.ref.name": "ubuntu", "org.opencontainers.image.version": "22.04"}, "Mounts": [{"Destination": "/etc/localtime", "Mode": "ro", "Propagation": "rprivate", "RW": false, "Source": "/etc/localtime", "Type": "bind"}, {"Destination": "/etc/timezone", "Mode": "ro", "Propagation": "rprivate", "RW": false, "Source": "/etc/timezone", "Type": "bind"}, {"Destination": "/var/lib/fluentd/data", "Driver": "local", "Mode": "rw", "Name": "fluentd_data", "Propagation": "", "RW": true, "Source": "/var/lib/docker/volumes/fluentd_data/_data", "Type": "volume"}, {"Destination": "/var/lib/kolla/config_files", "Mode": "ro", "Propagation": "rprivate", "RW": false, "Source": "/etc/kolla/fluentd", "Type": "bind"}, {"Destination": "/var/log/kolla", "Driver": "local", "Mode": "rw", "Name": "kolla_logs", "Propagation": "", "RW": true, "Source": "/var/lib/docker/volumes/kolla_logs/_data", "Type": "volume"}], "Names": ["/fluentd"], "NetworkSettings": {"Networks": {"host": {"Aliases": null, "DriverOpts": null, "EndpointID": "", "Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "IPAMConfig": null, "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "Links": null, "MacAddress": "", "NetworkID": ""}}}, "Ports": [], "State": "created", "Status": "Created", "changed": true, "msg": "Container timed out"}

1 0

[ironic] Announcement: Future of Metalsmith
by Jay Faulkner 02 Nov '23

02 Nov '23

Hi all, Just announcing that at the vPTG, the Ironic core team came to a consensus about the future of the metalsmith project. It was originally created to improve the user experience around provisioning Ironic nodes when using Ironic's API directly. At this, it was mostly successful and has completed this task. It's our intention to work on the Ironic API and python-ironicclient to provide a similarly simple deployment user experience, even for users without nova, integrating lessons learned from the metalsmith project. To this end; we are letting the overall community know we no longer expect to add additional functionality to metalsmith. We will ensure it remains maintained and well-tested until it's functionality has been duplicated, but all new work in this area will be directed at the primary Ironic API and Ironic client projects. To this end, and as discussed at the PTG, I've proposed a change to the Metalsmith readme, notifying users of this state. https://review.opendev.org/c/openstack/metalsmith/+/899761 This change, or this mailing list post, is a great place to comment if you have any comments on this decision. Thanks, Jay Faulkner Ironic PTL

3 3

Cinder Bug Report 2023-11-01
by Jon Bernard 01 Nov '23

01 Nov '23

Hello Argonauts, Cinder Bug Meeting Etherpad <https://etherpad.opendev.org/p/cinder-bug-squad-meeting> Undecided - hitachi and OEM : output REST API token <https://bugs.launchpad.net/cinder/+bug/2040966> - Status: New - hitachi AIX as host OS type does not work <https://bugs.launchpad.net/cinder/+bug/2041784> - Status: In Progress - The 'signature_verified' parameter should not be added to image when upload volume to image <https://bugs.launchpad.net/cinder/+bug/2041793> - Status: In Progress - Missing encryption param handling in Quobyte driver <https://bugs.launchpad.net/cinder/+bug/2042102> - Status: New Thanks, -- Jon

1 0

[manila] 2024.1 Caracal PTG summary
by Carlos Silva 01 Nov '23

01 Nov '23

Hello. Here is a summary of all topics we have discussed during last week's PTG and the AIs that were a result of the discussion. Thank you to all contributors that participated. If you missed one or more sessions, we got you covered. The Manila PTG, as well as the operator hour were recorded and the videos are available in the OpenStack Manila YouTube Channel [0]. Deferred deletion ------------------------- We discussed a bug that exists and the need for a deferred deletion approach. We agreed that the deferred deletion would be ideal in case the driver needs to do multiple things in the deletion and we don't want the operators to wait. #Agreed: The service should release user quotas as soon as the driver says it is processing the deletion; and handle errors internally AI: kpdev and NetApp/SAP will look into the deferred deletion implementation and provide a spec All things CephFS -------------------------- We discussed deprecating support for "standalone" NFS-Ganesha with the CephFS/NFS driver. We've supported using a clustered Ceph NFS service since the Zed release. We've also added upgrade helpers to migrate from a standalone NFS-Ganesha service to the Ceph NFS service; this code was backported to stable/2023.1 (Antelope) Testing and stabilization: We'll continue testing cephfs/nfs with standalone NFS-Ganesha until we drop support for ganesha's DBUS mechanism from the CephFS/NFS driver. IBM is currently testing the scale, performance and stability of clustered-nfs-ganesha and both, nfs-ganesha and cephadm are being updated accordingly Mulitnode devstack ceph in CI: - devstack testing of cephfs/nfs has been stable, we're trying to run this job on stable/2023.1 - AI: ashrodri will enable ingress service with cephadm Ceph NFS upgrades The code patches for upgrading to a cephadm deployed NFS cluster were merged and backported to stable/2023.1 (Antelope). When upgrading, access rules are reapplied and export locations will indicate the preferred export path for mounting shares. Deprecating use of DBUS API with NFS-Ganesha As the clustered nfs solution is hardened, and an upgrade mechanism is invented, we want to stop supporting the use of "standalone-nfs/ganesha" server with CephFS. AI: add a deprecation warning in the Caracal cycle and communicate it in the mailing list Multiple CephFS filesystems We discussed pros/cons of using a single driver representing multiple file systems as Manila storage pools - there are not a lot of benefits to this, especially considering the ease in deployment/configuration of the manila share services currently For multi-fs environments, the user experience to mount shares with Native CephFS is poor since the export paths no longer expose the filesystem name Operators suggested that we encode the filesystem name in export location metadata so automation tools (e.g.: manila CSI plugin, nova/virtiofs) can consume it programmatically AI: Elias Wimmer will report a bug against the CephFS driver AI: Ceph driver documentation will be updated regarding multi-fs and the behavior of the driver Ceph Mon address is never re-validated [3] Today, the only way to re-export a share is by restarting the manila-share manager service. We discussed an API based trigger to the ensure shares logic that will recalculate the export locations of the shares. We agreed that this implementation should add a micro-state to the share and that the API should deny other management operations on the shares when "ensuring". AI: carloss will propose a specification to implement this change Human readable export locations ----------------------------------------------- Export locations are constructed for machines, and they do not use human friendly names. NetApp proposed user customizable export locations a few releases ago [4] We discussed the proposal, and find it is still relevant - we brainstormed using a share type extra specification for customizing a prefix to the export location names We agreed that updating the export location of a share is bound to be disruptive and will not be part of the implementation. We may consider this enhancement if there is a very strong use case AI: NetApp folks will propose updates to the lite-spec and work on the implementation Share backups --------------------- A generic NFS implementation of share backups was introduced in the previous (Bobcat) cycle We discussed a share-manager driven advantaged share backup driver implementations, as well as the data service backup driver interface Cern found that the backup driver interface called back into the data service, and the driver interface needs changes AI: Luis from Cern will update the Backup driver to decouple the data manager interface on-demand in preparation for their implementation of the restic backup driver Tech debt -------------- During the tech debt session, gouthamr walked us through some multi-release tech debt items, such as the progress on scenario tests and the implementation of mechanisms to prevent race conditions. It is a good opportunity for someone in the community to step up and do some of the work, which can be a good technical challenge. AI: Discuss the specification in weekly IRC meetings and find new owner/s Barbican integration ----------------------------- Backend data encryption has been supported for several releases in Manila, but users don't have the ability to configure their encryption keys SAP is pursuing an integration with Barbican for Manila and has worked on a specification. #Agreed: Manila should have a Barbican/Castellan API layer to interact with the keystore, and share backends shouldn't have code to interact directly with Barbican. AI: kpdev will update the proposed spec [5] also documenting the interactions with Castellan Manila/Cinder cross-project session --------------------------------------------------- We organized a cross-project session with the Cinder team to discuss a couple of things: - An enhancement to the manila volume extend API calls in the generic driver: - as the Cinder APIs evolved and Manila would benefit from it. Rajat can work on the enhancement in case the Manila contributors don't have the bandwidth. - The way DHSS=True works in Manila - we brainstormed hypothetical support in Cinder for service instances/storage virtual machines and tenant driven volume networking - We also talked about the progress both the projects achieved with respect to Secure RBAC, OpenStackClient and OpenStackSDK. Expectations while submitting features -------------------------------------------------------- We are working on documenting the process one should go through while submitting features [6]. This also covers what are the deadlines and what reviewers expect in terms of feature submission. AI: carloss will update the new documentation with a checklist that will illustrate the definition of done for a feature Howcasts -------------- This is something we have been ideating on for a while. The idea is to produce short videos that will help newcomers to the project. We have an idea of what to work on [7] and help from the community would be appreciated. AI: we'll start the work on the howcasts and try to split the load within the community members SQLAlchemy 2.0 ----------------------- stephenfin has been working on the SQLAlchemy 2.0 changes for Manila and we have made good progress so far. One of the problems to test the changes is that we need to wait a whole entire CI run that can take up to 3 hours Agreed: - We'll work with stephenfin to reduce the amount of wait time to get CI results by reducing the tests using the dummy drivers - We'll review the changes and help figure out the issues with migration of share replicas and share instances queries Tech Debt Roundup - 2; multi-release efforts that have attention --------------------------------------------------------------------------- ---------------- Metadata spec We currently have metadata implemented for three resources (shares, share snapshots and share network subnets). We have open patches for access rules and export locations. AI: Update the export locations and access rules metadata changes and work on them during the Caracal cycle. OSC Support: We have parity between OSC and the manila shell client and we added a deprecation warning in the previous cycle and the removal will happen prior to the next SLURP release (E). This means that Manila's shell client will not be shipped as part of the D release. - More negative functional tests would be welcome, as we have implemented only the happy-paths during a hackathon. - AI: consider a OSC test hackathon during the Caracal release hackathon - AI: all documentation will be updated to utilize OSC in examples SDK We have completed a lot of this with the help of university students/outreachy interns. Most of the work that was pressing and features that were needed to integrate with other services were covered. AI: Continue working towards full parity between openstacksdk and manilaclient SDK Agreed: There are no plans to deprecate manilaclient SDK Share Transfers: We have share transfers between projects for DHSS=False, but we are still lacking support for DHSS=True. This is a cool technical challenge to work in Manila, and having someone willing to continue this work would be appreciated. Follow up on Resource locks ------------------------------------------ The two featureful changes for resource locks were merged in the Bobcat cycle (share resource locks and locks for access rules deletion and visibility restrictions) There are two changes yet to be merged, being the OpenStack SDK patch and the manila-tempest-plugin tests for the access rules locks. We brainstormed implementations of other resource locks, such as a visibility lock on sensitive fields of a security service, or deletion locks for snapshots, backups and replicas AI: We'll implement new resource locks opportunistically during the Caracal release Operator Hour --------------------- We have reflected over the results of the latest OpenStack user survey [9] and talked about the progress of the most requested features and also requested feedback from operators. We discussed the use cases for the disabling manila services feature, and chatted about one possible enhancement to this feature which would be to add backends as disabled and enable them later. AI: we should enhance our documentation to talk about start and stopping services, as well as the differences between stopping and disabling them. [0] https://www.youtube.com/@openstackmanila [1] https://github.com/openstack-k8s-operators/manila-operator/ - a replacement for TripleO [2] https://prow.ci.openshift.org/job-history/gs/origin-ci-test/pr-logs/directo… (Native CephFS) [3] https://bugs.launchpad.net/manila/+bug/1996793 [4] https://opendev.org/openstack/manila-specs/src/commit/baf6209da8e5b47228024… [5] https://review.opendev.org/c/openstack/manila-specs/+/898999 [6] https://review.opendev.org/c/openstack/manila/+/898855 [7] https://etherpad.opendev.org/p/manila-howcasts [8] https://etherpad.opendev.org/p/caracal-ptg-manila-operator-hour [9] https://drive.google.com/file/d/1ZM722TOX3ouP_7Cet2JdURFttTuI5ZLe/view?usp=… [10] https://etherpad.opendev.org/p/caracal-ptg-manila If you have questions or would like to follow-up on any of the topics, don't hesitate to reach out. Thank you! carloss

1 0

[manila] Nov 2nd upstream meeting is cancelled
by Carlos Silva 01 Nov '23

01 Nov '23

Hello! As some of our regular contributors will be on PTO tomorrow and we are following up on all-things-ptg, we are cancelling this week's weekly meeting. The next meeting will take place next week (Nov 9th). Thanks and regards, carloss

1 0

Volumes not deleting
by Russell Stather 01 Nov '23

01 Nov '23

Hi We have a juju openstack installation and we are finding that when we delete servers the associated volume doesnt get deleted. The volume stays as showing attached to a non existent server instance. We are having to marked them as not attached in the database and then the volume delete works. Any ideas what could be wrong here? Thanks Russell

2 1

Openstack volume deletion
by Vijay Thilak S 01 Nov '23

01 Nov '23

Hi, OpenStack Kolla Ansible has been installed on Ubuntu 20 and integrated with Ceph storage. The volume associated with the instance is not being deleted by Cinder when I terminate the instance with this option (Delete Volume on Instance Delete), which is causing me problems. I need to log in to the server and run the cinder commands to manually delete the volume that is not associated with instances. I would appreciate it if you could provide a solution. Thanks, Vijay

4 10

[kolla] Cancelling weekly meeting
by Michał Nasiadka 01 Nov '23

01 Nov '23

Hello Koalas, Weekly meeting today is cancelled due to public holiday in some of the countries including Poland. Best regards, Michal

1 0

[Octavia] weekly meeting 11/1 canceled
by Gregory Thiemonge 01 Nov '23

01 Nov '23

Hi Folks Many contributors are off today (public holiday), so we are canceling this week's meeting, Thanks

1 0

RE: openstack-discuss Digest, Vol 61, Issue 1 Openstack- Instance creation
by Asma Naz Shariq 01 Nov '23

01 Nov '23

Hi Openstack Team, When I tried to create instance in Openstack dashboard, I encountered below error: Error: Failed to perform requested operation on instance "Test-Instance", the instance has an error status: Please try again later [Error: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance f8cb7bb1-64db-4242-a26a-65f3568ff4e9.]. Can anyone guide me what is the root cause for this error? Regards, -----Original Message----- From: openstack-discuss-request(a)lists.openstack.org <openstack-discuss-request(a)lists.openstack.org> Sent: Wednesday, November 1, 2023 7:26 AM To: openstack-discuss(a)lists.openstack.org Subject: openstack-discuss Digest, Vol 61, Issue 1 Send openstack-discuss mailing list submissions to openstack-discuss(a)lists.openstack.org To subscribe or unsubscribe via email, send a message with subject or body 'help' to openstack-discuss-request(a)lists.openstack.org You can reach the person managing the list at openstack-discuss-owner(a)lists.openstack.org When replying, please edit your Subject line so it is more specific than "Re: Contents of openstack-discuss digest..." Today's Topics: 1. Re: [ops] Seeking Advice on Service Deployment and Layout for an 8-Node Cluster (Forrest Fuqua) ---------------------------------------------------------------------- Message: 1 Date: Wed, 1 Nov 2023 02:26:02 +0000 From: Forrest Fuqua <fffics(a)rit.edu> Subject: Re: [ops] Seeking Advice on Service Deployment and Layout for an 8-Node Cluster To: Karl Kloppenborg <kkloppenborg(a)resetdata.com.au>, "openstack-discuss(a)lists.openstack.org" <openstack-discuss(a)lists.openstack.org> Message-ID: <CO6PR16MB420923931283EAF219118489D0A7A(a)CO6PR16MB4209.namp rd16.prod.outlook.com> Content-Type: multipart/alternative; boundary="_000_CO6PR16MB420923 931283EAF219118489D0A7ACO6PR16MB4209namp_" I am currently doing COW images in Ceph right now, that seems to be the same thing as volume backed images, I have about 40 images I use ( I even published the set here:http://mirrors.rit.edu/oszoo/ ) Ill take a look at Openstack Helm, I've never deployed Kerb before. ________________________________ From: Karl Kloppenborg <kkloppenborg(a)resetdata.com.au> Sent: Tuesday, October 31, 2023 8:03 PM To: Forrest Fuqua <fffics(a)rit.edu>; openstack-discuss(a)lists.openstack.org <openstack-discuss(a)lists.openstack.org> Subject: Re: [ops] Seeking Advice on Service Deployment and Layout for an 8-Node Cluster Hi Forrest, One of the first recommendations I would suggest is to utilise volume-backed-images. https://docs.openstack.org/cinder/latest/admin/volume-backed-image.html If you have say a set of linux and windows images, you can use this to instead of RBD imaging, ceph will snapshot from the base image the RBD and this can provision in seconds. I would also suggest (and this is completely biased because I’m on the core team) looking at OpenstackHelm – we run this at scale with many thousands of VM’s. Thanks, Karl. From: Forrest Fuqua <fffics(a)rit.edu> Date: Wednesday, 1 November 2023 at 2:15 am To: Karl Kloppenborg <kkloppenborg(a)resetdata.com.au>, openstack-discuss(a)lists.openstack.org <openstack-discuss(a)lists.openstack.org> Subject: Re: [ops] Seeking Advice on Service Deployment and Layout for an 8-Node Cluster In response to Karl's request, Here is last year's major load I ran: http://mirrors.rit.edu/cptc/2023/virtual-machines/1-README.txt<http://mirrors.rit.edu/cptc/2023/virtual-machines/1-README.txt> The TL;DR its a mix of Windows and Linux VMs at about 4GB of Ram, and 4vCPU at about 50 machines per team, and having up to 50 teams at a time, with all the extras support VMs at various workloads, we do about 2500 VMs just for the yearly compitations All of these VMs will be under high load since they will be under a consistent cyberattack by students. The biggest issues I've had is Ceph starts to slow down on RBD creation of a COW image and Nova will timeout the creation, If it gets too back, entries like volumes, and VMs can get stuck in weird error states of error_deleting or some other weird state (I've had Volumes stuck attached to VMs that don't exist!) At peak, Netdata was showing about 500kIOP/s across the whole cluster before it really started to go north and went into a death spiral, A single windows VM was a 4 hour boot time. Ceph started showing slow writes with up to 40s commit latency When I was running a multi-node management, the MariaDB and RabbitMQ would get super mad and start dropping messages and database issues, having a single dedicated compute node has worked well so far. but I am thinking of moving away from Kolla and going to Ansible during my summer rebuild. ________________________________ From: Karl Kloppenborg <kkloppenborg(a)resetdata.com.au> Sent: Monday, October 30, 2023 6:44 PM To: Forrest Fuqua <fffics(a)rit.edu>; openstack-discuss(a)lists.openstack.org <openstack-discuss(a)lists.openstack.org> Subject: Re: [ops] Seeking Advice on Service Deployment and Layout for an 8-Node Cluster Hi Forrest, Based on the configuration specified, I will assume that 1.8TB is assignable to VM resources, less the management node (though honestly why not just share over the cluster, doesn’t seem like it’s a production workload?) However, (1.8TB*7)*1000 = 12600GB usable ram, with no oversubscription. (roughly 180-200GB each system for host and ceph services) Therefore 12600/3100 = ~4GB ram per VM at peak load, no oversubscription. Based on the CPU arch: 128*2 threads = 256 vCPU cores (which in this model I would count as vCPU cores, there’s a LOT of contention about what defines a vCPU, there’s many camps and religious arguments on this)… However, we want some space for the host so as a general thought, I will put 250 vCPU available in this calculation. 250*7 = 1750 vCPU cores avail. There are 3100-1750 = 1,350 deficit vCPU cores. As such to make up a 1vCPU core to VM at peak workload an oversubscription would be 1.8 1750*1.8 = 3150 vCPU available. Honestly, depending on these workloads and your requirements you can probably bust this subscription ratio up more and again, I would instead share the management workloads over the cluster more and induct the first node as another compute resource. I would like to know more about the workloads, what is the flavour of each VM, Is the workload disk intensive, cpu or memory, or io? In terms of running the workload, I don’t see an obvious issue with your nova config, the allocation ratio is high but that’s not going to probably be an issue here based on the numbers I mentioned. Ensuring host reservation memory is good, which I can see you have around 128GB in reservation and ram_allocation_ratio as 1.0 will ensure no OOM scenarios. Overall, I think your settings are sane and my only comment would be to add that management node into the compute mix if you can but otherwise without knowing more on the workload, seems fine. Have you done any scale testing? Spun up a workload of 3100 VMs? ( easiest way to do this would likely be to heat template…) then run an image with a default start workload of something you want to test with. --Karl. From: Forrest Fuqua <fffics(a)rit.edu> Date: Tuesday, 31 October 2023 at 5:40 am To: openstack-discuss(a)lists.openstack.org <openstack-discuss(a)lists.openstack.org> Subject: [ops] Seeking Advice on Service Deployment and Layout for an 8-Node Cluster Hi there, I'm currently exploring some strategies and seeking advice on the deployment and layout of services on an 8-node cluster. Each node is equipped with dual AMD EYPC 7702s, two 100G Links, and 2TB of RAM. Additionally, I've set up two 13TB NVME drives in a Ceph configuration for each node. At present, the setup involves one management node, while the others are dedicated to running Nova, Neutron, Glance, and Cinder. The workload, while fairly basic, involves handling substantial numbers. It's primarily used for one-off research projects and a few classes, serving around 60 end users. The most demanding scenario occurs during cybersecurity competitions when we need to deploy 3100 VMs at once. When this is spread out over a few days, it's manageable. However, there's a risk of overloading the system, leading to DiskIO issues and Database inconsistencies. Currently, I'm utilizing Kolla for deployment, and I've shared my configurations on this GitHub repository: Link to Configs<https://github.com/RIT-GCI-CyberRange/openstack-configs>. I'd greatly appreciate any insights or advice on optimizing this setup to handle the peak workload more efficiently and prevent potential performance issues. Thanks in advance for your help!

1 0

OpenInfra Live: Meet the Computing Force Network Working Group Nov 2 at 9 AM (CST) / 14:00 UTC
by Kristin Barrientos 01 Nov '23

01 Nov '23

Hi everyone, This week’s OpenInfra Live episode is brought to you by the open source community. Episode: China Mobile and other 10+ organizations are driving the effort of Computing Force Network in open source. In July of 2022, organizations like, China Mobile launched the Computing Force Network (CFN) Working Group within the OpenInfra Foundation. You may be wondering, what is Computing Force Network? What open source software is CFN working group going to deliver and how are CFN and related technology going to help the infrastructure? Join us to find out. Date and time: Nov 2, 2023 at 9:00 AM CST (14:00 UTC) You can watch us live on: YouTube: https://www.youtube.com/watch?v=Slc-kOao3BY LinkedIn: https://www.linkedin.com/events/meetthecomputingforcenetworkwor711866369563… WeChat: The recording will be posted on OpenInfra Foundation WeChat after the live stream Speakers: Qihui Zhao, Jie Niu, Xiaoqiao Geng, Changjin Wang, Jintao Wang, Lei Huang and Ildiko Vancsa Thanks, Kristin Barrientos Marketing Coordinator OpenInfra Foundation

1 0

[ironic] IntelIPMIManagement driver: Anyone using it?
by Jay Faulkner 01 Nov '23

01 Nov '23

Hi all, Ironic currently has an Intel IPMI Management driver ( https://docs.openstack.org/ironic/latest/admin/drivers/intel-ipmi.html ), a mild enhancement over our generic IPMI driver, which adds the ability to select an Intel Speed Select Performance Profile via an Ironic automation step. Ironic cores have no knowledge of this driver being used by any deployers of Ironic. We are evaluating it for deprecation and eventual removal. If you are interested in this driver remaining in the Ironic codebase, please let us know. Please note: this is not the same as our generic IPMI driver, which supports power control, boot device management, and is not being considered anytime soon for deprecation :). If we decide to mark the driver for removal, it'll obviously remain in the codebase for a long deprecation period as is our norm. Thanks, Jay Faulkner Ironic PTL

1 0

[largescale-sig] Next meeting: Nov 1, 15utc
by Kristin Barrientos 01 Nov '23

01 Nov '23

Hi everyone, The Large Scale SIG will be meeting this Wednesday in #openstack-operators on OFTC IRC, at 15UTC, our EU+US-friendly time. You can doublecheck how that UTC time translates locally at: https://www.timeanddate.com/worldclock/fixedtime.html?iso=20231004T15 Feel free to add topics to the agenda: https://etherpad.opendev.org/p/large-scale-sig-meeting Thanks, Kristin Barrientos Marketing Coordinator OpenInfra Foundation

1 0

[blazar] Cancelling the next IRC meeting
by Pierre Riteau 01 Nov '23

01 Nov '23

Hello, I am cancelling the next Blazar IRC meeting due to being out of the office. The next meeting will be on November 16. Best wishes, Pierre Riteau (priteau)

1 0

[horizon] Cancelling Tomorrow Weekly meeting (1st Nov 2023)
by vishal manchanda 01 Nov '23

01 Nov '23

Hello Team, I won't be able to chair tomorrow's Horizon weekly meeting due to some urgent work at that time. If anything urgent, please reach out to me and the Horizon core team on IRC. Thanks & regards, Vishal Manchanda

1 0

[cinder] 2024.1 (Caracal) PTG Summary
by Rajat Dhasmana 31 Oct '23

31 Oct '23

Hello, Please find the summary of 2024.1 (Caracal) PTG in the wiki page[1]. It includes summary and action items for each topic. More details can be found in the PTG etherpad[2]. I would recommend everyone to please look for the action items assigned to you to plan it according to the cycle deadlines. [1] https://wiki.openstack.org/wiki/CinderCaracalPTGSummary [2] https://etherpad.opendev.org/p/caracal-ptg-cinder Thanks Rajat Dhasmana

1 0

Green threads
by Gk Gk 31 Oct '23

31 Oct '23

Hi All, This may be out of place for this question, but I was reading about this bug https://bugs.launchpad.net/oslo.messaging/+bug/2009138, which led to this query. Do green threads in python need GIL ( global interpreter lock) for execution ? Also, can green threads take advantage of multiple processors ? If yes/no, why ?

1 0

[qa] Office hour day and time change
by Martin Kopec 31 Oct '23

31 Oct '23

Hi everyone, as discussed during the PTG [1] we are changing the date and time of our office hours. The office hour will be taking place on *Wednesdays at 17:00 UTC* [2]. The change is effective *from November 8th*. The meeting agenda is available here [3]. I'm cancelling office hours on Tuesday October 31st and November 7th. [1] https://etherpad.opendev.org/p/oct2023-ptg-qa [2] https://review.opendev.org/c/opendev/irc-meetings/+/899457 [3] https://wiki.openstack.org/wiki/Meetings/QATeamMeeting Thanks, -- Martin Kopec Principal Software Quality Engineer Red Hat EMEA IM: kopecmartin

1 1

capacity planning question.
by KK CHN 31 Oct '23

31 Oct '23

Hi list, I am in need of an infrastructure set up for data analytics / live video stream analytics application using big data and analytics technology.. The data is basically right now stored as structured data(no video streaming) in PostgresDatabase. ( Its an emergency call handling solution, In database which stores, caller info (address, mobile number, locations co-ordinates, emergency category metadata and dispatch information regarding rescue vehicles., Rescue vehicle location update (lat, long)every 30 seconds all are stored in the Postgres Database .. Input1 : I have to do an analytics on these data( say 600 GB for the last 2 years its the size grown from initial setup).To perform an analytical application development( using python and data analytics libraries, and displaying the results and analytical predication through a dashboard application.) Query 1. How much resource in terms of compute(GPU?(CPU) cores required for this analytical application? and memory ? And any specific type of storage(in memory like redis required ? ) etc, which I have to provision for these kind of application processing. ?? any hints most welcome.. Any more input required let me know I can provide if available. Input 2 In addition to the above I have to do video analytics from bodyworn cameras by police personnel, drone surveillance Videos from any emergency sites, patrol vehicle (from a mobile tablet device over 5G )live streaming of indent locations for few minutes ( say 3 to 5 minutes live streaming for each incident. ) There are 50 drones, 500 Emergency rescue service vehicles, 300 body worn camera personnels.. and roughly 5000 incidents / emergency incidents per day happening, which needs video streaming for at least 1000 incidents for a time duration of 4 to 5 minutes live streaming. Query2. What/(how many) kind of computing resources GPUs(CPUs)? RAM, Storage solutions I have to deploy in numbers( or cores of GPUs/how many/(CPUs)? RAM ? In Memory (Redis or similar ) or any other specific data storage mechanisms ? Any hints much appreciated.. Thank in advance, Krishane

1 0

[tacker][ptg] Caracal vPTG Summary
by Yasufumi Ogawa 31 Oct '23

31 Oct '23

Hi all, We had two days session for discussing new proposals for Caracal and reamined items couldn't fixed in the previous release [1]. Here is a summary of the discussion in Tacker team. # Revise FTs Gate tests, especially for functional tests, have been getting mess because of increasing amount of codes without active maintenance. For the issue, we have focused on reducing useless logs. And also re-design directory structure and testcase scenarios. We're going to continue the tasks and will fix it in this release. # Legacy API deprecating/obsoleting It was proposed to drop Tacker's legacy implementation in antelope, and will be completed in this release as planed. # Introducing Cilium CNI for high-performance network connectivity for VNF/CNF This proposal is for a new example app using Cilium CNI as a vRAN usecase in which VNF/CNF requires performance for network connectivity between VMs or containers. The first target is to confirm it's available to deploy with Tacker in a manner of VNF package. # Update requirements for monitoring We have introduced Prometheus for implementing FM/PM interfaces defined in ETSI-NFV standards from Yoga. This proposal is for more advanced feature than monitoring, called as observability, by using OpenTelemetry for considering future requirements. We agreed to implement an additional plugin dedicating to interwork between Otel collector and Tacker. # NFV Orchestration API v2.0 in Tacker Horizon Although many of v2 APIs in ETSI specifications have been implemented, tacker-horizon is not ready for the features. We'll update NFVO interfaces in Horizon to provide v2 features. Current legacy features are going to be dropped instead. # Improving Tacker Document As legacy implementations have been deprecated, documents are also removed. In addition, there are some other sections should be updated such as incorrect usecase guides don'twork, unmaintained operator's guides or so. # Tacker Tempest Plugin Tacker doesn't support tempest, but it's not good for considering if it leads to OpenStack vendors not including Tacker in their products, and also makes Tacker look unhealthy project. So, implement Tacker's tempest plugin and update our zuul jobs to use the plugin for. # Fix vimConnectionID in ResouceHandle In the current implementation, vimConnectionID differs from ETSI specs mistakenly. So, it's should be fixed soon and expected to be small changes. # Secure RBAC This item has been led by NEC team and development of UTs and FTs are under implemention as noted on the vPTG etherpad. However, we need to update ownership of the item for manpreetk moved to another organization. Confirm if NTT or NEC team own the item in caracal and share the result in IRC meeting. # Fix issues which were encounter in compliance test execution In next release of tacker (2024.1), NEC team proposing to make 'tacker-compliance-devstack-multinode-sol' job testing mandatory and voting after some issues occured in the test are fixed. # Maintain tosca-parser and heat-translator Assign team leaders for each projs temporarily for governance, Toshiaki Takahashi for tosca-parser and Ayumu Ueha heat-translator. Ask TC memebers anything requried to change the ownership. [1] https://etherpad.opendev.org/p/oct2023-ptg-tacker Thanks, Yasufumi

1 0

[tc] Weekly Meeting Tomorrow, Oct 31 1800 UTC in OFTC
by Jay Faulkner 31 Oct '23

31 Oct '23

Hi all, The OpenStackTechnical Committee will have our weekly meeting tomorrow, October 31, at 1800 UTC in #openstack-tc on OFTC. All are welcome to attend. The proposed agenda is below. Agenda Suggestions - Roll call - Follow up on tracked action items: https://meetings.opendev.org/meetings/tc/2023/tc.2023-10-17-18.00.html - JayF: Before next video meeting, write up a short document on pros/cons of moving TC video meetings to jitsi-meet. - Gate health check - Leaderless projects (gmann) - PTG follow-ups? - https://etherpad.opendev.org/p/tc-ptg-october-2023 - https://etherpad.opendev.org/p/tc-leaders-interaction-2024-1 - https://etherpad.opendev.org/p/oct2023-ptg-os-dbperf-crossproject - Open Discussion and Reviews Thank you, Jay Faulkner OpenStack TC Chair

1 0

[barbican] Weekly Meeting 31-10-2023 Canceled
by Grzegorz Grasza 30 Oct '23

30 Oct '23

Hi all, I'm canceling the meeting this Tuesday. I'm on a business trip this week. Thanks! / Greg

1 0

Re: Regarding Nova's inability to delete the Cinder volume for creating virtual machines (version Y)
by 无数的星球 30 Oct '23

30 Oct '23

Thank you for Gorka's reply! I have seen this document multiple times and it has not been taken seriously. Next, I will try.

1 0