November 2023 - openstack-discuss

[sunbeam][ptg] vPTG summary
by James Page 02 Nov '23

02 Nov '23

HI All The Sunbeam project participated in the vPTG last week; here is a short summary of the topics that we discussed and plans added for the Caracal cycle. # Full Cycle Retro ## Celebrations - Significant reduction in both the time taken to bootstrap and resize a deployed cluster. - Switch from Antelope to Bobcat in the charms was smooth and relatively quick. - Many new plugins delivered during the cycle, enabling lots more of the OpenStack ecosystem - telemetry, magnum, heat, barbican and designate. - Refactoring of ops-sunbeam to support both k8s and machine charms was a success - Documentation starting to look great and developing lots of depth - Lots of contributions to the open source tools we depend on including snapd, Juju (inc terraform provider), microk8s, traefik and prometheus charms. ## Learnings - Switching to Bobcat for snaps and terraform plans took a bit longer than it should have done - most likely down to this being a largely new process for the team. - Some impact on velocity due to bugs in dependencies outside of OpenStack - Release process needs to be better documented so more of the core team can execute it. - Plugin full functional test coverage still needs some work. ## Actions - Implementation of automated functional testing for plugins - Full documentation and improved automation of the release process # Debugging Problems ## Overview We had quite an extended conversation about how we help users debug both bugs and deployment specific issues in Sunbeam; the general observability feature we're working on will help in this area once a cloud is deployed but we need some extra features to help users before this stage of the deployment is reached. sunbeam inspect is a bit lightweight still; can supplement the information gathered with more extensive information collected from microk8s so we need to document this into the bug reporting guidance. ## Actions - Improve docs topic 'Inspecting the cluster' to include more detail of collecting debug information for reporting bugs - Update the guidance for reporting bugs in Launchpad projects to help users report better more qualified bugs # Monorepo for Sunbeam Charms ## Overview With our current approach of 1:1 mapping between charms and repositories, we're going to have a similar challenge to the OpenStack Charms project when we need to effect a change across the entire charmset (bumping to a new release, updating a dependency or alike). This generates a large number of reviews, and associated costs of testing, which then need to be managed through to landing by one of the core team. Liam proposed that we look at moving to a single mono-repo for all of the charms in the project, reducing the overhead of 'batch' type operations and making it generally easier to manage this part of Sunbeam. We'd also address integrated testing of a change that impacts multiple charms without having to model cross-repo dependencies. The resource required to complete a function test would increase in size, but reduce hugely in number when compared to our current approach. We'll need to figure out tooling to determine which charms are impacted by a specific change so that we only build, test and publish charms that need to be changed, rather than the entire charmset. Some projects in the wider charm ecosystem are using this approach to good effect. ## Actions - Liam to take this idea forward early during the Caracal cycle - will need some coordination and advice from the infra team (see next topic). # Integration testing ## Overview Although the Sunbeam charms do include functional testing, it's not a complete integration test of a full Sunbeam deployment. We'd like to achieve this goal but this will require test instances with more memory than we typically get right now. We discussed that this felt linked to the idea of the mono-repo for charms as it feels unreasonable to ask for increased instance sizes for and increasing set of repositories when we could achieve better resource usage efficiency with a mono-repo. We agreed to fold this into the mono-repo work early this cycle. # Multi-node testing ## Overview All of our multi-node testing is currently not executed as part of the project CI workflows - we really need to test this part of Sunbeam as part of our gating for changes. We discussed a few ideas in this area including use of LXD containers on a big instance type or use of the 3rd party CI provided by Canonical that currently supports the OpenStack Charms project. Both might present viable routes forward and will be considered during the coming cycle. # Managing OCI artefacts references in charms ## Overview The OCI resources used by the Sunbeam K8S charms are managed within a different project; we need to be aware of when new OCI's are published so we can feed these updates into the charms and publish new charm revisions and associated OCI resources. There is some prior art on github modelled as actions to support this type of workflow but we need something that will work across that boundary into the opendev gerrit infrastructure. ## Actions - Reach out to the opendev infrastructure team as to whether there is any prior work on this type of function. # Single Container Multiple Services ## Overview We've spent quite a bit of time this cycle reducing the physical footprint of the OCI resources we use for Sunbeam. We discussed whether this might be taken further by reducing the number of containers we run in each charm - this is made possible by the fact that the OCI entry point is managed using Pebble, which supports multiple service definitions so we can run multiple daemons within the same container. We agreed to prototype this in one of the more complicated charms to see how much benefit might be achieved by taking this approach. ## Actions - POC probably using the nova-k8s charm to assess benefits. We had around 6 active participants during our session with a few others dropping in to listen in on the conversation from time to time - thankyou to all of those who contributed to our discussions and I look forward to the next (v)PTG. Cheers James Sunbeam PTL

1 0

[kolla-ansible][16.1.0] Could not load namespace_haproxy_for_cell
by Satish Patel 02 Nov '23

02 Nov '23

Folks, I am deploying new cloud and planning to use 2023.1 with kolla-ansible tag 16.1.0 but I hit this error and after google I found this reference bug: https://bugs.launchpad.net/kolla-ansible/+bug/2039573 Very curious about the workaround here. How does it pass CI-CD if it's really a bug?

3 3

[nova][compute] HugePages question
by Satish Patel 02 Nov '23

02 Nov '23

Folks, What are the advantages and disadvantages to enable HugePages on compute nodes? I know that HugePages improve performance of VMs because of TLB lookup etc. But if it does improve in most cases then why people why default not turning on HugePage for deployment? Just trying to find out what could go wrong if I turn on by default HugePage for all deployment?

2 2

[kolla-ansible][16.1.0] Error container timeout
by Satish Patel 02 Nov '23

02 Nov '23

Folks, I am deploying the 2023.1 release of openstack using kolla-ansible (tag 16.1.0) . I noticed a very strange error during deployment. This is very close to this bug report: https://bugs.launchpad.net/kolla-ansible/+bug/2038914 RUNNING HANDLER [common : Restart fluentd container] ***************************************************************************************************************************** fatal: [os1-bos-ctrl02]: FAILED! => {"Command": "dumb-init --single-child -- kolla_start", "Created": 1698865177, "HostConfig": {"NetworkMode": "host"}, "Id": "ae17d4d276fd278d15c5df03fdee70eafbff66fecbc78ae0bbae64274daa910b", "Image": "registry.example.com/kolla/fluentd:2023.1-ubuntu-jammy", "ImageID": "sha256:41752d28100fd85074d112942f843ce144df8bb023d7bdf664df0ef7be9a50ea", "Labels": {"build-date": "20231029", "kolla_version": "16.1.1", "maintainer": "Kolla Project (https://launchpad.net/kolla)", "name": "fluentd", "org.opencontainers.image.ref.name": "ubuntu", "org.opencontainers.image.version": "22.04"}, "Mounts": [{"Destination": "/etc/localtime", "Mode": "ro", "Propagation": "rprivate", "RW": false, "Source": "/etc/localtime", "Type": "bind"}, {"Destination": "/etc/timezone", "Mode": "ro", "Propagation": "rprivate", "RW": false, "Source": "/etc/timezone", "Type": "bind"}, {"Destination": "/var/lib/fluentd/data", "Driver": "local", "Mode": "rw", "Name": "fluentd_data", "Propagation": "", "RW": true, "Source": "/var/lib/docker/volumes/fluentd_data/_data", "Type": "volume"}, {"Destination": "/var/lib/kolla/config_files", "Mode": "ro", "Propagation": "rprivate", "RW": false, "Source": "/etc/kolla/fluentd", "Type": "bind"}, {"Destination": "/var/log/kolla", "Driver": "local", "Mode": "rw", "Name": "kolla_logs", "Propagation": "", "RW": true, "Source": "/var/lib/docker/volumes/kolla_logs/_data", "Type": "volume"}], "Names": ["/fluentd"], "NetworkSettings": {"Networks": {"host": {"Aliases": null, "DriverOpts": null, "EndpointID": "", "Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "IPAMConfig": null, "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "Links": null, "MacAddress": "", "NetworkID": ""}}}, "Ports": [], "State": "created", "Status": "Created", "changed": true, "msg": "Container timed out"} fatal: [os1-bos-ctrl01]: FAILED! => {"Command": "dumb-init --single-child -- kolla_start", "Created": 1698865177, "HostConfig": {"NetworkMode": "host"}, "Id": "1ca559230483b490830dc2e38791fc8c567702d3280713a6f0c7bc45f2c8a038", "Image": "registry.example.com/kolla/fluentd:2023.1-ubuntu-jammy", "ImageID": "sha256:41752d28100fd85074d112942f843ce144df8bb023d7bdf664df0ef7be9a50ea", "Labels": {"build-date": "20231029", "kolla_version": "16.1.1", "maintainer": "Kolla Project (https://launchpad.net/kolla)", "name": "fluentd", "org.opencontainers.image.ref.name": "ubuntu", "org.opencontainers.image.version": "22.04"}, "Mounts": [{"Destination": "/etc/timezone", "Mode": "ro", "Propagation": "rprivate", "RW": false, "Source": "/etc/timezone", "Type": "bind"}, {"Destination": "/var/lib/fluentd/data", "Driver": "local", "Mode": "rw", "Name": "fluentd_data", "Propagation": "", "RW": true, "Source": "/var/lib/docker/volumes/fluentd_data/_data", "Type": "volume"}, {"Destination": "/var/lib/kolla/config_files", "Mode": "ro", "Propagation": "rprivate", "RW": false, "Source": "/etc/kolla/fluentd", "Type": "bind"}, {"Destination": "/var/log/kolla", "Driver": "local", "Mode": "rw", "Name": "kolla_logs", "Propagation": "", "RW": true, "Source": "/var/lib/docker/volumes/kolla_logs/_data", "Type": "volume"}, {"Destination": "/etc/localtime", "Mode": "ro", "Propagation": "rprivate", "RW": false, "Source": "/etc/localtime", "Type": "bind"}], "Names": ["/fluentd"], "NetworkSettings": {"Networks": {"host": {"Aliases": null, "DriverOpts": null, "EndpointID": "", "Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "IPAMConfig": null, "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "Links": null, "MacAddress": "", "NetworkID": ""}}}, "Ports": [], "State": "created", "Status": "Created", "changed": true, "msg": "Container timed out"} fatal: [os1-bos-ctrl03]: FAILED! => {"Command": "dumb-init --single-child -- kolla_start", "Created": 1698865177, "HostConfig": {"NetworkMode": "host"}, "Id": "8f3599681f56695b0e809acb5645d79deb3fbb476dc7bda5d75f838f67ed085d", "Image": "registry.example.com/kolla/fluentd:2023.1-ubuntu-jammy", "ImageID": "sha256:41752d28100fd85074d112942f843ce144df8bb023d7bdf664df0ef7be9a50ea", "Labels": {"build-date": "20231029", "kolla_version": "16.1.1", "maintainer": "Kolla Project (https://launchpad.net/kolla)", "name": "fluentd", "org.opencontainers.image.ref.name": "ubuntu", "org.opencontainers.image.version": "22.04"}, "Mounts": [{"Destination": "/etc/localtime", "Mode": "ro", "Propagation": "rprivate", "RW": false, "Source": "/etc/localtime", "Type": "bind"}, {"Destination": "/etc/timezone", "Mode": "ro", "Propagation": "rprivate", "RW": false, "Source": "/etc/timezone", "Type": "bind"}, {"Destination": "/var/lib/fluentd/data", "Driver": "local", "Mode": "rw", "Name": "fluentd_data", "Propagation": "", "RW": true, "Source": "/var/lib/docker/volumes/fluentd_data/_data", "Type": "volume"}, {"Destination": "/var/lib/kolla/config_files", "Mode": "ro", "Propagation": "rprivate", "RW": false, "Source": "/etc/kolla/fluentd", "Type": "bind"}, {"Destination": "/var/log/kolla", "Driver": "local", "Mode": "rw", "Name": "kolla_logs", "Propagation": "", "RW": true, "Source": "/var/lib/docker/volumes/kolla_logs/_data", "Type": "volume"}], "Names": ["/fluentd"], "NetworkSettings": {"Networks": {"host": {"Aliases": null, "DriverOpts": null, "EndpointID": "", "Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "IPAMConfig": null, "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "Links": null, "MacAddress": "", "NetworkID": ""}}}, "Ports": [], "State": "created", "Status": "Created", "changed": true, "msg": "Container timed out"}

1 0

[ironic] Announcement: Future of Metalsmith
by Jay Faulkner 01 Nov '23

01 Nov '23

Hi all, Just announcing that at the vPTG, the Ironic core team came to a consensus about the future of the metalsmith project. It was originally created to improve the user experience around provisioning Ironic nodes when using Ironic's API directly. At this, it was mostly successful and has completed this task. It's our intention to work on the Ironic API and python-ironicclient to provide a similarly simple deployment user experience, even for users without nova, integrating lessons learned from the metalsmith project. To this end; we are letting the overall community know we no longer expect to add additional functionality to metalsmith. We will ensure it remains maintained and well-tested until it's functionality has been duplicated, but all new work in this area will be directed at the primary Ironic API and Ironic client projects. To this end, and as discussed at the PTG, I've proposed a change to the Metalsmith readme, notifying users of this state. https://review.opendev.org/c/openstack/metalsmith/+/899761 This change, or this mailing list post, is a great place to comment if you have any comments on this decision. Thanks, Jay Faulkner Ironic PTL

3 3

Cinder Bug Report 2023-11-01
by Jon Bernard 01 Nov '23

01 Nov '23

Hello Argonauts, Cinder Bug Meeting Etherpad <https://etherpad.opendev.org/p/cinder-bug-squad-meeting> Undecided - hitachi and OEM : output REST API token <https://bugs.launchpad.net/cinder/+bug/2040966> - Status: New - hitachi AIX as host OS type does not work <https://bugs.launchpad.net/cinder/+bug/2041784> - Status: In Progress - The 'signature_verified' parameter should not be added to image when upload volume to image <https://bugs.launchpad.net/cinder/+bug/2041793> - Status: In Progress - Missing encryption param handling in Quobyte driver <https://bugs.launchpad.net/cinder/+bug/2042102> - Status: New Thanks, -- Jon

1 0

[manila] 2024.1 Caracal PTG summary
by Carlos Silva 01 Nov '23

01 Nov '23

Hello. Here is a summary of all topics we have discussed during last week's PTG and the AIs that were a result of the discussion. Thank you to all contributors that participated. If you missed one or more sessions, we got you covered. The Manila PTG, as well as the operator hour were recorded and the videos are available in the OpenStack Manila YouTube Channel [0]. Deferred deletion ------------------------- We discussed a bug that exists and the need for a deferred deletion approach. We agreed that the deferred deletion would be ideal in case the driver needs to do multiple things in the deletion and we don't want the operators to wait. #Agreed: The service should release user quotas as soon as the driver says it is processing the deletion; and handle errors internally AI: kpdev and NetApp/SAP will look into the deferred deletion implementation and provide a spec All things CephFS -------------------------- We discussed deprecating support for "standalone" NFS-Ganesha with the CephFS/NFS driver. We've supported using a clustered Ceph NFS service since the Zed release. We've also added upgrade helpers to migrate from a standalone NFS-Ganesha service to the Ceph NFS service; this code was backported to stable/2023.1 (Antelope) Testing and stabilization: We'll continue testing cephfs/nfs with standalone NFS-Ganesha until we drop support for ganesha's DBUS mechanism from the CephFS/NFS driver. IBM is currently testing the scale, performance and stability of clustered-nfs-ganesha and both, nfs-ganesha and cephadm are being updated accordingly Mulitnode devstack ceph in CI: - devstack testing of cephfs/nfs has been stable, we're trying to run this job on stable/2023.1 - AI: ashrodri will enable ingress service with cephadm Ceph NFS upgrades The code patches for upgrading to a cephadm deployed NFS cluster were merged and backported to stable/2023.1 (Antelope). When upgrading, access rules are reapplied and export locations will indicate the preferred export path for mounting shares. Deprecating use of DBUS API with NFS-Ganesha As the clustered nfs solution is hardened, and an upgrade mechanism is invented, we want to stop supporting the use of "standalone-nfs/ganesha" server with CephFS. AI: add a deprecation warning in the Caracal cycle and communicate it in the mailing list Multiple CephFS filesystems We discussed pros/cons of using a single driver representing multiple file systems as Manila storage pools - there are not a lot of benefits to this, especially considering the ease in deployment/configuration of the manila share services currently For multi-fs environments, the user experience to mount shares with Native CephFS is poor since the export paths no longer expose the filesystem name Operators suggested that we encode the filesystem name in export location metadata so automation tools (e.g.: manila CSI plugin, nova/virtiofs) can consume it programmatically AI: Elias Wimmer will report a bug against the CephFS driver AI: Ceph driver documentation will be updated regarding multi-fs and the behavior of the driver Ceph Mon address is never re-validated [3] Today, the only way to re-export a share is by restarting the manila-share manager service. We discussed an API based trigger to the ensure shares logic that will recalculate the export locations of the shares. We agreed that this implementation should add a micro-state to the share and that the API should deny other management operations on the shares when "ensuring". AI: carloss will propose a specification to implement this change Human readable export locations ----------------------------------------------- Export locations are constructed for machines, and they do not use human friendly names. NetApp proposed user customizable export locations a few releases ago [4] We discussed the proposal, and find it is still relevant - we brainstormed using a share type extra specification for customizing a prefix to the export location names We agreed that updating the export location of a share is bound to be disruptive and will not be part of the implementation. We may consider this enhancement if there is a very strong use case AI: NetApp folks will propose updates to the lite-spec and work on the implementation Share backups --------------------- A generic NFS implementation of share backups was introduced in the previous (Bobcat) cycle We discussed a share-manager driven advantaged share backup driver implementations, as well as the data service backup driver interface Cern found that the backup driver interface called back into the data service, and the driver interface needs changes AI: Luis from Cern will update the Backup driver to decouple the data manager interface on-demand in preparation for their implementation of the restic backup driver Tech debt -------------- During the tech debt session, gouthamr walked us through some multi-release tech debt items, such as the progress on scenario tests and the implementation of mechanisms to prevent race conditions. It is a good opportunity for someone in the community to step up and do some of the work, which can be a good technical challenge. AI: Discuss the specification in weekly IRC meetings and find new owner/s Barbican integration ----------------------------- Backend data encryption has been supported for several releases in Manila, but users don't have the ability to configure their encryption keys SAP is pursuing an integration with Barbican for Manila and has worked on a specification. #Agreed: Manila should have a Barbican/Castellan API layer to interact with the keystore, and share backends shouldn't have code to interact directly with Barbican. AI: kpdev will update the proposed spec [5] also documenting the interactions with Castellan Manila/Cinder cross-project session --------------------------------------------------- We organized a cross-project session with the Cinder team to discuss a couple of things: - An enhancement to the manila volume extend API calls in the generic driver: - as the Cinder APIs evolved and Manila would benefit from it. Rajat can work on the enhancement in case the Manila contributors don't have the bandwidth. - The way DHSS=True works in Manila - we brainstormed hypothetical support in Cinder for service instances/storage virtual machines and tenant driven volume networking - We also talked about the progress both the projects achieved with respect to Secure RBAC, OpenStackClient and OpenStackSDK. Expectations while submitting features -------------------------------------------------------- We are working on documenting the process one should go through while submitting features [6]. This also covers what are the deadlines and what reviewers expect in terms of feature submission. AI: carloss will update the new documentation with a checklist that will illustrate the definition of done for a feature Howcasts -------------- This is something we have been ideating on for a while. The idea is to produce short videos that will help newcomers to the project. We have an idea of what to work on [7] and help from the community would be appreciated. AI: we'll start the work on the howcasts and try to split the load within the community members SQLAlchemy 2.0 ----------------------- stephenfin has been working on the SQLAlchemy 2.0 changes for Manila and we have made good progress so far. One of the problems to test the changes is that we need to wait a whole entire CI run that can take up to 3 hours Agreed: - We'll work with stephenfin to reduce the amount of wait time to get CI results by reducing the tests using the dummy drivers - We'll review the changes and help figure out the issues with migration of share replicas and share instances queries Tech Debt Roundup - 2; multi-release efforts that have attention --------------------------------------------------------------------------- ---------------- Metadata spec We currently have metadata implemented for three resources (shares, share snapshots and share network subnets). We have open patches for access rules and export locations. AI: Update the export locations and access rules metadata changes and work on them during the Caracal cycle. OSC Support: We have parity between OSC and the manila shell client and we added a deprecation warning in the previous cycle and the removal will happen prior to the next SLURP release (E). This means that Manila's shell client will not be shipped as part of the D release. - More negative functional tests would be welcome, as we have implemented only the happy-paths during a hackathon. - AI: consider a OSC test hackathon during the Caracal release hackathon - AI: all documentation will be updated to utilize OSC in examples SDK We have completed a lot of this with the help of university students/outreachy interns. Most of the work that was pressing and features that were needed to integrate with other services were covered. AI: Continue working towards full parity between openstacksdk and manilaclient SDK Agreed: There are no plans to deprecate manilaclient SDK Share Transfers: We have share transfers between projects for DHSS=False, but we are still lacking support for DHSS=True. This is a cool technical challenge to work in Manila, and having someone willing to continue this work would be appreciated. Follow up on Resource locks ------------------------------------------ The two featureful changes for resource locks were merged in the Bobcat cycle (share resource locks and locks for access rules deletion and visibility restrictions) There are two changes yet to be merged, being the OpenStack SDK patch and the manila-tempest-plugin tests for the access rules locks. We brainstormed implementations of other resource locks, such as a visibility lock on sensitive fields of a security service, or deletion locks for snapshots, backups and replicas AI: We'll implement new resource locks opportunistically during the Caracal release Operator Hour --------------------- We have reflected over the results of the latest OpenStack user survey [9] and talked about the progress of the most requested features and also requested feedback from operators. We discussed the use cases for the disabling manila services feature, and chatted about one possible enhancement to this feature which would be to add backends as disabled and enable them later. AI: we should enhance our documentation to talk about start and stopping services, as well as the differences between stopping and disabling them. [0] https://www.youtube.com/@openstackmanila [1] https://github.com/openstack-k8s-operators/manila-operator/ - a replacement for TripleO [2] https://prow.ci.openshift.org/job-history/gs/origin-ci-test/pr-logs/directo… (Native CephFS) [3] https://bugs.launchpad.net/manila/+bug/1996793 [4] https://opendev.org/openstack/manila-specs/src/commit/baf6209da8e5b47228024… [5] https://review.opendev.org/c/openstack/manila-specs/+/898999 [6] https://review.opendev.org/c/openstack/manila/+/898855 [7] https://etherpad.opendev.org/p/manila-howcasts [8] https://etherpad.opendev.org/p/caracal-ptg-manila-operator-hour [9] https://drive.google.com/file/d/1ZM722TOX3ouP_7Cet2JdURFttTuI5ZLe/view?usp=… [10] https://etherpad.opendev.org/p/caracal-ptg-manila If you have questions or would like to follow-up on any of the topics, don't hesitate to reach out. Thank you! carloss

1 0

[manila] Nov 2nd upstream meeting is cancelled
by Carlos Silva 01 Nov '23

01 Nov '23

Hello! As some of our regular contributors will be on PTO tomorrow and we are following up on all-things-ptg, we are cancelling this week's weekly meeting. The next meeting will take place next week (Nov 9th). Thanks and regards, carloss

1 0

Volumes not deleting
by Russell Stather 01 Nov '23

01 Nov '23

Hi We have a juju openstack installation and we are finding that when we delete servers the associated volume doesnt get deleted. The volume stays as showing attached to a non existent server instance. We are having to marked them as not attached in the database and then the volume delete works. Any ideas what could be wrong here? Thanks Russell

2 1

Openstack volume deletion
by Vijay Thilak S 01 Nov '23

01 Nov '23

Hi, OpenStack Kolla Ansible has been installed on Ubuntu 20 and integrated with Ceph storage. The volume associated with the instance is not being deleted by Cinder when I terminate the instance with this option (Delete Volume on Instance Delete), which is causing me problems. I need to log in to the server and run the cinder commands to manually delete the volume that is not associated with instances. I would appreciate it if you could provide a solution. Thanks, Vijay

4 10