October 2022 - openstack-discuss - lists.openstack.org

network monitor
by Parsa Aminian 27 Oct '22

27 Oct '22

hello Im using openstack kolla-ansible . Could you please tell me how I can monitor instances' network usage ? 1-download and upload speed 2-traffic usage for example send and receive per month for each instance

3 3

[neutron] Neutron drivers meeting Oct 28th
by Rodolfo Alonso Hernandez 27 Oct '22

27 Oct '22

Hello Neutrinos: The drivers meeting will take place tomorrow at 14UTC. The agenda has one single topic: * https://bugs.launchpad.net/neutron/+bug/1994137: [RFE] Specify the precedence of port routes if multiple ports attached to a VM Regards.

1 0

[openstack-ansible][ptg] Antelope PTG results
by Dmitriy Rabotyagov 27 Oct '22

27 Oct '22

Hi everyone! As you might know, the OpenStack-Ansible team had Project Team Gathering on October 18, 2022 and we had quite good discussions. This email is a relatively short sum-up of decisions that were taken during it: * Add Zookepeer cluster deployment as coordination service. Coordination is required if you want to have active/active cinder-volume setup and also actively used by other projects, like Octavia or Designate. Zookeeper will be deployed in a separate set of containers for LXC path * Move out Ubuntu Jammy from experimental state except Ceph integration. This is because community repo does not have ceph packages built for Ubuntu 22.04, so we will switch default to use Ubuntu's native repository to get ceph from. With that you won't be able to pick what ceph version to use, but it will work for most usecases. * Continue supporting Ubuntu Focal (20.04) until Antelope release as upgrade path from Yoga to Antelope should exist and on TC PTG it was decided that it should be tested by projects on Ubuntu 20.04. * Update docs regarding level of support for integrated ceph-ansible deployment with openstack-ansible to address expectations on upgrade path. * Address concerns regarding Glance configuration with COW backends, with regards to OSSN notice: https://wiki.openstack.org/wiki/OSSN/OSSN-0090 * Switch distro path to experimental due to lack of maintainers. We should document that and mention that help is needed to keep it alive. If not - the feature can be removed in the future. * Cover OVN with TLS encryption. Leverage PKI role for that. * We also agreed to switch the default network driver used by OpenStack-Ansible to OVN as default. However, at the moment there is not much hands-on experience on OVN from the core team, so we can not switch default right now as we won't be efficiently helping with OVN deployments. So following switching plan was defined: ** Document reference OVN architecture that we are going to provide ** Document implementation of extra networks that are required for octavia, trove, etc. ** Switch AIO to OVN as default ** Add LXB jobs to neutron role ** Switch default for non-AIO ** Ensure that users who were relying on previous default (and using LXB) have proper override during upgrade. * Improve documentation for the Ironic role with adding a couple of scenarios. * Convert our dynamic inventory to an inventory plugin which will be installed during ansible bootstrap. * Create a repository for the skyline role. We are allowed to take https://github.com/jrosser/openstack-ansible-os_skyline as base and move it under openstack-ansible umbrella * Simplify documentation regarding provider_networks in openstack_user_config and suggest using neutron_provider_networks instead. * Continue working on improving our CI coverage, including returning to molecule testing but rely on the ansible and constraints version from the integrated repo. At the same time finish cleanup of old functional tests, including run_tests.sh, tox, etc. * Deprecate rsyslog_server/client roles as they are hardly used as of today. You can also check etherpad [1] where notes were taken during PTG and that we aim to update with progress on these points implementation: [1] https://etherpad.opendev.org/p/osa-antelope-ptg

1 0

[all][tc] Technical Committee next weekly meeting on 2022 Oct 27 at 1500 UTC
by Ghanshyam Mann 26 Oct '22

26 Oct '22

Hello Everyone, The technical Committee's next weekly meeting is scheduled for 2022 Oct 27, at 1500 UTC. If you would like to add topics for discussion, please add them to the below wiki page by Wednesday, Oct 26 at 2100 UTC. https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting -gmann

1 1

[cinder] Problem with restoring purestorage volume backup
by Ришат Азизов 26 Oct '22

26 Oct '22

Hello! We have a problem with cinder-backup with cephbackup driver and volume on purestorage. When backing up purestorage volume with cinder-backup with cephbackup driver it creates in ceph pool and everything is ok. But when we try to restore the backup, it is not restored with an error "rbd.ImageNotFound" in the screenshot attached to this email. This happens because the original image is not in rbd, it is in purestorage. It is not clear why the cinder is trying to look for a disk in the ceph. Could you please help with this? Thanks. Regards.

2 1

[cinder] Bug report from 10-12-2022 to 10-26-2022.
by Sofia Enriquez 26 Oct '22

26 Oct '22

This is a bug report from 10-12-2022 to 10-26-2022. Agenda: https://etherpad.opendev.org/p/cinder-bug-squad-meeting ----------------------------------------------------------------------------------------- Undecided - https://bugs.launchpad.net/cinder/+bug/1994018 "The volume multiattach and in-use after retyping another backend, then can not detach it." - https://bugs.launchpad.net/cinder/+bug/1994021 "Cinder cannot work when 1 node of 3 rabbit node cluster down." Low - https://bugs.launchpad.net/cinder/+bug/1993282 "Fail to get pools by volume_type filter key." Assigned to zhaoleilc. No fix proposed to master yet. - https://bugs.launchpad.net/os-brick/+bug/1994083 "FileNotFoundError: [Errno 2] No such file or directory: is raised when dmidecode is not installed." Fix proposed to os-brick master. Wishlist - https://bugs.launchpad.net/cinder/+bug/1994150 "Image signature verification does not verify certificates." Unassigned. - https://bugs.launchpad.net/cinder/+bug/1992493 "Cinder fails to backup/snapshot/clone/extend volumes when the pool is full." Unassigned. - https://bugs.launchpad.net/cinder/+bug/1992685 "Automate generation of snapshot transfer api-ref samples for MV 3.55." Assigned. No fix proposed to master yet. Invalid - https://bugs.launchpad.net/cinder/+bug/1993577 " [JovianDSS] Unable to provide target prefix through iscsi_target_prefix ." Cheers, -- Sofía Enriquez she/her Software Engineer Red Hat PnT <https://www.redhat.com> IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 0

[neutron] Neutron team meeting cancelled Nov 1st
by Rodolfo Alonso Hernandez 26 Oct '22

26 Oct '22

Hello Neutrinos: The next Neutron team meeting on November 1st is cancelled (All Saints' Day [1]). The next one will take place at the same time on November 8th. Regards. [1]https://en.wikipedia.org/wiki/All_Saints%27_Day

1 0

[manila] Antelope PTG Summary
by Carlos Silva 25 Oct '22

25 Oct '22

Hello, Zorillas and interested stackers. Thank you for the productive PTG we had last week. Here goes the summary for the past week's discussions. In case you would like to see the expanded versions, please refer to [0] or check out the recordings in the Manila YouTube Channel [11]. Different approaches for the images we use in our jobs - Recent changes in the Manila CI (related to the service image) made the jobs take longer to run and become more resource intensive, causing jobs to fail more often due to lack of resources in dsvm - We have been looking for approaches to tackle this issue - In the short term, if a job keeps failing due to such issues, we will split it into two jobs - One to run scenario tests (these tests spawn VMs) - One to run API tests - We agreed that this can get worse in the future with dsvm images that could be even more resource demanding, and that we will look for containerized approaches to try and solve this issue Share backup - (kpdev) The proposed approach for share backup has changed - An existing specification [1] was restored - The idea is to have new APIs for share backup and a new generic Backup driver in a similar way to what Cinder does. - This would allow backends to have their specific implementations for share backups - Reviews for this specification are ongoing Secure RBAC changes - Goutham has shared with the community all the good progress we had in Manila, and also mentioned what we did in Zed, after all the operator's feedback. - Now, we are entering a phase where we want to test a lot! - We have a CI jobs and some test cases covered, but we still want to have more coverage - During Antelope, we will promote a hackathon in order to accelerate the functional tests coverage Outreachy Internships - On this topic, ashrodri, fmount, gouthamr and carloss shared the two proposed Outreachy project that are Manila related - ashrodri and fmount are proposing a project to have an intern creating a multi-node test environment with Devstack and Ceph [2], as these tests are becoming more resource demanding - carloss and gouthamr are proposing a project for an intern to work in Manila UI [3]. The idea is to close down the gap of features between Manila UI and the Manila API. OpenStackSDK Manila Status - This has been an effort for some time, and during the Zed cycle some code was merged for integrating share groups and share group snapshots. - This is a good candidate project for an internship, and the idea is that we will be able to get some interns working on this project (potentially a project for a college capstone project) OSC Updates - Zed was the release we targeted to reach feature parity with the native client, and we made it! - We had an idea to add a deprecation warning to the native manila client, but that will need to wait - There is still a missing bit for adding the deprecation warning: having version autonegotiation working, which is something we are targeting for Antelope - OSC is now the primary focus of the project when doing implementations for CLIs. Bandit Testing / VMT - The Manila team was introduced to the VMT (vulnerability management team) and Bandit, and we had contributors sharing their ideas around having manila under the VMT and running bandit tests. - VMT - The audience agreed with the conditions to be under the overseen repos. - Goutham has volunteered to be the security liaison for Manila - Bandit - There were some errors pointed out in a preliminar test run [4]. - We will add a bandit non-voting job and file bugs against third party drivers that have issues pointed out by the new job - New bugs will be filed and will be distributed across community members All Things CephFS - Ingress aware NFS Ganesha - Ceph Orchestrator is now capable of creating an ingress service to front-end a cluster of active-active NFS Ganesha instances via a single load-balanced IP address - Previously, we would only use one single instance of NFS Ganesha. This introduces a SPOF and is not suitable for production environments without HA being handled externally (like TripleO deployments do). With this change, Manila CephFS NFS users would be able to deploy multiple NFS Ganesha servers and leverage inbuilt HA capabilities - Currently though manila client restrictions (access rules) will not work since NFS-Ganesha sees the ingress service's IP instead of the client's IP address. - ffilz proposed his design to support the PROXY protocol with NFS-Ganesha. Ceph Ingress can pass the Client's IP address to NFS-Ganesha [5] - An alternative would be to have cephadm assign stable IP addresses to ganesha instances - No driver changes are anticipated in either approach - AI: Investigating deploying ceph-ingress with devstack - Migration tool for Manila with CephFS NFS - After the new helper was introduced to the driver, we are able to interact with cephadm-deployed CephNFS service (which comes with a lot of benefits) - Now, we need to figure out how to upgrade deployments with CephFS NFS using the current helper (which interacts with NFS Ganesha through DBUS) - There are two main issues: - Representing a change in exports when migrating across NFS Servers (current use case) or when decommissioning NFS Servers - Representing exports from multiple servers - a special case enabling migration where an old server will eventually be decommissioned: - These issues were talked through and possible solutions were proposed and will be worked on in the following cycles. More details on [6] - DHSS=True for CephFS driver - Currently, CephFS driver, whether you'd like to use it for Native CephFS, NFS via DBUS, NFS via ceph-mgr only supports DHSS=False. The request of this feature was raised during the OpenInfra Summit in Berlin from several Manila users. - We discussed some alternatives for making it happen: - 1) Operator determines Ceph cluster limits, and creates isolated file systems and declares multiple manila backends with each filesystem - 2) Subvolume groups - pinning a subvolume group to mds can isolate/dedicate mds load - CephFS CI issues - The lack of resources in the dsvms is affecting the CephFS Native and NFS jobs, causing them to be unstable. The jobs often run into timeouts - The situation could get worse as Jammy Jellyfish packages are not available yet - We will ask the ceph-users if jammy release bits can be made available Oversubscription enhancements - Storage pools supporting thin provisioning are open to oversubscription. This caused some problems mentioned in [7]. - We have an open specification which we intend to merge in the Antelope cycle, as well as the changes to address such issue FIPS - We have shared our testing status with regards to FIPS. We have jobs merged on stable branches up to wallaby for both manila and python-manilaclient repositories. - The next steps would be a more in-depth code audit to identify non-compliant libraries and making our jobs voting - We agreed to make our jobs voting when the Ubuntu images supporting FIPS are out - For drivers using non FIPS compliant libraries, we will notify the maintainers Metadata API update - Metadata APIs for share snapshots were added during the Zed cycle - The goal for this cycle is to have functional testing and the CLI merged. Both patches are under review and expected to be merged not late in the release. Manila CSI - The CSI plugin's been pretty stable in the last six months [8] - There was a good talk presented at KubeCon by Robert Vasek [9] - The next steps include getting a fix for an issue involving long snapshot names in the CephFS backend and supporting volume expansion in the OpenShift Manila CSI driver operator Manila Configuration of VLAN Network information - A issue was found where Contrail Neutron Plugin did not return the VLAN ID during port allocation [10] - To tackle this issue, we have agreed to add metadata to the share network APIs so the administrators would be able to add VLAN as metadata and drivers would be able to consume it. *Better use of bug statuses* - Our bugs were stuck in "New" instead of "Confirmed" or "Triaged" and this could be misleading. - We agreed to tag the bugs as confirmed or triaged depending on the outcome of our triaging and not leave bugs as "new" - We will ask the bug assignees to update their open bugs in case one of them has the new status, so we can have a better visibility [0] https://etherpad.opendev.org/p/antelope-ptg-manila [1] https://review.opendev.org/c/openstack/manila-specs/+/330306 [2] https://www.outreachy.org/outreachy-december-2022-internship-round/communit… [3] https://www.outreachy.org/outreachy-december-2022-internship-round/communit… [4] https://gist.github.com/gouthampacha/c0d96966670956761b2b620be730efe2 [5] https://docs.google.com/document/d/1orjNjtEeeyRrgvbQuFU5BdsCnZG9gF3MymDCZl-… [6] https://etherpad.opendev.org/p/antelope-ptg-manila-cephfs [7] https://etherpad.opendev.org/p/manila_oversubscription_enhancements [8] https://github.com/kubernetes/cloud-provider-openstack/commits/master/pkg/c… [9] https://www.youtube.com/watch?v=XfpP9pBTXfY&t=1145s [10] https://bugs.launchpad.net/charm-manila/+bug/1987315 [11] https://www.youtube.com/playlist?list=PLnpzT0InFrqBzbSP6lcYDStKpbEl3GNHK Thanks, carloss

1 0

[Glance] Antelope PTG Summary
by Pranali Deore 25 Oct '22

25 Oct '22

Hello Everyone, We had our sixth virtual PTG between 18th OCt to 21st OCT 2022. Thanks to everyone who joined the virtual PTG sessions. Using bluejeans app we had all the discussion around different topics for glance, glance + cinder, glance + ceph, fips and secure RBAC. I have created etherpad[1] with notes from the session and also included the recordings of each session. Here is the short summary of the discussions. Tuesday, OCT 18th 2022 # Zed Retrospective On the positive note, we have merged a number of useful features this cycle. We managed to implement glance-download internal plugin to download the image from remote glance, Implemented support for immediate caching of an image, Extended the functionality of stores-detail API to expose store details of other stores and we have removed the scope scheck from scope_types for all project resources and done with phase 1 as per the revised community goal. In addition to that we have successfully organized the review party before each milestone to perform group review to cover the review load till the final milestone. On the other side we decided to organize a midcycle general cross project meetup/drivers meetup towards the end of the 2nd milestone to increase our presence in cross-projects. Recording: https://bluejeans.com/s/II_@CAqrZdd - Chapter 1 # Distributed responsibilities among cores/team This cycle also we have decided to follow the distributed leadership internally, we are going to distribute below responsibilities among the team, Release management:pranali Bug management: cyril Meetings: pranali Stable branch management: cyril and Erno Cross project communication: abhishekk Mailing lists: share responsibility, pranali PTG/summit preparation:pranali Vulnerability management: glance-coresec group Infra management: periodic-jobs - abhishekk migration of test jobs -abhishekk Recording: https://bluejeans.com/s/II_@CAqrZdd - Chapter 2 # Default Glance to configure multiple stores Glance has deprecated single store configuration since Stein Cycle and this cycle we are going to start putting our efforts to deploy glance using multistore by default and then remove the single store support from glance. Recording: https://bluejeans.com/s/II_@CAqrZdd - Chapter 3 # Add missing CLI support for some Glance API in Openstack Client The CLI support for all glance APIS is already there in GlanceClient and the similar CLI support we need to have in OpenstackClient, in this Cycle we are going to put efforts to have OSC support for all the missing glance APIs. Recording: https://bluejeans.com/s/II_@CAqrZdd - Chapter 4 # Glance Cache improvements, restrict duplicate downloads This is abou how we can avoid multiple downloading of the same image in cache on first download Spec : https://review.opendev.org/c/openstack/glance-specs/+/734683 We had this spec in the last cycle & decided to break the image into chunks and when the first request gets to the backend store it will start caching that image and if any other request comes in between and if caching is still in process it will read from the chunks created by the first request. But currently we have made one caching state to check if the image is still in caching but that was relying on md checksum to check if the image iterator has read the image completely or not but in new images we might not have checksum, multihash or size of the image, bcz if that is not present with the image we won't be able to change the state of image and thus we will never be able to resolve the issue of checking the image is in caching or not. Decided to dig more on the size verification & need to revisit this topic during mid-cycle meeting and update the spec with the solution for handling the multiple request and multiple chunks Recording: https://bluejeans.com/s/II_@CAqrZdd - Chapter 5 Wednesday OCT 19th , 2022 # Image uploads to the filesystem driver are not fully atomic No efficient way to reproduce this issue, so it's decided to mark it as 'Won't fix' Recording: https://bluejeans.com/s/j_UgZFw_jEV - Chapter 1 # DB migration constant change handling Till now we have all the migration scripts by the name of the cycle and since currently release has been change to 2023.1 which is going to break the migration test because when our DB sync tool runs it will check the initial version liberty and it finds the migration script from the liberty and traverse through all the directories till the current release and executes all the scripts available in that path. Decided to fix this by updating the data migration current release to '2023.1' and check with actual migration script to check whether there is any regression or not and check if it executes the scripts in serial manner. Recording: https://bluejeans.com/s/j_UgZFw_jEV - Chapter 2 # Configurable Soft Delete Stephen initiated this topic for nova and oslo.db but also sent out a mail for glance, cinder and for other projects, if we would be interested in the idea https://lists.openstack.org/pipermail/openstack-discuss/2022-October/030729… But Glance replies a lot on soft delete and doesn't allow hard delete as glance allows the user to specify the UUID of the image , so it's part of a security promise of an immutable images that you are not able to delete an image and then recreate image with same UUID right after, hence soft delete model can't be removed from Glance. Recording: https://bluejeans.com/s/j_UgZFw_jEV - Chapter 3 # Secure RBAC As per the revised community goal, till zed cycle we must have the project prosona implementation & drop the system scope which we have already done in glance. We have implemented project scope for image apis in wallaby & in Xena we have managed to move all policy checks to API layer and implemented project scope of metadef APIs. During zed cycle, we had discussed that like which apis should be exposed to system scope but after the operators feedback & as per the revised community goal it was decided to drop the system scope, we just had to remove the scope scheck from scope_types for all project resources, so we are done with Phase 1. In Antelope Cycle glance is going to switch the new defaults flag ON, once it is verified by tempest for all the services. Recording: https://bluejeans.com/s/j_UgZFw_jEV - Chapter 4 # Image Export with Metadata This is about exporting images with associated metadata for importing into another Glance deployment. This cycle we need some volunteers to work on this, the glance team will help in terms of reviews/finalizing the design etc otherwise we will revisit this in the next cycle. This session was not recorded as it was a small discussion # Fips Overview Path forward: - Investigate on the time outing of the existing glance fips job - Try to do the centos9 jobs stable till wallaby, so could possibly move to voting - Have ubuntu jobs working & running and make them stable Recording: https://bluejeans.com/s/j_UgZFw_jEV - Chapter 5 Thursday, OCT 20th, 2022 # Option needed to create image in erasure coded ceph pool To use the erasure coded ceph pool feature where the data will be held needs to be specified during image creation, a config option is needed. Decided to write a spec first describing this in detail and modify the existing devstack job for ceph. Recording: https://bluejeans.com/s/cNEJ@Yq_hv5 - Chapter 1 # Parallelization of RADOS image writes when creating an image This is for parallelization of image writes into ceph, by increasing the amount of data written at once. Decided to not have this without testing to measure if there's an actual gain. Recording: https://bluejeans.com/s/cNEJ@Yq_hv5 - Chapter 2 # Add chunk download support for rbd backend Rbd supports random reading, and the rbd driver is designed to support partial download. But the current version disables this feature, and doesn't implement the chunk support. There is potential future use for it, but there's nothing that users would gain from this. If not usable through the API then we will drop this. Recording: https://bluejeans.com/s/cNEJ@Yq_hv5 - chapter 3 # Operators hour No operator joined the session # Speedup upload images for Swift backend Upload images using Swift backend is slow due to the synchronous way of uploading fragments and which causes loading that can take several hours, especially in cases with very large images. Decided to update the spec with two implementations one as traditional & one for multithreaded one, by having a new configuration option ``swift_store_thread_pool_size`` to the Swift store backend. Recording: https://bluejeans.com/s/cNEJ@Yq_hv5 - Chapter 4 Friday, OCT 21st, 2022 # Cross project meet with Cinder # Glance Image Direct URL access Issue Glance has OSSN-0090[2] describing the security risk when you are operating glance with 'show_multiple_locations' or if the end user facing glance-api has 'show_image_direct_url' options set to true. When glance shares a common storage backend with nova & cinder, it is possible to open some known attack vendors by which malicious data modification can occur. Decided to fix this by going with the solution proposed by Rajat during last cycle, to remove the show_multiple_locations config option and to have below 2 new location APIs[3] which will replace the image-update mechanism for consumers like cinder and nova in glance, 1. Location ADD API: Design this api in way that the location will be added only once during image create when the image is in QUEUED state and no-one should be allowed to add location after the image is active This wouldn't require the 'service role' and a basic check on the glance side to check image status should suffice. 2. Location GET API: This will show all the locations associated with an existing image. Returns an empty list if an image contains no locations. This would still require the 'service role' since we don't want to expose locations to end users. Glance has dependency on keystone for 'service role' which is going to be implemented in this cycle as per phase 2 target mentioned in SRBAC community goals[4]. Recording : https://bluejeans.com/s/B4Rlifuwx_l - Chapter 1 You will find the detailed information about the same in the PTG etherpad[1] along with the recordings of the sessions and milestone wise priorities at the bottom of the etherpad. Kindly let me know if you have any questions about the same. [1]: https://etherpad.opendev.org/p/antelope-glance-ptg [2]: https://wiki.openstack.org/wiki/OSSN/OSSN-0090 [3]: https://specs.openstack.org/openstack/glance-specs/specs/zed/approved/glanc… [4]: https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rb… Thanks, Pranali

1 0

[cinder] Cancelling this weeks meeting 26 Oct
by Rajat Dhasmana 25 Oct '22

25 Oct '22

Hello Argonauts, Since we just had PTG last week, it seems reasonable to skip this week's meeting. Also I will be on leave tomorrow so cancelling this week's cinder meeting (26 October, 2022). - Rajat Dhasmana

1 0