- openstack-discuss - lists.openstack.org

RE: openstack-discuss Digest, Vol 61, Issue 1 Openstack- Instance creation
by Asma Naz Shariq 01 Nov '23

01 Nov '23

Hi Openstack Team, When I tried to create instance in Openstack dashboard, I encountered below error: Error: Failed to perform requested operation on instance "Test-Instance", the instance has an error status: Please try again later [Error: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance f8cb7bb1-64db-4242-a26a-65f3568ff4e9.]. Can anyone guide me what is the root cause for this error? Regards, -----Original Message----- From: openstack-discuss-request(a)lists.openstack.org <openstack-discuss-request(a)lists.openstack.org> Sent: Wednesday, November 1, 2023 7:26 AM To: openstack-discuss(a)lists.openstack.org Subject: openstack-discuss Digest, Vol 61, Issue 1 Send openstack-discuss mailing list submissions to openstack-discuss(a)lists.openstack.org To subscribe or unsubscribe via email, send a message with subject or body 'help' to openstack-discuss-request(a)lists.openstack.org You can reach the person managing the list at openstack-discuss-owner(a)lists.openstack.org When replying, please edit your Subject line so it is more specific than "Re: Contents of openstack-discuss digest..." Today's Topics: 1. Re: [ops] Seeking Advice on Service Deployment and Layout for an 8-Node Cluster (Forrest Fuqua) ---------------------------------------------------------------------- Message: 1 Date: Wed, 1 Nov 2023 02:26:02 +0000 From: Forrest Fuqua <fffics(a)rit.edu> Subject: Re: [ops] Seeking Advice on Service Deployment and Layout for an 8-Node Cluster To: Karl Kloppenborg <kkloppenborg(a)resetdata.com.au>, "openstack-discuss(a)lists.openstack.org" <openstack-discuss(a)lists.openstack.org> Message-ID: <CO6PR16MB420923931283EAF219118489D0A7A(a)CO6PR16MB4209.namp rd16.prod.outlook.com> Content-Type: multipart/alternative; boundary="_000_CO6PR16MB420923 931283EAF219118489D0A7ACO6PR16MB4209namp_" I am currently doing COW images in Ceph right now, that seems to be the same thing as volume backed images, I have about 40 images I use ( I even published the set here:http://mirrors.rit.edu/oszoo/ ) Ill take a look at Openstack Helm, I've never deployed Kerb before. ________________________________ From: Karl Kloppenborg <kkloppenborg(a)resetdata.com.au> Sent: Tuesday, October 31, 2023 8:03 PM To: Forrest Fuqua <fffics(a)rit.edu>; openstack-discuss(a)lists.openstack.org <openstack-discuss(a)lists.openstack.org> Subject: Re: [ops] Seeking Advice on Service Deployment and Layout for an 8-Node Cluster Hi Forrest, One of the first recommendations I would suggest is to utilise volume-backed-images. https://docs.openstack.org/cinder/latest/admin/volume-backed-image.html If you have say a set of linux and windows images, you can use this to instead of RBD imaging, ceph will snapshot from the base image the RBD and this can provision in seconds. I would also suggest (and this is completely biased because I’m on the core team) looking at OpenstackHelm – we run this at scale with many thousands of VM’s. Thanks, Karl. From: Forrest Fuqua <fffics(a)rit.edu> Date: Wednesday, 1 November 2023 at 2:15 am To: Karl Kloppenborg <kkloppenborg(a)resetdata.com.au>, openstack-discuss(a)lists.openstack.org <openstack-discuss(a)lists.openstack.org> Subject: Re: [ops] Seeking Advice on Service Deployment and Layout for an 8-Node Cluster In response to Karl's request, Here is last year's major load I ran: http://mirrors.rit.edu/cptc/2023/virtual-machines/1-README.txt<http://mirrors.rit.edu/cptc/2023/virtual-machines/1-README.txt> The TL;DR its a mix of Windows and Linux VMs at about 4GB of Ram, and 4vCPU at about 50 machines per team, and having up to 50 teams at a time, with all the extras support VMs at various workloads, we do about 2500 VMs just for the yearly compitations All of these VMs will be under high load since they will be under a consistent cyberattack by students. The biggest issues I've had is Ceph starts to slow down on RBD creation of a COW image and Nova will timeout the creation, If it gets too back, entries like volumes, and VMs can get stuck in weird error states of error_deleting or some other weird state (I've had Volumes stuck attached to VMs that don't exist!) At peak, Netdata was showing about 500kIOP/s across the whole cluster before it really started to go north and went into a death spiral, A single windows VM was a 4 hour boot time. Ceph started showing slow writes with up to 40s commit latency When I was running a multi-node management, the MariaDB and RabbitMQ would get super mad and start dropping messages and database issues, having a single dedicated compute node has worked well so far. but I am thinking of moving away from Kolla and going to Ansible during my summer rebuild. ________________________________ From: Karl Kloppenborg <kkloppenborg(a)resetdata.com.au> Sent: Monday, October 30, 2023 6:44 PM To: Forrest Fuqua <fffics(a)rit.edu>; openstack-discuss(a)lists.openstack.org <openstack-discuss(a)lists.openstack.org> Subject: Re: [ops] Seeking Advice on Service Deployment and Layout for an 8-Node Cluster Hi Forrest, Based on the configuration specified, I will assume that 1.8TB is assignable to VM resources, less the management node (though honestly why not just share over the cluster, doesn’t seem like it’s a production workload?) However, (1.8TB*7)*1000 = 12600GB usable ram, with no oversubscription. (roughly 180-200GB each system for host and ceph services) Therefore 12600/3100 = ~4GB ram per VM at peak load, no oversubscription. Based on the CPU arch: 128*2 threads = 256 vCPU cores (which in this model I would count as vCPU cores, there’s a LOT of contention about what defines a vCPU, there’s many camps and religious arguments on this)… However, we want some space for the host so as a general thought, I will put 250 vCPU available in this calculation. 250*7 = 1750 vCPU cores avail. There are 3100-1750 = 1,350 deficit vCPU cores. As such to make up a 1vCPU core to VM at peak workload an oversubscription would be 1.8 1750*1.8 = 3150 vCPU available. Honestly, depending on these workloads and your requirements you can probably bust this subscription ratio up more and again, I would instead share the management workloads over the cluster more and induct the first node as another compute resource. I would like to know more about the workloads, what is the flavour of each VM, Is the workload disk intensive, cpu or memory, or io? In terms of running the workload, I don’t see an obvious issue with your nova config, the allocation ratio is high but that’s not going to probably be an issue here based on the numbers I mentioned. Ensuring host reservation memory is good, which I can see you have around 128GB in reservation and ram_allocation_ratio as 1.0 will ensure no OOM scenarios. Overall, I think your settings are sane and my only comment would be to add that management node into the compute mix if you can but otherwise without knowing more on the workload, seems fine. Have you done any scale testing? Spun up a workload of 3100 VMs? ( easiest way to do this would likely be to heat template…) then run an image with a default start workload of something you want to test with. --Karl. From: Forrest Fuqua <fffics(a)rit.edu> Date: Tuesday, 31 October 2023 at 5:40 am To: openstack-discuss(a)lists.openstack.org <openstack-discuss(a)lists.openstack.org> Subject: [ops] Seeking Advice on Service Deployment and Layout for an 8-Node Cluster Hi there, I'm currently exploring some strategies and seeking advice on the deployment and layout of services on an 8-node cluster. Each node is equipped with dual AMD EYPC 7702s, two 100G Links, and 2TB of RAM. Additionally, I've set up two 13TB NVME drives in a Ceph configuration for each node. At present, the setup involves one management node, while the others are dedicated to running Nova, Neutron, Glance, and Cinder. The workload, while fairly basic, involves handling substantial numbers. It's primarily used for one-off research projects and a few classes, serving around 60 end users. The most demanding scenario occurs during cybersecurity competitions when we need to deploy 3100 VMs at once. When this is spread out over a few days, it's manageable. However, there's a risk of overloading the system, leading to DiskIO issues and Database inconsistencies. Currently, I'm utilizing Kolla for deployment, and I've shared my configurations on this GitHub repository: Link to Configs<https://github.com/RIT-GCI-CyberRange/openstack-configs>. I'd greatly appreciate any insights or advice on optimizing this setup to handle the peak workload more efficiently and prevent potential performance issues. Thanks in advance for your help!

1 0

OpenInfra Live: Meet the Computing Force Network Working Group Nov 2 at 9 AM (CST) / 14:00 UTC
by Kristin Barrientos 31 Oct '23

31 Oct '23

Hi everyone, This week’s OpenInfra Live episode is brought to you by the open source community. Episode: China Mobile and other 10+ organizations are driving the effort of Computing Force Network in open source. In July of 2022, organizations like, China Mobile launched the Computing Force Network (CFN) Working Group within the OpenInfra Foundation. You may be wondering, what is Computing Force Network? What open source software is CFN working group going to deliver and how are CFN and related technology going to help the infrastructure? Join us to find out. Date and time: Nov 2, 2023 at 9:00 AM CST (14:00 UTC) You can watch us live on: YouTube: https://www.youtube.com/watch?v=Slc-kOao3BY LinkedIn: https://www.linkedin.com/events/meetthecomputingforcenetworkwor711866369563… WeChat: The recording will be posted on OpenInfra Foundation WeChat after the live stream Speakers: Qihui Zhao, Jie Niu, Xiaoqiao Geng, Changjin Wang, Jintao Wang, Lei Huang and Ildiko Vancsa Thanks, Kristin Barrientos Marketing Coordinator OpenInfra Foundation

1 0

[ironic] IntelIPMIManagement driver: Anyone using it?
by Jay Faulkner 31 Oct '23

31 Oct '23

Hi all, Ironic currently has an Intel IPMI Management driver ( https://docs.openstack.org/ironic/latest/admin/drivers/intel-ipmi.html ), a mild enhancement over our generic IPMI driver, which adds the ability to select an Intel Speed Select Performance Profile via an Ironic automation step. Ironic cores have no knowledge of this driver being used by any deployers of Ironic. We are evaluating it for deprecation and eventual removal. If you are interested in this driver remaining in the Ironic codebase, please let us know. Please note: this is not the same as our generic IPMI driver, which supports power control, boot device management, and is not being considered anytime soon for deprecation :). If we decide to mark the driver for removal, it'll obviously remain in the codebase for a long deprecation period as is our norm. Thanks, Jay Faulkner Ironic PTL

1 0

[largescale-sig] Next meeting: Nov 1, 15utc
by Kristin Barrientos 31 Oct '23

31 Oct '23

Hi everyone, The Large Scale SIG will be meeting this Wednesday in #openstack-operators on OFTC IRC, at 15UTC, our EU+US-friendly time. You can doublecheck how that UTC time translates locally at: https://www.timeanddate.com/worldclock/fixedtime.html?iso=20231004T15 Feel free to add topics to the agenda: https://etherpad.opendev.org/p/large-scale-sig-meeting Thanks, Kristin Barrientos Marketing Coordinator OpenInfra Foundation

1 0

[blazar] Cancelling the next IRC meeting
by Pierre Riteau 31 Oct '23

31 Oct '23

Hello, I am cancelling the next Blazar IRC meeting due to being out of the office. The next meeting will be on November 16. Best wishes, Pierre Riteau (priteau)

1 0

[horizon] Cancelling Tomorrow Weekly meeting (1st Nov 2023)
by vishal manchanda 31 Oct '23

31 Oct '23

Hello Team, I won't be able to chair tomorrow's Horizon weekly meeting due to some urgent work at that time. If anything urgent, please reach out to me and the Horizon core team on IRC. Thanks & regards, Vishal Manchanda

1 0

[cinder] 2024.1 (Caracal) PTG Summary
by Rajat Dhasmana 31 Oct '23

31 Oct '23

Hello, Please find the summary of 2024.1 (Caracal) PTG in the wiki page[1]. It includes summary and action items for each topic. More details can be found in the PTG etherpad[2]. I would recommend everyone to please look for the action items assigned to you to plan it according to the cycle deadlines. [1] https://wiki.openstack.org/wiki/CinderCaracalPTGSummary [2] https://etherpad.opendev.org/p/caracal-ptg-cinder Thanks Rajat Dhasmana

1 0

Green threads
by Gk Gk 31 Oct '23

31 Oct '23

Hi All, This may be out of place for this question, but I was reading about this bug https://bugs.launchpad.net/oslo.messaging/+bug/2009138, which led to this query. Do green threads in python need GIL ( global interpreter lock) for execution ? Also, can green threads take advantage of multiple processors ? If yes/no, why ?

1 0

[qa] Office hour day and time change
by Martin Kopec 31 Oct '23

31 Oct '23

Hi everyone, as discussed during the PTG [1] we are changing the date and time of our office hours. The office hour will be taking place on *Wednesdays at 17:00 UTC* [2]. The change is effective *from November 8th*. The meeting agenda is available here [3]. I'm cancelling office hours on Tuesday October 31st and November 7th. [1] https://etherpad.opendev.org/p/oct2023-ptg-qa [2] https://review.opendev.org/c/opendev/irc-meetings/+/899457 [3] https://wiki.openstack.org/wiki/Meetings/QATeamMeeting Thanks, -- Martin Kopec Principal Software Quality Engineer Red Hat EMEA IM: kopecmartin

1 1

capacity planning question.
by KK CHN 30 Oct '23

30 Oct '23

Hi list, I am in need of an infrastructure set up for data analytics / live video stream analytics application using big data and analytics technology.. The data is basically right now stored as structured data(no video streaming) in PostgresDatabase. ( Its an emergency call handling solution, In database which stores, caller info (address, mobile number, locations co-ordinates, emergency category metadata and dispatch information regarding rescue vehicles., Rescue vehicle location update (lat, long)every 30 seconds all are stored in the Postgres Database .. Input1 : I have to do an analytics on these data( say 600 GB for the last 2 years its the size grown from initial setup).To perform an analytical application development( using python and data analytics libraries, and displaying the results and analytical predication through a dashboard application.) Query 1. How much resource in terms of compute(GPU?(CPU) cores required for this analytical application? and memory ? And any specific type of storage(in memory like redis required ? ) etc, which I have to provision for these kind of application processing. ?? any hints most welcome.. Any more input required let me know I can provide if available. Input 2 In addition to the above I have to do video analytics from bodyworn cameras by police personnel, drone surveillance Videos from any emergency sites, patrol vehicle (from a mobile tablet device over 5G )live streaming of indent locations for few minutes ( say 3 to 5 minutes live streaming for each incident. ) There are 50 drones, 500 Emergency rescue service vehicles, 300 body worn camera personnels.. and roughly 5000 incidents / emergency incidents per day happening, which needs video streaming for at least 1000 incidents for a time duration of 4 to 5 minutes live streaming. Query2. What/(how many) kind of computing resources GPUs(CPUs)? RAM, Storage solutions I have to deploy in numbers( or cores of GPUs/how many/(CPUs)? RAM ? In Memory (Redis or similar ) or any other specific data storage mechanisms ? Any hints much appreciated.. Thank in advance, Krishane

1 0

[tacker][ptg] Caracal vPTG Summary
by Yasufumi Ogawa 30 Oct '23

30 Oct '23

Hi all, We had two days session for discussing new proposals for Caracal and reamined items couldn't fixed in the previous release [1]. Here is a summary of the discussion in Tacker team. # Revise FTs Gate tests, especially for functional tests, have been getting mess because of increasing amount of codes without active maintenance. For the issue, we have focused on reducing useless logs. And also re-design directory structure and testcase scenarios. We're going to continue the tasks and will fix it in this release. # Legacy API deprecating/obsoleting It was proposed to drop Tacker's legacy implementation in antelope, and will be completed in this release as planed. # Introducing Cilium CNI for high-performance network connectivity for VNF/CNF This proposal is for a new example app using Cilium CNI as a vRAN usecase in which VNF/CNF requires performance for network connectivity between VMs or containers. The first target is to confirm it's available to deploy with Tacker in a manner of VNF package. # Update requirements for monitoring We have introduced Prometheus for implementing FM/PM interfaces defined in ETSI-NFV standards from Yoga. This proposal is for more advanced feature than monitoring, called as observability, by using OpenTelemetry for considering future requirements. We agreed to implement an additional plugin dedicating to interwork between Otel collector and Tacker. # NFV Orchestration API v2.0 in Tacker Horizon Although many of v2 APIs in ETSI specifications have been implemented, tacker-horizon is not ready for the features. We'll update NFVO interfaces in Horizon to provide v2 features. Current legacy features are going to be dropped instead. # Improving Tacker Document As legacy implementations have been deprecated, documents are also removed. In addition, there are some other sections should be updated such as incorrect usecase guides don'twork, unmaintained operator's guides or so. # Tacker Tempest Plugin Tacker doesn't support tempest, but it's not good for considering if it leads to OpenStack vendors not including Tacker in their products, and also makes Tacker look unhealthy project. So, implement Tacker's tempest plugin and update our zuul jobs to use the plugin for. # Fix vimConnectionID in ResouceHandle In the current implementation, vimConnectionID differs from ETSI specs mistakenly. So, it's should be fixed soon and expected to be small changes. # Secure RBAC This item has been led by NEC team and development of UTs and FTs are under implemention as noted on the vPTG etherpad. However, we need to update ownership of the item for manpreetk moved to another organization. Confirm if NTT or NEC team own the item in caracal and share the result in IRC meeting. # Fix issues which were encounter in compliance test execution In next release of tacker (2024.1), NEC team proposing to make 'tacker-compliance-devstack-multinode-sol' job testing mandatory and voting after some issues occured in the test are fixed. # Maintain tosca-parser and heat-translator Assign team leaders for each projs temporarily for governance, Toshiaki Takahashi for tosca-parser and Ayumu Ueha heat-translator. Ask TC memebers anything requried to change the ownership. [1] https://etherpad.opendev.org/p/oct2023-ptg-tacker Thanks, Yasufumi

1 0

[tc] Weekly Meeting Tomorrow, Oct 31 1800 UTC in OFTC
by Jay Faulkner 30 Oct '23

30 Oct '23

Hi all, The OpenStackTechnical Committee will have our weekly meeting tomorrow, October 31, at 1800 UTC in #openstack-tc on OFTC. All are welcome to attend. The proposed agenda is below. Agenda Suggestions - Roll call - Follow up on tracked action items: https://meetings.opendev.org/meetings/tc/2023/tc.2023-10-17-18.00.html - JayF: Before next video meeting, write up a short document on pros/cons of moving TC video meetings to jitsi-meet. - Gate health check - Leaderless projects (gmann) - PTG follow-ups? - https://etherpad.opendev.org/p/tc-ptg-october-2023 - https://etherpad.opendev.org/p/tc-leaders-interaction-2024-1 - https://etherpad.opendev.org/p/oct2023-ptg-os-dbperf-crossproject - Open Discussion and Reviews Thank you, Jay Faulkner OpenStack TC Chair

1 0

[barbican] Weekly Meeting 31-10-2023 Canceled
by Grzegorz Grasza 30 Oct '23

30 Oct '23

Hi all, I'm canceling the meeting this Tuesday. I'm on a business trip this week. Thanks! / Greg

1 0

Re: Regarding Nova's inability to delete the Cinder volume for creating virtual machines (version Y)
by 无数的星球 30 Oct '23

30 Oct '23

Thank you for Gorka's reply! I have seen this document multiple times and it has not been taken seriously. Next, I will try.

1 0

Regarding Nova's inability to delete the Cinder volume for creating virtual machines (version Y)
by 无数的星球 30 Oct '23

30 Oct '23

When creating a virtual machine in the dashboard, create a volume and choose to delete the virtual machine while also deleting the volume. When deleting the virtual machine, there is no normal uninstallation of the volume and the volume is not deleted. The relevant error logs are shown in the image, but the openstack CLI can delete its volume. The specific commands are as follows. CLI： source /etc/keystone/admin-openrc.sh (Verify password file) openstack volume set --detached　191e555c-3947-4928-be46-9f09e2190877（volumeID） openstack volume delete 191e555c-3947-4928-be46-9f09e2190877（volumeID） It seems that Nova is unable to interact with the Cinder API to delete(or detached) commands, but I am not very professional. I don't know if it's a bug? Note that I have seen similar bug patch files, but they are invalid. URL:   BUG page   https://bugs.launchpad.net/nova/+bug/2016173   Page for patching files   https://review.opendev.org/c/openstack/nova/+/880399 Is not working!!!

2 1

[openstack-ansible][ptg] OpenStack-Ansible 2024.1 (Caracal) vPTG Summary
by Dmitriy Rabotyagov 30 Oct '23

30 Oct '23

Hey everyone, We had quite a productive discussion on the PTG and I will provide a summary of them below. It was divided into multiple sections, like things we need to finish for 2023.2 (Bobcat) - since OpenStack-Ansible has trailing release model we have some time to wrap things up; plans for 2024.1 (Caracal) and some things that moved out of scope until better times. You can also find original PTG with discussions here [1] Finalizing 2023.1 (Bobcat): * Documentation on using pretty endpoint names (ie identity.cloud.org instead of cloud.org:5000). For 2024.1 we will try to simplify the way of using such naming to make it more convenient for end-users. * Finalize support of RabbitMQ Quorum queues for all services [2]. With that they will be disabled by default at least for 2023.2 release and we will revise the defaults for 2024.1 * Implement openstack_resources common rolem that will handle different resources creation, like OpenStack users, projects, networks, images, aggregates, etc. It will be also used in roles like Tempest, Octavia, Trove and Magnum to create such resources, instead of each role implementing its own flow of doing that. We agreed to add a separate playbook that will allow deployers to leverage the role and run it after setup-openstack. * Debian 12 support. Most patches have landed, but we also need to reflect it's support in documentation * Review and release Skyline role. Tough discussion was about defaults and how Skyline should co-exist with Horizon. As a result we decided to deploy both options, when both are defined in the inventory (as usual), but the playbook that is run last will basically configure HAProxy backends to itself, fully neglecting other available services. So if you want to have both Skyline and Horizon running and functional - you will need to apply some overrides for that. There are also no guarantees that we will be able to merge and release the role in time, so we may end up with making it as part of OSA only for 2024.1. Scoped improvements for 2024.1 (Caracal): * Adoption of PKI role to be used with PIPE modules, that will allow the role to be pointed to any file (like encrypted with ansible-vault) or external storage that supports lookup, instead of plain text files on a deploy host. With that we will also take a look at using Hishcorp Vault as a PKI source and adding it as another driver in addition to the "standalone" one. * Natively supporting multi-AZ deployments for Neutron. At the moment for OVS/LXB you need to define some overrides to make that working, while for OVN you would need to manually execute a bunch of commands. POC patch has been already proposed [3], but it still needs some love. * Finalize support of OVN driver for Octavia. Initial patch has been proposed, but it also needs some love, despite it works in general [4] * Resume work on ProxySQL implementation as a replacement for HAProxy balancing for MariaDB deployment. That should improve load distribution between MariaDB backends. * Allow to select OS for LXC containers. At the moment we're forcing to use the host OS, while in some cases it might be preferred to use a different one, like during OS upgrades or switching the OpenStack deployment from one distro to another. * With that we are excited about incus (LXD fork) having the first release and it being packaged with traditional packages (only .deb at the moment). This opens the door for implementation of incus as a manager for LXC containers, that may simplify our flows and day2 operations significantly. * Invest some time into decreasing the runtime, both for CI/CD and for end-user deployments. * In case improvements for Quorum Queues will land to the oslo.messaging [5] - add a set of variables to roles for nicely controlling the behaviour. [1] https://etherpad.opendev.org/p/osa-caracal-ptg [2] https://review.opendev.org/q/topic:osa%252Fquorum_queues [3] https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/858271 [4] https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/868462 [5] https://review.opendev.org/q/topic:bug-2031497

1 0

[Glance] Caracal PTG Summary
by Pranali Deore 30 Oct '23

30 Oct '23

Hello Everyone, We had our eighth virtual PTG between 23rd OCt to 26th OCT 2023. Thanks to everyone who joined the virtual PTG sessions. Using Opendev Meetpad we had all the discussion around different topics for glance. I have created etherpad[1] with notes from the session and also included the recordings of each session. Here is the short summary of the discussions. *Tuesday , OCT 24th 2023* # Bobcat Retrospective On a positive note we have completed few important things like adding support for weighting mechanism to store, support for RBD driver to move images to the trash if they cannot be deleted immediately due to having snapshots, putting efforts in bridging the gap between glanceclient and openstackclient and started new location API work which will replace the image-update mechanism for consumers like cinder and nova for OSSN-0090 and OSSN-0065. On the other side we decided to speedup the work we've had in flight and we decided to have recent patches reviews meeting once in a month and backlogs review parties before each milestone to speedup the reviews. Recording: https://drive.google.com/drive/u/0/folders/1oYrgVLV5L89keDZLx7r1JWcG8pk65918 - Bobcat Retrospective # Common Tempest Failures for Glance Glance has faced so many random CI failure and timeouts since bobcat mid cycle, though progress has been made on general issues and few of the project-specific issue also been addressed like post failures causing by image conversion failure, tried reducing concurrency but still we see some random timeouts, one of the common tempest failure has been observed that some glance related jobs like glance-multistore-cinder-import, nova-ceph-multistore are failing with race condition while saving image data, there is a short term solution proposed by sending image_id while creating the image to ensure that each test can utilize own image and race condition could be avoided. https://docs.google.com/document/d/1S41X3FSjqGF87ehTvqLEC9QsDIkULjS1NiqCagZ… Decided to reproduce 1st failure from the gdoc and debug more on failures Recording: https://drive.google.com/drive/u/0/folders/1oYrgVLV5L89keDZLx7r1JWcG8pk65918 - Common tempest failures # Add missing CLI support for some Glance API in Openstack Client The migration work for adding missing CLI was initiated in the antelope cycle. We have added the SDK support for cache API , images API and metadef API(in progress). On the client side support for cache API and images API has been added and for metadef is in progress. This cycle we have decided to cover the remaining client and sdk support for APIs by distributing the work among the team. Recordings: https://drive.google.com/drive/u/0/folders/1oYrgVLV5L89keDZLx7r1JWcG8pk65918 - Adding missing OSC support # Default Glance to configure multiple stores Glance has deprecated single stores configuration since Stein cycle and will remove the support of same in upcoming cycle by dividing the wrok in small parts like remove/migrate single store unit/functional tests. Goal for this cycle: list down the work we have to do with regards to tests. and migrate unit/functional test from "single-store only" to "multistore-only" and send email to the mailing list to urge users to migrate to multi-store. Recording: https://drive.google.com/drive/u/0/folders/1oYrgVLV5L89keDZLx7r1JWcG8pk65918 - Default glane to multistore support.webm *Wednesday, OCT 25th 2023* # Remove Location Strategy In Bobcat we added support for the weighing mechanism to store. So we can safely remove the location strategy and by default use the latest weighing mechanism which will be very easy and efficient. Decided to deprecate location strategy in C and remove it in D, notify operators and users on ML and have a spec for the same in Caracal Cycle. Recording: https://drive.google.com/drive/u/0/folders/1oYrgVLV5L89keDZLx7r1JWcG8pk65918 - Remove Location Strategy # Improve process of restoring image to active state which is set as 'pending_delete' At the moment glance supports delaying of image-data from the store by setting delayed_delete to True and when image gets deleted it actually keeps image data in the backend and then Scrubber daemon deletes it permanently later. To restore the image, the deployer needs to set the UUID of that image in glance-api.conf to the 'restore' option under the Default section. If there are multiple images that need to be restored then this process will be kind of time consuming as each time a new Id will be needed to set in the configuration file and rerun the scrubber if it is not running as Daemon. Decided to introduce admin only API to restore the image and restrict Scrubber daemon to delete the images which are in pending_delete state. The restore part from the scrubber for a couple of cycles until the new API is stable.The restore option from the scrubber will be deprecated in this cycle and then will be removed in the following cycle. Recording: https://drive.google.com/drive/u/0/folders/1oYrgVLV5L89keDZLx7r1JWcG8pk65918 - Restore Image # Deprecate cachemanage middleware Cache operations (CR(u)D) is now part of glance API. Even though we need to depend on cache/cachemanage middleware to enable it. There are two middlewares related to caching, one is `cache` and other is `cachemanage` and we can now get rid of `cachemanage` middleware and glance-cache-manage command line utility. Agreed on deprecating cachmanage middleware in C & remove in D cycle. Decided to have a spec lite for the same after verifying how the cache APIs will respond if cache middleware is not enabled. Recording: https://drive.google.com/drive/u/0/folders/1oYrgVLV5L89keDZLx7r1JWcG8pk65918 - Deprecate Cachemanage Middleware # Use mysql database for caching instead of sqlite There are two different database(s) used for glance, MySql for regular operations and sqlite for caching related operations. We have decided to use single database i.e. MySql for all glance related operations so that it will reduce the efforts to maintain both databases. The Spec is up for review now: https://review.opendev.org/c/openstack/glance-specs/+/899367/4 Recording: https://drive.google.com/drive/u/0/folders/1oYrgVLV5L89keDZLx7r1JWcG8pk65918 - Centralize DB for Caching *Thursday, OCT 26th 2023* # Unified limits Topic was discussed and decided to postpone it to the next cycle since no one volunteered to work on it. # New Location APIs We have decided to have support of new location api which will replace the image-update mechanism for consumers like cinder and nova for OSSN-0090 and OSSN-0065. The implementation patches are up for review along with the unit/functional tests, documentation, api version bump, client side support and tempest coverage[2] The Poc patches of nova and cinder are also there to test this new location api functionality. In this cycle we are going to work on some missing test scenarios & verify PoC patches of consumers like nova and cinder are working with new location api functionality. Recording: https://drive.google.com/drive/u/0/folders/1oYrgVLV5L89keDZLx7r1JWcG8pk65918 - New Location API Apart from above topics you can find other miscellaneous topics discussed in PTG etherpad [1] along with the recordings of the sessions and milestone wise priorities at the end. If you have any questions/suggestions then please join us in our weekly meeting(each Thursday on #openstack-meeting irc channel at 1400 UTC) [1]: https://etherpad.opendev.org/p/oct2023-ptg-glance#L10 [2]: https://review.opendev.org/q/topic:New-Location-Apis Thanks & Regards, Pranali Deore

1 0

[qa][ptg] QA Caracal vPTG Summary
by Martin Kopec 30 Oct '23

30 Oct '23

Hi everyone, here is a summary of the discussions. The full agenda can be found here [1]. == Bobcat Retrospective == Considering the still shrinking number of active contributors, we still manage making releases and doing code reviews. However, less contributors make it hard to react to sudden failures or provide as stable jobs as possible - that's e.g. the case of the Centos 9 topic. == Stop supporting Centos 9 Stream? == No, Centos 9 stays. As concluded during the session, Centos 9 is required for FIPS as other distros (their latest versions) are far behind with supporting FIPS. Apart from that, CS9 is required for Manila, Cinder integration with Ceph as Ceph is tested on Centos 9 too. The biggest pain point is that Centos9 Stream is not versioned and thus it's hard to build a reliable gate. One of the goals for the cycle is improving the connection between Centos and OpenStack community - it's beneficial for both sides. One of the ways might be a 3rd party CI or simply a solution that would provide a quick feedback to the Centos group, e.g. by running Devstack with the default configuration, but at the same time wouldn't affect all the jobs on the OpenStack side that run the distro. This way we could at least avoid simple issues found right away (e.g. some deps issues). The QA team is happy to assist. == Triage previous and write new priority items == The list of the priority items planned for this cycle as well as the ones which were carried over from the previous cycles, can be found here [2]. [1] https://etherpad.opendev.org/p/oct2023-ptg-qa [2] https://etherpad.opendev.org/p/qa-caracal-priority Thank you all for attending the PTG and contributing to the discussion, -- Martin Kopec Principal Software Quality Engineer Red Hat EMEA IM: kopecmartin

1 0

[interop][ptg] Interop Caracal vPTG Summary
by Martin Kopec 30 Oct '23

30 Oct '23

Hi everyone, here is a summary of the discussions. The full agenda can be found here [1]. We had only one topic on the agenda - a new view on interoperability testing - don't confuse it with interop which is a certification program. The program just uses the interop tests for the certification purposes. The tests that are part of interop were long ago considered as enough for testing interoperability. However, as it's been pointed out as well as discussed many times already [2][3], the tests don't guarantee interoperability of tested clouds - they test only basic API thus they can be used for the certification program (related to OpenStack brand) at most. Interoperability is subjective, every user may be looking for a different level of interoperability. Read the ending of the article [2]. If there is someone who is interested in interoperability testing in OpenStack or even someone who requires that for various reasons, we could do something like this [4] - we could gather various tests from various testing platforms that test specific functionalities. It would provide anyone with metrics and thus allow them to compare clouds. It would be like when we're shopping, e.g. for a new computer - we have various metrics that help us decide which one suits us better like dimensions, weight, RAM, CPU, storage space etc. From the last PTG (and several one before) it, however, doesn't look like the community is interested in running actual interoperability testing. Let us know if you disagree, we don't want to develop something that is not needed by anyone. [1] https://etherpad.opendev.org/p/oct2023-ptg-interop [2] https://superuser.openinfra.dev/articles/new-view-on-interoperability-in-op… [3] https://etherpad.opendev.org/p/bobcat-ptg-interop [4] https://etherpad.opendev.org/p/openstack-interop-2.0 Thank you all for attending the PTG and contributing to the discussion, -- Martin Kopec Principal Software Quality Engineer Red Hat EMEA IM: kopecmartin

1 0

[neutron] Bugs deputy report October 23rd - 29th
by Miguel Lavalle 29 Oct '23

29 Oct '23

Hi Neutrinos, Here's this week's bugs deputy report: Critical ====== High ==== https://bugs.launchpad.net/neutron/+bug/2040166 [OVN] The neutron_sync_mode conf is ignored for SB DB sync. Assignee: Lucas Gomes. https://bugs.launchpad.net/neutron/+bug/2040172 [OVN] OvnSbSynchronizer - clean/delete segmenthostmappings for unrelated hosts. Assignee: Harald Jensas. Proposed fix: https://review.opendev.org/c/openstack/neutron/+/899077 https://bugs.launchpad.net/neutron/+bug/2041609 FIP update removes QoS policy. Assignee Lajos Katona. Proposed fix: https://review.opendev.org/c/openstack/neutron/+/899469 Medium ====== https://bugs.launchpad.net/neutron/+bug/2040517 host not removed from table ml2_vxlan_endpoints with the agent delete. Needs and owner https://bugs.launchpad.net/neutron/+bug/2041612 [n-d-r] DRAgent removed from bgp speaker automatically. Needs verification and owner Low === RFEs ==== https://bugs.launchpad.net/neutron/+bug/2040001 [N-D-R] Change neighbor connect mode when the BGP peer does not support passive mode. Assignee: Tiago Pires. This RFE was initially discussed during this week's vPTG. Proposed spec: https://review.opendev.org/c/openstack/neutron-specs/+/899210 Undecided ======== Invalid ======== https://bugs.launchpad.net/neutron/+bug/2040242 [ipam] Why prevent first ip < (subnet.first + 1) if version of subnet is ipv6. Assignee Liu Xie. Proposed fix: https://review.opendev.org/c/openstack/neutron/+/899374 https://bugs.launchpad.net/neutron/+bug/2040457 [allocation_pools] Need re-generation new pools from cidr and gateway-ip after update one subnet with empty 'allocation_pools'. Assignee: Liu Xie. Proposed fix: https://review.opendev.org/c/openstack/neutron/+/899373

1 0

Regarding Nova's inability to delete the Cinder volume for creating virtual machines (version Y)
by 无数的星球 28 Oct '23

28 Oct '23

When creating a virtual machine in the dashboard, create a volume and choose to delete the virtual machine while also deleting the volume. When deleting the virtual machine, there is no normal uninstallation of the volume and the volume is not deleted. The relevant error logs are shown in the image, but the openstack CLI can delete its volume. The specific commands are as follows. CLI： source /etc/keystone/admin-openrc.sh (Verify password file) openstack volume set --detached　191e555c-3947-4928-be46-9f09e2190877（volumeID） openstack volume delete 191e555c-3947-4928-be46-9f09e2190877（volumeID） It seems that Nova is unable to interact with the Cinder API to delete(or detached) commands, but I am not very professional. I don't know if it's a bug?

1 0

horizon - dashboard behavior - shows elements from previous project
by Miguel Casique Meneses 27 Oct '23

27 Oct '23

There is a horizon chart behaving quite strangely version: horizon-0.3.2 To recreate the issue: login to openstack >> change project (using dropdown) >> show containers (odd behaviour, such as change to old project, or lists containers from previous project) >> show instances (odd behaviour, changes projects, or at least shows the wrong project in dropdown) Basically, it sometimes shows the elements from the previous browsed project and not the current one that has been selected. If we reload the page, it goes back to project that has been selected, but sometimes it does not The logs are not showing nothing relevant, but we can see when project has been swapped

3 2

[Glance] Caracal PTG Updates
by Pranali Deore 27 Oct '23

27 Oct '23

Hello Everyone, We have concluded Glance Caracal vPTG yesterday. I will share the summary early next week. You can find tentative milestone wise priorities for glance in PTG etherpad[1]. Join us in the weekly meeting on Thursday if you have any doubts/suggestions. Thanks & Regards, Pranali [1]: https://etherpad.opendev.org/p/oct2023-ptg-glance#L277

1 0

[keystone] Weekly Meeting 11/1 Canceled
by Dave Wilde 27 Oct '23

27 Oct '23

Hello all, I'm on PTO the week of 11/1 so I'm canceling the Weekly meeting as well as the reviewathon on 11/3. Thanks! /Dave

1 0

[vPTG][Security SIG] Agenda / Planning etherpad
by Tony Breeds 26 Oct '23

26 Oct '23

Hello All, I'ms sorry for getting this out with so little notice. I've pulled together an agenda for the os-security SIG meeting at the vPTG When: https://www.timeanddate.com/worldclock/fixedtime.html?iso=2023-10-27T16:00:… Where: https://meetpad.opendev.org/oct2023-ptg-os-security Planning Etherpad: https://etherpad.opendev.org/p/oct2023-ptg-os-security Agenda: * Previous PTGs carryover ** https://etherpad.opendev.org/p/vancouver-june2023-os-security-sig *** Xstatic --https://review.opendev.org/c/openstack/xstatic-jquery-migrate/+/878923 ** https://etherpad.opendev.org/p/march2023-ptg-os-security *** image encryption pop-up team (Luzi) **** castellan consumers api is in place now, unblocking significant parts of the implementation * Should we meet outside of the PTG events [10 mins] * https://security.openstack.org/ Updates ** Sorting of OSSNs -- Something along the lines of -- https://review.opendev.org/c/openstack/releases/+/887691 ** Remove references to storyboard? * Other business? * VMT Business * Open public security bug triage/processing -- http://tiny.cc/public-ossn Yours Tony.

1 0