- openstack-discuss - lists.openstack.org

[nova][cinder] future of rebuild without reimaging
by Bence Romsics 06 Apr '23

06 Apr '23

Hi All! We have users who use 'rebuild' on volume booted servers before nova microversion 2.93, relying on the behavior that it keeps the volume as is. And they would like to keep doing this even after the openstack distro moves to a(n at least) zed base (sometime in the future). As a naive user, it seems to me both behaviors make sense. I can easily imagine use cases for rebuild with and without reimaging. However since the implementation of https://specs.openstack.org/openstack/nova-specs/specs/zed/implemented/volu… rebuild without reimaging is only possible using an old microversion (<2.93). With that change merged, rebuild without reimaging seems to be a somewhat less than fully supported feature. A few examples of what I mean by that: First, there's this warning: https://opendev.org/openstack/python-openstackclient/src/commit/5eb89e4ca1c… In which it is unclear to me what exactly will become an error in a future release. Rebuild with a different image? Or any rebuild with microversion <2.93? Then old nova microversions may get dropped. Though what I heard from nova folks, this is unlikely to happen. Then there are a few hypothetical situations like: a) Rebuild gets a new api feature (in a new microversion) which can never be combined with the do-not-reimage behavior. b) Rebuild may have a bug, whose fix requires a microversion bump. This again can never be combined with the old behavior. What do you think, are these concerns purely theoretical or real? If we would like to keep having rebuild without reimaging, can we rely on the old microversion indefinitely? Alternatively shall we propose and implement a nova spec to explicitly expose the choice in the rebuild api (just to express the idea: osc server rebuild --reimage|--no-reimage)? If the topic is worth further discussion beyond the ML, I can also bring it to the nova ptg. Thanks in advance, Bence Romsics (rubasov) ps: I'll be afk for a few days, but I'll follow up next Tuesday.

7 16

[qa][ptg] QA Bobcat vPTG summary
by Martin Kopec 06 Apr '23

06 Apr '23

Hi everyone, here is a summary of the discussions. The full agenda can be found here [1]. == Antelope Retrospective == Good things: kudos to all who are still around and help with reviews, patches, bug fixes Bad things: unstable gate due to tox update, timeouts, job flakiness (mainly multinode job) == uWSGI is in the maintenance mode == This doesn't require an immediate action as it's still in the maintenance mode = any bugs should be fixed and until we need new features we don't have to rush into replacing it. We should have it on the watchlist though and potentially test some alternatives - I've put this as one of the priority items for this cycle [2]. == FIPS interop testing == We had a FIPS goal since the last PTG [3]. The patches [4] and [5] were to have FIPS on ubuntu and these [6] and [7] are to add FIPS on a centos job. The next goal is to have a FIPS job on Rocky linux as it's more stable than centos. == Retiring Patrole == We are going to retire Patrole project. 2 main facts that speak in favor of retiring: Fact 1: plugins have enough RBAC tests (and are writing more to test SRBAC) - therefore patrole is not needed, the testing coverage got by the plugins is enough. Fact 2: patrole gates have been broken for a long time and no one was affected by that. == Additional PyPi maintainer cleanup == Only 2 projects from QA left with additional maintainers: * openstack/os-performance-tools - additional maintainer SpamapS (Clint Byrum) * openstack/patrole - additional maintainer DavidPurcell If you know any of them, please, tell them to reach out to me. I've sent emails to them, but maybe the email addresses I found aren't checked regularly anymore. == Tempest/Devstack gate stability == Several precautions have been done such as reduce mysql memory [8] or reduce the number of tests executed per job, increase of a timeout etc [9]. However, those precautions are mainly workarounds. The important note is that the instability was only highlighted by our testing, not caused by it. Many problems are inefficiencies in openstack software, e.g.: * openstack-client issues a new token for every operation * python startup time - entrypoint lookup * privesp processes in aggregate use more memory than nova (every service runs their own copy) * inefficient ansible code, etc Any volunteers who would like to first analyse the situation, find which of those inefficiencies are the biggest culprits and how big effect they have on the system? [1] https://etherpad.opendev.org/p/qa-bobcat-ptg [2] https://etherpad.opendev.org/p/qa-bobcat-priority [3] https://etherpad.opendev.org/p/antelope-ptg-interop [4] https://review.opendev.org/c/zuul/zuul-jobs/+/866881 [5] https://review.opendev.org/c/zuul/zuul-jobs/+/873893 [6] https://review.opendev.org/c/openstack/devstack/+/871606 [7] https://review.opendev.org/c/openstack/tempest/+/873697 [8] https://review.opendev.org/c/openstack/devstack/+/873646 [9] https://review.opendev.org/q/topic:bug%252F2004780 Regards, -- Martin Kopec Principal Software Quality Engineer Red Hat EMEA IM: kopecmartin

1 0

[cinder] 2023.2 Bobcat PTG Summary
by Rajat Dhasmana 06 Apr '23

06 Apr '23

Hello, Please find the summary of 2023.2 Bobcat PTG in the wiki page[1]. It includes recordings, summary and action items for each topic. More details can be found in the PTG etherpad[2]. I would recommend everyone to please look for the action items assigned to you to plan it according to the cycle deadlines. [1] https://wiki.openstack.org/wiki/CinderBobcatPTGSummary [2] https://etherpad.opendev.org/p/bobcat-ptg-cinder Thanks Rajat Dhasmana

1 0

[cinder]multi backend SAN and Ceph
by Nguyễn Hữu Khôi 06 Apr '23

06 Apr '23

Hello. I have a question that could we use both SAN and Ceph for multi backend? If yes can we change volume type from SAN to Ceph and otherwise? Thanks.

3 3

[Glance] Weekly Meeting Cancelled
by Pranali Deore 05 Apr '23

05 Apr '23

Hi All, Canceling today's glance weekly meeting, I'm on sick leave today. See you next week ! Thanks, Pranali D

1 0

[cinder]cannot lauch instance with cinder volume
by Nguyễn Hữu Khôi 05 Apr '23

05 Apr '23

Hello guys. I am using kolla-ansible yoga to deploy an openstack cluster, but when I launch an instance with volume, it has errors. but I can create volume first and boot from it. I am using Powerstore 5000Tb. Is it a bug? 672 ERROR oslo_messaging.rpc.server [req-1a822ccd-1d3a-4f9a-a453-22d04f07f816 47cefefdadbe4bb584d8cf983de71eb5 708d7af8178143aab263d3942a124ec8 - - -] Exception during message handling: cinder.exception.ImageCopyFailure: Failed to copy image to volume: Bad or unexpected response from the storage volume backend API: Unable to fetch connection information from backend: Bad or unexpected response from the storage volume backend API: Failed to create PowerStore host . 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/volume/driver.py", line 1045, in _attach_volume 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server conn = self.initialize_connection(volume, properties) 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/volume/drivers/dell_emc/powerstore/driver.py", line 153, in initialize_connection 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server return self.adapter.initialize_connection(volume, connector, **kwargs) 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/volume/drivers/dell_emc/powerstore/adapter.py", line 272, in initialize_connection 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server connection_properties = self._connect_volume(volume, connector) 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/volume/drivers/dell_emc/powerstore/adapter.py", line 613, in _connect_volume 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server chap_credentials, volume_lun = self._create_host_and_attach( 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/volume/drivers/dell_emc/powerstore/adapter.py", line 602, in _create_host_and_attach 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server host, chap_credentials = self._create_host_if_not_exist(connector) 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/decorator.py", line 232, in fun 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/coordination.py", line 195, in _synchronized 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server return f(*a, **k) 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/volume/drivers/dell_emc/powerstore/adapter.py", line 446, in _create_host_if_not_exist 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server host = self.client.create_host(host_name, ports) 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/volume/drivers/dell_emc/powerstore/client.py", line 302, in create_host 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server raise exception.VolumeBackendAPIException(data=msg) 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server cinder.exception.VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Failed to create PowerStore host . Nguyen Huu Khoi

1 1

[kolla-ansible][yoga][neutron][neutron-dhcp]ERRORS.... Need to understand
by Franck VEDEL 05 Apr '23

05 Apr '23

Hello. I come back to you to try to understand my problem. My installation is based on kolla-ansible, Yoga, Centos-Stream. For some time, without understanding why, neutron_dhcp no longer works on external networks and the networks of my students' projects. This generates many error messages, which are difficult to understand. Could this be a bug in this version? Before, it worked very well, but I have the impression that when the number of network increases (i’m not sure), from a moment, dhcp does not work anymore. What is the way to find the problem? Here are examples of errors that won't stop even if no one creates more instances of them 2023-04-04 20:36:52.870 7 ERROR neutron.agent.dhcp.agent device, namespace, scope, broadcast) 2023-04-04 20:36:52.870 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 272, in _wrap 2023-04-04 20:36:52.870 7 ERROR neutron.agent.dhcp.agent r_call_timeout) 2023-04-04 20:36:52.870 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 215, in remote_call 2023-04-04 20:36:52.870 7 ERROR neutron.agent.dhcp.agent raise exc_type(*result[2]) 2023-04-04 20:36:52.870 7 ERROR neutron.agent.dhcp.agent neutron.privileged.agent.linux.ip_lib.InvalidArgument: Invalid parameter/value used on interface tap73cd30e2-35, namespace qdhcp-395f1edb-de6d-4dd8-b31b-69a53780c012. 2023-04-04 20:36:52.870 7 ERROR neutron.agent.dhcp.agent 2023-04-04 20:36:52.871 7 INFO neutron.agent.dhcp.agent [-] Finished network 395f1edb-de6d-4dd8-b31b-69a53780c012 dhcp configuration 2023-04-04 20:36:52.872 7 INFO neutron.agent.dhcp.agent [-] Starting network 8db375e0-28d2-4379-9330-397dc9b4f2b1 dhcp configuration 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent [-] Unable to enable dhcp for 8db375e0-28d2-4379-9330-397dc9b4f2b1.: neutron.privileged.agent.linux.ip_lib.InvalidArgument: Invalid parameter/value used on interface tap1c220d25-b9, namespace qdhcp-8db375e0-28d2-4379-9330-397dc9b4f2b1. 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent Traceback (most recent call last): 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/dhcp/agent.py", line 218, in call_driver 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent rv = getattr(driver, action)(**action_kwargs) 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/dhcp.py", line 274, in enable 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent common_utils.wait_until_true(self._enable, timeout=300) 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/common/utils.py", line 717, in wait_until_true 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent while not predicate(): 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/dhcp.py", line 286, in _enable 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent interface_name = self.device_manager.setup(self.network) 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/dhcp.py", line 1779, in setup 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent namespace=network.namespace) 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/interface.py", line 152, in init_l3 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent device.addr.add(ip_cidr) 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 537, in add 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent add_broadcast) 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 823, in add_ip_address 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent device, namespace, scope, broadcast) 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 272, in _wrap 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent r_call_timeout) 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 215, in remote_call 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent raise exc_type(*result[2]) 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent neutron.privileged.agent.linux.ip_lib.InvalidArgument: Invalid parameter/value used on interface tap1c220d25-b9, namespace qdhcp-8db375e0-28d2-4379-9330-397dc9b4f2b1. 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent Thanks a lot Franck

2 4

[ironic][ptg] 2023.2 Work Items
by Jay Faulkner 05 Apr '23

05 Apr '23

Hey all, As usual, I documented items we talked about working on in Bobcat in a spec. Please go have a look and review it: https://review.opendev.org/c/openstack/ironic-specs/+/879381 This will hopefully be published very soon at https://specs.openstack.org/openstack/ironic-specs/#themes as our 2023.2 work items. Thanks everyone for participating in our vPTG this cycle! -- Jay Faulkner Ironic PTL

1 0

[tripleo] hold rechecks content provider gate blocker
by Marios Andreou 05 Apr '23

05 Apr '23

Hello tripleo we have a gate blocker at [1] hitting the centos-9-content-provider jobs. The tripleo CI team is tracking this - please hold rechecks until it we have a resolution or workaround regards, marios [1] https://bugs.launchpad.net/tripleo/+bug/2015309

1 1

[kolla] weekly meeting cancelled today
by Michał Nasiadka 05 Apr '23

05 Apr '23

Hello Koalas, Weekly meeting cancelled today. Please note that our weekly meeting time for next week and onwards has changed to 13:00 UTC due to DST changes. Best regards, Michal -- Michał Nasiadka mnasiadka(a)gmail.com

1 0

[neutron][OpenStack-ansible] Performance issues with trunk ports
by John Bartelme 05 Apr '23

05 Apr '23

Hello, I'm currently experiencing some pretty severe performance issues with my openstack-ansible deployed cluster(yoga) while deploying trunk ports and I'm looking for some help determining what might be the cause of this poor performance. In my simplest case I'm deploying 2 servers each with one trunk port each. The first trunk has 2 subports and the second 6 subports. Both servers also have 3 other regular ports. When deploying the first trunk port its subports are often provisioned quickly and the second trunk port takes anywhere from 30 seconds to 18 minutes. This happens even when I isolate neutron-server to a single physical machine with 44(88 threads) and 256GB ram. Further diagnosis has shown me some things i didn't quite understand. My deployment with OpenStack-ansible deploys neutron-server with 16 uWSGI processes and neutron-rpc-server with 16 rpc workers. However the way that the trunk RPC server is implemented it is only run on the parent RPC thread and instead runs in all of the uWSGI processes as well. This means that most of my trunk RPC calls are being handled by the uWSGI instead of the RPC workers. When the parent RPC thread handles the trunk port creation calls I constantly see creation times of 1-1.5 seconds. I've isolated it so that this thread does all of the trunk RPC calls and this works quite well but this doesn't seem ideal. What could be causing such poor performance in the uWSGI side of the house? I'm having a really hard time getting a good feeling for what might be slowing it down so much. I'm wondering if it could be green thread preemption but I really don't know. I've tried setting 'enable-threads' false for uWSGI but I don't think that is improving performance. Putting the profiled decorator on update_subport_bindings shows different places taking longer every time, but in general a lot of time(tottime, i.e. not subfunction time) spent in webob/dec.py(__call__), paste/urlmap.py(__call__), webob/request.py(call_application),webob/request.py(send). What else can I do to try and look for why this is taking so long? As a side question it seems counterintuitive that the uWSGI handles most of the trunk RPC calls and not the RPC workers? A couple other notes about my environment that could indicate my challenges: I had to disable rabbitmq heartbeats for neutron as they kept not getting sent reliably and connections were terminated. I tried with heartbeat_in_pthread both true and false but still had issues. It looks like nova also sometimes experiences this but not near as often. I was overzealous with my vxlan ranges in my first configuration and gave it a range of 10,000,000 not realizing that would create that many rows in the database. Looking into that I saw that pymysql in my cluster takes 3.5 minutes to retrieve those rows. mysql CLI only takes 4 seconds. Perhaps that is just the overhead of pymysql? I've greatly scaled down the vxlan range now. I'm provisioning the 2 servers with a heat template that contains around 200 custom resources. For 198 of the resources they are set to conditionally not create any OpenStack native resources. Deploying this template of mostly no-op resources still takes about 3 minutes. Horizon works but almost every page load take a few seconds to load. I'm not sure if that is normal or not. Thanks for any help anyone can provide. john

4 8

Does Openstack support the notion of cinder backend in maintenance mode?
by wodel youchi 05 Apr '23

05 Apr '23

Hi, When you have multiple cinder backends, and you need to take down one of them for maintenance purposes, what is the best way to proceed? Does Openstack have the notion of : cinder backend in maintenance mode? Regards.

3 6

[cinder] 2023.2 Bobcat Meeting Etherpad
by Rajat Dhasmana 04 Apr '23

04 Apr '23

Hello Argonauts, As we're done with the 2023.2 Bobcat PTG, we will be moving to a new etherpad for cinder meetings for the 2023.2 Bobcat development cycle. Please add topics (if you have any) to the etherpad for the meeting on 05 April, 2023 (Today). Etherpad: https://etherpad.opendev.org/p/cinder-bobcat-meetings Thanks Rajat Dhasmana

1 0

[manila] Bobcat vPTG summary
by Carlos Silva 04 Apr '23

04 Apr '23

Hello, Zorillas and interested stackers! Thank you for attending last week's PTG. We had productive discussions and came up with good plans for this cycle! Here is a summary of the topics we spoke about during the PTG. The etherpad with all the notes is available in [0]. If you would like to see the recordings for those discussions, they are available on the Manila Youtube channel [1]. - *Share Transfer Future development* - Share transfers feature was introduced in Antelope but it has a couple of possible enhancements, such as: transferring shares that have snapshots, replicated shares, share networks alongside its share servers and shares. - Snapshots shouldn't be very difficult to cover but we would need to keep in mind quota allocations while doing so. - Transferring shares with replicas is the bigger challenge, considering we need to work with modifying the access rules. - The next step would be to work on transferring share networks. - AIs: Haixin plans to work on this feature during Bobcat. - *S-RBAC changes in Bobcat* - For the Bobcat cycle, we will focus on: - Ensure we are landing even more tests and enabling the oslo policy defaults. - Modify the testing we are doing to use the reader role to test read-only API, opposed to the member role. - Audit all cross service interactions and isolate actions performed by the "service" user. - Push for the completion of unit tests for Secure RBAC - AIs: Manila team will continue promoting events to get tests reviewed and implemented - *Cross project discussion with Nova for VirtIO FS * - Manila and Nova team discussed approaches for avoiding shares to be deleted while they are attached to instances. - We have agreed that an admin/service level API for locking/unlocking shares deletion would be the best solution to avoid shares being deleted while mounted. This API would ideally be a 'locked by <consumer>' action instead of a simple boolean - As part of this cross-project effort, the Manila team has also been focusing on getting more APIs available in OpenStack SDK. We have been making great progress with it and we plan to have all APIs nova is using merged in the OpenStack SDK by the bobcat cycle. - AIs: - Manila team to write a spec describing the behavior of the locking/unlocking API for share. - Manila team to focus on having all APIs nova is using in OpenStack SDK by the Bobcat cycle. - *Scalable NFS Ganesha for CephFS - state of the kitchen* - We have shared a status of what was our focus during the previous release and some important bugs that have been fixed. - Updates on moving from a standalone ganesha to a cephadm deployed: - There are three changes being worked on at the moment, after past discussions: - Ensure that the shares access rules will be applied when moving from the old to the new server - Ensure that the share exists in both old and new ganesha servers and to reapply all access rules in both - Change to represent export locations from both old and new export locations while both cephfs_nfs_cluster_id and ganesha_server_ips are present in the backend. - With these three changes we would cover all the scenarios considering release upgrades. - The changes are being tested and need feedback. - A job for automated testing is in progress. We are targeting to have a multi-node CI job that will be capable of deploying ganesha using cephadm and run tests using it. - AI: carloss and gouthamr will pursue closure for the three changes related to the upgrades and ashrodri will continue to work on the multi-node testing job. - *SQLAlchemy 2.0 and other DB challenges/changes* - DB testing is failing sporadically in the CI and we were discussing possible changes to avoid rechecks caused by sporadic failures. - We intend to work on missing indexes for queries, so we can gain performance in the database. - Implementing __repr__ to manila models - Other people are already doing this and this could be of huge help for debugging, as one of the operators suggested in the session - We will investigate and apply this in Manila - AIs: gouthamr will open bugs for tracking the work on the joinedloads and a blueprint for the missing indexes. - Sqlalchemy 2.0 changes have already started and we have a list of changes yet to merge. stephenfin is doing a great work on the changes. - AIs: manila team to review the changes proposed by stephenfin - *Create share from replicated snapshot / keep source share on replica promote* - Users of this feature are looking for a way to test the inactive replicas before actually promoting them and if everything is correct, they continue with the promotion. - One suggestion to get this working would be having a replication pause API, which would make the inactive instance able to be mounted - We do think this feature could be relevant and some suggestions from the Manila team are: - Letting users know if they can pause/unpause replication even before they create their share replicas - Making users able to know about errors in the API (something similar to the check mechanism on security-service-update) - AIs: carthaca and team will work on a spec and will discuss it further with the Manila team. - *Tech debt:* - We brought up the current issues with our CI, mostly caused by tests being very resource intensive. - We discussed approaches for scenario testing and thought it is worth it trying to create containers to mount share during scenario tests instead of spawning VMs. - Also, the audience has mentioned that it is difficult to keep track of all of these issues and it would be better if we had them documented somewhere, and that we could bring these up more often. - Migration to Ubuntu 22 is yet to complete for some jobs due to failures in package installation and the usage of quagga. - AIs: - carloss will work on a wiki with tech debt items and will bring this up more often in the manila weekly meetings, so if people are available in that time frame to work on them, they will have more chances to raise their hands. - carloss and gouthamr will work to fix the issues with jobs yet to migrate to Ubuntu 22 - *Scheduler data placement based on vendor specific tool* - NetApp has a tool called Active IQ that uses machine learning to identify what is the best pool of disks to place a request based on custom parameters (latency, iops and so on) - They are thinking of trying to have this tool interact with Manila, so it would also benefit from their tool when NetApp is the only storage vendor in the cloud. - The suggestion is to have a new weigher that will make calls to the storage based on the request and use Active IQ response to identify where the share should be placed. - Testing this feature could be a challenge, considering that it's not straightforward to foresee what a weigher would return in a given moment - AIs: felipe_rodriguess alongside the NetApp team will work on a spec so this can be further discussed. [0] https://etherpad.opendev.org/p/manila-bobcat-ptg [1] https://www.youtube.com/watch?v=gefbYrzRD04&list=PLnpzT0InFrqDW_djitgCiFzwh… Thank you! carloss

1 0

[neutron] bug deputy report March 27th - April 2nd
by Miguel Lavalle 04 Apr '23

04 Apr '23

Here's last week's bugs deputy report: Medium ====== https://bugs.launchpad.net/neutron/+bug/2012944 neutron-remove-duplicated-port-bindings does not write to stdout. Valid bug. Assigned to mlavalle https://bugs.launchpad.net/neutron/+bug/2013326 Trying to activate port binding as regular user causes error 500. Assigned to slaweq Low: ========== https://bugs.launchpad.net/neutron/+bug/2012912 [neutron-lib][stable] Wallaby CI n-t-p, test "test_qos_dscp_create_and_update" failing. This bug is addressed by https://review.opendev.org/c/openstack/neutron-lib/+/879441. No further action seems likely RFEs ==== https://bugs.launchpad.net/neutron/+bug/2013228 [rfe] Non-admin users unable to create SR-IOV switch dev ports. Discussed during the PTG during the Neutron / Nova session, pending further discussion in drivers meeting https://bugs.launchpad.net/neutron/+bug/2013540 [RFE] Add support for Napatech LinkVirt SmartNICs. Discussed during the PTG, pending further discussion in drivers meeting

1 0

OpenInfra Live - April 6 at 9 a.m. CT / 14:00 UTC
by Kristin Barrientos 04 Apr '23

04 Apr '23

Hi everyone, This week’s OpenInfra Live episode is brought to you by the OpenStack Large Scale SIG. Episode: Large Scale Ops Deep Dive: Société Générale In the "Large Scale Ops Deep Dive" series, a panel of OpenStack operators invites special guests to talk about their deployment and discuss their operations. For this episode, our guest will be Guillaume Allard from Société Générale, one of France’s largest banks. Speakers: Guillaume Allard, Arnaud Morin, Thierry Carrez, Belmiro Moreira, Mohammed Naser and Kristin Barrientos Date and time: April 6, 2023, at 9 a.m. CT (14:00 UTC) You can watch us live on: YouTube: https://www.youtube.com/live/ZIjpEZSu1gw?feature=share LinkedIn: https://www.linkedin.com/video/event/urn:li:ugcPost:7043641363350908928/ WeChat: recording will be posted on OpenStack WeChat after the live stream Have an idea for a future episode? Share it now at ideas.openinfra.live <https://openinfrafoundation.formstack.com/forms/openinfralive>. Thanks, Kristin Barrientos Marketing Coordinator OpenInfra Foundation

1 0

[horizon][domain][yoga][kolla-ansible] horizon display failed to display domain tab
by James Leong 04 Apr '23

04 Apr '23

Hi all, I am playing around with the domain. I have used kolla-ansible as a deployment tool for deploying the yoga version of OpenStack. However, after deploying a fresh OpenStack, I was not able to view the domain tab on my dashboard. Every time when I want to create a new domain, I have to go to my terminal and create it manually without any GUI. Is there any other configuration I have to enable in the globals.yml file other than setting "horizon_keystone_multidomain: "True"? Thanks for your help James

3 4

[policy][rbac] RBAC 2023.2 Bobcat cycle vPTG discussions summary
by Ghanshyam Mann 04 Apr '23

04 Apr '23

Hello Everyone, We discussed the RBAC goal on Tuesday. I am summarizing the discussion here. Goal document: https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rb… Tracking: https://etherpad.opendev.org/p/rbac-goal-tracking Current progress: ============= Phase-1 (project personas and drop system scope): ------------------------------------------------------------ Projects completed: * Nova * Neutron * Glance * Manila * Ironic (no change needed) * Octavia * Placement * Service that completed Phase 1 in Zed and enabled scope and new defaults by default (enforce_scope=True & enforce_new_defaults=True by default): ** Nova ** Glance Projects in progress: * Cinder (almost completed. not using system scope, but have not added scope=['project'] to the default rules (currently, no scope is specified) * Magnum ** https://review.opendev.org/c/openstack/magnum/+/874945 * Tacker ** https://review.opendev.org/q/topic:bp%252Fimplement-project-personas Pending work (for phase-2|3): * Keystone implements a new default role called manager: ** https://review.opendev.org/c/openstack/keystone/+/822601 * Keystone implements a new default role called service: ** https://review.opendev.org/c/openstack/keystone/+/863420 Phase-2 (service role): -------------------------- In-progress: * Keystone: ** bootstrap support for servicerole: *** https://review.opendev.org/c/openstack/keystone/+/863420 ** bootstrap support for manager role: *** https://review.opendev.org/c/openstack/keystone/+/822601 *Nova ** https://review.opendev.org/c/openstack/nova/+/864594 Other discussion: ============= *Service role We discussed the service role and how the policy will add the service role. It is correct to add service as well as user role as default if that API is supposed to be called by the service as well as the user role. For example, if Manila is talking to Nova, Cinder, or Neutron via APIs, it needs to use a service role to interact, and Nova, Cinder, and Neutron can update such API rules to allow for service roles also. * Update the goal timeline for removing deprecated rules as per the SLURP release. ** Need at least 1 SLURP release between enabling the new default and removal. ** Action: gmann to update this in the goal document. *Testing: ** Manila has a lot of tests and running on the stable release ** Tempest and devstack ready to implement the test ** The current job with nova, cinder, neutron, glance with the scope and new default enable *** https://zuul.openstack.org/builds?job_name=tempest-full-enforce-scope-new-d… * Tempest now has a project reader/member same project_id * Related sessions: ** Tacker *** Title: Secure RBAC: Implement support of project-personas in Tacker [Continue from Antelope release] (manpreetk) *** Etherpad: https://etherpad.opendev.org/p/tacker-bobcat-ptg#L136 ** Glance *** Title: Secure RBAC *** Etherpad: https://etherpad.opendev.org/p/glance-bobcat-ptg#L53 ** Neutron *** Title: (slaweq) Secure RBAC - phase 2 description and review of the existing API calls *** Etherpad: https://etherpad.opendev.org/p/neutron-bobcat-ptg#L358 <feel free to add here if I missed any related sessions> I will continue holding the biweekly meeting to discuss progress and any query on RBAC. - https://wiki.openstack.org/wiki/Consistent_and_Secure_Default_Policies_Popu… -gmann

1 0

[keystone] Weekly Meeting Cancelled (04-Apr 2023)
by Dave Wilde 04 Apr '23

04 Apr '23

Hi folks, I'm cancelling the Keystone weekly meeting this week (04-Apr 2023) due to a conflict that I have. Please reach out to me on IRC or via email if you need anything. Thanks! /Dave

1 0

[neutron][ovn-bgp-agent] Make ovn-bgp-agent a Neutron stadium project
by Rodolfo Alonso Hernandez 04 Apr '23

04 Apr '23

Hello all: During the last PTG, it was discussed and accepted to migrate the ovn-bgp-agent project [1] to the Neutron stadium umbrella. This mail is a heads-up for anyone using it. In the next days (maybe today), Lucas will send the corresponding patches for "governance" and "project-config" repositories in order to start this migration. If you have any questions, please ask them in this email or via IRC in the #openstack-neutron channel. Regards. [1]https://opendev.org/x/ovn-bgp-agent

1 0

[nova][cinder] Providing ephemeral storage to instances - Cinder or Nova
by Christian Rohmann 04 Apr '23

04 Apr '23

Hello OpenStack-discuss, I am currently looking into how one can provide fast ephemeral storage (backed by local NVME drives) to instances. There seem to be two approaches and I would love to double-check my thoughts and assumptions. 1) *Via Nova* instance storage and the configurable "ephemeral" volume for a flavor a) We currently use Ceph RBD als image_type (https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.im…) so instance images are stored in Ceph, not locally on disk. I believe this setting will also cause ephemeral volumes (destination_local) to be placed on a RBD and not /var/lib/nova/instances? Or is there a setting to set a different backend for local block devices providing "ephemeral" storage? So RBD for the root disk and a local LVM VG for ephemeral? b) Will an ephemeral volume also be migrated when the instance is shutoff as with live-migration? Or will there be an new volume created on the target host? I am asking because I want to avoid syncing 500G or 1T when it's only "ephemeral" and the instance will not expect any data on it on the next boot. c) Is the size of the ephemeral storage for flavors a fixed size or just the upper bound for users? So if I limit this to 1T, will such a flavor always provision a block device with his size? I suppose using LVM this will be thin provisioned anyways? 2) *Via Cinder*, running cinder-volume on each compute node to provide a volume type "ephemeral", using e.g. the LVM driver a) While not really "ephemeral" and bound to the instance lifecycle, this would allow users to provision ephemeral volume just as they need them. I suppose I could use backend specific quotas (https://docs.openstack.org/cinder/latest/cli/cli-cinder-quotas.html#view-bl…) to limit the number of size of such volumes? b) Do I need to use the instance locality filter (https://docs.openstack.org/cinder/latest/contributor/api/cinder.scheduler.f…) then? c) Since a volume will always be bound to a certain host, I suppose this will cause side-effects to instance scheduling? With the volume remaining after an instance has been destroyed (beating the purpose of it being "ephemeral") I suppose any other instance attaching this volume will be scheduling on this very machine? Is there any way around this? Maybe a driver setting to have such volumes "self-destroy" if they are not attached anymore? d) Same question as with Nova: What happens when an instance is live-migrated? Maybe others also have this use case and you can share your solution(s)? Thanks and with regards Christian

3 4

[Neutron] CI meeting 04.04.2023 cancelled
by Slawek Kaplonski 04 Apr '23

04 Apr '23

Hi, I have overlapping internal meeting this week and will not be able to run Neutron CI meeting. So lets cancel it and meet next week. See You there. -- Slawek Kaplonski Principal Software Engineer Red Hat

1 0

[neutron] Neutron drivers meeting
by Rodolfo Alonso Hernandez 04 Apr '23

04 Apr '23

Hello Neutrinos: Next Neutron drivers meeting (April 7) will be cancelled. This is Easter week and most people will have Friday as a holiday. See you next week.

1 0

[nova] 2023.2 Bobcat PTG summary
by Sylvain Bauza 03 Apr '23

03 Apr '23

Hey folks, that's a wrap ! I'm happy to say we had a very productive PTG week with around more than 12 contributors each day. It was nice to see new faces. As a reminder, please don't open the main etherpad link [1] if you use a Web browser with an automatic translation modification feature. If you do it, then it would internally translate all the phrases directly in the etherpad as if you were directly modifying the etherpad by yourself. Please also make sure you don't accidentally remove or modify lines . In order to prevent any misusage, this etherpad is a read-only copy of the main one : https://etherpad.opendev.org/p/r.22d95e20f7133350dcad573c015ed7da Last point but not the least, I'm a human who usually writes bugs [2], including when it's about writing a report of the past week's discussions and the actions taken. Don't feel offended or irritated if you read silly notes of the very beloved chat you had, please rather fix it the opensource way by correcting my errors with a reply on that thread. Enough chat, please grab a coffee [3] (as it's a quite verbose email) and jump straight on below. ### Operator hour ### (specific and distinct read-only etherpad here : https://etherpad.opendev.org/p/r.f94e6dda608450b6a4a5c9afc76e8a88 ) Yet again, we missed a few more operators in this room (at least we had 3 of them, thanks for those that were there !). I'm still wondering why they are not joining a free session where they can discuss their concerns or some wanted features. Anyway, that's what we discussed : - for the moment, no operators have tried to use the Yoga unified limits feature. - operators are happy with the new default RBAC scopes. We could create a small and quick admin-guide section for explaining that and pointing to the oslo.policy documentation. - operators would love to use the (possibly) new virtiofs feature from Bobcat for Manila shares (if we are able to merge it). That being said, they think they couldn't provide it for their environments until Nova can live-migrate the instances using the shares (which is not possible as for the moment, even QEMU doesn't support it) - we also discussed a pain point from an operator and then we asked him to provide a bug report (related to the vm state continuing to be ACTIVE after shelving). For the moment, we can't reproduce the problem. ### Cross-project discussions ### ## Manila/Nova ### # Prevent shares deletion while it's attached to an instance - Nova could add some traits to reflect the share type. - the Manila team will create a manila spec for adding a service-level (or admin-level) lock API with a 'locked-by' parameter (or using service tokens if set) ### Neutron/Nova ### Read Neutron summary email [4] if you want more details. - We agreed on a delete_on_termination feature for ports. We need to see if we can use some already existing Neutron API (tags) or if we need a whole new API. - We are quite OK with the Napatech LinkVirt SmartNics feature, but we need to review the spec and we also want some third-party CI support for it. - the Neutron team accepted to return a HTTP409 Conflict return code if a duplicated port binding action is done by Nova and the Nova team agreed on handling it. - About the SRIOV port tracking in Placement, the Nova team needs to help the spec owner on how to work on it as it's a large and difficult design (but we agreed on the first part of the design). ### Glance/Nova ### Glance wants to add a new API for Image Direct URL access. Nova was quite OK with this new API, but the Nova team was having some concerns about upgrades and client calls to the new API. The Manila team agreed to create a nova spec for discussing those. ### Cinder/Nova ### - The Nova team is quite OK with the usecase for NFV encryption. Since the design is quite simple, maybe this could be a specless feature, just using traits for telling to Placement which computes support NFV encryption. We're OK to test this with cinder NFV jobs and maybe also testing it with a Nova periodic job depending on the implementation. - The Cinder team agreed on using its volume metadata API for tracking the cinder volumes hardware models. ### Horizon/Nova ### Horizon wanted to call the Placement API. We agreed on using the openstackSDK for this as Placement doesn't have python bindings directly. Some new methods may be added against the SDK for calling specific Placement APIs if we think we may need those (instead of directly calling Placement with the standard HTTP verbs and the API URIs) ### Nova-specific topics ### # Antelope retrospective and process-related discussions Shorter cycle than Zed, 6 features landed (same as Zed), 46 contributors (down by 8 compared to Zed), 27 bugfixes (vs. 54 for Zed). We were happy to not have a RC2 for the first time in the Nova history. The bug triage backlog kept small. - We want to discuss on a better review cadence to not look at all the implementations by the last weeks. - We want to get rid of Storyboard stories in Placement. We found and agreed on a way forward. - We will set 5-years-old-and-more bug reports as Incomplete, so bug reporters have 90 days to confirm that the bug is still present on master. - We will add more contrib documentation for explaining more how to find Gerrit changes for one-off reviewers and how to create some changes - We agreed on no longer automatically accepting an already-approved spec from a previous cycle if no code is still not existing. - All efforts on porting our client calls to the SDK should be tagged in Launchpad with the 'sdk' tag. - A Bobcat schedule will be proposed on next Tuesday weekly meeting with feature and spec deadlines + some review days. - We also discussed what we should discuss in the Vancouver PTG. # CI failures continuous improvement - We want to test some alternative image but cirros (alpine as a potential alternative) with a small footprint in some specific nova jobs - We need to continue investigating on the volume detach/attach failures. We may create a specific job that would do serialized volume checks as a canary job for forcing the job to fail more frequently in order to dig down further. - (related) Sylvain will propose a Vancouver PTG cross-project session for a CI debugging war room experience. # The new release cadence We had long arguments whether we should hold deprecations and removals this cycle, given the releasenotes tooling isn't in place yet. As a reminder, since Bobcat is a non-SLURP release, operators are able to skip it and just look at the C releasenotes so we want to make sure we forward-port upgrade notes for them. For the moment, no consensus in place, we defer the outcome into a subsequent nova meeting and in the meantime, we raised the point to the TC for guidance needs. # The problem with hard affinity group policies - We agreed on the fact that hard affinity/anti-affinity policies aren't ideal (operators, if you read me, please prefer soft affinity/anti-affinity policies for various reasons I won't explain here) but since we support those policies and the use case is quite understandable, we need to find a way to unblock instances that can't be easily-migrated from those kind of groups (eg. try to move your instances off a host with hard Affinity between them, good luck) - A consensus has been reached to propose a new parameter for live-migrate, evacuate and migrate that would only skip the hard affinity checks. A backlog spec will be written capturing this consensus. - Since a group policy may be violated, we'll also propose a new server group API modification that would show if a group has its policy violated. # Unified limits next steps - We agreed on the fact we should move on and enable by default unified limits in a close future, but we first need to provide a bit of tooling to help the migration. - tool could be a nova-manage command that would provide the existing quota limits as a yaml file so we could inject those into keystone (to be further defined during Bobcat) - a nova-status upgrade check may be nice for yelling if operators forgot to define their limits in keystone before upgrading to the release (potentially C) that would default to unified limits - testing in the grenade job is a feat. # Frequently written objects may lead to exhaustion of DB primary keys - We agreed on having all new primary keys to have BigInteger as data type. Existing keys should use BigInt but there is a huge upgrade impact to mitigate. - Operators will be sollicited to know their preferences between a costly-at-once data migration vs. a rolling data online migration for existing tables. - We will document which DB tables are safe to reindex (since their PKs isn't normalized as a FK somewhere else) - A backlog spec will capture all of the above # Leftover volumes relationship when you directly call Cinder instead of Nova for detaching - Yup, this is a known issue and we should discuss this with the Cinder team (unfortunately, we didn't did it this week) - We maybe should have a nova-manage script that would cleanup the BDMs but we're afraid of leaking some volume residues in the compute OS - Anyway, we need to understand the main scope and we also need to discuss about the preconditions ('when' should we call this tool and 'what' this tool should be doing ?). Probably a spec. # Misc but not the leasc (heh) - we will fill a rfe bug for lazy-loading instance name from system metadata or update the nested field in the instance object (no additional DB write) - we will add a new policy for cold-migrate specifically when a host is passed as a parameter (admin-only as a default) - We agreed on continuing the effort on compute node hostname robustification (the Bobcat spec seems not controversial) - We agreed on a few things about the server show command and the use of the SDK. Some existing OSC patch may require a bit of rework but it would cover most of the concerns we discussed. - we should fix the post-copy issue when you want to live-migrate a paused instance by removing the post-copy migration flag if so. - we could enable virtio-blk trim support by default. It's both a bugfix (we want to remove an exception) and a new feature (we want to enable it by default) so we'll discuss whether we need a specless blueprint by a next meeting. - we also discuss about a Generic VDPA support feature for Nova, and we asked the owner to provide us a spec for explaining the usecase. ### That's it. ### Yet again, I'm impressed. If you read that phrase, you're done with reading. I just hope your coffee was good and the time you took reading that email was worth it. If you have questions or want to reach me directly on IRC, that's simple : I'm bauzas on #openstack-nova. HTH, -Sylvain (on behalf of the whole Nova community) [1] https://etherpad.opendev.org/p/nova-bobcat-pt g (please remove the empty char between 'pt' and 'g') [2] Unfortunately, I'm very good at creating bugs as I discovered that based on my 10-year unfortunate contributions to OpenStack. [3] Or a tea, or whatever you prefer. [4] https://lists.openstack.org/pipermail/openstack-discuss/2023-April/033115.h…

1 1

[kolla-ansible][zed][15.1.0] Horizon failed to load trunk and QoS
by Stefan Bryzga 03 Apr '23

03 Apr '23

Hi all, Some time ago I deployed a small cluster with kolla-ansible v15.1.0. Before deployment I enabled neutron_qos and neutron_trunk flags in /etc/kolla/globals.yml. Both functions work fine via cli and I can create new entries via dashboard however horizon won't display created trunks/qos on the page (It shows "No items to display"). In web browser console I get this error while loading page: TypeError: Cannot read properties of undefined (reading 'data') at addTrackBy (output.0a9483334bbd.js:1115:59) at processQueue (output.43a09e34a303.js:1581:786) at output.43a09e34a303.js:1585:110 at Scope.$digest (output.43a09e34a303.js:1636:200) at Scope.$apply (output.43a09e34a303.js:1643:269) at done (output.43a09e34a303.js:1356:123) at completeRequest (output.43a09e34a303.js:1375:20) at XMLHttpRequest.requestLoaded (output.43a09e34a303.js:1368:1) 'Possibly unhandled rejection: {}' Does anyone knows how to fix this? Should I modify any configs to display new entries in horizon? Best Regards, Stefan

1 0